The most important change in my working practices in recent months, making your life a little better

Over the past couple of years, we've seen a dramatic shift in the way Plone software (and Zope software, and, really, Python software in general) is packaged. Our code is now broken into dozens of packages, pulled in via a complex set of dependencies, and managed via powerful tools like Buildout.

However, the part of this whole story that excites me the most is the humble README.txt, combined with the practice of turning it into the "long description" in setup.py that appears on PyPI.

I didn't always appreciate this quite so much, but that little bit of documentation is the minimum you must do if you release something. It is not a substitute for more narrative documentation, which helps people learn and understand how the pieces fit together. However, the package README is the first line of defence for reference documentation. Once people have an idea about how your software fits together, they will find the package and expect to be told enough about to use it without having to resort to code archeology.

Let's look at some examples. First, there's the narrative "how does it all fit together" type documentation. This needs to be kept separate, and must be discoverable independently of code structure. A good example of mine is the Dexterity manuals.

Next, a bad example: The README for plone.theme. Ironically, I was trying to remember how to hook a browser layer up to a CMF theme (the raison d'etre of plone.theme) and got rather annoyed by the total lack of information in that README. I then realised who wrote the package. D'oh.

A better example is collective.beaker, a package for using the Beaker session and caching management framework in Zope 2. Notice how you both find out what the package is for, and how it is used. There are installation instructions, too, and hints on things like testing, where this package has some unusual implications.

More recently, I've started doing this very seriously, for example in the as-yet unreleased update to plone.app.registry and the new-and-exciting plone.app.caching. My coding cycle is now to do the interfaces module first, then code and tests in tandem, and then documentation. I normally start writing the README while waiting for test runs to complete. This cycle is then repeated, so that I update and revise the README as I refactor code. Finally, I read the rendered README through once as a sanity check before doing a release to PyPI.

Importantly, the documentation for the packages above has helped make them substantially better. By trying to explain how they worked, I found troubling inconsistencies and omissions that I'd failed to spot in my unit tests or interfaces.

Put differently, if something cannot be documented properly, it's crap. Find out for yourself before you foist it on everyone else, as Chris McDonough would no doubt agree with.

I'll leave you with some practical advice:

• Have a README.txt in your package and make sure it's being included as the long_description in setup.py. The standard ZopeSkel templates takes care of this for you.
• Make sure your README makes sense to someone other than yourself. Start by explaining what the package is for and why it exists. Mention briefly how to install it. Then explain the main use cases and how they are achieved. Even a few bullet points will go a long way. If your package is mostly user-facing, explain where people should expect to find its functionality in the UI. You can reference interfaces or code as necessary, but give people a fighting chance to see the big picture first.
• Your README is not a doctest. You can have doctests too, and if you genuinely believe they are useful to someone trying to understand your package, you can append them too to the long_description after the README. They must be separate files. Doctests are often good for low-level documentation (and sometimes more appropriate for really basic packages that have no end-user or integrator relevance). But they are not by themselves sufficient for documentation, and can be hugely off-putting by tricking the reader into thinking they can figure out how to use the package, when in fact it only shows contrived test cases based on unrealistic test setup. (Yes, I haven't always been of this persuasion, but I've seen it go wrong too many times).
• Make sure your README is valid reStructuredText. I dislike reST, but I've grown to tolerate it, and it's what PyPI renders. In TextMate, the reStructuredText bundle helps a lot, as I can now preview the rendered text with Cmd+Alt+Ctrl+P.
• Have a HISTORY.txt in the long_description too (the standard templates again set this up for you), and maintain a change log once you've released the package. I cannot overstate how important it is to have a clear picture of how a package has changed, and in which versions new features or bug fixes were introduced.

At EuroPython last year, Matt Hamilton gave a talk on using Zope Page Templates (ZPT) outside of Zope. ZPT uses the Template Attribute Language (TAL) to create dynamic templates that you can use in your own web applications, reporting frameworks, documentation systems or any other project.

Why TAL?

The point of this post isn't to go into detail about why you should use ZPT/TAL. Suffice to say that TAL:

• Makes well-formed XHTML easy
• Ensures that you close all elements and quote attributes
• Escapes all ampersands by default & -> &

The Django Templating Language:

<ul>
  {% for name in row %} 
    <li>{{name}}</li>
  {% endfor %}
</ul>

TAL:

<ul>
  <li tal:repeat="name row"
      tal:content="name">
    Dummy data
  </li>
</ul>

Using ZPT in your own project

There are three steps to using ZPT in your own project.

• Install ZPT (via the zope.pagetemplate package and its dependencies)
• Create a template file
• Render the template file using the data from your application

Install zope.pagetemplate

I recommend using virtualenv for each new application.

# virtualenv zptdemo
# cd zptdemo
# bin/easy_install zope.pagetemplate

Create a template

mytemplate.pt

<html> 
  <body> 
    <h1>Hello World</h1>
    <div tal:condition="python:foo == 'bar'">
      <ul>
        <li tal:repeat="item rows" tal:content="item" /> 
      </ul> 
    </div>
  </body>
</html>

Render the template

mycode.py

from zope.pagetemplate.pagetemplatefile \
    import PageTemplateFile

my_pt = PageTemplateFile('mytemplate.pt')
context = {'rows': ['apple', 'banana', 'carrot'], 
           'foo':'bar'}
print my_pt.pt_render(namespace=context)

And that's it. This will generate the following:

Hello World

• apple
• banana
• carrot

Google App Engine

This is all very well if you're able to install your own packages, but Google App Engine (GAE) doesn't allow you to do this. You can, however, include packages with your application, which is what we'll do here. When installing zope.pagetemplate into your virtual environment, you may have noticed that the following packages were installed:

• zope.pagetemplate
• zope.i18nmessageid
• zope.interface
• zope.tal
• zope.tales

We can extract all these files and put them into our GAE application.

I've already done this for you, and you can grab a copy of the 'zope' folder on GitHub here. Put this folder into the top level of your application.

Create a template as described above, and then place the Python code that renders the template into your application's RequestHandler. It should end up looking something like this:

class MainHandler(webapp.RequestHandler):

  def get(self):
    my_pt = PageTemplateFile('main.pt') 
    context = {'rows': ['apple', 'banana', 'carrot'], 
               'foo':'bar'}
    self.response.out.write(my_pt.pt_render(namespace=context))

At this point you have ZPT working in GAE and you can take advantage of all the features that ZPT and TAL provide.

The application on GitHub can be download, tested and even deployed straight to Google App Engine without modification. You can see an example of it running here.

update: it's also possible to use Chameleon on Google App Engine, which as Martin says may be easier and faster.

Chris McDonough has written a book on repoze.bfg! repoze.bfg, or simply BFG, is a web framework written in Python. It’s of Zope pedigree but borrows many ideas from Pylons and Django as well. It’s simple yet powerful, agile, fast, has close to 100% test coverage and is extremely well documented.

Most importantly, however, BFG is true to its motto “pay only for what you eat.” You can use SQLAlchemy, Zope’s object store ZODB, or some other persistence mechanism. You can use Zope-style object traversal, Routes-style URL mapping or a combination of both. You can use Zope Page Templates, Genshi (both through the extremely fast Chameleon engine), Jinja, or your favourite templating language. You can write your application in an extensible manner using declarative configuration, or you can just use the Python API. You can deploy on Google App Engine, Apache and mod_wsgi or anywhere else that supports WSGI. And you needn’t worry about learning about the stuff you’re not using.

BFG is well worth a look, and so is

I’ve enjoyed seeing some writeups on requests/second and memory usage for upcoming versions of Plone.  It’s great to see things trending in that direction.  Hopefully with some tough choices and deprecation, more gains can be made (just my personal opinion.)

I thought I’d give a primitive try at the same numbers for KARL, the collaboration application atop BFG that we’ve been working on and deploying to customers.

Using the ‘ab -n 100 -c 2′ on my first gen MacBook 2 GHz, 2 Gb of RAM, I leveled off at just over 134 requests per second.  Memory usage was 31 Mb.

Obviously it’s not an apples-apples comparison.  The feature set is smaller.  Although we do have cataloging, text search, workflow, security, and the like, there’s a ton of stuff we don’t do.  We’re an end-user application with specific features, versus a framework.

On the other hand, all requests in KARL are authenticated and fully-dynamic.  So the 137 rps above?  That’s our slow number: authenticated, personalized, security-aware, fully dynamic.

For more fun, we recently built an ugly, cheap Core i5 box in the Agendaless office for $600, with 4 Gb of RAM.  In production we deploy under modwsgi, so we fired it up to have 3 processes (for 3 of the four cores).  We also have a script that lets us bulk load 300 sample communities, each containing a bunch of content.

That’s a bit more realistic of a test, since we start paying the price of having content in the catalog.

In that “with content” test, we got 349 requests/second.

Sometime soon we’re going to think a bit harder about a more realistic test.  Pounding the same URL over and over as the same user just doesn’t mean squat.  Well, it’s valuable in so much as it is a veto: if your numbers are pathetically low on the fastest-possible “test”, it’s only going to get worse.  We are slowing building up some Funkload scripts that cover a scenario which includes different users, different activities, and some writes as well as reads.

We need this as we are evaluating various KARL ideas in 2010.  First and foremost, we bought a solid-state disk for the test box.  We had a query (prefix match on text search, where only one letter was entered) which blew up our system previously.  Think, 60+ seconds.  That time fell down to 2 with the SSD.

Next, we’d like to see some before/after on RelStorage using some real-world scenarios.  Finally, I’d like to see some before/after on repoze.pgtextindex, where we swap out just one of our catalog index types (the text one) with transactional text indexing in Postgresql.