The Fastest WSGI Server for Zope

I have been planning to compare mod_wsgi with paste.httpserver, which Zope 3 uses by default.  I guessed the improvement would be small since parsing HTTP isn’t exactly computationally intensive.  Today I finally had a good chance to perform the test on a new linode virtual host.

The difference blew me away.  I couldn’t believe it at first, so I double-checked everything.  The results came out about the same every time, though:


I used the ab command to run this test, like so:

ab -n 1000 -c 8 http://localhost/

The requests are directed at a simple Zope page template with no dynamic content (yet), but the Zope publisher, security, and component architecture are all deeply involved.  The Paste HTTP server handles up to 276 requests per second, while a simple configuration of mod_wsgi handles up to 1476 per second.  Apparently, Graham‘s beautiful Apache module is over 5 times as fast for this workload.  Amazing!

Well, admittedly, no… it’s not actually amazing. I ran this test on a Xen guest that has access to 4 cores.  I configured mod_wsgi to run 4 processes, each with 1 thread. This mod_wsgi configuration has no lock contention.  The Paste HTTP server lets you run multiple threads, but not multiple processes, leading to enormous contention for Python’s global interpreter lock.  The Paste HTTP server is easier to get running, but it’s clearly not intended to compete with the likes of mod_wsgi for production use.

I confirmed this explanation by running “ab -n 1000 -c 1 http://localhost/”; in this case, both servers handled just under 400 requests per second.  Clearly, running multiple processes is a much better idea than running multiple threads, and with mod_wsgi, running multiple processes is now easy.  My instance of Zope 3 is running RelStorage 1.1.3 on MySQL.  (This also confirms that the MySQL connector in RelStorage can poll the database at least 1476 times per second.  That’s good to know, although even higher speeds should be attainable by enabling the memcached integration.)

I mostly followed the repoze.grok on mod_wsgi tutorial, except that I used zopeproject instead of Repoze or Grok.  The key ingredient is the WSGI script that hits my Zope application to handle requests.  Here is my WSGI script (sanitized):

# set up sys.path.
code = open('/opt/myapp/bin/myapp-ctl').read()
exec code

# load the app
from paste.deploy import loadapp
zope_app = loadapp('config:/opt/myapp/deploy.ini')

def application(environ, start_response):
    # translate the path
    path = environ['PATH_INFO']
    host = environ['SERVER_NAME']
    port = environ['SERVER_PORT']
    scheme = environ['wsgi.url_scheme']
    environ['PATH_INFO'] = (
        '/myapp/++vh++%s:%s:%s/++%s' % (scheme, host, port, path))
    # call Zope
    return zope_app(environ, start_response)

This script is mostly trivial, except that it modifies the PATH_INFO variable to map the root URL to a folder inside Zope. I’m sure the same path translation is possible with Apache rewrite rules, but this way is easier, I think.

How to Fix the MySQL Write Speed

Last time I ran the RelStorage performance tests, the write speed to a MySQL database appeared to be slow and getting slower.  I suspected, however, that all I needed to do was tune the database.  Today I changed some InnoDB configuration parameters from the defaults.  The simple changes solved the MySQL performance problem completely.

The new 10K chart, using RelStorage 1.1.3 on Debian Sid with Python 2.4 and the same hardware as before:

I added the following lines to my.cnf to get this speed:

innodb_data_file_path = ibdata1:10M:autoextend

This is similar to the configuration suggested by the InnoDB documentation for a 512 MB database server.  Even if you have a 16 GB server, I would suggest starting with the settings for a 512 MB server, then watch what happens to the RAM and CPU on the database server when you connect all of your client machines simultaneously.  You want to leave at least half the RAM available for disk cache and usage spikes.

Not all of these changes are related to speed.  The innodb_file_per_table option just seems like a good idea because it makes tables visible on the filesystem, which should improve manageability.  I think it might improve cache locality as well.

With these changes to my.cnf, ZEO, PostgreSQL, and MySQL all perform about the same for writes, with MySQL having a slight lead.  I suspect all three are hitting hardware and kernel limits.  I think the differences would be more pronounced on higher-end storage hardware.

A big caveat: It’s risky to change InnoDB settings unless you’re familiar with all the effects.  Some changes break compatibility with existing table data.  Get to know the InnoDB documentation very well before you change these settings, and make backups using mysqldump, as always.

Meanwhile, Oracle XE continues to write slowly and ZEO read performance is so bad that it’s off the chart.  I bet ZEO read performance could be improved with some simple optimizations somewhere, but I don’t have an incentive to fix that. 🙂  Perhaps it has been fixed in ZODB 3.9.

RelStorage Support

I am more than happy to support RelStorage as best I can by email.  Every time I do, however, I always get a nagging feeling that I could help RelStorage users a lot better if we set up a short term support contract.  I would very much appreciate a chance to optimize their system by testing the performance of different configurations.  When the communication is limited to email, neither of us gets a chance to discover how we might help each other better.

So if you’re a RelStorage user and your database is growing by tens of gigabytes, please seriously consider a short term support contract with my little company.  A little tuning or code revision in the right place could yield orders of magnitude performance gains.  I really want to help you directly.  Contact me at shane (at) willowrise (dot) com.

Limits of zope.pipeline

I’m starting to get a sense of what publisher functionality I can put in a WSGI pipeline and what I shouldn’t.

The pipeline is very useful for specifying the order things should happen.  For example, the error handling should be as early in the pipeline as possible, so it can handle many kinds of errors, but it has to come after the pipeline element that opens and closes the root database connection.  Constraints like that have never been expressed clearly in the current publisher.

I was planning to encapsulate the <base> tag mangling logic in a simple pipeline step, but I’ve studied how it currently works and I realize now that WSGI doesn’t provide a good abstraction for the kind of heuristics Zope uses to makes the <base> tag logic fast.  I am considering several choices:

  1. Split the base tag handling between a pipeline element and a new adapter.
  2. Add short-lived output filter hooks to the response, similar to the traversal_hooks I added to requests, which I think turned out quite nice.
  3. Stick to the original plan, which might cause performance problems since Zope would then have to buffer potentially large output streams.

I need to choose the pattern that maximizes clarity for readers.  #1 and #2 are very similar.  #1 is less direct and thus more ambiguous than #2, but #1 is used more often in Zope code.


I’ve been working on a new revision of the Zope publishing framework.  The goal is to make the publisher comprehensible.  Since I helped design the current publisher, I don’t mind saying that the current design really stinks.  We made it extensible in a way that breeds ravioli code.  I find it difficult to follow the sequence of interactions in the code without concentrating hard and memorizing a lot.  That is not acceptable.

Now that I’m working with Zope most of the time, I can try to clean up the mess.  I decided to try out a Repoze-like design that builds on a WSGI pipeline.  (WSGI did not exist at the time we invented the current zope.publisher.)  I’m trying hard to maintain compatibility with the current Zope publisher, so I’m copying from Zope rather than Repoze, but taking ideas from Repoze as much as possible.

I’ve been pretty happy with the new code, but it wasn’t until last night that I felt a click that told me this new design can work well.  The key is to use the Zope component architecture to build the pipeline dynamically.  In order to cope with all the strange things the publisher has to do, the system varies the pipeline according to the request type.  Now any developer should be able to easily plug in pipeline elements with ZCML.  People can also build a static pipeline if they want, but I think that would involve a fair amount of work to maintain across software upgrades.  I expect the standard dynamically configured pipeline to have 12 stages or more.

The best part is the new design seems to have fewer concepts than the old design.  IPublication is gone, along with publication factories; now we just have request factories.  Dependencies should be under control as well.

I’ve been checking the sketched code into my sandbox on  I hope to get the code working soon so I can get feedback on the new design.

Redesign of zope.publisher

Here are my current thoughts on what we should do with zope.publisher.

  • Most packages that depend on zope.publisher only use its interfaces and two base classes (BrowserView and BrowserPage).  Those packages don’t care about anything else that zope.publisher offers.  Therefore, to reduce dependency burdens and to make the zope.publisher package easier to explain, I think zope.publisher should be reduced to providing only interfaces and those two base classes.
  • The IPublication interface and all its implementations should die, to be replaced with a WSGI pipeline.  This should make the publication process dramatically easier to understand and customize.
  • Request and response objects should just hold information; they should have no interesting behavior such as traverse().
  • We should create and use two new keys in the WSGI environment, “zope.request” and “zope.response”.  These will implement at least IRequest and IResponse, respectively.  (“transaction” and “zope.interaction” are also possibilities.)
  • The WSGI-compatible pipeline should be capable of doing everything currently does.  We should use Repoze packages in that pipeline where possible.
  • Most of the code in should be moved to a better-named package like zope.fileresource or something.  IMHO, the name “” conveys an understanding that does not match the contents of the package at all.
  • I think all of and should be moved out of and into WSGI pipeline elements.  (This is less clear to me than the rest of the plan.)

How would others feel about that plan?  It’s a lot of changes, but it could be done in phases, and it seems like a big improvement.  I probably need to elaborate more.  It’s still crystalizing in my head.


Back in 2002-2004, I was part of the team that redesigned a lot of code for Zope 3.  The publisher was one of the first things we redesigned.  While I’m sure ZPublisher started out pretty clean when it was first coded, over the years it had grown massive special cases and odd dependencies.  For Zope 3, we reduced the publisher to something clean again and provided a way for it to grow as requirements grew.

Unfortunately, it seems the growth strategy did not work.  The publisher I see now in Zope 3 is spread out among at least 3 packages (zope.publisher,, and it’s quite a tangle.  Sure there are interfaces, but a lot of code calls undocumented methods.  It especially saddens me that the publisher implements its own version of object traversal rather than use zope.traversing.

We can’t let this code languish.  Remember what the P in Zope stands for?  We have to get this right.

So what went wrong?  Perhaps we didn’t provide the right kind of extensibility.  We provided a series of documented hooks (the IPublication interface) that applications like Zope could easily override.  Wait, did I say “applications like Zope”?  Over the past 5 years, most Zope development has focused on building solid libraries with minimal dependencies, rather than building another application like Zope 2.  The audience has changed, so the requirements have changed.

I think zope.publisher is a classical framework with all the classical problems of a framework. I suppose we could try to solve the problem with another cleanup.  We could mash together zope.publisher,, and in one package, which would untangle a lot, but that package would probably have a lot of unnecessary dependencies.  Then we could factor stuff out of that new package to reduce the dependencies.  After all that work, though, the publisher would still be a framework.  I expect that it would yet again accumulate special cases quickly.

The Repoze team may have provided a different solution to our publisher woes.  They have built a system based on WSGI filter pipelines.  I’ve looked at Repoze and I’m very intrigued.  For example, if you want your application to support Zope-style transactions, all you have to do is include a transaction manager in the pipeline; no extra baggage will come along, now that the transaction package has been split from ZODB.  It seems like most functions of the current publisher could be rebuilt on this design.  This is a different kind of extensibility that could control cruft accumulation much better.

So, should we replace zope.publisher with Repoze packages?  How would others feel about that, particularly the Repoze team?  It’s really fortunate that the Repoze packages are BSD licensed rather than GPL, otherwise I would not even consider this.

This is not yet a proposal.  This is just a little discussion that happens to span the planet.  The next step, if this idea doesn’t get shot down, is to create a more formal proposal.  The proposal would specify which Zope release should deprecate the current publisher packages.  (3.5? 3.6?  I don’t know.)

keas.pbpersist and keas.pbstate

Remember how I was talking about serializing data in ZODB using Google Protocol Buffers instead of pickles?  Well, Keas Inc. suggested the idea and asked me to work on it.  The first release of a package combination that implements the idea is ready:

  • keas.pbstate 0.1.1 (on PyPI), which helps you write classes that store all state in a Protocol Buffer message.
  • A patch for ZODB (available at that makes it possible to plug in serializers other than ZODB’s standard pickle format.
  • keas.pbpersist 0.1 (on PyPI), which registers a special serializer with ZODB so that ProtobufState objects get stored without any pickling.

This code is new, but the tests all pass and the coverage tests claim that every line of code is run by the tests at least once.  I hope these packages get used so we can find out their flaws.  I did my best to make keas.pbstate friendly, but there are some areas, object references in particular, where I don’t like the current syntax.  I don’t know if this code is fast–optimization would be premature!

I should mention that while working with protobuf, I got the feeling that C++ is the native language and Java and Python are second class citizens.  I wonder if I’d have the same feeling with other serialization formats.

RelStorage 1.1.2

This release has two new useful features, one for performance, one for safety.

The performance feature is that if you use both the cache-servers and poll-interval options, RelStorage will use the cache to distribute basic change notifications.  That means we get to lighten the load on the database using the poll-interval, yet changes should still be seen instantly on all clients.  Yay! 🙂

The only drawback I expect is that caching makes debugging more difficult.  Still, this option should help people build enormous clusters, like the one my current customer was planning to build, although I got word today that they have changed their mind.

The new safety feature is the pack-dry-run option, which lets you run only the nondestructive pre_pack phase to get a list of everything that would be deleted by the pack phase.  This should be particularly useful if you’re trying out packing for the first time on a big database.  My current customer would have benefited from this too.

I also fixed a bug that caused the pack code to not remove as much old stuff as it should and I started using PyPI instead of the wiki as the main web page.  Using PyPI means I have to maintain only one README, which gets translated automatically into the PyPI page.  Until now I’ve had to maintain both the README and the wiki page.

Patched ZODB3 Eggs Available

This week, I put up some ZODB3 eggs and source distributions with the patch required for RelStorage already applied. I built both ZODB3-3.8.1-polling and ZODB3-3.7.3-polling.  I even made eggs for Windows developers who have not yet taken the time to set up MinGW. 😉

Developers can use this web site in buildout.cfg to incorporate RelStorage in their applications.  Feel free to mirror the site if you need to.