Month: February 2009

Limits of zope.pipeline

I’m starting to get a sense of what publisher functionality I can put in a WSGI pipeline and what I shouldn’t.

The pipeline is very useful for specifying the order things should happen.  For example, the error handling should be as early in the pipeline as possible, so it can handle many kinds of errors, but it has to come after the pipeline element that opens and closes the root database connection.  Constraints like that have never been expressed clearly in the current publisher.

I was planning to encapsulate the <base> tag mangling logic in a simple pipeline step, but I’ve studied how it currently works and I realize now that WSGI doesn’t provide a good abstraction for the kind of heuristics Zope uses to makes the <base> tag logic fast.  I am considering several choices:

  1. Split the base tag handling between a pipeline element and a new adapter.
  2. Add short-lived output filter hooks to the response, similar to the traversal_hooks I added to requests, which I think turned out quite nice.
  3. Stick to the original plan, which might cause performance problems since Zope would then have to buffer potentially large output streams.

I need to choose the pattern that maximizes clarity for readers.  #1 and #2 are very similar.  #1 is less direct and thus more ambiguous than #2, but #1 is used more often in Zope code.

Anecdotal Evidence

I like to believe that I am a competent software developer in both Python and Java.  As a competent developer, I find that certain things are generally much easier than other things.

For instance, I just spent a frustrating week working out how to install a Shibboleth identity provider, yet I never got it working quite satisfactorily.  Then, after spending 30 minutes with Python-openid, I had an identity provider server running and I already felt quite confident with it.

It’s as if Java and Python are not really in the same league.  There is no way to prove that, though.  Sometimes Java wins anyway.

java.lang.IllegalStateException: No match found

That error means a Java regular expression did not match the input text.

If you get this while trying to install Shibboleth, it’s because the installer requires a host name with three parts, such as “shib.example.com”.  “localhost” and “localhost.localdomain” will not suffice.

zope.pipeline

I’ve been working on a new revision of the Zope publishing framework.  The goal is to make the publisher comprehensible.  Since I helped design the current publisher, I don’t mind saying that the current design really stinks.  We made it extensible in a way that breeds ravioli code.  I find it difficult to follow the sequence of interactions in the code without concentrating hard and memorizing a lot.  That is not acceptable.

Now that I’m working with Zope most of the time, I can try to clean up the mess.  I decided to try out a Repoze-like design that builds on a WSGI pipeline.  (WSGI did not exist at the time we invented the current zope.publisher.)  I’m trying hard to maintain compatibility with the current Zope publisher, so I’m copying from Zope rather than Repoze, but taking ideas from Repoze as much as possible.

I’ve been pretty happy with the new code, but it wasn’t until last night that I felt a click that told me this new design can work well.  The key is to use the Zope component architecture to build the pipeline dynamically.  In order to cope with all the strange things the publisher has to do, the system varies the pipeline according to the request type.  Now any developer should be able to easily plug in pipeline elements with ZCML.  People can also build a static pipeline if they want, but I think that would involve a fair amount of work to maintain across software upgrades.  I expect the standard dynamically configured pipeline to have 12 stages or more.

The best part is the new design seems to have fewer concepts than the old design.  IPublication is gone, along with publication factories; now we just have request factories.  Dependencies should be under control as well.

I’ve been checking the sketched code into my sandbox on svn.zope.org.  I hope to get the code working soon so I can get feedback on the new design.

Redesign of zope.publisher

Here are my current thoughts on what we should do with zope.publisher.

  • Most packages that depend on zope.publisher only use its interfaces and two base classes (BrowserView and BrowserPage).  Those packages don’t care about anything else that zope.publisher offers.  Therefore, to reduce dependency burdens and to make the zope.publisher package easier to explain, I think zope.publisher should be reduced to providing only interfaces and those two base classes.
  • The IPublication interface and all its implementations should die, to be replaced with a WSGI pipeline.  This should make the publication process dramatically easier to understand and customize.
  • Request and response objects should just hold information; they should have no interesting behavior such as traverse().
  • We should create and use two new keys in the WSGI environment, “zope.request” and “zope.response”.  These will implement at least IRequest and IResponse, respectively.  (“transaction” and “zope.interaction” are also possibilities.)
  • The WSGI-compatible pipeline should be capable of doing everything zope.app.publication currently does.  We should use Repoze packages in that pipeline where possible.
  • Most of the code in zope.app.publisher should be moved to a better-named package like zope.fileresource or something.  IMHO, the name “zope.app.publisher” conveys an understanding that does not match the contents of the package at all.
  • I think all of zope.app.publication and zope.app.http should be moved out of zope.app and into WSGI pipeline elements.  (This is less clear to me than the rest of the plan.)

How would others feel about that plan?  It’s a lot of changes, but it could be done in phases, and it seems like a big improvement.  I probably need to elaborate more.  It’s still crystalizing in my head.

Repozublisher

Back in 2002-2004, I was part of the team that redesigned a lot of code for Zope 3.  The publisher was one of the first things we redesigned.  While I’m sure ZPublisher started out pretty clean when it was first coded, over the years it had grown massive special cases and odd dependencies.  For Zope 3, we reduced the publisher to something clean again and provided a way for it to grow as requirements grew.

Unfortunately, it seems the growth strategy did not work.  The publisher I see now in Zope 3 is spread out among at least 3 packages (zope.publisher, zope.app.publisher, zope.app.publication) and it’s quite a tangle.  Sure there are interfaces, but a lot of code calls undocumented methods.  It especially saddens me that the publisher implements its own version of object traversal rather than use zope.traversing.

We can’t let this code languish.  Remember what the P in Zope stands for?  We have to get this right.

So what went wrong?  Perhaps we didn’t provide the right kind of extensibility.  We provided a series of documented hooks (the IPublication interface) that applications like Zope could easily override.  Wait, did I say “applications like Zope”?  Over the past 5 years, most Zope development has focused on building solid libraries with minimal dependencies, rather than building another application like Zope 2.  The audience has changed, so the requirements have changed.

I think zope.publisher is a classical framework with all the classical problems of a framework. I suppose we could try to solve the problem with another cleanup.  We could mash together zope.publisher, zope.app.publisher, and zope.app.publication in one package, which would untangle a lot, but that package would probably have a lot of unnecessary dependencies.  Then we could factor stuff out of that new package to reduce the dependencies.  After all that work, though, the publisher would still be a framework.  I expect that it would yet again accumulate special cases quickly.

The Repoze team may have provided a different solution to our publisher woes.  They have built a system based on WSGI filter pipelines.  I’ve looked at Repoze and I’m very intrigued.  For example, if you want your application to support Zope-style transactions, all you have to do is include a transaction manager in the pipeline; no extra baggage will come along, now that the transaction package has been split from ZODB.  It seems like most functions of the current publisher could be rebuilt on this design.  This is a different kind of extensibility that could control cruft accumulation much better.

So, should we replace zope.publisher with Repoze packages?  How would others feel about that, particularly the Repoze team?  It’s really fortunate that the Repoze packages are BSD licensed rather than GPL, otherwise I would not even consider this.

This is not yet a proposal.  This is just a little discussion that happens to span the planet.  The next step, if this idea doesn’t get shot down, is to create a more formal proposal.  The proposal would specify which Zope release should deprecate the current publisher packages.  (3.5? 3.6?  I don’t know.)