Back in 2002-2004, I was part of the team that redesigned a lot of code for Zope 3. The publisher was one of the first things we redesigned. While I’m sure ZPublisher started out pretty clean when it was first coded, over the years it had grown massive special cases and odd dependencies. For Zope 3, we reduced the publisher to something clean again and provided a way for it to grow as requirements grew.
Unfortunately, it seems the growth strategy did not work. The publisher I see now in Zope 3 is spread out among at least 3 packages (zope.publisher, zope.app.publisher, zope.app.publication) and it’s quite a tangle. Sure there are interfaces, but a lot of code calls undocumented methods. It especially saddens me that the publisher implements its own version of object traversal rather than use zope.traversing.
We can’t let this code languish. Remember what the P in Zope stands for? We have to get this right.
So what went wrong? Perhaps we didn’t provide the right kind of extensibility. We provided a series of documented hooks (the IPublication interface) that applications like Zope could easily override. Wait, did I say “applications like Zope”? Over the past 5 years, most Zope development has focused on building solid libraries with minimal dependencies, rather than building another application like Zope 2. The audience has changed, so the requirements have changed.
I think zope.publisher is a classical framework with all the classical problems of a framework. I suppose we could try to solve the problem with another cleanup. We could mash together zope.publisher, zope.app.publisher, and zope.app.publication in one package, which would untangle a lot, but that package would probably have a lot of unnecessary dependencies. Then we could factor stuff out of that new package to reduce the dependencies. After all that work, though, the publisher would still be a framework. I expect that it would yet again accumulate special cases quickly.
The Repoze team may have provided a different solution to our publisher woes. They have built a system based on WSGI filter pipelines. I’ve looked at Repoze and I’m very intrigued. For example, if you want your application to support Zope-style transactions, all you have to do is include a transaction manager in the pipeline; no extra baggage will come along, now that the transaction package has been split from ZODB. It seems like most functions of the current publisher could be rebuilt on this design. This is a different kind of extensibility that could control cruft accumulation much better.
So, should we replace zope.publisher with Repoze packages? How would others feel about that, particularly the Repoze team? It’s really fortunate that the Repoze packages are BSD licensed rather than GPL, otherwise I would not even consider this.
This is not yet a proposal. This is just a little discussion that happens to span the planet. The next step, if this idea doesn’t get shot down, is to create a more formal proposal. The proposal would specify which Zope release should deprecate the current publisher packages. (3.5? 3.6? I don’t know.)
For the Plone project we already decided to ditch the current (in our case ZPublisher) and go straight to the repoze version. I’m not sure if and when such a move could be the right one for either Zope2 or Zope3 itself, though.
I really like the Repoze approach as well. The question is whether we need to officially bless it in a Zope release. Zope 3 releases are dead IMHO. We should just make it easy for people to cobble the right eggs together and have a Zopeish system running on top of the Repoze publisher. We’re probably not too far away from this.
By the way, you mention zope.traversing. It has a very unfortunate name. Its name should really be zope.pathresolve or something like that, because that’s what it does: resolving object paths, such as the ones you have in TALES. It’s by no means a URL traversal machinery. This lives (rightfully!) in zope.app.publication.
The most significant differences you should know about when considering whether to replace Zope2’s ZPublisher with repoze.zope2 are:
– standard_error_message doesn’t work (you need to wire up a WSGI exception handler instead)
– error_log doesn’t work (replaced via repoze.errorlog middleware)
– zope2 stream iterators may or may not work (it’s possible under repoze to return any iterable, like a file handle, instead)
We’d of course be fine with it becoming the default Zope2 publisher; I think Plone’s usage of it will make this more palatable.
As far as Z3 goes, we’ve more or less ditched it for repoze.bfg, which implements its own (much simpler) traversal scheme for publishing, so I don’t know that we have the fortitude to help much there.
The repoze.grok package already does the minimal bit of sanding to make a Grok
app run atop the Repoze stack:
http://svn.repoze.org/repoze.grok/
In essence, the only real requirement is to suppress zope.publisher’s attempt to
do error logging / handling:
– The package supplyies a no-op IErrorReportingUtility (repoze.grok.bbb.dumbErrorReporting)
to be used when error logging is handled by WSGI middleware.
– The package also provides middleware which sets the ‘wsgi.handleErrors’ key in the
WSGI environment, disabling Z3’s own internal handlers.
With those pieces configured, pretty much any Z3 app should run (although I haven’t
tested this with recent versions of the Z3 packages).
I’ve been thinking about this topic for a while.
Can we really conclude that the current framework is that bad before we got a good look at it? I actually never really have, as it’s so scattered it’s hard to get good grasp on it. So I’m actually in favor of trying out the mashing up the various publisher related packages into a single place. This would preserve backwards compatibility, which is important as we likely have a ton of code that depends on this stuff. I’m also quite convinced it’d reduce dependencies quite a bit. Finally I’m not convinced that we’ll go through another period of special casing after that – after all the current publisher has been more or less stable for quite a while.
Could a repoze-based publisher be devised that’s backwards compatible enough not to give us an enormous headache? It’d need to be a combination of a powerful publisher, a lot of porting of code, and a lot of documentation that instructs people how to port things over. That’s potentially a lot more work than mashing together what we have now and harder to accomplish in an evolutionary fashion.
So while that may be the destination, I’d like this possibly intermediary step of a mashed-up publisher to be actually taken first.
Anyway, I may be wrong. To make me happier with such a step we’d need:
* a clear indication we have committed volunteers to do the work
* a pretty good answer for how to evolve codebases forward somehow
Philip: There is a lot of duplication between zope.traversing and the publisher packages. I really think the publisher should be using the same path resolution machinery as everything else uses (which might mean we need to expand the path resolution machinery a bit). I can’t think of a good reason why it should be different.
Chris and Tres: It’s interesting that Repoze suppresses the error reporting mechanism. That might not be a good choice for the wider Zope audience.
Martijn: I wouldn’t mind starting the mashup myself. In fact, I’m starting to see how a reunited publisher might incorporate Repoze concepts. I would work to get others involved after I’ve started, I think.
I’d love to see you tackle the mashup project, Shane. I have been thinking about starting such a project (with an eye on possible rewrites too) for a while, and talking to Tres about the much simpler Repoze publisher also makes the publisher an attractive topic. It fits quite well with the whole dependency minimization project too.
Concerning ZPT path traversal versus publishing traversal, there’s one difference I can think of; path traversal allows traversing to attributes and publishing traversal doesn’t ordinarily do that. But perhaps that’s a superficial difference. I do agree it’d be nice if the two were unified. On the other hand over time I’ve become less and less a fan of path traversal in ZPT. People are attracted to it as it supposedly keeps templates simpler, but on the other hand path traversal can still have arbitrary side effects and is (to me) sometimes a bit unpredictable compared to Python. A large part of the reason it actually feels more simple is because you don’t have to to type “python:” so it’s shorter, but that’s of course an illusion. I think if we just made ‘python:’ the default we might have a simpler world; we could then look at actually pure push templates for the set of use cases where you want your templating language to be limited. But that’s a digression from the main discussion. 🙂
Anyway, I’d be happy to be involved in such a project if I can be of any assistance.
FTR, the reason we don’t want Zope to catch all errors is generally because it’s useful (and sometimes necessary) to have some exceptions bubble up to middleware. For example, we have a repoze.retry middleware which catches (by default) ZODB’s ConflictError and retries the request. It’d of course be possible to only allow certain exceptions to escape, I think, and still retain some control by way of Zope exception views for most errors.
OTOH, it’s awful nice to be able to use the current crop of WSGI exception handling middleware during development, for example http://turbogears.org/2.0/docs/modules/thirdparty/weberror.html . And most of the error cases that do escape Zope in production are usually 500 errors; it’s not too painful to theme just those within some middleware handler in general; they shouldn’t be happening anyway usually.
@martijn: -re- making “python:” default, then repoze.bfg does just that when it uses ZPT; in fact, there is no path: expression.
Matlhe, cool to hear that. We should explore a configuration to do the same in Grok at some point.