I’ve been working on a new revision of the Zope publishing framework. The goal is to make the publisher comprehensible. Since I helped design the current publisher, I don’t mind saying that the current design really stinks. We made it extensible in a way that breeds ravioli code. I find it difficult to follow the sequence of interactions in the code without concentrating hard and memorizing a lot. That is not acceptable.
Now that I’m working with Zope most of the time, I can try to clean up the mess. I decided to try out a Repoze-like design that builds on a WSGI pipeline. (WSGI did not exist at the time we invented the current zope.publisher.) I’m trying hard to maintain compatibility with the current Zope publisher, so I’m copying from Zope rather than Repoze, but taking ideas from Repoze as much as possible.
I’ve been pretty happy with the new code, but it wasn’t until last night that I felt a click that told me this new design can work well. The key is to use the Zope component architecture to build the pipeline dynamically. In order to cope with all the strange things the publisher has to do, the system varies the pipeline according to the request type. Now any developer should be able to easily plug in pipeline elements with ZCML. People can also build a static pipeline if they want, but I think that would involve a fair amount of work to maintain across software upgrades. I expect the standard dynamically configured pipeline to have 12 stages or more.
The best part is the new design seems to have fewer concepts than the old design. IPublication is gone, along with publication factories; now we just have request factories. Dependencies should be under control as well.
I’ve been checking the sketched code into my sandbox on svn.zope.org. I hope to get the code working soon so I can get feedback on the new design.
Hi Shane, this is looking very exciting!
How useful do you think your new publisher will be as an independent package?
Alternatively, how many other packages will it pull in as dependencies?
Thank you!
The current plan is that very few packages will pull in zope.pipeline as a dependency; most packages will rely on zope.publisher instead. The zope.publisher package will contain interfaces and not much else.
The zope.pipeline package will contain many small WSGI middleware applications. It should be easy to break those out as needed.
TWELVE STAGES?
In other words: when can we expect the Zope Publisher + Plone combo to perform as fast as repoze.bfg performed in relation to the video about repoze.bfg that was posted on the Internets the other day?
Hi Shane,
I took a look at zope.pipeline. Nice!
I can see the utility of *choosing* a pipeline based on request type. However, I’m a little dubious of *constructing* a pipeline dynamically based on request type. I think it’s always useful for the user to be able to add and remove components from any pipeline statically, even if they have to make many pipelines to service a particular goal. I *think* this is the direction you’re headed.
I also hope that the middleware bits can get generalized and broken out into packages that are consumable by folks other than Zope people, and that non-zope.pipeline middleware can be used within zope.pipeline. This may influence how pipelines are constructed (some middleware bits require more configuration than a name).
Rudd-O: part of the idea behind this project is people will be able to reduce the pipeline to fit their needs, making it faster. I don’t know how much faster.
Chris: the zope.pipeline.entry module caches the pipeline by request type (it uses tuple(providedBy(request)) as the key), so it should be plenty fast. Static pipelines should be easy because everything in zope.pipeline.apps is intended to be usable directly by tools like Paste Deploy. We might break out some of those applications into their own packages or even just replace them with Repoze apps.
The “more configuration than a name” requirement is still bothering me a little, though. I haven’t yet figured out what I should do about that. I invented a solution but I’m not happy with it.
Rudd-O: Zope + Plone will never perfom at that level because they do a lot more than just rendering the website and adding dynamic bits, which is basically what the repoze.bfg video showed. The point of the video was that sometimes you don’t need all the extras that Plone gives you for the front end, but still want a great application for managing your content.
Shane: it’s great to see you working in Zope again!
Update on my last comment: Tonight I found a nice simplification that makes it easy to pass application constructor parameters. I also figured out I can move the pipeline cache out of module scope. Yay!
Hi Shane,
zope.pipeline is starting to look good! It looks like a really elegant way to compose a pipeline of endware.
Shane, great start and I’m really happy to see this work finally happened – the publisher has been on the “to overhaul” list for a while. I assume you’re going to focus on adding tests next, as there are none right now, right? Another thing I saw is that some apps pull in some dependencies such as zope.app.security that we will need to look at.
Once your design has stabilized the grok project will investigate creating a grokker to register pipeline stages. 🙂
Chris: “endware”, huh? Yeah, middleware isn’t quite the right name, but I’m not sure about endware either. Maybe it will grow on me.
Martijn: right, no tests yet, and I suspect there are even syntax errors. 🙂 Tests are next.
Hi Shane,
Great to see you working on Zope again!
I think I’d like to understand slightly better how this relates to the Repoze stuff. Conceptually, at least, there seems to be some overlap, though I understand the backwards compatibility question may make some things different. Still, middleware like repoze.who and repoze.tm2 sound like good candidates for re-use rather than re-invention. These are already used in the wild outside Zope-ish applications.
Martin
P.S.
We should get your blog onto planet.plone.org. If you want that, you can put in a ticket on http://dev.plone.org/plone.org with an RSS URL.
Repoze is not (and is not intended to be) a replacement for zope.publisher, while zope.pipeline is. I hope we can integrate Repoze components into zope.pipeline, but it’s not time for that yet. It’s easier to start by just pulling all of the code from zope.publisher, while looking at Repoze for design ideas.
Martin, I think this is the road to least reinvention for the time being, as Shane’s refactoring the existing code to be WSGI-ish, instead of replacing them wholesale with components that do something similar but in a different way. I agree with Shane we should look at those, but later, when things that work now have been improved first.
Projecting the pipeline concept to it’s “logical conclusion” leads to pondering the “growing pains” of managing an exponentially expanding number of possible pathways, as more and more WSGI goodies appear on the horizon.
The PyKE (Python Knowledge Engine) may offer a useful way to capture and store (compile as “plans”) any number of “dynamic” pipelines, based on declarative rules about introspectable (is that a word!?) features. Ultimately, that’s what PyKE is for, to enable compiling an arbitrarily large range of possible solutions into re-usable plans, matched to present conditions via a rule-base.
The excerpt below is submitted strictly as an _example_ of the kinds of code re-use that PyKE makes possible. Dynamic is wonderful, but once you’ve crunched through a solution, store it! Then, call it back on demand, when similar rules apply. Think WSGI widgets as “functions”, piped together as defined in the appropriate plan. If such a plan doesn’t already exist, it may be “discovered”, and _next_time_, it will already exist …
“it occurred to me that backward-chaining could be used to automatically figure out how to join database tables together and generate SQL statements.
And if the backward-chaining rules could see which substitution variables are needed by an HTML templating system, it could automatically generate the SQL to get these data and build the code to update the template.
It seemed that it would no longer be necessary to include anything that looks like code in the HTML templates. The graphic designers could just add simple attributes to their tags and the backward-chaining system would figure out the rest. This would mean that the programmers don’t need to modify the HTML templates, and the graphic designers could maintain full ownership of the HTML.
I had a WSGI front-end that would simply assert the data passed to it as facts.
The forward-chaining rules took these starting facts, parsed the cookie information, form information, browser information, and url, determined whether the user was logged in, figured out which client the request was for, established all of this as additional facts and activated the appropriate rule base for this client.
Then the WSGI front-end simply asked for a proof of the process() goal and executed the resulting plan function which returned the final HTTP status codes and HTML document.”
http://pyke.sourceforge.net/PyCon2008-paper.html#initial-results
Thanks for all the Zope,
Jerry S.