Compositing in Plone

Seven years ago, I started a project called CompositePage.  It added a drag and drop UI to Zope 2.  Others built on it and created CompositePack.  I applaud them for their efforts.

However, today I think the CompositePage UI pattern is the wrong idea.  The CompositePage UI pattern is to render the page with extra markup that allows people with the necessary privileges to move objects around the page.  The idea works technically, but it incurs a lot of technical issues for template authors and it is frequently difficult to explain to users.

I saw an implementation of a much better idea a few years ago.  The better idea is to present a greatly simplified representation of the page when in design mode.  Ideally, the simplified page should contain only basic colors and fuzzy text, so as to draw the user’s attention to the widgets, not the page.

Now it just occurred to me that CompositePage could implement this idea easily by letting developers define two templates per layout: the real template and the representative design template.  Why didn’t I think of that before?  That would be quite friendly.

RelStorage 1.4.0b1 and zodbshootout

I just released RelStorage 1.4.0b1.  New features:

  • More documentation.
  • Support for history-free storage on PostgreSQL, MySQL, and Oracle.  This reduces the need to pack and makes RelStorage more appropriate for session storage.
  • Speed.  New tests prompted several optimizations that reduced the effect of network latency in both read and write operations.  Memcached support is now integrated in a much better way.
  • Support for asynchronous database replication.  Previous versions of RelStorage worked with MySQL replication, but did not keep ZODB caches in sync when failing over to a slave that was slightly out of date.
  • The Oracle adapter now uses PL/SQL for speed and lock timeouts.  Lock timeouts are important for preventing cluster lockup.
  • Moved the speed test script into a separate package named zodbshootout, making it easier for developers and administrators to run comparative performance tests.
  • The adapter code is more modular, making it easier to support new kinds of databases and database adapter modules.

The zodbshootout script tells me this release of RelStorage is faster than ever.  It reports objects read or written per second, so unlike the previous charts I’ve made, bigger is now better.  Here are the results:

relstorage-speed

PostgreSQL now beats MySQL in some of the tests.  Oracle (not on this chart) is now looking pretty good too.

The new features led to far more automated tests.  My private Buildbot, which tests RelStorage with several combinations of Python, ZODB, and operating systems (in virtual private servers), now takes 2 hours to run all the tests.  Maybe I need to upgrade that server or investigate the possibility of making Buildbot launch an Amazon EC2 instance.

The previous release was 1.3.0b1, which added ZODB blob support.  Several customers asked for new features right after I released 1.3.0b1, so I decided to jump to version 1.4.0b1 rather than finalize the 1.3 series.  The 1.2 series has had more extensive testing, so use that for a while if you have troubles with 1.4.0b1.

This new release should be particularly interesting for Plone users, since Plone is always hungry for faster infrastructure.

RelStorage 1.3.0b1, Now With Blob Support

I have just released two versions of RelStorage. Version 1.3.0b1 adds full support for ZODB blobs stored on the filesystem. Version 1.2.0 is currently the better choice if you’re upgrading a production system and don’t need blob support.

People have been asking for blob support for months. I am glad to finally get it done, with a little help from a customer. With blob support, now we can easily store large artifacts on the filesystem, while keeping all metadata in the database.

To celebrate the new release, I have created a sample buildout.cfg that builds Plone with RelStorage, PostgreSQL, and blob support. (Thanks goes to Hanno Schlichting, who released a compatible version of plone.recipe.zope2instance only moments after I requested it.) Here it is:

[buildout]
parts = plone zope2 instance zopepy
find-links =
    http://dist.plone.org
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads
    http://packages.willowrise.org
eggs =
    elementtree
    PILwoTk
    RelStorage
    psycopg2
versions = versions

[versions]
ZODB3 = 3.8.3-polling
RelStorage = 1.3.0b1

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
products = ${plone:products}
eggs =
    ${buildout:eggs}
    ${plone:eggs}
    plone.app.blob
zcml = plone.app.blob
rel-storage =
    type postgresql
    dsn dbname='plone' user='plone' host='localhost' password='plone'
    blob-dir var/blobs

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${instance:zope2-location}/lib/python
scripts = zopepy zodbconvert

P.S. I have been told that a very prominent Plone developer recently configured RelStorage with master/slave replication on MySQL, and that it works smoothly. I expect him to announce his success soon!

Book Review: Practical Plone 3

Packt Publishing asked me to review their new book, Practical Plone 3: A Beginner’s Guide to Building Powerful Websites. The book impressed me, but not in the way I expected at first.

As I read the instructions in chapter two about how to install Plone, I considered the experience my Dad would have gone through if he had this book when we were setting up Plone to run his company’s web site. My Dad is a power user, but not a programmer or systems administrator, so with this book, he probably would have installed Plone himself on a spare Windows computer. This book would have provided him enough direction to set up a lot of the functionality he needed, without my help. He would have immediately started publishing pages with Plone’s many features.

However, I imagine that a short time later, something would go seriously wrong. The computer’s IP address would change because the DHCP lease expired, the database would lose some transactions due to some misbehaving application, or a mischevious virus would rename files with a “py” extension to “rb”. All of those problems are outside Plone’s control, so this book does not try to address them.

Plone beginners like my Dad are not prepared to handle the problems that occur when a computer is used as a web server. In that light, I wondered if it really is possible to run Plone (or any content management system) without deep technical experience. I thought for a moment that this book is not for beginners after all.

Even after that logic, I decided I still want very much to give my Dad a copy of this book next time we set up a Plone web site. He will read it to find out what the latest version of Plone can do. He will install it on his own computer for his own education, but I will set up the production web site on a server.

The first twelve chapters (250 pages) are intended for Plone users. Beginners will enjoy all of those chapters, I think. As I read them, I even picked up a few things I haven’t learned, like how to use content rules.

I think beginners might struggle the most with chapter nine, which explains how to control workflow. Controlling workflow in Plone is not as easy as other Plone functions, because Plone falls back to the less polished Zope Management Interface for workflow design. Matt Bowen handled the difficult topic gracefully.

The rest of the book (almost 300 pages) is for developers, not power users. The contrast is sharp. While the first half of the book tells the reader what buttons to push, the second half tells the reader how to modify their Buildout configuration and what to type in a terminal session.

Each chapter is written by a different author. I noticed two interesting effects of multiple authorship. First, each author is enthusiastic about the particular topic, so even LDAP (which I generally find quite boring) gets a chapter of quality treatment. Second, there is more redundancy than you would find in most technical books, but redundancy is probably good in this case.

I do have one quibble with the book’s organization. When the technology behind Plone was invented, CSS was still a baby and browsers did not support it well. Back then, changing a site’s appearance meant changing nested tables in HTML, so the developers of Zope (including myself) invented ways to manage that task. That is how the portal_skins tool came about. The theming chapters explain how to use the latest version of that technology.

Today, we can expect all of our customers to use browsers that support CSS, so the chapters on theming should start by explaining how to customize the web site’s CSS. Developers will make much faster progress that way than if they have to learn the many theming-related abstractions Plone has today.

In conclusion, Practical Plone 3 is more than just a beginner’s book. I plan to use this book as a communication tool with my Plone customers. The book is a menu telling my customers who are beginners to Plone what we can set up together without a lot of work. I will also use it to help developers come up to speed on Plone.

How to Install Plone with RelStorage and MySQL

These step by step instructions describe how to install Plone on Ubuntu with RelStorage connected to MySQL as the main database. Familiarity with Linux systems administration is expected. Update: These instructions were revised in August 2009 for Plone 3.2.3 and RelStorage 1.2.0.

Continue reading How to Install Plone with RelStorage and MySQL

An Abominable, Crazy Idea

What if I were to take a Plone install, delete all of the Zope 2 code (leaving mainly Zope 3 stuff), then mangle the code (including CMF) until Plone works again?

I know I’m foolish for saying this, but my intuition tells me I could actually do it, especially if I got a lot of help from others.  I think it would take 1 to 3 months of full time effort.  I would redesign all sorts of stuff in the process.  The resulting code would not be compatible with any existing site; the necessary migration scripts would take many more months, I think.  But the code would be enough to create new sites from scratch and put them on the web.  The intent would be to make Plone more maintainable.

It would be really crazy… but fun, I think.

RelStorage 1.1 beta 1

This release has greatly optimized packing. This time for real, I hope! 🙂

At this point, I am confident in the correctness of the pack algorithm. Version 1.0 had it right as well. However, version 1.0 was naive about just how many object states might be packed all at once. It is unwise to attempt to delete gigabytes of data in millions of rows with a single, simple delete statement; it turns out that can take multiple days to complete. Even worse is to do that while holding the commit lock.

So version 1.1 packs one transaction at a time and periodically releases the commit lock. It also has numerous other optimizations in the pack algorithm that should make packing faster and less obtrusive. So, ZODB/Zope/Plone users of the world, please take the beta release for a spin. I’ll help if you have problems with it.

Update: I did not know until now that advisory locks were only recently added to PostgreSQL.  So if you’re using PostgreSQL, at least version 8.2 is required for this beta.  I think I’ll relax that requirement before RelStorage version 1.1.

RelStorage: New MySQL Adapter

I was interested in adding MySQL 5.0+ support to RelStorage, so I went ahead and did it. The code is in Subversion now.

The tests I’ve run suggest the MySQL adapter is already a lot faster than ZEO and both of the other adapters. The MySQL adapter has consistently finished the tests 40% faster than the PostgreSQL adapter. I haven’t tested PostgreSQL 8.3 yet; perhaps that will make a difference. I don’t know yet how the reliability of each adapter compares.

The port took all day. (My wife and I both have a cold, making it unwise to go to work today.) I didn’t think it would take that long, but I had to slow down to figure out the strange relationship between locks and transactions in MySQL. The LOCK TABLE statement is full of surprises! I ended up using GET_LOCK and RELEASE_LOCK and row-level locks instead.

So after editing queries to fit MySQL’s rules and fixing miscellaneous details, RelStorage now has a third database adapter. I’m excited to see what happens next. For one thing, MySQL support could have a major positive effect on Plone.

By the way, MySQLdb version 1.2.2 is required. Version 1.2.1 has a bug involving BLOBs; RelStorage depends heavily on BLOBs.

RelStorage: A New ZODB Storage

I’m writing RelStorage, a new storage implementation for ZODB / Zope / Plone. RelStorage replaces PGStorage. I’ve put up a RelStorage Wiki page and the zodb-dev mailing list has been discussing it. There is no stable release yet, but a stable release is planned for this month.

While performance is not the main goal (reliability and scalability are more important), I was pleasantly surprised to discover last week that creating a Plone 3 site in RelStorage on PostgreSQL 8.1 is a bit faster than doing the same thing in FileStorage, the default ZODB storage. Clearly, the PostgreSQL team is doing a great job!

Several years ago I put together an early prototype of PGStorage. I recall discovering that PostgreSQL was terribly slow at storing a lot of BLOBs. I read about the soon-to-come TOAST feature, but I wasn’t sure it would solve the problem, so I discarded the whole idea for years. Today, PostgreSQL seems to have no problem at all with this kind of work. It sure has come a long way.

RelStorage also connects to Oracle 10g. According to benchmarks, Oracle has a slight performance advantage, perhaps due to the “read only” isolation mode that Oracle provides. It might be useful for PostgreSQL to get that feature too.

I’m considering setting up a MySQL adapter for RelStorage as well. When the database is in MySQL and Zope is running in mod_wsgi, we could say that the “P” in LAMP stands for Plone!