RelStorage 1.2.0b2 Released

This release works with unpatched versions of ZODB 3.9!  A big thank-you to Jim Fulton for including support for RelStorage in ZODB.  This release also continues to support patched versions of ZODB 3.7 and 3.8.

I have been doing a lot of testing, and I have found MySQL 5.1.34 to be a lot more stable than earlier releases of MySQL 5.1, so I am now declaring MySQL 5.1.34 and above supportable, meaning that if you ask questions about it, I am no longer going to request that you revert to MySQL 5.0. 🙂

Finally, I recently expanded my private RelStorage Buildbot to include a Windows XP slave.  After solving a couple of minor test glitches, the test results are now all consistently green on 4 platforms:

  • Debian Etch, 32 bit (Python 2.4.4, MySQL 5.0.32, PostgreSQL 8.1.17, Oracle 10g XE)
  • Debian Lenny, 32 bit (Python 2.5.2, MySQL 5.0.51a, PostgreSQL 8.3.7, Oracle 10g XE)
  • Debian Lenny, 64 bit (same as above but no Oracle)
  • Windows XP, 32 bit (Python 2.6.2, MySQL 5.1.34, PostgreSQL 8.3.7)

I’m thinking about adding another Linux slave that runs MySQL 5.1 and Python 2.6.

Anyway, enjoy the release!

P.S. You may be wondering why I released 1.2.0b2 instead of 1.2.0b1.  A little slip ruined the web page on PyPI, so I fixed the slip and skipped to the next version number.

How to Install Plone with RelStorage and MySQL

These step by step instructions describe how to install Plone on Ubuntu with RelStorage connected to MySQL as the main database. Familiarity with Linux systems administration is expected. Update: These instructions were revised in August 2009 for Plone 3.2.3 and RelStorage 1.2.0.

Continue reading How to Install Plone with RelStorage and MySQL

RelStorage for Sessions?

I’ve been working on a document explaining how to install Plone with RelStorage, starting from a basic Linux server.  As always, the basic procedure is simple, but there are all sorts of interesting little complications.  One detail that bugged me today is the need for a shared session database.

Session storage is a little different from normal storage because keeping a history of session state becomes expensive quickly.  For session storage, I think we still want all the goodness of ZODB transactions, conflict detection, distributed caching, and so on, but in this case, the ability to undo is pointless and the need to pack is a liability.

The database schema I’ve been using in RelStorage is a mismatch for this need.  A history-free storage should not have a “transaction” table, there should be no need for MD5 sums and prev_tid pointers, and the compound primary keys consisting of oid and tid should become simple primary keys indexed by oid.  A history-free storage still needs garbage collection, but not packing.

I’m thinking that the main RelStorage class will need few changes to support history-free storages, while the database adapter class will change so much that it would be best to just create a different adapter.  I like that.  It won’t be possible to switch history on and off without an export and import operation, but that seems reasonable.  To put a positive spin on the new adapters, I think I’ll call the new adapters “packless”, like the old BerkeleyDB storage.

I think that’s a good plan.  While the purpose of the packless adapters is initially session storage, they will certainly be also usable for other databases, including the main database.  I expect them to be slightly simpler and faster than the history-preserving adapters we have now.

I encourage anyone interested to leave a comment, even if all you wish to say is “I want that!” 🙂

How to Fix the MySQL Write Speed

Last time I ran the RelStorage performance tests, the write speed to a MySQL database appeared to be slow and getting slower.  I suspected, however, that all I needed to do was tune the database.  Today I changed some InnoDB configuration parameters from the defaults.  The simple changes solved the MySQL performance problem completely.

The new 10K chart, using RelStorage 1.1.3 on Debian Sid with Python 2.4 and the same hardware as before:

I added the following lines to my.cnf to get this speed:

innodb_data_file_path = ibdata1:10M:autoextend

This is similar to the configuration suggested by the InnoDB documentation for a 512 MB database server.  Even if you have a 16 GB server, I would suggest starting with the settings for a 512 MB server, then watch what happens to the RAM and CPU on the database server when you connect all of your client machines simultaneously.  You want to leave at least half the RAM available for disk cache and usage spikes.

Not all of these changes are related to speed.  The innodb_file_per_table option just seems like a good idea because it makes tables visible on the filesystem, which should improve manageability.  I think it might improve cache locality as well.

With these changes to my.cnf, ZEO, PostgreSQL, and MySQL all perform about the same for writes, with MySQL having a slight lead.  I suspect all three are hitting hardware and kernel limits.  I think the differences would be more pronounced on higher-end storage hardware.

A big caveat: It’s risky to change InnoDB settings unless you’re familiar with all the effects.  Some changes break compatibility with existing table data.  Get to know the InnoDB documentation very well before you change these settings, and make backups using mysqldump, as always.

Meanwhile, Oracle XE continues to write slowly and ZEO read performance is so bad that it’s off the chart.  I bet ZEO read performance could be improved with some simple optimizations somewhere, but I don’t have an incentive to fix that. 🙂  Perhaps it has been fixed in ZODB 3.9.

RelStorage Support

I am more than happy to support RelStorage as best I can by email.  Every time I do, however, I always get a nagging feeling that I could help RelStorage users a lot better if we set up a short term support contract.  I would very much appreciate a chance to optimize their system by testing the performance of different configurations.  When the communication is limited to email, neither of us gets a chance to discover how we might help each other better.

So if you’re a RelStorage user and your database is growing by tens of gigabytes, please seriously consider a short term support contract with my little company.  A little tuning or code revision in the right place could yield orders of magnitude performance gains.  I really want to help you directly.  Contact me at shane (at) willowrise (dot) com.

keas.pbpersist and keas.pbstate

Remember how I was talking about serializing data in ZODB using Google Protocol Buffers instead of pickles?  Well, Keas Inc. suggested the idea and asked me to work on it.  The first release of a package combination that implements the idea is ready:

  • keas.pbstate 0.1.1 (on PyPI), which helps you write classes that store all state in a Protocol Buffer message.
  • A patch for ZODB (available at that makes it possible to plug in serializers other than ZODB’s standard pickle format.
  • keas.pbpersist 0.1 (on PyPI), which registers a special serializer with ZODB so that ProtobufState objects get stored without any pickling.

This code is new, but the tests all pass and the coverage tests claim that every line of code is run by the tests at least once.  I hope these packages get used so we can find out their flaws.  I did my best to make keas.pbstate friendly, but there are some areas, object references in particular, where I don’t like the current syntax.  I don’t know if this code is fast–optimization would be premature!

I should mention that while working with protobuf, I got the feeling that C++ is the native language and Java and Python are second class citizens.  I wonder if I’d have the same feeling with other serialization formats.

RelStorage 1.1.2

This release has two new useful features, one for performance, one for safety.

The performance feature is that if you use both the cache-servers and poll-interval options, RelStorage will use the cache to distribute basic change notifications.  That means we get to lighten the load on the database using the poll-interval, yet changes should still be seen instantly on all clients.  Yay! 🙂

The only drawback I expect is that caching makes debugging more difficult.  Still, this option should help people build enormous clusters, like the one my current customer was planning to build, although I got word today that they have changed their mind.

The new safety feature is the pack-dry-run option, which lets you run only the nondestructive pre_pack phase to get a list of everything that would be deleted by the pack phase.  This should be particularly useful if you’re trying out packing for the first time on a big database.  My current customer would have benefited from this too.

I also fixed a bug that caused the pack code to not remove as much old stuff as it should and I started using PyPI instead of the wiki as the main web page.  Using PyPI means I have to maintain only one README, which gets translated automatically into the PyPI page.  Until now I’ve had to maintain both the README and the wiki page.

Patched ZODB3 Eggs Available

This week, I put up some ZODB3 eggs and source distributions with the patch required for RelStorage already applied. I built both ZODB3-3.8.1-polling and ZODB3-3.7.3-polling.  I even made eggs for Windows developers who have not yet taken the time to set up MinGW. 😉

Developers can use this web site in buildout.cfg to incorporate RelStorage in their applications.  Feel free to mirror the site if you need to.

What Would ZODB + Paxos Look Like?

I just learned about the Paxos algorithm. I think we might be able to use it to create a fully distributed version of ZODB. I found a document that explains Paxos in simple terms.  Now I’m interested in learning about any ideas and software that might support integration of Paxos into ZODB.  I would also like to know how much interest people have in such a project.

I think ZODB’s transaction layer already implements a sort of squashed version of Paxos, but it’s not currently possible to separate the pieces to make it distributed.  To me, “distributed ZODB” means multiple servers accept writes while assuring consistency at all times.  I also require sub-millisecond response timing on the majority of read operations, since that is what ZODB applications have come to rely upon.  I suspect the speed requirement disqualifies systems like CouchDB.

ZODB + Protobuf Looking Good

Today I drafted a module that mixes ZODB with Protocol Buffers.  It’s looking good!  I was hoping not to use metaclasses, but metaclasses solved a lot of problems, so I’ll keep them unless they cause deeper problems.

The code so far lets you create a class like this:

class Tower(Persistent):
    __metaclass__ = ProtobufState
    protobuf_type = Tower_pb

    def __init__(self): = "Rapunzel's"
        self.height = 1000.0
        self.width = 10.0

    def __str__(self):
        return '%s %fx%f %d' % (
  , self.height, self.width, self.build_year)

The only special lines are the second and third.  The second line says the class should use a metaclass that causes all persistent state to be stored in a protobuf message.  The third line says which protobuf class to use.  The protobuf class comes from a generated Python module (not shown).

If you try to store a string in the height attribute, Google’s code raises an error immediately, which is probably a nice new benefit.  Also, this Python code does not have to duplicate the content of the .proto file.  Did you notice the build_year attribute?  Nothing sets it, but since it’s defined in the message schema, it’s possible to read the default value at any time.  Here is the message schema, by the way:

message Tower {
    required string name = 1;
    required float width = 2;
    required float height = 3;
    optional int32 build_year = 4 [default = 1900];

The metaclass creates a property for every field defined in the protobuf message schema.  Each property delegates the storage of an attribute to a message field.  The metaclass also adds __getstate__, __setstate__, and __new__ methods to the class; these are used by the pickle module.  Since ZODB uses the pickle module, the serialization of these objects in the database is now primarily a protobuf message.  Yay!  I accomplished all of this with only about 100 lines of code, including comments. 🙂

I have a solution for subclassing that’s not beautiful, but workable.  Next, I need to work on persistent references and _p_changed notification.  Once I have those in place, I intend to release a pre-alpha and change ZODB to remove the pickle wrapper around database objects that are protobuf messages.  At that point, when combined with RelStorage, ZODB will finally store data in a language independent format.

As a matter of principle, important data should never be tied to any particular programming language.  Still, the fact that so many people use ZODB even without language independence is a testament to how good ZODB is.

Jonathan Ellis pointed out Thrift, an alternative to protobuf.  Thrift looks like it could be good, but my customer is already writing protobuf code, so I’m going to stick with that for now.  I suspect almost everything I’m doing will be be applicable to Thrift anyway.

I know what some of you must be thinking: what about storing everything in JSON like CouchDB?  I’m sure it could be done, but I don’t yet see a benefit.  Maybe someone will clue me in.