RelStorage 1.0 and Measurements

RelStorage 1.0 is out!

To celebrate, I thought I’d post some measurements. They are the results of a speed test I’ve been running on RelStorage. I created the speed test for the purpose of identifying performance problems. The test certainly helped, but I found that the results varied dramatically between runs, so don’t take these too seriously.

The test hardware is an AMD 5200+ (2 cores, 2.6 GHz). The kernel is 64 bit Linux 2.6.24, but userland is a 32 bit Debian “sid” environment in a chroot area (because it’s much easier to install Oracle in 32 bit Debian than in 64 bit Gentoo). The test is running on Python 2.4 and ZODB 3.7.1 (from Zope 2.10.5). The contenders are ZEO + FileStorage, PostgreSQL 8.3, MySQL 5.0.51a, and Oracle 10g XE. I am using the default configuration of all databases.

The first test shows the time spent per transaction to write 100 objects, followed by the average time spent to read those 100 objects in a different transaction. The time shown is the best of 3 runs, where each run measures the mean duration of each of 10 transactions.

This chart says a few interesting things:

• MySQL is the champ in both read and write speed. I tried hard to make PostgreSQL match, but I haven’t yet found a way. This measurement is backed up by other general observations such as the speed of the unit tests. Both databases run the same 100+ tests, but MySQL runs them in about 17 seconds, while PostgreSQL runs them in about 29 seconds.
• Oracle 10g XE, unlike the standard Oracle product, is not intended for production. It’s limited in several ways. This test is probably not fair to Oracle, because I expect the standard edition of Oracle to fare better.
• The ZEO + FileStorage results become unpredictable as the concurrency increases. I have a strong suspicion why that occurs: the ZEO client storage often crashes with a segfault! The speed test automatically recovers from the crash and re-runs the test, but I don’t feel like I can trust the results when ZEO is segfaulting. The crash occurs in zrpc when a lot of clients connect at once.
• PostgreSQL write speed was only a little slower than ZEO in this test.

The second test shows the time spent per transaction to write and read 10,000 objects.

In this case, MySQL still leads in read speed, but both ZEO and PostgreSQL beat MySQL in write speed. Surprisingly, the ZEO read speed turns out to be the slowest of all the tests, perhaps because 10,000 objects do not fit in the ZEO cache.

The ZEO measurements have major aberrations on the right side of the chart. This may be caused by the fact that during these tests, the ZEO client segfaults on nearly every test run. Fortunately, the fault is in the client, not the server, and these tests put unusually high stress on the ZEO connection setup and teardown code. ZEO connection setup and teardown usually only occurs when Zope starts or stops.

While the primary goal for RelStorage is reliability, not performance, it’s nice to see test results that suggest RelStorage can compete and often win.

UPDATE 12/2008: I clarified the statement above about Oracle 10g XE.