Always Run the RelStorage Tests

I would like to advise all users of RelStorage to run the RelStorage test suite on their staging and production servers before running any application that uses RelStorage.  There are a number of ways to misconfigure the database and the tests will reveal many kinds of issues.  The test suite is in very good shape and should pass every test, every time.

Here are a few MySQL-specific misconfigurations that the tests will reveal:

  • An incorrect database adapter.  For example, Debian Etch still ships MySQLdb 1.2.1, causing most of the tests to fail.
  • Insufficient packet size.  In my.cnf, you should increase max_allowed_packet from 16M to at least 32M.  This parameter limits the maximum size of an object stored by RelStorage.
  • Insufficient space in /tmp.  You need a lot more than 16M available, but linux-vserver limits /tmp to 16M by default.

To run the suite, do something like this:

cd /path/to/relstorage
export PYTHONPATH=`pwd`:/path/to/ZODB/src
python relstorage/tests/testmysql.py  # or testpostgresql.py or testoracle.py

Personally, every time I run these tests on a new database, I discover something misconfigured.  It is worth the time to run the tests.

11 thoughts on “Always Run the RelStorage Tests”

  1. RelStorage “seems” to be working, but the unit test seem to fail horribly .. any ideas?

    ……….WARNING:relstorage:Reconnecting load_conn: (2006, ‘MySQL server has gone away’)
    …ERROR:ZODB.ConflictResolution:Unexpected error
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/ConflictResolution.py”, line 207, in tryToResolveConflict
    resolved = resolve(old, committed, newstate)
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/ConflictResolution.py”, line 50, in _p_resolveConflict
    raise AttributeError(“no attribute (testing conflict resolution)”)
    AttributeError: no attribute (testing conflict resolution)
    .ERROR:ZODB.ConflictResolution:Unexpected error
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/ConflictResolution.py”, line 207, in tryToResolveConflict
    resolved = resolve(old, committed, newstate)
    TypeError: _p_resolveConflict() takes exactly 3 arguments (4 given)
    ..WARNING:relstorage:Storage KeyError on oid 1: creation has been undone; Current transaction is 250641370987609020; Recent object tids: [250641370987609020L, 250641370987609019L]
    …………FE..F…………………..FE.F……….WARNING:relstorage:Storage KeyError on oid 1: creation has been undone; Current transaction is 250641372132933637; Recent object tids: [250641372132933637L, 250641372132933636L, 250641372132933635L, 250641372132933634L, 250641372132933633L, 250641372132933632L]
    ……EFE….WARNING:relstorage:Storage KeyError on oid 1: creation has been undone; Current transaction is 250641372276099211; Recent object tids: [250641372276099211L, 250641372276099210L, 250641372276099209L, 250641372276099208L]
    ………..EE..EE
    ======================================================================
    ERROR: checkIterateWhileWriting (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/IteratorStorage.py”, line 153, in checkIterateWhileWriting
    self.assertRaises(StopIteration, iterator.next)
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/unittest.py”, line 320, in failUnlessRaises
    callableObj(*args, **kwargs)
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/RelStorage-1.1c1-py2.4.egg/relstorage/relstorage.py”, line 985, in next
    params = self._transactions[self._index]
    IndexError: list index out of range

    ======================================================================
    ERROR: checkPackUnlinkedFromRoot (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 578, in checkPackUnlinkedFromRoot
    tid = log[0][‘id’]
    IndexError: list index out of range

    ======================================================================
    ERROR: checkTransactionExtensionFromIterator (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/IteratorStorage.py”, line 89, in checkTransactionExtensionFromIterator
    self.assertEqual(txn.extension, {})
    AttributeError: ‘RecordIterator’ object has no attribute ‘extension’

    ======================================================================
    ERROR: checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/TransactionalUndoStorage.py”, line 503, in checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot
    tid = log[0][‘id’]
    IndexError: list index out of range

    ======================================================================
    ERROR: checkRestoreWithMultipleObjectsInUndoRedo (__main__.MySQLToFile)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/RecoveryStorage.py”, line 196, in checkRestoreWithMultipleObjectsInUndoRedo
    self.compare(self._storage, self._dst)
    File “/usr/src/RelStorage/relstorage/tests/reltestbase.py”, line 478, in compare
    self.assertRaises(IndexError, txn2.next)
    AttributeError: ‘TransactionRecord’ object has no attribute ‘next’

    ======================================================================
    ERROR: checkSimpleRecovery (__main__.MySQLToFile)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/RecoveryStorage.py”, line 35, in checkSimpleRecovery
    self.compare(self._storage, self._dst)
    File “/usr/src/RelStorage/relstorage/tests/reltestbase.py”, line 478, in compare
    self.assertRaises(IndexError, txn2.next)
    AttributeError: ‘TransactionRecord’ object has no attribute ‘next’

    ======================================================================
    ERROR: checkRestoreWithMultipleObjectsInUndoRedo (__main__.FileToMySQL)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/RecoveryStorage.py”, line 196, in checkRestoreWithMultipleObjectsInUndoRedo
    self.compare(self._storage, self._dst)
    File “/usr/src/RelStorage/relstorage/tests/reltestbase.py”, line 466, in compare
    recs1 = [(r.oid, r) for r in txn1]
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py”, line 1681, in next
    h = self._read_data_header(pos)
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/FileStorage/format.py”, line 143, in _read_data_header
    self._file.seek(pos)
    ValueError: I/O operation on closed file

    ======================================================================
    ERROR: checkSimpleRecovery (__main__.FileToMySQL)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/RecoveryStorage.py”, line 35, in checkSimpleRecovery
    self.compare(self._storage, self._dst)
    File “/usr/src/RelStorage/relstorage/tests/reltestbase.py”, line 466, in compare
    recs1 = [(r.oid, r) for r in txn1]
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py”, line 1681, in next
    h = self._read_data_header(pos)
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/FileStorage/format.py”, line 143, in _read_data_header
    self._file.seek(pos)
    ValueError: I/O operation on closed file

    ======================================================================
    FAIL: checkIterateRepeatedly (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/IteratorStorage.py”, line 135, in checkIterateRepeatedly
    self.assertEquals(0, len(list(transactions)))
    AssertionError: 0 != 1

    ======================================================================
    FAIL: checkLoadBefore (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/RevisionStorage.py”, line 62, in checkLoadBefore
    assert prev < middle < cur # else the snooze() trick failed
    AssertionError

    ======================================================================
    FAIL: checkPackUndoLog (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 649, in checkPackUndoLog
    self.assertEqual(1,len(self._storage.undoLog()))
    AssertionError: 1 != 0

    ======================================================================
    FAIL: checkPackWithMultiDatabaseReferences (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 315, in checkPackWithMultiDatabaseReferences
    self.assertEqual(len(self._storage), 1)
    AssertionError: 0 != 1

    ======================================================================
    FAIL: checkTransactionalUndoAfterPack (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.9.0a7-py2.4-linux-x86_64.egg/ZODB/tests/TransactionalUndoStorage.py”, line 449, in checkTransactionalUndoAfterPack
    eq(len(info2), 2)
    AssertionError: 0 != 2

    ———————————————————————-
    Ran 100 tests in 31.947s

    FAILED (failures=5, errors=8)

  2. I have not yet tested with the pre-releases of ZODB 3.9. I imagine there are subtle differences that require changes to RelStorage. I don’t recommend that configuration.

  3. Many thanks for that, the default for buildout is 3.9, even tho’ it says 3.8 .. I’d not realised someone had screwed the repo’s. I’ve manually installed ZODB 3.8 and that’s fixed most of the problems … I’m still left with this however, any ideas?

    …………WARNING:relstorage:Reconnecting load_conn: (2006, ‘MySQL server has gone away’)
    …ERROR:ZODB.ConflictResolution:Unexpected error
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/ConflictResolution.py”, line 207, in tryToResolveConflict
    resolved = resolve(old, committed, newstate)
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/tests/ConflictResolution.py”, line 50, in _p_resolveConflict
    raise AttributeError(“no attribute (testing conflict resolution)”)
    AttributeError: no attribute (testing conflict resolution)
    .ERROR:ZODB.ConflictResolution:Unexpected error
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/ConflictResolution.py”, line 207, in tryToResolveConflict
    resolved = resolve(old, committed, newstate)
    TypeError: _p_resolveConflict() takes exactly 3 arguments (4 given)
    …………………………………………F…………………………………..
    ======================================================================
    FAIL: checkPackWithMultiDatabaseReferences (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB/tests/PackableStorage.py”, line 329, in checkPackWithMultiDatabaseReferences
    assert(len(self._storage) == 1)
    AssertionError

    ———————————————————————-
    Ran 106 tests in 17.510s

    FAILED (failures=1)

  4. That’s a new test in ZODB that I first saw this week. It is slightly incorrect: like the other tests, it needs to permit the storage to return a length of 0. I am about to check in a change to that test.

  5. Mmm, I’ve got a very strange issue now .. I’ve moved to a production box .. and the same buildout / config is now giving 4 failures and 2 errors … while the test box just gives one fail… I’ve redone it about 10 times now with different combinations and can’t see the difference .. don’t suppose you can spot anything ???

    Only difference I can see is a move from x86 to x86_64 (i.e. architecture) , everything else is the same .. tried easy_install relstoreage and also SVN ..
    Could it be RelStorage doesn’t work on 64 bit?

    ======================================================================
    ERROR: checkPackUnlinkedFromRoot (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 558, in checkPackUnlinkedFromRoot
    tid = log[0][‘id’]
    IndexError: list index out of range

    ======================================================================
    ERROR: checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-x86_64.egg/ZODB/tests/TransactionalUndoStorage.py”, line 506, in checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot
    tid = log[0][‘id’]
    IndexError: list index out of range

    ======================================================================
    FAIL: checkLoadBefore (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-x86_64.egg/ZODB/tests/RevisionStorage.py”, line 62, in checkLoadBefore
    assert prev < middle < cur # else the snooze() trick failed
    AssertionError

    ======================================================================
    FAIL: checkPackUndoLog (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 629, in checkPackUndoLog
    self.assertEqual(1,len(self._storage.undoLog()))
    AssertionError: 1 != 0

    ======================================================================
    FAIL: checkPackWithMultiDatabaseReferences (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 329, in checkPackWithMultiDatabaseReferences
    assert(len(self._storage) == 1)
    AssertionError

    ======================================================================
    FAIL: checkTransactionalUndoAfterPack (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/opt/Plone-3.1/Python-2.4/lib/python2.4/site-packages/ZODB3-3.8.1-py2.4-linux-x86_64.egg/ZODB/tests/TransactionalUndoStorage.py”, line 452, in checkTransactionalUndoAfterPack
    eq(len(info2), 2)
    AssertionError: 0 != 2

    ———————————————————————-
    Ran 106 tests in 34.836s

    FAILED (failures=4, errors=2)

  6. Ok, just installed on another 32 bit machine, got the one failure scenario, which sounds like it’s not a problem. In which case, it points at either RelStorage or the test suite not working in a 64 bit environment ????

  7. Ok, absolutely clean build on 64 bit, identical to 32 bit being very careful .. 1 failure on 32 bit whereas I get 4 on 64 bit with 2 errors.
    Any idea if this is just the tests, is this safe to use?

  8. Tried 3.8.0 (as opposed to 3.8.1) same problem except “checkPackWithMultiDatabaseReferences” no longer fails.

    ======================================================================
    ERROR: checkPackUnlinkedFromRoot (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 535, in checkPackUnlinkedFromRoot
    tid = log[0][‘id’]
    IndexError: list index out of range

    ======================================================================
    ERROR: checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-x86_64.egg/ZODB/tests/TransactionalUndoStorage.py”, line 503, in checkTransactionalUndoAfterPackWithObjectUnlinkFromRoot
    tid = log[0][‘id’]
    IndexError: list index out of range

    ======================================================================
    FAIL: checkLoadBefore (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-x86_64.egg/ZODB/tests/RevisionStorage.py”, line 62, in checkLoadBefore
    assert prev < middle < cur # else the snooze() trick failed
    AssertionError

    ======================================================================
    FAIL: checkPackUndoLog (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-x86_64.egg/ZODB/tests/PackableStorage.py”, line 606, in checkPackUndoLog
    self.assertEqual(1,len(self._storage.undoLog()))
    AssertionError: 1 != 0

    ======================================================================
    FAIL: checkTransactionalUndoAfterPack (__main__.MySQLTests)
    ———————————————————————-
    Traceback (most recent call last):
    File “/usr/lib/python2.5/site-packages/ZODB3-3.8.0-py2.5-linux-x86_64.egg/ZODB/tests/TransactionalUndoStorage.py”, line 449, in checkTransactionalUndoAfterPack
    eq(len(info2), 2)
    AssertionError: 0 != 2

    ———————————————————————-
    Ran 105 tests in 33.523s

    FAILED (failures=3, errors=2)

  9. RelStorage has been tested successfully on 64 bit. Please verify you’re running MySQLdb 1.2.2 and that MySQL actually used the InnoDB backend for MySQL. (Use ‘show table status;’.) Also, please describe the operating system.

  10. Ok,

    1. Yes, 1.2.2 as per the docs
    2. MySQL is the Ubuntu standard which supports InnoDB and MyISAM
    3. Database is relstoragetest, tables are created by the test routines, 3 are InnoDB, 6 are MyISAM
    (innodb are current_object, object_state and transaction)
    4. Operating system is a Linux 2.6.21 kernel, Redhat Xen client running on Xen 3.2, system is production, 100% uptime, has been running Plone 3.x for six months.

    I’ve done the tests again from scratch on 32bit and 64bit, same results … fyi, I’ve loaded a complex site up and it “seems” to “run” on the 64 bit ….

    mysql> show table status;
    +——————-+——–+———+————+——+—————-+————-+——————+————–+———–+—————-+———————+———————+————+——————-+———-+—————-+———————————————————————————-+
    | Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
    +——————-+——–+———+————+——+—————-+————-+——————+————–+———–+—————-+———————+———————+————+——————-+———-+—————-+———————————————————————————-+
    | current_object | InnoDB | 10 | Compact | 1 | 16384 | 16384 | 0 | 16384 | 0 | NULL | 2008-12-12 15:57:08 | NULL | NULL | latin1_swedish_ci | NULL | | InnoDB free: 75776 kB; (`zoid` `tid`) REFER `relstoragetest/object_state`(`zoid` |
    | new_oid | MyISAM | 10 | Fixed | 1 | 9 | 9 | 2533274790395903 | 2048 | 0 | 3 | 2008-12-12 16:20:42 | 2008-12-12 16:20:42 | NULL | latin1_swedish_ci | NULL | | |
    | object_ref | MyISAM | 10 | Fixed | 0 | 0 | 0 | 7036874417766399 | 1024 | 0 | NULL | 2008-12-12 15:57:08 | 2008-12-12 16:20:41 | NULL | latin1_swedish_ci | NULL | | |
    | object_refs_added | MyISAM | 10 | Fixed | 0 | 0 | 0 | 2533274790395903 | 1024 | 0 | NULL | 2008-12-12 15:57:08 | 2008-12-12 16:20:41 | NULL | latin1_swedish_ci | NULL | | |
    | object_state | InnoDB | 10 | Compact | 3 | 5461 | 16384 | 0 | 32768 | 0 | NULL | 2008-12-12 15:57:08 | NULL | NULL | latin1_swedish_ci | NULL | | InnoDB free: 75776 kB |
    | pack_object | MyISAM | 10 | Fixed | 0 | 0 | 0 | 5066549580791807 | 1024 | 0 | NULL | 2008-12-12 16:20:41 | 2008-12-12 16:20:41 | NULL | latin1_swedish_ci | NULL | | |
    | pack_state | MyISAM | 10 | Fixed | 0 | 0 | 0 | 4785074604081151 | 1024 | 0 | NULL | 2008-12-12 16:20:41 | 2008-12-12 16:20:41 | NULL | latin1_swedish_ci | NULL | | |
    | pack_state_tid | MyISAM | 10 | Fixed | 0 | 0 | 0 | 2533274790395903 | 1024 | 0 | NULL | 2008-12-12 16:20:41 | 2008-12-12 16:20:41 | NULL | latin1_swedish_ci | NULL | | |
    | transaction | InnoDB | 10 | Compact | 4 | 4096 | 16384 | 0 | 0 | 0 | NULL | 2008-12-12 15:57:08 | NULL | NULL | latin1_swedish_ci | NULL | | InnoDB free: 75776 kB |
    +——————-+——–+———+————+——+—————-+————-+——————+————–+———–+—————-+———————+———————+————+——————-+———-+—————-+———————————————————————————-+
    9 rows in set (0.00 sec)

Comments are closed.