RelStorage 1.3.0b1, Now With Blob Support

I have just released two versions of RelStorage. Version 1.3.0b1 adds full support for ZODB blobs stored on the filesystem. Version 1.2.0 is currently the better choice if you’re upgrading a production system and don’t need blob support.

People have been asking for blob support for months. I am glad to finally get it done, with a little help from a customer. With blob support, now we can easily store large artifacts on the filesystem, while keeping all metadata in the database.

To celebrate the new release, I have created a sample buildout.cfg that builds Plone with RelStorage, PostgreSQL, and blob support. (Thanks goes to Hanno Schlichting, who released a compatible version of plone.recipe.zope2instance only moments after I requested it.) Here it is:

[buildout]
parts = plone zope2 instance zopepy
find-links =
    http://dist.plone.org
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads
    http://packages.willowrise.org
eggs =
    elementtree
    PILwoTk
    RelStorage
    psycopg2
versions = versions

[versions]
ZODB3 = 3.8.3-polling
RelStorage = 1.3.0b1

[plone]
recipe = plone.recipe.plone

[zope2]
recipe = plone.recipe.zope2install
url = ${plone:zope2-url}

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
products = ${plone:products}
eggs =
    ${buildout:eggs}
    ${plone:eggs}
    plone.app.blob
zcml = plone.app.blob
rel-storage =
    type postgresql
    dsn dbname='plone' user='plone' host='localhost' password='plone'
    blob-dir var/blobs

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${instance:zope2-location}/lib/python
scripts = zopepy zodbconvert

P.S. I have been told that a very prominent Plone developer recently configured RelStorage with master/slave replication on MySQL, and that it works smoothly. I expect him to announce his success soon!

10 thoughts on “RelStorage 1.3.0b1, Now With Blob Support”

  1. What would it take to store the blobs in the database instead of filesystem? Several databases loves big objects, and can stream with iterators.

  2. My experience is that it’s wiser to put big files on the filesystem if you can, because flexibility is required to solve the problems that occur when storing multi-gigabyte files. When I last worked with very large files, I had to choose XFS for one system where write speed was most important (we saturated gigabit wires), and Ext3 for another system where longevity was most important. Databases obscure those kind of decisions.

    That said, I will happily add support for storing blobs in the database when I get paid to do it. 🙂

  3. What is the architecture of the blog storage i.e. the naming convention? The reason I as is that I have a .gov.au client who wants to store about 1 million files and Im looking at Plone Storage options. The concern is that it would not work to put them all in one folder so I would want some control over how they are stored or be sure its done in a way that can scale.

  4. Tom: The blob storage formats currently available for ZODB are “lawn” and “bushy”. The lawn format puts all files in a directory, while the default bushy format creates an 8 level subdirectory hierarchy. The lawn format can handle as many objects as the filesystem can.

    It would be easy to expand the available formats. For example, it would be nice to have a format that spreads blobs over a fixed number of similarly-sized volumes using a hash of the object ID. That could be done in a matter of hours.

  5. It seems that blob support in relstorage is not the same as when using clientstorage/zeo? With zeo you define a blob-dir on the server and local file system blob-dir for the clients. That means that any client can always get a copy of the blob files no matter where they are located. They don’t have to access a shared file system. But it seems that relstorage just sends a link to the blob to the relational database.

    We recently just added relstorage in our application and it works very nicely without the authentication issues we had with zeo. However our zodb clients are not using shared filesystems, so the above breaks the blob support we had.

  7. Yes, that’s what I thought. But how come this is not required in the zeo blob implementation? Clientstorage seems to place whatever blobs are accessed as required on the local file system, but a centralised copy is also kept on the server. This just happens to make more sense for what I am trying to do, where clients cannot access a common NFS share. Is there any reason why relstorage couldn’t support this scheme too?
    Sorry if I’m not using the right terminology.

  10. Mikko, RelStorage 1.5.0a1 has the blob support you’re looking for. You can now store the blobs in Postgres. Like ZEO, an unshared blob cache directory is required.

Comments are closed.