Bootstrap.py versus pkg_resources.py

I’ve been using zc.buildout quite a bit over the past month.  Although it has been working, it has been doing strange things like using the wrong version of zope.interface.  Yesterday I finally figured out why, and today I found a possible solution.

It turns out that Ubuntu (8.10) provides a package called python-pkg-resources.  At least one Ubuntu package (Snowballz, a strategy game written in Python) pulls in that package automatically.  It installs a pkg_resources module in Python’s site-packages directory, but it does not install the rest of setuptools.

I can understand why Ubuntu chose to split up setuptools, but that choice causes havoc for the bootstrap.py module people use to install zc.buildout.  Here is what bootstrap.py is supposed to do:

1. Download ez_setup.py and run it.
2. ez_setup tries to import the pkg_resources module, but fails.
3. The setuptools package is not found, so ez_setup downloads setuptools in a temporary directory.
4. ez_setup alters sys.path to include the new setuptools package.
5. bootstrap.py imports the pkg_resources module from the version of setuptools just downloaded.
6. Ask pkg_resources about the installed setuptools package.
7. Use setuptools to install zc.buildout.

Here is what bootstrap.py actually does when pkg_resources.py exists in the site-packages directory (differences emphasized):

1. Download ez_setup.py and run it.
2. ez_setup successfully imports the pkg_resources module from site-packages.
3. The setuptools package is not found, so ez_setup downloads setuptools in a temporary directory.
4. ez_setup alters sys.path to include the new setuptools package.
5. boostrap.py continues to use the previously imported pkg_resources module.
6. Ask pkg_resources about the installed setuptools package.
7. pkg_resources does not find setuptools because pkg_resources does not notice the change to sys.path.  bootstrap.py fails.

At first, following ideas I gleaned from various posts about zc.buildout, I worked around this by deleting the setuptools egg and the pkg_resources module from site-packages.  I didn’t know exactly why this helped until I studied the problem.  It turns out that bootstrap.py was just not written to cope with a system-wide installation of pkg_resources.

Now I think I recognize another bad choice that zc.buildout has been making.  zc.buildout generates a “bin” directory full of Python scripts.  Those scripts prepend egg directories and egg zip files to sys.path before doing their work.  I noticed that sometimes the list of paths to prepend includes “/usr/lib/python2.5/site-packages”, which is already on sys.path.  I now suspect that whenever zc.buildout includes paths like that, it’s wrong, and the cause is a mixup involving a system-wide installation of pkg_resources, setuptools, or some other foundational package.

Here is a possible way to fix bootstrap.py.  Just before the “import pkg_resources” line, add this:

del sys.modules[‘pkg_resources’]

This solved the bootstrap.py problem for me.  Altering sys.modules is rarely a good idea, but this might be a good exception to the rule.  I don’t believe we need to catch KeyError because ez_setup should have imported pkg_resources already.

Beyond this, there is probably more work to do to make zc.buildout produce correct scripts.

Whoever said computers behave logically must have been joking or delusional.  The people who provide the software never fully agree with each other–nor even themselves!