Tag: Buildout

PyCon 2010: I Want to Present Something

I’ve been racking my brains to find something to present at PyCon 2010. I have been trying to find something good to present since PyCon 2003, when I last presented at PyCon.  (I talked about Ape, the Adaptable Persistence Engine for Zope.)  I really liked the experience of presenting and it led to a lot of interesting conversations. Since then, however, nothing has really struck me as a good idea for a presentation. The right idea has to fit at least these criteria:

  • It’s something I’m good at.
  • At least a handful of PyCon attendees would like to learn more about it in a presentation.
  • I can’t spend weeks to prepare for the presentation.

This year, I was planning to do a really fun presentation by writing some Python scripts for controlling a RepRap, then I was going to present the hardware and software.  That didn’t work out, however, because the warping issues with ABS (a type of plastic) are just too severe to print useful parts, so my RepRap has sat idle.  (I’m now considering PLA, but it’s a newish and expensive material.)

Here are a few other ideas for a presentation topic.  What do you think?  Any other ideas?

  • RelStorage.  I could talk about future plans, why I think it is an improvement over ZEO, and why you should use it.  There seem to be a lot of people with questions about RelStorage, so it would be nice to have some time to answer the questions in a big room so others can hear the answers.
  • KARL and BFG.  This would probably be a team presentation.  KARL is some fairly interesting software that I got to help develop this year.  I could talk about the software, along with the development experience and style.  In KARL we made a conscious choice to ignore certain apparent DRY violations, leading to significant productivity gains.
  • A friendly introduction to Buildout.  Buildout is a tool that a lot of developers need, but don’t know it yet.  The function that Buildout performs is as important as version control and automated testing.  Come find out why Buildout is far better than a pile of Makefiles.
  • An introduction to Buildout (zc.buildout) for people familiar with Apache Maven.  Buildout and Maven fill approximately the same niche, but for different audiences.  (Buildout for Python, Maven for Java.)  Maybe there are Mavenites at the conference who would like to switch to a more Python centric system.
  • A discussion of text indexing in Plone and BFG.  This might be a narrow topic, but I find it interesting and important.  I have found ways to reduce complex 90 second text searches to 1 second.  The solution is not pure Python, unfortunately. 😉  I have also thought about how to expand into areas like faceted search/browse functionality.

Feedback encouraged!

How to Install Plone with RelStorage and MySQL

These step by step instructions describe how to install Plone on Ubuntu with RelStorage connected to MySQL as the main database. Familiarity with Linux systems administration is expected. Update: These instructions were revised in August 2009 for Plone 3.2.3 and RelStorage 1.2.0.

Continue reading How to Install Plone with RelStorage and MySQL

Easy Workaround for zc.buildout

Problem: running “python bootstrap.py” or “bin/buildout” often produces scripts that mix up the Python package search path due to some packages being installed system-wide.  Version conflicts result.

Workaround: use “python -S bootstrap.py” and “python -S bin/buildout”.  Magically, no more version conflicts.

I wish I had thought of that before.  Duh!

Update: Another tip for new zc.buildout users I’ve been meaning to mention is that you should create a preferences file in your home directory so that downloads and eggs are cached centrally.  This makes zc.buildout much friendlier.  Do this:

mkdir ~/.buildout
echo "[buildout]" >> ~/.buildout/default.cfg
echo "eggs-directory = $HOME/.buildout/eggs" >> ~/.buildout/default.cfg
echo "download-cache = $HOME/.buildout/cache" >> ~/.buildout/default.cfg

It seems a bit silly that zc.buildout doesn’t have these settings by default.  They make zc.buildout behave a lot like Apache Maven, which is what a lot of Java shops are using these days.  Both zc.buildout and Maven are great tools once you get to know them, but both are a real pain to understand at first.

Bootstrap.py versus pkg_resources.py

I’ve been using zc.buildout quite a bit over the past month.  Although it has been working, it has been doing strange things like using the wrong version of zope.interface.  Yesterday I finally figured out why, and today I found a possible solution.

It turns out that Ubuntu (8.10) provides a package called python-pkg-resources.  At least one Ubuntu package (Snowballz, a strategy game written in Python) pulls in that package automatically.  It installs a pkg_resources module in Python’s site-packages directory, but it does not install the rest of setuptools.

I can understand why Ubuntu chose to split up setuptools, but that choice causes havoc for the bootstrap.py module people use to install zc.buildout.  Here is what bootstrap.py is supposed to do:

  1. Download ez_setup.py and run it.
  2. ez_setup tries to import the pkg_resources module, but fails.
  3. The setuptools package is not found, so ez_setup downloads setuptools in a temporary directory.
  4. ez_setup alters sys.path to include the new setuptools package.
  5. bootstrap.py imports the pkg_resources module from the version of setuptools just downloaded.
  6. Ask pkg_resources about the installed setuptools package.
  7. Use setuptools to install zc.buildout.

Here is what bootstrap.py actually does when pkg_resources.py exists in the site-packages directory (differences emphasized):

  1. Download ez_setup.py and run it.
  2. ez_setup successfully imports the pkg_resources module from site-packages.
  3. The setuptools package is not found, so ez_setup downloads setuptools in a temporary directory.
  4. ez_setup alters sys.path to include the new setuptools package.
  5. boostrap.py continues to use the previously imported pkg_resources module.
  6. Ask pkg_resources about the installed setuptools package.
  7. pkg_resources does not find setuptools because pkg_resources does not notice the change to sys.path.  bootstrap.py fails.

At first, following ideas I gleaned from various posts about zc.buildout, I worked around this by deleting the setuptools egg and the pkg_resources module from site-packages.  I didn’t know exactly why this helped until I studied the problem.  It turns out that bootstrap.py was just not written to cope with a system-wide installation of pkg_resources.

Now I think I recognize another bad choice that zc.buildout has been making.  zc.buildout generates a “bin” directory full of Python scripts.  Those scripts prepend egg directories and egg zip files to sys.path before doing their work.  I noticed that sometimes the list of paths to prepend includes “/usr/lib/python2.5/site-packages”, which is already on sys.path.  I now suspect that whenever zc.buildout includes paths like that, it’s wrong, and the cause is a mixup involving a system-wide installation of pkg_resources, setuptools, or some other foundational package.

Here is a possible way to fix bootstrap.py.  Just before the “import pkg_resources” line, add this:

del sys.modules[‘pkg_resources’]

This solved the bootstrap.py problem for me.  Altering sys.modules is rarely a good idea, but this might be a good exception to the rule.  I don’t believe we need to catch KeyError because ez_setup should have imported pkg_resources already.

Beyond this, there is probably more work to do to make zc.buildout produce correct scripts.

Whoever said computers behave logically must have been joking or delusional.  The people who provide the software never fully agree with each other–nor even themselves!