Tag: Python

RelStorage 1.2.0b2 Released

This release works with unpatched versions of ZODB 3.9!  A big thank-you to Jim Fulton for including support for RelStorage in ZODB.  This release also continues to support patched versions of ZODB 3.7 and 3.8.

I have been doing a lot of testing, and I have found MySQL 5.1.34 to be a lot more stable than earlier releases of MySQL 5.1, so I am now declaring MySQL 5.1.34 and above supportable, meaning that if you ask questions about it, I am no longer going to request that you revert to MySQL 5.0. 🙂

Finally, I recently expanded my private RelStorage Buildbot to include a Windows XP slave.  After solving a couple of minor test glitches, the test results are now all consistently green on 4 platforms:

  • Debian Etch, 32 bit (Python 2.4.4, MySQL 5.0.32, PostgreSQL 8.1.17, Oracle 10g XE)
  • Debian Lenny, 32 bit (Python 2.5.2, MySQL 5.0.51a, PostgreSQL 8.3.7, Oracle 10g XE)
  • Debian Lenny, 64 bit (same as above but no Oracle)
  • Windows XP, 32 bit (Python 2.6.2, MySQL 5.1.34, PostgreSQL 8.3.7)

I’m thinking about adding another Linux slave that runs MySQL 5.1 and Python 2.6.

Anyway, enjoy the release!

P.S. You may be wondering why I released 1.2.0b2 instead of 1.2.0b1.  A little slip ruined the web page on PyPI, so I fixed the slip and skipped to the next version number.

Anecdotal Evidence

I like to believe that I am a competent software developer in both Python and Java.  As a competent developer, I find that certain things are generally much easier than other things.

For instance, I just spent a frustrating week working out how to install a Shibboleth identity provider, yet I never got it working quite satisfactorily.  Then, after spending 30 minutes with Python-openid, I had an identity provider server running and I already felt quite confident with it.

It’s as if Java and Python are not really in the same league.  There is no way to prove that, though.  Sometimes Java wins anyway.

Easy Workaround for zc.buildout

Problem: running “python bootstrap.py” or “bin/buildout” often produces scripts that mix up the Python package search path due to some packages being installed system-wide.  Version conflicts result.

Workaround: use “python -S bootstrap.py” and “python -S bin/buildout”.  Magically, no more version conflicts.

I wish I had thought of that before.  Duh!

Update: Another tip for new zc.buildout users I’ve been meaning to mention is that you should create a preferences file in your home directory so that downloads and eggs are cached centrally.  This makes zc.buildout much friendlier.  Do this:

mkdir ~/.buildout
echo "[buildout]" >> ~/.buildout/default.cfg
echo "eggs-directory = $HOME/.buildout/eggs" >> ~/.buildout/default.cfg
echo "download-cache = $HOME/.buildout/cache" >> ~/.buildout/default.cfg

It seems a bit silly that zc.buildout doesn’t have these settings by default.  They make zc.buildout behave a lot like Apache Maven, which is what a lot of Java shops are using these days.  Both zc.buildout and Maven are great tools once you get to know them, but both are a real pain to understand at first.

Egg Patching Solution #3

Martijn Faasen suggested this solution in a comment on my previous post and I think it’s the best.  I created a new service:

http://packages.willowrise.org

I simply posted a patched ZODB3 source distribution on a virtual-hosted server.  The first tarball, “ZODB3-3.8.1-polling-serial.tar.gz”, includes both the invalidation polling patch and the framework I created for plugging in data serialization formats other than pickles, but in the near future I plan to also post distributions with just the polling patch and some eggs for Windows users.

It would not make sense for me to post the patched tarballs and eggs on PyPI because I don’t want people to pull these patched versions accidentally.  Pulling these needs to be an explicit step.

Thanks to setuptools and zc.buildout, it turns out that creating a Python code distribution server is a piece of cake.  The buildout process scans the HTML pages on distribution servers for <a> links.  Any link that points to a tarball or egg with a version number is considered a candidate.  A static web site can easily fulfill these requirements. I imagine it gets deeper than that, but for now, that’s all I need.

To use this tarball, buildout.cfg just needs to include lines something like:

[buildout]
find-links = http://packages.willowrise.org
versions = versions

[versions]
ZODB3 = 3.8.1-polling-serial

zc.buildout does the rest.

It took a while to find this solution because, upon encountering the need to distribute patched eggs, I guessed it would be difficult to set up and maintain my own package distribution server. I also guessed setuptools had no support for patches in its versioning scheme. I’m glad I was completely wrong.

By the way, Ian suggested pip as a solution, but I don’t yet see how it helps. I am interested. I hope to see more of pip on Ian’s great blog.

Home Grown RepRap Software

After I switched my RepRap to run on g-codes a few weeks ago, I started writing little Python scripts to do various things like lay down a raft, shut the extruder off and move the platform out of the way, extrude in preparation for a build, and send a g-code file to the controller.  All of the scripts rely on a common module that sends a series of g-codes to the controller and waits for acknowledgements.  The code has grown and become more interesting:

  • I optimized the communication by having the host send enough commands to keep the controller’s 128 byte serial buffer full.  The host does this by sending commands before an acknowledgement has been received for commands sent earlier.  This works surprisingly well and helps the controller handle many commands per second, which is important for drawing short segments.
  • I added code that automatically resumes a build where it left off if the controller restarts during the build.  I implemented this because my controller spontaneously restarted several times this week in the middle of a build.  I implemented this feature by creating a simple, fast machine simulator in the host.  To resume a build, the code resets the controller, sends the g-codes necessary to put the machine back in the state recorded by the simulator, then resubmits the commands the controller never acknowledged.  This also works surprisingly well.
  • I created a g-code runner that can resume any stopped build.  I implemented this because yesterday my controller not only spontaneously restarted, it actually lost its firmware in the process!  (I then discovered and fixed a loosely connected ground wire which could have been the culprit all along.)  Now I can stop a build at any time, turn off the power, disconnect the USB cable, clean the heater, restart the host computer, turn on the power and reconnect, upload a new firmware, run some g-code to test and prime the RepRap, and finally resume printing exactly where I left off.  It’s kind of magical to see it actually work.

So now I have worked around the most common ways a print can fail partway through.  This should make it much easier to work through problems with large builds.  I can even stop a build with ctrl-C, shut everything off, go to bed, and resume the build another day!  I definitely need to start doing that. 😉

By the way, I had to make a minor update to the g-code firmware to make it possible to restore all of the machine state: the G92 code needs to notice and use the X, Y, and Z parameters.  The fact that no one has yet done this to the firmware in Subversion suggests that none of the other g-code runners can resume a build like mine can.  So I guess this code might be useful to the community.  Speak up if you’re interested.