python

Currently I'm having lots of fun experimenting with WSGI, repoze and Deliverance. But while it's nice to know that it works in a development setup (i.e. deployed with paster) I needed to be sure it would work well in a production environment. And while there are already instructions floating around on how to deploy it with Apache and mod_wsgi, I wanted to know whether I could deploy WSGI-based sites using my trusted workhorse nginx.

Since nginx is much more monolithic than Apache (which is one reason why it can be so noticably more efficient than Apache in certain situations) you can't just drop in a plugin or module. Instead, you must compile nginx from sources and add the module at compile time.

The projects I'm working on will be deployed on Ubuntu and FreeBSD, but of course I will want to be able to test the same setup on my OS X development machine. So I've begun my tests with Ubuntu and OS X. Since nginx is available in all three systems' packaging system, my strategy is to install nginx via its respective package (which will integrate it nicely with start- and shutdown scripts) and simply replace the nginx binary with a self-compiled version that includes mod_wsgi.

So, here it goes: download and expand the sources for nginx (currently 0.5.35) and mod_wsgi (currently version 0.0.6):

wget http://sysoev.ru/nginx/nginx-0.5.35.tar.gz
wget http://hg.mperillo.ath.cx/nginx/mod_wsgi/archive/0.0.6.tar.gz
tar xzf nginx-0.5.35.tar.gz
tar xzf 0.0.6.tar.gz

*By the way, it seems to be a feature of mercurial unknown to the author of mod_wsgi Manlio Perillo to provide .tgz archives not only for the tip but also for each tag. Currently the tip of mod_wsgi doesn't compile on Mac OS X so I'm sticking with version 0.0.6 which has proven to be stable and contains the config fixes for Mac OS X.*

For both Ubuntu and Mac OS X we will need to explicitly tell the mod_wsgi plugin to use Python 2.4 rather than the default 2.5 version that comes with both systems, since I'm intending to run Zope based applications:

$EDITOR mod_wsgi-0.0.6/config

Change the second line of the file to:

PYTHON='python2.4'

Ubuntu

On Ubuntu you will need to install the following packages:

sudo apt-get install gcc
sudo apt-get install python2.4-dev
sudo apt-get install libxslt-dev
sudo apt-get install libssl-dev
sudo apt-get install libpcre3-dev

To take advantage of the start- and stop mechanisms provided by the official nginx package, let's first install that:

sudo apt-get install nginx

Now we can change into the nginx source directory and configure the build process to replace the packaged version of nginx with one that includes mod_wsgi like so:

cd nginx-0.5.35
./configure --add-module=../mod_wsgi-0.0.6/ --prefix=/usr/local --sbin-path=/usr/sbin \
    --conf-path=/etc/nginx/nginx.conf --with-http_ssl_module

You should receive a summary that looks like this:

Configuration summary
  + threads are not used
  + using system PCRE library
  + using system OpenSSL library
  + md5 library is not used
  + sha1 library is not used
  + using system zlib library

  nginx path prefix: "/usr/local"
  nginx binary file: "/usr/sbin"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/usr/local/logs/nginx.pid"
  nginx error log file: "/usr/local/logs/error.log"
  nginx http access log file: "/usr/local/logs/access.log"
  nginx http client request body temporary files: "/usr/local/client_body_temp"
  nginx http proxy temporary files: "/usr/local/proxy_temp"
  nginx http fastcgi temporary files: "/usr/local/fastcgi_temp"

Make sure, the packaged instance of nginx is not running (we won't be able to replace it, otherwise):

sudo /etc/init.d/nginx stop

Now you can do the usual make ; sudo make install dance.

Before starting up the instance, we still need to run setup.py from the mod_wsgi folder:

cd ../mod_wsgi-0.0.6/
sudo python2.4 setup.py --prefix=/usr/local/ --sbin-path=/usr/sbin/ --conf-path=/etc/nginx/

Now you can start up your instance:

sudo /etc/init.d/nginx start

Mac OS X

On Mac OS X you will need to have the Developer Tools and MacPorts installed and the install the following in addition:

sudo port install python2.4
sudo port install libxslt # has libxml2 as auto-dependency
sudo port install py-libxml2
sudo port install nginx

For the configure process to find the 2.4 python libraries I found I needed to copy them to /opt/local/lib, as otherwise nginx would load the libraries of the system's 2.5 version at startup time which would throw mod_wsgi off track.

cp /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/libpython2.4.dylib /opt/local/lib/

Now we can configure it to match the nginx version from the ports collection like so:

./configure --add-module=../mod_wsgi-0.0.6/ --prefix=/opt/local --conf-path=etc/nginx/nginx.conf --sbin-path=sbin/ --with-http_ssl_module

Again, here's the summary output you should expect:

Configuration summary
  + threads are not used
  + using system PCRE library
  + using system OpenSSL library
  + md5 library is not used
  + sha1 library is not used
  + using system zlib library

  nginx path prefix: "/opt/local"
  nginx binary file: "/opt/local/sbin/"
  nginx configuration file: "/opt/local/etc/nginx/nginx.conf"
  nginx pid file: "/opt/local/logs/nginx.pid"
  nginx error log file: "/opt/local/logs/error.log"
  nginx http access log file: "/opt/local/logs/access.log"
  nginx http client request body temporary files: "/opt/local/client_body_temp"
  nginx http proxy temporary files: "/opt/local/proxy_temp"
  nginx http fastcgi temporary files: "/opt/local/fastcgi_temp"

Now you can do the usual make ; sudo make install dance.

Before starting up the instance, we still need to run setup.py from the mod_wsgi folder:

cd ../mod_wsgi-0.0.6/
sudo python2.4 setup.py --prefix=/opt/local/ --sbin-path=/opt/local/sbin/ --conf-path=/opt/local/etc/nginx/

Now we're finally ready to fire up our new instance. While testing and developing I can't be bothered to use launchctl so I chose a more pedestrian approach:

sudo killall nginx ; sudo /opt/local/sbin/nginx

Now you can take a look at the sample nginx.conf file provided in the examples directory of mod_wsgi to take the provided WSGI demos for a spin and, of course, to serve as a starting point to get your own apps running. Next I'll be looking at getting repoze.plone and repoze.grok running behind nginx+mod_wsgi, so stay tuned.

I've been toying with the idea of writing a web frontend for BitTorrent for a while now (Grok looks like a likely contender for that) but since the 'official' python based implementation of BitTorrent is simply much too inefficient I had to first come up with a Python library to some other BitTorrent implementation. In the end, I wrote my own library for the highly commendable Transmission client. And after some clean-up and polish work today, I'm happy to announce a release candidate.

The README, the documentation and the tests live all in the same document and the whole thing should be pretty straightforward to use. I'm curious, as to whether some other pythonistas might find this useful, too. We'll see.

Often, when writing a Plone Product I come across small methods that I need to implement, that actually have no dependency upon Plone itself but are too trivial and/or specific as to warrant putting them into a separate package. If that's the case for you, too, you will undoubtedly start to get impatient when testing these methods, because with each friggin' testrun an entire zope instance is created and torn down again. This I have become used to and accept it as part of life as Zope developer, if the code I'm testing actually needs that setup. But it just kept bugging me that I need to wait so long for tests that actually execute in under one second. So I've poked around today and came up with the following scenario:

1. put your methods into a separate module that has no import dependencies from zope
2. add a test runner to that module to execute its tests without starting up zope
3. create a test suite for your product that will pick up the tests of your module
   

1. standalone module

def batch(items=[], batch=0, batchsize=5):
    """returns a batch from the given items

        >>> items = range(12)
        >>> batch(items, 3, 5)
        Traceback (most recent call last):
            ...
        IndexError: No batch number 3

        >>> batch(items, 0, 5)
        [0, 1, 2, 3, 4]

        >>> batch(items, 1, 5)
        [5, 6, 7, 8, 9]

        >>> batch(items, 2, 5)
        [10, 11]

        >>> batch(items, 2, 4)
        [8, 9, 10, 11]
    """
    if batch >= batchnum(len(items), batchsize):
        raise IndexError, \
            "No batch number %d" % batch
    else:
        return items[batch * batchsize:(batch+1) * batchsize]

from Products.PRODUCTNAME.batching import batch

2. add a test runner

if __name__ == "__main__":
    import doctest
    doctest.testmod()

3. Zope integration

import unittest
from unittest import TestSuite
from zope.testing.doctestunit import DocTestSuite

def test_suite():
    return TestSuite((
        DocTestSuite('Products.PRODUCTNAME.batching'),
        ))

if __name__ == '__main__':
    unittest.main(defaultTest="test_suite")

from unittest import TestSuite, main
from Testing.ZopeTestCase import ZopeDocTestSuite

def test_remote_browsing():
    """
    By importing directly from zope.testbrowser.browser we can use
    the Browser API also for testing remote (non-Zope) sites:

      >>> from zope.testbrowser.browser import Browser
      >>> browser = Browser()
      >>> browser.open('http://en.wikipedia.org/wiki/Monty_Python')
      >>> browser.url
      'http://en.wikipedia.org/wiki/Monty_Python'
      >>> browser.contents
      '...Dead Parrot...'
   """

def test_suite():
    return ZopeDocTestSuite()

if __name__ == '__main__':
    main(defaultTest='test_suite')

Sean (Kelly? No, Fulmer!) pointed me to an interesting article by Micheal Thornhill (who, sadly, seems to have stopped blogging) in a comment to my previous entry on using pdb with tab-completion. I've had this article in my TO READ list for several months now but was always put off by the sheer magnitude and multitude of tips it contained, but today (since I should actually be doing my income taxes...) I ventured into it in search for the one snippet of code, that would simply allow me to use ipython as a shell for accessing a ZEO server i.e. as replacement for Ploneshell. And after going through this shell script I came up with the following snippet that will let me do that (and only that!) simply by pointing it to a ZOPE instance:

#!/bin/bash
ZOPE_HOME=$1
if [ -z $ZOPE_HOME ] 
    then echo "usage: ipython-zope.sh <PATH-TO-ZOPE_HOME>"
    exit ;
fi
PYTHON=`which python`
set -x
$PYTHON -i -c "from Zope2 import configure;configure \ 
  ('$ZOPE_HOME/etc/zope.conf');import Zope2; app=Zope2.app(); \
  ns={'__name__':'blah','app':app}; \
  import IPython;IPython.Shell.IPShell(user_ns=ns).mainloop(sys_exit=1);"

Note: I've inserted newlines to preserve a reasonable linewidth, you can download a executable version of the script here.

Now, if only somebody could figure out a way to use ipython as a shell for pdb, that would be... for lack of a better word: awesome ;-)