Currently I'm having lots of fun experimenting with WSGI, repoze and Deliverance. But while it's nice to know that it works in a development setup (i.e. deployed with paster) I needed to be sure it would work well in a production environment. And while there are already instructions floating around on how to deploy it with Apache and mod_wsgi, I wanted to know whether I could deploy WSGI-based sites using my trusted workhorse nginx.

Since nginx is much more monolithic than Apache (which is one reason why it can be so noticably more efficient than Apache in certain situations) you can't just drop in a plugin or module. Instead, you must compile nginx from sources and add the module at compile time.

The projects I'm working on will be deployed on Ubuntu and FreeBSD, but of course I will want to be able to test the same setup on my OS X development machine. So I've begun my tests with Ubuntu and OS X. Since nginx is available in all three systems' packaging system, my strategy is to install nginx via its respective package (which will integrate it nicely with start- and shutdown scripts) and simply replace the nginx binary with a self-compiled version that includes mod_wsgi.

So, here it goes: download and expand the sources for nginx (currently 0.5.35) and mod_wsgi (currently version 0.0.6):

wget http://sysoev.ru/nginx/nginx-0.5.35.tar.gz
wget http://hg.mperillo.ath.cx/nginx/mod_wsgi/archive/0.0.6.tar.gz
tar xzf nginx-0.5.35.tar.gz
tar xzf 0.0.6.tar.gz

*By the way, it seems to be a feature of mercurial unknown to the author of mod_wsgi Manlio Perillo to provide .tgz archives not only for the tip but also for each tag. Currently the tip of mod_wsgi doesn't compile on Mac OS X so I'm sticking with version 0.0.6 which has proven to be stable and contains the config fixes for Mac OS X.*

For both Ubuntu and Mac OS X we will need to explicitly tell the mod_wsgi plugin to use Python 2.4 rather than the default 2.5 version that comes with both systems, since I'm intending to run Zope based applications:

$EDITOR mod_wsgi-0.0.6/config

Change the second line of the file to:

PYTHON='python2.4'

Ubuntu

On Ubuntu you will need to install the following packages:

sudo apt-get install gcc
sudo apt-get install python2.4-dev
sudo apt-get install libxslt-dev
sudo apt-get install libssl-dev
sudo apt-get install libpcre3-dev

To take advantage of the start- and stop mechanisms provided by the official nginx package, let's first install that:

sudo apt-get install nginx

Now we can change into the nginx source directory and configure the build process to replace the packaged version of nginx with one that includes mod_wsgi like so:

cd nginx-0.5.35
./configure --add-module=../mod_wsgi-0.0.6/ --prefix=/usr/local --sbin-path=/usr/sbin \
    --conf-path=/etc/nginx/nginx.conf --with-http_ssl_module

You should receive a summary that looks like this:

Configuration summary
  + threads are not used
  + using system PCRE library
  + using system OpenSSL library
  + md5 library is not used
  + sha1 library is not used
  + using system zlib library

  nginx path prefix: "/usr/local"
  nginx binary file: "/usr/sbin"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/usr/local/logs/nginx.pid"
  nginx error log file: "/usr/local/logs/error.log"
  nginx http access log file: "/usr/local/logs/access.log"
  nginx http client request body temporary files: "/usr/local/client_body_temp"
  nginx http proxy temporary files: "/usr/local/proxy_temp"
  nginx http fastcgi temporary files: "/usr/local/fastcgi_temp"

Make sure, the packaged instance of nginx is not running (we won't be able to replace it, otherwise):

sudo /etc/init.d/nginx stop

Now you can do the usual make ; sudo make install dance.

Before starting up the instance, we still need to run setup.py from the mod_wsgi folder:

cd ../mod_wsgi-0.0.6/
sudo python2.4 setup.py --prefix=/usr/local/ --sbin-path=/usr/sbin/ --conf-path=/etc/nginx/

Now you can start up your instance:

sudo /etc/init.d/nginx start

Mac OS X

On Mac OS X you will need to have the Developer Tools and MacPorts installed and the install the following in addition:

sudo port install python2.4
sudo port install libxslt # has libxml2 as auto-dependency
sudo port install py-libxml2
sudo port install nginx

For the configure process to find the 2.4 python libraries I found I needed to copy them to /opt/local/lib, as otherwise nginx would load the libraries of the system's 2.5 version at startup time which would throw mod_wsgi off track.

cp /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/libpython2.4.dylib /opt/local/lib/

Now we can configure it to match the nginx version from the ports collection like so:

./configure --add-module=../mod_wsgi-0.0.6/ --prefix=/opt/local --conf-path=etc/nginx/nginx.conf --sbin-path=sbin/ --with-http_ssl_module

Again, here's the summary output you should expect:

Configuration summary
  + threads are not used
  + using system PCRE library
  + using system OpenSSL library
  + md5 library is not used
  + sha1 library is not used
  + using system zlib library

  nginx path prefix: "/opt/local"
  nginx binary file: "/opt/local/sbin/"
  nginx configuration file: "/opt/local/etc/nginx/nginx.conf"
  nginx pid file: "/opt/local/logs/nginx.pid"
  nginx error log file: "/opt/local/logs/error.log"
  nginx http access log file: "/opt/local/logs/access.log"
  nginx http client request body temporary files: "/opt/local/client_body_temp"
  nginx http proxy temporary files: "/opt/local/proxy_temp"
  nginx http fastcgi temporary files: "/opt/local/fastcgi_temp"

Now you can do the usual make ; sudo make install dance.

Before starting up the instance, we still need to run setup.py from the mod_wsgi folder:

cd ../mod_wsgi-0.0.6/
sudo python2.4 setup.py --prefix=/opt/local/ --sbin-path=/opt/local/sbin/ --conf-path=/opt/local/etc/nginx/

Now we're finally ready to fire up our new instance. While testing and developing I can't be bothered to use launchctl so I chose a more pedestrian approach:

sudo killall nginx ; sudo /opt/local/sbin/nginx

Now you can take a look at the sample nginx.conf file provided in the examples directory of mod_wsgi to take the provided WSGI demos for a spin and, of course, to serve as a starting point to get your own apps running. Next I'll be looking at getting repoze.plone and repoze.grok running behind nginx+mod_wsgi, so stay tuned.

When doing maintenance work on a deployment site it is always a good idea (and in good taste) to show visitors a human-readable page explaining that you'll be back soon[tm] instead of letting them see ugly tracebacks etc.

Easy, right? Just put a static HTML file and configure it as the site root and you're done, right? Well, not quite. Because that will show the page to all visitors of the root of your site, but everybody else will get a 404.

Well, regular expressions and nginx to the rescue. The following configuration snippet displays the same index.html page to any request, not just the front page. I now keep it commented out in all my config files to be able to quickly turn on a maintenance message if something requires some fixing:

server {
    listen 80;
    server_name foobar.tld;

    root /usr/local/www/foobar/;
    index  index.html index.htm;
    location ^~ /index.html {
    }
    location / {
        rewrite "^/(.+)$" / last;
    }
}

The crucial part is the (empty) ^~ /index.html location that simply exists to stop an endless recursion that would take place otherwise. It also means, that your HTML file must be named index.html and reside in /usr/local/www/foobar/ on the file system.

Any non-trivial deployment of Plone usually consists of at least a web server 'sitting in front of' it, often enough accompanied by some sort of caching service. For my own purposes I have settled on nginx for the former and varnish for the latter[1].

They have proven to be a reasonably good team for me, especially in combination with recent versions of CacheFu, Plone's de facto default caching product. By 'reasonably good' I mean, that varnish and CacheFu in their 'normal' setup (i.e. with the 'official' Plone .vcl config) offer a reasonable trade-off between performance and potentially stale content.

I noticed this behaviour particularly when looking at Zope's access log (i.e. var/log/primary-Z2.log) while accessing my latest Plone installation at wahlcomputer.ccc.de. There was much too much going on, as theoretically there shouldn't be any requests reaching the backend until some content had changed. In essence, the site behaved much slower than it would need to by answering unnecessarily to requests that varnish should be able to satisfy.

However, in order to debug the caching and purging of content in detail I needed a local Mac OS X based setup that mirrored that on the FreeBSD box serving the site as closely, as possible. I habitually took notes while setting it up on my desktop machine and am now retracing my steps as I attempt to duplicate it on my Macbook (Just like a bug isn't fixable until it's reproducible, I never consider my admin work done until I have been able to do it at least twice). I'm confident that somebody out there will find the following notes useful, too (even if it's just myself six months^wweeks down the road)...

Since I'm a happy user of the macports collection already anyway, I let it do the 'heavy lifting' of actually installing nginx and varnish. In addition I provided a launchd startup item for varnish and also added a host entry for wahlcomputer to enable virtual hosting for nginx and varnish. Here it goes:

sudo port install nginx
sudo port install varnish

DNS

Before continuing the setup I first wanted to make sure that I could use the domain name wahlcomputer to access my machine. Instead of doing it 'properly' by adding a new record to OpenLDAP and thus possibly opening up what could amount to yet another can of worms I simply tried to edit ye' good ol' /etc/hosts file and added the following line:

127.0.0.1   wahlcomputer

And sure enough, even in the days of 'OpenLDAP FTW!!' Mac OS X still honours entries made to the hosts file so I could move on.

nginx

Next, I configured the freshly installed nginx. As the port already comes with a launchd start up item, I just needed to load it:

sudo launchctl load /opt/local/etc/LaunchDaemons/org.macports.nginx/org.macports.nginx.plist

It's my practice to leave the default files as untouched as possible and put all my configurations into separate files, that I simply include:

sudo mv /opt/local/etc/nginx/nginx.conf.default  /opt/local/etc/nginx/nginx.conf
sudo mkdir /opt/local/etc/nginx/includes
sudo touch /opt/local/etc/nginx/includes/wahlcomputer.conf
sudo mkdir /opt/local/var/log/nginx/

To make nginx aware of those files I added the following line just before the final closing }:

include etc/nginx/includes/*.conf;

Here's its contents in its full, unabridged glory (and VHM gore):

server {
    server_name wahlcomputer.ccc.de wahlcomputer;
    location / {
        proxy_pass http://127.0.0.1:8071/VirtualHostBase/http/wahlcomputer:80/foo/VirtualHostRoot/;
    }
}

For the changes to take effect, I needed to restart nginx like so (why, oh why, does launchctl with all its bells and whistles not have a simple reload or restart command?!):

sudo launchctl stop org.macports.nginx
sudo launchctl start org.macports.nginx

Now visiting http://wahlcomputer/ showed the front page of the Plone site and I could move on to varnish:

varnish

Unlike nginx, the varnish port (currently) does not come with a startup item, so I made my very own (by blatantly copying from the nginx setup):

sudo mkdir /opt/local/etc/varnish
sudo touch /opt/local/etc/varnish/default.vcl
sudo mkdir /opt/local/etc/LaunchDaemons/org.macports.varnish
sudo touch /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist
sudo chown root:wheel /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist
sudo chmod 644 /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist
sudo ln -s /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist /Library/LaunchDaemons/
sudo mkdir /opt/local/var/varnish
sudo chown -R _www /opt/local/var/varnish

Looking at varnishd's man page - HTTP accelerator daemon - Linux Manual - Digipedia") I saw that there are some options that must be passed in upon start up. I did this by adding them to the ProgramArguments array in the /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>KeepAlive</key>
    <false/>
    <key>Debug</key>
    <false/>
    <key>Label</key>
    <string>varnishd</string>
    <key>OnDemand</key>
    <false/>
    <key>GroupName</key>
    <string>staff</string>
    <key>UserName</key>
    <string>_www</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/local/sbin/varnishd</string>
        <string>-a</string>
        <string>localhost:6081</string>
        <string>-T</string>
        <string>localhost:6082</string>
        <string>-f</string>
        <string>/opt/local/etc/varnish/default.vcl</string>
    </array>
    <key>RunAtLoad</key>
    <false/>
</dict>
</plist>

Now all that was missing was to add some directives to the currently still empty file /opt/local/etc/varnish/default.vcl. It ended up being too long to quote here in its entirety, instead I will simply point you to a nice variation of the 'official zope-plone.vcl that I found that sports some neat enhancements.

Finally I could start up varnish:

sudo launchctl load /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist

In the nginx wahlcomputer.conf file all I needed to change was the port number from 8071 (the Zope instance) to 6081 (varnish).

By keeping a tail on the access logs of nginx and my Zope instance and looking at the response headers with firebug I can now start to tweak my setup to my heart's content. Stay tuned.

[1] A bit of an explanation is perhaps necessary as to why even bother with two types of machinery in front of Plone. The short answer is 'separation of concerns'. While varnish could theoretically take over most of the features that I use nginx for (namely, URL-rewriting, logging and static content delivery) I still would need it for SSL/HTTPS. Also, the logging feature of varnish requires an extra varnishlog process anyway.