Blog

lxml on Mac OS X

Tom Lazar — Thu, 25 Sep 2008 11:33:20 +0000

For a current project I needed to include (the most excellent Python XML library) lxml. Getting the right version to run on Mac OS X has been notoriously difficult, though. Easy-installing it was broken (and currently is, for me, as well) and while I have found multiple how-tos, they all dealt with the problem by installing it manually from sources and patching the configure process etc. pp. That would, of course, defeat the whole purpose of using buildout in the first place.

Cheeseshop to the rescue: there is a recipe for installing lxml along with the proper versions of libxml and libxslt, however, its default values didn't work for me. So for the record: the following buildout snippet will get you non-segfaulting lxml on (at least) Mac OS X 10.5.x and Ubuntu linux. YMMV.

parts =
    [...]
    lxml

eggs =
    [...]
    lxml==2.1.2

[lxml]
recipe=plone.recipe.lxml
egg = 
    lxml==2.1.2

libxml2_url = 
    http://xmlsoft.org/sources/libxml2-2.7.1.tar.gz

libxslt_url =
    http://xmlsoft.org/sources/libxslt-1.1.24.tar.gz

p.s. this should be good news for all folks working with Deliverance. Perhaps now I could revisit that...

Netboot installing Ubuntu on a SunFire X2200

Tom Lazar — Mon, 02 Jun 2008 18:10:00 +0000

After receiving my new toy at work three days ago, a SunFire X2200 with 16Gb RAM, today I finally got around to set it up.

Now, while my server OS of choice is definitely FreeBSD this particular machine is going to be used as a development machine for a Plone project which will eventually be hosted at a provider who doesn't support FreeBSD but instead Ubuntu. Also, the machine came without an optical drive. So after a few fruitless attempts to boot from an USB stick, I simply stuck to what I already know and chose a TFTP based route using Mac OS X's built-in tftp server.

Basically, I just followed my previous how-to on how to install FreeBSD on a (headless, keyboardless) soekris machine. Only, this time, I installed the dhcp server from mac ports, instead of compiling it manually:

sudo port install dhcp

Then I downloaded the Ubuntu Server ISO image and mounted it.

In my dhcpd.conf I modified the root-path and filename options thus:

option root-path "/Volumes/Ubuntu-Server 8./install/netboot/";
filename "pxelinux.0";

Next modified the (existing) /System/Library/LaunchDaemons/tftp.plist so that it looked like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" 
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>InitGroups</key>
    <true/>
    <key>Label</key>
    <string>com.apple.tftpd</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/libexec/tftpd</string>
        <string>-i</string>
        <string>/Volumes/Ubuntu-Server 8./install/netboot</string>
        <string>-u</string>
        <string>root</string>
        <string>-s</string>
        <string>/Volumes/Ubuntu-Server 8./install/netboot</string>
        <string>-l</string>
    </array>
    <key>Sockets</key>
    <dict>
        <key>Listeners</key>
        <dict>
            <key>SockServiceName</key>
            <string>tftp</string>
            <key>SockType</key>
            <string>dgram</string>
        </dict>
    </dict>
    <key>inetdCompatibility</key>
    <dict>
        <key>Wait</key>
        <true/>
    </dict>
</dict>
</plist>

Now all I needed to do was to start both daemons...

sudo /opt/local/sbin/dhcpd 
sudo service tftp start

...finally fire up the machine and press F12 at the boot prompt and Bob was my proverbial uncle.

blueprint css + jquery

Tom Lazar — Sat, 26 Apr 2008 17:01:13 +0000

I'm using the blueprint CSS framework for a new site that I'm working on and am rather enjoying the process (as far as that is humanly possible, given that it by definition involves working with CSS... cough).

One of the neat features of blueprint is a compressor script that not only renders all the various CSS bits that you want according to the given column numbers and sizes for each project into one neat (and compact) CSS file, but it also lets you define semantic classes and ids, so you can use something like class="navigation" instead of class="span-6 last". And on top of that, the compressor script will also generate a PNG image that conforms to the column size and padding you've selected. This proved to be really helpful in debugging the pages. Simply add the class showgrid to your outer most container div and presto!

When demonstrating the layout to the client, though, I wanted a less obtrusive way of switching the grid on and off and I thus came up with little jQuery script to generate a toggle switch for each page. And since jQuery will be the standard javascript library for Plone from version 3.1 onward, anyway I thought I might as well share the following snippet:

$(document).ready(function() {
    $("body").append("<div id='debug'>turn grid: <a href='' id='togglegrid'>" 
    + gridstate() + "</a></div>";);
    $("#debug").css("position", "absolute");
    $("#debug").css("bottom", "0");
    $("#togglegrid").click(toggle_grid);
});

function toggle_grid () {
    $(".container").toggleClass("showgrid");
    $("#togglegrid").text(gridstate());
    return false;
}

function gridstate () {
    if ($(".container").hasClass("showgrid")) {
        return 'off';
    } else {
        return 'on';
    };
}

P.S. I'm not exactly a fan of the Javascript language but I do appreciate jQuery a lot and for the sanity that it partially restores when working with Javascript. The snippet above is a good example of that in my opinion. Javascript can't get any more "pythonic" than that IMHO ;-) (But please prove me wrong!)

Connecting Plone to Mac OS X Server with LDAP

Tom Lazar — Fri, 18 Apr 2008 19:38:43 +0000

The idea is, of course, that all (or just some) of your OS X Server users can authenticate against a Plone instance using the same credentials that they use to access all the other services (usually filesharing).

Requirements

I'm assuming a buildout based setup, so you will need to add the following bits to your buildout.cfg:

[buildout]
parts =
    ...
    productdistros
    openldap
...
[openldap]
recipe = zc.recipe.cmmi
url = http://www.openldap.org/software/download/OpenLDAP/openldap-release/openldap-2.3.27.tgz 
extra_options= --disable-slapd --disable-backends
...
[productdistros]
recipe = plone.recipe.distros
urls =
    ...
    http://plone.org/products/ploneldap/releases/1.0/PloneLDAP-bundle-1.0.tar.gz

nested-packages =
    ...
    PloneLDAP-bundle-1.0.tar.gz

You will also need python-ldap, for which even some eggs exist, however, I found that they didn't work on my test server (Ubuntu, 64bit) as they seem to have some .so files that assume a 32bit architecture (just a wild guess on my part), so instead I just installed it via apt-get (Ubuntu really has good support for Python2.4 based packages btw, no wonder it's so popular among Zopistas and Plonistas!)

sudo apt-get install python2.4-ldap

Now you can run ./bin/buildout and restart your instance.

Adding the plug-in

In the ZMI, navigate to your Plone instance's acl_users and add a Plone LDAP Plugin from the upper right hand select box. Obviously, filling out the following form with exactly the right values is the trickiest bit of the entire operation, so I've tried to make the example values as self-evident as possible. They all assume that the FQDN of your Mac OS X server is my.ldap.server.tld, so it should be a no-brainer to substitute all values according to your own setup.

Set all three mappings (for Login Name Attribute, User ID Attribute and RDN Attribute) to UID (uid).
Users Base DN to cn=users,dc=my,dc=ldap,dc=server,dc=tld
Groups Base DN to cn=groups,dc=my,dc=ldap,dc=server,dc=tld
Manager DN to uid=diradmin,cn=users,dc=my,dc=ldap,dc=server,dc=tld. You will obviously need to substitute diradmin for the id you chose when setting up the OpenDirectory server. Hint: it's the same id you use to log into the Workgroup Manager ;-)
I have switched off encryption and SSL in my tests, so no guarantees that it will work with encryption (my Plone instance is running inside a VMware Fusion instance on the OS X Server itself, so I didn't see any need to bother with encryption, for a change)

Configuring the plug-in

Now you need to click on the newly created plugin at /plone/acl_users/ldap and activate all functionalities.

Still at /plone/acl_users/ldap click on Properties and User_Management and move the ldap plug-in to the top in both forms.

Finally, navigate to /plone/acl_users/ldap/acl_users and change the value for User object classes to posixAccount.

You now should be able to log into the Plone site using the credentials of a OS X Server user.

nginx + mod_wsgi + python2.4

Tom Lazar — Wed, 16 Apr 2008 12:28:22 +0000

Currently I'm having lots of fun experimenting with WSGI, repoze and Deliverance. But while it's nice to know that it works in a development setup (i.e. deployed with paster) I needed to be sure it would work well in a production environment. And while there are already instructions floating around on how to deploy it with Apache and mod_wsgi, I wanted to know whether I could deploy WSGI-based sites using my trusted workhorse nginx.

Since nginx is much more monolithic than Apache (which is one reason why it can be so noticably more efficient than Apache in certain situations) you can't just drop in a plugin or module. Instead, you must compile nginx from sources and add the module at compile time.

The projects I'm working on will be deployed on Ubuntu and FreeBSD, but of course I will want to be able to test the same setup on my OS X development machine. So I've begun my tests with Ubuntu and OS X. Since nginx is available in all three systems' packaging system, my strategy is to install nginx via its respective package (which will integrate it nicely with start- and shutdown scripts) and simply replace the nginx binary with a self-compiled version that includes mod_wsgi.

So, here it goes: download and expand the sources for nginx (currently 0.5.35) and mod_wsgi (currently version 0.0.6):

wget http://sysoev.ru/nginx/nginx-0.5.35.tar.gz
wget http://hg.mperillo.ath.cx/nginx/mod_wsgi/archive/0.0.6.tar.gz
tar xzf nginx-0.5.35.tar.gz
tar xzf 0.0.6.tar.gz

*By the way, it seems to be a feature of mercurial unknown to the author of mod_wsgi Manlio Perillo to provide .tgz archives not only for the tip but also for each tag. Currently the tip of mod_wsgi doesn't compile on Mac OS X so I'm sticking with version 0.0.6 which has proven to be stable and contains the config fixes for Mac OS X.*

For both Ubuntu and Mac OS X we will need to explicitly tell the mod_wsgi plugin to use Python 2.4 rather than the default 2.5 version that comes with both systems, since I'm intending to run Zope based applications:

$EDITOR mod_wsgi-0.0.6/config

Change the second line of the file to:

PYTHON='python2.4'

Ubuntu

On Ubuntu you will need to install the following packages:

sudo apt-get install gcc
sudo apt-get install python2.4-dev
sudo apt-get install libxslt-dev
sudo apt-get install libssl-dev
sudo apt-get install libpcre3-dev

To take advantage of the start- and stop mechanisms provided by the official nginx package, let's first install that:

sudo apt-get install nginx

Now we can change into the nginx source directory and configure the build process to replace the packaged version of nginx with one that includes mod_wsgi like so:

cd nginx-0.5.35
./configure --add-module=../mod_wsgi-0.0.6/ --prefix=/usr/local --sbin-path=/usr/sbin \
    --conf-path=/etc/nginx/nginx.conf --with-http_ssl_module

You should receive a summary that looks like this:

Configuration summary
  + threads are not used
  + using system PCRE library
  + using system OpenSSL library
  + md5 library is not used
  + sha1 library is not used
  + using system zlib library

  nginx path prefix: "/usr/local"
  nginx binary file: "/usr/sbin"
  nginx configuration file: "/etc/nginx/nginx.conf"
  nginx pid file: "/usr/local/logs/nginx.pid"
  nginx error log file: "/usr/local/logs/error.log"
  nginx http access log file: "/usr/local/logs/access.log"
  nginx http client request body temporary files: "/usr/local/client_body_temp"
  nginx http proxy temporary files: "/usr/local/proxy_temp"
  nginx http fastcgi temporary files: "/usr/local/fastcgi_temp"

Make sure, the packaged instance of nginx is not running (we won't be able to replace it, otherwise):

sudo /etc/init.d/nginx stop

Now you can do the usual make ; sudo make install dance.

Before starting up the instance, we still need to run setup.py from the mod_wsgi folder:

cd ../mod_wsgi-0.0.6/
sudo python2.4 setup.py --prefix=/usr/local/ --sbin-path=/usr/sbin/ --conf-path=/etc/nginx/

Now you can start up your instance:

sudo /etc/init.d/nginx start

Mac OS X

On Mac OS X you will need to have the Developer Tools and MacPorts installed and the install the following in addition:

sudo port install python2.4
sudo port install libxslt # has libxml2 as auto-dependency
sudo port install py-libxml2
sudo port install nginx

For the configure process to find the 2.4 python libraries I found I needed to copy them to /opt/local/lib, as otherwise nginx would load the libraries of the system's 2.5 version at startup time which would throw mod_wsgi off track.

cp /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/libpython2.4.dylib /opt/local/lib/

Now we can configure it to match the nginx version from the ports collection like so:

./configure --add-module=../mod_wsgi-0.0.6/ --prefix=/opt/local --conf-path=etc/nginx/nginx.conf --sbin-path=sbin/ --with-http_ssl_module

Again, here's the summary output you should expect:

Configuration summary
  + threads are not used
  + using system PCRE library
  + using system OpenSSL library
  + md5 library is not used
  + sha1 library is not used
  + using system zlib library

  nginx path prefix: "/opt/local"
  nginx binary file: "/opt/local/sbin/"
  nginx configuration file: "/opt/local/etc/nginx/nginx.conf"
  nginx pid file: "/opt/local/logs/nginx.pid"
  nginx error log file: "/opt/local/logs/error.log"
  nginx http access log file: "/opt/local/logs/access.log"
  nginx http client request body temporary files: "/opt/local/client_body_temp"
  nginx http proxy temporary files: "/opt/local/proxy_temp"
  nginx http fastcgi temporary files: "/opt/local/fastcgi_temp"

Now you can do the usual make ; sudo make install dance.

Before starting up the instance, we still need to run setup.py from the mod_wsgi folder:

cd ../mod_wsgi-0.0.6/
sudo python2.4 setup.py --prefix=/opt/local/ --sbin-path=/opt/local/sbin/ --conf-path=/opt/local/etc/nginx/

Now we're finally ready to fire up our new instance. While testing and developing I can't be bothered to use launchctl so I chose a more pedestrian approach:

sudo killall nginx ; sudo /opt/local/sbin/nginx

Now you can take a look at the sample nginx.conf file provided in the examples directory of mod_wsgi to take the provided WSGI demos for a spin and, of course, to serve as a starting point to get your own apps running. Next I'll be looking at getting repoze.plone and repoze.grok running behind nginx+mod_wsgi, so stay tuned.

Today is Naked CSS Day

Tom Lazar — Wed, 09 Apr 2008 15:27:03 +0000

Out of a whim I decided to join today's CSS Naked Day. After all, as a Plone Site there is no need to be ashamed :-)

Deliverance á la WSGI for Plone

Tom Lazar — Tue, 08 Apr 2008 18:01:42 +0000

Deliverance is the New Kid on the Block[tm] in the Plone world. I find the whole concept very intriguing and have decided to give it a try on both of my current new projects. (No risk, no fun, eh!?)

One of the two projects is smaller in scope where I am both the Zope/Plone developer and the HTML/CSS handyman and the other a larger affair together with another Zopista and an external web agency with up to five more people, none of which is experienced with - or much interested in - Plone, so I am curious, how Deliverance will hold up in these two quite different scenarios.

Now, if you just want to dip your toes into the proverbial water, I suggest you check out Martin Aspeli's Deliverance recipe (does that man ever sleep?) However, Martin's recipe, along with any other tutorial that I found floating about, uses deliverance in its so-called proxy mode where a second daemon is fired up that mediates between the web- and application server. This is not optimal, of course, as it pretty much defeats the whole point of the WSGI concept, so I set out to create a setup, where Plone would be themed by Deliverance configured as a WSGI middleware. Forthwith my notes:

I conducted my test on Mac OS X 10.5.2 with Python 2.4.5 installed via Macports. I didn't use virtualenv (although I should have) but had to make sure throughout the whole process, that python2.4 was used (as opposed to Leopard's default 2.5.1). To do that, I installed easy_install and setuptools explicitly for 2.4:

wget http://peak.telecommunity.com/dist/ez_setup.py
sudo python2.4 ez_setup.py
sudo python ez_setup.py -U setuptools

Before dealing with Deliverance I set up a WSGI enabled Plone instance. The buildout provided by repoze.plone proved to be the best candidate to do so (believe me!)

svn co http://svn.repoze.org/buildouts/repoze.plone/trunk repoze.plone
cd repoze.plone
python2.4 bootstrap.py
./bin/buildout

I then fired up the new instance as per instructions:

./bin/supervisord
./bin/addzope2user admin admin

I then could add a Plone instance foo at localhost:8080 just as always. I then stopped the zope instance so:

./bin/supervisorctl stop zope

and then restarted it using paster:

./bin/paster serve etc/zope2.ini

and sure enough, localhost:8080/foo was still there!

Now the only thing missing was Deliverance itself. To make it available for the instance, I simply easy_installed it into my system (this is the part, where next time I'll use virtualenv!):

sudo easy_install-2.4 Deliverance

This enabled me to add the following snippet to etc/zope2.ini:

[filter:deliverance]
paste.filter_app_factory = deliverance.wsgimiddleware:make_filter
theme_uri = file:///%(here)s/layout.html
rule_uri = file:///%(here)s/rules.xml

To use the deliverance as a filter in the WSGI stack I just had to add it somewhere. I chose to add it just after zope2 itself, so that my pipeline section looked like so:

[pipeline:main]
pipeline = egg:Paste#cgitb
           egg:Paste#httpexceptions
           egg:repoze.retry#retry
           egg:repoze.tm#tm
           egg:repoze.vhm#vhm_xheaders 
           errorlog
           deliverance
           zope2

Before restarting zope (or rather 'reserving it with paster') I had to add the rulefiles mentioned in the filter declaration. Eager as I was at this point, I simply borrowed the trivial default rule and theme from Martin's recipe, spiced it up a little with one of repoze.org's own rule files and created a etc/rules.xml file that looked like this:

<?xml version="1.0" encoding="UTF-8"?>
<rules xmlns:xi="http://www.w3.org/2001/XInclude" 
    xmlns="http://www.plone.org/deliverance">
    <prepend theme="//head" content="//head/link" nocontent="ignore" /> 
    <prepend theme="//head" content="//head/style" nocontent="ignore" /> 
    <append theme="//head" content="//head/script" nocontent="ignore" />    
    <append theme="//head" content="//head/meta" nocontent="ignore" />
    <append-or-replace 
        theme="//head" 
        content="//head/title"
        nocontent="ignore" />
    <append-or-replace theme="//div[@id='content']"
        content="//div[@id='region-content']"
        nocontent="ignore" />
</rules>

and a etc/layout.html file that looked like this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15">
<title>The Theme</title>
</head>
<body>
<h1>A theme</h1>
<div id="content">
</div>
<hr>
<address></address>
<!-- hhmts start -->Last modified: Fri Mar 16 09:33:37 CDT 2007 <!-- hhmts end -->
</body> </html>
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
    <title>layout</title>
</head>
<body>
</body>
</html>

Now I could stop the foreground paster process, fire it up again, hit reload in my browser and voila! Bob was my proverbial Uncle!

Next, I'll give mod_wsgi a try. But, of course, not the one for Apache but... yup, there's a mod_wsgi for nginx, too, yay! Stay tuned...

P.S. I'd like to take this opportunity to thank Chris, Tres and Paul for their efforts to connect Zope and Plone with non-techies and non-Zopistas. I'm very confident that you guys are creating some serious groundwork here that will benefit us immensely during the next eight years! Rock on!

Using the OKI C9800 on a Mac

Tom Lazar — Fri, 04 Apr 2008 12:03:31 +0000

Earlier this week I bought a printer for the new project I'm working on. A big, bad-ass OKI 9800 A3 color printer for the price of a small car -- not my money -- the client's money ;-).

Since we're a Mac-only shop here, I made sure it's got postscript3 and ethernet, so that connectivity wouldn't be a problem. Well, it was. Of course. Long story short: if we used the offical, OKI supplied PPD, the printer wouldn't print. It only could be coaxed into printing when using the generic postscript PPD -- which of course knew nothing about the A3 paper size or any of the other nifty features.

After spending three days(!) with the OKI hotline (spent mainly with re-stating our problem and configuration to various operators) I finally hit gold. In the form of a friendly, competent and female(!) operator (Hallo, Frau Walther!) who didn't just resort to OKI's ticket database but (gasp!) to her memory. She actually remembered a similar case a while back, told me to hold, came back 30 seconds later and ever since we're able to print here!

Turns out, that OKI has been delivering a faulty PPD - ever since march 2005! In the end it's just a single % character that's missing. I have to ask myself: WTF did they not update that file? Anyway, I'm posting this so that any other fellow OKI users can find this (because until now googling for the error Missing filter "application/vnd.cups-postscript 0 fierycupsfilter" didn't deliver any results. Hopefully, this will have changed).

Below you can find a diff of the the PPD or, for your convenience, you can download a patched version right here from tomster.org.

@@ -2947,5 +2947,5 @@

 *% PPD Last Modified 2005.03.10
 *% OS 10.2 PPD DB version 4/10/03 JDF
-*cupsFilter: "application/vnd.cups-postscript 0 fierycupsfilter"
+*% cupsFilter: "application/vnd.cups-postscript 0 fierycupsfilter"
 *% End of PPD file

URL-dependent skinswitching in Plone 3

Tom Lazar — Wed, 02 Apr 2008 10:02:30 +0000

I was just about to explain to somebody via email how to switch skins per url in Plone 3 when I realised I should rather blog it and send him the link instead of just explain it to him privately, so here it goes...

Add a file skinswitcher.py to the top-level of your product with the following contents:

def setskin(site, event):
    if event.request.URL.find('127.0.0.1') > -1 \
        or event.request.URL.startswith('https'):
        site.changeSkin('NuPlone', event.request)

then include the following snippet in your top-level configure.zcml:

<subscriber
    for="Products.CMFPlone.interfaces.IPloneSiteRoot
         zope.app.publication.interfaces.IBeforeTraverseEvent"
    handler=".skinswitcher.setskin" />

This particular example simply enables the NuPlone skin when the site is accessed via https or 127.0.0.1 (instead of localhost) and otherwise uses the default skin which is my usecase 99% of the time but it should be easy now to modify skinswitcher.py to your particular needs.

Site wide maintenance message

Tom Lazar — Fri, 21 Mar 2008 22:56:26 +0000

When doing maintenance work on a deployment site it is always a good idea (and in good taste) to show visitors a human-readable page explaining that you'll be back soon[tm] instead of letting them see ugly tracebacks etc.

Easy, right? Just put a static HTML file and configure it as the site root and you're done, right? Well, not quite. Because that will show the page to all visitors of the root of your site, but everybody else will get a 404.

Well, regular expressions and nginx to the rescue. The following configuration snippet displays the same index.html page to any request, not just the front page. I now keep it commented out in all my config files to be able to quickly turn on a maintenance message if something requires some fixing:

server {
    listen 80;
    server_name foobar.tld;

    root /usr/local/www/foobar/;
    index  index.html index.htm;
    location ^~ /index.html {
    }
    location / {
        rewrite "^/(.+)$" / last;
    }
}

The crucial part is the (empty) ^~ /index.html location that simply exists to stop an endless recursion that would take place otherwise. It also means, that your HTML file must be named index.html and reside in /usr/local/www/foobar/ on the file system.

nginx and varnish on Mac OS X

Tom Lazar — Sun, 16 Mar 2008 19:24:56 +0000

Any non-trivial deployment of Plone usually consists of at least a web server 'sitting in front of' it, often enough accompanied by some sort of caching service. For my own purposes I have settled on nginx for the former and varnish for the latter[1].

They have proven to be a reasonably good team for me, especially in combination with recent versions of CacheFu, Plone's de facto default caching product. By 'reasonably good' I mean, that varnish and CacheFu in their 'normal' setup (i.e. with the 'official' Plone .vcl config) offer a reasonable trade-off between performance and potentially stale content.

I noticed this behaviour particularly when looking at Zope's access log (i.e. var/log/primary-Z2.log) while accessing my latest Plone installation at wahlcomputer.ccc.de. There was much too much going on, as theoretically there shouldn't be any requests reaching the backend until some content had changed. In essence, the site behaved much slower than it would need to by answering unnecessarily to requests that varnish should be able to satisfy.

However, in order to debug the caching and purging of content in detail I needed a local Mac OS X based setup that mirrored that on the FreeBSD box serving the site as closely, as possible. I habitually took notes while setting it up on my desktop machine and am now retracing my steps as I attempt to duplicate it on my Macbook (Just like a bug isn't fixable until it's reproducible, I never consider my admin work done until I have been able to do it at least twice). I'm confident that somebody out there will find the following notes useful, too (even if it's just myself six months^wweeks down the road)...

Since I'm a happy user of the macports collection already anyway, I let it do the 'heavy lifting' of actually installing nginx and varnish. In addition I provided a launchd startup item for varnish and also added a host entry for wahlcomputer to enable virtual hosting for nginx and varnish. Here it goes:

sudo port install nginx
sudo port install varnish

DNS

Before continuing the setup I first wanted to make sure that I could use the domain name wahlcomputer to access my machine. Instead of doing it 'properly' by adding a new record to OpenLDAP and thus possibly opening up what could amount to yet another can of worms I simply tried to edit ye' good ol' /etc/hosts file and added the following line:

127.0.0.1   wahlcomputer

And sure enough, even in the days of 'OpenLDAP FTW!!' Mac OS X still honours entries made to the hosts file so I could move on.

nginx

Next, I configured the freshly installed nginx. As the port already comes with a launchd start up item, I just needed to load it:

sudo launchctl load /opt/local/etc/LaunchDaemons/org.macports.nginx/org.macports.nginx.plist

It's my practice to leave the default files as untouched as possible and put all my configurations into separate files, that I simply include:

sudo mv /opt/local/etc/nginx/nginx.conf.default  /opt/local/etc/nginx/nginx.conf
sudo mkdir /opt/local/etc/nginx/includes
sudo touch /opt/local/etc/nginx/includes/wahlcomputer.conf
sudo mkdir /opt/local/var/log/nginx/

To make nginx aware of those files I added the following line just before the final closing }:

include etc/nginx/includes/*.conf;

Here's its contents in its full, unabridged glory (and VHM gore):

server {
    server_name wahlcomputer.ccc.de wahlcomputer;
    location / {
        proxy_pass http://127.0.0.1:8071/VirtualHostBase/http/wahlcomputer:80/foo/VirtualHostRoot/;
    }
}

For the changes to take effect, I needed to restart nginx like so (why, oh why, does launchctl with all its bells and whistles not have a simple reload or restart command?!):

sudo launchctl stop org.macports.nginx
sudo launchctl start org.macports.nginx

Now visiting http://wahlcomputer/ showed the front page of the Plone site and I could move on to varnish:

varnish

Unlike nginx, the varnish port (currently) does not come with a startup item, so I made my very own (by blatantly copying from the nginx setup):

sudo mkdir /opt/local/etc/varnish
sudo touch /opt/local/etc/varnish/default.vcl
sudo mkdir /opt/local/etc/LaunchDaemons/org.macports.varnish
sudo touch /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist
sudo chown root:wheel /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist
sudo chmod 644 /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist
sudo ln -s /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist /Library/LaunchDaemons/
sudo mkdir /opt/local/var/varnish
sudo chown -R _www /opt/local/var/varnish

Looking at varnishd's man page - HTTP accelerator daemon - Linux Manual - Digipedia") I saw that there are some options that must be passed in upon start up. I did this by adding them to the ProgramArguments array in the /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>KeepAlive</key>
    <false/>
    <key>Debug</key>
    <false/>
    <key>Label</key>
    <string>varnishd</string>
    <key>OnDemand</key>
    <false/>
    <key>GroupName</key>
    <string>staff</string>
    <key>UserName</key>
    <string>_www</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/local/sbin/varnishd</string>
        <string>-a</string>
        <string>localhost:6081</string>
        <string>-T</string>
        <string>localhost:6082</string>
        <string>-f</string>
        <string>/opt/local/etc/varnish/default.vcl</string>
    </array>
    <key>RunAtLoad</key>
    <false/>
</dict>
</plist>

Now all that was missing was to add some directives to the currently still empty file /opt/local/etc/varnish/default.vcl. It ended up being too long to quote here in its entirety, instead I will simply point you to a nice variation of the 'official zope-plone.vcl that I found that sports some neat enhancements.

Finally I could start up varnish:

sudo launchctl load /opt/local/etc/LaunchDaemons/org.macports.varnish/org.macports.varnishd.plist

In the nginx wahlcomputer.conf file all I needed to change was the port number from 8071 (the Zope instance) to 6081 (varnish).

By keeping a tail on the access logs of nginx and my Zope instance and looking at the response headers with firebug I can now start to tweak my setup to my heart's content. Stay tuned.

[1] A bit of an explanation is perhaps necessary as to why even bother with two types of machinery in front of Plone. The short answer is 'separation of concerns'. While varnish could theoretically take over most of the features that I use nginx for (namely, URL-rewriting, logging and static content delivery) I still would need it for SSL/HTTPS. Also, the logging feature of varnish requires an extra varnishlog process anyway.

Chaosradio Express on Plone and Zope

Tom Lazar — Sun, 16 Mar 2008 16:07:56 +0000

Two weeks ago I sat down with Tim Pritlove and he interviewed me about Plone and Zope for his Chaosradio Express podcast.

I thoroughly enjoyed the experience, largely due to the fact that Tim proved to be an excellent host. He gave me enough leeway to ramble while at the same time providing enough structure for the process to let me relax and be myself. Thanks Tim!

In the end we talked over one and a half hours and I can only assume that one has to have a genuine interest in the content matter to listen to it all the way through but afterwards I still felt that I had left much unsaid.

I tried to be honest and upfront about Zope and Plone's weaknesses while still conveying that they both are truly great systems with even greater communities. The interview was conducted in German, though, so I figure the percentage of the audience of this blog that will find it interesting is probably rather small, but nonetheless, here's the link to the episode.

The state of the Union^WLinguaPlone

Tom Lazar — Fri, 07 Mar 2008 19:23:39 +0000

Today I made an offer to a prospective client for a multi-lingual, Plone based website. Since it's been a while since I've had any hands-on experience with Plone's de-facto standard product for managing multi-lingual content LinguaPlone I decided I'd best take the current offerings for a spin before I make any (legally binding!) promises.

On first sight, things regarding multi-lingual content are currently a bit confusing. While there is a stable release of LinguaPlone that offers support for Plone 3.0, there have also been recent developments in Plone itself that have broken LinguaPlone (for instance 3.0.5 didn't seem to fare particularily well in this regard).

As always, when feeling uncertain or confused about something Plone related, I turned to #plone, where luckily jensens and baekholt were online at the moment and it soon transpired, that things aren't actually quite as bleak as it seemed and further testing on my part today confirmed that. So, without further ado, here's the lodown:

use Plone >= 3.0.6
install plone.browserlayer
use LinguaPlone 2.1/svn
use PloneLanguageTool >= 2.1
don't use the default Language selector portlet provided by Plone

The last item requires a bit of explanation. In its current implementation the default language selector portlet only switches the language of the interface, but doesn't actually change the context to the translated version in the selected language. This, obviously still needs to be fixed, but is just cosmetical in nature. And if the offer that I just sent out will be accepted, I might even do it myself ;-)

Other than the non-working language selector all the scenarios I played through today worked like a charm, though. To switch languages I needed to manually enter the URL of an object that I knew to be of the target language, but since the new version of PLT keeps content and language interface always in sync, nothing else was required (and the fix for the selection tool shouldn't involve much else, either for that reason).

LinguaPlone has come a long way to tackle a truly ambitious problem and I'm very sure, as of today, that Plone will prove to be one of the best solutions for managing multi-lingual content on the web. Again.

Skin dependent resources

Tom Lazar — Wed, 05 Mar 2008 19:31:03 +0000

For the soon-to-be-released Voting machines info site of the CCC I'm using the current trunk of NuPlone for both its public facing pages, as well as for the editors.

NuPlone has a neat new feature called multiresolution layout, which is great, especially for editors. Basically it means, if you have a bigger screen or wider window you get a three column layout, if not, the layout is based on two columns (you can test that feature here at tomster.org).

However, for the CCC site the feature proved to be more confusing than useful for public visitors, so I wanted to make sure that the JS resource, responsible for that feature was only rendered, if the NuPlone skin was used.

The initial idea, was to include an empty multi-resolution.js file with the Wahlcomputer skin but it turns out, it's possible to evaluate the name of the current skin in the conditional expression of a resource. In this case I used the following snippet inside jsregistry.xml:

<javascript cacheable="True" 
    compression="safe" 
    cookable="True"
    enabled="True" 
    expression="python:portal.getCurrentSkinName()=='NuPlone'" 
    id="multi-resolution.js" 
    inline="False"/>

Instead of the default empty expression I used python:portal.getCurrentSkinName()=='NuPlone' and bingo!

Launchd in a nutshell

Tom Lazar — Wed, 27 Feb 2008 23:29:27 +0000

Today I needed to make a fetchmail process automatically start on a Mac OS X Server instance.

Now, ever since Version 10.4 this is the domain of launchd. But compared to what I'm used to in FreeBSD, Apple's own introduction to launchd seemed rather daunting.

My reasong was something along the lines of "If I can't use something as straightforward as rc I might as well get out the big guns and use daemontools" (which already e.g. takes care of my IMAP Server and, of course, my name servers.)

Well, to make a long story short, once I got over the fact, that I need to write XML to configure startup items everything else went rather smoothly. In the end, to have fetchmail launched automatically at startup I needed the following markup in /Library/LaunchDaemons/fetchmail.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>GroupName</key>
    <string>staff</string>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>fetchmail</string>
    <key>Program</key>
    <string>/usr/bin/fetchmail</string>
    <key>ProgramArguments</key>
    <array>
        <string>--fetchmailrc=/Users/diradmin/.fetchmailrc</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>Umask</key>
    <integer>7</integer>
    <key>UserName</key>
    <string>diradmin</string>
    <key>WorkingDirectory</key>
    <string>/Users/diradmin/</string>
</dict>
</plist>

(Since the fetchmail process won't be launched via a logged in user, I needed to point it explicitly to the config file I developed for it using the ProgramArguments key.)

Next, I needed to make sure that the abovementioned file had the right ownership and permissions:

sudo chown root:wheel /Library/LaunchDaemons/fetchmail.plist
sudo chmod 644 /Library/LaunchDaemons/fetchmail.plist

Unlike with rc or daemontools, however, just because I now had a perfectly fine config file sitting in its proper place didn't mean anything would happen next time the machine booted. No, I still needed to explicitly tell launchd to use this item using its companion launchctl command:

sudo launchctl load fetchmail.plist

To test the item I then proceeded to manually start it:

sudo launchctl start fetchmail

Note, that this time I didn't state fetchmail.plist as above but rather fetchmail, which refers to the Label key in the XML.

And sure enough, fetchmail was now up and running and was doing its thing. Since I had set KeepAlive to <true/> I could kill it and not even a second later, it was back up and running (with a new PID, of course). Nice. Just like daemontools.

While I'm glad that this all went rather smoothly for me I must say that I much prefer rc and daemontools, though. launchd seems over-engineered. I find that snippet above positively ugly. Even for XML. I mean, come on, what's this supposed to be: <key>RunAtLoad</key><true/>?

But to be fair, Apple specifically designed launchd to handle more than just start up items but to replace "init, rc, the init.d and rc.d scripts, SystemStarter (Mac OS X), inetd and xinetd, atd, crond and watchdogd", to quote its Wikipedia entry.

Anyway, from now on I'm probably just going to use the snippet above as boilerplate and get over with it ;-)

Oh, and Tim? Sorry about the XML bashing. I got nothing against XML per se. In fact I wouldn't know what to do without it. I just don't think it's a good choice for configuration files that are created and read by humans. (And perhaps, neither do you.) Even Zope is moving away from ZCML ;-)