I recently had to ship an application which was primarily made of a couple of standalone Retro-based web servers, all mounted on a Pound HTTP proxy. Pound is much better than Apache for proxying, first because it's faster, easier to configure, but also because Apache will often block and give 503 - Service Not Available when you back-end web-service crashes while serving a request. Pound goes away with that nicely, and has the added bonus of periodically checking the health of your back-end web servers.

More recently, I've been working on a project using the fashionable 'redis' and 'mongodb' databases. In both cases, I was faced with the problem of automatically starting and monitoring non-forking processes, which did not come with any /etc/init.d/ management script.

I've tried Monit, but really spent two days trying to get it to automatically start and monitor processes that I was "daemonizing" with the daemonize command. The main issue I had was that Monit would not restart the daemon if the pid was still present, and was apparently unable to check if the pid actually matched a running process -- so it was just assuming that the daemon was running if there was a pid file. It's maybe me using Monit wrongly, but I gave up after two frustrating days.

I then found supervisord which despite a lack of buzz, is actually a pretty decent solution for this particular problem: it expects your command to be non-forking, and will manage all the daemonizing and monitoring aspects itself. The documentation is not so great, but writing a rule to automatically start and monitor a service is as simple as this:

[program:services-redis]
priority=10
directory=/opt/services/redis
command=/opt/services/redis/redis-server /opt/services/redis/redis.conf
user=root
redirect_stderr=true

if you want to try it, here is how to install it:

sudo aptitude install python python-setuptools
sudo easy_install supervisord

then vi /etc/supervisord.conf

[supervisord]
logfile=/var/log/supervisord
loglevel=info
pidfile=/var/run/supervisord.pid
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
http://localhost:9001
[inet_http_server]
port=9001

and just append whatever service configuration you'd like. Here's what you'd do for Apache:

[program:apache2]
command=/usr/sbin/apache2 -c "ErrorLog /dev/stdout" -DFOREGROUND
redirect_stderr=true

Again, note that supervisord asks your programs to not fork (ie. run in foreground), and will do the daemonization for you. As such, it's a good replacement to init.d as well as Monit, and because it offers command-line, Web, XML-RPC and Python interop, it makes it a very good candidate for managing your services across a various range of server nodes.

The only thing I'd like to have is to be able to retrieve is per-process metrics, such as cpu, memory, number of threads -- but these are already available in the /proc filesystem on Linux.

So to sum this up, if you have to make sure a set of programs are always running, and if you want a simple way to configure which of these programs have to run, supervisord is your friend !

PS: If you want to automatically start supervisor, here how to set it up with init.d on Ubuntu 9.10