daemontools

The daemontools package provides a suite of utilities to reliably and securely start, run, log, and control server programs known as daemons. Dan Bernstein, the author of daemontools, explains the rationale behind daemontools quite well.

compiling daemontools

Official source: http://cr.yp.to/daemontools.html
Install instructions: http://cr.yp.to/daemontools/install.html

Lots of information is available at http://thedjbway.org/daemontools.html, including some daemontools patches that are useful in any daemontools installation -- specifically the daemontools sigq12.patch. Also useful is the daemontools errno.patch which is required for compilation on many modern systems. The errno.patch is included in the daemontools logging patch below.

In addition to the sigq12.patch, we have patched daemontools to "chew its own bubblegum" so that svscan employs the same logging framework that it provides to services that are run under supervise.

The daemontools logging.patch directs svscan output through multilog or other logger which supervise monitors. With this patch, if a directory (or symlink to a directory) named /service/.svscan exists, then svscan will direct its stdout and stderr to the supervised service run from the /service/.svscan/log/ directory. This simplifies svscan startup scripts and replaces the use of readproctitle given in djb's examples, which does not work on some platforms, e.g. FreeBSD 5.1 notes thedjbway.org in booting svscan. Redirecting svscan output through multilog also provides a much better solution than other svscan startup scripts which direct output exclusively to console, causing problems to go undetected for a long time if the server is in a closet somewhere and nobody looks at console.

our favorite way to boot svscan

We split services into two groups, those local services that should always run -- such as local DNS resolver and local MTA/antivirus/antispam -- and those public networked services that should run only when the server is in full production (all the time, of course!). These include SSH, web, MTA on public IP addresses, IMAP, POP, authoritative DNS, etc. The /usr/local/service-local/ and /usr/local/service-public/ directories both contain symlinks to appropriate service directories in /var/service/ including the symlinks
   /usr/local/service-local/.svscan -> /var/service/svscan-local
   /usr/local/service-public/.svscan -> /var/service/svscan-public
and those target directories each have a log/ subdirectory that runs multilog. The output of any service that does not have its own log/ subdirectory will end up in these logs if the service does not redirect its output elsewhere.

Our /etc/inittab contains the following:

SL:2345:respawn:/usr/local/sbin/svscan.rc start /usr/local/service-local
SP:345:respawn:/usr/local/sbin/svscan.rc start /usr/local/service-public

SY:S0126:once:/usr/local/sbin/svscan.rc stop  /usr/local/service-public
SZ:S016:once:/usr/local/sbin/svscan.rc stop  /usr/local/service-local

/usr/local/sbin/svscan.rc is just as straightforward. It takes a "start" or "stop" argument and an optional service directory, with the default being /service. First, the supervisors for the loggers are set to exit when the loggers exit. Then, the services are shut down and their supervisors are told to exit, at which point the loggers will read EOF and will themselves exit along with their supervisors. This sequence ensures that log entries will not be lost when the services and their loggers are shut down. Note that for this to succeed, svscan must not be running. If it is, it must be killed prior to running this script, or else svscan will simply respawn the services and their supervisors. The runlevel commands in /etc/inittab above take care of that for us. (Note that if a logger dies during the shutdown, it will not be restarted, and so some log entries may be lost, but this is still better than svc -dx /service/* /service/*/log.)

#!/bin/sh

PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
SERVICES=${2:-"/service"}

exec </dev/null >/dev/null 2>&1

for log_dir in "$SERVICES"/*/log "$SERVICES"/.svscan/log; do
    if [ -d "$log_dir" ] \
       && svok "$log_dir" \
       && svstat "$log_dir" | grep -q ": up"
    then
        svc -ox "$log_dir"
    fi
done

svc -dx "$SERVICES"/*

[ "$1" = "start" ] && exec env - PATH=$PATH svscan "$SERVICES"

While the svscan.rc above is short and sweet, the script can be extended to do more than just svc -dx. For example, the script could check for a $SERVICES/*/shutdown script in each service directory, and if present, run the shutdown script. Such a svscan.rc script might look like this (untested):

#!/bin/sh

PATH=/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin
SERVICES=${2:-"/service"}

exec </dev/null >/dev/null 2>&1

for log_dir in "$SERVICES"/*/log "$SERVICES"/.svscan/log; do
    if [ -d "$log_dir" ] \
       && svok "$log_dir" \
       && svstat "$log_dir" | grep -q ": up"
    then
        svc -ox "$log_dir"
    fi
done

for service_dir in "$SERVICES"/*; do
    if [ -d "$service_dir" ] \
       && svok "$service_dir" \
       && svstat "$service_dir" | grep -q ": up"
    then
        [ -x "$service_dir"/shutdown ] && "$service_dir"/shutdown
        svc -dx "$service_dir"
    fi
done

[ "$1" = "start" ] && exec env - PATH=$PATH svscan "$SERVICES"

using daemontools