What we needs from daemons

In my post of yesterday I noted some things about the init scripts, small niceties that init scripts should do in Gentoo for them to work properly and to solve the issue of migrating pid files to /run. Today I’d like to add a few more notes of what I wish all daemons out there implemented at the very least.

First of all, while some people prefer for the daemon to not fork and background by itself, I honestly prefer it to — it makes so many things so much easier. But if you fork, wait till the forked process completed initialization before exiting! The reason why I’m saying this is that, unfortunately, it’s common for a daemon to start up, fork, then load its configuration file and find out there’s a mistake … leading to a script that thinks that the daemon started properly, while no process is left running. In init scripts, --wait allows you to tell the start-stop-daemon tool to wait for a moment to see if the daemon could start at all, but it’s not so nice, because you have to find the correct wait time empirically, and in almost every case you’re going to run longer than needed.

If you will background by yourself, please make sure that you create a pidfile to tell the init system which ID to signal to stop — and if you do have such a pidfile, please do not make it configurable on the configuration file, but set a compiled-in default and eventually allow an override at runtime. The runtime override is especially welcome if your software is supposed to have multiple instances configured on the same box — as then a single pidfile would conflict. Not having it configured on a file means that you no longer need to hack up a parser for the configuration file to be able to know what the user wants, but you can rely on either the default or your override.

Also if you do intend to support multiple instances of the same daemon make sure that you allow multiple configuration files to be passed in by he command-line. This simplifies a lot the whole handling of multiple-instances, and should be mandatory in that situation. Make sure you don’t re-use paths in that case either.

If you have messages you should also make sure that they are sent to syslog — please do not force, or even default, everything to log files! We have tons of syslog implementations, and at least the user does not have to guess which one of many files is going to be used for the messages from your last service start — at this point you probably guessed that there are a few things I hope to rectify in Munin 2.1.

I’m pretty sure that there are other concerns that could come up, but for now I guess this would be enough for me to have a much simpler life as an init script maintainer.

Small talk about my experience with MongoDB

I’m interrupting the series of Ruby rants (somewhat) to talk about something that is slightly related but not too much. I’ve already written about my plan of writing a system to index and manage the boxes that I manage at various customers’ places. This system is half written for me to have something neater than GIT and HTML to manage the data, and half to get up-to-date with modern Rails development.

One of the decisions I made was to try for once a NoSQL approach. I had many questions on why I did it that way and the answer was for me pretty simple actually: I didn’t want to spend time in designing the relationships first and write the code later. The nice part about using MongoDB was that I’m able to add and remove attributes when I like, and still query for them without fetching huge amount of data to process Ruby-side.

Honestly, after seeing the kind of queries that Rails concocts to get me data that is normalised, but requires multiple many-to-many relationships to be resolved, I’m quite sure that it can’t be worse with MongoDB than it is with PostgreSQL, for this kind of data.

Of course it’s not all positive sides with MongoDB; beside the obnoxious requirement of a JavaScript engine (and there aren’t many), which is linked to the use of v8 (which is not ABI-stable, and thus each update is a rebuild and restart), I had some bad experience yesterday, but not something extreme, luckily. On the one server I use MongoDB on, I recently configured the locale settings in the environment — but I forgot to re-execute locale-gen so the locale was broken; as it turns out, Boost throws an exception in that case, and MongoDB does not try to manage it, aborting instead.

I guess the main blame here is on the init script that does not report an execution failure: the service is noted as started, and then crashed, which is technically correct, but not what you expect the init script to tell you. I guess if I have more time I should try to get more Unix-style daemon support to mongod so that it can be integrated better with our start-stop-daemon rather than left with the hacky script that it’s using now.

Add to that missing support for using the syslog protocol and you can probably figure out that the thing that worries me the most about MongoDB is the usual sense of “let’s ignore every previous convention” which seems to come with NoSQL projects. Luckily at least this project feels like technically, rather than politically, driven, which means I should be able to get them to at least partially implement those features that would make it a “good citizen” in an Unix environment.

Sigh, I really wish I had a bit more time to hack at it, since I know the few rough spots I found should be easily polished with a week or so of work; unfortunately I guess that’ll have to wait. Maybe after my coming US trip I’ll be able to take a couple of weeks to dedicate to myself rather than customers.

Extending libdaemon

As Luca wrote, since I came home from the hospital I’ve been working quite a bit on LScube, in particular working on porting feng to use glib, which is a very interesting thing considering that it also allowed to reduce the dependencies (no more need for libgcrypt for instance, which was a problem to me since it didn’t use pkg-config and I find that a bit medieval nowadays).

One other thing that feng lacked was support for running as a daemon; as git-daemon shown recently, having proper support is much better than faking it through start-stop-daemon.

A proper support for running as a Unix daemon mean quite a few different things: proper logging to either stdout/stderr (when not running as daemon) or syslog (when running as daemon); pid file generation (a file, usually in /var/run, where the process ID of the running daemon is written); proper check for forking (so that the user is told if the daemon failed to start; check for if the daemon is running already if trying to execute it again on the same pid file (multiple instances, where allowed, should use multiple pid files); privilege dropping (you don’t want your daemons to run with root privileges, although the ones who provide network service might need those privileges at once to open privileged ports (below the 1024 boundary).

Implementing all these things can easily be a boring task considering that is almost the same for each daemon you might be writing (feng is not the only daemon in lscube). Luckily, someone already thought about this and started working on a solution: Lennart’s libdaemon, which he used for the Avahi daemon.

But as it happens, although libdaemon 0.12 (which is not even the latest version) already provides enough of the basics needed for feng, there are a few things that would require further wrapping around in the code of feng itself. This includes verbosity handling (you might not want to have information output on stdout/stderr; while with syslog you can easily configure your logging to reduce the output, on stdout/stderr you either implement verbosity or you get everything that is logged through the daemon_log() function), and privileges dropping.

The nice thing of working with good maintainers, though, is that they are open to suggestions and extensions, so thanks to Lennart availability, I started implementing those parts that I was lacking. Hopefully in a not-so-long time, there will be a new libdaemon release, which feng will use, that will provide all the needed functionalities.

I really like this approach since it allows to do the things right once and for all, upstream, without having to reinvent the wheel every time, especially considering that it’s not rare that, trying to reinvent the wheel, you get it -octarine- octagonal rather than round.

For those who would like to take a look at the work in progress of my extensions, you can find them on my git repository together with some autotools fixes and some patches for OpenSolaris compatibility.