Random quality

RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

xkcd’s Random Number comic © Randall Munroe

We all know that random numbers might not be very random unless you are very careful. Indeed, as the (now old) Debian OpenSSL debacle, a not-enough-random random number generator can be a huge breach in your defences. The other problem is that if you want really random numbers you need a big pool of entropy otherwise code requiring a huge chunk of random bytes would stall until enough data is available.

Luckily there are a number of ways to deal with this; one is to use the EntropyKey while other involves either internal sources of entropy (which is what timer_entropyd and haveged do), or external ones (audio_entropyd, but a number of custom circuitry and software exist as well). These fill in the entropy pool, hopefully at a higher rate than it is depleted, providing random data that is still of high quality (there are other options such as prngd, but as far as I can tell those are slightly worse in term of quality).

So, the other day I was speaking with Jaervosz, who’s also an EntropyKey user, and we were reflecting on whether, if there is not enough entropy during crypto operations, the process would stall or cause the generation to be less secure. In most cases, this shouldn’t be a problem: any half-decent crypto software will make sure not to process pseudo-random numbers (this is why OpenSSL key generation tells you to move your mouse or something).

What we ended up wondering about, was how much software uses /dev/urandom (that re-uses the entropy when it’s starving) rather than /dev/random (which blocks on entropy starvation). Turns out there are quite a few. For instance on my systems, I know that Samba uses /dev/urandom, and so does netatalk — neither of which make me very happy.

A few ebuilds allow you to choose which one you want to use through the (enabled-by-default) urandom USE flag… but these I noted above aren’t among those. I suppose, one thing we could be doing would be going over a few ebuilds and see if we can make it configurable which one to use.. for those of us who make sure to have a stable source of entropy, this change should be a very good way to be safe.

Are you wondering if any of your mission-critical services are using /dev/urandom ? Try this:

# fuser -v /dev/{,u}random
                     USER        PID ACCESS COMMAND
/dev/random:         root      12527 F.... ekey-egd-linux
/dev/urandom:        root      10129 f.... smbd
                     root      10141 f.... smbd
                     root      10166 f.... afpd
                     flame     12356 f.... afpd

Also, if you want to make sure that any given service is started only after the entropy services, you can simply make it depend on the virtual service entropy (provided by haveged, or ekeyd if set to kernel output, or ekey-egd-linux if set to EGD output). A quick way to do so without having to edit the init script yourself, is to add the following line to /etc/conf.d/$SERVICENAME:

rc_after="entropy"

Changes to the netatalk ebuild

You might remember a few weeks ago I bought a new system to use as a local fileserver, keeping among other things a complete copy of the distfiles mirrors (and more than that) to use with the tinderbox and the other boxes. But that system is not only limited to serve (via NFS) the distfiles for my systems, it’s also set up to serve as main storage point for customers’ data as well as my own data, including my music library (which is primarily managed by iTunes, but is consumed by Banshee as well) — before this, a rsync task copied the iTunes library over Yamato to then serve it to yet another box.. you probably see how duplicating this data is not something I’m happy about.

Well, turns out that Samba can’t get me anything over 50Mbit of bandwidth over a gigabit network, when copying over the data, not nice (and this is without considering that each time I copy over files with Windows 7 I have to answer “Try again” as it reports a network-busy error…). So I decided to try again Netatalk, that I used to use a long time ago, and the ebuild of which I have partly wrote myself in the past — the result has been satisfying: 200Mbit, which is still not gigabit, but it’s still four times as fast as Samba.

Now, since my two OS X systems (okay, three with my mother’s) are all running Lion, what I needed was the 2.2 series (2.2.1 release) with Avahi support, not a problem, just need the ~arch version for now. But after using it for a week or so, I started seeing a number of issues that needed to be addressed. The most obvious of which was that restarting Avahi didn’t cause the AFP daemon to restart, which in turn meant that OS X was unable to connect to the fileserver after reboot. D’oh.

The Netatalk ebuild seems to have been maintained on life support for a while, and simply bumped without checking out all of its internals, so I decided to give it a look. The cause of the issue I hit myself was obvious: upstream-provided init script wasn’t designed to be well-integrated into Gentoo. This is common: upstream projects provide some init scripts that do work with Gentoo but don’t follow our guidelines, which would be especially difficult since we do lack good documentation on how to properly write init scripts for the Gentoo init system.

Thankfully, after an afternoon toying with the init scripts and the ebuild, I was able to get a new revision of the ebuild in tree that, while causing a few changes in behaviour, should be much easier to deal with in the future. Since there isn’t an official documentation for Netatalk in Gentoo, it should be a good idea to document here the changes, so that if somebody is confused by the new ebuild, or has comments on what I have done, it can be used as a reference.

The most noticeable difference from the old design is that the new ebuild installs split init scripts for the services, rather than using the single /etc/init.d/netatalk script. This is important for two main reasons: it no longer risk leaving daemons running if they were enabled earlier and stopped later, and it allows to check out the service’s status daemon per daemon.

In the default configuration (the details of which I’ll get to in a moment), netatalk ebuild installs two services: afpd and cnid_metad; the former is the actual file server daemon, the latter is its backend: it provides the CNID metadata for the volumes, and is basically a huge database process. Having the two separate is handy: you no longer need to reload the database when changing the configuration file, which could be a waste of time if you have huge volumes, or volumes with a huge number of files. And most importantly, only afpd talks to the network and needs avahi support, anything else is, well, backend.

The new init scripts don’t rely on the old /etc/netatalk/netatalk.conf configuration file; this happens by design, as the services don’t really share many settings: most of the variables were used to tell which services to start, and the rest were mostly used to pass custom options to the daemons themselves. In the case of afpd almost all the settings passed were configurable in the afpd.conf setting file as well, and the end result is that using that configuration is the suggested method, instead of passing the options via the init script.

The only option that is shared across different services is related to the AppleTalk protocol, which is what I’ll be talking about now. While most users of Netatalk will likely only need the AFP service, which is the file server itself, the package implements also some other services to be used in AppleTalk-based networks. For those who don’t know the details of this, AppleTalk was Apple’s own local network protocol, and was, and sometimes is, used with Ethernet networks with a “wrapper” called EtherTalk.

Apple discontinued support for this technology with the Snow Leopard (10.6) release; and even Leopard itself preferred TCP/IP over AppleTalk, whenever possible, so the uses of that protocol (or to be precise, of DDP) are pretty rare. One of my customers still have a Linux box configured with EtherTalk, because their Mutoh large format printer was configured using that.. but even that’s going to change at some point in the future. With this in mind, there is the other noticeable difference from before.

While netatalk before installed everything for AppleTalk users by default, now there is an appletalk USE flag that needs to be enabled: without that the atalkd service is not installed, nor are the a2boot, timelord and papd services, that are used, respectively, for network boot, time synchronisation and print server hosting (after all, CUPS is now developed by Apple itself). This is another good reason why the services are now split: those three services are only installed if you want AppleTalk support, and they all depend on the atalkd service; afpd on the other hand will use atalkd only if you configure it to do so, which is not the default for most systems.

But I said that AppleTalk users had shared options between services, and that is true: the name and zone for the atalkd service need to be configured; this is now done through /etc/conf.d/atalkd (and you have to follow the same setting with the extra parameters for afpd if you want that to run over AppleTalk). By default it’ll export the host’s short name as the AppleTalk host name, and will use no zone, and that should do the trick for almost every user out there who is still stuck with AppleTalk, for what I can tell.

And to complete the discussion for what concerns AppleTalk, there is one catch: the Netatalk package needs kernel help to support the protocol stack; this means that it should check the kernel configuration and give you hints on what to enable and why. Unfortunately I have no real idea of which settings need to be enabled, so I didn’t add any check for that right now. If you wish to send a patch to do so, it’ll be very welcome.

But not all changes are related to the services: new USE flags were introduced to deal with access control lists (ACLs) and user space quota, and the old XFS USE flag has been removed: it made sense back in the early days of kernel 2.6 as not everybody had the same set of Linux kernel headers in the system, but nowadays, that is only legacy, so support for XFS-style quota is enabled whenever Quotas are enabled as well, just like it was supposed to. Very basic support for LDAP sharing of users and group is also present now, but like for Kerberos, it’ll need a complex testing network to actually work as it was intended, so if you notice anything wrong, please report. And if you only need AFP to work with modern OS X installation, feel free to disable the ssl USE flag, as that one only adds the old, DHX 1 user access module (UAM), which is replaced by the libgcrypt-based DHX2 for modern systems.

The change I’m probably most happy about, though, is the replacement of the statically-linked libatalk library with a shared object, that is used both by the utility and as an exported interface (I’m not sure if it should have been exported, but right now it is, so there..). This allows to cut the size of the package from over 8MB down to 4MB, with debug information included, which means also a smaller memory footprint when you have more than one service started (and you always have at least two, possibly three, at any time).

Unfortunately this one has brought at least one issue, which is now fixed: since static linking is non-transitive but also non-asneeded-influenced, external library dependencies were not expressed on the common library, but rather on the tools themselves; this was a problem now, so for a little while the tcpd USE flag couldn’t produce a final link… I fixed that one, but there might be other issues with other combination of USE flags, but I guess this is why we call it “testing”…

Anyhow… if you have suggestions to provide or bugs to report, please don’t refrain from leaving a comment here or opening a bug in our bugzilla — for a while at least, I’ll be the netatalk dedicated maintainer.. again… Next week’s tasks include sending the changes upstream so that we won’t need to keep patching this forever.

Gentoo Updates

Ok I had just a little time to update my blog lately so it’s time to get a bit of an update.

First, a good and a bad news: the good one is the one everyone already talked about, the software patents in Europe are cancelled for now; the bad one is the yesterday’s event in London.. I think there’s nothing I can say that hasn’t already been said.

Moving to more Gentooish topics, vlc is still driving me mad with mozilla support (it requires some .so libs we don’t have in gecko-sdk because it needs -fPIC love).
For now latest version has slp and mozilla support disabled. The first one will remain until a new version can fix it to use the latest API, the latter is the one I’m trying to take care of.

Also, I took over netatalk (package for file and printer sharing between *BSD/Linux and MacOS with its original protocol), adding version 2.0.3 which fixes most of the bugs open on bugzilla (also the one with future dates in 64-bit arches), and which is available now also for ~amd64.

libtorrent and rtorrent are still releasing one version every week or little more, and as I couldn’t wait for the latest version to be in portage for more than a month to mark it stable, so I took the latest version of the two packages which had big bugfixes (0.6.⁴⁄₀.2.4) and marked it stable for amd64 (my “maintained” arch).

Third news: I’ll take care of media-tv/xdtv as soon as I have the amd64 support for mmx/sse extensions sorted out with upstream. Just wait a couple of days and you’ll find xawdecode’s successor on portage with ~amd64 (it can seem absurd, but it won’t have ~x86 until someone will test it as I don’t have x86 hardware to test it on).

I think I’m starting maintaining more useful packages :)