My distributed nightmare

For Gentoo/FreeBSD I ended up having three extra boxes, besides my main workstation (Enterprise) and my laptop (Intrepid) in my home office, all of which shares a good deal of data, as they all need a Portage tree, a Distfiles directory, a few overlays and a bit of data like the repositories I work with so that I can just update them on one box and build on two or three.

To avoid having to copy the data on all of them (which other than being slow and impractical would also reduce a lot the space on disk I have, and for all of them but the sparc it’s actually quite a bad thing), I used to use NFS to share most of the data between them, after all they are all on the same network segment and that is not a big problem.

Unfortunately, I knew for a while that nfs-utils-1.0.8 and later versions’ server conflicts with FreeBSD’s client. For some reason, it’s returning a lot of «Stale NFS file handle» when accessing more than an handful of files (which means I can’t merge anything, nor I can do my usual daily backups of git and typo). I left the newer version masked locally up to now, but now nfs-utils was broken by a libevent bump, and it didn’t rebuild, not sure why and I don’t feel like wasting my time trying to keep nfs-utils-1.0.7 alive.

So I’ve started looking for alternative solutions, unfortunately is far from being easy.

I first looked for alternative network/distributed filesystems. Unfortunately the requirement is that it works fine on both Linux (server) and FreeBSD (client), which turns away AFS and Luster, both of which are not supported on FreeBSD — okay, AFS is somewhat supported, but a lot of noise about problems doesn’t really make me expect anything good out of it. What I see that could still work is CIFS or CodaFS.

The first I ruled out because I tried it before, and not only it’s far from being practical, is also slow as hell on a 100MBit network. Only the latter is left.

And again, a problem. net-fs/coda is currently available only on PPC and x86. When I tried just building a package of it on AMD64, it failed because the DEPEND/RDEPEND settings are wrong. When I wanted to merge it, two packages failed because of —as-needed related issues, when I finally came to merge net-fs/coda I understood why only those arches had it marked: it’s 64-bit unsafe.

But this is just the tip of the iceberg. Coda and its related packages currently in portage are quite old, compared to the latest versions available on the site. Maybe they already fixed the two —as-needed problems I found and fixed myself tonight, maybe they also fixed the 64-bit uncleanness, as there is a new minor release (6.1 versus 6.0). I’m not sure if Griffon26 just stopped using coda or if there are other reasons not to have it updated in Portage, but in the next days I might take a look to improve the situation in my overlay at least (where you can also find the two —as-needed fixes).

If codafs won’t allow me to get what I need (a central repository that can be off or online in general, but only online when used), I’ll have to find an alternative solution. The easiest route would be to make Farragut also handle the NFS serving, but considering that I use the data mostly from Enterprise while I develop, it’s likely that I can’t allow it to run on an old and slow IDE drive, and also the size of my current share is not going to fit into it. It’s also bad that I’d have to keep all this data in the WebServer.

My only solution here would be to get another box to run as an NFS server and maybe something more, something running FreeBSD and that wouldn’t do much noise so that it can stay online together with Farragut. Unfortunately I don’t think I can afford a new box just yet, maybe in a couple of months.. in the mean time if NFS is broken, I simply won’t upgrade Farragut for a while, and find a different way to send the backups from one box to the other for safety.