New devbox running

I announced it in February that Excelsior, which ran the Tinderbox, was no longer at Hurricane Electric. I have also said I’ll start on working on a new generation Tinderbox, and to do that I need a new devbox, as the only three Gentoo systems I have at home are the laptops and my HTPC, not exactly hardware to run compilation all the freaking time.

So after thinking of options, I decided that it was much cheaper to just rent a single dedicated server, rather than a full cabinet, and after asking around for options I settled for Online.net, because of price and recommendation from friends. Unfortunately they do not support Gentoo as an operating system, which makes a few things a bit more complicated. They do provide you with a rescue system, based on Ubuntu, which is enough to do the install, but not everything is easy that way either.

Luckily, most of the configuration (but not all) was stored in Puppet — so I only had to rename the hosts there, changed the MAC addresses for the LAN and WAN interfaces (I use static naming of the interfaces as lan0 and wan0, which makes many other pieces of configuration much easier to deal with), changed the IP addresses, and so on. Unfortunately since I didn’t start setting up that machine through Puppet, it also meant that it did not carry all the information to replicate the system, so it required some iteration and fixing of the configuration. This also means that the next move is going to be easier.

The biggest problem has been setting up correctly the MDRAID partitions, because of GRUB2: if you didn’t know, grub2 has an automagic dependency on mdadm — if you don’t install it it won’t be able to install itself on a RAID device, even though it can detect it; the maintainer refused to add an USE flag for it, so you have to know about it.

Given what can and cannot be autodetected by the kernel, I had to fight a little more than usual and just gave up and rebuilt the two (/boot and / — yes laugh at me but when I installed Excelsior it was the only way to get GRUB2 not to throw up) arrays as metadata 0.90. But the problem was being able to tell what the boot up errors were, as I have no physical access to the device of course.

The Online.net server I rented is a Dell server, that comes with iDRAC for remote management (Dell’s own name for IPMI, essentially), and Online.net allows you to set up connections to through your browser, which is pretty neat — they use a pool of temporary IP addresses and they only authorize your own IP address to connect to them. On the other hand, they do not change the default certificates, which means you end up with the same untrustable Dell certificate every time.

From the iDRAC console you can’t do much, but you can start up the remove, JavaWS-based console, which reminded me of something. Unfortunately the JNLP file that you can download from iDRAC did not work on either Sun, Oracle or IcedTea JREs, segfaulting (no kidding) with an X.509 error log as last output — I seriously thought the problem was with the certificates until I decided to dig deeper and found this set of entries in the JNLP file:

 <resources os="Windows" arch="x86">
   <nativelib href="https://idracip/software/avctKVMIOWin32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLWin32.jar" download="eager"/>
 </resources>
 <resources os="Windows" arch="amd64">
   <nativelib href="https://idracip/software/avctKVMIOWin64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLWin64.jar" download="eager"/>
 </resources>
 <resources os="Windows" arch="x86_64">
   <nativelib href="https://idracip/software/avctKVMIOWin64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLWin64.jar" download="eager"/>
 </resources>
  <resources os="Linux" arch="x86">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  </resources>
  <resources os="Linux" arch="i386">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  </resources>
  <resources os="Linux" arch="i586">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  </resources>
  <resources os="Linux" arch="i686">
    <nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
  </resources>
  <resources os="Linux" arch="amd64">
    <nativelib href="https://idracip/software/avctKVMIOLinux64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
  </resources>
  <resources os="Linux" arch="x86_64">
    <nativelib href="https://idracip/software/avctKVMIOLinux64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
  </resources>
  <resources os="Mac OS X" arch="x86_64">
    <nativelib href="https://idracip/software/avctKVMIOMac64.jar" download="eager"/>
   <nativelib href="https://idracip/software/avctVMAPI_DLLMac64.jar" download="eager"/>
  </resources>

Turns out if you remove everything but the Linux/x86_64 option, it does fetch the right jar and execute the right code without segfaulting. Mysteries of Java Web Start I guess.

So after finally getting the system to boot, the next step is setting up networking — as I said I used Puppet to set up the addresses and everything, so I had working IPv4 at boot, but I had to fight a little longer to get IPv6 working. Indeed IPv6 configuration with servers, virtual and dedicated alike, is very much an unsolved problem. Not because there is no solution, but mostly because there are too many solutions — essentially every single hosting provider I ever used had a different way to set up IPv6 (including none at all in one case, so the only option was a tunnel) so it takes some fiddling around to set it up correctly.

To be honest, Online.net has a better set up than OVH or Hetzner, the latter being very flaky, and a more self-service one that Hurricane, which was very flexible, making it very easy to set up, but at the same time required me to just mail them if I wanted to make changes. They document for dibbler, as they rely on DHCPv6 with DUID for delegation — they give you a single /56 v6 net that you can then split up in subnets and delegate independently.

What DHCPv6 in this configuration does not give you is routing — which kinda make sense, as you can use RA (Route Advertisement) for it. Unfortunately at first I could not get it to work. Turns out that, since I use subnets for the containerized network, I enabled IPv6 forwarding, through Puppet of course. Turns out that Linux will ignore Route Advertisement packets when forwarding IPv6 unless you ask it nicely to — by setting accept_ra=2 as well. Yey!

Again this is the kind of problems that finding this information took much longer than it should have been; Linux does not really tell you that it’s ignoring RA packets, and it is by far not obvious that setting one sysctl will disable another — unless you go and look for it.

Luckily this was the last problem I had, after that the server was set up fine and I just had to finish configuring the domain’s zone file, and the reverse DNS and the SPF records… yes this is all the kind of trouble you go through if you don’t just run your whole infrastructure, or use fully cloud — which is why I don’t consider self-hosting a general solution.

What remained is just bits and pieces. The first was me realizing that Puppet does not remove the entries from /etc/fstab by default, so I noticed that the Gentoo default /etc/fstab file still contains the entries for CD-ROM drives as well as /dev/fd0. I don’t remember which was the last computer with a floppy disk drive that I used, let alone owned.

The other fun bit has been setting up the containers themselves — similarly to the server itself, they are set up with Puppet. Since the server used to be running a tinderbox, it used to also host a proper rsync mirror, it was just easier, but I didn’t want to repeat that here, and since I was unable to find a good mirror through mirrorselect (longer story), I configured Puppet to just provide to all the containers with distfiles.gentoo.org as their sync server, which did not work. Turns out that our default mirror address does not have any IPv6 hosts on it ­– when I asked Robin about it, it seems like we just don’t have any IPv6-hosted mirror that can handle that traffic, it is sad.

So anyway, I now have a new devbox and I’m trying to set up the rest of my repositories and access (I have not set up access to Gentoo’s repositories yet which is kind of the point here.) Hopefully this will also lead to more technical blogging in the next few weeks as I’m cutting down on the overwork to relax a bit.

No more envelopes!

And finally the job is complete! Tomorrow I’ll probably pass reading Gibson’s Neuromancer to relax, as I’m the tiredness personified.

Now, after having to take a forced break yesterday to fix a nasty KDE regression with KSpell2 (you know, zander, ranting won’t bring you anywhere…), I’m working on the next ebuild for Amarok, 1.4.4.

With this version, there is a partial Karma device support (as I wrote here); but I didn’t get much requests, and no devs seem to have a Rio Karma device (as far as I remember).. if you’re interested in this, we might be able to come up with a proxied maintainership, for the whole Karma support of course.

I’ll be adding an mtp useflag, at least experimentally, as the support should now be improved, and there seems to be always more devices supported by that protocol… libmtp itself seems to be fine on FreeBSD, although it needs build fixes I didn’t prepare yet, and I cannot test it at all (not having a MTP device). I just hope nothing will break, if it will, I’ll just remove the flag.

Also, now that 1.4.3-r1 is stable everywhere, I’ll be removing 1.4.1, and 1.3.8 soon, this means that there won’t be anymore an Amarok that won’t depend on Ruby, and that there won’t be exscalibar anymore (in fact I’ll be asking Hanno to send a last rites as soon as Amarok 1.4.1 will be out of the tree). Yes, I know it could have been better if I got moodbar in the tree first, but I didn’t get too many requests, and I’m still not keen on merging GStreamer at the moment.

Remaining on topic of media players, I’d like to thank nenolod from Audacious team for the help diverting the flames and hatemails about XMMS removal (and thanks Mike for the NiceMail :) I was really happy to receive it, just hadn’t had time to reply you yet because of the job). You never know, maybe one day we’ll have an Audacious engine for Amarok…

And about the devbox I talked about in the past days, seems like we might have an ace up our sleeves to get something, so that I don’t really have to find an extra job to find one :) If that’s the case, I’ll use some of my current job’s money to buy a couple of extra disks, for enterprise, and move the current ones to farragut so that I can get better performance out of it… but I’ll be waiting for that too, because I don’t have much money and I need to decide if I want some 250G drives, or instead faster drives.

Tiredness 2

Seems like one tiredness wasn’t enough. Today I resumed labelling envelopes, and tomorrow I’ll be printing receipts, and the day after will be closing the envelops and afterward attaching the receipt to them. It’s a pita of a work, but I’m paid for it (not that well actually, but better than nothing).

Of course there’s also the boring stuff with XMMS. Of course the news that it will go way from Gentoo finally was misrepresented and misunderstood as me forcing users with all the rights on this world to use Audacious just for the sake of it. I know it’s only a vocal minority that is currently complaining, but I want to make it clear.

I was an XMMS user, and it took me quite a while before jumping on the Beep Media Player bandwagon, after that I used Audacious for a while and then moved to Amarok for good.. this latter move was also due to my move from a random listener to a more assiduous one (as I’ve started ripping my CDs and playing them from harddisk). This means I don’t have a personal grudge against XMMS (I used to use it too), and I don’t have anything against GTK itself this time (well, I don’t like GTK, but it’s not like I’m forcing everybody to use Qt… I think everybody has his taste). I’m not an Audacious guy either, although I do like it for random listening, and I’m glad to help from time to time (warning fixes, porting on FreeBSD, translations, …), I think that they are doing a good deal of work, so thanks a lot to Chainsaw, nenolod and the rest of the developers.

Why I was happy of getting rid of XMMS then? Because I think we should have phased it out last year already.. there were way more bugs open, okay, but I think Luis invested too much time on such a dead project, and although I’m not sure of this, I’d be ready to bet that this contributed on his stress. As Tony said, this is the XMMS curse, every one of the developers maintaining XMMS in the past ended up exhausted. This is quite easy to understand, although from one site the wide availability of XMMS plugins means that you can play almost everything and to almost every output system, and you can do quite interesting things with the misc plugins, the combinations are so many that you cannot test for them. Also, maintaining a project dead upstream is basically suicidal, as you end up maintaining every part of the code, not just the packaging and building.

This is not the first time XMMS faces this destiny. Last year I already announced the phase out plan, but then Luis took it over and so it was saved from the coming doom. Unfortunately as you can see, Luis only postponed the final cvs rm, as now we’re at the same point. This is why I wasn’tgoing to write once again a maintainer requrest on gentoo-dev and waiting to hear before going with the mask. Nobody in the sound herd is going to take care of it, and any other person wanting to do that should first be able to fix all the bugs for good and then make sure he has at least one fall back when he’s going to retire… as I doubt anyone is going to do that, XMMS is going away for good.

As I said, I won’t be putting it in an official overlay, because having an official overlay with unsupported software is just silly, a bad move in PR and a bad move in QA. I won’t stop anybody going to create an XMMS overlay, but be assured that we won’t support it in any way, and the way it currently works (faad and flac using the xmms useflag to build the plugin), if the libraries are overlaid, we’re not going to support you with them either.

After all, I think there are waaaay more users that will be happy with the reduced sync time than there will be pissed off by the removal of such an old piece of broken code like XMMS. If it worked for you it does not mean it works for everyone!

Uhm, I started this entry thinking of writing something about my plans for Gentoo/FreeBSD, but then I talked about XMMS up to now, so I’ll try to be short now. First of all, adding enterprise as buildbox for Prakesh helped, at least a bit, kdelibs took just 2 hours instead of 3, it’s still a lot, but at least a bit better than before. As farragut is still on 6.1, I need to find a way to be able to install a cross-compiler to 6.2 on a 6.1 system… right now it does not work that well, but I’m going to improve the situation soon. I’m also thinking of using an old 533 Pentium3 as an extra distcc box, although it won’t help that much, I’m sure.

Also, I now realise that the only way to make sure that Gentoo/FreeBSD can be considered a first class citizen of the Gentoo project, is to provide access for the developers.. a simple way to do this would be having a box online, with a lot of space on disk, a jail for generic usage for developers and one each developer if they are going to do something more complex projects. The problem is that right now Gentoo/FreeBSD can be pretty tricky and needing a manual tinkering, which means someone who understand G/FBSD should have direct access to the box.. I do have a friend who could house the box for me free of charge, but still remains the problem of the box itself.. needs to be at least a dual processor I suppose, and with big disks, being able to leave at least the core system on mirroring… which means it will take many months for me to be able to even think of buying it… I don’t even have the money to buy a new box for me ;)

But anyway the idea is not totally discarded yet.. I’d like to hear if someone can provide me with some estimate of how much I could end up spending, it might make my mind clearer.. after that maybe I might end up asking if anybody wants to contribute… but as I said, I feel that would be really only the last choice I have, unless people feel really motivated to.