I announced it in February that Excelsior, which ran the Tinderbox, was no longer at Hurricane Electric. I have also said I’ll start on working on a new generation Tinderbox, and to do that I need a new devbox, as the only three Gentoo systems I have at home are the laptops and my HTPC, not exactly hardware to run compilation all the freaking time.
So after thinking of options, I decided that it was much cheaper to just rent a single dedicated server, rather than a full cabinet, and after asking around for options I settled for Online.net, because of price and recommendation from friends. Unfortunately they do not support Gentoo as an operating system, which makes a few things a bit more complicated. They do provide you with a rescue system, based on Ubuntu, which is enough to do the install, but not everything is easy that way either.
Luckily, most of the configuration (but not all) was stored in Puppet — so I only had to rename the hosts there, changed the MAC addresses for the LAN and WAN interfaces (I use static naming of the interfaces as lan0 and wan0, which makes many other pieces of configuration much easier to deal with), changed the IP addresses, and so on. Unfortunately since I didn’t start setting up that machine through Puppet, it also meant that it did not carry all the information to replicate the system, so it required some iteration and fixing of the configuration. This also means that the next move is going to be easier.
The biggest problem has been setting up correctly the MDRAID partitions, because of GRUB2: if you didn’t know, grub2 has an automagic dependency on mdadm — if you don’t install it it won’t be able to install itself on a RAID device, even though it can detect it; the maintainer refused to add an USE flag for it, so you have to know about it.
Given what can and cannot be autodetected by the kernel, I had to fight a little more than usual and just gave up and rebuilt the two (/boot and / — yes laugh at me but when I installed Excelsior it was the only way to get GRUB2 not to throw up) arrays as metadata 0.90. But the problem was being able to tell what the boot up errors were, as I have no physical access to the device of course.
The Online.net server I rented is a Dell server, that comes with iDRAC for remote management (Dell’s own name for IPMI, essentially), and Online.net allows you to set up connections to through your browser, which is pretty neat — they use a pool of temporary IP addresses and they only authorize your own IP address to connect to them. On the other hand, they do not change the default certificates, which means you end up with the same untrustable Dell certificate every time.
From the iDRAC console you can’t do much, but you can start up the remove, JavaWS-based console, which reminded me of something. Unfortunately the JNLP file that you can download from iDRAC did not work on either Sun, Oracle or IcedTea JREs, segfaulting (no kidding) with an X.509 error log as last output — I seriously thought the problem was with the certificates until I decided to dig deeper and found this set of entries in the JNLP file:
<resources os="Windows" arch="x86">
<nativelib href="https://idracip/software/avctKVMIOWin32.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLWin32.jar" download="eager"/>
</resources>
<resources os="Windows" arch="amd64">
<nativelib href="https://idracip/software/avctKVMIOWin64.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLWin64.jar" download="eager"/>
</resources>
<resources os="Windows" arch="x86_64">
<nativelib href="https://idracip/software/avctKVMIOWin64.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLWin64.jar" download="eager"/>
</resources>
<resources os="Linux" arch="x86">
<nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
</resources>
<resources os="Linux" arch="i386">
<nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
</resources>
<resources os="Linux" arch="i586">
<nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
</resources>
<resources os="Linux" arch="i686">
<nativelib href="https://idracip/software/avctKVMIOLinux32.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLLinux32.jar" download="eager"/>
</resources>
<resources os="Linux" arch="amd64">
<nativelib href="https://idracip/software/avctKVMIOLinux64.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
</resources>
<resources os="Linux" arch="x86_64">
<nativelib href="https://idracip/software/avctKVMIOLinux64.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLLinux64.jar" download="eager"/>
</resources>
<resources os="Mac OS X" arch="x86_64">
<nativelib href="https://idracip/software/avctKVMIOMac64.jar" download="eager"/>
<nativelib href="https://idracip/software/avctVMAPI_DLLMac64.jar" download="eager"/>
</resources>
Turns out if you remove everything but the Linux/x86_64 option, it does fetch the right jar and execute the right code without segfaulting. Mysteries of Java Web Start I guess.
So after finally getting the system to boot, the next step is setting up networking — as I said I used Puppet to set up the addresses and everything, so I had working IPv4 at boot, but I had to fight a little longer to get IPv6 working. Indeed IPv6 configuration with servers, virtual and dedicated alike, is very much an unsolved problem. Not because there is no solution, but mostly because there are too many solutions — essentially every single hosting provider I ever used had a different way to set up IPv6 (including none at all in one case, so the only option was a tunnel) so it takes some fiddling around to set it up correctly.
To be honest, Online.net has a better set up than OVH or Hetzner, the latter being very flaky, and a more self-service one that Hurricane, which was very flexible, making it very easy to set up, but at the same time required me to just mail them if I wanted to make changes. They document for dibbler, as they rely on DHCPv6 with DUID for delegation — they give you a single /56 v6 net that you can then split up in subnets and delegate independently.
What DHCPv6 in this configuration does not give you is routing — which kinda make sense, as you can use RA (Route Advertisement) for it. Unfortunately at first I could not get it to work. Turns out that, since I use subnets for the containerized network, I enabled IPv6 forwarding, through Puppet of course. Turns out that Linux will ignore Route Advertisement packets when forwarding IPv6 unless you ask it nicely to — by setting accept_ra=2 as well. Yey!
Again this is the kind of problems that finding this information took much longer than it should have been; Linux does not really tell you that it’s ignoring RA packets, and it is by far not obvious that setting one sysctl will disable another — unless you go and look for it.
Luckily this was the last problem I had, after that the server was set up fine and I just had to finish configuring the domain’s zone file, and the reverse DNS and the SPF records… yes this is all the kind of trouble you go through if you don’t just run your whole infrastructure, or use fully cloud — which is why I don’t consider self-hosting a general solution.
What remained is just bits and pieces. The first was me realizing that Puppet does not remove the entries from /etc/fstab by default, so I noticed that the Gentoo default /etc/fstab file still contains the entries for CD-ROM drives as well as /dev/fd0. I don’t remember which was the last computer with a floppy disk drive that I used, let alone owned.
The other fun bit has been setting up the containers themselves — similarly to the server itself, they are set up with Puppet. Since the server used to be running a tinderbox, it used to also host a proper rsync mirror, it was just easier, but I didn’t want to repeat that here, and since I was unable to find a good mirror through mirrorselect (longer story), I configured Puppet to just provide to all the containers with distfiles.gentoo.org as their sync server, which did not work. Turns out that our default mirror address does not have any IPv6 hosts on it – when I asked Robin about it, it seems like we just don’t have any IPv6-hosted mirror that can handle that traffic, it is sad.
So anyway, I now have a new devbox and I’m trying to set up the rest of my repositories and access (I have not set up access to Gentoo’s repositories yet which is kind of the point here.) Hopefully this will also lead to more technical blogging in the next few weeks as I’m cutting down on the overwork to relax a bit.