The end of an era, the end of the tinderbox

I’m partly sad, but for the most part this is a weight that goes away from my shoulders, so I can’t say I’m not at least in part joyful of it, even though the context in which this is happening is not exactly what I expected.

I turned off the Gentoo tinderbox, never to come back. The S3 storage of logs is still running, but I’ve asked Ian to see if he can attach everything at his pace, so I can turn off the account and be done with it.

Why did this happen? Well, it’s a long story. I already stopped running it for a few months because I got tired of Mike behaving like a child, like I already reported in 2012 by closing my bugs because the logs are linked (from S3) rather than attached. I already made my position clear that it’s a silly distinction as the logs will not disappear in the middle of nowhere (indeed I’ll keep the S3 bucket for them running until they are all attached to Bugzilla), but as he keeps insisting that it’s “trivial” to change the behaviour of the whole pipeline, I decided to give up.

Yes, it’s only one developer, and yes, lots of other developers took my side (thanks guys!), but it’s still aggravating to have somebody who can do whatever he likes without reporting to anybody, ignoring Council resolutions, QA (when I was the lead) and essentially using Gentoo as his personal playground. And the fact that only two people (Michał and Julian) have been pushing for a proper resolution is a bit disappointing.

I know it might feel like I’m taking my toys and going home — well, that’s what I’m doing. The tinderbox has been draining on my time (little) and my money (quite more), but those I was willing to part with — draining my motivation due to assholes in the project was not in the plans.

In the past six years that I’ve been working on this particular project, things evolved:

  • Originally, it was a simple chroot with a looping emerge, inspected with grep and Emacs, running on my desktop and intended to catch --as-needed failures. It went through lots of disks, and got me off XFS for good due to kernel panics.
  • It was moved to LXC, which is why the package entered the Gentoo tree, together with the OpenRC support and the first few crude hacks.
  • When I started spendig time in Los Angeles for a customer, Yamato under my desk got replaced with Excelsior which was crowdfounded and hosted, for two years straight, by my customer at the time.
  • This is where the rewrite happened, from attaching logs (which I could earlier do with more or less ease, thanks to NFS) to store them away and linking instead. This had to do mostly with the ability to remote-manage the tinderbox.
  • This year, since I no longer work for the company in Los Angeles, and instead I work in Dublin for a completely different company, I decided Excelsior was better off on a personal space, and rented a full 42 unit cabinet with Hurricane Electric in Fremont, where the server is still running as I type this.

You can see that it’s not that ’m trying to avoid spending time to engineer solutions. It’s just that I feel that what Mike is asking is unreasonable, and the way he’s asking it makes it unbearable. Especially when he feigns to care about my expenses — as I noted in the previously linked post, S3 is dirty cheap, and indeed it now comes down to $1/month given to Amazon for the logs storage and access, compared to $600/month to rent the cabinet at Hurricane.

Yes, it’s true that the server is not doing only tinderboxing – it also is running some fate instances, and I have been using it as a development server for my own projects, mostly open-source ones – but that’s the original use for it, and if it wasn’t for it I wouldn’t be paying so much to rent a cabinet, I’d be renting a single dedicated server off, say, Hetzner.

So here we go, the end of the era of my tinderbox. Patrick and Michael are still continuing their efforts so it’s not like Gentoo is left without integration test, but I’m afraid it’ll be harder for at least some of the maintainers who leveraged the tinderbox heavily in the past. My contract with Hurricane expires in April; at that point I’ll get the hardware out of the cabinet, and will decide what to do with it — it’s possible I’ll donate the server (minus harddrives) to Gentoo Foundation or someone else who can use it.

My involvement in Gentoo might also suffer from this; I hopefully will be dropping one of the servers I maintain off the net pretty soon, which will be one less system to build packages for, but I still have a few to take care of. For the moment I’m taking a break: I’ll soon send an email that it’s open season on my packages; I locked my bugzilla account already to avoid providing harsher responses in the bug linked at the top of this post.

Munin, HP servers and APC powerstrips

Yes, I know I start to get boring.

Today I spent at least half of my work day working on Munin plugins to monitor effectively some of the equipment we currently have at our co-location. This boils down to two metered APC powerstrips PDUs (let’s use their term, silly as it might sound). I think it’s worth to note the difference: APC provides switched and metered PDUs; the former should allow for having per-plug load data, and powering on and off of the single plug; the latter (what we have here) is much cheaper, does not allow you to turn them on and off, and simply give you a reading of the load per-phase. Given that our co-location has only single-phase power, we only get a reading per strip, which is still okay, it gives us more information than we had before at least.

Now, there are a few funny things with these strips: they have a network interface, which is cool, but they don’t use DHCP by default! You either have to set them up with the serial interface (which obviously is still very serial, not an USB adapter — and my laptop doesn’t have any serial port), or use a Windows software (which is actually written in Java and spend 98% of the install time copying an extra install of the JRE to the drive), or finally note down the MAC address when you install them, and then poisoning a system’s ARP table to “fake” an IP to the strip, sealing the deal by sending a 113 bytes ICMP packet to the strip via ping … no there is no use for a watermelon or a chimp, sorry Luca.

After finally completing the IP settings, I had to find my way to get the data out; the strips support either SNMPv1 or SNMPv3 — I discarded the former simply because it’s extremely insecure and I’d rather not even have that around, so I set up an user for munin. Next problem? snmpwalk did not report any useful data. The reason is actually quite simple: it doesn’t know which OIDs to probe for. Download the MIB data from APC and install it in the system, and it’s much happier.

Then I had to write a plugin for it. Which wasn’t too bad; the data is simple, too bad I couldn’t find a way to get, through SNMP, the high limit of current drain on the strip — it did report the configured (default) limits for near-overload and overload, which makes it very nice to set them up in Munin. Unfortunately only after writing the plugin I found out that the Munin contrib repository had already not one but two plugins trying to do the same. Neither is very good with it though: neither supported Munin’s SNMP framework, one had a very unclear licensing situation (which is unfortunately common on the contrib repository), and used sh and net-snmp’s command-line utilities to access the strip.

So after adding my plugin, and removing the two bad ones, I also looked into cleaning up the contrib tree a little bit. It’s far from perfect, there are still miscategorized plugins and duplicates, and others (such as one of the net-p2p/transmission monitors) which rely on external script files instead of being written in a single one. But at least I was able to remove and recategorize enough of them that it starts to make some sense. If you’re a Munin user and would like for Gentoo to provide more, better plugins, then please take your time to see which of the plugins currently in the contrib tree are trying to reimplement something and failing at it (lots of them I’m afraid will be, especially those related to APC UPSes), and get rid of them. There is also work to be done to bring even only the documentation of the plugins up to speed with the format used by Munin proper, and this is without talking about improving them to follow the right code style or anything.

I also spent some time improving my IPMI plugin (which you can find now on the contrib repository if you’re not a Gentoo user – if you’re a Gentoo user it takes the place of the original IPMI plugins shipped with Munin – after I removed all the others that were trying to do the same thing sometimes with twice as many lines of code than mine), and now it can monitor foreign hosts as well. How is this useful? Well, among other things it lets you monitor Windows boxes and other boxes where you either lack access or you can’t install any IPMI tool (I have a couple of systems that are running RHEL4 to monitor, okay?).

One interesting thing I learnt out of this experience is that it makes total sense to monitor voltages at least on HP servers. Beside the idea of monitoring for a PSU gone wrong, HP has one probe set to the CMOS battery, which is a 3V CR2032 Lithium Battery which will provide decreasing voltage, and thus will show in the list when it has to be replaced — unfortunately it also seems like their newest servers don’t have a probe there, which is bad (Excelsior has a VBAT which seems to be just the same thing).

This is all for today!

A good reason not to use network bridges

So one of the things I’m working on for my job is to look to set up Linux Containers to separate some applications — yes I know I’m the one who said that they are not ready for prime time but please note that what I was saying is that I wouldn’t give root inside a container to anybody I would trust — which is not the same as to say that they are not extremely useful to limit the resource consumption of various applications.

Anyway, there is one thing that has to be considered, of which I already quickly wrote about : networking. The simplest way to set up a LXC host, if your network is a private one, with a DHCP server or something along those lines, is to create one single bridge between your public network interface and the host-side of virtual Ethernet pairs — this has one unfortunate side effect: to make it working, it puts the network interface in promiscuous mode, which means that it receives all the packets directed to any other interface, which slows it down quite a bit.

So how do you solve the issue? Well, I’m honestly not sure whether macvlan improves the situation, I’m afraid not. What I decided for Excelsior, since it is not on a private network, was to set up an internal bridge, and have static IP addresses set to internal IPs. When i need to jump into one of the containers, I simply use the main public IP as an SSH jumphost and then connect to the correct address. I described the setup before although I made then a further change so now I don’t have to bother with the private IP addresses in the configuration file: I use the public IPv6 AAAA record for the containers, which simply resolve as usual once inside my jumphosts.

Of course with the exception of jumphosts, that kind of settings, which involve using NAT on iptables, has no way to receive connections from the outside.

So what other options are there? One thing I’ve been thinking about was to use a level-3 managed switch and set it to route a subnet to the LXC host — but that wouldn’t fly too much. So at the end the question would be “what is it that I need access on the containers form the outside?” and the answer is simply “the websites”. The containers provide a number of services, but only the websites are mapped to the outside. So, do I need IPs that are even partially public? Not really.

The solution I’m planning right now is that I’ll set up a box with either an Apache reverse-proxy or some other reverse proxy (depending on how much we want to handle on the proxy itself), and have that contact the internal containers, the same way it would be if you had one reverse proxy on the Internet, and the servers on the internal network.

I guess at some point I should overhaul the LXC wiki page for what concerns networking; I already spent some time to remove some duplicated content and actually sync it with what’s going on on the ebuild…

More on the SuperMicro iKVM

In my previous post I narrated my adventure trying to get a memtest running on Excelsior — at the end I was able to run it through a modified JNLP, and one more open port. I was suggested to look into one particular Java application from Supermicro that does not require using the JNLP at all, and instead installs as a client, giving access to more features such as the SOL and generic IPMI control…

Unfortunately it seems like the installer (InstallShield for Linux, written in Java — what the heck?) is extremely picky as to which JRE it find and which one it wants, so I haven’t been able to even test it. But at least I got some details going.

I basically listened with Wireshark to what’s going on with the JNLP; the interface uses four combined interfaces, between the browser and Java: port 80, 443, 5900 and 623. The first two are the HTTP/HTTPS interfaces (the JNLP downloads the JARs from HTTPS even thought hey are available on HTTP as well); the third is the VNC/RFB default port, while the fourth is the one that I haven’t understood yet. It’s some kind of USB over IP protocol, and seem to send raw USB data over the wire, standing to the USBC/USBS instances in the trace, but at the same time it doesn’t seem like it’s using it at runtime, as I see no traffic on that port after I connect the ISO file.

The RFB protocol used seems to be the standard one using TightVNC extensions for authentication — I guess they actually used TightVNC’s code for it. The problem with the authentication is that for whatever the reason, it’s not a clear user/password auth. Instead it uses some hash or unique identifier, which changes each time I connect to the web interface — I’m not sure if it’s a hash, it’s definitely not an OTP (as I can start multiple instances of the javaws applet without having to re-download the JNLP), or just a nonce-based authentication, but it’s used both as user and as password.

Edit: actually I had a hunch while looking into it and I confirmed that what it uses is the same SID saved as a cookie after my login on the web interface. Now if I could get the iKVM viewer to work on my system and I could see how that one connects…

The USB over IP protocol is interesting as well; it doesn’t seem to use a standardised port, and Wireshark has no clue as to what’s going on there. As I said I can see USBC and USBS as literals within the traffic as well as the data for the ISO and some other not-well-explained things — I’ll have to work more on that, possibly with smaller files, and without the RFB in the trace.

Does anybody else have clues about this kind of protocols? For what I can tell the firmware for my board’s IPMI contains a copy of Linux (this is what nmap said as well), but I see no released sources for it, nor an offer for them on the zip file I downloaded. I wonder if I should just mail SuperMicro to ask them about it.

Trouble with memtest

So there is something that doesn’t feel extremely right on the tinderbox — as in there are a few logs spotting ICE and a few killed Ruby processes that make no sense at all. This is usually indication of bad memory. Now it is true that I ran a one-pass test of the memory when I got the system, and it didn’t spot anything, and that this does not happen consistently, so I wanted to give it a 24 hours memory testing — it should be easy thanks to a serial console and the memtest86+ software, no?

Well, no. Let’s start with a bit of background on the setup. Excelsior, the server that is running the two tinderbox instances, is a Supermicro barebone, which integrates an IPMI 2.0 SBMC. This allows me to control it just fine from here (Italy) while the server is back in Los Angeles. At least in theory; while the server is on a public IP, the IPMI interface is connected to the VPN of my employer back in the US, so to actually connect to it I have to jump through an SSH host — which is easy done on Linux — but not on Windows.

The serial console, by the way, is tremendously easy to get by as you can simply SSH into the IPMI IP and you can use some semi-standard commands to get to it, in particular you just need to run cd /system1/sol1 and start. Unfortunately, my first blind setup of grub (2) and the Linux kernel was wrong, as I set them to output to ttyS0 — while the IPMI is forwarding you ttyS1. And finding how to set up grub2 to use a serial console wasn’t easy.

What you have to do is editing /etc/default/grub and add this:

GRUB_TERMINAL="serial"
GRUB_SERIAL_COMMAND="serial --unit=1 --speed=115200"

This will set it to use the second serial interface (--unit=1) as 115200,8n1 (8n1 is the default). And there you go. Actually the command editing seems to work more reliably on the serial console than on the display.

So this is done, what’s the next? Well, next is getting memtest to work — it doesn’t help that the pre-compiled binary provided by the upstream project is not able to start. The problem is a “too small lower memory” which is caused by a combination of grub, BIOS and the compiled file itself. While for some systems it’s enough to use a custom compiled version such as the one provided by sys-apps/memtest86+, on Excelsior that didn’t work. So I had to go with the fallback: the ebuild installs both an old-style Linux kernel bootable binary and a netbsd-style binary; as far as I can tell the latter does not support boot parameters though.

The correct way to boot that particular method on Grub 2 is editing /etc/grub.d/40_custom and add:

menuentry "Memtest86+" {
    insmod bsd
    knetbsd /boot/memtest86plus/memtest.netbsd
}

For those wondering, yes I’m working as we speak on ebuilds that install the grub2 extra configuration file by themselves, and you should have them by the end of the day. This involves both memtest86+ and, as you’ll see in a moment, memtest86 itself. This will make it much easier to deal with the packages.

Ah of course this has to be built on my laptop, because both memtest86+ and memtest86 require a multilib compiler as they are built 32-bit. Sigh.

Unfortunately, not even that is good enough for my system. While with this code it boots, instead of refusing to do so, it seems to get stuck during initialisation and the test never starts. But how do I know that, given that memtest by default does not output to serial, and when it does it outputs on serial 0?

Well, the IPMI interfaces actually has what they call an iKVM written in Java, not to be confused with another Java IKVM — the problem with it is that it doesn’t work with IcedTea and thus you have to use Oracle’s JRE to run it; the bug involves not only Supermicro systems, but also Dell and others. Why it hasn’t been solved on the IcedTea side is something I have no idea about.

While the package uses a standard RFB/VNC protocol, it implements authentication in a non-standard way for what I can tell, so I can’t simply login as I’d like to do. It also probably either has some extensions or an extra signalling protocol, as it can be used to set up “virtual media” such as virtual CD-Roms and virtual floppy images.

Now, this latter detail should give me enough to deal with the memtest issue, as I’d just have to connect a virtual ISO of memtest to get it to work but … Java segfaults (it uses a native library) the moment I try to do so! I have yet to check whether this is simply because it’s trying to use a signalling port that is unavailable, but it doesn’t feel very likely.

Okay so memtest86+ boots as a NetBSD-style kernel, but it doesn’t seem like it’s able to do anything — what about the original project? Memtest86 is still alive and kicking, and released a new version last October (called 4.0b but versioned 4.0s) which supports “up to 32 cores and 8 TB of memory” — reading such release notes I’m afraid that the reason why Memtest86+ doesn’t work is simply that it’s too much memory, or too many cores (32 cores, 64GB of memory).

Unfortunately Memtest86 doesn’t seem to build a NetBSD kernel style binary, so I can’t boot that either. Which means I’m stuck.

Interestingly, Memtest86+ released a 5.00beta back in May — unfortunately there are two big problems: there is no download for the NetBSD kernel, and most worrisome, they didn’t release the sources. Given the project is GPL-2 and includes code from the original Memtest86 project (which the maintainer of Memtest86+ has no way to claim rights to), this is a license violation.

So now I’m using the user-space memtester, which seems to work fine even on hardened kernels, with the caveat that it doesn’t allow to test the full range of memory but just a little piece of it at a time. Sigh. No easy way out, eh?

The latest development in Tinderboxing: semi-automated bug filing

Beside criticising Intel for their x32 ABI I’ve spent the last few days working hard on the Tinderbox to make sure that the GCC 4.7 failures are identified quickly.

While the analysis of the logs is now vastly automated compared to earlier efforts, I’m still not using automated scripts for bug filing, and I file them by hand. How did I do that? Well, before I used delicious to store a number of bug templates for situations such as fortified sources, but this is no longer a viable option… for one simple reason. When the service was bought off Yahoo!, instead of focusing on the one reason why I’m pretty sure a number of users stopped using it (namely, the lack of a decent extension for Chrome), they decided to re-make it in a more “web 2.0” fashion…

… the problem is that it came with what I would define “web 0.2” developers, and the data import caused corruption in the (very long) URLs of those bug templates.

I had people contribute possible ways to deal with automated bug filings from the tinderbox, but all of these were fragile, because the interface between Bugzilla and the scripts has changed already too many times, and you need to be able to create the bug to attach a file to it (okay it’s not strictly true but it’s the case most of the times).

What did I decide to do then? Well, the output of the logs analysis is displayed as a webpage, through Sinatra, and when I’m doing the analysis of the logs I can come up with a little more data than before… and the very same system I used to file the bugs with templates can be used with a simple link. This means no worrying about Bugzilla authentication, credential sharing, bug filing and attachment adding.

What happens now is that near to every link for the build log, I have a [B] (which I’d love to replace with a 16×16 icon of a bug if somebody feels like finding or drawing one! — I could also use a wrench with a red bar to indicate a log with build failure, and a magnifying glass similarly barred for test failures). This is actually a link to an half-filled template, with the URL field pointing to the build log, the package name already set in the subject, and most importantly the maintainers already properly assigned.

The only remaining issue was having a way to tell the template which emerge --info posting, so for now it’s simply using a file on the system, with as name the hostname from which the log has been sent — this way I can easily set up multiple tinderboxes, after all the emerge --info output doesn’t change that often. The alternative would be to have a Portage feature that always outputs the full info at the top of the build log.

And to finish this off — I said before I can set up multiple tinderboxes with this new box. Indeed next week I’ll set a new one up, amd64 stable with hardened (right now it’s targeting ~arch and only runs hardened kernel). I was considering setting one up for x32 but then I would just end up filing bugs over bugs over bugs for no real gain and I don’t think the rest of the guys will like that.

The virt-manager pain

So I’m still trying to come up with a decent way to parse those logs. Actually I lost a whole batch of them because I forgot to rename them before processing, but that made me realize that the best thing I can do is process the logs outside of the tinderbox itself. But this is a topic for another time.

Another thing I’m trying to set up is a virtual machine to test the new x32 ABI that Mike has made available in Gentoo. This is important for the tinderbox as well as for FATE (which is already being run for a standard Gentoo Hardened setup on that very same hardware). Unfortunately I can’t use this via LXC, yet — simply because the kernel currently running does not support the x32 executables yet.

This means that I have to use KVM and go full-virtualisation. Thanks to Tim, I’ve got a modified SysRescueCD ISO that should let me take care of the install. Unfortunately, this is not easy to deal with for a number of reasons, still.

The first is that virt-manager is just slow, and painful, as some kind of slow and painful death. The whole idea of using a frontend that connects through SSH is that you don’t want to “feel” the lag… but virt-manager makes you feel the lag even more than a command-line SSH connection. I’m under the impression that the guys who work on that kind of stuff only ever tried this on a local connection, and never from the other side of the world. I mean, I understand you might have concurrency issues, but do you really have to make me wait for two minutes to switch from CPU settings to Memory settings to Disk setting when editing the VM?

The second issue is that even though I was able to set up a testing VM for x32… qemu doesn’t like the additional instruction sets (SIMD) that Bulldozer comes with; something within the C library causes every x32 binary to be killed with SIGILL (Illegal Instruction). The problem is likely in some of the indirect-binding functions that are being used — my guess based on the NEWS file of the 2.15 release is strcasecmp() which has been optimised through the use of SSE4.2 and AVX (both of which are available on that server) — I have a 34 written, half drawn post about this kind of optimisations in my queue, I’ll see if I can post it over the weekend.

The end result is that I spent the most part of three hours on virt-manager before accepting that the way to go is to update the host’s kernel and just run the usual container. Just so you know, the final step that “creates” the VM (which is not the LVM allocation step!) took me over half an hour. This is silly, what the heck was it doing during that time?

Oh and yes, two years afterwards virt-manager still keeps defunct ssh processes around because it never reaps them (check the comments).

Right now I’m trying to get this to work with LXC, but I’m not having much luck with it either; and yes I did update the init script to handle correctly the x32 containers, that didn’t work correctly before… it might have some problems if you’re going to use this on SPARC, because I’m not handling those properly yet, but this is (again) a topic for another time.

Tinderboxing again

Finally Excelsior is doing what it has been bought to do: tinderboxing. Well, not really to full capacity but I’ve got enough pieces of the puzzle in place that it should soon start building regularly.

What the tinderbox set up is doing now is actually a limited run: it’s building the reverse dependencies of virtual/ffmpeg to find out which packages actually require libpostproc, so that we can fix their dependencies with libav — to follow up on Tomáš’s post I’m also strongly attached to libav, being, together with Luca, part of the development team of the project.

Speaking about libav, one of the things that since a couple of days ago Excelsior is doing, is running FATE so to both cover AMD Bulldozer’s architecture and Gentoo Hardened. For those who do not know, FATE is an automated testing environment that configures, builds, and run the tests for libav and then send the result to a central server for analysis. The whole build takes three minutes on Excelsior. Thanks to libav’s highly-parallelized makefiles.

There is only one problem with the Tinderbox, and is once again the logs analysis. To be honest this time I have some idea on how to proceed: the first step is to replace the grep command I have used before with a script that produces an HTML output of the log file. Yes many people said before that HTML is not a good idea for this kind of thing: since nobody else has helped me writing a better log analysis tool, that’s going to be enough.

So what has to happen is reading line by line the input log, then create an HTML file with (numbered) lines, marked with a specific (CSS) file that makes the row red. As I said before it would be nice to have some kind of javascript to jump from one error line to the other. Until this is all set, though, just creating the HTML file would be enough.

The next step would probably be getting the HTML output on S3 (for easy access), and write it down on a database that actually give you an index of the error logs — for those who wonder, using s3fs as PORT_LOGDIR is not a good idea at all. I guess tomorrow I might try to find some time to work on this. I could find some time on the weekend, given that my original idea (biking around with Luca) is disabled due to me falling and (lightly) injuring myself last night.

Securing Tinderbox network access

For those who wonder why Excelsior is now built for a week and it’s still not running in full capacity, the reason is relatively simple: I’m still trying to figure out how to handle network security so that I can give other developers access to it, and not risk that we have a security breach in other places.

The first problem is that the current setup I need to use and the final one will be very different: right now the system is connected to the internal network of my employer, while in production it will be DMZ’d, with a public IP available straight to it. The latter setup will let me set up an IPv6 tunnel, which means all the internal containers will be available by themselves, while the current one prevents me to set it up at all.

Previously this was much easier to deal with simply because I had control over an external firewall myself, which took care of most of the filtering, which combined with LXC networking options made it decently easy to deal with. Unfortunately this time I have to do without this.

One of the things that might not be obvious from the post above, nor from the documentation, is that the three setups I have drawn are the ones that require the least amount of configuration on the host-side: if you use the standard bridge, you just need to start net.br0 (or equivalent), and LXC will connect the host-side virtual Ethernet pair to the bridge by itself. If you’re using the MACVLAN modes instead, the containers will piggy-back the interface of the host, and then rely on the external router/firewall/switch to deal with the configuration — by the way I’m glad that man lxc.conf now actually says that you’re supposed to have a reflective switch to use those modes. You can probably guess it’s not that great an idea if you expect lots of internal communication.

What I’m going to do for now is setting up a system that is defended as much as possible by depth, with iptables carrying out enough filtering that I should be realistically safe. Unfortunately just iptables is not enough and what you need is iptables and ebtables (for Ethernet Bridging filtering), to make sure that the containers’ don’t dupe your IPs or something.

The idea is that one IPv6 is enabled, you can jump straight into the tinderboxes, which is required to control it, but until then, the host system acts as a jump host through a custom scponly setup, which only allows forwarding, as ProxyCommand, port 22 of others boxes within that same system.

I’d like to show more documentation of what I’m trying to do, and what I achieved already, but to do so, I’m afraid I’ll be needing some more … visual examples, as it’s very hard to explain it in words, while it should be much more clearer with a series of drawings. I guess I’ll start working on them soonish, maybe if Luca can package Synfig for me…

For now, this is enough, I guess.

Hardware identification, version bumps, Excelsior

Let’s start with the good news: most of Excelsior has arrived and it’s already set up. The only thing that is missing is … the CPUs, which are coming in from Philadelphia, and should arrive here tomorrow, standing to Amazon’s tracking. As I said, the server will be co-located by my current employer, so that’s one issue not to worry about.

Without Yamato, it turns out that my ability to bump the version of my own packages is vastly reduced, mostly because I don’t want to install packages such as MongoDB on this laptop just to test out Ruby Gems, and at the same time I don’t want to have too many extraneous packages installed. Luckily this means that starting tomorrow we should be all ready to start the install phase.

One of the things I’ve been keeping busy with was the split hardware IDs package — sys-apps/hwids, which I’m bumping weekly. This from one side makes it much less important to use the (now gone) network cron scripts to update the IDs files, and on the other allows people who don’t want their systems to access the network directly to be kept up-to-date with the files themselves. This is the first week I’m skipping over the bump, simply because … there is no new content!

I’ve added a new device today to the USB IDs database though so that should mean that next week we might have an update. And tomorrow I’ll probably update it with the possibly missing subsystem IDs for the devices on Excelsior, which will go to the PCI IDs database where I already sent my laptop’s and one of the local server’s subsystems.

Speaking about device identification I can understand why Kay thinks that it might be better to have a general database of everything, instead of multiple small databases… for instance it would be nice if I could just update one database with the IDs of my new external HDD (WD My Passport), and let smartctl know that it has to connect to it with the SAT method, instead of having to write it on a page and then remember about it myself. Speaking about which, WD still is my favourite HDD vendor.

Anyway, thanks once more to all the people who helped the new Excelsior to be built; tomorrow I’ll post a few more details about it, including some photos hopefully, as I’ve got my camera with me as well. There has actually been some trouble with the SSDs and the mounting bays, which I think would be a valuable lesson not only for me.