Remotely upgrading BIOS on a SuperMicro servers

So it’s that time of the year again when you have to run a few BIOS updates; and you know I love updating BIOS. In this case the victim is the very Excelsior that I bought to run the Gentoo tinderbox, and which I’m hosting to my expenses at Hurricane Electric in Freemont. This is important to say, because it means I have no direct access to the hardware, and nobody else can help me there.

First, I decided to get rid of the low-hanging fruit: the IPMI sideboard. I couldn’t find release notes for the update, but it went from version 2.05 to 3.16, and it was released at the end of June 2014, so it’s pretty recent, and one would expect quite a bit of bug fixes, given that I had trouble running its iKVM client two years ago already, and I tried to get more information about its protocol without success.

For this part, SuperMicro is strangely helpful: the downloaded archive (once I was able to actually download it, it took me a few tries, the connection would break midway through the download) is three times the size of the actual firmware, because it comes with Windows, DOS and, helpfully, Linux update utilities in the same file. Which would be endearing if it wasn’t that the Linux utilities require the tools to access the BMC raw, which is not fun to do with grsec present, and if I’m not mistaken it also conflicts with the in-kernel IPMI drivers. Also, the IPMI remote interface can handle the firmware update “quite well” by itself.

I say “quite well” because it seems like ATEN (the IPMI vendor for SuperMicro and another long list of server manufacturers) still has not mastered the ability to keep configuration consistent across firmware upgrades. You know, the kind of things that most of our home routers or cable TV boxes manage nearly all the time. And all my home computers’ BIOSes. So instead it requires you to reset the full configuration when you upgrade the firmware of the device. Which is fine because you can use bmc-config from userspace to dump the configuration (minus the passwords) and reset it back. If the configuration it downloads is actually accepted as valid, which in my case it wasn’t. But I only needed to set an admin password and IP address and routing, so…

The second step is to update the BIOS of the server itself. And here is where the fun part starts. You have two options to download: a Windows executable or a ZIP archive. I downloaded both. The Windows executable is supposed to build a floppy to upgrade the BIOS. Not a floppy image, a floppy. It does indeed not work in Wine, but I tried! I settled for the ZIP file.

So the iKVM I just upgraded supports “virtual media” — which means you can give it files on your computer that it’ll use instead of floppy disks or CD-Rom. This is very neat, but it comes with some downsides. At first I wanted to give it a floppy disk image via HTTP, and use the Serial-over-SSH to enter the boot menu and do stuff. This has worked like a charm for me before, but it comes with limitations: the HTTP-based virtual media configuration for this iKVM only supports 1.44MB images for floppy disks, and uses SMB to access shared ISO files for the CD-Rom. It makes sense, you can’t expect the iKVM board to have enough RAM to store a whole 650MB image for the CD-Rom, now, can you?

The answer is of course to build a BIOS upgrade floppy disk, which is after all what their own Windows tool would have done (albeit on a real disk rather than an image — do people really still use floppy disks on Opteron servers?) Too bad that there is a small problem with that plan: the BIOS itself is over 2MB uncompressed. It does not fit in a floppy disk unformatted, let alone in the formatted disk, with the update utility, the kernel and the command interpreter. So no way to do that easily. The astutes of you after reading the blog post will probably figure out there was a shortcut here, but I’ll comment on that later.

So I decided to go back to the iKVM client, the Java WebStart one. Now luckily the new release of it works fine with IcedTea, unlike the previous one. Unfortunately it still is not working out of the box. Indeed it was only by chance that I found someone else who faced and solved that problem by adding a couple of lines to the XML file that defines the JavaWS environment.

The configuration dialog for the client allows providing a local ISO file as well as configuring the address of a SMB share with it: it implements its own transmission protocol, and the floppy option is now “floppy/USB”, so I thought maybe I could just give it my 1GB FreeDOS-based BIOS disk … nope, no luck. It still is limited to 1.44MB it seems.

So I had to get a working ISO and do the upgrade. It should be pretty straightforward, right? I even found a blog post showing how to do so, based on Gentoo, as it says to emerge isomaster! Great! Unfortunately, FreeDOS download page now provides you with a great way to download … installers. They no longer provide links to download live CDs. It makes total sense, right? Who doesn’t think every day “I should stop procrastinating and finally install FreeDOS on my laptop!”?

After some fiddling around I found a tiny non-descript directory in their FTP mirrors that includes fdfull.iso which is the actual LiveCD. So I download that, copy the BIOS and the updater into the CD and set it in the virtual media settings for the iKVM, and then boot. The CD does not boot unattended. If you leave it unattended it’ll ask you what you want to do and default to boot from the HDD. If you do remember to type 1 and press enter at the SysLinux prompt, it’ll then default to install (again, why did I bother installing Gentoo on the NUC when I could have used FreeDOS like the cool kids?).

Instead, you can choose to boot FreeDOS from the LiveCD with no support and no drivers, with just the CD drivers, with the CD drivers and HIMEM, with the CD drivers, HIMEM and EMM386. I decided I wanted to play it safe since it’s a BIOS update, and booted with just the CD drivers (I needed the BIOS files after all). Turns out that while this is an option at the boot menu, it will never work, as the CDROM driver needs to be loaded high, MSCDEX.EXE style. So reboot once again with HIMEM (and EMM386 because rebooting into that CD is painful — among other things, the iKVM client loses focus every time the framebuffer size changes. And it changes. all. the freaking. time. during boot up), and try to access the CD… unfortunately the driver does not support USB CD-Rom (like the one the iKVM provides me with) but just IDE ones, so no access to the ISO for me. Nope.

But I’m not totally unprepared for this kind of situations. As a failsafe device I left in one of the USB slots a random flashdrive I got at Percona Live reflashed with SysRescueCD — it was one of those because I had gotten a couple while over there, and I forgot to bring one from home. Since SysRescueCD uses VFAT by default, and the BIOS shows the USB devices transparently to DOS, it was visible as C:.

Here is the shortcut from above: I should have just used a random FreeDOS boot floppy image, and just copied the BIOS onto that particular thumbdrive, back before I started using Java. It would have been much simpler. But I forgot about that thumbdrive and I was hoping I would be able to start bootdisk. Now I know better.

Of course that meant I had to finish booting the system in Linux proper, mount the USB flashdrive, scp over the BIOS update, and reboot. It took much longer than I wanted. But still much less than all the time trying to get FreeDOS into booting the right thing.

By the way, even if I were to find a way to access the CD with the bios updater, it wouldn’t have helped. I did not notice that, but the BIOS updater script FLASH.BAT wants to be smart, and it renames AFUDOS.SMC to AFUDOS.EXE, runs it, then renames it back. Which means that it’s not usable from read-only media. Oh yeah and it requires a parameter, but if you don’t provide it, it’ll tell you that the command is unknown (because it still passes a bunch of flags to the flash utility), and tells you to reboot.

Now this is only the problem with getting the BIOS updated. What happened after would probably deserve a separate post, but I’ll try to be short and write that down here. After update, of course I got the usual “CMOS checksum error, blah blah, Press F1 to enter Setup.”. So I went and reconfigured the few parameters that I wanted different from the defaults, for instance put the SATA controller into AHCI mode (IDE compatibility, seriously?) and disabled the IDE controller itself, plus enabled SVM which on the defaults is still disabled.

The system booted fine, but no network cards were to be found. Not a driver problem, they would not appear on the PCI bus. To be precise, now that I compared this with the actual running setup, the two PLX PCI-E switches were not to be found on the bus, the network cards just happen to be connected to them. I tried playing with all the configuration options for the network cards, for PCI, for PnP and ACPI. Nope. Tried a full remote power cycle, nope. I was already considering the costs of adding a riser card and a multiport card to the server (wouldn’t have helped, as I noted the PLX were gone), when Janne suggested doing a reset to defaults anyway. It worked. The settings were basically the same I left before (looks like “optimal defaults” do include SVM), just changed SATA back to AHCI (yes, even in optimal settings, it defaults to IDE compatibility).

It worked. But then only three out of five drives were recognized (one of the SSDs was disconnected when I was there in April and I hadn’t had time to go back there in June to put it back). One more reboot into BIOS to disable the IDE controller altogether and the rest of the HDDs appeared, together with their LVM group.

Yes, I know, SuperMicro suggests you to only do BIOS updates when they require you to. On the other hand, I was hoping for an SVM/IOMMU fix which is not there still, as KVM made it complain, so I wanted to try. I was used to Tyan, that Yamato still uses, that basically told you “don’t bother us with support requests unless you updated your BIOS” — a policy that I find much more pleasant, especially given other experience with other hardware (such as Intel’s NUC requiring a BIOS update before it can use Intel’s WiFi cards….).

Why my Munin plugins are now written in Perl

This post is an interlude between Gentoo-related posts. The reason is that I have one in drafts that requires me to produce some results that I have not yet, so it’ll have to wait for the weekend or so.

You might remember that my original IPMI plugin was written in POSIX sh and awk, rather than bash and gawk as the original one. Since then, the new plugin (that as it turns out might become part of the 2.1 series but not to replace both the old ones, since RHEL and Fedora don’t package a new enough version of Freeipmi) has been rewritten in Perl, so using neither sh nor awk. Similarly, I’ve written a new plugin for sensors which I also wrote in Perl (although in this case the original one also used it).

So why did I learn a new language (since I never programmed in Perl before six months ago) just to get these plugins running? Well, as I said in the other post, the problem was calling the same command so many times, which is why I wanted to go multigraph — but when dealing with variables, sticking to POSIX sh is a huge headache. One of the common ways to handle this is to save to a temporary directory the output of a command and parse that multiple times, but that’s quite a pain, as it might require I/O to disk, and it also means that you have to execute more and more commands. Doing the processing in Perl means that you can save things in variables, or even just parse it once and split it into multiple objects, to be later used for output, which is what I’ve been doing for parsing FreeIPMI’s output.

But why Perl? Well, Munin itself is written in Perl, so while my usual language of choice is Ruby, the plugins are much more usable if doing it in Perl. Yes, there are some alternative nodes written in C and shell, but in general it’s a safe bet that these plugins will be executed on a system that at least supports Perl — the only system I can think of that wouldn’t be able to do so would be OpenWRT, but that’s a whole different story.

There are a number of plugins written in Python and Ruby, some in the official package, but most in the contrib repository and they could use some rewriting. Especially those that use net-snmp or other SNMP libraries, instead of Munin’s Net::SNMP wrapper.

But while the language is of slight concern, some of the plugins could use some rewriting simply to improve their behaviour. As I’ve said, using multigraphs it’s possible to reduce the number of times that the plugin is executed, and thus the number of calls to the backend, whatever that is (a program, or access to /sys), so in many cases plugins that support multiple “modes” or targets through wildcarding can be improved by making them a single plugin. In some cases, it’s even possible to reduce multiple plugins into one, as I did to the various apache_* plugins shipping with Munin itself, replaced on my system with apache_status as provided by the contrib repository, that fetches the server status page only once and then parses it to produce the three graphs that were, before that, created by three different plugins with three different fetches.

Another important trick up our sleeves while working on Munin plugins is dirty config which basically means that (under indication from the node itself), you can make the plugin output the values as well as the configuration itself during the config execution — this saves you one full trip to the node (to fetch the data), and usually that also means it saves you from having to send one more call to the backend. In particular with these changes my IPMI plugin went from requiring six calls to ipmi-sensors per update, for the three graphs, to just one. And since it’s either IPMI on the local bus (which might require some time to access) or over LAN (which takes more time), the difference is definitely visible both in timing, and in traffic — in particular one of the servers at my day job is monitoring another seven servers (which can’t be monitored through the plugin locally), which means that we went from 42 to 7 calls per update cycle.

So if you use Munin, and either have had timeout issues in the past or recently, or you have some time at hand to improve some plugins, you might want to follow what I’ve been doing, and start improving or re-writing plugins to support multigraph or dirtyconfig, and thus improve its performance.

Revenge of the HP Updates

Just shy of three months ago I was fighting with updating the iLO firmware (IPMI and extras) and as I recounted, even when you select downloads for RHEL (which is a supported operating system on those boxes), you’re given a Windows executable file, which you have to extract. But at least, you can use the file you extract, to update the IPMI firmware remotely.

Well, if it wasn’t for a small little issue that the fans are going to get stuck at 14 KRPM until you also update the BIOS. It wasn’t obvious how much of a problem that is until we got to the co-location last week and… “What on Earth is this noise?” “I think it’s our servers!” screamed on the backside of the cabinet.

Since one of the servers also had some other hardware issues (one of the loops that keep the chipset’s heatsink gave way — I glued it back and applied a new layer of thermal paste after scraping the old one), we ended up bringing it back to the office, where today, after repairing it and booting, it became obvious that we couldn’t leave it running at any time with that kind of noise. So it was time to update the BIOS. Which is easier said than done.

Step one is finding the correct download — the first one I found turned out to be wrong, but it took me some time to understand that, because BIOS update has to be done with DOS. And that brought me back to a very old post of mine (well, not that old, it’s just an year and a half ago, now that I see), and its follow-up which came with a downloadable 383KB compressed, 2GB uncompressed bootable FreeDOS image. Since getting sysrescuecd’s FreeDOS to do anything other than booting and playing its own demos was impossible.

So when I actually get to run the executable in the FreeDOS image … what I come to is an extremely stupid tool (that, I remember you, will not work on Windows XP, Vista or 7) to create an USB drive to update the BIOS… lol, whut?

The correct download is, once again, for Windows even when you select RHEL4, and it auto-extracts in a multitude of files that include the BIOS itself some four different times, and would provide some sort of network update, as well as “flat files” (which you can use with FreeDOS), a Windows updater, an ISO file, and an utility to build an USB stick to update the BIOS itself.

If you count the fact that this is for a server running Linux, you now just involved two more operating systems. And the next trip to the co-lo we’ve got work cut out for us, updating server by server the BIOS, and the IPMI firmware (hoping that the new firmware actually have a reliable SOL connection, among others).

But to avoid being all too negative with HP, it’s still better than trying to do standard sysadmin work on an Apple OS X Server install on a Mac Mini. OS X Server combines UNIX’s friendliness, with Windows’s remote management capabilities, Solaris’s hardware support, and AIX software availability. But that’s a topic for another post.

Munin, sensors and IPMI

In my previous post about Munin I said that I was still working on making sure that the async support would reach Gentoo in a way that actually worked. Now with version 2.0.7-r5 this is vastly possible, and it’s documented on the Wiki for you all to use.

Unfortunately, while testing it, I found out that one of the boxes I’m monitoring, the office’s firewall, was going crazy if I used the async spooled node, reporting fan speeds way too low (87 RPMs) or way too high (300K), and with similar effects on the temperatures as well. This also seems to have caused the fans to go out of control and run constantly at their 4KRPM instead of their usual 2KRPM. The kernel log showed that there was something going wrong with the i2c access, which is what the sensors program uses.

I started looking into the sensors_ plugin that comes with Munin, which I knew already a bit as I fixed it to match some of my systems before… and the problem is that for each box I was monitoring, it would have to execute sensors six times: twice for each graph (fan speed, temperature, voltages), one for config and one for fetching the data. And since there is no way to tell it to just fetch some of the data instead of all of it, it meant many transactions had to go over the i2c bus, all at the same time (when using munin async, the plugins are fetched in parallel). Understanding that the situation is next to unsolvable with that original code, and having one day “half off” at work, I decided to write a new plugin.

This time, instead of using the sensors program, I decided to just access /sys directly. This is quite faster and allows to pinpoint what data you need to fetch. In particular during the config step, there is no reason to fetch the actual value, which saves many i2c transactions even just there. While at it, I also made it a multigraph plugin, instead of the old wildcard one, so that you only need to call it once, and it’ll prepare, serially, all the available graphs: in addition to those that were supported before, which included power – as it’s exposed by the CPUs on Excelsior – I added a few that I haven’t been able to try but are documented by the hwmon sysfs interface, namely current and humidity.

The new plugin is available on the contrib repository – which I haven’t found a decent way to package yet – as sensors/hwmon and is still written in Perl. It’s definitely faster, has fewer dependencies and it’s definitely more reliable at leas ton my firewall. Unfortunately, there is one feature that is missing: sensors would sometimes report an explicit label for temperature data.. but that’s entirely handled in userland. Since we’re reading the data straight from the kernel, most of those labels are lost. For drivers that do expose those labels, such as coretemp, they are used, though.

Also we lose the ability to ignore the values from the get-go, like I described before but you can’t always win. You’ll have to ignore the graph data from the master instead. Otherwise you might want to find a way to tell the kernel to not report that data. The same probably is true for the names, although unfortunately…

[temp*_label] Should only be created if the driver has hints about what this temperature channel is being used for, and user-space doesn’t. In all other cases, the label is provided by user-space.

But I wouldn’t be surprised if it was possible to change that a tinsy bit. Also, while it does forfeit some of the labeling that the sensors program do, I was able to make it nicer when anonymous data is present — it wasn’t so rare to have more than one temp1 value as it was the first temperature channel for each of the (multiple) controllers, such as the Super I/O, ACPI Thermal Zone, and video card. My plugin outputs the controller and the channel name, instead of just the channel name.

After I’ve completed and tested my hwmon plugin I moved on to re-rewrite the IPMI plugin. If you remember the saga I first rewrote the original ipmi_ wildcard plugin in freeipmi_, including support for the same wildcards as ipmisensor_, so that instead of using OpenIPMI (and gawk), it would use FreeIPMI (and awk). The reason was that FreeIPMI can cache SDR information automatically, whereas OpenIPMI does have support, but you have to tackle it manually. The new plugin was also designed to work for virtual nodes, akin to the various SNMP plugins, so that I could monitor some of the servers we have in production, where I can’t install Munin, or I can’t install FreeIPMI. I have replaced the original IPMI plugin, which I was never able to get working on any of my servers, with my version on Gentoo for Munin 2.0. I expect Munin 2.1 to come with the FreeIPMI-based plugin by default.

Unfortunately, like for the sensors_ plugin, my plugin was calling the command six times per host — although this allows you to filter for the type of sensors you want to receive data for. And that became even worse when you have to monitor foreign virtual nodes. How do I solve that? I decided to rewrite it to be multigraph as well… but shell script then was difficult to handle, which means that it’s now also written in Perl. The new freeipmi, non-wildcard, virtual node-capable plugin is available in the same repository and directory as hwmon. My network switch thanks me for that.

Of course unfortunately the async node still does not support multiple hosts, that’s something for later on. In the mean time though, it does spare me lots of grief and I’m happy I took the time working on these two plugins.

Updating HP iLO 2.x

As I wrote yesterday I’ve been doing system and network administration work here in LA as well, and I’ve set up Munin and Icinga to warn me when something required maintenance.

Now some of the first probes that Munin forwarded to Icinga we knew already about (in another post I wrote of how the CMOS battery ran out on two of the servers), but one was something that bothered me before as well: one of the boxes only has one CPU on board and it reports a value of 0 instead of N/A.

So I decided to look into updating the firmware of the DL140 G3 and see if it would help us at all; the original firmware on IPMI device was 2.10 while the latest one available is 2.21. Neither support firmware update via HTML. The firmware download, even when selecting the RedHat Enterprise Linux option is a Windows EXE file (not an auto-extract archive, which you can extract from Linux, but their usual full-fledged setup software to extract in C:SWSetup). When you extract it, you’re presented with instructions on how to build an USB key which you can then use to update the firmware via FreeDOS…

You can guess I wasn’t amused.

After searching around a bit more I found out that there is a way to update this over the network. It’s described in HP’s advanced iLO usage guide, and seems to work fine, but it also requires another step to be taken in Windows (or FreeDOS): you have to use the ROMPAQ.EXE utility to decompress the compressed firmware image.

*I wonder, why does HP provide you with two copies of the compressed firmware image, for a grand total of 3MB, instead of only one of the uncompressed one (2MB)? I suppose the origin of the compressed image is to be found in the 1.44MB floppy disk size limitation, but nowadays you don’t use floppies… oh well.*

After you have the uncompressed image, you have to set up a TFTP server.. which luckily I already had laying around from when I updated the firmware of the APC powerstrips discussed in one of the posts linked above. So I just added the IPMI firmware image, and moved on to the next step.

The next step consists of connecting via telnet to the box and issue two commands: cd map1/firmware1 followed by load -source //$serverip/$filename -oemhpfiletype csr … the file is downloaded via TFTP and the BMC rebooted. Afterwards you have to clear out the SDR cache of FreeIPMI as ipmi-sensors wouldn’t work otherwise.

This did fix the critical notification I was receiving .. to a point. First of all, the fan speed has still bogus thresholds (and I’m not sure if it’s a bug in FreeIPMI or one in the firmware at this point) as it reports the upper limits instead of the lower ones). Second of all the way it fixed the misreported CPU thermal sensor is by … not reporting any temperature off either thermal sensor! Now both CPU temperatures are gone and only ambient temperature is available. D’oh!

Another funky issue is that I’m still fighting to get Munin to tell Icinga that “everything’s okay” — the way Munin contacts send_nsca is connected to the limits so if there are no limits that are present, it seems like it simply doesn’t report anything at all. This is something else I have to fix this week.

Now back to doing the firmware updates on the remaining boxes…

Update: turns out HP updates are worse than the original firmware in some ways. Not only the CPU Thermal Diodes are no longer represented, but the voltages lost their thresholds altogether! The end result of which is that now it says that it’s all a-ok! Even if the 3V battery is reported at 0.04V!. Which basically means that I have to set my own limits on things, but at least it should work as intended afterwards.

Oh and the DL160 G6? First of all, this time the firmware update has a web interface… to tell it which file to request from which TFTP server. Too bad that all the firmware updates that I can run on my systems require the bootcode to be updated as well, which means we’ll have to schedule some maintenance time when I come back from VDDs.

Munin, HP servers and APC powerstrips

Yes, I know I start to get boring.

Today I spent at least half of my work day working on Munin plugins to monitor effectively some of the equipment we currently have at our co-location. This boils down to two metered APC ~~powerstrips~~ PDUs (let’s use their term, silly as it might sound). I think it’s worth to note the difference: APC provides switched and metered PDUs; the former should allow for having per-plug load data, and powering on and off of the single plug; the latter (what we have here) is much cheaper, does not allow you to turn them on and off, and simply give you a reading of the load per-phase. Given that our co-location has only single-phase power, we only get a reading per strip, which is still okay, it gives us more information than we had before at least.

Now, there are a few funny things with these strips: they have a network interface, which is cool, but they don’t use DHCP by default! You either have to set them up with the serial interface (which obviously is still very serial, not an USB adapter — and my laptop doesn’t have any serial port), or use a Windows software (which is actually written in Java and spend 98% of the install time copying an extra install of the JRE to the drive), or finally note down the MAC address when you install them, and then poisoning a system’s ARP table to “fake” an IP to the strip, sealing the deal by sending a 113 bytes ICMP packet to the strip via ping … no there is no use for a watermelon or a chimp, sorry Luca.

After finally completing the IP settings, I had to find my way to get the data out; the strips support either SNMPv1 or SNMPv3 — I discarded the former simply because it’s extremely insecure and I’d rather not even have that around, so I set up an user for munin. Next problem? snmpwalk did not report any useful data. The reason is actually quite simple: it doesn’t know which OIDs to probe for. Download the MIB data from APC and install it in the system, and it’s much happier.

Then I had to write a plugin for it. Which wasn’t too bad; the data is simple, too bad I couldn’t find a way to get, through SNMP, the high limit of current drain on the strip — it did report the configured (default) limits for near-overload and overload, which makes it very nice to set them up in Munin. Unfortunately only after writing the plugin I found out that the Munin contrib repository had already not one but two plugins trying to do the same. Neither is very good with it though: neither supported Munin’s SNMP framework, one had a very unclear licensing situation (which is unfortunately common on the contrib repository), and used sh and net-snmp’s command-line utilities to access the strip.

So after adding my plugin, and removing the two bad ones, I also looked into cleaning up the contrib tree a little bit. It’s far from perfect, there are still miscategorized plugins and duplicates, and others (such as one of the net-p2p/transmission monitors) which rely on external script files instead of being written in a single one. But at least I was able to remove and recategorize enough of them that it starts to make some sense. If you’re a Munin user and would like for Gentoo to provide more, better plugins, then please take your time to see which of the plugins currently in the contrib tree are trying to reimplement something and failing at it (lots of them I’m afraid will be, especially those related to APC UPSes), and get rid of them. There is also work to be done to bring even only the documentation of the plugins up to speed with the format used by Munin proper, and this is without talking about improving them to follow the right code style or anything.

I also spent some time improving my IPMI plugin (which you can find now on the contrib repository if you’re not a Gentoo user – if you’re a Gentoo user it takes the place of the original IPMI plugins shipped with Munin – after I removed all the others that were trying to do the same thing sometimes with twice as many lines of code than mine), and now it can monitor foreign hosts as well. How is this useful? Well, among other things it lets you monitor Windows boxes and other boxes where you either lack access or you can’t install any IPMI tool (I have a couple of systems that are running RHEL4 to monitor, okay?).

One interesting thing I learnt out of this experience is that it makes total sense to monitor voltages at least on HP servers. Beside the idea of monitoring for a PSU gone wrong, HP has one probe set to the CMOS battery, which is a 3V CR2032 Lithium Battery which will provide decreasing voltage, and thus will show in the list when it has to be replaced — unfortunately it also seems like their newest servers don’t have a probe there, which is bad (Excelsior has a VBAT which seems to be just the same thing).

This is all for today!

Munin again, sorry!

Okay this might start to be boring, but I’m still working on Munin, and that means you have to read (or not) another post on the topic.

So last time I was talking about Munin and I was intending to write about SNMP, but with one thing and another I ended up just writing about IPMI.

The only thing I want to put in clear now about SNMP is that I’m still working on it — the main issue seems to be that the Munin plugins have a default timeout of 10 seconds, but the multigraph plugin that should be used for mapping multiple SNMP interface takes about 24 seconds for the switch I’d like to monitor here at my workplace. This should technically be solvable by using the new async daemon support, but this requires, in turn, support for the SSH transport, which can’t be used without relaxing the security applied to munin user (by making it loggable). There is also another point I have to make: I intend to modify the plugin and make a second one that actually allows to graph all the interfaces on one single entry, so that it actually allows me for more interesting data.

For what concerns IPMI instead: the new version of Munin 2 in Gentoo replaces both the IPMI plugins (ipmi_ and ipmi_sensor_) with my version based off FreeIPMI, which now not only outperforms the original ones (thanks to FreeIPMI caching, which is enabled by default), but also outfeatures the original plugins! Here’s the gist of it:

by using FreeIPMI, the plugin is shorter, much shorter, as the output is malleable to script handling;
I’ve made the script accept both the names used by ipmi_ and ipmi_sensor_ — both reported the same data, but one used bash and gawk (the GNU version only), while the other used python;
thanks to Kent I was able to get enough data to make it report power and current as well as temperatures and fans; I still have to implement voltage that was not implemented in the previous plugins either;
thanks to Albert, the new 1.2 series of FreeIPMI has support for threshold output — which was the remaining missing feature; I’ve implemented it in the plugin and I’ve patched 1.1.6 (and 1.1.7) to support the option as well;
and since I cared … the new version uses POSIX-compatible sh syntax and POSIX awk syntax instead of the GNU variants of both.

My next development is going to be supporting what they call “foreign hosts” on the plugin, so you can actually get Munin to monitor an IPMI SBMC instead of the local BMC interface. This will probably come soonish in Gentoo together with the support for asyncd.

What remains now is finding a way to package the contributed plugins which needs to be available, especially since Steve said he doesn’t want new plugins in the main package, and everything else has to come through the new contrib repository.

And yes, I still have to fix HTML generations, Justin, I know. I’m trying to find what the heck is wrong with it.

Munin, SNMP and IPMI

You might wonder why I’m working so much on Munin, given that I should be worried about other things such as my own job… well, turns out that I’m using Munin at my job and my bosses are so impressed with actually having a monitor on the resource (while they originally sceptical we needed it), that I can easily cut down more time on the payroll to work on improvements.

The funny thing is that we’re actually developing proprietary software based on Free Software components (don’t worry, we respect the licenses!) but our main FLOSS contributions are probably outside the area of development for which our business is based on. I guess this is just the way it is, after all, the open source contributions of Facebook have little to do with social networking.

Anyway while trying to set up monitor for the servers and devices we care about, I was trying to solve the issue of knowing what the sensors reading are on the servers, which are mostly HP (with the exception of my Excelsior, which is monitored, yes, but on a different Munin anyway) and for which lm_sensors is useless: the data is not fed to the main system but rather to the IPMI management board.

Thankfully, there are a number of different tools that allow you to access that IPMI data, and Munin already had a plugin, ipmi_ that uses ipmitool to fetch the sensors’ data. The problem with it is that the fetching takes time, and if the plugin doesn’t reply in 10 seconds, Munin will consider it as not available. On the HP server I started setting this up, the reply time is well over 10 seconds.

To get around this limitation, I decided to take a different approach: I wrote my own plugin. Or rather I rewrote the original plugin using FreeIPMI which I know well (I maintain it in Gentoo and I sent patches upstream before). And this seems to be a win on all counts.

First of all, the ipmi-sensors command caches the so-called SDR data, which describe the sensors available on a system, which means that after the first execution, it doesn’t spend as much time parsing what it’s receiving. Then, since version 1 at least, it has a number of parameters that make it very easy to filter the output and receive it in a format that is suitable for script-based parsing. In particular you can filter what type of sensors you want data from (Temperature or Fan), ignore the values that are not available (e.g.: missing fans), all together with having the output in CSV form.

The net result is that instead of a page-long gawk script to parse the lines, filter them, generate unique names and so on, I’m using a couple of awk commands — yes I tested them with mawk, but they also use a very very simple syntax, so I’m not surprised it’ll work for a very long time. Basically now the plugin is very fast, very short, and very simple. And instead of just expecting to always have both temperature and fans data, this time it actually checks before suggesting anything at all.

Unfortunately, right now, it has two missing features that are present in the original plugin: the first is critical and warning thresholds that are not printed by the current version of FreeIPMI. This is okay, because likely the next release will have a switch to print those as well (which means that it’ll reach feature parity for those two sensors). The other issue is that the original plugin has an undocumented support for power metering.

Unfortunately none of my boards support power metering so I’m stuck without that kind of data, and I can’t be sure how to implement it back right now — the ticket refers that HP servers have the data, but none of the ones I have here report it. But they do report a few temperatures…

Munin graph of FreeIPMI-reported temperatures.

But with the exception of that, the plugin is much better than the one that was there before. Hopefully at some point it’ll be in the default Munin set instead of just being available for Gentoo users.