FreeIPMI

Why my Munin plugins are now written in Perl

This post is an interlude between Gentoo-related posts. The reason is that I have one in drafts that requires me to produce some results that I have not yet, so it’ll have to wait for the weekend or so.

You might remember that my original IPMI plugin was written in POSIX sh and awk, rather than bash and gawk as the original one. Since then, the new plugin (that as it turns out might become part of the 2.1 series but not to replace both the old ones, since RHEL and Fedora don’t package a new enough version of Freeipmi) has been rewritten in Perl, so using neither sh nor awk. Similarly, I’ve written a new plugin for sensors which I also wrote in Perl (although in this case the original one also used it).

So why did I learn a new language (since I never programmed in Perl before six months ago) just to get these plugins running? Well, as I said in the other post, the problem was calling the same command so many times, which is why I wanted to go multigraph — but when dealing with variables, sticking to POSIX sh is a huge headache. One of the common ways to handle this is to save to a temporary directory the output of a command and parse that multiple times, but that’s quite a pain, as it might require I/O to disk, and it also means that you have to execute more and more commands. Doing the processing in Perl means that you can save things in variables, or even just parse it once and split it into multiple objects, to be later used for output, which is what I’ve been doing for parsing FreeIPMI’s output.

But why Perl? Well, Munin itself is written in Perl, so while my usual language of choice is Ruby, the plugins are much more usable if doing it in Perl. Yes, there are some alternative nodes written in C and shell, but in general it’s a safe bet that these plugins will be executed on a system that at least supports Perl — the only system I can think of that wouldn’t be able to do so would be OpenWRT, but that’s a whole different story.

There are a number of plugins written in Python and Ruby, some in the official package, but most in the contrib repository and they could use some rewriting. Especially those that use net-snmp or other SNMP libraries, instead of Munin’s Net::SNMP wrapper.

But while the language is of slight concern, some of the plugins could use some rewriting simply to improve their behaviour. As I’ve said, using multigraphs it’s possible to reduce the number of times that the plugin is executed, and thus the number of calls to the backend, whatever that is (a program, or access to /sys), so in many cases plugins that support multiple “modes” or targets through wildcarding can be improved by making them a single plugin. In some cases, it’s even possible to reduce multiple plugins into one, as I did to the various apache_* plugins shipping with Munin itself, replaced on my system with apache_status as provided by the contrib repository, that fetches the server status page only once and then parses it to produce the three graphs that were, before that, created by three different plugins with three different fetches.

Another important trick up our sleeves while working on Munin plugins is dirty config which basically means that (under indication from the node itself), you can make the plugin output the values as well as the configuration itself during the config execution — this saves you one full trip to the node (to fetch the data), and usually that also means it saves you from having to send one more call to the backend. In particular with these changes my IPMI plugin went from requiring six calls to ipmi-sensors per update, for the three graphs, to just one. And since it’s either IPMI on the local bus (which might require some time to access) or over LAN (which takes more time), the difference is definitely visible both in timing, and in traffic — in particular one of the servers at my day job is monitoring another seven servers (which can’t be monitored through the plugin locally), which means that we went from 42 to 7 calls per update cycle.

So if you use Munin, and either have had timeout issues in the past or recently, or you have some time at hand to improve some plugins, you might want to follow what I’ve been doing, and start improving or re-writing plugins to support multigraph or dirtyconfig, and thus improve its performance.

2012-12-20 by Flameeyes English Technical 4

Munin, sensors and IPMI

In my previous post about Munin I said that I was still working on making sure that the async support would reach Gentoo in a way that actually worked. Now with version 2.0.7-r5 this is vastly possible, and it’s documented on the Wiki for you all to use.

Unfortunately, while testing it, I found out that one of the boxes I’m monitoring, the office’s firewall, was going crazy if I used the async spooled node, reporting fan speeds way too low (87 RPMs) or way too high (300K), and with similar effects on the temperatures as well. This also seems to have caused the fans to go out of control and run constantly at their 4KRPM instead of their usual 2KRPM. The kernel log showed that there was something going wrong with the i2c access, which is what the sensors program uses.

I started looking into the sensors_ plugin that comes with Munin, which I knew already a bit as I fixed it to match some of my systems before… and the problem is that for each box I was monitoring, it would have to execute sensors six times: twice for each graph (fan speed, temperature, voltages), one for config and one for fetching the data. And since there is no way to tell it to just fetch some of the data instead of all of it, it meant many transactions had to go over the i2c bus, all at the same time (when using munin async, the plugins are fetched in parallel). Understanding that the situation is next to unsolvable with that original code, and having one day “half off” at work, I decided to write a new plugin.

This time, instead of using the sensors program, I decided to just access /sys directly. This is quite faster and allows to pinpoint what data you need to fetch. In particular during the config step, there is no reason to fetch the actual value, which saves many i2c transactions even just there. While at it, I also made it a multigraph plugin, instead of the old wildcard one, so that you only need to call it once, and it’ll prepare, serially, all the available graphs: in addition to those that were supported before, which included power – as it’s exposed by the CPUs on Excelsior – I added a few that I haven’t been able to try but are documented by the hwmon sysfs interface, namely current and humidity.

The new plugin is available on the contrib repository – which I haven’t found a decent way to package yet – as sensors/hwmon and is still written in Perl. It’s definitely faster, has fewer dependencies and it’s definitely more reliable at leas ton my firewall. Unfortunately, there is one feature that is missing: sensors would sometimes report an explicit label for temperature data.. but that’s entirely handled in userland. Since we’re reading the data straight from the kernel, most of those labels are lost. For drivers that do expose those labels, such as coretemp, they are used, though.

Also we lose the ability to ignore the values from the get-go, like I described before but you can’t always win. You’ll have to ignore the graph data from the master instead. Otherwise you might want to find a way to tell the kernel to not report that data. The same probably is true for the names, although unfortunately…

[temp*_label] Should only be created if the driver has hints about what this temperature channel is being used for, and user-space doesn’t. In all other cases, the label is provided by user-space.

But I wouldn’t be surprised if it was possible to change that a tinsy bit. Also, while it does forfeit some of the labeling that the sensors program do, I was able to make it nicer when anonymous data is present — it wasn’t so rare to have more than one temp1 value as it was the first temperature channel for each of the (multiple) controllers, such as the Super I/O, ACPI Thermal Zone, and video card. My plugin outputs the controller and the channel name, instead of just the channel name.

After I’ve completed and tested my hwmon plugin I moved on to re-rewrite the IPMI plugin. If you remember the saga I first rewrote the original ipmi_ wildcard plugin in freeipmi_, including support for the same wildcards as ipmisensor_, so that instead of using OpenIPMI (and gawk), it would use FreeIPMI (and awk). The reason was that FreeIPMI can cache SDR information automatically, whereas OpenIPMI does have support, but you have to tackle it manually. The new plugin was also designed to work for virtual nodes, akin to the various SNMP plugins, so that I could monitor some of the servers we have in production, where I can’t install Munin, or I can’t install FreeIPMI. I have replaced the original IPMI plugin, which I was never able to get working on any of my servers, with my version on Gentoo for Munin 2.0. I expect Munin 2.1 to come with the FreeIPMI-based plugin by default.

Unfortunately, like for the sensors_ plugin, my plugin was calling the command six times per host — although this allows you to filter for the type of sensors you want to receive data for. And that became even worse when you have to monitor foreign virtual nodes. How do I solve that? I decided to rewrite it to be multigraph as well… but shell script then was difficult to handle, which means that it’s now also written in Perl. The new freeipmi, non-wildcard, virtual node-capable plugin is available in the same repository and directory as hwmon. My network switch thanks me for that.

Of course unfortunately the async node still does not support multiple hosts, that’s something for later on. In the mean time though, it does spare me lots of grief and I’m happy I took the time working on these two plugins.

2012-10-25 by Flameeyes English Technical 1

Updating HP iLO 2.x

As I wrote yesterday I’ve been doing system and network administration work here in LA as well, and I’ve set up Munin and Icinga to warn me when something required maintenance.

Now some of the first probes that Munin forwarded to Icinga we knew already about (in another post I wrote of how the CMOS battery ran out on two of the servers), but one was something that bothered me before as well: one of the boxes only has one CPU on board and it reports a value of 0 instead of N/A.

So I decided to look into updating the firmware of the DL140 G3 and see if it would help us at all; the original firmware on IPMI device was 2.10 while the latest one available is 2.21. Neither support firmware update via HTML. The firmware download, even when selecting the RedHat Enterprise Linux option is a Windows EXE file (not an auto-extract archive, which you can extract from Linux, but their usual full-fledged setup software to extract in C:SWSetup). When you extract it, you’re presented with instructions on how to build an USB key which you can then use to update the firmware via FreeDOS…

You can guess I wasn’t amused.

After searching around a bit more I found out that there is a way to update this over the network. It’s described in HP’s advanced iLO usage guide, and seems to work fine, but it also requires another step to be taken in Windows (or FreeDOS): you have to use the ROMPAQ.EXE utility to decompress the compressed firmware image.

*I wonder, why does HP provide you with two copies of the compressed firmware image, for a grand total of 3MB, instead of only one of the uncompressed one (2MB)? I suppose the origin of the compressed image is to be found in the 1.44MB floppy disk size limitation, but nowadays you don’t use floppies… oh well.*

After you have the uncompressed image, you have to set up a TFTP server.. which luckily I already had laying around from when I updated the firmware of the APC powerstrips discussed in one of the posts linked above. So I just added the IPMI firmware image, and moved on to the next step.

The next step consists of connecting via telnet to the box and issue two commands: cd map1/firmware1 followed by load -source //$serverip/$filename -oemhpfiletype csr … the file is downloaded via TFTP and the BMC rebooted. Afterwards you have to clear out the SDR cache of FreeIPMI as ipmi-sensors wouldn’t work otherwise.

This did fix the critical notification I was receiving .. to a point. First of all, the fan speed has still bogus thresholds (and I’m not sure if it’s a bug in FreeIPMI or one in the firmware at this point) as it reports the upper limits instead of the lower ones). Second of all the way it fixed the misreported CPU thermal sensor is by … not reporting any temperature off either thermal sensor! Now both CPU temperatures are gone and only ambient temperature is available. D’oh!

Another funky issue is that I’m still fighting to get Munin to tell Icinga that “everything’s okay” — the way Munin contacts send_nsca is connected to the limits so if there are no limits that are present, it seems like it simply doesn’t report anything at all. This is something else I have to fix this week.

Now back to doing the firmware updates on the remaining boxes…

Update: turns out HP updates are worse than the original firmware in some ways. Not only the CPU Thermal Diodes are no longer represented, but the voltages lost their thresholds altogether! The end result of which is that now it says that it’s all a-ok! Even if the 3V battery is reported at 0.04V!. Which basically means that I have to set my own limits on things, but at least it should work as intended afterwards.

Oh and the DL160 G6? First of all, this time the firmware update has a web interface… to tell it which file to request from which TFTP server. Too bad that all the firmware updates that I can run on my systems require the bootcode to be updated as well, which means we’ll have to schedule some maintenance time when I come back from VDDs.

2012-08-22 by Flameeyes English Technical 2

Munin again, sorry!

Okay this might start to be boring, but I’m still working on Munin, and that means you have to read (or not) another post on the topic.

So last time I was talking about Munin and I was intending to write about SNMP, but with one thing and another I ended up just writing about IPMI.

The only thing I want to put in clear now about SNMP is that I’m still working on it — the main issue seems to be that the Munin plugins have a default timeout of 10 seconds, but the multigraph plugin that should be used for mapping multiple SNMP interface takes about 24 seconds for the switch I’d like to monitor here at my workplace. This should technically be solvable by using the new async daemon support, but this requires, in turn, support for the SSH transport, which can’t be used without relaxing the security applied to munin user (by making it loggable). There is also another point I have to make: I intend to modify the plugin and make a second one that actually allows to graph all the interfaces on one single entry, so that it actually allows me for more interesting data.

For what concerns IPMI instead: the new version of Munin 2 in Gentoo replaces both the IPMI plugins (ipmi_ and ipmi_sensor_) with my version based off FreeIPMI, which now not only outperforms the original ones (thanks to FreeIPMI caching, which is enabled by default), but also outfeatures the original plugins! Here’s the gist of it:

by using FreeIPMI, the plugin is shorter, much shorter, as the output is malleable to script handling;
I’ve made the script accept both the names used by ipmi_ and ipmi_sensor_ — both reported the same data, but one used bash and gawk (the GNU version only), while the other used python;
thanks to Kent I was able to get enough data to make it report power and current as well as temperatures and fans; I still have to implement voltage that was not implemented in the previous plugins either;
thanks to Albert, the new 1.2 series of FreeIPMI has support for threshold output — which was the remaining missing feature; I’ve implemented it in the plugin and I’ve patched 1.1.6 (and 1.1.7) to support the option as well;
and since I cared … the new version uses POSIX-compatible sh syntax and POSIX awk syntax instead of the GNU variants of both.

My next development is going to be supporting what they call “foreign hosts” on the plugin, so you can actually get Munin to monitor an IPMI SBMC instead of the local BMC interface. This will probably come soonish in Gentoo together with the support for asyncd.

What remains now is finding a way to package the contributed plugins which needs to be available, especially since Steve said he doesn’t want new plugins in the main package, and everything else has to come through the new contrib repository.

And yes, I still have to fix HTML generations, Justin, I know. I’m trying to find what the heck is wrong with it.

2012-08-04 by Flameeyes English Technical 0

Munin, SNMP and IPMI

You might wonder why I’m working so much on Munin, given that I should be worried about other things such as my own job… well, turns out that I’m using Munin at my job and my bosses are so impressed with actually having a monitor on the resource (while they originally sceptical we needed it), that I can easily cut down more time on the payroll to work on improvements.

The funny thing is that we’re actually developing proprietary software based on Free Software components (don’t worry, we respect the licenses!) but our main FLOSS contributions are probably outside the area of development for which our business is based on. I guess this is just the way it is, after all, the open source contributions of Facebook have little to do with social networking.

Anyway while trying to set up monitor for the servers and devices we care about, I was trying to solve the issue of knowing what the sensors reading are on the servers, which are mostly HP (with the exception of my Excelsior, which is monitored, yes, but on a different Munin anyway) and for which lm_sensors is useless: the data is not fed to the main system but rather to the IPMI management board.

Thankfully, there are a number of different tools that allow you to access that IPMI data, and Munin already had a plugin, ipmi_ that uses ipmitool to fetch the sensors’ data. The problem with it is that the fetching takes time, and if the plugin doesn’t reply in 10 seconds, Munin will consider it as not available. On the HP server I started setting this up, the reply time is well over 10 seconds.

To get around this limitation, I decided to take a different approach: I wrote my own plugin. Or rather I rewrote the original plugin using FreeIPMI which I know well (I maintain it in Gentoo and I sent patches upstream before). And this seems to be a win on all counts.

First of all, the ipmi-sensors command caches the so-called SDR data, which describe the sensors available on a system, which means that after the first execution, it doesn’t spend as much time parsing what it’s receiving. Then, since version 1 at least, it has a number of parameters that make it very easy to filter the output and receive it in a format that is suitable for script-based parsing. In particular you can filter what type of sensors you want data from (Temperature or Fan), ignore the values that are not available (e.g.: missing fans), all together with having the output in CSV form.

The net result is that instead of a page-long gawk script to parse the lines, filter them, generate unique names and so on, I’m using a couple of awk commands — yes I tested them with mawk, but they also use a very very simple syntax, so I’m not surprised it’ll work for a very long time. Basically now the plugin is very fast, very short, and very simple. And instead of just expecting to always have both temperature and fans data, this time it actually checks before suggesting anything at all.

Unfortunately, right now, it has two missing features that are present in the original plugin: the first is critical and warning thresholds that are not printed by the current version of FreeIPMI. This is okay, because likely the next release will have a switch to print those as well (which means that it’ll reach feature parity for those two sensors). The other issue is that the original plugin has an undocumented support for power metering.

Unfortunately none of my boards support power metering so I’m stuck without that kind of data, and I can’t be sure how to implement it back right now — the ticket refers that HP servers have the data, but none of the ones I have here report it. But they do report a few temperatures…

But with the exception of that, the plugin is much better than the one that was there before. Hopefully at some point it’ll be in the default Munin set instead of just being available for Gentoo users.

2012-07-27 by Flameeyes English Technical 3