Remotely upgrading BIOS on a SuperMicro servers

So it’s that time of the year again when you have to run a few BIOS updates; and you know I love updating BIOS. In this case the victim is the very Excelsior that I bought to run the Gentoo tinderbox, and which I’m hosting to my expenses at Hurricane Electric in Fremont. This is important to say, because it means I have no direct access to the hardware, and nobody else can help me there.

First, I decided to get rid of the low-hanging fruit: the IPMI sideboard. I couldn’t find release notes for the update, but it went from version 2.05 to 3.16, and it was released at the end of June 2014, so it’s pretty recent, and one would expect quite a bit of bug fixes, given that I had trouble running its iKVM client two years ago already, and I tried to get more information about its protocol without success.

For this part, SuperMicro is strangely helpful: the downloaded archive (once I was able to actually download it, it took me a few tries, the connection would break midway through the download) is three times the size of the actual firmware, because it comes with Windows, DOS and, helpfully, Linux update utilities in the same file. Which would be endearing if it wasn’t that the Linux utilities require the tools to access the BMC raw, which is not fun to do with grsec present, and if I’m not mistaken it also conflicts with the in-kernel IPMI drivers. Also, the IPMI remote interface can handle the firmware update “quite well” by itself.

I say “quite well” because it seems like ATEN (the IPMI vendor for SuperMicro and another long list of server manufacturers) still has not mastered the ability to keep configuration consistent across firmware upgrades. You know, the kind of things that most of our home routers or cable TV boxes manage nearly all the time. And all my home computers’ BIOSes. So instead it requires you to reset the full configuration when you upgrade the firmware of the device. Which is fine because you can use bmc-config from userspace to dump the configuration (minus the passwords) and reset it back. If the configuration it downloads is actually accepted as valid, which in my case it wasn’t. But I only needed to set an admin password and IP address and routing, so…

The second step is to update the BIOS of the server itself. And here is where the fun part starts. You have two options to download: a Windows executable or a ZIP archive. I downloaded both. The Windows executable is supposed to build a floppy to upgrade the BIOS. Not a floppy image, a floppy. It does indeed not work in Wine, but I tried! I settled for the ZIP file.

So the iKVM I just upgraded supports “virtual media” — which means you can give it files on your computer that it’ll use instead of floppy disks or CD-Rom. This is very neat, but it comes with some downsides. At first I wanted to give it a floppy disk image via HTTP, and use the Serial-over-SSH to enter the boot menu and do stuff. This has worked like a charm for me before, but it comes with limitations: the HTTP-based virtual media configuration for this iKVM only supports 1.44MB images for floppy disks, and uses SMB to access shared ISO files for the CD-Rom. It makes sense, you can’t expect the iKVM board to have enough RAM to store a whole 650MB image for the CD-Rom, now, can you?

The answer is of course to build a BIOS upgrade floppy disk, which is after all what their own Windows tool would have done (albeit on a real disk rather than an image — do people really still use floppy disks on Opteron servers?) Too bad that there is a small problem with that plan: the BIOS itself is over 2MB uncompressed. It does not fit in a floppy disk unformatted, let alone in the formatted disk, with the update utility, the kernel and the command interpreter. So no way to do that easily. The astutes of you after reading the blog post will probably figure out there was a shortcut here, but I’ll comment on that later.

So I decided to go back to the iKVM client, the Java WebStart one. Now luckily the new release of it works fine with IcedTea, unlike the previous one. Unfortunately it still is not working out of the box. Indeed it was only by chance that I found someone else who faced and solved that problem by adding a couple of lines to the XML file that defines the JavaWS environment.

The configuration dialog for the client allows providing a local ISO file as well as configuring the address of a SMB share with it: it implements its own transmission protocol, and the floppy option is now “floppy/USB”, so I thought maybe I could just give it my 1GB FreeDOS-based BIOS disk … nope, no luck. It still is limited to 1.44MB it seems.

So I had to get a working ISO and do the upgrade. It should be pretty straightforward, right? I even found a blog post showing how to do so, based on Gentoo, as it says to emerge isomaster! Great! Unfortunately, FreeDOS download page now provides you with a great way to download … installers. They no longer provide links to download live CDs. It makes total sense, right? Who doesn’t think every day “I should stop procrastinating and finally install FreeDOS on my laptop!”?

After some fiddling around I found a tiny non-descript directory in their FTP mirrors that includes fdfull.iso which is the actual LiveCD. So I download that, copy the BIOS and the updater into the CD and set it in the virtual media settings for the iKVM, and then boot. The CD does not boot unattended. If you leave it unattended it’ll ask you what you want to do and default to boot from the HDD. If you do remember to type 1 and press enter at the SysLinux prompt, it’ll then default to install (again, why did I bother installing Gentoo on the NUC when I could have used FreeDOS like the cool kids?).

Instead, you can choose to boot FreeDOS from the LiveCD with no support and no drivers, with just the CD drivers, with the CD drivers and HIMEM, with the CD drivers, HIMEM and EMM386. I decided I wanted to play it safe since it’s a BIOS update, and booted with just the CD drivers (I needed the BIOS files after all). Turns out that while this is an option at the boot menu, it will never work, as the CDROM driver needs to be loaded high, MSCDEX.EXE style. So reboot once again with HIMEM (and EMM386 because rebooting into that CD is painful — among other things, the iKVM client loses focus every time the framebuffer size changes. And it changes. all. the freaking. time. during boot up), and try to access the CD… unfortunately the driver does not support USB CD-Rom (like the one the iKVM provides me with) but just IDE ones, so no access to the ISO for me. Nope.

But I’m not totally unprepared for this kind of situations. As a failsafe device I left in one of the USB slots a random flashdrive I got at Percona Live reflashed with SysRescueCD — it was one of those because I had gotten a couple while over there, and I forgot to bring one from home. Since SysRescueCD uses VFAT by default, and the BIOS shows the USB devices transparently to DOS, it was visible as C:.

Here is the shortcut from above: I should have just used a random FreeDOS boot floppy image, and just copied the BIOS onto that particular thumbdrive, back before I started using Java. It would have been much simpler. But I forgot about that thumbdrive and I was hoping I would be able to start bootdisk. Now I know better.

Of course that meant I had to finish booting the system in Linux proper, mount the USB flashdrive, scp over the BIOS update, and reboot. It took much longer than I wanted. But still much less than all the time trying to get FreeDOS into booting the right thing.

By the way, even if I were to find a way to access the CD with the bios updater, it wouldn’t have helped. I did not notice that, but the BIOS updater script FLASH.BAT wants to be smart, and it renames AFUDOS.SMC to AFUDOS.EXE, runs it, then renames it back. Which means that it’s not usable from read-only media. Oh yeah and it requires a parameter, but if you don’t provide it, it’ll tell you that the command is unknown (because it still passes a bunch of flags to the flash utility), and tells you to reboot.

Now this is only the problem with getting the BIOS updated. What happened after would probably deserve a separate post, but I’ll try to be short and write that down here. After update, of course I got the usual “CMOS checksum error, blah blah, Press F1 to enter Setup.”. So I went and reconfigured the few parameters that I wanted different from the defaults, for instance put the SATA controller into AHCI mode (IDE compatibility, seriously?) and disabled the IDE controller itself, plus enabled SVM which on the defaults is still disabled.

The system booted fine, but no network cards were to be found. Not a driver problem, they would not appear on the PCI bus. To be precise, now that I compared this with the actual running setup, the two PLX PCI-E switches were not to be found on the bus, the network cards just happen to be connected to them. I tried playing with all the configuration options for the network cards, for PCI, for PnP and ACPI. Nope. Tried a full remote power cycle, nope. I was already considering the costs of adding a riser card and a multiport card to the server (wouldn’t have helped, as I noted the PLX were gone), when Janne suggested doing a reset to defaults anyway. It worked. The settings were basically the same I left before (looks like “optimal defaults” do include SVM), just changed SATA back to AHCI (yes, even in optimal settings, it defaults to IDE compatibility).

It worked. But then only three out of five drives were recognized (one of the SSDs was disconnected when I was there in April and I hadn’t had time to go back there in June to put it back). One more reboot into BIOS to disable the IDE controller altogether and the rest of the HDDs appeared, together with their LVM group.

Yes, I know, SuperMicro suggests you to only do BIOS updates when they require you to. On the other hand, I was hoping for an SVM/IOMMU fix which is not there still, as KVM made it complain, so I wanted to try. I was used to Tyan, that Yamato still uses, that basically told you “don’t bother us with support requests unless you updated your BIOS” — a policy that I find much more pleasant, especially given other experience with other hardware (such as Intel’s NUC requiring a BIOS update before it can use Intel’s WiFi cards….).

Updating firmware on a Crucial M4 drive, with Linux, no luck

When you think of SSD manufacturers, it might be obvious to think of them as Linux friendly, given they target power users, and Linux users are for the most part are power users. Seems like this is not that true for Crucial. My main personal laptop has, since last year, a 64GB Crucial M4 SSD – given I’ve not been using a desktop computer for a while it does start to feel small, but that’s a different point – which underwent a couple of firmware update since then. In particular, there is a new release 070H that is supposed to fix some nasty power saving issues.

Crucial only provide firmware update utilities for Windows 7 and 8 (two different versions of them), and then they have an utility for “Windows and Mac” — the latter is actually a ZIP file that contains an ISO file… well, I don’t have a CD to burn with me, so my first option was to run the Windows 7 file from my Windows 7 install, which resides on the external SATA harddrive. No luck with that, from what I read on the forums what the upgrader does is simply setting up an EFI application to boot, and then reboot. Unfortunately in my case there are two EFI partitions, because of the two bootable drives, and that most likely messes up with the upgrader.

Okay strike one, let’s look at the ISO file. The ISO is very simple and very small.. it basically is just an ISOLINUX tree that uses memdisk to launch a 2.88MB image file (2.88MB is a semi-standard floppy disk size, which never really went popular for real disks, but has been leveraged by most virtual floppy disk images in bootable CD-Roms for the expanded size). Okay what’s in the image file then? Nothing surprising, it’s a FreeDOS image, with Crucial’s own utility and the firmware.

So if you remember, I had some experience with trying to update BIOS through FreeDOS images, and I have my trusty USB stick with FreeDOS arriving on Tuesday with most of my personal effects that were in Italy waiting for me to find a final place to move my stuff on. But I wanted to see if I could try to boot the image file without needing the FreeDOS stick, so I checked. Grub2 does not include a direct way to load image — the reference to memdisk in their manual refers to the situation where you’re loading a standalone or rescue Grub image, nothing to do with what we care about.

The most obvious way to run the image is through SYSLINUX’s memdisk loader. Debian even has a package that allows you to just drop the images in /boot/images and adds them to the grub menu. Quick and easy, no? Well, no. The problem is that memdisk needs to be loaded with linux16 — and, well, Grub 2 does not support it if you’re booting via EFI, like I am.

I guess I’ll wait until Tuesday night, when my BIOS disk will be here, and I’ll just use it to update the SSD. It should solve the issue once and for all.

Revenge of the HP Updates

Just shy of three months ago I was fighting with updating the iLO firmware (IPMI and extras) and as I recounted, even when you select downloads for RHEL (which is a supported operating system on those boxes), you’re given a Windows executable file, which you have to extract. But at least, you can use the file you extract, to update the IPMI firmware remotely.

Well, if it wasn’t for a small little issue that the fans are going to get stuck at 14 KRPM until you also update the BIOS. It wasn’t obvious how much of a problem that is until we got to the co-location last week and… “What on Earth is this noise?” “I think it’s our servers!” screamed on the backside of the cabinet.

Since one of the servers also had some other hardware issues (one of the loops that keep the chipset’s heatsink gave way — I glued it back and applied a new layer of thermal paste after scraping the old one), we ended up bringing it back to the office, where today, after repairing it and booting, it became obvious that we couldn’t leave it running at any time with that kind of noise. So it was time to update the BIOS. Which is easier said than done.

Step one is finding the correct download — the first one I found turned out to be wrong, but it took me some time to understand that, because BIOS update has to be done with DOS. And that brought me back to a very old post of mine (well, not that old, it’s just an year and a half ago, now that I see), and its follow-up which came with a downloadable 383KB compressed, 2GB uncompressed bootable FreeDOS image. Since getting sysrescuecd’s FreeDOS to do anything other than booting and playing its own demos was impossible.

So when I actually get to run the executable in the FreeDOS image … what I come to is an extremely stupid tool (that, I remember you, will not work on Windows XP, Vista or 7) to create an USB drive to update the BIOS… lol, whut?

The correct download is, once again, for Windows even when you select RHEL4, and it auto-extracts in a multitude of files that include the BIOS itself some four different times, and would provide some sort of network update, as well as “flat files” (which you can use with FreeDOS), a Windows updater, an ISO file, and an utility to build an USB stick to update the BIOS itself.

If you count the fact that this is for a server running Linux, you now just involved two more operating systems. And the next trip to the co-lo we’ve got work cut out for us, updating server by server the BIOS, and the IPMI firmware (hoping that the new firmware actually have a reliable SOL connection, among others).

But to avoid being all too negative with HP, it’s still better than trying to do standard sysadmin work on an Apple OS X Server install on a Mac Mini. OS X Server combines UNIX’s friendliness, with Windows’s remote management capabilities, Solaris’s hardware support, and AIX software availability. But that’s a topic for another post.

On filesystem wiping

I have been talking about BIOS with “facebook friends” today, and Davide (from KDE) asked me if I could send him a copy of my FreeDOS stick, or at least the instructions on how to do so which obviously were on this very blog.

At first I wanted to just send him a compressed copy of the stick itself; then I remembered that I’m not that stupid and I checked out the content of the actual image. The problem is that even if I remove the files as they appear on the filesystem, FAT does not clear them out until the space is needed again; given that my USB stick has been used on almost every imaginable operating system (Windows XP, Vista, 7, OS X Snow Leopard and Lion, and of course Linux in a number of variants I don’t want to start thinking about), and that if it held a hundred megabytes it was very strange (the stick itself is 2GB in size, I didn’t have anything smaller at the time, nor I have now), you can easily tell that the space was rarely reclaimed.

This made it a problem in two very different ways: from one side, the compression couldn’t be very good, as the deleted files were still there and would account for hundreds of megabytes at least, as I’ve been using the same key for almost an year, and I had many boxes to update the BIOS of in the mean time, both mine and customers’; on the other side, some of the data in that image would include serial numbers for both Windows and Avira licenses I installed on customers’ systems (many times I only used one stick to bring the data in and out from a system: that stick).

In my mind the task to perform was clear: wipe out the empty areas of the filesystem so that they are all zero’d: saves on the compression and makes sure that the data is not going to be leaked. So I checked dosfstools and … no luck, there is no utility to do this. I’m not surprised, I don’t have a particular liking for dosfstools: back when it was picked up by Debian, I decided to help out and I worked for a few days on the buildsystem and code to update and improve it … after a quick note of “next release, before we want to just push in the changes that everybody applied”, it seems to simply have been ignored. I didn’t care much about working more on that project so I left it behind.

Googling around I found this project on sourceforge that seemed to be designed to do exactly what I needed. If you look at the page, and you’re used to both FLOSS projects and fake projects, you probably are wondering now if it’s legit or not. I asked that to myself as well, but I gave it a try in a VM, and as far as I can see, there is nothing malicious in it as it is, at least for what vfat is concerned; I explicitly didn’t want to try other filesystems.

Aside; if you are not wondering on the legitimacy of the project I linked, and don’t understand why I would say that… well it’s not really your average FLOSS project that boast “100% free/clean” certifications from websites such as GearDownload and Softpedia… being FLOSS should suffice.

Unfortunately, while the project does exactly what it’s meant to be.. saying that it does so sub-optimally is saying too little. There are quite a few issues with the project that makes me thing I’d be better off rewriting it from scratch than trying to get it fixed – and I’m not referring ot the website, as much as I have an opinion about it, I don’t think it’s my place to judge that, after all my own website is not really updated in … too long a time – starting from the use of a library that, albeit interesting, doesn’t seem to even have a single release, using even the private headers!

Okay I shouldn’t be ungrateful about this, since it did save my day, but it makes me cringe to think I can’t package it for others. And the fact that if you were to try running the tool on a file image of a 2GB USB stick, it would probably take a day or two to complete. What’s the matter? The library it uses, ttf-lib, uses fsync() over the device – a loop device in my case – each time it writes a sector… a 512bytes sector. You can easily tell it’s not the smartest move to do. I was quite bad already that it used write() over 512 sectors one at a time, but if it also synced the device, the tool becomes so slow it’s not funny! Couple of comment markers in the code to avoid the fsync, and the tool takes just a few moments. Of course if it actually waited for a discontinuity before writing to disk, it would probably work much better.

There are a few more issues with these two projects, but at least they seem to have a decent start. I should probably contact the authors and point them in the right direction for them to be packaged in Gentoo and other distributions.

Oh and as a final note, once compressed with xz -9e, the wiped image is just 384KiB, while the unwiped image is 610MiB. Yes that’s almost the side of a CD-ROM, after compression…. the small one can be downloaded here for those of you who want to use it. Sooner or later I’ll have to make a version for 1GiB sticks, I think my mother still has one or two of those I should be able to use, rather than waste a 2GiB stick that way.

BIOSing away

On my so-called dayjob (that more than often ends up keeping me up late at night), I often have to update motherboards’ BIOS data — it happened more than once that USB didn’t really work correctly with some device until the BIOS was updated. I have written about it before and fortunately since then, flashrom started supporting a huge number of motherboards, making it the first choice to update BIOS chips, when they are supported.

Now, when they aren’t, I’m not much of an expert in adding support for them; I have been able to send one patch (which hasn’t been applied still unfortunately) to simply list an extra motherboard, but since most of the boards I have at hand are not mine, too often I cannot go on and try — most of the time I don’t even have the time to do so.

The procedure I have written two and a half years ago would still produce a working bootable USB sticks, but it’s messy; googling around I found an (italian) tutorial (Update (2016-04-29): this has been replaced by an English tutorial) that based itself on installing FreeDOS itself on an USB stick through qemu. And that gave me a very nice idea to solve this cleanly without requiring strange software or dd commands.

For completeness sake, I’d like to add that SysRescueCD’s FreeDOS bootable image seems to crash with invalid instruction errors on almost all the systems I tried it on. No clue as to why.

You need an USB stick of at most 2GB; this ensures that it can be used with FAT16, for maximum compatibility, and the trick will only work properly on systems that can properly boot an USB strick in HDD compatibility mode, which seems to be any system capable of starting off an USB stick at all. In 2GB you can easily add quite a few firmware files, which is something I’m happy with, even if most of the time I don’t need them again after the first time.

For what concerns software, you only need qemu, a copy of FreeDOS (both the Full and Base versions would do), and some good old remembrance of DOS commands). Start up QEmu with the USB stick as the harddrive:

# qemu -hda /dev/sdm -cdrom fdbasecd.iso -boot d

Then use FDISK (the DOS variant) to create a single primary partition in the drive. Using the DOS FDISK command ensures that you will have a DOS-compatible partition without having to deal with sizes, alignments, types, and so on.

After re-booting FreeDOS from the CD with the newly-created partition, simply format it, from FreeDOS, and make it bootable:

A:> FORMAT /S C:

This copies the minimum required files to boot the disk up, which are well enough to actually update a BIOS. Contrarily to the tutorial I linked above, I don’t have any intention to install FreeDOS, which then would require editing the startup files to disable things. The straight boot is just what I need.

Also, a little funny note here. Today’s box to set up had a MSI motherboard; beside “International” being spelt wrong in the original 1.00 BIOS’s vendor strings, the specs page of the board on MSI’s site states very clearly that “for chipset limitations” neither Windows 98 nor ME can be used on the board… given it’s a Pentium4-era board, that went without saying for me. But on the BIOS update documentation, they still insist on telling you to use a Windows 98 or ME boot disk! Isn’t that lovely?

At any rate, now I have my BIOS update stick, and I’m pretty happy with it. Love FreeDOS.

Updating a BIOS without floppies, Windows or CDs

One regression I have with Yamato is that the BIOS cannot be updated through the BIOS itself, like I could do with my previous ASUS motherboard (and Award BIOS). The AMI bios of the Tyan motherboard doesn’t support that.

But since I don’t want to use my Windows installation to do the job (I use it only for work and to play more recent games that don’t play in Wine), I decided to look up something that could allow me to do the upgrade using only Free Software, my Gentoo Linux install, and the files provided by Tyan.

Caution: I make no warranty that the procedure will work properly, if this post gets published it means I was able to get it working myself but it might not work for you! Please remember that you do this at your risk!

I found an interesting post about FreeDOS and USB drives, which made my life much more easier. But as I’m using Linux and in particular Gentoo, I revised a bit the instructions :)

first of all, merge the software we’re going to need, this means sys-boot/makebootfat (from my overlay) and app-arch/libarchive (to extract an ISO file later on);
proceed then to create the directory where to put the files that will go on the virtual floppy, let’s call it “bios-update”;
download the new BIOS from Tyan’s website, and put the content of the zip archive in “bios-update”;
download FreeDOS (I suggest you to use the torrent download rather that ibiblio that is so slow!);
extract the ISO, with bsdtar it’s just the same as extracting a standard tar file: xf and all;
copy freedos/setup/odin/{command.com,kernel.sys} from the FreeDOS ISO to the “bios-update” directory;
from freedos/packages/src_base/kernels.zip extract the files source/ukernel/boot/fat{12,16,32lba}.bin.
run the makebootfat command with a commandline similar to this: makebootfat -o /dev/sde -E 255 -1 fat12.bin -2 fat16.bin -3 fat32lba.bin -m /usr/share/syslinux/mbr.bin bios-update; note that /dev/sde has to be changed to your USB device.

Now just restart your system with the USB drive in it and run the flash utility as provided by your manufacturer.

As an alternative you may rename an eventual batch file provided by your manufacturer to autoexec.bat so that it is ran at boot; I have no idea (yet) how to stop FreeDOS from asking date and time, but whatever, just press enter at them. You don’t usually have to worry about infinite loops of BIOS updates, as once the update is done, I’ve noticed all BIOS, after update, require you to fill in the configuration again.

I’m tempted to streamline this to a script or make the makebootfat ebuild fetch the needed files out of the FreeDOS ISO, or create an ebuild that install the basic FreeDOS files needed to create the boot disk. But maybe another time.