Inspecting and knowing your firmware images

Update: for context, here’s the talk I was watching while writing this post.

Again posting about the Enigma conference. Teddy Reed talked about firmware security, in particular based on pre-boot EFI services. The video will be available at some point, it talks in details about osquery (which I’d like to package for Gentoo), but also has a lower-key announcement of something I found very interesting: VirusTotal is now (mostly) capable of scanning firmware images of various motherboard manufacturers.

The core of this implementation leverages two open-source tools: uefi_firmware by Teddy himself, and UEFITool by Nikolaj Schlej. They are pretty good but since this is still in the early stages, there are still a few things to iron out.

For instance, when I first scanned the firmware of my home PC it was reported with a clearly marker of malware, which made me suspicious – and indeed got ASUS to take notice and look into it themselves – but it looks like it was a problem with parsing the file, Teddy’s looking into it.

On the other hand, sticking with ASUS, my ZenBook shows in its report the presence of CompuTrace — luckily for me I don’t run this on Windows.

This tool is very interesting under many different point of views, because not only it will (maybe in due time, as firmware behaviour analysis improves) provide information about possibly-known malware (such as CompuTrace) in a firmware upgrade, before you apply it, but even before you even buy the computer.

And this is not just about malware. The information that VirusTotal provides (or to be precise the tools behind it) include information about certificates, which for instance told me that my home PC would allow me to install Ubuntu under SecureBoot, since the Canonical certificate is present — or, according to Matthew Garrett, it will allow an Ubuntu signed bootloaded to boot just about anything defeating SecureBoot altogether.

Unfortunately this only works for manufacturers that provide raw firmware updates right now. ASUS and Intel both do that, but for instance Dell devices will provide the firmware upgrade only as a Windows (or DOS) executable. Some old extraction instructions exist, but they are out of date. Thankfully, Nikolaj pointed me at a current script that works at least for my E6510 laptop — which by the way also has CompuTrace.

That script, though, fails with my other Dell laptop, a Vostro 3750 — in that case, you can get your hands on the BIOS image by simply executing it with Wine (it will fail with an obscure error message) and then fetching it from Wine’s temporary folder. Similarly, it does not work with the updater for the XPS 13 (which I’m considering buying to replace the Zenbook), and in this case Wine is not of enough help (because it does not

Unfortunately that script does not work with the more recent laptops such as the XPS13 that I’m considering buying, so I should possibly look into extending it if I can manage to get it work, although Nikolaj with much more experience than me tried and failed to get a valid image out of it.

To complete the post, I would like to thank Teddy for pointing the audience to Firmware Security — I know I’ll be reading a lot more about that soon!

Remotely upgrading BIOS on a SuperMicro servers

So it’s that time of the year again when you have to run a few BIOS updates; and you know I love updating BIOS. In this case the victim is the very Excelsior that I bought to run the Gentoo tinderbox, and which I’m hosting to my expenses at Hurricane Electric in Fremont. This is important to say, because it means I have no direct access to the hardware, and nobody else can help me there.

First, I decided to get rid of the low-hanging fruit: the IPMI sideboard. I couldn’t find release notes for the update, but it went from version 2.05 to 3.16, and it was released at the end of June 2014, so it’s pretty recent, and one would expect quite a bit of bug fixes, given that I had trouble running its iKVM client two years ago already, and I tried to get more information about its protocol without success.

For this part, SuperMicro is strangely helpful: the downloaded archive (once I was able to actually download it, it took me a few tries, the connection would break midway through the download) is three times the size of the actual firmware, because it comes with Windows, DOS and, helpfully, Linux update utilities in the same file. Which would be endearing if it wasn’t that the Linux utilities require the tools to access the BMC raw, which is not fun to do with grsec present, and if I’m not mistaken it also conflicts with the in-kernel IPMI drivers. Also, the IPMI remote interface can handle the firmware update “quite well” by itself.

I say “quite well” because it seems like ATEN (the IPMI vendor for SuperMicro and another long list of server manufacturers) still has not mastered the ability to keep configuration consistent across firmware upgrades. You know, the kind of things that most of our home routers or cable TV boxes manage nearly all the time. And all my home computers’ BIOSes. So instead it requires you to reset the full configuration when you upgrade the firmware of the device. Which is fine because you can use bmc-config from userspace to dump the configuration (minus the passwords) and reset it back. If the configuration it downloads is actually accepted as valid, which in my case it wasn’t. But I only needed to set an admin password and IP address and routing, so…

The second step is to update the BIOS of the server itself. And here is where the fun part starts. You have two options to download: a Windows executable or a ZIP archive. I downloaded both. The Windows executable is supposed to build a floppy to upgrade the BIOS. Not a floppy image, a floppy. It does indeed not work in Wine, but I tried! I settled for the ZIP file.

So the iKVM I just upgraded supports “virtual media” — which means you can give it files on your computer that it’ll use instead of floppy disks or CD-Rom. This is very neat, but it comes with some downsides. At first I wanted to give it a floppy disk image via HTTP, and use the Serial-over-SSH to enter the boot menu and do stuff. This has worked like a charm for me before, but it comes with limitations: the HTTP-based virtual media configuration for this iKVM only supports 1.44MB images for floppy disks, and uses SMB to access shared ISO files for the CD-Rom. It makes sense, you can’t expect the iKVM board to have enough RAM to store a whole 650MB image for the CD-Rom, now, can you?

The answer is of course to build a BIOS upgrade floppy disk, which is after all what their own Windows tool would have done (albeit on a real disk rather than an image — do people really still use floppy disks on Opteron servers?) Too bad that there is a small problem with that plan: the BIOS itself is over 2MB uncompressed. It does not fit in a floppy disk unformatted, let alone in the formatted disk, with the update utility, the kernel and the command interpreter. So no way to do that easily. The astutes of you after reading the blog post will probably figure out there was a shortcut here, but I’ll comment on that later.

So I decided to go back to the iKVM client, the Java WebStart one. Now luckily the new release of it works fine with IcedTea, unlike the previous one. Unfortunately it still is not working out of the box. Indeed it was only by chance that I found someone else who faced and solved that problem by adding a couple of lines to the XML file that defines the JavaWS environment.

The configuration dialog for the client allows providing a local ISO file as well as configuring the address of a SMB share with it: it implements its own transmission protocol, and the floppy option is now “floppy/USB”, so I thought maybe I could just give it my 1GB FreeDOS-based BIOS disk … nope, no luck. It still is limited to 1.44MB it seems.

So I had to get a working ISO and do the upgrade. It should be pretty straightforward, right? I even found a blog post showing how to do so, based on Gentoo, as it says to emerge isomaster! Great! Unfortunately, FreeDOS download page now provides you with a great way to download … installers. They no longer provide links to download live CDs. It makes total sense, right? Who doesn’t think every day “I should stop procrastinating and finally install FreeDOS on my laptop!”?

After some fiddling around I found a tiny non-descript directory in their FTP mirrors that includes fdfull.iso which is the actual LiveCD. So I download that, copy the BIOS and the updater into the CD and set it in the virtual media settings for the iKVM, and then boot. The CD does not boot unattended. If you leave it unattended it’ll ask you what you want to do and default to boot from the HDD. If you do remember to type 1 and press enter at the SysLinux prompt, it’ll then default to install (again, why did I bother installing Gentoo on the NUC when I could have used FreeDOS like the cool kids?).

Instead, you can choose to boot FreeDOS from the LiveCD with no support and no drivers, with just the CD drivers, with the CD drivers and HIMEM, with the CD drivers, HIMEM and EMM386. I decided I wanted to play it safe since it’s a BIOS update, and booted with just the CD drivers (I needed the BIOS files after all). Turns out that while this is an option at the boot menu, it will never work, as the CDROM driver needs to be loaded high, MSCDEX.EXE style. So reboot once again with HIMEM (and EMM386 because rebooting into that CD is painful — among other things, the iKVM client loses focus every time the framebuffer size changes. And it changes. all. the freaking. time. during boot up), and try to access the CD… unfortunately the driver does not support USB CD-Rom (like the one the iKVM provides me with) but just IDE ones, so no access to the ISO for me. Nope.

But I’m not totally unprepared for this kind of situations. As a failsafe device I left in one of the USB slots a random flashdrive I got at Percona Live reflashed with SysRescueCD — it was one of those because I had gotten a couple while over there, and I forgot to bring one from home. Since SysRescueCD uses VFAT by default, and the BIOS shows the USB devices transparently to DOS, it was visible as C:.

Here is the shortcut from above: I should have just used a random FreeDOS boot floppy image, and just copied the BIOS onto that particular thumbdrive, back before I started using Java. It would have been much simpler. But I forgot about that thumbdrive and I was hoping I would be able to start bootdisk. Now I know better.

Of course that meant I had to finish booting the system in Linux proper, mount the USB flashdrive, scp over the BIOS update, and reboot. It took much longer than I wanted. But still much less than all the time trying to get FreeDOS into booting the right thing.

By the way, even if I were to find a way to access the CD with the bios updater, it wouldn’t have helped. I did not notice that, but the BIOS updater script FLASH.BAT wants to be smart, and it renames AFUDOS.SMC to AFUDOS.EXE, runs it, then renames it back. Which means that it’s not usable from read-only media. Oh yeah and it requires a parameter, but if you don’t provide it, it’ll tell you that the command is unknown (because it still passes a bunch of flags to the flash utility), and tells you to reboot.

Now this is only the problem with getting the BIOS updated. What happened after would probably deserve a separate post, but I’ll try to be short and write that down here. After update, of course I got the usual “CMOS checksum error, blah blah, Press F1 to enter Setup.”. So I went and reconfigured the few parameters that I wanted different from the defaults, for instance put the SATA controller into AHCI mode (IDE compatibility, seriously?) and disabled the IDE controller itself, plus enabled SVM which on the defaults is still disabled.

The system booted fine, but no network cards were to be found. Not a driver problem, they would not appear on the PCI bus. To be precise, now that I compared this with the actual running setup, the two PLX PCI-E switches were not to be found on the bus, the network cards just happen to be connected to them. I tried playing with all the configuration options for the network cards, for PCI, for PnP and ACPI. Nope. Tried a full remote power cycle, nope. I was already considering the costs of adding a riser card and a multiport card to the server (wouldn’t have helped, as I noted the PLX were gone), when Janne suggested doing a reset to defaults anyway. It worked. The settings were basically the same I left before (looks like “optimal defaults” do include SVM), just changed SATA back to AHCI (yes, even in optimal settings, it defaults to IDE compatibility).

It worked. But then only three out of five drives were recognized (one of the SSDs was disconnected when I was there in April and I hadn’t had time to go back there in June to put it back). One more reboot into BIOS to disable the IDE controller altogether and the rest of the HDDs appeared, together with their LVM group.

Yes, I know, SuperMicro suggests you to only do BIOS updates when they require you to. On the other hand, I was hoping for an SVM/IOMMU fix which is not there still, as KVM made it complain, so I wanted to try. I was used to Tyan, that Yamato still uses, that basically told you “don’t bother us with support requests unless you updated your BIOS” — a policy that I find much more pleasant, especially given other experience with other hardware (such as Intel’s NUC requiring a BIOS update before it can use Intel’s WiFi cards….).

Updating firmware on a Crucial M4 drive, with Linux, no luck

When you think of SSD manufacturers, it might be obvious to think of them as Linux friendly, given they target power users, and Linux users are for the most part are power users. Seems like this is not that true for Crucial. My main personal laptop has, since last year, a 64GB Crucial M4 SSD – given I’ve not been using a desktop computer for a while it does start to feel small, but that’s a different point – which underwent a couple of firmware update since then. In particular, there is a new release 070H that is supposed to fix some nasty power saving issues.

Crucial only provide firmware update utilities for Windows 7 and 8 (two different versions of them), and then they have an utility for “Windows and Mac” — the latter is actually a ZIP file that contains an ISO file… well, I don’t have a CD to burn with me, so my first option was to run the Windows 7 file from my Windows 7 install, which resides on the external SATA harddrive. No luck with that, from what I read on the forums what the upgrader does is simply setting up an EFI application to boot, and then reboot. Unfortunately in my case there are two EFI partitions, because of the two bootable drives, and that most likely messes up with the upgrader.

Okay strike one, let’s look at the ISO file. The ISO is very simple and very small.. it basically is just an ISOLINUX tree that uses memdisk to launch a 2.88MB image file (2.88MB is a semi-standard floppy disk size, which never really went popular for real disks, but has been leveraged by most virtual floppy disk images in bootable CD-Roms for the expanded size). Okay what’s in the image file then? Nothing surprising, it’s a FreeDOS image, with Crucial’s own utility and the firmware.

So if you remember, I had some experience with trying to update BIOS through FreeDOS images, and I have my trusty USB stick with FreeDOS arriving on Tuesday with most of my personal effects that were in Italy waiting for me to find a final place to move my stuff on. But I wanted to see if I could try to boot the image file without needing the FreeDOS stick, so I checked. Grub2 does not include a direct way to load image — the reference to memdisk in their manual refers to the situation where you’re loading a standalone or rescue Grub image, nothing to do with what we care about.

The most obvious way to run the image is through SYSLINUX’s memdisk loader. Debian even has a package that allows you to just drop the images in /boot/images and adds them to the grub menu. Quick and easy, no? Well, no. The problem is that memdisk needs to be loaded with linux16 — and, well, Grub 2 does not support it if you’re booting via EFI, like I am.

I guess I’ll wait until Tuesday night, when my BIOS disk will be here, and I’ll just use it to update the SSD. It should solve the issue once and for all.

Revenge of the HP Updates

Just shy of three months ago I was fighting with updating the iLO firmware (IPMI and extras) and as I recounted, even when you select downloads for RHEL (which is a supported operating system on those boxes), you’re given a Windows executable file, which you have to extract. But at least, you can use the file you extract, to update the IPMI firmware remotely.

Well, if it wasn’t for a small little issue that the fans are going to get stuck at 14 KRPM until you also update the BIOS. It wasn’t obvious how much of a problem that is until we got to the co-location last week and… “What on Earth is this noise?” “I think it’s our servers!” screamed on the backside of the cabinet.

Since one of the servers also had some other hardware issues (one of the loops that keep the chipset’s heatsink gave way — I glued it back and applied a new layer of thermal paste after scraping the old one), we ended up bringing it back to the office, where today, after repairing it and booting, it became obvious that we couldn’t leave it running at any time with that kind of noise. So it was time to update the BIOS. Which is easier said than done.

Step one is finding the correct download — the first one I found turned out to be wrong, but it took me some time to understand that, because BIOS update has to be done with DOS. And that brought me back to a very old post of mine (well, not that old, it’s just an year and a half ago, now that I see), and its follow-up which came with a downloadable 383KB compressed, 2GB uncompressed bootable FreeDOS image. Since getting sysrescuecd’s FreeDOS to do anything other than booting and playing its own demos was impossible.

So when I actually get to run the executable in the FreeDOS image … what I come to is an extremely stupid tool (that, I remember you, will not work on Windows XP, Vista or 7) to create an USB drive to update the BIOS… lol, whut?

The correct download is, once again, for Windows even when you select RHEL4, and it auto-extracts in a multitude of files that include the BIOS itself some four different times, and would provide some sort of network update, as well as “flat files” (which you can use with FreeDOS), a Windows updater, an ISO file, and an utility to build an USB stick to update the BIOS itself.

If you count the fact that this is for a server running Linux, you now just involved two more operating systems. And the next trip to the co-lo we’ve got work cut out for us, updating server by server the BIOS, and the IPMI firmware (hoping that the new firmware actually have a reliable SOL connection, among others).

But to avoid being all too negative with HP, it’s still better than trying to do standard sysadmin work on an Apple OS X Server install on a Mac Mini. OS X Server combines UNIX’s friendliness, with Windows’s remote management capabilities, Solaris’s hardware support, and AIX software availability. But that’s a topic for another post.

On filesystem wiping

I have been talking about BIOS with “facebook friends” today, and Davide (from KDE) asked me if I could send him a copy of my FreeDOS stick, or at least the instructions on how to do so which obviously were on this very blog.

At first I wanted to just send him a compressed copy of the stick itself; then I remembered that I’m not that stupid and I checked out the content of the actual image. The problem is that even if I remove the files as they appear on the filesystem, FAT does not clear them out until the space is needed again; given that my USB stick has been used on almost every imaginable operating system (Windows XP, Vista, 7, OS X Snow Leopard and Lion, and of course Linux in a number of variants I don’t want to start thinking about), and that if it held a hundred megabytes it was very strange (the stick itself is 2GB in size, I didn’t have anything smaller at the time, nor I have now), you can easily tell that the space was rarely reclaimed.

This made it a problem in two very different ways: from one side, the compression couldn’t be very good, as the deleted files were still there and would account for hundreds of megabytes at least, as I’ve been using the same key for almost an year, and I had many boxes to update the BIOS of in the mean time, both mine and customers’; on the other side, some of the data in that image would include serial numbers for both Windows and Avira licenses I installed on customers’ systems (many times I only used one stick to bring the data in and out from a system: that stick).

In my mind the task to perform was clear: wipe out the empty areas of the filesystem so that they are all zero’d: saves on the compression and makes sure that the data is not going to be leaked. So I checked dosfstools and … no luck, there is no utility to do this. I’m not surprised, I don’t have a particular liking for dosfstools: back when it was picked up by Debian, I decided to help out and I worked for a few days on the buildsystem and code to update and improve it … after a quick note of “next release, before we want to just push in the changes that everybody applied”, it seems to simply have been ignored. I didn’t care much about working more on that project so I left it behind.

Googling around I found this project on sourceforge that seemed to be designed to do exactly what I needed. If you look at the page, and you’re used to both FLOSS projects and fake projects, you probably are wondering now if it’s legit or not. I asked that to myself as well, but I gave it a try in a VM, and as far as I can see, there is nothing malicious in it as it is, at least for what vfat is concerned; I explicitly didn’t want to try other filesystems.

Aside; if you are not wondering on the legitimacy of the project I linked, and don’t understand why I would say that… well it’s not really your average FLOSS project that boast “100% free/clean” certifications from websites such as GearDownload and Softpedia… being FLOSS should suffice.

Unfortunately, while the project does exactly what it’s meant to be.. saying that it does so sub-optimally is saying too little. There are quite a few issues with the project that makes me thing I’d be better off rewriting it from scratch than trying to get it fixed – and I’m not referring ot the website, as much as I have an opinion about it, I don’t think it’s my place to judge that, after all my own website is not really updated in … too long a time – starting from the use of a library that, albeit interesting, doesn’t seem to even have a single release, using even the private headers!

Okay I shouldn’t be ungrateful about this, since it did save my day, but it makes me cringe to think I can’t package it for others. And the fact that if you were to try running the tool on a file image of a 2GB USB stick, it would probably take a day or two to complete. What’s the matter? The library it uses, ttf-lib, uses fsync() over the device – a loop device in my case – each time it writes a sector… a 512bytes sector. You can easily tell it’s not the smartest move to do. I was quite bad already that it used write() over 512 sectors one at a time, but if it also synced the device, the tool becomes so slow it’s not funny! Couple of comment markers in the code to avoid the fsync, and the tool takes just a few moments. Of course if it actually waited for a discontinuity before writing to disk, it would probably work much better.

There are a few more issues with these two projects, but at least they seem to have a decent start. I should probably contact the authors and point them in the right direction for them to be packaged in Gentoo and other distributions.

Oh and as a final note, once compressed with xz -9e, the wiped image is just 384KiB, while the unwiped image is 610MiB. Yes that’s almost the side of a CD-ROM, after compression…. the small one can be downloaded here for those of you who want to use it. Sooner or later I’ll have to make a version for 1GiB sticks, I think my mother still has one or two of those I should be able to use, rather than waste a 2GiB stick that way.

Lesson learnt: flashrom is not for the faint of heart

I have recently written about using FreeDOS to perform BIOS updates and I stated that before going that route I have been using flashrom – coming out of the coreboot project – that allows for hot update of BIOS without requiring booting into DOS at all.

The project in the past months started supporting pretty well a long list of motherboards, including one for which I added support myself (an ASUS Pentium 4 board I had to update for a customer, one where USB support didn’t seem to work properly without a BIOS update), but the other day I hit a very nasty problem linked to it.

I wanted to update the BIOS of the small frontend system I use as a desktop (Yamato is now working headless), even though the only thing that it would improve is USB 3 support and I have no USB 3 device (yet); on the board (ASUS based on nVidia ION platform) flashrom worked quite fine, so I flashed the BIOS, restarted, removed the backup and… just then I noticed I had no network.

The problem is that the Realtek 8169 integrated on the board reads its MAC address from somewhere in the BIOS flash; when updating with the ASUS tools it would have been preserved, but when using flashrom, well, it doesn’t know what it shouldn’t erase and rewrite. This caused the MAC address on that box to be overwritten with garbage. Luckily it doesn’t really matter, for that system, since I simply used the mac_eth0 setting to change the mac address to the one it used before, so that the same IP addresses would be used.

Speaking with the developers of the software suggested me that there are many ways in which the mac address could be encoded into the BIOS, and also noted that the same is true for DMI and SMBIOS data, which is no issue for me most of the time, since I don’t have any (I am my own OEM), but can be quite a problem for customer’s computers.

Also, clumsiness ants that I were to delete the backup file I read from the previously-used BIOS before I noticed the lack of network access, otherwise I would have been able to compare it with the stock BIOS (for the same version) provided by ASUS; looking for the new mac address as a binary string in the data as read back by flashrom, with the help of bgrep (which is not in Portage, does anybody know if there is something like that in Portage already or if I should rather add that one?) does not yield any useful result, either, so I can’t seem to be able to find where it’s actually stored.

This probably also explains why Yamato reports 88:aa:99:bb:dd:ee and 60:50:40:30:20:10 as mac addresses.. and I thought it was the BMC’s fault! I guess I should re-flash the Tyan firmwares and see if I can reprogram the network cards (if I remember correctly, their utilities allow to do a permanent change of mac address). Could possibly make the BMC-over-LAN access more reliable, it doesn’t seem to be much right now.

At any rate, I’m not going to suggest against the use of flashrom, but definitely I suppose I learnt to trust it a bit less, and be more wary about it than I was before; I’m happy it happened with a box of mine and not a customer’s, that could have been quite an issue.

BIOSing away

On my so-called dayjob (that more than often ends up keeping me up late at night), I often have to update motherboards’ BIOS data — it happened more than once that USB didn’t really work correctly with some device until the BIOS was updated. I have written about it before and fortunately since then, flashrom started supporting a huge number of motherboards, making it the first choice to update BIOS chips, when they are supported.

Now, when they aren’t, I’m not much of an expert in adding support for them; I have been able to send one patch (which hasn’t been applied still unfortunately) to simply list an extra motherboard, but since most of the boards I have at hand are not mine, too often I cannot go on and try — most of the time I don’t even have the time to do so.

The procedure I have written two and a half years ago would still produce a working bootable USB sticks, but it’s messy; googling around I found an (italian) tutorial (Update (2016-04-29): this has been replaced by an English tutorial) that based itself on installing FreeDOS itself on an USB stick through qemu. And that gave me a very nice idea to solve this cleanly without requiring strange software or dd commands.

For completeness sake, I’d like to add that SysRescueCD’s FreeDOS bootable image seems to crash with invalid instruction errors on almost all the systems I tried it on. No clue as to why.

You need an USB stick of at most 2GB; this ensures that it can be used with FAT16, for maximum compatibility, and the trick will only work properly on systems that can properly boot an USB strick in HDD compatibility mode, which seems to be any system capable of starting off an USB stick at all. In 2GB you can easily add quite a few firmware files, which is something I’m happy with, even if most of the time I don’t need them again after the first time.

For what concerns software, you only need qemu, a copy of FreeDOS (both the Full and Base versions would do), and some good old remembrance of DOS commands). Start up QEmu with the USB stick as the harddrive:

# qemu -hda /dev/sdm -cdrom fdbasecd.iso -boot d

Then use FDISK (the DOS variant) to create a single primary partition in the drive. Using the DOS FDISK command ensures that you will have a DOS-compatible partition without having to deal with sizes, alignments, types, and so on.

After re-booting FreeDOS from the CD with the newly-created partition, simply format it, from FreeDOS, and make it bootable:

A:> FORMAT /S C:

This copies the minimum required files to boot the disk up, which are well enough to actually update a BIOS. Contrarily to the tutorial I linked above, I don’t have any intention to install FreeDOS, which then would require editing the startup files to disable things. The straight boot is just what I need.

Also, a little funny note here. Today’s box to set up had a MSI motherboard; beside “International” being spelt wrong in the original 1.00 BIOS’s vendor strings, the specs page of the board on MSI’s site states very clearly that “for chipset limitations” neither Windows 98 nor ME can be used on the board… given it’s a Pentium4-era board, that went without saying for me. But on the BIOS update documentation, they still insist on telling you to use a Windows 98 or ME boot disk! Isn’t that lovely?

At any rate, now I have my BIOS update stick, and I’m pretty happy with it. Love FreeDOS.

Dell was a definite mistake, and an expensive one

This week, my newly-bought laptop arrived; as I noted I was looking for a computer that had a Trackpoint device, to avoid touching the touchpad while I’m writing. This brought me to exactly two viable alternatives:

Lenovo, with their Thinkpad, had a good track record with Linux; they also have usually a decent price (doesn’t mean they are cheap but that they are worth their pricetag) and in general they were my first choice;
Dell and the Latitude E65xx series was the second choice; they didn’t have such a good track record, but most people didn’t complain much about them, beside for minor annoyances.

Again, as I said above, Lenovo doesn’t sell directly in Italy, who knows why; unfortunately this is also the cause for their price to be quite higher in Italy in general. Also, I don’t have many options for Lenovo resellers in my area; the nearest one (50Km from here) sent me the price for “the model I asked for” after a week I asked for it: after VAT it was €300 more than the same from Lenovo UK. I also quoted this being the model I asked for because it really wasn’t!

Not only they still forced me to pick up a Windows license (okay, not too nice but I can live with it), but they also forgot to upgrade the RAM to 4GiB as I asked and, quite a bad one, they didn’t provide me with an integrated smartcard reader. Indeed, the first time around the guy at the sales department of the reseller asked me if I was sure to ask for a BTO about that, given that all T510 already had (translated, but literal) “reader of smartcards of SD type, those for photo cameras”. When the price came, they actually were proposing me to buy a T510 and a smartcard reader, from GemAlto and Lenovo branded. Not a ExpressCard reader though, an USB reader. I guess you can tell what the problem is with that; if not, the problem was that I already have an USB card reader, it’s just difficult to use on, say, a train.

So I surveyed the Dell options; they didn’t allow me to forgo on Windows 7 from the website, so I called the hotline and asked for a sales person to help me out; they refused right away to switch my keyboard for an US one, which was unfortunate but bearable, then they told me that they’d let me know if it was possible to avoid Windows; in two days they came back with an offer… for the E6510 with Ubuntu Linux rather than Windows 7, base options (no 4GiB memory, which I asked for, and no webcam, fingerprint reader, RFID reader, which I didn’t ask for and I didn’t/don’t need)… but at a €500 premium over the same laptop from the website with Windows 7. For the same price, I could get the highest-level option, with Core i7, all the extras and so on.

So the Dell arrived, and I started installing Gentoo on it; beside a series of quirks, which I googled up, such as the smartcard reader needing patching of OpenSC/OpenCT to work, or the fact that the RFID and fingerprint readers not working under Linux (with Daniel confirming that there is realistically no hope of getting the new ones to work on Linux any time soon), everything seemed to go fine. The network card is definitely stable and fast; the wireless card required me to create a new firmware ebuild because of the many ucode ebuilds we have in portage, mine was missing, but it was a matter of minutes.

The problem started when it came down to run Xorg. The nVidia card works quite fine with the drivers, so that’s not a problem, but the touchpad, oh the touchpad. For apparently no reason first, the touchpad was being recognised as a simple mouse; and contrarily to what most people told me about the Thinkpad laptops, the touchpad and the trackpoint in Dell’s hardware are not separate (hardware) devices; they appear as separate devices in Linux because the ALPS driver in the kernel splits them after parsing their different protocols. Unfortunately, both the E6400 and the E6500 Latitude models have a new Alps Electric combined GlidePoint device with a protocol that is, as of now, unsupported by the Linux kernel; Ubuntu submitted some patches for it to the kernel but none that works yet. Right now the device is seen as a standard PS/2 mouse and entirely handled in BIOS, without special features or settings.

The second problem came when it was time to turn off the laptop: halting the system causes it to reboot instead; I thought the problem was related to ACPI, so I looked up if there were BIOS updates, and lo and behold, two were around. Unfortunately, applying them is not the usual matter of running flashrom, on a laptop. Dell used to develop Linux tools for their laptops, including firmware update software; as far as I know they were the first vendor doing so. Unfortunately, they either stopped, or this model is not covered, and the only BIOS updates are provided bundled within the Windows-based flasher (which most likely is not the real flasher at all, given that the system reboots itself before flashing both BIOS and the firmware of the Embedded Controller.

Now, not everything goes bad, luckily for me. I mailed Matthew Garrett, to ask him for pointers on what I could try to get it at least to halt properly, and he’s suggested me a git tree to try which I’m now building. He also pointed out that there is at least some work going on to solve the Alps touchpad problem (which makes me hopeful that this will be properly solved before end of the year). And of course he didn’t have to tell me why the external monitor button prints a ‘p’ since I knew the unfortunate reasons already.

On the bright side, the hardware looks, by itself, tremendously nice: the keyboard, while not as good as the Apple Aluminiums is quite solid and nice to write on, the touchpad is not as invasive as on the MacBook Pro, the monitor is gorgeous and it has all kind of expansion ports including eSATA. The battery lasts a lot even on Linux and even without setting up the governors properly, and so on so forth. I just hope the few problems will smooth themselves out soon.

Updating a BIOS without floppies, Windows or CDs

One regression I have with Yamato is that the BIOS cannot be updated through the BIOS itself, like I could do with my previous ASUS motherboard (and Award BIOS). The AMI bios of the Tyan motherboard doesn’t support that.

But since I don’t want to use my Windows installation to do the job (I use it only for work and to play more recent games that don’t play in Wine), I decided to look up something that could allow me to do the upgrade using only Free Software, my Gentoo Linux install, and the files provided by Tyan.

Caution: I make no warranty that the procedure will work properly, if this post gets published it means I was able to get it working myself but it might not work for you! Please remember that you do this at your risk!

I found an interesting post about FreeDOS and USB drives, which made my life much more easier. But as I’m using Linux and in particular Gentoo, I revised a bit the instructions :)

first of all, merge the software we’re going to need, this means sys-boot/makebootfat (from my overlay) and app-arch/libarchive (to extract an ISO file later on);
proceed then to create the directory where to put the files that will go on the virtual floppy, let’s call it “bios-update”;
download the new BIOS from Tyan’s website, and put the content of the zip archive in “bios-update”;
download FreeDOS (I suggest you to use the torrent download rather that ibiblio that is so slow!);
extract the ISO, with bsdtar it’s just the same as extracting a standard tar file: xf and all;
copy freedos/setup/odin/{command.com,kernel.sys} from the FreeDOS ISO to the “bios-update” directory;
from freedos/packages/src_base/kernels.zip extract the files source/ukernel/boot/fat{12,16,32lba}.bin.
run the makebootfat command with a commandline similar to this: makebootfat -o /dev/sde -E 255 -1 fat12.bin -2 fat16.bin -3 fat32lba.bin -m /usr/share/syslinux/mbr.bin bios-update; note that /dev/sde has to be changed to your USB device.

Now just restart your system with the USB drive in it and run the flash utility as provided by your manufacturer.

As an alternative you may rename an eventual batch file provided by your manufacturer to autoexec.bat so that it is ran at boot; I have no idea (yet) how to stop FreeDOS from asking date and time, but whatever, just press enter at them. You don’t usually have to worry about infinite loops of BIOS updates, as once the update is done, I’ve noticed all BIOS, after update, require you to fill in the configuration again.

I’m tempted to streamline this to a script or make the makebootfat ebuild fetch the needed files out of the FreeDOS ISO, or create an ebuild that install the basic FreeDOS files needed to create the boot disk. But maybe another time.