Saving a non-booting Asus UX31A laptop

I have just come back from a long(ish) trip through UK and US, and decided it’s time for me to go back to some simple OSS tasks, while I finally convince myself to talk about the doubts I’m having lately.

To start on that, I tried to turn on my laptop, the Asus UX31A I got three and a half years ago. It didn’t turn on. This happened before, so I just left it to charge and tried again. No luck.

Googling around I found a number of people with all kind of problems about it, and one of them is something getting stuck at the firmware level. Given how I had found a random problem with PCIE settings in my previous laptop, that would make it reboot every time I turned it off, but only if the power was still plugged in, I was not completely surprised. Unfortunately following the advice I read (take off the battery and power over AC) didn’t help.

I knew it was not the (otherwise common) problem with the power plug, because when I plugged the cable in, the Yubikey Neo-n would turn on, which means power arrived to the board fine.

Then I remembered two things: one of the advices was about the keyboard, and the keyboard itself has had problems before (the control key sometimes would stop working for half an hour at a time.) Indeed, once I re-seated the keyboards’ ribbon cable, it turned on again, yay!

But here’s the other problem: the laptop would turn on, the caps-lock LED on and stay there. And even letting the main battery run out would not be enough to return it to working conditions. What to do? Well, I got a hunch, and turned out to be right.

One of the things that I tried before was to remove the CMOS battery — either I kept it out not long enough to properly clear, or something else went wrong, but it turned out that removing the CMOS battery allowed the system to start up — but that would mean no RTC, which is not great, if you start the laptop without an Internet connection.

The way I solved it was as follows:

disconnect the CMOS battery;
start up the laptop;
enter “BIOS” (EFI) setup;
make any needed change (such as time);
“Save and exit”;
let the laptop boot up;
connect the CMOS battery.

Yes this does involve running the laptop without the lower plate for a while, be careful about it, but to the other hand, it did save my laptop from being stomped on, on the ground out of sheer rage.

Inspecting and knowing your firmware images

Update: for context, here’s the talk I was watching while writing this post.

Again posting about the Enigma conference. Teddy Reed talked about firmware security, in particular based on pre-boot EFI services. The video will be available at some point, it talks in details about osquery (which I’d like to package for Gentoo), but also has a lower-key announcement of something I found very interesting: VirusTotal is now (mostly) capable of scanning firmware images of various motherboard manufacturers.

The core of this implementation leverages two open-source tools: uefi_firmware by Teddy himself, and UEFITool by Nikolaj Schlej. They are pretty good but since this is still in the early stages, there are still a few things to iron out.

For instance, when I first scanned the firmware of my home PC it was reported with a clearly marker of malware, which made me suspicious – and indeed got ASUS to take notice and look into it themselves – but it looks like it was a problem with parsing the file, Teddy’s looking into it.

On the other hand, sticking with ASUS, my ZenBook shows in its report the presence of CompuTrace — luckily for me I don’t run this on Windows.

This tool is very interesting under many different point of views, because not only it will (maybe in due time, as firmware behaviour analysis improves) provide information about possibly-known malware (such as CompuTrace) in a firmware upgrade, before you apply it, but even before you even buy the computer.

And this is not just about malware. The information that VirusTotal provides (or to be precise the tools behind it) include information about certificates, which for instance told me that my home PC would allow me to install Ubuntu under SecureBoot, since the Canonical certificate is present — or, according to Matthew Garrett, it will allow an Ubuntu signed bootloaded to boot just about anything defeating SecureBoot altogether.

Unfortunately this only works for manufacturers that provide raw firmware updates right now. ASUS and Intel both do that, but for instance Dell devices will provide the firmware upgrade only as a Windows (or DOS) executable. Some old extraction instructions exist, but they are out of date. Thankfully, Nikolaj pointed me at a current script that works at least for my E6510 laptop — which by the way also has CompuTrace.

That script, though, fails with my other Dell laptop, a Vostro 3750 — in that case, you can get your hands on the BIOS image by simply executing it with Wine (it will fail with an obscure error message) and then fetching it from Wine’s temporary folder. Similarly, it does not work with the updater for the XPS 13 (which I’m considering buying to replace the Zenbook), and in this case Wine is not of enough help (because it does not

Unfortunately that script does not work with the more recent laptops such as the XPS13 that I’m considering buying, so I should possibly look into extending it if I can manage to get it work, although Nikolaj with much more experience than me tried and failed to get a valid image out of it.

To complete the post, I would like to thank Teddy for pointing the audience to Firmware Security — I know I’ll be reading a lot more about that soon!

Dealing with closed-source firmware and latent bugs

Another day, another customer with issues with the closed-source firmware of one of their devices, even if this time it’s not as bad as the one who got angry yesterday when I dared suggest I would have liked to see the device’s firmware to see if I could isolate the bug.

This time the device is a RIP server for Xerox printers, developed by EFI (no, not the one Matthew Garrett is fighting with). In particular, it’s the server that comes with a DocuColor 250 “copier” a very old customer of mine has recently acquired.

This copier, which is actually a much higher end multi-function printer, is replacing, for my customer, the previous DocuPrint 700 copier, which also has an EFI RIP server. The 700 is actually the more recent one, but that’s my customer’s choice to go with an older version, and thus I have no say in the matter.

As you can probably guess, both devices come with a scanner (otherwise they wouldn’t be called copiers), and both scan to so-called “mailboxes”, which are on-RIP storage areas with predefined configuration values. One of the most important things to keep in mind with these mailboxes, though, is that they can be set to transfer the scanned files to a different storage area through the network. On the DP700 RIP, I was able to set it up to use CIFS to send the data on a shared folder on their backup server – after they updated to OS X Lion, since Apple replaced Samba with a SMB2-only server – but the same trick wouldn’t work with the DC250, since that version, much older, does not support CIFS at all.

Instead, to push files out, this version only supports FTP, which is generally nice and easy: just set it up on the same server, to access the same shared directory, and you’re done.. or not? Turns out that the release notes for the RIP firmware state explicitly this:

Scan to FTP not supported for all FTP servers

Scan to FTP is not supported for all FTP servers. If an FTP server does not reply to a NLST command with a 4xx or 5xx response, scan to FTP from the Fiery does not work correctly. The WS_FTP server is known to have this issue.

While these three lines are all the documentation you can find about the topic, a quick sniffing around gave me a good idea of what the problem is. When writing the files to the FTP server, the RIP software is trying to find a unique name to use; instead of using the current date/time to generate one, it tries over and over until it finds a free one. The problem is that to do so, it expects the NLST (short listing) command to return an error if there is no file matching the name… which is not what RFC959 (the specifications of FTP) tells you to do. VSFTPd does it exactly like WS_FTP is said to do above: it will return a success message.. and an empty list.

Again, this is the kind of problems that I would like to be able to fix, but since this is a closed-source, proprietary firmware, it’s out of the question that I’d be able to (besides, the developers know the problem exists already, as they wrote it down, yet there is no fix available). Combine this with the obnoxious experience of “local” developers who pretend to be holy and get angry I dare to think I’m as good as them, and you can probably guess why I’m a Free Software developer.

Hardened and EFI aren’t buddies

For my work I store a huge amount of data in my systems, starting with a number of customers’ systems’ backups, which take the best part of 1TB of storage to preserve for the usual year or two that I usually store them (customers are very happy that way). This unfortunately means that even the 5TB of space present in Yamato is quite restrictive for my business. It’s for this reason, and others, that I ordered the components to build myself a 12TB storage server.

For said server I decided to go with an AMD Fusion board, by Sapphire, both because I’ve been disappointed by Intel’s Atom (and nVidia’s ION), and because the price was right for a board with 5 SATA ports and an eSATA one — four 3TB Western Digital Caviar Green disks, one Kingston SSD that I still had laying around, plus a WD MyBook Studio II that is currently connected to Yamato. In particular, for those asking, the fact that most of the “high end” Atoms – such as the D510 I have on my frontend – do not support CPU speed throttling is appalling; the AMD Fusion board I have do not have said problem.

Anyway, the board itself looks good and works fine — and it was curious for me to notice that it uses UEFI 2.1, rather than BIOS, as its firmware.. it’s the first desktop-class board I found with such a setup (admittedly I haven’t bought many, lately). Also, while the IOMMU feature is disabled by default, the help text for it in the configuration pages states that it is useful to translate I/O operations under Linux.. yes it explicitly mentions Linux in the help text! Kudos to them!

Out of experience with Yamato, when you have so many disks with a complex situation (RAID1 and LVM), booting can be troublesome; in Yamato’s case a bit more, as the order the disks are listed on depends among other things by the order in which they are detected by the kernel, which can change between version and version since the box uses three different controllers and three different drivers. For this reason I was hoping to use UEFI-style booting through grub2 rather than the classic old BIOS boot…

Turns out it wasn’t that good an idea; even with scarabeus’s guide, I was unable to get it to work; it seems like to set it up properly you need to boot an EFI-capable kernel (which is not the one in SysRescueCD, that I usually use to install my systems), and boot it EFI-style. If you were to boot it without EFI support, you wouldn’t be able to find the EFI variables in the /sys hierarchy anyway. I ended up discarding the whole idea of that, since at the end grub2 can make sense of the RAID devices and can find my rootfs just fine.

But there is one thing to note here that is quite important. When I configured my kernel for the first time, I enabled EFI support; then I enabled the hardened features (GRSec and Pax)… and then when I booted it the first time I went to look for the EFI variables without finding them — I didn’t yet understand that booting via PC BIOS wouldn’t have shown them anyway! Turns out that the KERNEXEC option that is so troublesome with virtualisation … is also incompatible with EFI. If you enable that option (which both the server and workstation configurations in hardened-sources enable), the EFI support in the kernel is entirely shut off.

So it appears using EFI is still too soon: we lack tools stable enough, to begin with, and documentation (I had to rely on ArchLinux’s documentation for the most part, but even that is not very clear and refers to black magic tricks; I refrain on writing documentation on the topic myself because I don’t understand all of it at all, even after reading Matthew Garret’s posts on the topic). And it seems to require the kernel to perform unsafe operations (RWX mappings), which is definitely not a good thing.

I start to understand Matthew’s feelings in regard of EFI: it might end up with a more normal boot process once the transition is complete, but between limitations with backward compatibility and no real good feature that is missing – with exception of nvram support – it doesn’t look like it’s something we should be paying much attention to, for now.

Runabout and Intrepid

I’m telling the tale of two new boxes in my network, that are being tested and used all day long.

Intrepid is, as you might have already read, my new MacBook Pro Core2Duo, on which today I started to install Gentoo Linux (AMD64 of course). Unfortunately, well, I got into a little problem when I started installing, because I partitioned using fdisk after running BootCamp, so then GPT and MBR were out of sync, and both OSX and rEFIt weren’t sure of what was going on. I then decided to copy all the content of the installed root and then restarted from scratch.

As I wanted to have a swap partition (just to be safe, or if I ever decide to use software suspend), I then decided to first create the Windows partition with BootCamp, and then fiddle with the partitions directly on the GPT. Thanks to Daniel Ostrow, I then used the same method used on IA64 (GNU parted), and the result is, well, that the partitioning worked! YAI! :)

One problem I have now is that rEFIt shows as bootable also the swap partition, but that can be worked around somehow, I hope (actually I’m just trying to see if by using parted to set the boot flag off on sda3 and then on on sda4 it would help.. no…).

Runabout instead is a new name, a new box… if you can call that a box, it’s a Nokia E61 smartphone, that I bought to get back my 3 number while I’m without the V1075… afterward, it will become my work phone. It was suggested to me by a friend of mine, Andrea, who has similar tastes to mine (although I don’t usually like Nokia as much as he do); I decided to spend money and buy this on his suggestion, as I also got the first payment for my current job, which allowed me to spend a bit more money than usual for once.

Although I was skeptic at the start, after playing with the E61 for a while, especially with the WLAN settings, the email setup, the Web browsing, I had to agree with my friend, it is an interesting device. I also found a PuTTY port that will allow me to connect via SSH to my boxes directly from the cellphone (that’s the final tool for a geek system administrator!), and an IRC client, both Free Software, which is pretty good.

The hostname (Runabout) I reserved some time ago for whenever I had a PDA, and seems the time came, finally :)