On filesystem wiping

I have been talking about BIOS with “facebook friends” today, and Davide (from KDE) asked me if I could send him a copy of my FreeDOS stick, or at least the instructions on how to do so which obviously were on this very blog.

At first I wanted to just send him a compressed copy of the stick itself; then I remembered that I’m not that stupid and I checked out the content of the actual image. The problem is that even if I remove the files as they appear on the filesystem, FAT does not clear them out until the space is needed again; given that my USB stick has been used on almost every imaginable operating system (Windows XP, Vista, 7, OS X Snow Leopard and Lion, and of course Linux in a number of variants I don’t want to start thinking about), and that if it held a hundred megabytes it was very strange (the stick itself is 2GB in size, I didn’t have anything smaller at the time, nor I have now), you can easily tell that the space was rarely reclaimed.

This made it a problem in two very different ways: from one side, the compression couldn’t be very good, as the deleted files were still there and would account for hundreds of megabytes at least, as I’ve been using the same key for almost an year, and I had many boxes to update the BIOS of in the mean time, both mine and customers’; on the other side, some of the data in that image would include serial numbers for both Windows and Avira licenses I installed on customers’ systems (many times I only used one stick to bring the data in and out from a system: that stick).

In my mind the task to perform was clear: wipe out the empty areas of the filesystem so that they are all zero’d: saves on the compression and makes sure that the data is not going to be leaked. So I checked dosfstools and … no luck, there is no utility to do this. I’m not surprised, I don’t have a particular liking for dosfstools: back when it was picked up by Debian, I decided to help out and I worked for a few days on the buildsystem and code to update and improve it … after a quick note of “next release, before we want to just push in the changes that everybody applied”, it seems to simply have been ignored. I didn’t care much about working more on that project so I left it behind.

Googling around I found this project on sourceforge that seemed to be designed to do exactly what I needed. If you look at the page, and you’re used to both FLOSS projects and fake projects, you probably are wondering now if it’s legit or not. I asked that to myself as well, but I gave it a try in a VM, and as far as I can see, there is nothing malicious in it as it is, at least for what vfat is concerned; I explicitly didn’t want to try other filesystems.

Aside — if you are not wondering on the legitimacy of the project I linked, and don’t understand why I would say that… well it’s not really your average FLOSS project that boast “100% free/clean” certifications from websites such as GearDownload and Softpedia… being FLOSS should suffice.

Unfortunately, while the project does exactly what it’s meant to be.. saying that it does so sub-optimally is saying too little. There are quite a few issues with the project that makes me thing I’d be better off rewriting it from scratch than trying to get it fixed – and I’m not referring ot the website, as much as I have an opinion about it, I don’t think it’s my place to judge that, after all my own website is not really updated in … too long a time – starting from the use of a library that, albeit interesting, doesn’t seem to even have a single release, using even the private headers!

Okay I shouldn’t be ungrateful about this, since it did save my day, but it makes me cringe to think I can’t package it for others. And the fact that if you were to try running the tool on a file image of a 2GB USB stick, it would probably take a day or two to complete. What’s the matter? The library it uses, ttf-lib, uses fsync() over the device – a loop device in my case – each time it writes a sector… a 512bytes sector. You can easily tell it’s not the smartest move to do. I was quite bad already that it used write() over 512 sectors one at a time, but if it also synced the device, the tool becomes so slow it’s not funny! Couple of comment markers in the code to avoid the fsync, and the tool takes just a few moments. Of course if it actually waited for a discontinuity before writing to disk, it would probably work much better.

There are a few more issues with these two projects, but at least they seem to have a decent start. I should probably contact the authors and point them in the right direction for them to be packaged in Gentoo and other distributions.

Oh and as a final note, once compressed with xz -9e, the wiped image is just 384KiB, while the unwiped image is 610MiB. Yes that’s almost the side of a CD-ROM, after compression…. the small one can be downloaded here for those of you who want to use it. Sooner or later I’ll have to make a version for 1GiB sticks, I think my mother still has one or two of those I should be able to use, rather than waste a 2GiB stick that way.