USB capturing in 2020

The vast majority of the glucometer devices I reverse the protocol of use USB to connect to a computer. You could say that all of those that I successfully reversed up to now are USB based. Over the years, the way I capture USB packets to figure out a protocol changed significantly, starting from proprietary Windows-based sniffers, and more recently involving my own opensource trace tools. The process evolution was not always intentional — in the case of USBlyzer, it was pretty much dead a few years after I started using it, plus the author refused to document the file format, and by then even my sacrificial laptop was not powerful enough to keep running all the tools I needed.

I feel I’m close to another step on the evolution of my process, and once again it’s not because of me looking to improve the process as much as is the process not working on modern tools. Let me start by explaining what the situation is, because there are two nearly separate issues at play here.

The first issue is that either OpenSuse or the kernel changed the way the debugfs is handled. For those who have not looked at this before, debugfs is what lives in /sys/kernel/debug, and provides the more modern interface for usbmon access; the old method via /dev/usbmonX is deprecated, and Wireshark will not even show up the ability to capture USB packets without debugfs. Previously, I was able to manually change the ownership of the usbmon debugfs paths to my user, and started Wireshark as user to do the capturing, but as of January 2020, it does not seem to be possible to do that anymore: the debugfs mount is only accessible to root.

Using Wireshark as root is generally considered a really bad idea, because it has a huge attack surface, in particular when doing network captures, where the input would literally be to the discretion of external actors. It’s a tinsy bit safer when capturing USB because even when the device is fairly unknown, the traffic is not as controllable, so I would have flinched, but not terribly, to use Wireshark as root — except that I can’t sudo wireshark and have it paint on X. So the remaining alternative is to use tshark, which is a terminal utility that implements the same basics as Wireshark.

Unfortunately here’s the second problem: the last time I ran a lot of captures was when I was working on the Beurer glucometer (which I still haven’t gotten back to, because Linux 5.5 is still unreleased at the time of writing, and that’s the first version that’s not going to go into a reset loop with the device), and I was doing that work from my laptop, and that’s relevant. While the laptop’s keyboard and touchpad are USB, the ports are connected to a different bus internally. Since usbmon interfaces are set by bus, that made it very handy: I only needed to capture on the “ports” bus, and no matter how much and what I typed, it wouldn’t interfere in my captures at all.

You can probably see where this is going: I’m now using a NUC on my desk, with an external keyboard and the Elecom trackball (because I did manage to hurt my wrist while working on the laptop, but that’s a story for another post). And now all the USB 2.0 ports are connected to the same bus. Capturing the bus means getting all the events for keypresses, mouse movements, and so on.

If you have some experience with tcpdump or tshark, you’d think that this is an easy problem to solve: it’s not uncommon having to capture network packets from an SSH connection, which you want to exclude from the capture itself. And the solution for that is to apply a capture filter, such as port not 22.

Unfortunately, it looks like libpcap (which means Wireshark and tshark) does not support capture filters on usbmon. The reasoning provided is that since the capture filters for network are implemented in BPF, there’s no fallback for usbmon that does not have any BPF capabilities in the kernel. I’m not sure about the decision, but there you go. You could also argue that adding BPF to usbmon would be interesting to avoid copying too much data from the kernel, but that’s not something I have particular interest in exploring right now.

So how do you handle this? The suggested option is to capture everything, then use Wireshark to select a subset of packets and save the capture again. This should allow you to have a limited capture that you can share without risking having shared a keylogger off your system. But it also made me think a bit more.

The pcapng format, which Wireshark stores usbmon captures in, is a fairly complicated one, because it can include a lot of different protocol information, and it has multiple typed blocks to store things like hardware interface descriptions. But for USB captures, there’s not much use in the format: not only the Linux and Windows captures (the latter via usbpcap) are different formats altogether, but also the whole interface definition is, as far as I can tell, completely ignored. Instead, if you need a device descriptor, you need to scan the capture for a corresponding request (which usbmon-tools now does.)

I’m now considering just providing a simpler format to store captured data with usbmon-tools, either a simple 1:1 conversion from pcapng, with each packet just size-prefixed, and a tool to filter down the capture on the command line (because honestly, having to load Wireshark to cut down a capture is a pain), or a more complicated format that can store the descriptors separately, and maybe bundle/unbundle them across captures so that you can combine multiple fragments later. If I was in my bubble, I would be using protocol buffers, but that’s not particularly friendly to integrate in a Python module, as far as I can tell. Particularly if you want to be able to use the tools straight out of the git clone.

I guess that since I’m already using construct, I could instead design my own simplistic format. Or maybe I could just bite the bullet, use base64-encoded bytearrays, and write the whole capture session out in JSON.

As I said above, pcapng supports Windows and Linux captures differently: on Linux, the capture format is effectively the wire format of usbmon, while on Linux, it’s the format used by usbpcap. While I have not (yet, at the time of writing) added support to usbmon-tools to load the usbpcap captures, I don’t see why it shouldn’t work out that way. If I do manage to load usbpcap files, though, I would need a custom format to copy these to.

If anyone has a suggestion I’m open to them. One thing that I may try is to use Protocol Buffers but submit the generated source files to parse and serialize the object.

Working with usbmon captures

Two years ago I posted some notes on how I do USB sniffing. I have not really changed much since then, although admittedly I have not spent much time reversing glucometers in that time. But I’m finally biting the bullet and building myself a better setup.

The reasons why I’m looking for a new setup are multiple: first of all, I now have a laptop that is fast enough to run a Windows 10 VM (with Microsoft’s 90 days evaluation version). Second, the proprietary software I used for USB sniffing has not been updated since 2016 — and they still have not published any information about their CBCF format, despite their reason being stated as:

Unfortunately, there is no such documentation and I’m almost sure will
never be. The reason is straightforward – every documented thing
should stay the same indefinitely. That is very restrictive.

At this point, keeping my old Dell Vostro 3750 as a sacrificial machine just for reverse engineering is not worth it anymore. Particularly when you consider that it started being obsoleted by both software (Windows 10 appears to have lost the ability to map network shares easily, and thus provide local-network backups), and hardware (the Western Digital SSD that I installed on it can’t be updated — their update package only works for UEFI boot systems, and while technically that machine is UEFI, it only supports the CSM boot).

When looking at a new option for my setup, I also want to be able to publish more of my scripts and tooling, if nothing else because I would feel more accomplished by knowing that even the side effects of working on these projects can be reused. So this time around I want to focus on all open source tooling, and build as much of the tools to be suitable for me to release as part of my employer’s open source program, which basically means not include any device-specific information within the tooling.

I started looking at Wireshark and its support for protocol dissectors. Unfortunately it looks like USB payloads are a bit more complicated, and dissector support is not great. So once again I’ll be writing a bunch of Python scripts to convert the captured data into some “chatter” files that are suitable for human consumption, at least. So I started to take a closer look at the usbmon documentation (the last time I looked at this was over ten years ago), and see if I can process that data directly.

To be fair, Wireshark does make it much nicer to get the captures out, since the text format usbmon is not particularly easy to parse back into something you can code with — and it is “lossy” when compared with the binary structures. With that, the first thing to focus on is to support the capture format Wireshark generates, which is pcapng, with one particular (out of many) USB capture packet structures. I decided to start my work from that.

What I have right now, is an (incomplete) library that can parse a pcapng capture into objects that are easier to play with in Python. Right now it loads the whole content into memory, which might or might not be a bad limitation, but for now it will do. I guess it would also be nice if I can find a way to integrate this with Colaboratory, which is a tool I only have vague acquaintance with, but would probably be great for this kind of reverse engineering, as it looks a lot like the kind of stuff I’ve been doing by hand. That will probably be left for the future.

The primary target right now is for me to be able to reconstruct the text format of usbmon given the pcapng capture. This would at least tell me that my objects are not losing details in the construction. Unfortunately this is proving harder than expected, because the documentation of usbmon is not particularly clear, starting from the definition of the structure, that mixes sized (u32) and unsized (unsigned int) types. I hope I’ll be able to figure this out and hopefully even send changes to improve the documentation.

As you might have noticed from my Twitter rants, I maintain that the documentation needs an overhaul. From mention of “easy” things, to the fact that the current suggested format (the binary structures) is defined in terms of the text format fields — except the text format is deprecated, and the kernel actually appears to produce the text format based on the binary structures. There are also quite a few things that are not obviously documented in the kernel docs, so you need to read the source code to figure out what they mean. I’ll try rewriting sections of the documentation.

Keep reading the blog to find updates if you have interests in this.

Reverse engineering notes: USB sniffing

You have probably by now read a number of the posts I wrote about reverse engineering glucometers. And while I have complained about the lack of documentation, and maintain a repository of reverse-engineered protocols, I have not really shared the tools I’m using for my work.

The first problem with this is that I’m using a closed-source USB sniffer. I’m not proud of it, but it proved itself useful and worth the price, since the alternative that Microsoft suggests (Message Analyzer) appears not to be working for me, and USBpcap is not supported on Windows 10 (at the time of writing).

Update 2020-10-24: looks like the Message Analyzer was retired in 2019, and is no longer available to download. But USBpcap does now support Windows 10, so there you go.

The native file format of USBlyzer is a CFBF container, but it also includes the ability to export the sniff to text CSV. Originally, I had to fight quite a bit with that, because version 2.1 of the tool produced a mostly unserviceable CSV – in particular the actual packet data was in an unmarked column – but the current version (2.2) is actually decent enough.

I have been working on these CSV, parsing them into a Python structure, and then manipulating them to produce what I refer to as “chatter” files, which is the format you see in my blog posts usually. These are just hexdumps (using the hexdump module) prefixed with a direction of the packet and the packet number, to make it easier to refer to the raw trace. The scripts I’ve used for this translation have evolved quite a bit, from a set of copy-pasted CSV parsing to building a dedicated module for the parsing, to the latest version that separates the idea of reassembling the higher-level protocol packets from actually producing the “chatter” file.

All of these scripts are yet to be released, but I hope to be able to do very soon. I’m also planning to start working a way to access the original CFBF (.ulz) files without requiring the UI to convert to CSV, as that should make my life significantly easier, as it avoids the most boring step in the process, which relies on the Windows UI. Luckily, there is an olefile module that allows you to access these files and the streams in them, I just not have started looking into what the structure of the content is.

I did contact the original developers of the software, and ask them to publish the file format specifications, since they should not contain any special sauce of their business case, but they told me they won’t do that, because it would require them to set in stone the format and never change it again. I told them I disagree on the stance, but it is their decision to make. So instead, I’ll be spending some time figuring this out in the future, feel free to keep reading this blog until I get more details on it.

One of the goals I’d like to have is the ability to convert the USBlyzer traces (either from CSV or their own format) to pcap format so that I can analyze it in Wireshark. This should allow me to make queries such as “Show me all the packets that have the fourth bytes as a value higher than 2” — which is indeed something I have been doing recently, in very convoluted ways. Unfortunately when I started looking on how to do this, I found out two things that made me very unhappy.

The first is that there isn’t a single way to store USB sniff in pcap format, but two. One is the “classic” usbmon one that you can get by building and loading the usbmon module in Linux, and starting Wireshark. The other is the one used by USBpcap to save the information from Windows. These have different formats and (as far as I can tell) there is no easy way to convert between the two. I’m also not sure if the standard Wireshark dissectors apply to that.

The other problem is that the USBpcap format itself is so Windows specific that, despite documenting the whole format on its website, it relies on some constant values coming from the Windows SDK. And you can imagine that depending on the Windows SDK for a Python utility to convert between two file formats is not a great idea. Luckily for me, Wine also has a header with the same constants, but being able to copy that code into the conversion utility means there’s a bit of work to do to make sure I can publish it under the proper license — as I would like to keep every tool’s license the most liberal I can.

You could be asking why on Earth I’m not using virtual machines and just good old standard usbmon to solve my problem. And the answer is two-fold: the laptop I’m using to do the development is just not powerful enough to run Windows 10 on a virtual machine, particularly when having to run Java software such as is the case for Contour, and on the other hand I don’t have enough USB ports to dedicate one to the virtual machine for attaching the devices.

I have an alternative plan, which involves using a BeagleBone Black and USBProxy, but I have not started on that project, among other things because it requires a bit of a complicated setup with USB Ethernet devices and external chargers. So it’s planned, but not sure when I’ll get to that.

Also, speaking of Wireshark, a quick check by using usbmon with my tool and the FreeStyle Libre (because I travel with that, so it’s easier to test on the road), tells me that here is something not quite correct in the HID dissector. In particular it looks like it only recognizes the first report sent by the device as an HID packet, when the response is fragmented into multiple packets. I need to spend some time to track that problem down, and possibly figure out how difficult it would be for me to build further dissectors that can reassemble the higher-level protocol, the way TCP and SCTP sessions can be displayed in Wireshark already.

Sniffing on an Android phone with Wireshark

In my review of the iHealth glucometer I pointed out that I did indeed check if the app talked with the remote service over TLS or not. This was important because if it didn’t, it meant it was sending medical information over plaintext. There are a few other things that can go wrong, they can for instance not validate the certificate provided over TLS, effectively allowing MITM attacks to succeed, but that’s a different story altogether, so I won’t go there for now.

What I wanted to write about is some notes about my experience, if nothing else because it took me a while to get all the fragments ready, and I could not find a single entry anywhere that would explain what the error message I was receiving was about.

First of all, this is about the Wireshark tool, and Android phones, but at the end of the day you’ll find something that would work almost universally with a bunch of caveats. So make sure you get your Wireshark installed, and make sure you never run it as root for your own safety.

Rick suggested to look into the androiddump tool that comes with Wireshark; on Gentoo that requires enabling the right USE flag. This uses the extcap interface to “fetch” the packets to display from a remote source. I like this idea among other things because it splits the displaying/parsing from the capturing. As I’ll show later, this is not the only useful tool using the interface.

There are multiple interfaces that androiddump can capture from; that does include the logcat output, that makes it very useful when you’re debugging an application in realtime, but what I cared about was sniffing the packets from the interfaces on the device itself. This kept failing with the following error:

Error by extcap pipe: ERROR: Broken socket connection.

And no further debugging information available. Googling for a good half hour didn’t bring me anywhere, I even started strace‘ing the process (to the point that Wireshark crashed in a few situations!) until I finally managed to figure out the right -incantation- invokation of the androiddump tool… that had no more information even in verbose mode, but at least it told me what it was trying to do.

The explanation is kind of simple: this set of interfaces is effetively just a matrioska of interfaces. Wireshark calls into extcap, that calls into androiddump, that calls into adb, that calls into tcpdump on the device.

And here is the problem: my device (a Sony Xperia XA from 3 Ireland) has indeed a tcpdump command, but the only thing it does is returning 1 as return value, and that’s it. No error message and not even a help output to figure out if you need to enable somethihng. I have not dug into the phone much more because I was already kind of tired of having to figure out pieces of the puzzle that are not obvious at all, so I looked for alternative approaches.

Depending on the working system you use to set the capture up, you may be able to set up your computer to be an access point, and connect the phone to it. But this is not easy particularly on a laptop with already-oversubscribed USB ports. So I had to look for alternatives.

On the bright side, my router is currently running OpenWRT (with all the warts it has). Which means I have som leeway on the network access already. Googling around would suggest setting up a tee: tell iptables to forward a copy of every single packet coming from or to the phone to another mac address. This is relativel expensive, and no reliable over WiFi networks anyway, beside increasing congestion on an already busy network.

I opted instead to use another tool that is available in extcap: ssh-based packet captures. In Gentoo these require the sshdump and libssh USE flags enabled. With this interface, Wireshark effectively opens a session via SSH to the router, and runs tcpdump on it. It can also use dumpcap or tshark, which are Wireshark-specific tools, and would be significantly more performant, but there is no build for them on OpenWRT so that does not help either.

While this actually increases the amount of traffic over WiFi compared to the tee option, it does so over a reliable channel, and it allows you to apply capture filters, as well as start and stop capture as needed. I ended up going for this option, and the good thing with this is that if you know the hardware addresses of your devices, you can now very easily sniff any of the connected clients just by filtering on that particular address, which opens for interesting discoveries. But that’s for another day.

Merging the hardware data

First off, thanks to Greg who actually pointed me to the right blog post which I referred to in my previous one: this one by Matt Domsch.

Edit 2012/03/21: changed the URL to Matt’s blog post as it was moved.

And again thanks to Greg for letting me get in touch with Matt, and we now are discussing the issue of shared hardware data.

As an experiment, I’ve added to my overlay a live GIT ebuild for hwdata, And to make use of that, I also added ebuils for a few packages to use that data rather than installing their own or using others’ data: pciutils and usbutils (which now don’t install their own data), lshw (which also installed its own data before), hal (rather than using pciutils’ and usbutils’; note that I made hwdata not install the compressed copies at all, it only installs the uncompressed ones at the moment), sysfsutils (see also bug #208651), systemsettings (KDE4, it installed its own usb.ids), and KControl (KDE 3.5, it used usbutils’ usb.ids).

I don’t know at the moment if there are more users of usb.ids and pci.ids, if there are, please tell me and I’ll prepare modified ebuilds for that too so that the experiment can be completed. Also note that all the related ebuilds have KEYWORDS="", so they are never merged in unrequested when you use my overlay, they also got -r99.

There are other kind of data we might like to have in the hwdata package, for instance the oui database, which in my system is duplicated in both Wireshark and lshw, and which can be installed by both KControl and systemsettings when enabling FireWire support.

Now, continuing this way we might actually be able to finally have everything done in one place, and shared across the plane, rathe than having multiple copies of data :)

A misc weekend

So today is the first week without Gentoo work for me. I admit I feel a bit disappointed, I used to do a lot of stuff and now that I can’t do them anymore (for my choice too), I feel like I’m useless. But I’m resisting the temptation to feel my overlay with a lot of ebuilds for the bumps that I would have done myself previously.

First of all, I wish to thank the anonymous “bla” who commented on my previous post about the lack of analysers for the usbmon output, suggesting me Wireshark. I really didn’t know they were working on supporting USB sniffing, and yes that is nice and stops me from deciding to write an analyser myself. Unfortunately, the current weekly snapshot of libpcap and the current subversion trunk of Wireshark does not get along well together: they simply don’t allow me to sniff at all, I get a series of errors instead.

Talking about the Chinese adapter, I found something on their site but I’m not really sure if it’s a specification or simply a description of the driver interface. Nonetheless, I’ve been able to identify a few patterns in the logs I was able to get with usbmon (analysed by hand), and I now know two major type of commands that are sent to the device: a pair of set and get commands used to set and get a 16 bit word out of at least three «registers»… I’m not sure if they are registers at all, they only are three 16-bit words which have another 16-bit word assigned to them, and which is set and get with some vendor-type requests. Unfortunately, I haven’t been able to get a hold of their actual meaning yet.

I’ve also found a sort of “commit” request, or a “get internal status” command, that tells me I haven’t initialised the device correctly yet: when handled by the Windows drivers, this request always return 0x9494, while here it returns values like 0xf4f4 or similar, never the actual 0x9494 word I’m expecting.

I didn’t work on Rust in the past few days but I have been thinking a lot about it, as David is being the first user of it, and submitted me a few reports of features currently missing, which I didn’t think of before, namely support for multiple constructors, and proper support for sub-classes. I haven’t worked on this yet, but I’ll do soon. He also provided me a patch that I merged, but I’ll have to write a series of testcase for that too, so that I can actually be sure that Rust continue behaving as needed.

Today I also decided to look a bit more on xine, as lately I’ve been neglecting it a lot. I’ve tried to cleanup my Wavpack demuxer a bit more, and I just finished a Valgrind (massif) run while playing two wavpack files one after the other. The results are somewhat interesting: xine does not leak memory, as the memory consumption remains constant between the two playbacks, but it also seems to allocate about ten megabytes of memory in _x_video_decode that are not used at all, as I’ve disabled the effect plugins, and so there should be no video at all. About seven more megabytes are wasted in the overlay handling, which also shows that the video output is not correctly disabled while playing audio tracks, with xine-ui at least.

I’ll be running a callgrind now also to test where time is actually lost, it’s good now that I have KCacheGrind displaying properly the callgraph, thanks to Albert Astals Cid (TSDgeos from KDE). One thing that I’m afraid of is that xine does not really have good performances on lossless files because it uses too small buffers, at least by default; I have this doubt because while mp3s play mostly fine – with mad decoder, not FFmpeg’s – it skips a lot for both Flac and WavPack files. This is especially seen when you run it through Valgrind, as the code is obviously slowed down and shows the need for improvements.

If only there was support in ruby-prof for Valgrind-style output to analyse the output with KCacheGrind, I could be improving Rust too. But that will have to wait I’m afraid.

Oh well.