USB Captures Yak Shaving

Months ago I complained about the state of USB captures solutions in 2020. One of the issues it that you can’t easily provide a capture filter to libpcap, because they don’t want to implement user-mode capturing, and Linux does not provide BPF-based filtering for usbmon.

While I do still find it an interesting idea to add BPF filtering there, my kernel-fu is still fairly limited, and I thought I would start with something easier: filtering in userspace with a custom capture program. This also got me a bit more comfortable with the actual capture API, that I have been ignoring for the most part.

As I said before, languages are tools, and I could have tried implementing the tool in a different programming language. But on the other hand, I’m trying to get this done to integrate with the rest of the chatter-extraction tools I released as part of usbmon-tools, so why straying away (too much) from the path? Well, turns out that the usbmon interface is a bit too complicated to implement in pure Python, but Cython makes for a good extended language for it, and it’s something I’m familiar enough with — including for something fairly similar with the SGIO implementation.

It was yet another interesting exercise in Yak Shaving though. Beside the documentation being obtuse at times, and trying to explain the interfaces in their chronological order, with the most useful once last, I found myself partially stumped when I realised that the ioctl() constants you have to use to get any useful information are not available on any userspace header of the Linux kernel! Indeed, it seems the main implementation of usbmon, as part of libpcap, just copies enough of the structures to be able to read the information — and, by the way, does not actually follow the documented process: it sets a value for the buffer size, rather than getting the one that is already set.

I’ve now engaged to make sure that the structures and constants are available to userspace, because at the very least that needs to be addressed properly. I’ve also added unrolled constants for the two ioctl calls that are needed to set the capture up, which keep the amount of copy-paste from kernel headers to a minimum.

While I have committed a monitoring tool that allows printing the output of packets, this is far from the end. It only outputs text format right now, it doesn’t do URB re-tagging, and it only does naïve filters. My next few steps will likely involve getting python-pcapng write support merged in, and start writing pcapng file with the new tool. Then I can start looking at a more common, more interesting filtering set.

Once the capturing is properly taken care of, I have two main needs that I need to address, in the toolset: one is to be able to unpack PL2303 serial protocols — because the programmer that is failing me is using PL2303 and I would like to see how the conversation with the bootloader is going. While the stcgal tool has debug output, having a general chatter printer feels like it would be useful in the future. The other is USB Mass Storage parsing and inspecting, because I need it for the beurer, but also because I would like to turn some of my past reverse engineering blog posts into a talk, and I would like to have some more examples of how the tools make it easier to find the meat of the information.

So yeah that’s where a Sunday went for me…

USB capturing in 2020

The vast majority of the glucometer devices I reverse the protocol of use USB to connect to a computer. You could say that all of those that I successfully reversed up to now are USB based. Over the years, the way I capture USB packets to figure out a protocol changed significantly, starting from proprietary Windows-based sniffers, and more recently involving my own opensource trace tools. The process evolution was not always intentional — in the case of USBlyzer, it was pretty much dead a few years after I started using it, plus the author refused to document the file format, and by then even my sacrificial laptop was not powerful enough to keep running all the tools I needed.

I feel I’m close to another step on the evolution of my process, and once again it’s not because of me looking to improve the process as much as is the process not working on modern tools. Let me start by explaining what the situation is, because there are two nearly separate issues at play here.

The first issue is that either OpenSuse or the kernel changed the way the debugfs is handled. For those who have not looked at this before, debugfs is what lives in /sys/kernel/debug, and provides the more modern interface for usbmon access; the old method via /dev/usbmonX is deprecated, and Wireshark will not even show up the ability to capture USB packets without debugfs. Previously, I was able to manually change the ownership of the usbmon debugfs paths to my user, and started Wireshark as user to do the capturing, but as of January 2020, it does not seem to be possible to do that anymore: the debugfs mount is only accessible to root.

Using Wireshark as root is generally considered a really bad idea, because it has a huge attack surface, in particular when doing network captures, where the input would literally be to the discretion of external actors. It’s a tinsy bit safer when capturing USB because even when the device is fairly unknown, the traffic is not as controllable, so I would have flinched, but not terribly, to use Wireshark as root — except that I can’t sudo wireshark and have it paint on X. So the remaining alternative is to use tshark, which is a terminal utility that implements the same basics as Wireshark.

Unfortunately here’s the second problem: the last time I ran a lot of captures was when I was working on the Beurer glucometer (which I still haven’t gotten back to, because Linux 5.5 is still unreleased at the time of writing, and that’s the first version that’s not going to go into a reset loop with the device), and I was doing that work from my laptop, and that’s relevant. While the laptop’s keyboard and touchpad are USB, the ports are connected to a different bus internally. Since usbmon interfaces are set by bus, that made it very handy: I only needed to capture on the “ports” bus, and no matter how much and what I typed, it wouldn’t interfere in my captures at all.

You can probably see where this is going: I’m now using a NUC on my desk, with an external keyboard and the Elecom trackball (because I did manage to hurt my wrist while working on the laptop, but that’s a story for another post). And now all the USB 2.0 ports are connected to the same bus. Capturing the bus means getting all the events for keypresses, mouse movements, and so on.

If you have some experience with tcpdump or tshark, you’d think that this is an easy problem to solve: it’s not uncommon having to capture network packets from an SSH connection, which you want to exclude from the capture itself. And the solution for that is to apply a capture filter, such as port not 22.

Unfortunately, it looks like libpcap (which means Wireshark and tshark) does not support capture filters on usbmon. The reasoning provided is that since the capture filters for network are implemented in BPF, there’s no fallback for usbmon that does not have any BPF capabilities in the kernel. I’m not sure about the decision, but there you go. You could also argue that adding BPF to usbmon would be interesting to avoid copying too much data from the kernel, but that’s not something I have particular interest in exploring right now.

So how do you handle this? The suggested option is to capture everything, then use Wireshark to select a subset of packets and save the capture again. This should allow you to have a limited capture that you can share without risking having shared a keylogger off your system. But it also made me think a bit more.

The pcapng format, which Wireshark stores usbmon captures in, is a fairly complicated one, because it can include a lot of different protocol information, and it has multiple typed blocks to store things like hardware interface descriptions. But for USB captures, there’s not much use in the format: not only the Linux and Windows captures (the latter via usbpcap) are different formats altogether, but also the whole interface definition is, as far as I can tell, completely ignored. Instead, if you need a device descriptor, you need to scan the capture for a corresponding request (which usbmon-tools now does.)

I’m now considering just providing a simpler format to store captured data with usbmon-tools, either a simple 1:1 conversion from pcapng, with each packet just size-prefixed, and a tool to filter down the capture on the command line (because honestly, having to load Wireshark to cut down a capture is a pain), or a more complicated format that can store the descriptors separately, and maybe bundle/unbundle them across captures so that you can combine multiple fragments later. If I was in my bubble, I would be using protocol buffers, but that’s not particularly friendly to integrate in a Python module, as far as I can tell. Particularly if you want to be able to use the tools straight out of the git clone.

I guess that since I’m already using construct, I could instead design my own simplistic format. Or maybe I could just bite the bullet, use base64-encoded bytearrays, and write the whole capture session out in JSON.

As I said above, pcapng supports Windows and Linux captures differently: on Linux, the capture format is effectively the wire format of usbmon, while on Linux, it’s the format used by usbpcap. While I have not (yet, at the time of writing) added support to usbmon-tools to load the usbpcap captures, I don’t see why it shouldn’t work out that way. If I do manage to load usbpcap files, though, I would need a custom format to copy these to.

If anyone has a suggestion I’m open to them. One thing that I may try is to use Protocol Buffers but submit the generated source files to parse and serialize the object.

Introducing usbmon-tools

A couple of weeks ago I wrote some notes about my work in progress to implement usbmon captures handling code, and pre-announced I was going to publish more of my extraction/inspection scripts.

The good news is that the project is now released, and you can find it on GitHub as usbmon-tools with an Apache 2.0 license, and open to contributions (with a CLA, sorry about that part). This is the first open source project I release using my employer’s releasing process (for other projects, I used the IARC process instead), and I have to say I’m fairly pleased with the results.

This blog post is meant mostly as a way to explain what’s going on my head regarding this project, with the hope that contributors can help it become reality. Or that they can contribute other ideas to it, even when they are not part of my particular plans.

I want to start with a consideration on the choice of language. usbmon-tools is written in Python 3. And in particular it is restricted to Python 3.7, because I wanted to have access to type annotations, which I found extremely addictive at work. I even set up Travis CI to run mypy as part of the integration tests for the repository.

For other projects I tend to be more conservative, and wait for Debian stable to have a certain version before requiring that as a minimum, but as this is a toolset for developers primarily, I’m going to expect its public to be able to deal with Python 3.7 as the requirement. This version was released nearly a year ago, and that should be plenty of time for people to have one at hand.

As for what the project should achieve in my view, is an easy way for developers to dissect an USB snooping trace. I started by building a simplistic tool that recreates a text format trace from the pcapng file, based on the official documentation of usbmon in the kernel (I have some patches to improve on that, too, but that probably will become a post in by itself next week). It’s missing isochronous support, and it’s not totally tested, but it at least gave me a few important insight on the format itself, including the big caveat that the “id” (or tag) of the URBs is not unique.

Indeed, I think that alone is one of the most important pieces of the puzzle in the library: in addition to parsing the pcapng file itself, the library can re-tag the events so that they get a real unique identifier (UUID), making it significantly easier to analyze the traces.

My next steps on the project are to write a more generic tool to convert a USB capture into what I call my “chatter format” (similar to the one I used to discuss serial protocols), and a more specific one that converts HID traces (because HID is a more defined protocol, and we can go a level deeper in exposing this into a human-readable source). I’m also considering if it would be within reach to provide the tool a HID descriptor blob, parse it and have it used to parse the HID traffic based on it. It would make some debugging particularly easier, for instance the stuff I did when I was fixing the ELECOM DEFT trackball.

I would also love to be able to play with a trace in a more interactive manner, for instance by loading this into Jupyter notebook, so that I could try parsing the blobs interactively, but unless someone with more experience with those contributes the code, I don’t expect I’ll have much time for it.

Pull requests are more than welcome!

Working with usbmon captures

Two years ago I posted some notes on how I do USB sniffing. I have not really changed much since then, although admittedly I have not spent much time reversing glucometers in that time. But I’m finally biting the bullet and building myself a better setup.

The reasons why I’m looking for a new setup are multiple: first of all, I now have a laptop that is fast enough to run a Windows 10 VM (with Microsoft’s 90 days evaluation version). Second, the proprietary software I used for USB sniffing has not been updated since 2016 — and they still have not published any information about their CBCF format, despite their reason being stated as:

Unfortunately, there is no such documentation and I’m almost sure will
never be. The reason is straightforward – every documented thing
should stay the same indefinitely. That is very restrictive.

At this point, keeping my old Dell Vostro 3750 as a sacrificial machine just for reverse engineering is not worth it anymore. Particularly when you consider that it started being obsoleted by both software (Windows 10 appears to have lost the ability to map network shares easily, and thus provide local-network backups), and hardware (the Western Digital SSD that I installed on it can’t be updated — their update package only works for UEFI boot systems, and while technically that machine is UEFI, it only supports the CSM boot).

When looking at a new option for my setup, I also want to be able to publish more of my scripts and tooling, if nothing else because I would feel more accomplished by knowing that even the side effects of working on these projects can be reused. So this time around I want to focus on all open source tooling, and build as much of the tools to be suitable for me to release as part of my employer’s open source program, which basically means not include any device-specific information within the tooling.

I started looking at Wireshark and its support for protocol dissectors. Unfortunately it looks like USB payloads are a bit more complicated, and dissector support is not great. So once again I’ll be writing a bunch of Python scripts to convert the captured data into some “chatter” files that are suitable for human consumption, at least. So I started to take a closer look at the usbmon documentation (the last time I looked at this was over ten years ago), and see if I can process that data directly.

To be fair, Wireshark does make it much nicer to get the captures out, since the text format usbmon is not particularly easy to parse back into something you can code with — and it is “lossy” when compared with the binary structures. With that, the first thing to focus on is to support the capture format Wireshark generates, which is pcapng, with one particular (out of many) USB capture packet structures. I decided to start my work from that.

What I have right now, is an (incomplete) library that can parse a pcapng capture into objects that are easier to play with in Python. Right now it loads the whole content into memory, which might or might not be a bad limitation, but for now it will do. I guess it would also be nice if I can find a way to integrate this with Colaboratory, which is a tool I only have vague acquaintance with, but would probably be great for this kind of reverse engineering, as it looks a lot like the kind of stuff I’ve been doing by hand. That will probably be left for the future.

The primary target right now is for me to be able to reconstruct the text format of usbmon given the pcapng capture. This would at least tell me that my objects are not losing details in the construction. Unfortunately this is proving harder than expected, because the documentation of usbmon is not particularly clear, starting from the definition of the structure, that mixes sized (u32) and unsized (unsigned int) types. I hope I’ll be able to figure this out and hopefully even send changes to improve the documentation.

As you might have noticed from my Twitter rants, I maintain that the documentation needs an overhaul. From mention of “easy” things, to the fact that the current suggested format (the binary structures) is defined in terms of the text format fields — except the text format is deprecated, and the kernel actually appears to produce the text format based on the binary structures. There are also quite a few things that are not obviously documented in the kernel docs, so you need to read the source code to figure out what they mean. I’ll try rewriting sections of the documentation.

Keep reading the blog to find updates if you have interests in this.

CP2110 Update for 2019

The last time I wrote about the CP2110 adapter was nearly a year ago, and because I have had a lot to keep me busy since, I have not been making much progress. But today I had some spare cycles and decided to take a deeper look starting from scratch again.

What I should have done properly since then would have been procuring myself a new serial dongle, as I was not (and still am) not entirely convinced about the quality of the CH341 adapter I’m using. I think I used that serial adapter successfully before, but maybe I didn’t and I’ve been fighting with ghosts ever since. This counts double as, silly me, I didn’t re-read my own post when I resumed working on this, and been scratching my head at nearly exactly the same problems as last time.

I have some updates first. The first of which is that I have some rough-edged code out there on this GitHub branch. It does not really have all the features it should, but it at least let me test the basic implementation. It also does not actually let you select which device to open — it looks for the device with the same USB IDs as I have, and that might not work at all for you. I’ll be happy to accept pull requests to fix more of the details, if anyone happen to need something like this too — once it’s actually in a state where it can be merged, I’ll be doing a squash commit and send a pull request upstream with the final working code.

The second is that while fighting with this, and venting on Twitter, Saleae themselves put me on the right path: when I said that Logic failed to decode the CP2110→CH341 conversation at 5V but worked when they were set at 3.3V, they pointed me at the documentation of threshold voltage, which turned out to be a very good lead.

Indeed, when connecting the CP2110 at 5V alone, Logic reports a high of 5.121V, and a low of ~-0.12V. When I tried to connect it with the CH341 through the breadboard full of connections, Logic reports a low of nearly 3V! And as far as I can tell, the ground is correctly wired together between the two serial adapters — they are even connected to the same USB HUB. I also don’t think the problem is with the wiring of the breadboard, because the behaviour is identical when just wiring the two adapters together.

So my next step has been setting up the BeagleBone Black I bought a couple of years ago and shelved into a box. I should have done that last year, and I would probably have been very close to have this working in the first place. After setting this up (which is much easier than it sounds), and figuring out from the BeagleBoard Wiki the pinout (and a bit of guesswork on the voltage) of its debug serial port, I could confirm the data was being sent to the CP2110 right — but it got all mangled on print.

The answer was that the HID buffered reads are… complicated. So instead of deriving most of the structure from the POSIX serial implementation, I lifted it from the RFC2217 driver, that uses a background thread to loop the reads. This finally allowed me to use the pySerial miniterm tool to log in and even dmesg(!) the BBB over the CP2110 adapter, which I consider a win.

Tomorrow I’ll try polishing the implementation to the point where I can send a pull request. And then I can actually set up to look back into the glucometer using it. Because I had an actual target when I started working on this, and was not just trying to get this to work for the sake of it.

Updates on Silicon Labs CP2110

One month ago I started the yak shave of supporting the Silicon Labs CP2110 with a fully opensource stack, that I can even re-use for glucometerutils.

The first step was deciding how to implement this. While the device itself supports quite a wide range of interfaces, including a GPIO one, I decided that since I’m only going to be able to test and use practically the serial interface, I would at least start with just that. So you’ll probably see the first output as a module for pyserial that implements access to CP2110 devices.

The second step was to find an easy way to test this in a more generic way. Thankfully, Martin Holzhauer, who commented on the original post, linked to an adapter by MakerSpot that uses that chip (the link to the product was lost in the migration to WordPress, sigh), which I then ordered and received a number of weeks later, since it had to come to the US and clear customs through Amazon.

All of this was the easy part, the next part was actually implementing enough of the protocol described in the specification, so that I could actually send and receive data — and that also made it clear that despite the protocol being documented, it’s not as obvious as it might sound — for instance, the specification says that the reports 0x01 to 0x3F are used to send and receive data, but it does not say why there are so many reports… except that it turns out they are actually used to specify the length of the buffer: if you send two bytes, you’ll have to use the 0x02 report, for ten bytes 0x0A, and so on, until the maximum of 63 bytes as 0x3F. This became very clear when I tried sending a long string and the output was impossible to decode.

Speaking of decoding, my original intention was to just loop together the CP2110 device with a CH341 I bought a few years ago, and have them loop data among each other to validate that they work. Somehow this plan failed: I can get data from the CH341 into the CP2110 and it decodes fine (using picocom for the CH341, and Silicon Labs own binary for the CP2110), but I can’t seem to get the CH341 to pick up the data sent through the CP2110. I thought it was a bad adapter, but then I connected the output to my Saleae Logic16 and it showed the data fine, so… no idea.

The current status is:

I know the CH341 sends out a good signal;
I know the CP2110 can receive a good signal from the CH341, with the Silicon Labs software;
I know the CP2110 can send a good signal to the Saleae Logic16, both with the Silicon Labs software and my tiny script;
I can’t get the CH341 to receive data from the CP2110.

Right now the state is still very much up in the air, and since I’ll be travelling quite a bit without a chance to bring with me the devices, there probably won’t be any news about this for another month or two.

Oh and before I forget, Rich Felker gave me another interesting idea: CUSE (Character Devices in User Space) is a kernel-supported way to “emulate” in user space devices that would usually be implemented in the kernel. And that would be another perfect application for this: if you just need to use a CP2110 as an adapter for something that needs to speak with a serial port, then you can just have a userspace daemon that implements CUSE, and provide a ttyUSB-compatible device, while not requiring short-circuiting the HID and USB-Serial subsystems.

Reverse Engineering and Serial Adapter Protocols

In the comments to my latest post on the Silicon Labs CP2110, the first comment got me more than a bit upset because it was effectively trying to mansplain to me how a serial adapter (or more properly an USB-to-UART adapter) works. Then I realized there’s one thing I can do better than complain and that is providing even more information on this for the next person who might need them. Because I wish I knew half of what I know now back when I tried to write the driver for ch314.

So first of all, what are we talking about? UART is a very wide definition for any interface that implements serial communication that can be used to transmit between a host and a device. The word “serial port” probably bring different ideas to mind depending on the background of a given person, whether it is mice and modems connected to PCs, or servers’ serial terminals, or programming interfaces for microcontrollers. For the most part, people in the “consumer world” think of serial as RS-232 but people who have experience with complex automation systems, whether it is home, industrial, or vehicle automation, have RS-485 as their main reference. None of that actually matters, since these standards mostly deal with electrical or mechanical standards.

As physical serial ports on computer stopped appearing many years ago, most of the users moved to USB adapters. These adapters are all different between each other and that’s why there’s around 40KSLOC of serial adapters drivers in the Linux kernel (according to David’s SLOCCount). And that’s without counting the remaining 1.5KSLOC for implementing CDC ACM which is the supposedly-standard approach to serial adapters.

Usually the adapters are placed either directly on the “gadget” that needs to be connected, which expose a USB connector, or on a cable used to connect to it, in which case the device usually has a TRS or similar connectors. The TRS-based serial cables appeared to become more and more popular thanks to osmocom as they are relatively inexpensive to build, both as cables and as connectors onto custom boards.

Serial interface endpoints in operating systems (/dev/tty{S,USB,ACM}* on Linux, COM* on Windows, and so on) do not only transfer data between host and device, but also provides configuration of parameters such as transmission rate and “symbol shape” — you may or may not have heard references to something like “9600n8” which is a common way to express the transmission protocol of a serial interface: 9600 symbols per second (“baud rate”), no parity, 8-bit per symbol. You can call these “out of band” parameters, as they are transmitted to the UART interface, but not to the device itself, and they are the crux of the matter of interacting with these USB-to-UART adapters.

I already wrote notes about USB sniffing, so I won’t go too much into detail there, but most of the time when you’re trying to figure out what the control software sends to a device, you start by taking a USB trace, which gives you a list of USB Request Blocks (effectively, transmission packets), and you get to figure out what’s going on there.

For those devices that use USB-to-UART adapters and actually use the OS-provided serial interface (that is, COM* under Windows, where most of the control software has to run), you could use specialised software to only intercept the communication on that interface… but I don’t know of any such modern software, while there are at least a few well-defined interface to intercept USB communication. And that would not work for software that access the USB adapter directly from userspace, which is always the case for Silicon Labs CP2110, but is also the case for some of the FTDI devices.

To be fair, for those devices that use TRS, I actually have considered just intercepting the serial protocol using the Saleae Logic Pro, but beside being overkill, it’s actually just a tiny fraction of the devices that can be intercepted that way — as the more modern ones just include the USB-to-UART chip straight onto the device, which is also the case for the meter using the CP2110 I referenced earlier.

Within the request blocks you’ll have not just the serial communication, but also all the related out-of-band information, which is usually terminated on the adapter/controller rather than being forwarded onto the device. The amount of information changes widely between adapters. Out of those I have had direct experience, I found one (TI3420) that requires a full firmware upload before it would start working, which means recording everything from the moment you plug in the device provides a lot more noise than you would expect. But most of those I dealt with had very simple interfaces, using Control transfers for out-of-band configuration, and Bulk or Interrupt¹ transfers for transmitting the actual serial interface.

With these simpler interfaces, my “analysis” scripts (if you allow me the term, I don’t think they are that complicated) can produce a “chatter” file quite easily by ignoring the whole out of band configuration. Then I can analyse those chatter files to figure out the device’s actual protocol, and for the most part it’s a matter of trying between one and five combinations of transmission protocol to figure out the right one to speak to the device — in glucometerutils I have two drivers using 9600n8 and two drivers using 38400n8. In some cases, such as the TI3420 one, I actually had to figure out the configuration packet (thanks to the Linux kernel driver and the datasheet) to figure out that it was using 19200n8 instead.

But again, for those, the “decoding” is just a matter to filtering away part of the transmission to keep the useful parts. For others it’s not as easy.

0029 <<<< 00000000: 30 12                                             0.

0031 <<<< 00000000: 05 00                                             ..

0033 <<<< 00000000: 2A 03                                             *.

0035 <<<< 00000000: 42 00                                             B.

0037 <<<< 00000000: 61 00                                             a.

0039 <<<< 00000000: 79 00                                             y.

0041 <<<< 00000000: 65 00                                             e.

0043 <<<< 00000000: 72 00                                             r.

This is an excerpt from the chatter file of a session with my Contour glucometer. What happens here is that instead of buffering the transmission and sending a single request block with a whole string, the adapter (FTDI FT232RL) sends short burts, probably to reduce latency and keep a more accurate serial protocol (which is important for device that need accurate timing, for instance some in-chip programming interfaces). This would be also easy to recompose, except it also comes with

0927 <<<< 00000000: 01 60                                             .`

0929 <<<< 00000000: 01 60                                             .`

0931 <<<< 00000000: 01 60                                             .`

which I’m somehow sceptical they come from the device itself. I have not paid enough attention yet to figure out from the kernel driver whether this data is marked as coming from the device or is some kind of keepalive or synchronisation primitive of the adapter.

In the case of the CP2110, the first session I captured starts with:

0003 <<<< 00000000: 46 0A 02                                          F..

0004 >>>> 00000000: 41 01                                             A.

0006 >>>> 00000000: 50 00 00 4B 00 00 00 03  00                       P..K.....

0008 >>>> 00000000: 01 51                                             .Q

0010 >>>> 00000000: 01 22                                             ."

0012 >>>> 00000000: 01 00                                             ..

0014 >>>> 00000000: 01 00                                             ..

0016 >>>> 00000000: 01 00                                             ..

0018 >>>> 00000000: 01 00                                             ..

and I can definitely tell you that the first three URBs are not sent to the device at all. That’s because HID (the higher-level protocol that CP2110 uses on top of USB) uses the first byte of the block to identify the “report” it sends or receives. Checking these against AN434 give me a hint of what’s going on:

report 0x46 is “Get Version Information” — CP2110 always returns 0x0A as first byte, followed by a device version, which is unspecified; probably only used to confirm that the device is right, and possibly debugging purposes;
report 0x41 is “Get/Set UART Enabled” — 0x01 just means “turn on the UART”;
report 0x50 is “Get/Set UART Config” — and this is a bit more complex to parse: the first four bytes (0x00004b00) define the baud rate, which is 19200 symbols per second; then follows one byte for parity (0x00, no parity), one for flow control (0x00, no flow control), one for the number of data bits (0x03, 8-bit per symbol), and finally one for the stop bit (0x00, short stop bit); that’s a long way to say that this is configured as 19200n8.
report 0x01 is the actual data transfer, which means the transmission to the device starts with 0x51 0x22 0x00 0x00 0x00 0x00.

This means that I need a smarter analysis script that understands this protocol (which may be as simple as just ignoring anything that does not use report 0x01) to figure out what the control software is sending.

And at the same time, it needs code to know how “talk serial” to this device. Usually the out-of-bad configuration is done by a kernel driver: you ioctl() the serial device to the transmission protocol you need, the driver sends the right request block to the USB endpoint. But in the case of the CP2110 device, there’s no kernel driver implementing this, at least per Silicon Labs design: since HID devices are usually exposed to userland, and in particular to non-privileged applications, sending and receiving the reports can be done directly from the apps. So indeed there is no COM* device exposed on Windows, even with the drivers installed.

Could someone (me?) write a Linux kernel driver that expose CP2110 as a serial, rather than HID, device? Sure. It would require fiddling around with the HID subsystem a bit to have it ignore the original device, and that means it’ll probably break any application built with Silicon Labs’ own development kit, unless someone has a suggestion on how to have both interfaces available at the same time, while it would allow accessing those devices without special userland code. But I think I’ll stick with the idea of providing a Free and Open Source implementation of the protocol, for Python. ~~And maybe add support for it to pyserial to make it easier for me to use it.~~ pyserial support CP2110 devices since commit 8b24cbb6131a97a3e91aabe5299c834a75ae8964.

¹ l these terms make more sense if you have at least a bit of knowledge of USB works behind the scene, but I don’t want to delve too much into that.

Yak Shaving: Silicon Labs CP2110 and Linux

One of my favourite passtimes in the past years has been reverse engineering glucometers for the sake of writing an utility package to export data to it. Sometimes, in the quest of just getting data out of a meter I end up embarking in yak shaves that are particularly bothersome, as they are useful only for me and no one else.

One of these yak shaves might be more useful to others, but it will have to be seen. I got my hands on a new meter, which I will review later on. This meter has software for Windows to download the readings, so it’s a good target for reverse engineering. What surprised me, though, was that once I connected the device to my Linux laptop first, it came up as an HID device, described as an “USB HID to UART adapter”: the device uses a CP2110 adapter chip by Silicon Labs, and it’s the first time I saw this particular chip (or even class of chip) in my life.

Effectively, this device piggybacks the HID interface, which allows vendor-specified protocols to be implemented in user space without needing in-kernel drivers. I’m not sure if I should be impressed by the cleverness or disgusted by the workaround. In either case, it means that you end up with a stacked protocol design: the glucometer protocol itself is serial-based, implemented on top of a serial-like software interface, which converts it to the CP2110 protocol, which is encapsulated into HID packets, which are then sent over USB…

The good thing is that, as the datasheet reports, the protocol is available: “Open access to interface specification”. And indeed in the download page for the device, there’s a big archive of just-about-everything, including a number of precompiled binary libraries and a bunch of documents, among which figures AN434, which describe the full interface of the device. Source code is also available, but having spot checked it, it appears it has no license specification and as such is to be considered proprietary, and possibly virulent.

So now I’m warming up to the idea of doing a bit more of yak shaving and for once trying not to just help myself. I need to understand this protocol for two purposes: one is obviously having the ability to communicate with the meter that uses that chip; the other is being able to understand what the software is telling the device and vice-versa.

This means I need to have generators for the host side, but parsers for both. Luckily, construct should make that part relatively painless, and make it very easy to write (if not maintain, given the amount of API breakages) such a parser/generator library. And of course this has to be in Python because that’s the language my utility is written in.

The other thing that I realized as I was toying with the idea of writing this is that, done right, it can be used together with facedancer, to implement the gadget side purely in Python. Which sounds like a fun project for those of us into that kind of thing.

But since this time this is going to be something more widely useful, and not restricted to my glucometer work, I’m now looking to release this using a different process, as that would allow me to respond to issues and codereviews from my office as well as during the (relatively little) spare time I have at home. So expect this to take quite a bit longer to be released.

At the end of the day, what I hope to have is an Apache 2 licensed Python library that can parse both host-to-controller and controller-to-host packets, and also implement it well enough on the client side (based on the hidapi library, likely) so that I can just import the module and use it for a new driver. Bonus points if I can sue this to implement a test fake framework to implement the tests for the glucometer.

In all of this, I want to make sure to thank Silicon Labs for releasing the specification of the protocol. It’s not always that you can just google up the device name to find the relevant protocol documentation, and even when you do it’s hard to figure out if it’s enough to implement a driver. The fact that this is possible surprised me pleasantly. On the other hand I wish they actually released their code with a license attached, and possibly a widely-usable one such as MIT or Apache 2, to allow users to use the code directly. But I can see why that wouldn’t be particularly high in their requirements.

Let’s just hope this time around I can do something for even more people.

Reverse engineering notes: USB sniffing

You have probably by now read a number of the posts I wrote about reverse engineering glucometers. And while I have complained about the lack of documentation, and maintain a repository of reverse-engineered protocols, I have not really shared the tools I’m using for my work.

The first problem with this is that I’m using a closed-source USB sniffer. I’m not proud of it, but it proved itself useful and worth the price, since the alternative that Microsoft suggests (Message Analyzer) appears not to be working for me, and USBpcap is not supported on Windows 10 ~~(at the time of writing)~~.

Update 2020-10-24: looks like the Message Analyzer was retired in 2019, and is no longer available to download. But USBpcap does now support Windows 10, so there you go.

The native file format of USBlyzer is a CFBF container, but it also includes the ability to export the sniff to text CSV. Originally, I had to fight quite a bit with that, because version 2.1 of the tool produced a mostly unserviceable CSV – in particular the actual packet data was in an unmarked column – but the current version (2.2) is actually decent enough.

I have been working on these CSV, parsing them into a Python structure, and then manipulating them to produce what I refer to as “chatter” files, which is the format you see in my blog posts usually. These are just hexdumps (using the hexdump module) prefixed with a direction of the packet and the packet number, to make it easier to refer to the raw trace. The scripts I’ve used for this translation have evolved quite a bit, from a set of copy-pasted CSV parsing to building a dedicated module for the parsing, to the latest version that separates the idea of reassembling the higher-level protocol packets from actually producing the “chatter” file.

All of these scripts are yet to be released, but I hope to be able to do very soon. I’m also planning to start working a way to access the original CFBF (.ulz) files without requiring the UI to convert to CSV, as that should make my life significantly easier, as it avoids the most boring step in the process, which relies on the Windows UI. Luckily, there is an olefile module that allows you to access these files and the streams in them, I just not have started looking into what the structure of the content is.

I did contact the original developers of the software, and ask them to publish the file format specifications, since they should not contain any special sauce of their business case, but they told me they won’t do that, because it would require them to set in stone the format and never change it again. I told them I disagree on the stance, but it is their decision to make. So instead, I’ll be spending some time figuring this out in the future, feel free to keep reading this blog until I get more details on it.

One of the goals I’d like to have is the ability to convert the USBlyzer traces (either from CSV or their own format) to pcap format so that I can analyze it in Wireshark. This should allow me to make queries such as “Show me all the packets that have the fourth bytes as a value higher than 2” — which is indeed something I have been doing recently, in very convoluted ways. Unfortunately when I started looking on how to do this, I found out two things that made me very unhappy.

The first is that there isn’t a single way to store USB sniff in pcap format, but two. One is the “classic” usbmon one that you can get by building and loading the usbmon module in Linux, and starting Wireshark. The other is the one used by USBpcap to save the information from Windows. These have different formats and (as far as I can tell) there is no easy way to convert between the two. I’m also not sure if the standard Wireshark dissectors apply to that.

The other problem is that the USBpcap format itself is so Windows specific that, despite documenting the whole format on its website, it relies on some constant values coming from the Windows SDK. And you can imagine that depending on the Windows SDK for a Python utility to convert between two file formats is not a great idea. Luckily for me, Wine also has a header with the same constants, but being able to copy that code into the conversion utility means there’s a bit of work to do to make sure I can publish it under the proper license — as I would like to keep every tool’s license the most liberal I can.

You could be asking why on Earth I’m not using virtual machines and just good old standard usbmon to solve my problem. And the answer is two-fold: the laptop I’m using to do the development is just not powerful enough to run Windows 10 on a virtual machine, particularly when having to run Java software such as is the case for Contour, and on the other hand I don’t have enough USB ports to dedicate one to the virtual machine for attaching the devices.

I have an alternative plan, which involves using a BeagleBone Black and USBProxy, but I have not started on that project, among other things because it requires a bit of a complicated setup with USB Ethernet devices and external chargers. So it’s planned, but not sure when I’ll get to that.

Also, speaking of Wireshark, a quick check by using usbmon with my tool and the FreeStyle Libre (because I travel with that, so it’s easier to test on the road), tells me that here is something not quite correct in the HID dissector. In particular it looks like it only recognizes the first report sent by the device as an HID packet, when the response is fragmented into multiple packets. I need to spend some time to track that problem down, and possibly figure out how difficult it would be for me to build further dissectors that can reassemble the higher-level protocol, the way TCP and SCTP sessions can be displayed in Wireshark already.

Free Idea: a QEMU Facedancer fuzzer

This post is part of a series of free ideas that I’m posting on my blog in the hope that someone with more time can implement. It’s effectively a very sketched proposal that comes with no design attached, but if you have time you would like to spend learning something new, but no idea what to do, it may be a good fit for you.

Update (2017-06-19): see last paragraph.

You may already be familiar with the Facedancer, the USB fuzzer originally designed and developed by Travis Goodspeed of PoC||GTFO fame. If you’re not, in no so many words, it’s a board (and framework) that allows you to simulate the behaviour of any USB device. It works thanks to a Python framework which supports a few other board and could – theoretically – be expanded to support any device with gadgetfs support (such as the BeagleBone Black I have at home, but I digress).

I have found about this through Micah’s videos, and I have been thinking for a while to spend some time to do the extension to gadgetfs so I can use it to simulate glucometers with their original software — as in particular it should allow me to figure out which value represent what, by changing what is reported to the software and see how it behaves. While I have had no time to do that yet, this is anyway a topic for a different post.

The free idea I want to give instead is to integrate, somehow, the Facedancer framework with QEMU, so that you can run the code behind a Facedancer device as if it was connected to a QEMU guest, without having to use any hardware at all. A “Virtualdancer”, which would not only obviate the need for hardware in the development phase (if the proof of concept of a facedancer were to require an off-site usage) but also would integrate more easily into fuzzing projects such as Bochspwn or TriforceAFL.

In particular, I have interest not only in writing simulated glucometers for debugging purposes (although a testsuite that requires qemu and a simulated device may be a bit overkill), but also in simulating HID devices. You may remember that recently I had to fix my ELECOM trackball, and this is not the first time I have to deal with broken HID descriptor. I have spent some more time looking into the Linux HID subsystem and I’m trying to figure out if I can make some simplifications here and there (again, topic for another time), so having a way to simulate an HID device with strange behaviour and see if my changes fix it or not would be extremely beneficial.

Speaking of HID, and report descriptors in particular, Alex Ionescu (of ReactOS fame) at REcon pointed out that there appear to be very few reported security issues with HID report descriptor parsing, in particular for Windows, which seems strange given how parsing those descriptor is very hard, and in particular there are very seriously broken descriptors out there. This would be another very interesting surface for a QEMU-based dancer software, to run through a number of broken HID report descriptor and send data to see how the system behaves. I would be very surprised if there is no bug in particular on the many small and random drives that apply workarounds such as the one I did for ELECOM.

Anyway, as I said I haven’t even had time to make the (probably minor) modifications to the framework to support BBB (which I already have access to), so you can imagine I’m not going to be working on this any time soon, but if you feel like working on some USB code, why not?

Update (2017-07-19): I pointed Travis at this post over Twitter, and he showed me vUSBf. While this does not have the same interface as facedancer, it proves that there is a chance to provide a virtual USB device implemented in Python.

As a follow up, Binyamin Sharet linked to Umap2 which supports fuzzing on top of facedancer, but does not support qemu as it is.

While it’s not quite (yet) what I had in mind, it proves that it is a feasible goal, and that there is already some code out there getting very close!