USB Captures Yak Shaving

Months ago I complained about the state of USB captures solutions in 2020. One of the issues it that you can’t easily provide a capture filter to libpcap, because they don’t want to implement user-mode capturing, and Linux does not provide BPF-based filtering for usbmon.

While I do still find it an interesting idea to add BPF filtering there, my kernel-fu is still fairly limited, and I thought I would start with something easier: filtering in userspace with a custom capture program. This also got me a bit more comfortable with the actual capture API, that I have been ignoring for the most part.

As I said before, languages are tools, and I could have tried implementing the tool in a different programming language. But on the other hand, I’m trying to get this done to integrate with the rest of the chatter-extraction tools I released as part of usbmon-tools, so why straying away (too much) from the path? Well, turns out that the usbmon interface is a bit too complicated to implement in pure Python, but Cython makes for a good extended language for it, and it’s something I’m familiar enough with — including for something fairly similar with the SGIO implementation.

It was yet another interesting exercise in Yak Shaving though. Beside the documentation being obtuse at times, and trying to explain the interfaces in their chronological order, with the most useful once last, I found myself partially stumped when I realised that the ioctl() constants you have to use to get any useful information are not available on any userspace header of the Linux kernel! Indeed, it seems the main implementation of usbmon, as part of libpcap, just copies enough of the structures to be able to read the information — and, by the way, does not actually follow the documented process: it sets a value for the buffer size, rather than getting the one that is already set.

I’ve now engaged to make sure that the structures and constants are available to userspace, because at the very least that needs to be addressed properly. I’ve also added unrolled constants for the two ioctl calls that are needed to set the capture up, which keep the amount of copy-paste from kernel headers to a minimum.

While I have committed a monitoring tool that allows printing the output of packets, this is far from the end. It only outputs text format right now, it doesn’t do URB re-tagging, and it only does naïve filters. My next few steps will likely involve getting python-pcapng write support merged in, and start writing pcapng file with the new tool. Then I can start looking at a more common, more interesting filtering set.

Once the capturing is properly taken care of, I have two main needs that I need to address, in the toolset: one is to be able to unpack PL2303 serial protocols — because the programmer that is failing me is using PL2303 and I would like to see how the conversation with the bootloader is going. While the stcgal tool has debug output, having a general chatter printer feels like it would be useful in the future. The other is USB Mass Storage parsing and inspecting, because I need it for the beurer, but also because I would like to turn some of my past reverse engineering blog posts into a talk, and I would like to have some more examples of how the tools make it easier to find the meat of the information.

So yeah that’s where a Sunday went for me…

USB capturing in 2020

The vast majority of the glucometer devices I reverse the protocol of use USB to connect to a computer. You could say that all of those that I successfully reversed up to now are USB based. Over the years, the way I capture USB packets to figure out a protocol changed significantly, starting from proprietary Windows-based sniffers, and more recently involving my own opensource trace tools. The process evolution was not always intentional — in the case of USBlyzer, it was pretty much dead a few years after I started using it, plus the author refused to document the file format, and by then even my sacrificial laptop was not powerful enough to keep running all the tools I needed.

I feel I’m close to another step on the evolution of my process, and once again it’s not because of me looking to improve the process as much as is the process not working on modern tools. Let me start by explaining what the situation is, because there are two nearly separate issues at play here.

The first issue is that either OpenSuse or the kernel changed the way the debugfs is handled. For those who have not looked at this before, debugfs is what lives in /sys/kernel/debug, and provides the more modern interface for usbmon access; the old method via /dev/usbmonX is deprecated, and Wireshark will not even show up the ability to capture USB packets without debugfs. Previously, I was able to manually change the ownership of the usbmon debugfs paths to my user, and started Wireshark as user to do the capturing, but as of January 2020, it does not seem to be possible to do that anymore: the debugfs mount is only accessible to root.

Using Wireshark as root is generally considered a really bad idea, because it has a huge attack surface, in particular when doing network captures, where the input would literally be to the discretion of external actors. It’s a tinsy bit safer when capturing USB because even when the device is fairly unknown, the traffic is not as controllable, so I would have flinched, but not terribly, to use Wireshark as root — except that I can’t sudo wireshark and have it paint on X. So the remaining alternative is to use tshark, which is a terminal utility that implements the same basics as Wireshark.

Unfortunately here’s the second problem: the last time I ran a lot of captures was when I was working on the Beurer glucometer (which I still haven’t gotten back to, because Linux 5.5 is still unreleased at the time of writing, and that’s the first version that’s not going to go into a reset loop with the device), and I was doing that work from my laptop, and that’s relevant. While the laptop’s keyboard and touchpad are USB, the ports are connected to a different bus internally. Since usbmon interfaces are set by bus, that made it very handy: I only needed to capture on the “ports” bus, and no matter how much and what I typed, it wouldn’t interfere in my captures at all.

You can probably see where this is going: I’m now using a NUC on my desk, with an external keyboard and the Elecom trackball (because I did manage to hurt my wrist while working on the laptop, but that’s a story for another post). And now all the USB 2.0 ports are connected to the same bus. Capturing the bus means getting all the events for keypresses, mouse movements, and so on.

If you have some experience with tcpdump or tshark, you’d think that this is an easy problem to solve: it’s not uncommon having to capture network packets from an SSH connection, which you want to exclude from the capture itself. And the solution for that is to apply a capture filter, such as port not 22.

Unfortunately, it looks like libpcap (which means Wireshark and tshark) does not support capture filters on usbmon. The reasoning provided is that since the capture filters for network are implemented in BPF, there’s no fallback for usbmon that does not have any BPF capabilities in the kernel. I’m not sure about the decision, but there you go. You could also argue that adding BPF to usbmon would be interesting to avoid copying too much data from the kernel, but that’s not something I have particular interest in exploring right now.

So how do you handle this? The suggested option is to capture everything, then use Wireshark to select a subset of packets and save the capture again. This should allow you to have a limited capture that you can share without risking having shared a keylogger off your system. But it also made me think a bit more.

The pcapng format, which Wireshark stores usbmon captures in, is a fairly complicated one, because it can include a lot of different protocol information, and it has multiple typed blocks to store things like hardware interface descriptions. But for USB captures, there’s not much use in the format: not only the Linux and Windows captures (the latter via usbpcap) are different formats altogether, but also the whole interface definition is, as far as I can tell, completely ignored. Instead, if you need a device descriptor, you need to scan the capture for a corresponding request (which usbmon-tools now does.)

I’m now considering just providing a simpler format to store captured data with usbmon-tools, either a simple 1:1 conversion from pcapng, with each packet just size-prefixed, and a tool to filter down the capture on the command line (because honestly, having to load Wireshark to cut down a capture is a pain), or a more complicated format that can store the descriptors separately, and maybe bundle/unbundle them across captures so that you can combine multiple fragments later. If I was in my bubble, I would be using protocol buffers, but that’s not particularly friendly to integrate in a Python module, as far as I can tell. Particularly if you want to be able to use the tools straight out of the git clone.

I guess that since I’m already using construct, I could instead design my own simplistic format. Or maybe I could just bite the bullet, use base64-encoded bytearrays, and write the whole capture session out in JSON.

As I said above, pcapng supports Windows and Linux captures differently: on Linux, the capture format is effectively the wire format of usbmon, while on Linux, it’s the format used by usbpcap. While I have not (yet, at the time of writing) added support to usbmon-tools to load the usbpcap captures, I don’t see why it shouldn’t work out that way. If I do manage to load usbpcap files, though, I would need a custom format to copy these to.

If anyone has a suggestion I’m open to them. One thing that I may try is to use Protocol Buffers but submit the generated source files to parse and serialize the object.

Introducing usbmon-tools

A couple of weeks ago I wrote some notes about my work in progress to implement usbmon captures handling code, and pre-announced I was going to publish more of my extraction/inspection scripts.

The good news is that the project is now released, and you can find it on GitHub as usbmon-tools with an Apache 2.0 license, and open to contributions (with a CLA, sorry about that part). This is the first open source project I release using my employer’s releasing process (for other projects, I used the IARC process instead), and I have to say I’m fairly pleased with the results.

This blog post is meant mostly as a way to explain what’s going on my head regarding this project, with the hope that contributors can help it become reality. Or that they can contribute other ideas to it, even when they are not part of my particular plans.

I want to start with a consideration on the choice of language. usbmon-tools is written in Python 3. And in particular it is restricted to Python 3.7, because I wanted to have access to type annotations, which I found extremely addictive at work. I even set up Travis CI to run mypy as part of the integration tests for the repository.

For other projects I tend to be more conservative, and wait for Debian stable to have a certain version before requiring that as a minimum, but as this is a toolset for developers primarily, I’m going to expect its public to be able to deal with Python 3.7 as the requirement. This version was released nearly a year ago, and that should be plenty of time for people to have one at hand.

As for what the project should achieve in my view, is an easy way for developers to dissect an USB snooping trace. I started by building a simplistic tool that recreates a text format trace from the pcapng file, based on the official documentation of usbmon in the kernel (I have some patches to improve on that, too, but that probably will become a post in by itself next week). It’s missing isochronous support, and it’s not totally tested, but it at least gave me a few important insight on the format itself, including the big caveat that the “id” (or tag) of the URBs is not unique.

Indeed, I think that alone is one of the most important pieces of the puzzle in the library: in addition to parsing the pcapng file itself, the library can re-tag the events so that they get a real unique identifier (UUID), making it significantly easier to analyze the traces.

My next steps on the project are to write a more generic tool to convert a USB capture into what I call my “chatter format” (similar to the one I used to discuss serial protocols), and a more specific one that converts HID traces (because HID is a more defined protocol, and we can go a level deeper in exposing this into a human-readable source). I’m also considering if it would be within reach to provide the tool a HID descriptor blob, parse it and have it used to parse the HID traffic based on it. It would make some debugging particularly easier, for instance the stuff I did when I was fixing the ELECOM DEFT trackball.

I would also love to be able to play with a trace in a more interactive manner, for instance by loading this into Jupyter notebook, so that I could try parsing the blobs interactively, but unless someone with more experience with those contributes the code, I don’t expect I’ll have much time for it.

Pull requests are more than welcome!

Working with usbmon captures

Two years ago I posted some notes on how I do USB sniffing. I have not really changed much since then, although admittedly I have not spent much time reversing glucometers in that time. But I’m finally biting the bullet and building myself a better setup.

The reasons why I’m looking for a new setup are multiple: first of all, I now have a laptop that is fast enough to run a Windows 10 VM (with Microsoft’s 90 days evaluation version). Second, the proprietary software I used for USB sniffing has not been updated since 2016 — and they still have not published any information about their CBCF format, despite their reason being stated as:

Unfortunately, there is no such documentation and I’m almost sure will
never be. The reason is straightforward – every documented thing
should stay the same indefinitely. That is very restrictive.

At this point, keeping my old Dell Vostro 3750 as a sacrificial machine just for reverse engineering is not worth it anymore. Particularly when you consider that it started being obsoleted by both software (Windows 10 appears to have lost the ability to map network shares easily, and thus provide local-network backups), and hardware (the Western Digital SSD that I installed on it can’t be updated — their update package only works for UEFI boot systems, and while technically that machine is UEFI, it only supports the CSM boot).

When looking at a new option for my setup, I also want to be able to publish more of my scripts and tooling, if nothing else because I would feel more accomplished by knowing that even the side effects of working on these projects can be reused. So this time around I want to focus on all open source tooling, and build as much of the tools to be suitable for me to release as part of my employer’s open source program, which basically means not include any device-specific information within the tooling.

I started looking at Wireshark and its support for protocol dissectors. Unfortunately it looks like USB payloads are a bit more complicated, and dissector support is not great. So once again I’ll be writing a bunch of Python scripts to convert the captured data into some “chatter” files that are suitable for human consumption, at least. So I started to take a closer look at the usbmon documentation (the last time I looked at this was over ten years ago), and see if I can process that data directly.

To be fair, Wireshark does make it much nicer to get the captures out, since the text format usbmon is not particularly easy to parse back into something you can code with — and it is “lossy” when compared with the binary structures. With that, the first thing to focus on is to support the capture format Wireshark generates, which is pcapng, with one particular (out of many) USB capture packet structures. I decided to start my work from that.

What I have right now, is an (incomplete) library that can parse a pcapng capture into objects that are easier to play with in Python. Right now it loads the whole content into memory, which might or might not be a bad limitation, but for now it will do. I guess it would also be nice if I can find a way to integrate this with Colaboratory, which is a tool I only have vague acquaintance with, but would probably be great for this kind of reverse engineering, as it looks a lot like the kind of stuff I’ve been doing by hand. That will probably be left for the future.

The primary target right now is for me to be able to reconstruct the text format of usbmon given the pcapng capture. This would at least tell me that my objects are not losing details in the construction. Unfortunately this is proving harder than expected, because the documentation of usbmon is not particularly clear, starting from the definition of the structure, that mixes sized (u32) and unsized (unsigned int) types. I hope I’ll be able to figure this out and hopefully even send changes to improve the documentation.

As you might have noticed from my Twitter rants, I maintain that the documentation needs an overhaul. From mention of “easy” things, to the fact that the current suggested format (the binary structures) is defined in terms of the text format fields — except the text format is deprecated, and the kernel actually appears to produce the text format based on the binary structures. There are also quite a few things that are not obviously documented in the kernel docs, so you need to read the source code to figure out what they mean. I’ll try rewriting sections of the documentation.

Keep reading the blog to find updates if you have interests in this.

A misc weekend

So today is the first week without Gentoo work for me. I admit I feel a bit disappointed, I used to do a lot of stuff and now that I can’t do them anymore (for my choice too), I feel like I’m useless. But I’m resisting the temptation to feel my overlay with a lot of ebuilds for the bumps that I would have done myself previously.

First of all, I wish to thank the anonymous “bla” who commented on my previous post about the lack of analysers for the usbmon output, suggesting me Wireshark. I really didn’t know they were working on supporting USB sniffing, and yes that is nice and stops me from deciding to write an analyser myself. Unfortunately, the current weekly snapshot of libpcap and the current subversion trunk of Wireshark does not get along well together: they simply don’t allow me to sniff at all, I get a series of errors instead.

Talking about the Chinese adapter, I found something on their site but I’m not really sure if it’s a specification or simply a description of the driver interface. Nonetheless, I’ve been able to identify a few patterns in the logs I was able to get with usbmon (analysed by hand), and I now know two major type of commands that are sent to the device: a pair of set and get commands used to set and get a 16 bit word out of at least three «registers»… I’m not sure if they are registers at all, they only are three 16-bit words which have another 16-bit word assigned to them, and which is set and get with some vendor-type requests. Unfortunately, I haven’t been able to get a hold of their actual meaning yet.

I’ve also found a sort of “commit” request, or a “get internal status” command, that tells me I haven’t initialised the device correctly yet: when handled by the Windows drivers, this request always return 0x9494, while here it returns values like 0xf4f4 or similar, never the actual 0x9494 word I’m expecting.

I didn’t work on Rust in the past few days but I have been thinking a lot about it, as David is being the first user of it, and submitted me a few reports of features currently missing, which I didn’t think of before, namely support for multiple constructors, and proper support for sub-classes. I haven’t worked on this yet, but I’ll do soon. He also provided me a patch that I merged, but I’ll have to write a series of testcase for that too, so that I can actually be sure that Rust continue behaving as needed.

Today I also decided to look a bit more on xine, as lately I’ve been neglecting it a lot. I’ve tried to cleanup my Wavpack demuxer a bit more, and I just finished a Valgrind (massif) run while playing two wavpack files one after the other. The results are somewhat interesting: xine does not leak memory, as the memory consumption remains constant between the two playbacks, but it also seems to allocate about ten megabytes of memory in _x_video_decode that are not used at all, as I’ve disabled the effect plugins, and so there should be no video at all. About seven more megabytes are wasted in the overlay handling, which also shows that the video output is not correctly disabled while playing audio tracks, with xine-ui at least.

I’ll be running a callgrind now also to test where time is actually lost, it’s good now that I have KCacheGrind displaying properly the callgraph, thanks to Albert Astals Cid (TSDgeos from KDE). One thing that I’m afraid of is that xine does not really have good performances on lossless files because it uses too small buffers, at least by default; I have this doubt because while mp3s play mostly fine – with mad decoder, not FFmpeg’s – it skips a lot for both Flac and WavPack files. This is especially seen when you run it through Valgrind, as the code is obviously slowed down and shows the need for improvements.

If only there was support in ruby-prof for Valgrind-style output to analyse the output with KCacheGrind, I could be improving Rust too. But that will have to wait I’m afraid.

Oh well.

Sheer luck, or lack of?

Okay, so today I spent the whole day following a bug for what I’m handling for my job, and when evening came, I was tired and in need of something that relaxed me.

Missing still the information to actually start taking seriously the idea of building model ships, and missing a jigsaw (I looked for some the other day at the supermarket, but I can’t find a subject that I like, most of them are paintings of the Renaisance, but I’m not really into that kind of art, I never liked studying it when I was at school, but I admit that graphic arts are not really something I’m into in general, but if it had to be a painting, I would have preferred finding something from Magritte.. I still remember seeing one of his works at the Guggenheim Musem here in Venice some time ago and being quite impressed; surrealism can be a kind of graphic art that I make myself like, it’s actually interesting), my best choice to get a relaxing night was to start looking for the W.ch USB serial converter driver.

I installed the only Windows I still have a license of (Windows 98) on a virtual machine, and set up usbmon in the kernel so that I could use it (it would be nice if the usbmon module depended on the debugfs support, as the latter is needed to use the former), and then started fighting to get the device up. After some usual Win98 driver setup (install driver, insert peripheral, remove driver, remove peripheral, re-insert peripheral, install driver, reboot, curse, remove peripheral, remove driver and so on), I’ve started having a working COM3 port with the adapter. Too bad that anything I tried to type was sent two or more times over the wires, but I knew VMware was far from perfect with USB.

Anyway, I started looking at the output of the usbmon tracing, and I was able to isolate some of the control messages used to set the parameters up (that are basically the only thing I need, as the actual I/O seems to happen just like PL2303 code (or usbserial) handles it); I’ve got the commit sequence and the okay string that I should expect from the driver, but I still have to make sense of all of them.

Unfortunately the USBmon (note the case) utility is written in Java. I say unfortunately simply because I still can’t get Sun’s JRE to work with XCB-enabled libX11, so I can’t run the utility on my box.

After some tries, which finally resulted in an hard lockup of Enterprise, which is now shut down waiting for me resuming it tomorrow, I finally decided that the next days I’ll start by writing some tools that might help me during the development of the driver, and maybe someone else in the future.

The first tool will be a serial signal generator: basically it would be a simple app that repeated the same string over and over and over on a serial port, using the configured baud rate, bits, parity and stop bits. This is useful because I can connect the adapter in look with my real serial port, leave the generator open on the serial port, and try sending different commands to the adapter till I get the signal as I need it, then I can move to try another parameter, for instance changing the stop bits, and try the commands again until I get the same parameters as configured in the generator. Right now I’ve used for this a simple C source file that sent “Flameeyes” string with the hardcoded parameters, I’d like to be able to set the parameters at runtime so you don’t have to stop and start it every time, but I’m not yet sure of which interface I could use for that.

The second tool is the one that most likely will be helpful if I can get it as good as I hope, and it would be a Ruby-based ncurses frontend for reading the data out of usbmon. First it would show the data in a more structured fashion, with the list of read packets that can be actually inspected without having to copy them around, with a ringbuffer for the read packets, and a different buffer for particular packets you want to inspect while continuing recording, and if I can with some kind of filters so that I can pinpoint the request for the single device, and then start discarding stuff like the I/O operations I don’t care about while trying to set the control stuff.

It’s far from being an easy task (especially since I never used ncurses), and I know I probably won’t be able to complete it as I want to, but at least I can try to provide a start if someone else will need to read that usbmon data in a more comprehensible shape. Right now SnoopyPro on Windows has a simpler view of the data, although it can really improve, especially to get a decent output to print and study.

But now, to actually make some sense out of the title I used for this blog, I have to tell you what happened while I was debugging. When I was ready to start messing with the parameters of the device, I wanted to be able to get no serial connection, and just start with the pristine adapter status, so I took one of the three remaining adapters (I ordered four from the same place) and connected it to Enterprise, launched picocom (that I started using as a test – you can find it in my overlay if you want it) on the ttyUSB0 device and.. found it not being there. An lsusb run later I seen that the adapter I just took was actually a PL2303-based device, and after looking at the other two adapters, I discovered that out of the four adapters I bought, only one has actually the W.ch chip on itself, and that is the one I chosen when I wanted to try them out, luck or lack of it?

But anyway I won’t stop working on the driver, I’ll probably take it more easy though, as three serial ports are enough for now, I ordered one more “just in case”, and that proven a good idea. I have my job to complete, so I’ll dedicate to the driver only my free time, compatibly with the other projects I’m working on. Not only I can’t really try to get the adapter replaced (I ordered it from an Hong Kong reseller on e-bay, the price to sending it back would probably be higher than I paid for it, besides I can’t really complain, the adapters were cheap and they never told me which chip they used), but I have no intention to: writing a driver for this adapter is an interesting challenge, it can be useful to other people, might be useful to me again in the future (I might need more serial adapters, and I’ll probably just order them as I ordered these ones, so I might find more W.ch-based devices), and it’s a nice way to start kernel hacking, as it’s a simple enough device.