Working with usbmon captures

Two years ago I posted some notes on how I do USB sniffing. I have not really changed much since then, although admittedly I have not spent much time reversing glucometers in that time. But I’m finally biting the bullet and building myself a better setup.

The reasons why I’m looking for a new setup are multiple: first of all, I now have a laptop that is fast enough to run a Windows 10 VM (with Microsoft’s 90 days evaluation version). Second, the proprietary software I used for USB sniffing has not been updated since 2016 — and they still have not published any information about their CBCF format, despite their reason being stated as:

Unfortunately, there is no such documentation and I’m almost sure will
never be. The reason is straightforward – every documented thing
should stay the same indefinitely. That is very restrictive.

At this point, keeping my old Dell Vostro 3750 as a sacrificial machine just for reverse engineering is not worth it anymore. Particularly when you consider that it started being obsoleted by both software (Windows 10 appears to have lost the ability to map network shares easily, and thus provide local-network backups), and hardware (the Western Digital SSD that I installed on it can’t be updated — their update package only works for UEFI boot systems, and while technically that machine is UEFI, it only supports the CSM boot).

When looking at a new option for my setup, I also want to be able to publish more of my scripts and tooling, if nothing else because I would feel more accomplished by knowing that even the side effects of working on these projects can be reused. So this time around I want to focus on all open source tooling, and build as much of the tools to be suitable for me to release as part of my employer’s open source program, which basically means not include any device-specific information within the tooling.

I started looking at Wireshark and its support for protocol dissectors. Unfortunately it looks like USB payloads are a bit more complicated, and dissector support is not great. So once again I’ll be writing a bunch of Python scripts to convert the captured data into some “chatter” files that are suitable for human consumption, at least. So I started to take a closer look at the usbmon documentation (the last time I looked at this was over ten years ago), and see if I can process that data directly.

To be fair, Wireshark does make it much nicer to get the captures out, since the text format usbmon is not particularly easy to parse back into something you can code with — and it is “lossy” when compared with the binary structures. With that, the first thing to focus on is to support the capture format Wireshark generates, which is pcapng, with one particular (out of many) USB capture packet structures. I decided to start my work from that.

What I have right now, is an (incomplete) library that can parse a pcapng capture into objects that are easier to play with in Python. Right now it loads the whole content into memory, which might or might not be a bad limitation, but for now it will do. I guess it would also be nice if I can find a way to integrate this with Colaboratory, which is a tool I only have vague acquaintance with, but would probably be great for this kind of reverse engineering, as it looks a lot like the kind of stuff I’ve been doing by hand. That will probably be left for the future.

The primary target right now is for me to be able to reconstruct the text format of usbmon given the pcapng capture. This would at least tell me that my objects are not losing details in the construction. Unfortunately this is proving harder than expected, because the documentation of usbmon is not particularly clear, starting from the definition of the structure, that mixes sized (u32) and unsized (unsigned int) types. I hope I’ll be able to figure this out and hopefully even send changes to improve the documentation.

As you might have noticed from my Twitter rants, I maintain that the documentation needs an overhaul. From mention of “easy” things, to the fact that the current suggested format (the binary structures) is defined in terms of the text format fields — except the text format is deprecated, and the kernel actually appears to produce the text format based on the binary structures. There are also quite a few things that are not obviously documented in the kernel docs, so you need to read the source code to figure out what they mean. I’ll try rewriting sections of the documentation.

Keep reading the blog to find updates if you have interests in this.

Reverse Engineering and Serial Adapter Protocols

In the comments to my latest post on the Silicon Labs CP2110, the first comment got me more than a bit upset because it was effectively trying to mansplain to me how a serial adapter (or more properly an USB-to-UART adapter) works. Then I realized there’s one thing I can do better than complain and that is providing even more information on this for the next person who might need them. Because I wish I knew half of what I know now back when I tried to write the driver for ch314.

So first of all, what are we talking about? UART is a very wide definition for any interface that implements serial communication that can be used to transmit between a host and a device. The word “serial port” probably bring different ideas to mind depending on the background of a given person, whether it is mice and modems connected to PCs, or servers’ serial terminals, or programming interfaces for microcontrollers. For the most part, people in the “consumer world” think of serial as RS-232 but people who have experience with complex automation systems, whether it is home, industrial, or vehicle automation, have RS-485 as their main reference. None of that actually matters, since these standards mostly deal with electrical or mechanical standards.

As physical serial ports on computer stopped appearing many years ago, most of the users moved to USB adapters. These adapters are all different between each other and that’s why there’s around 40KSLOC of serial adapters drivers in the Linux kernel (according to David’s SLOCCount). And that’s without counting the remaining 1.5KSLOC for implementing CDC ACM which is the supposedly-standard approach to serial adapters.

Usually the adapters are placed either directly on the “gadget” that needs to be connected, which expose a USB connector, or on a cable used to connect to it, in which case the device usually has a TRS or similar connectors. The TRS-based serial cables appeared to become more and more popular thanks to osmocom as they are relatively inexpensive to build, both as cables and as connectors onto custom boards.

Serial interface endpoints in operating systems (/dev/tty{S,USB,ACM}* on Linux, COM* on Windows, and so on) do not only transfer data between host and device, but also provides configuration of parameters such as transmission rate and “symbol shape” — you may or may not have heard references to something like “9600n8” which is a common way to express the transmission protocol of a serial interface: 9600 symbols per second (“baud rate”), no parity, 8-bit per symbol. You can call these “out of band” parameters, as they are transmitted to the UART interface, but not to the device itself, and they are the crux of the matter of interacting with these USB-to-UART adapters.

I already wrote notes about USB sniffing, so I won’t go too much into detail there, but most of the time when you’re trying to figure out what the control software sends to a device, you start by taking a USB trace, which gives you a list of USB Request Blocks (effectively, transmission packets), and you get to figure out what’s going on there.

For those devices that use USB-to-UART adapters and actually use the OS-provided serial interface (that is, COM* under Windows, where most of the control software has to run), you could use specialised software to only intercept the communication on that interface… but I don’t know of any such modern software, while there are at least a few well-defined interface to intercept USB communication. And that would not work for software that access the USB adapter directly from userspace, which is always the case for Silicon Labs CP2110, but is also the case for some of the FTDI devices.

To be fair, for those devices that use TRS, I actually have considered just intercepting the serial protocol using the Saleae Logic Pro, but beside being overkill, it’s actually just a tiny fraction of the devices that can be intercepted that way — as the more modern ones just include the USB-to-UART chip straight onto the device, which is also the case for the meter using the CP2110 I referenced earlier.

Within the request blocks you’ll have not just the serial communication, but also all the related out-of-band information, which is usually terminated on the adapter/controller rather than being forwarded onto the device. The amount of information changes widely between adapters. Out of those I have had direct experience, I found one (TI3420) that requires a full firmware upload before it would start working, which means recording everything from the moment you plug in the device provides a lot more noise than you would expect. But most of those I dealt with had very simple interfaces, using Control transfers for out-of-band configuration, and Bulk or Interrupt¹ transfers for transmitting the actual serial interface.

With these simpler interfaces, my “analysis” scripts (if you allow me the term, I don’t think they are that complicated) can produce a “chatter” file quite easily by ignoring the whole out of band configuration. Then I can analyse those chatter files to figure out the device’s actual protocol, and for the most part it’s a matter of trying between one and five combinations of transmission protocol to figure out the right one to speak to the device — in glucometerutils I have two drivers using 9600n8 and two drivers using 38400n8. In some cases, such as the TI3420 one, I actually had to figure out the configuration packet (thanks to the Linux kernel driver and the datasheet) to figure out that it was using 19200n8 instead.

But again, for those, the “decoding” is just a matter to filtering away part of the transmission to keep the useful parts. For others it’s not as easy.

0029 <<<< 00000000: 30 12                                             0.

0031 <<<< 00000000: 05 00                                             ..

0033 <<<< 00000000: 2A 03                                             *.

0035 <<<< 00000000: 42 00                                             B.

0037 <<<< 00000000: 61 00                                             a.

0039 <<<< 00000000: 79 00                                             y.

0041 <<<< 00000000: 65 00                                             e.

0043 <<<< 00000000: 72 00                                             r.

This is an excerpt from the chatter file of a session with my Contour glucometer. What happens here is that instead of buffering the transmission and sending a single request block with a whole string, the adapter (FTDI FT232RL) sends short burts, probably to reduce latency and keep a more accurate serial protocol (which is important for device that need accurate timing, for instance some in-chip programming interfaces). This would be also easy to recompose, except it also comes with

0927 <<<< 00000000: 01 60                                             .`

0929 <<<< 00000000: 01 60                                             .`

0931 <<<< 00000000: 01 60                                             .`

which I’m somehow sceptical they come from the device itself. I have not paid enough attention yet to figure out from the kernel driver whether this data is marked as coming from the device or is some kind of keepalive or synchronisation primitive of the adapter.

In the case of the CP2110, the first session I captured starts with:

0003 <<<< 00000000: 46 0A 02                                          F..

0004 >>>> 00000000: 41 01                                             A.

0006 >>>> 00000000: 50 00 00 4B 00 00 00 03  00                       P..K.....

0008 >>>> 00000000: 01 51                                             .Q

0010 >>>> 00000000: 01 22                                             ."

0012 >>>> 00000000: 01 00                                             ..

0014 >>>> 00000000: 01 00                                             ..

0016 >>>> 00000000: 01 00                                             ..

0018 >>>> 00000000: 01 00                                             ..

and I can definitely tell you that the first three URBs are not sent to the device at all. That’s because HID (the higher-level protocol that CP2110 uses on top of USB) uses the first byte of the block to identify the “report” it sends or receives. Checking these against AN434 give me a hint of what’s going on:

report 0x46 is “Get Version Information” — CP2110 always returns 0x0A as first byte, followed by a device version, which is unspecified; probably only used to confirm that the device is right, and possibly debugging purposes;
report 0x41 is “Get/Set UART Enabled” — 0x01 just means “turn on the UART”;
report 0x50 is “Get/Set UART Config” — and this is a bit more complex to parse: the first four bytes (0x00004b00) define the baud rate, which is 19200 symbols per second; then follows one byte for parity (0x00, no parity), one for flow control (0x00, no flow control), one for the number of data bits (0x03, 8-bit per symbol), and finally one for the stop bit (0x00, short stop bit); that’s a long way to say that this is configured as 19200n8.
report 0x01 is the actual data transfer, which means the transmission to the device starts with 0x51 0x22 0x00 0x00 0x00 0x00.

This means that I need a smarter analysis script that understands this protocol (which may be as simple as just ignoring anything that does not use report 0x01) to figure out what the control software is sending.

And at the same time, it needs code to know how “talk serial” to this device. Usually the out-of-bad configuration is done by a kernel driver: you ioctl() the serial device to the transmission protocol you need, the driver sends the right request block to the USB endpoint. But in the case of the CP2110 device, there’s no kernel driver implementing this, at least per Silicon Labs design: since HID devices are usually exposed to userland, and in particular to non-privileged applications, sending and receiving the reports can be done directly from the apps. So indeed there is no COM* device exposed on Windows, even with the drivers installed.

Could someone (me?) write a Linux kernel driver that expose CP2110 as a serial, rather than HID, device? Sure. It would require fiddling around with the HID subsystem a bit to have it ignore the original device, and that means it’ll probably break any application built with Silicon Labs’ own development kit, unless someone has a suggestion on how to have both interfaces available at the same time, while it would allow accessing those devices without special userland code. But I think I’ll stick with the idea of providing a Free and Open Source implementation of the protocol, for Python. And maybe add support for it to pyserial to make it easier for me to use it.

All these terms make more sense if you have at least a bit of knowledge of USB works behind the scene, but I don’t want to delve too much into that.
^[return]

Yak Shaving: Silicon Labs CP2110 and Linux

One of my favourite passtimes in the past years has been reverse engineering glucometers for the sake of writing an utility package to export data to it. Sometimes, in the quest of just getting data out of a meter I end up embarking in yak shaves that are particularly bothersome, as they are useful only for me and no one else.

One of these yak shaves might be more useful to others, but it will have to be seen. I got my hands on a new meter, which I will review later on. This meter has software for Windows to download the readings, so it’s a good target for reverse engineering. What surprised me, though, was that once I connected the device to my Linux laptop first, it came up as an HID device, described as an “USB HID to UART adapter”: the device uses a CP2110 adapter chip by Silicon Labs, and it’s the first time I saw this particular chip (or even class of chip) in my life.

Effectively, this device piggybacks the HID interface, which allows vendor-specified protocols to be implemented in user space without needing in-kernel drivers. I’m not sure if I should be impressed by the cleverness or disgusted by the workaround. In either case, it means that you end up with a stacked protocol design: the glucometer protocol itself is serial-based, implemented on top of a serial-like software interface, which converts it to the CP2110 protocol, which is encapsulated into HID packets, which are then sent over USB…

The good thing is that, as the datasheet reports, the protocol is available: “Open access to interface specification”. And indeed in the download page for the device, there’s a big archive of just-about-everything, including a number of precompiled binary libraries and a bunch of documents, among which figures AN434, which describe the full interface of the device. Source code is also available, but having spot checked it, it appears it has no license specification and as such is to be considered proprietary, and possibly virulent.

So now I’m warming up to the idea of doing a bit more of yak shaving and for once trying not to just help myself. I need to understand this protocol for two purposes: one is obviously having the ability to communicate with the meter that uses that chip; the other is being able to understand what the software is telling the device and vice-versa.

This means I need to have generators for the host side, but parsers for both. Luckily, construct should make that part relatively painless, and make it very easy to write (if not maintain, given the amount of API breakages) such a parser/generator library. And of course this has to be in Python because that’s the language my utility is written in.

The other thing that I realized as I was toying with the idea of writing this is that, done right, it can be used together with facedancer, to implement the gadget side purely in Python. Which sounds like a fun project for those of us into that kind of thing.

But since this time this is going to be something more widely useful, and not restricted to my glucometer work, I’m now looking to release this using a different process, as that would allow me to respond to issues and codereviews from my office as well as during the (relatively little) spare time I have at home. So expect this to take quite a bit longer to be released.

At the end of the day, what I hope to have is an Apache 2 licensed Python library that can parse both host-to-controller and controller-to-host packets, and also implement it well enough on the client side (based on the hidapi library, likely) so that I can just import the module and use it for a new driver. Bonus points if I can sue this to implement a test fake framework to implement the tests for the glucometer.

In all of this, I want to make sure to thank Silicon Labs for releasing the specification of the protocol. It’s not always that you can just google up the device name to find the relevant protocol documentation, and even when you do it’s hard to figure out if it’s enough to implement a driver. The fact that this is possible surprised me pleasantly. On the other hand I wish they actually released their code with a license attached, and possibly a widely-usable one such as MIT or Apache 2, to allow users to use the code directly. But I can see why that wouldn’t be particularly high in their requirements.

Let’s just hope this time around I can do something for even more people.

Anyone working on motherboard RGB controllers?

I have been contacted by email last week by a Linux user, probably noticing my recent patch for the gpio_it87 driver in the kernel. They have been hoping my driver could extend to IT7236 chips that are used in a number of gaming motherboards for controlling RGB LEDs.

Having left the case modding world after my first and only ThermalTake chassis – my mother gave me hell for the fans noise, mostly due to the plexiglass window on the side of the case – I still don’t have any context whatsoever on what the current state of these boards is, whether someone has written generic tools to set the LEDs, or even UIs for them. But it was an interesting back and forth of looking for leads into figuring out what is needed.

The first problem is, like most of you who already know a bit about electrical engineering and electronics, that clearly the IT7236 chip is clearly not the same series as the IT87xx chips that my driver supports. And since they are not the same series, they are unlikely to share the same functionality.

The IT87xx series chips are Super I/O controllers, which mean they implement functionality such as floppy-disk controllers, serial and parallel ports and similar interfaces, generally via the LPC bus. You usually know these chip names because these need to be supported by the kernel for them to show up in sensors output. In addition to these standard devices, many controllers include at least a set of general purpose I/O (GPIO) lines. On most consumer motherboards these are not exposed in any way, but boards designed for industrial applications, or customized boards tend to expose those lines easily.

Indeed, I wrote the gpio_it87 driver (well, actually adapted and extended it from a previous driver), because the board I was working on in Los Angeles had one of those chips, and we were interested in having access to the GPIO lines to drive some extra LEDs (and possibly in future versions more external interfaces, although I don’t know if anything was made of those). At the time, I did not manage to get the driver merged; a couple of years back, LaCie manufactured a NAS using a compatible chip, and two of their engineers got my original driver (further extended) merged into the Linux kernel. Since then I only submitted one other patch to add another ID for a compatible chip, because someone managed to send me a datasheet, and I could match it to the one I originally used to implement the driver as having the same behaviour.

Back to the original topic, the IT7236 chip is clearly not a Super I/O controller. It’s also not an Environmental Control (EC) chip, as I know that series is actually IT85xx, which is what my old laptop had. Somewhat luckily though, a “Preliminary Specifications” datasheet for that exact chip is available online from a company that appears to distribute electronics component in the general sense. I’m not sure if that was intentional or not, but having the datasheet is always handy of course.

According to these specifications, the IT7236xFN chips are “Touch ASIC Cap Button Controllers”. And indeed, ITE lists them as such. Comparing this with a different model in the same series shows that probably LED driving was not their original target, but they came to be useful for that. These chips also include an MCU based on a 8051 core, similarly to their EC solution — this makes them, and in particular the datasheet I found earlier, a bit more interesting to me. Unfortunately the datasheet is clearly amended to be the shorter version, and does not include a programming interface description.

Up to this point this tells us exactly one thing only: my driver is completely useless for this chip, as it implements specifically the Super I/O bus access, and it’s unlikely to be extensible to this series of chips. So a new driver is needed and some reverse engineering is likely to be required. The user who wrote me also gave me two other ITE chip names found on the board they have: IT8792⁰ and IT8686 (which appears to be a PWN fan controller — I couldn’t find it on the ITE website at all). Since the it87 (hwmon) driver is still developed out-of-kernel on GitHub, I checked and found an issue that appears to describe a common situation for gaming motherboards: the fans are not controlled with the usual Super I/O chip, but with a separate (more accurate?) one, and that suggests that the LEDs are indeed controlled by another separate chip, which makes sense. The user ran strings on the UEFI/BIOS image and did indeed find modules named after IT8790 and IT7236 (and IT8728 for whatever reason), to confirm this.

None of this brings us any closer to supporting it though, so let’s take a loop at the datasheet, and we can see that the device has an I²C bus, instead of the LPC (or ISA) bus used by Super I/O and the fan controller. Which meant looking at i2cdev and lsi2c. Unfortunately the output can only see that there are things connected to the bus, but not what they are.

This leaves us pretty much dry. Particularly me since I don’t have hardware access. So my suggestion has been to consider looking into the Windows driver and software (that I’m sure the motherboard manufacturer provides), and possibly figure out if they can run in a virtualized environment (qemu?) where I²C traffic can be inspected. But there may be simpler, more useful or more advanced tools to do most of this already, since I have not spent any time on this particular topic before. So if you know of any of them, feel free to leave a comment on the blog, and I’ll make sure to forward them to the concerned user (since I have not asked them if I can publish their name I’m not going to out them — they can, if they want, leave a comment with their name to be reached directly!).

Diabetes management software, online apps, and my projects

So my previous post with glucometerutils news got picked up by Hackaday, and though the comments ended up mostly talking about the (more physical, less practical) note about fiddling with the glucometers hardware themselves (which would suggest me the editor should probably have avoided moving the spotlight in the post, but never mind), I ended up replying to a few comments that were actually topical, to the point that I thought I should be writing about this more extensively.

In the comments, someone brought up Tidepool, which is a no-profit in California that develops what to me appears to be its own data storage and web application for diabetics. This is not far from what Glucosio is meant to be — and you might remember that an interaction with them, had me almost leave open source development, at least for what diabetes is concerned.

The problem with both projects, and a number of others that I’ve been pointed to over the years, is that I find most of them either not practical or web-oriented, or a mixture of the two. With not practical I mean that while building an “universal glucometer” capable of using any random strip is an interesting proposal, it does nothing to improve the patients’ life, and it actually can significantly increase the risks of misreading values and thus, risk the life of the user. For this reason, plus the fact that I do not have enough of a biochemistry understanding to figure out how to evaluate the precision of the meters that are already certified, I don’t invest any time looking into these projects.

Web-based applications such as Tidepool and similar are also far from my interests. I do not have a personal problem with accessing my blood sugar readouts for the sake of research, but I do have some concerns about which actors are allowed access to them. So in particular a startup like Glucosio is not someone I’d be particularly fond of giving access to my data to. Tidepool may be a no-profit, but that does not really make me feel much better, particularly because I would expect that an US-based no-profit would not have gone through all the possible data processing requirements of EU legislation, unlike, say, Abbott. I have already written a lot about why I don’t find self-hosting a good solution so I don’t think I need to spend much time on it here.

Except, there is one extra problem with those apps that require you to set up your own instance — like some of the people who have not been waiting some time ago. While running an app for my own interest may sound like an interesting thing to do, particularly if I want to build up the expertise to run complicated web app stacks, my personal ultimate goal is to have my doctor know what my blood sugar levels are over time. This is the whole point why I started that tool, I wanted to be able to output a PDF that my doctor could see without having to jump around a number of hoops to produce it — I failed to do so, but in part because I lost interest after I started using the awesome Accu-Chek Mobile.

If I were to tell my doctor «Log in on this site here with these credentials and you can see my readouts» he might actually do it, but mostly because of novelty and because he entertains my geekery around trying different meters and solutions. If he started to get this request from dozens of his patients, not only he’d have to keep a password managers just to deal with credentials, but he’d probably just couldn’t have the time to deal with it. The LibreLink app does have the ability to share data with a few services, and he did suggest me to look into diasend, but it looks like it got merged into something else that might or might not work for now, so I gave up.

Now, here is an interesting prospect, and why such apps are not completely worthless in my opinion. If the protocols are open to be used, and the apps are open source and can be set up by anyone, there is space for the doctors to have their own instance set up so that their patients can upload their data. Unfortunately, the idea that being open source this does not involve a significant investment in time and money is patently false. Particularly for important data like this, there has to be proper security, starting from every session being encrypted with TLS, and the data encrypted at rest (it is ironic that neither Tidepool nor Glucosio, at the time of writing, use TLS for their main websites). So I still don’t expect doctors in the public sector to be using these technologies any time soon. But on the other hand, there are more and more apps for this being built by the diabetes tech companies, so maybe we’ll see something happening in the future.

Where does this leave my project? Well, to begin with it’s not a single project but two of them. glucometerutils was born as a proof of concept and is still a handy tool to have. If someone manages to implement output to HTML or to PDF of the data, that would make it a very useful piece of software that does not need to interact with any remote, online application. The protocols repository serves a distinct need: it provides a way for more people to contribute to this ecosystem without requiring each of them to invest significant time in reversing the protocols, or getting in bed with the manufacturers, which – I can only guess – involves NDAs, data-sharing agreements, and similar bureaucracy that most hobbyist developers can’t afford.

Indeed, I know of at least one app, built for iOS, proprietary and commercial (as in, you have to pay for it), that has built support for meters thanks to my repository (and the author gave back in form of corrections and improvements on the documentation!). This is perfectly in line with my reasons to even have such a repository. I don’t care if the consumers and contributors to the repository build closed-source tools, as long as they share the knowledge on how to get to the data. And after that, may the best tool win.

As I said before, smartphones are no longer a luxury and for many people they are the only way they can access the Internet. It makes sense that the same way, for many diabetics it is their only way to analyse their readouts. This is why Contour Next One comes with Bluetooth and a nice app, and why there even are standard Bluetooth specification for glucometers (GLP/GLS) and continuous monitors (CGMP/CGMS). If my work on an open-source tool brings more people the ability to manage their diabetes, even with closed-source software, I’ll consider myself satisfied.

Now, there is one more interesting bit with Tidepool, though: they actually publish a Chrome-based uploader app that is able to download data from many more glucometers than my own tool (and the intersection between the two is minimal). This is great! But, as it happens, it comes with a little bit of a downside: the drivers are not documented at all. I confirmed the reason is that the access to the various meters’ protocols is subject to NDA — so while they can publish the code that access those meters, they cannot publish the specs of the protocols themselves, and that appears to include in-code comments that would make it easy to read what’s going on.

So, one of the things I’m going to do is read through those drivers, and try to write a protocol spec for the meters. It appears that they have a driver for Contour Next meters, which may or may not work for the Contour Next One which I’ve been trying to reverse engineer — I know there is at least one other open-source implementation of accessing data from Contour Next meters, but the other one is GPL-2 and, like OpenGlucose, I’ve avoided looking too closely to the code.

Projects such as Tidepool are extremely important to provide a proper alternative to the otherwise closed-garden of proprietary cloud diabetes management software. And if they become simple, and secure enough to set up, it is possible that some of the doctors will start providing their own instances where their patients can upload the readings, and that will make them also practical. But for now, to me they are only a good source of confrontation to figure out a way forward for my own tools.

Reverse engineering notes: USB sniffing

You have probably by now read a number of the posts I wrote about reverse engineering glucometers. And while I have complained about the lack of documentation, and maintain a repository of reverse-engineered protocols, I have not really shared the tools I’m using for my work.

The first problem with this is that I’m using a closed-source USB sniffer. I’m not proud of it, but it proved itself useful and worth the price, since the alternative that Microsoft suggests (Message Analyzer) appears not to be working for me, and USBpcap is not supported on Windows 10.

The native file format of USBlyzer is a CFBF container, but it also includes the ability to export the sniff to text CSV. Originally, I had to fight quite a bit with that, because version 2.1 of the tool produced a mostly unserviceable CSV – in particular the actual packet data was in an unmarked column – but the current version (2.2) is actually decent enough.

I have been working on these CSV, parsing them into a Python structure, and then manipulating them to produce what I refer to as “chatter” files, which is the format you see in my blog posts usually. These are just hexdumps (using the hexdump module) prefixed with a direction of the packet and the packet number, to make it easier to refer to the raw trace. The scripts I’ve used for this translation have evolved quite a bit, from a set of copy-pasted CSV parsing to building a dedicated module for the parsing, to the latest version that separates the idea of reassembling the higher-level protocol packets from actually producing the “chatter” file.

All of these scripts are yet to be released, but I hope to be able to do very soon. I’m also planning to start working a way to access the original CFBF (.ulz) files without requiring the UI to convert to CSV, as that should make my life significantly easier, as it avoids the most boring step in the process, which relies on the Windows UI. Luckily, there is an olefile module that allows you to access these files and the streams in them, I just not have started looking into what the structure of the content is.

I did contact the original developers of the software, and ask them to publish the file format specifications, since they should not contain any special sauce of their business case, but they told me they won’t do that, because it would require them to set in stone the format and never change it again. I told them I disagree on the stance, but it is their decision to make. So instead, I’ll be spending some time figuring this out in the future, feel free to keep reading this blog until I get more details on it.

One of the goals I’d like to have is the ability to convert the USBlyzer traces (either from CSV or their own format) to pcap format so that I can analyze it in Wireshark. This should allow me to make queries such as “Show me all the packets that have the fourth bytes as a value higher than 2” — which is indeed something I have been doing recently, in very convoluted ways. Unfortunately when I started looking on how to do this, I found out two things that made me very unhappy.

The first is that there isn’t a single way to store USB sniff in pcap format, but two. One is the “classic” usbmon one that you can get by building and loading the usbmon module in Linux, and starting Wireshark. The other is the one used by USBpcap to save the information from Windows. These have different formats and (as far as I can tell) there is no easy way to convert between the two. I’m also not sure if the standard Wireshark dissectors apply to that.

The other problem is that the USBpcap format itself is so Windows specific that, despite documenting the whole format on its website, it relies on some constant values coming from the Windows SDK. And you can imagine that depending on the Windows SDK for a Python utility to convert between two file formats is not a great idea. Luckily for me, Wine also has a header with the same constants, but being able to copy that code into the conversion utility means there’s a bit of work to do to make sure I can publish it under the proper license — as I would like to keep every tool’s license the most liberal I can.

You could be asking why on Earth I’m not using virtual machines and just good old standard usbmon to solve my problem. And the answer is two-fold: the laptop I’m using to do the development is just not powerful enough to run Windows 10 on a virtual machine, particularly when having to run Java software such as is the case for Contour, and on the other hand I don’t have enough USB ports to dedicate one to the virtual machine for attaching the devices.

I have an alternative plan, which involves using a BeagleBone Black and USBProxy, but I have not started on that project, among other things because it requires a bit of a complicated setup with USB Ethernet devices and external chargers. So it’s planned, but not sure when I’ll get to that.

Also, speaking of Wireshark, a quick check by using usbmon with my tool and the FreeStyle Libre (because I travel with that, so it’s easier to test on the road), tells me that here is something not quite correct in the HID dissector. In particular it looks like it only recognizes the first report sent by the device as an HID packet, when the response is fragmented into multiple packets. I need to spend some time to track that problem down, and possibly figure out how difficult it would be for me to build further dissectors that can reassemble the higher-level protocol, the way TCP and SCTP sessions can be displayed in Wireshark already.

Hardware Review: ELECOM DEFT Trackball

I know that by this point it feels like I’m collecting half-done reverse engineering projects, but sometimes you need some time off from one project to figure out how another is going to behave, so I have temporarily shelved my Accu-Chek Mobile and instead took a day to look at a different problem altogether.

I like trackballs, and with exception of gaming purposes, I consider them superior to mice: no cables that get tangled, no hands to move around the desk, much less space needed to keep clear, and so on. On laptops I prefer TrackPoint™ Style Pointers, which are rare to find, but on desktop I am a happy user of trackballs. Unfortunately my happiness has been having trouble as of late, because finding good trackballs is getting harder. My favourite device would still be Logitech’s Cordless Optical Trackman, but it was discontinued years ago, and it’s effectively impossible to find to buy. Amazon has second-hand listings for hundreds of dollars! I’m still kicking myself in the face for having dropped mine on the floor (literally) while I packed to move to Dublin, completely destroying it.

The new Logitech offerings appear to be all in the realm of thumb-operated “portable” trackballs, such as the M570, which I have and don’t particularly like. An alternative manufacturer that is easy to find both online and in store is Kensington, and I do indeed own an Orbit with the scroll ring, but it’s in my opinion too avant-garde and inconvenient. So I have been mostly unhappy.

But last year a colleague, also a trackball user, suggested me to look into the ELECOM DEFT Trackball (also available wired).

ELECOM, for those who may not be familiar with it, is a Japanese hardware company, that would sell everything from input devices to ultra flat network cables. If you have not been to Japan, it may be interesting to know that there is effectively a parallel world of hardware devices that you would not find in Europe or the USA, which makes a visit to Yodobashi Camera a must-see for every self-respecting geek.

I got the DEFT last year, and loved it. I’ve been using it at work all the time, because that’s where I mostly use a non-gaming input device anyway, but recently I started working from home a bit more often (it’s a long story) and got myself a proper setup for it, with a monitor, keyboard and, for a while, the M570 I noted above. I decided then to get myself two more of the DEFT, one to use with my work from home setup, and the other to use with my personal laptop while I work on reverse engineering.

Note here: I made a huge mistake. In both cases I ordered them from eBay directly from Japan, so I had to deal with the boring customs and VAT modules on my end. Which is not terrible, since the An Post depot is about ten minutes away from my apartment and my office, but it’s still less nice than just receiving the package directly. The second order, I ended up receiving two “Certified Frustration Free” packages, so I checked and indeed these devices are available on Amazon Japan. As I found out a few weeks ago for a completely different product, there is a feature called AmazonGlobal, which is not available for all products but would have been for these. With AmazonGlobal, the customs and VAT charges are taken care by Amazon, so there is no need for me to go out of my way and pay cash to An Post. And as it happens, if you don’t want to sign up for an account with Amazon Japan (which somehow is not federated to the others), you can just look for the same product on the USA version of Amazon, and AmazonGlobal applies just the same.

The trackball has a forefinger-operated ball (although ELECOM also makes a thumb-operated trackball), the usual left/middle/right buttons, a scroll wheel that “tilts” (badly) horizontally, and three function buttons at the top of the mouse (which I’ll go back to later). It also has two switches, one on the side, that has a red or blue area showing depending on how you pull it, and one on the bottom that is marked with the power symbol, H and L. Unfortunately, the manual leaflet that comes with the device is all in Japanese, which meant I had to get the help of Google Translate with my phone’s camera.

The switch on the side selects the DPI of the ball tracking (750 for blue, 1500 for red), while the one at the bottom appear to be a “power-assist” for the ball — it warns that the H version will use more battery.

As I said before, the trackball has three function buttons (marked Fn1, Fn2, Fn3) on the top. These are, for what I could tell, configurable through the Windows and Mac application, and they were indeed not seen by Linux at all, neither through xev nor through evtest. So I set myself up to reverse whichever protocol they used to configure this — I expected something similar to the Anker/Holtek gaming mouse I’m also working on, where the software programs the special event ID in the device directly.

The software can be downloaded on ELECOM’s website, although the whole page is in Japanese. On the other hand, the software itself is (badly) translated to English, so fewer yaks to shave there. Unfortunately when I tried using the app with the USB sniffer open… I could find nothing whatsoever. Turns out the app is actually handling all of that in software, rather than programming the hardware. So why did it not work on Linux? Well, I think that may be the topic for another post, since it turned out to require a kernel patch (which I sent, but can’t currently quite find it in the archives. I think that writing a blog post about it is going to be fairly useful, given that I had to assemble documentation that I found on at least a half dozen different sites.

Other things that may be relevant to know about the ELECOM is that somehow the 2.4GHz connection is sometimes a bit unstable. At the office, I cannot connect the receiver on the USB behind the monitor, because otherwise it skips beats. At home instead I have to put it there, because if I try to connect it directly to the Anker USB-C adapter I use with my Chromebook, the same problem happens. Ironically, the Microsoft receiver has the opposite problem: if I connect it behind the monitor at home, the keyboard sometimes get stuck repeating the same key over and over again. But again, that’s a topic for another time.

Reverse Engineering Notes: Accu-Chek Mobile

A couple of years ago, upon suggestion by a reader of this blog, I switched my glucometer to an Accu-Chek Mobile by Roche. I have not even looked into reverse engineering it at the time, as the killer application of that meter was not needing software at all. Indeed all the data is made available over a USB Mass Storage interface as CSV and graphs.

While there is a secondary interface available on the device to connect to software on PC, you had to order a copy of the download software online and receive it physically to be able to download the data to a computer, which I still find kind of silly for a device that is designed to have a standard USB connector.

Things changed recently, as Roche joined Abbott (and probably more to come) on the bandwagon of “show us yours”: upload your blood glucose reading to their cloud systems, and they will help you managing diabetes. I guess this is what happens when people are not waiting. I’m not particularly fond of uploading my health information to the cloud, but signing up for this service also meant being able to grab a proper chatter of the protocol over USB.

The software I’m using right now for snooping over the USB connection is USBlyzer which is proprietary and closed source — I’m not proud, but it gets its job done. I have been thinking of switching of a hardware snooping solution, and I bought a BeagleBone Black to use for that, but I have not started working on that due to time.

So instead, I have collected over time a set of libraries and utilities that operate on the CSV files that the software export (particularly as with version 2.2 they are actually quite well written, the older version was messier to post-process). I should look into publishing this collection, and I promise I’ll give it a try to publish before end of this year.

One of the tools I have prints out the “chatter”, coded hexdumps with include direction information to make it easier for me to read the stream. The first run of it was a bit noisy, but a quick check told me that what I’m interested in is bulk transfers (rather than control transfers, which are the most basic), so I filtered for those only, and then the first thing became obvious very quickly.

The maximum length of the bulk transfers in the trace is 64 bytes, which corresponds to the maximum size of bulk transfers for full speed endpoints. But the chatter shows the device sending multiple packets back from a single command, which is not unusual, as you can’t fit much blood sugar data in 64 bytes. And as usual when there is fragmentation, the actual data transfer size is coded somewhere at the beginning of the message.

0023 >>>> 00000000: E3 00 00 2C 00 03 50 79  00 26 80 00 00 00 80 00  ...,..Py.&......
0023 >>>> 00000010: 80 00 00 00 00 00 00 00  80 00 00 00 00 08 00 60  ...............`
0023 >>>> 00000020: 19 00 01 08 00 00 00 00  00 01 01 01 00 00 00 00  ................

0025 <<<< 00000000: E7 00 01 0A 01 08 00 00  01 01 01 02 00 00 FF FF  ................
0025 <<<< 00000010: FF FF 0D 1C 00 F8 50 00  00 05 00 F2 00 06 00 01  ......P.........
0025 <<<< 00000020: 00 04 00 24 09 2F 00 04  00 02 71 BC 0A 46 00 02  ...$./....q..F..
0025 <<<< 00000030: F0 40 09 96 00 02 08 52  0A 55 00 0C 00 02 00 08  .@.....R.U......
0025 <<<< 00000040: 09 90 00 08 0A 4C 00 02  00 06 00 02 00 04 00 24  .....L.........$
0025 <<<< 00000050: 09 2F 00 04 00 02 71 D0  0A 46 00 02 F0 40 09 96  ./....q..F...@..
0025 <<<< 00000060: 00 02 08 52 0A 55 00 0C  00 02 00 08 09 90 00 08  ...R.U..........
0025 <<<< 00000070: 0A 4C 00 02 00 05 00 03  00 03 00 1E 09 2F 00 04  .L.........../..
0025 <<<< 00000080: 00 80 71 D8 0A 46 00 02  F0 40 0A 55 00 0C 00 02  ..q..F...@.U....
0025 <<<< 00000090: 00 08 09 90 00 08 0A 66  00 02 00 05 00 04 00 03  .......f........
0025 <<<< 000000A0: 00 1E 09 2F 00 04 00 80  72 48 0A 46 00 02 F0 48  .../....rH.F...H
0025 <<<< 000000B0: 0A 55 00 0C 00 02 00 08  09 90 00 08 0A 49 00 02  .U...........I..
0025 <<<< 000000C0: 00 3D 00 05 00 08 00 46  0A 4D 00 02 98 20 09 43  .=.....F.M... .C
0025 <<<< 000000D0: 00 02 00 00 09 41 00 04  00 00 17 70 09 44 00 04  .....A.....p.D..
0025 <<<< 000000E0: 00 00 01 E5 09 53 00 02  00 00 0A 57 00 12 00 10  .....S.....W....
0025 <<<< 000000F0: 50 61 74 69 65 6E 74 20  52 65 73 75 6C 74 73 00  Patient Results.
0025 <<<>>> 00000000: E7 00 00 0E 00 0C 00 01  01 03 00 06 00 00 00 00  ................
0047 >>>> 00000010: 00 00                                             ..

0049 <<<< 00000000: E7 00 00 F6 00 F4 00 01  02 03 00 EE 00 00 00 08  ................
0049 <<<< 00000010: 00 E8 09 28 00 0E 00 06  52 6F 63 68 65 00 00 04  ...(....Roche...
0049 <<<< 00000020: 31 32 30 35 09 84 00 0A  00 08 00 60 19 04 B5 1B  1205.......`....
0049 <<<< 00000030: DF 5C 0A 44 00 02 40 00  09 2D 00 78 00 04 00 74  ..D..@..-.x...t
0049 <<<< 00000040: 00 01 00 00 00 18 73 65  72 69 61 6C 2D 6E 75 6D  ......serial-num
0049 <<<< 00000050: 62 65 72 3A 20 30 30 31  38 32 36 36 35 32 00 04  ber: 001826652..
0049 <<<< 00000060: 00 00 00 14 73 77 2D 72  65 76 20 4D 45 3A 20 56  ....sw-rev ME: V
0049 <<<< 00000070: 30 33 2E 31 33 20 20 20  00 05 00 00 00 16 66 77  03.13   ......fw
0049 <<<< 00000080: 2D 72 65 76 69 73 69 6F  6E 3A 20 56 30 33 2E 39  -revision: V03.9
0049 <<<< 00000090: 30 20 20 20 00 06 00 00  00 1A 70 72 6F 74 6F 63  0   ......protoc
0049 <<<< 000000A0: 6F 6C 2D 72 65 76 69 73  69 6F 6E 3A 20 52 50 43  ol-revision: RPC
0049 <<<< 000000B0: 20 31 2E 30 09 87 00 08  20 17 02 24 21 07 00 00   1.0.... ..$!...
0049 <<<< 000000C0: 0A 45 00 10 C0 00 1F 00  FF FF FF FF 00 64 00 00  .E...........d..
0049 <<<< 000000D0: 00 00 00 00 0A 4B 00 16  00 02 00 12 02 01 00 08  .....K..........
0049 <<<< 000000E0: 01 05 00 01 00 02 20 11  02 02 00 02 00 00 0A 5A  ...... ........Z
0049 <<<< 000000F0: 00 08 00 01 00 04 10 11  00 01                    ..........

As you can see in this particular exchange, bytes at offset 2-3 represent a (big-endian) length for the whole transfer. You just keep reading until that is complete.

While I have not (yet at the time of writing) figured out what the command and response actually convey, one thing that is kind of obvious is that there is some kind of (type-)length-value encoding at play, although in a bit of a funny way.

All records with type 0xE700 appear to have two-level lengths, as you can see on the two responses and the second command: in red it’s the length of the packet, in magenta the same length minus two (which matches the size of the length itself). There are also a number of strings, some zero terminated (Roche) and some not (1205), but still encoded with a 16-bit length in front of them.

The next thing to figure out in these cases is whether there is a checksum or not. For effectively all the meters I have reverse engineered up to now, except for maybe the cheap Korean one, include a checksum somewhere. I checked the chatter and found that there are a number of packets that appear to include the same information, but adding a checksum to the packet showed them different.

Once I dumped the (recomposed) packets to binary files, I noticed a number of packets with the same sizes. hexdump, wdiff and colordiff make it very easy to tell what changed between them. It didn’t quite look like a cryptographic checksum, as changing one byte would replace it with a very different number, but it didn’t quite match up with a “dumb” checksum of all the bytes values.

A couple of diff later, it become obvious.

[flameeyes@saladin Accu-Chek]$ wdiff <(hexdump -C 0039-0034-in) <(hexdump -C 0049-0046-in) | colordiff
00000000  e7 00 00 f6 00 f4 00 [-2d-] {+01+}  02 03 00 ee 00 00 00 08  [-|.......-........|-]  {+|................|+}
00000010  00 e8 09 28 00 0e 00 06  52 6f 63 68 65 00 00 04  |...(....Roche...|
00000020  31 32 30 35 09 84 00 0a  00 08 00 60 19 04 b5 1b  |1205.......`....|
00000030  df 5c 0a 44 00 02 40 00  09 2d 00 78 00 04 00 74  |..D..@..-.x...t|
00000040  00 01 00 00 00 18 73 65  72 69 61 6c 2d 6e 75 6d  |......serial-num|
00000050  62 65 72 3a 20 30 30 31  38 32 36 36 35 32 00 04  |ber: 001826652..|
00000060  00 00 00 14 73 77 2d 72  65 76 20 4d 45 3a 20 56  |....sw-rev ME: V|
00000070  30 33 2e 31 33 20 20 20  00 05 00 00 00 16 66 77  |03.13   ......fw|
00000080  2d 72 65 76 69 73 69 6f  6e 3a 20 56 30 33 2e 39  |-revision: V03.9|
00000090  30 20 20 20 00 06 00 00  00 1a 70 72 6f 74 6f 63  |0   ......protoc|
000000a0  6f 6c 2d 72 65 76 69 73  69 6f 6e 3a 20 52 50 43  |ol-revision: RPC|
000000b0  20 31 2e 30 09 87 00 08  20 17 02 24 21 [-06 53-] {+07 00+} 00  | 1.0.... [-..$!.S.|-] {+..$!...|+}
000000c0  0a 45 00 10 c0 00 1f 00  ff ff ff ff 00 64 00 00  |.E...........d..|
000000d0  00 00 00 00 0a 4b 00 16  00 02 00 12 02 01 00 08  |.....K..........|
000000e0  01 05 00 01 00 02 20 11  02 02 00 02 00 00 0a 5a  |...... ........Z|
000000f0  00 08 00 01 00 04 10 11  00 01                    |..........|
000000fa
[flameeyes@saladin Accu-Chek]$ wdiff <(hexdump -C 0039-0034-in) <(hexdump -C 0073-0068-in) | colordiff
00000000  e7 00 00 f6 00 f4 00 [-2d-] {+02+}  02 03 00 ee 00 00 00 08  [-|.......-........|-]  {+|................|+}
00000010  00 e8 09 28 00 0e 00 06  52 6f 63 68 65 00 00 04  |...(....Roche...|
00000020  31 32 30 35 09 84 00 0a  00 08 00 60 19 04 b5 1b  |1205.......`....|
00000030  df 5c 0a 44 00 02 40 00  09 2d 00 78 00 04 00 74  |..D..@..-.x...t|
00000040  00 01 00 00 00 18 73 65  72 69 61 6c 2d 6e 75 6d  |......serial-num|
00000050  62 65 72 3a 20 30 30 31  38 32 36 36 35 32 00 04  |ber: 001826652..|
00000060  00 00 00 14 73 77 2d 72  65 76 20 4d 45 3a 20 56  |....sw-rev ME: V|
00000070  30 33 2e 31 33 20 20 20  00 05 00 00 00 16 66 77  |03.13   ......fw|
00000080  2d 72 65 76 69 73 69 6f  6e 3a 20 56 30 33 2e 39  |-revision: V03.9|
00000090  30 20 20 20 00 06 00 00  00 1a 70 72 6f 74 6f 63  |0   ......protoc|
000000a0  6f 6c 2d 72 65 76 69 73  69 6f 6e 3a 20 52 50 43  |ol-revision: RPC|
000000b0  20 31 2e 30 09 87 00 08  20 17 02 24 21 [-06 53-] {+07 02+} 00  | 1.0.... [-..$!.S.|-] {+..$!...|+}
000000c0  0a 45 00 10 c0 00 1f 00  ff ff ff ff 00 64 00 00  |.E...........d..|
000000d0  00 00 00 00 0a 4b 00 16  00 02 00 12 02 01 00 08  |.....K..........|
000000e0  01 05 00 01 00 02 20 11  02 02 00 02 00 00 0a 5a  |...... ........Z|
000000f0  00 08 00 01 00 04 10 11  00 01                    |..........|
000000fa
[flameeyes@saladin Accu-Chek]$ wdiff <(hexdump -C 0039-0034-in) <(hexdump -C 0087-0084-in) | colordiff
00000000  e7 00 00 f6 00 f4 00 [-2d-] {+04+}  02 03 00 ee 00 00 00 08  [-|.......-........|-]  {+|................|+}
00000010  00 e8 09 28 00 0e 00 06  52 6f 63 68 65 00 00 04  |...(....Roche...|
00000020  31 32 30 35 09 84 00 0a  00 08 00 60 19 04 b5 1b  |1205.......`....|
00000030  df 5c 0a 44 00 02 40 00  09 2d 00 78 00 04 00 74  |..D..@..-.x...t|
00000040  00 01 00 00 00 18 73 65  72 69 61 6c 2d 6e 75 6d  |......serial-num|
00000050  62 65 72 3a 20 30 30 31  38 32 36 36 35 32 00 04  |ber: 001826652..|
00000060  00 00 00 14 73 77 2d 72  65 76 20 4d 45 3a 20 56  |....sw-rev ME: V|
00000070  30 33 2e 31 33 20 20 20  00 05 00 00 00 16 66 77  |03.13   ......fw|
00000080  2d 72 65 76 69 73 69 6f  6e 3a 20 56 30 33 2e 39  |-revision: V03.9|
00000090  30 20 20 20 00 06 00 00  00 1a 70 72 6f 74 6f 63  |0   ......protoc|
000000a0  6f 6c 2d 72 65 76 69 73  69 6f 6e 3a 20 52 50 43  |ol-revision: RPC|
000000b0  20 31 2e 30 09 87 00 08  20 17 02 24 [-21 06 53-] {+20 55 15+} 00  | 1.0.... [-..$!.S.|-] {+..$ U..|+}
000000c0  0a 45 00 10 c0 00 1f 00  ff ff ff ff 00 64 00 00  |.E...........d..|
000000d0  00 00 00 00 0a 4b 00 16  00 02 00 12 02 01 00 08  |.....K..........|
000000e0  01 05 00 01 00 02 20 11  02 02 00 02 00 00 0a 5a  |...... ........Z|
000000f0  00 08 00 01 00 04 10 11  00 01                    |..........|
000000fa

It may become much clearer if you go back at the first dump and observe the part highlighted in blue: it’s 8 bytes that represent 20 17 02 24 21 07 00 00. To help understanding this, you should know that I was looking at this around 9pm on February 24th, 2017. Indeed, these bytes effectively represent the date and time in binary-coded decimal, which is not something I was expecting to see, but make sense.

Once you know this, it’s easy to tell that there is no checksum in the messages, and that is one less problem to worry about. Indeed, when looking for the “big” packets, I could find the telltale representation of fixed-size records with what looked like (and I confirmed being) glucometer readings (in mg/dL, even though the device is mmol/l based).

If you’re wondering why the peculiar change in time on the last part of the diff, the reason is quite simple: the software noted that the time on the device didn’t match the time on the computer, and asked me to sync it. Which means I also know the command to set the time now.

Looking at the commands, there are a few more things that are interesting to see:

0037 >>>> 00000000: E7 00 00 0E 00 0C 00 2D  01 03 00 06 00 00 00 00  .......-........
0037 >>>> 00000010: 00 00                                             ..

0047 >>>> 00000000: E7 00 00 0E 00 0C 00 01  01 03 00 06 00 00 00 00  ................
0047 >>>> 00000010: 00 00                                             ..

0071 >>>> 00000000: E7 00 00 0E 00 0C 00 02  01 03 00 06 00 00 00 00  ................
0071 >>>> 00000010: 00 00                                             ..

0081 >>>> 00000000: E7 00 00 1A 00 18 00 03  01 07 00 12 00 00 0C 17  ................
0081 >>>> 00000010: 00 0C 20 17 02 24 20 54  57 00 00 00 00 00        .. ..$ TW.....

0085 >>>> 00000000: E7 00 00 0E 00 0C 00 04  01 03 00 06 00 00 00 00  ................
0085 >>>> 00000010: 00 00                                             ..

0095 >>>> 00000000: E7 00 00 14 00 12 00 05  01 07 00 0C 00 05 0C 0D  ................
0095 >>>> 00000010: 00 06 00 01 00 02 00 00                           ........

The time change command is 0081 (and I highlighted in green the new time, also provided as BCD). The remaining commands appear to be querying some information about the device. Commands 0037, 0047, 0071 and 0085 are exactly the same, except as I found out initially, no packet was identical. In blue I highlighted what appears to be a packet counter of sorts. I’m not sure why it starts at 0x2D, but after that it appears to increment normally, although only after 0xE7 commands (there appear to be a handful more).

Unfortunately this does not cover enough of the protocol yet, but it’s a good starting point for a few hours spent trying to prod things around on a Friday night (what an exciting life I live). I also managed to find how the device is reporting the readings, in blocks of less than 1KB records, but I have not figure out how the software knows when to stop asking for them. In this case it definitely is handy that I have so many readings on the device — this is probably the glucometer I used the most, and I still think is the best blood-reading glucometer, for handiness and results.

Stay tuned for more details, and hopefully to see a spec for the protocol soon, too.

Reverse Engineering is just the first step

Last year I said that reverse engineering obsolete systems is useful giving as an example adding Coreboot support for very old motherboards that are simpler and whose components are more likely to have been described somewhere already. One thing that I realized I didn’t make very clear in that post is that there is an important step on reverse engineering: documenting. As you can imagine from this blog, I think that documenting the reverse engineering processes and results are important, but I found out that this is definitely not the case for everybody.

On the particularly good side, going to 33c3 had a positive impression on me. Talks such as The Ultimate GameBoy Talk were excellent: Michael Steil did an awesome job at describing a lot of the unknown details of Nintendo’s most popular handheld. He also did a great job at showing practical matters, such as which tricks did various games use to implement things that at first sight would look impossible. And this is only one of his talks, he has a series that is going on year after year, I’ve watched his talk about the Commodore 64, and the only reason why it’s less enjoyable to watch is that the recording quality suffers from the ages.

In other posts I already referenced Micah’s videos. These have also been extremely nice to start watching, as she does a great job at explaining complex concepts, and even the “stream of consciousness” streams are very interesting and a good time to learn new tricks. What attracted me to her content, though, is the following video:

I have been using Wacom tablets for years, and I had no idea how they really worked behind the scene. Not only she does a great explanation of the technology in general, but the teardown of the mouse was also awesome with full schematics and explanation of the small components. No wonder I have signed up for her Patreon right away: she deserve to be better known and have a bigger following. And if funding her means spreading more knowledge around, well, then I’m happy to do my bit.

For the free software, open source and hacking community, reverse engineering is only half the process. The endgame is not for one person to know exactly how something works, but rather for the collectivity to gain more insight on things, so that more people have access to the information and can even improve on it. The community needs not only to help with that but also to prioritise projects that share information. And that does not just mean writing blogs about things. I said this before: blogs don’t replace documentation. You can see blogs as Micah’s shop-streaming videos, while documentation is more like her video about the tablets: they synthesize documentation in actually usable form, rather than just throwing information around.

I have a similar problem of course: my blog posts are not usually a bit of a stream of consciousness and they do not serve an useful purpose to capture the factual state of information. Take for example my post about reverse engineering the OneTouch Verio and its rambling on, then compare it with the proper protocol documentation. The latter is the actual important product, compared to my ramblings, and that is the one I can be proud of. I would also argue that documenting these things in a easily consumable form is more important than writing tools implementing them as those only cover part of the protocol and in particular can only leverage my skills, that do not involve statistical, pharmaceutical or data visualisation skills.

Unfortunately there are obstacles to these idea of course. Sometimes, reverse engineering documentation is attacked by manufacturer even more than code implementing the same information. So for instance while I have some information I still haven’t posted about a certain gaming mouse, I already know that the libratbag people do not want documentation of the protocols in their repository or wiki, because it causes them more headaches than the code. And then of course there is the problem of hosting this documentation somewhere.

I have been pushing my documentation on GitHub, hoping nobody causes a stink, but the good thing about using git rather than Wiki or similar tools is exactly that you can just move it around without losing information. This is not always the case: a lot of documentation is still nowadays only available either as part of code itself, or on various people’s homepages. And there are at least two things that can happen with that, the first is the most obvious and morbid one: the author of the documentation dies, and the documentation disappears once their domain registration expires, or whatever else, or if the homepage is at a given university or other academic endeavour, it may very well be that the homepage gets to disappear before the person anyway.

I know a few other alternatives to store this kind of data have been suggested, including common wiki akin to Wikipedia, but allowing for original research, but I am still uncertain that is going to be very helpful. The most obvious thing I can think of, is making sure these information can actually be published in books. And I think that at least No Starch Press has been doing a lot for this, publishing extremely interesting books including Designing BSD Rootkits and more recently Rootkits and Bootkits which is still in Early Access. A big kudos to Bill for this.

From my side, I promise I’ll try to organize my findings of anything I’ll work on in the best of my ability, and possibly organize it in a different form than just a blog, because the community deserves better.

Reverse engineering the FreeStyle Libre CGM, chapter 1

I have already reviewed the Abbott FreeStyle Libre continuous glucose monitor, and I have hinted that I already started reverse engineering the protocol it uses to communicate with the (Windows) software. I should also point out that for once the software does provide significant value, as they seem to have spent more effort in the data analysis than any other part of it.

Please note that this is just a first part for this device. Unlike the previous blog posts, I have not managed yet to get even partial information downloaded with my script as I write and post this. Indeed, if you, as you read this, have any suggestion of things I have not tried yet, please do let me know.

Since at this point it’s getting common, I’ve started up the sniffer, and sniffed starting from the first transaction. As it is to be expected, the amount of data in these transactions is significantly higher than that of the other glucometers. Even if you were taking seven blood samples a day for months with one of the other glucometers, it’s going to take a much longer time to get the same amount of readings as this sensor, which takes 96 readings a day by itself, plus the spot-checks and added notes and information to comment them.

The device itself presents itself as a standard HID device, which is a welcome change from the craziness of SCSI-based hidden message protocols. The messages within are of course not defined in any standard of course, so inspecting them become interesting.

It took me a while to figure out what the data that the software was already decoding for me meant. At first I thought I would have to use magic constant and libusb to speak raw USB to the device — indeed, a quick glance around Xavier’s work showed me that there were plently of similarities, and he’s including quite a few magical consants in that code. Luckily for me, after managing to query the device with python-libusb1, which was quite awkward as I also had to fix it to work, I realized that I was essentially reimplementing hidraw access.

After rewriting the code to use /dev/hidraw1 (which makes it significantly simpler), I also managed to understand that the device uses exactly the same initialization procedure as the FreeStyle InsuLinx that Xavier already implemented, and similar but not identical command handling (some of the commands match, and some even match the Optium, at least in format.)

Indeed the device seem to respond to two general classes of commands: text-commands and binary commands, the first device I reverse engineer with such a hybrid protocol. Text commands also have the same checksumming as both the Optium and Neo protocols.

The messages are always transferred in 64-bytes packets, even though the second byte of the message declares the actual significant length, which can be even zero. Neither the software nor the device zero out their buffers before writing the new command/response packets, so there is lots of noise in those packets.

I’ve decided that the custom message framing and its usage of HID is significant enough to warrant being documented by itself so I did that for now, although I have not managed to complete the reverse engineering of the protocol.

The remaining of the protocol kept baffling me. Some of the commands appear to include a checksum, and are ignored if they are not sent correctly. Others actually seem to append to an error buffer that you can somehow access (but probably more by mistake than design) and in at least one case I managed to “crash” the device, which asked me to turn it off and on again. I have thus decided to stop trying to send random messages to it for a while.

I have not been pouring time on this as much as I was considering doing before, what with falling for a bad flu, being oncall, and having visitors in town, so I have only been looking at traces from time to time, particularly recording all of them as I downloaded more data out of it. What still confuses me is that the commands sent from the software are not constant across different calls, but I couldn’t really make much heads or tails of it.

Then yesterday I caught a break — I really wanted to figure out at least if it was encoding or compressing the data, so I started looking for at least a sequence of numbers, by transcribing the device’s logbook into hexadecimal and looking in the traces for them.

This is not as easy as it might sound, because I have a British device — in UK, Ireland and Australia the measure of blood sugar is given in mmol/l rather than the much more common mg/dl. There is a stable conversion between the two units (you multiply the former by 18 to get the latter), but this conversion usually happens on display. All the devices I have used up to now have been storing and sending over the wire values in mg/dl and only converted when the data is shown, usually by providing some value within the protocol to specify that the device is set to use a given unit measure. Because of this conversion issue, and the fact that I only had access to the values mmol/l, I usually had two different options for each of the readings, as I wasn’t sure how the rounding happened.

The break happened when I was going through the software’s interface, trying to get the latest report data to at least match the reading timing difference, so that I could look for what might appear like a timestamp in the transcript. Instead, I found the “Export” function. The exported file is a comma-separated values file, which includes all readings, including those by the sensor, rather than just the spot-checks I could see from the device interface and in the export report. Not only that, but it includes a “reading ID”, which was interesting because it started from a value a bit over 32000, and is not always sequential. This was lucky.

I imported the CSV to Google Sheets, then added columns next to the ID and glucose readings. The latter were multiplied by 18 to get the value in mg/dl (yes the export feature still uses mmol/l, I think it might be some certification requirement), and then convert the whole lot to hexadecimal (hint: Google Sheets and LibreOffice have a DEC2HEX function that do that for you.) Now I had something interesting to search for: the IDs.

Now, I have to point out that the output I have from USBlyzer is a CSV file that includes the hexdump of the USB packets that are being exchanged. I already started writing a set of utilities (too rough to be published though) to convert those into a set of binary files (easier to bgrep or binwalk them) or hexdump-like transcripts (easier to recognize strings.) I wrote both a general “full USB transcript” script as well as a “Verio-specific USB transcript” while I was working on my OneTouch meter, so I wrote one for the Abbott protocol, too.

Because of the way that works, of course, it is not completely obvious if any value which is not a single byte is present, by looking at the text transcript, as it might be found on the message boundary. One would think they wouldn’t, since that means there are odd-sized records, but indeed that is the case for this device at least. Indeed it took me a few tries of IDs found in the CSV file to find one in the USB transcript.

And even after finding one the question was to figure out the record format. What I have done in the past when doing binary format reverse engineering was to print on a piece of paper a dump of the binary I’m looking at, and start doodling on it trying to mark similar parts of the message. I don’t have a printer in Dublin, so I decided to do a paperless version of the same, by taking a screenshot of a fragment of transcript, and loading it into a drawing app on my tablet. It’s not quite as easy, but it does making sharing results easier and thanks to layers it’s even easier to try and fail.

I made a mistake with the screenshot by not keeping the command this was a reply to in the picture — this will become more relevant later. Because of the size limit in the HID-based framing protocol Abbott uses, many commands reply with more than one message – although I have not understood yet how it signals a continuation – so in this case the three messages (separated by a white line) are in response to a single command (which by the way is neither the first or the last in a long series.)

The first thing I wanted to identify in the response was all the reading IDs, the one I searched for is marked in black in the screenshot, the others are marked in the same green tone. As you can see they are not (all) sequential; the values are written down as little-endian by the way. The next step was to figure out the reading values, which are marked in pink in the image. While the image itself has no value that is higher than 255, thus using more than bytes to represent them, not only it “looked fair” to assume little endian. It was also easy to confirm as (as noted in my review) I did have a flu while wearing the sensor, so by filtering for readings over 14 mmol/L I was able to find an example of a 16-bit reading.

The next thing I noted was the “constant” 0C 80 which might include some flags for the reading, I have not decoded it yet, but it’s an easy way to find most of the other IDs anyway. Following from that, I needed to find an important value, as it could allow decoding many other record types just by being present: the timestamp of the reading. The good thing with timestamps is that they tend to stay similar for a relative long time: the two highest bytes are the same for most of a day, and the highest of those is usually the same for a long while. Unfortunately looking for the hex representation of the Unix timestamp at the time yield nothing, but that was not so surprising, given how I found usage of a “newer” epoch in the Verio device I looked at earlier.

Now, since I have the exported data I know not only the reading ID but also the timestamp it reports it at, which does not include seconds. I also know that since the readings are (usually) taken at 15 minutes intervals, if they are using seconds since a given epoch the numbers should be incrementing by 900 between readings. Knowing this and doing some mental pattern matching it became easy to see where the timestamps have been hiding, they are marked in blue in the image above. I’ll get back to the epoch.

At this point, I still have not figured out where the record starts and ends — from the image it might appear that it starts with the record ID, but remember I took this piece of transcript mid-stream. What I can tell is that the length of the record is not only not a multiple of eight (the bytes in hexdump are grouped by eight) but it is odd, which, by itself, is fairly odd (pun intended.) This can be told by noticing how the colouring crosses the mid-row spacing, for 0c 80, for reading values and timestamps alike.

Even more interesting, not only the records can cross the message boundaries (see record 0x8fe0 for which the 0x004b value is the next message over), but even do the fields. Indeed you can see on the third message the timestamp ends abruptly at the end of the message. This wouldn’t be much of an issue if it wasn’t that it provides us with one more piece of information to decode the stream.

As I said earlier, timestamps change progressively, and in particular reading records shouldn’t usually be more than 900 seconds apart, which means only the lower two bytes change that often. Since the device uses little-endian to encode the numbers, the higher bytes are at the end of the encoded sequence, which means 4B B5 DE needs to terminate with 05, just like CC B8 DE 05 before it. But the next time we encounter 05 is in position nine of the following message, what gives?

The first two bytes of the message, if you checked the protocol description linked earlier, describe the message type (0B) and the number of significant bytes following (out of the usb packet), in this case 3E means the whole rest of the packet is significant. Following that there are six bytes (highlighted turquoise in the image), and here is where things get to be a bit more confusing.

You can actually see how discarding those six bytes from each message now gives us a stream of records that are at least fixed length (except the last one that is truncated, which means the commands are requesting continuous sequences, rather than blocks of records.) Those six bytes now become interesting, together with the inbound command.

The command that was sent just before receiving this response was 0D 04 A5 13 00 00. Once again the first two bytes are only partially relevant (message type 0D, followed by four significant bytes.) But A5 13 is interesting, since the first message of the reply starts with 13 A6, and the next three message increment the second byte each. Indeed, the software follows these with 0D 04 A9 13 00 00, which matches the 13 A9 at the start of the last response message.

What the other four bytes mean is still quite the mystery. My assumption right now is that they are some form of checksum. The reason is to be found in a different set of messages:

>>>> 00000000: 0D 04 5F 13 00 00                                 .._...

<<<< 00000000: 0B 3E 10 5F 34 EC 5A 6D  00 00 00 00 00 00 00 00  .>._4.Zm........
<<<< 00000010: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................

<<<< 00000000: 0B 3E 10 60 34 EC 5A 6D  00 00 00 00 00 00 00 00  .>.`4.Zm........
<<<< 00000010: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................

<<<< 00000000: 0B 3E 10 61 34 EC 5A 6D  00 00 00 00 00 00 00 00  .>.a4.Zm........
<<<< 00000010: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................

<<<< 00000000: 0B 3E 10 62 34 EC 5A 6D  00 00 00 00 00 00 00 00  .>.b4.Zm........
<<<< 00000010: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000030: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................

<<<< 00000000: 0B 3E 10 63 E8 B6 84 09  00 00 00 00 00 00 00 00  .>.c............
<<<< 00000010: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
<<<< 00000020: 00 00 00 00 9A 39 65 70  99 51 09 30 4D 30 30 30  .....9ep.Q.0M000
<<<< 00000030: 30 37 52 4B 35 34 00 00  01 00 02 A0 9F DE 05 FC  07RK54..........

In this set of replies, there are two significant differences compared to the ones with record earlier. The first is that while the command lists 5F 13 the replies start with 10 5F, so that not only 13 becomes 10, but 5F is not incremented until the next message, making it unlikely for the two bytes to form a single 16-bit word. The second is that there are at least four messages with identical payload (fifty-six bytes of value zero). And despite the fourth byte of the message changing progressively, the following four bytes are staying the same. This makes me think it’s a checksum we’re talking about, although I can’t for the life of me figure out which at first sight. It’s not CRC32, CRC32c nor Adler32.

By the way, the data in the last message relates to the list of sensors the devices has seen — 9ep.Q.0M00007RK54 is the serial number, and A0 9F DE 05 is the timestamp of it initializing.

Going back to the epoch, which is essentially the last thing I can talk about for now. The numbers above clearly shower in a different range than the UNIX timestamp, which would start with 56 rather than 05. So I used the same method I used for the Verio, and used a fixed, known point in time, got the timestamp from the device and compared with its UNIX timestamp. The answer was 1455392700 — which is 2012-12-31T00:17:00+00:00. It would make perfect sense, if it wasn’t 23 hours and 43 minutes away from a new year…

I guess that is all for now, I’m still trying to figure out how the data is passed around. I’m afraid that what I’m seeing from the software looks like it’s sending whole “preferences” structures that change things at once, which makes it significantly more complicated to understand. It’s also not so easy to tell how the device and software decide the measure unit as I don’t have access to logs of a mg/dl device.