Abbott, the Libre 2, and the takedown

A few people today messaged and mentioned me on twitter regarding the news that Abbott has requested the takedown of something related to their Libre 2. I gave a quick hot take on this on Twitter, but I guess it’s worth having something in long form to be referenced, since I’m sure this will be talked about a lot more, not least because of the ominous permalink chosen by Boing Boing (“they-literally-own-you”) and the fact that, game of telephone style, the news went from the original takedown, to Reddit phrasing it as “Abbott asserts copyright on your data”, which is both silly and untrue.

So let’s start with a bit of background, that most of the re-posters of this story probably don’t know much about. The Libre 2 is an upgrade on the FreeStyle Libre system that I wrote a lot about and that I use daily. It comes with both a reader device and with support in the LibreLink app for both Android and (on more recent iPhones) iOS. The main difference with the Libre system is that the sensors provide both NFC and BLE capabilities, with the ability to proactively notify of high- or low-blood sugar conditions, that the old NFC-only sensors cannot provide, which is more similar to CGM solutions like Dexcom‘s.

In both the Libre and Libre 2 systems, the sensors don’t report blood sugar values, like in most classic glucometers. Instead they report a number of “raw” values, including from a number of temperature sensors. There’s a great explanation of these from Pierre Vandevenne, here and here. To get a real blood sugar measurement, you need to apply some algorithm, that Abbott still refines. The algorithm is what I usually refer to as “secret sauce”, and is implemented in both the reader’s firmware and the LibreLink app itself.

Above I used the word “something” to refer to what was taken down. The reason why I say that is that Boing Boing in the title straight up calls this a “tool” — but when you read the linked post from the affected person, it is described as “details of how to patch the LibreLink app”. Since I have not seen what the repository was before it was taken down, I have no idea which one to believe exactly. In either case, it looks like Abbott does not like someone to effectively leverage their “secret sauce” to use in a different application, but in particular, it does not look like we’re talking about something like glucometerutils, that implemented the protocol “clean”, without derivation off the original software.

Indeed, Boing Boing seems to make a case that this is equivalent of implementing a file format: «[…] just because Apple’s Pages can read Word docs, it doesn’t mean that Pages is a derivative of MS Office.» Except that it’s not as clear cut. If you implemented support for one format by copying the implementation code into your software, that actually would make it a derivative work, quite obviously. In this case, if I am to believe the original report instead, the taken down content were instructions to modify Abbott’s app — and not a redistribution of it. Since I’m not a lawyer, I have no idea where that stands, but it’s clearly not as black-and-white as Boing Boing appears to make it.

As I said on twitter, this does not affect either of my projects, since neither is relying on the original software, and are rather descriptions of the protocols. They also don’t include any information or support for the Libre 2, since the protocol appears to have changed. There’s an open issue with discussion, but it also appears that this time Abbott is using some encryption on the protocol. And that might be an interesting problem, as someone might have to get up close and personal with the code to figure that part out — but if that’s the case, we’re back at needing a clean-room design for implementing it.

I also want to quote Pierre explicitly from the posts I linked above:

[…] in the Libre FRAM, what we are seeing is a real “raw” signal. While the measure of the glucose signal itself is fairly reliable, it is heavily post-processed by the Libre firmware. Specifically – and in no particular order – temperature compensation, delay compensation, de-noising… all play a role. That understanding and, to some extent, my MD training, led me to extreme caution and prevented me from releasing my “solution”, which I knew to be both incomplete and unable to handle some error conditions.
The main driver behind my decision was the well known “first do no harm” (primum non nocere) motto, an essential part of the Hippocratic Oath which I symbolically took. I still stick by it today. […]
[…]
Today, there are a lot of add-on devices that aim to transform the Libre into a full CGM. To be honest, in general, I do not like either the results they provide or their (in)convenience. None of those I have tried delivered results that would lead to an approval by a regulatory agency, none of them were stable for long periods of time. But, apparently, patients still feel they are helpful and there is now a thriving community that aims at improving them.
Pierre Vandevenne

While I have not sworn a Hippocratic Oath myself, I have similar concerns to Pierre, and I have explicitly avoided documenting the sensors’ protocol, and I won’t be merging code that tries to read them directly, even if provided.

And when it comes to copyright issues, I do weigh them fairly heavily: they are the fundamental way that Free Software even works, by respecting licenses. So I will prefer someone to provide me with the description of Abbott’s encryption protocol, rather than an implementation of it where I may be afraid of a “poisonous tree.”

Glucometer notes: GlucoRx Nexus

This is a bit of a strange post, because it would be a glucometer review, except that I bought this glucometer a year and a half ago, teased a review, but don’t actually remember if I ever wrote any notes for it. While I may be able to get a new feel for the device to write a review, I don’t even know if the meter is still being distributed, and a few of the things I’m going to write here suggest me that it might not be the case, but who knows.

I found the Nexus as an over-the-counter boxed meter at my local pharmacy, in London. To me it appears like the device was explicitly designed to be used by the elderly, not just because of the large screen and numbers, but also because it comes with a fairly big lever to drop out the test strip, something I had previously only seen in the Sannuo meter.

This is also the first meter I see with an always-on display — although it seems that the backlight turns on only when the device is woken up, and otherwise is pretty much unreadable. I guess they can afford this type of display given that the meter is powered by 2 AAA batteries, rather than CR2032 like others.

As you may have guessed by now from the top link about the teased review, this is the device that uses a Silicon Labs CP2110 HID-to-UART adapter, for which I ended up writing a pyserial driver, earlier this year. The software to download the data seems to be available from the GlucoRx website for Windows and Mac — confusingly, the website you actually download the file from is not GlucoRx’s but Taidoc’s. TaiDoc Technology Corporation being named on the label under the device, together with MedNet GmbH. A quick look around suggests TaiDoc is a Taiwanese company, and now I’m wondering if I’m missing a cultural significance around the test strips, or blood, and the push-out lever.

I want to spend a couple notes about the Windows software, which is the main reason why I don’t know if the device is still being distributed. The download I was provided today was for version 5.04.20181206 – which presumes the software was still being developed as of December last year – but it does not seem to be quite tested to work on Windows 10.

The first problem is that that the Windows Defender malware detection tool actually considers the installer itself as malware. I’m not sure why, and honestly I don’t care: I’m only using this on a 90-days expiring Windows 10 virtual machine that barely has access to the network. The other problem, is that when you try to run the setup script (yes, it’s a script, it even opens a command prompt), it tries to install the redistributable for .NET 3.5 and Crystal Reports, fail and error out. If you try to run the setup for the software itself explicitly, you’re told you need to install .NET 3.5, which is fair, but then it opens a link from Microsoft’s website that is now not found and giving you a 404. Oops.

Setting aside these two annoying, but not insurmountable problems, what remains is to figure out the protocol behind the scenes. I wrote a tool that reads a pcapng file and outputs the “chatter”, and you can find it in the usbmon-tools repository. It’s far from perfect and among other things it still does not dissect the actual CP2110 protocol — only the obvious packets that I know include data traffic to the device itself.

This is enough to figure out that the serial protocol is one of the “simplest” that I have seen. Not in the sense of being easy to reverse, but rather in term of complexity of the messages: it’s a ping-pong protocol with fixed-length 8-bytes messages, of which the last one is a simple checksum (sum-modulo-8-bit), a fixed start byte of 0x51, and a fixed end with a bit for host-to-device and device-to-host selection. Adding to the first nibble of the message to always have the same value (2), it brings down the amount of data to be passed for each message to 34-bit. Which is a pretty low amount of information even when looking at simple information as glucose readings.

At any rate, I think I already have a bit of the protocol figured out. I’ll probably finish it over the next few days and the weekend, and then I’ll post the protocol in the usual repository. Hopefully if there are other users of this device they can be well served by someone writing a tool to download the data that is not as painful to set up as the original software.

Introducing usbmon-tools

A couple of weeks ago I wrote some notes about my work in progress to implement usbmon captures handling code, and pre-announced I was going to publish more of my extraction/inspection scripts.

The good news is that the project is now released, and you can find it on GitHub as usbmon-tools with an Apache 2.0 license, and open to contributions (with a CLA, sorry about that part). This is the first open source project I release using my employer’s releasing process (for other projects, I used the IARC process instead), and I have to say I’m fairly pleased with the results.

This blog post is meant mostly as a way to explain what’s going on my head regarding this project, with the hope that contributors can help it become reality. Or that they can contribute other ideas to it, even when they are not part of my particular plans.

I want to start with a consideration on the choice of language. usbmon-tools is written in Python 3. And in particular it is restricted to Python 3.7, because I wanted to have access to type annotations, which I found extremely addictive at work. I even set up Travis CI to run mypy as part of the integration tests for the repository.

For other projects I tend to be more conservative, and wait for Debian stable to have a certain version before requiring that as a minimum, but as this is a toolset for developers primarily, I’m going to expect its public to be able to deal with Python 3.7 as the requirement. This version was released nearly a year ago, and that should be plenty of time for people to have one at hand.

As for what the project should achieve in my view, is an easy way for developers to dissect an USB snooping trace. I started by building a simplistic tool that recreates a text format trace from the pcapng file, based on the official documentation of usbmon in the kernel (I have some patches to improve on that, too, but that probably will become a post in by itself next week). It’s missing isochronous support, and it’s not totally tested, but it at least gave me a few important insight on the format itself, including the big caveat that the “id” (or tag) of the URBs is not unique.

Indeed, I think that alone is one of the most important pieces of the puzzle in the library: in addition to parsing the pcapng file itself, the library can re-tag the events so that they get a real unique identifier (UUID), making it significantly easier to analyze the traces.

My next steps on the project are to write a more generic tool to convert a USB capture into what I call my “chatter format” (similar to the one I used to discuss serial protocols), and a more specific one that converts HID traces (because HID is a more defined protocol, and we can go a level deeper in exposing this into a human-readable source). I’m also considering if it would be within reach to provide the tool a HID descriptor blob, parse it and have it used to parse the HID traffic based on it. It would make some debugging particularly easier, for instance the stuff I did when I was fixing the ELECOM DEFT trackball.

I would also love to be able to play with a trace in a more interactive manner, for instance by loading this into Jupyter notebook, so that I could try parsing the blobs interactively, but unless someone with more experience with those contributes the code, I don’t expect I’ll have much time for it.

Pull requests are more than welcome!

Working with usbmon captures

Two years ago I posted some notes on how I do USB sniffing. I have not really changed much since then, although admittedly I have not spent much time reversing glucometers in that time. But I’m finally biting the bullet and building myself a better setup.

The reasons why I’m looking for a new setup are multiple: first of all, I now have a laptop that is fast enough to run a Windows 10 VM (with Microsoft’s 90 days evaluation version). Second, the proprietary software I used for USB sniffing has not been updated since 2016 — and they still have not published any information about their CBCF format, despite their reason being stated as:

Unfortunately, there is no such documentation and I’m almost sure will
never be. The reason is straightforward – every documented thing
should stay the same indefinitely. That is very restrictive.

At this point, keeping my old Dell Vostro 3750 as a sacrificial machine just for reverse engineering is not worth it anymore. Particularly when you consider that it started being obsoleted by both software (Windows 10 appears to have lost the ability to map network shares easily, and thus provide local-network backups), and hardware (the Western Digital SSD that I installed on it can’t be updated — their update package only works for UEFI boot systems, and while technically that machine is UEFI, it only supports the CSM boot).

When looking at a new option for my setup, I also want to be able to publish more of my scripts and tooling, if nothing else because I would feel more accomplished by knowing that even the side effects of working on these projects can be reused. So this time around I want to focus on all open source tooling, and build as much of the tools to be suitable for me to release as part of my employer’s open source program, which basically means not include any device-specific information within the tooling.

I started looking at Wireshark and its support for protocol dissectors. Unfortunately it looks like USB payloads are a bit more complicated, and dissector support is not great. So once again I’ll be writing a bunch of Python scripts to convert the captured data into some “chatter” files that are suitable for human consumption, at least. So I started to take a closer look at the usbmon documentation (the last time I looked at this was over ten years ago), and see if I can process that data directly.

To be fair, Wireshark does make it much nicer to get the captures out, since the text format usbmon is not particularly easy to parse back into something you can code with — and it is “lossy” when compared with the binary structures. With that, the first thing to focus on is to support the capture format Wireshark generates, which is pcapng, with one particular (out of many) USB capture packet structures. I decided to start my work from that.

What I have right now, is an (incomplete) library that can parse a pcapng capture into objects that are easier to play with in Python. Right now it loads the whole content into memory, which might or might not be a bad limitation, but for now it will do. I guess it would also be nice if I can find a way to integrate this with Colaboratory, which is a tool I only have vague acquaintance with, but would probably be great for this kind of reverse engineering, as it looks a lot like the kind of stuff I’ve been doing by hand. That will probably be left for the future.

The primary target right now is for me to be able to reconstruct the text format of usbmon given the pcapng capture. This would at least tell me that my objects are not losing details in the construction. Unfortunately this is proving harder than expected, because the documentation of usbmon is not particularly clear, starting from the definition of the structure, that mixes sized (u32) and unsized (unsigned int) types. I hope I’ll be able to figure this out and hopefully even send changes to improve the documentation.

As you might have noticed from my Twitter rants, I maintain that the documentation needs an overhaul. From mention of “easy” things, to the fact that the current suggested format (the binary structures) is defined in terms of the text format fields — except the text format is deprecated, and the kernel actually appears to produce the text format based on the binary structures. There are also quite a few things that are not obviously documented in the kernel docs, so you need to read the source code to figure out what they mean. I’ll try rewriting sections of the documentation.

Keep reading the blog to find updates if you have interests in this.

Reverse Engineering and Serial Adapter Protocols

In the comments to my latest post on the Silicon Labs CP2110, the first comment got me more than a bit upset because it was effectively trying to mansplain to me how a serial adapter (or more properly an USB-to-UART adapter) works. Then I realized there’s one thing I can do better than complain and that is providing even more information on this for the next person who might need them. Because I wish I knew half of what I know now back when I tried to write the driver for ch314.

So first of all, what are we talking about? UART is a very wide definition for any interface that implements serial communication that can be used to transmit between a host and a device. The word “serial port” probably bring different ideas to mind depending on the background of a given person, whether it is mice and modems connected to PCs, or servers’ serial terminals, or programming interfaces for microcontrollers. For the most part, people in the “consumer world” think of serial as RS-232 but people who have experience with complex automation systems, whether it is home, industrial, or vehicle automation, have RS-485 as their main reference. None of that actually matters, since these standards mostly deal with electrical or mechanical standards.

As physical serial ports on computer stopped appearing many years ago, most of the users moved to USB adapters. These adapters are all different between each other and that’s why there’s around 40KSLOC of serial adapters drivers in the Linux kernel (according to David’s SLOCCount). And that’s without counting the remaining 1.5KSLOC for implementing CDC ACM which is the supposedly-standard approach to serial adapters.

Usually the adapters are placed either directly on the “gadget” that needs to be connected, which expose a USB connector, or on a cable used to connect to it, in which case the device usually has a TRS or similar connectors. The TRS-based serial cables appeared to become more and more popular thanks to osmocom as they are relatively inexpensive to build, both as cables and as connectors onto custom boards.

Serial interface endpoints in operating systems (/dev/tty{S,USB,ACM}* on Linux, COM* on Windows, and so on) do not only transfer data between host and device, but also provides configuration of parameters such as transmission rate and “symbol shape” — you may or may not have heard references to something like “9600n8” which is a common way to express the transmission protocol of a serial interface: 9600 symbols per second (“baud rate”), no parity, 8-bit per symbol. You can call these “out of band” parameters, as they are transmitted to the UART interface, but not to the device itself, and they are the crux of the matter of interacting with these USB-to-UART adapters.

I already wrote notes about USB sniffing, so I won’t go too much into detail there, but most of the time when you’re trying to figure out what the control software sends to a device, you start by taking a USB trace, which gives you a list of USB Request Blocks (effectively, transmission packets), and you get to figure out what’s going on there.

For those devices that use USB-to-UART adapters and actually use the OS-provided serial interface (that is, COM* under Windows, where most of the control software has to run), you could use specialised software to only intercept the communication on that interface… but I don’t know of any such modern software, while there are at least a few well-defined interface to intercept USB communication. And that would not work for software that access the USB adapter directly from userspace, which is always the case for Silicon Labs CP2110, but is also the case for some of the FTDI devices.

To be fair, for those devices that use TRS, I actually have considered just intercepting the serial protocol using the Saleae Logic Pro, but beside being overkill, it’s actually just a tiny fraction of the devices that can be intercepted that way — as the more modern ones just include the USB-to-UART chip straight onto the device, which is also the case for the meter using the CP2110 I referenced earlier.

Within the request blocks you’ll have not just the serial communication, but also all the related out-of-band information, which is usually terminated on the adapter/controller rather than being forwarded onto the device. The amount of information changes widely between adapters. Out of those I have had direct experience, I found one (TI3420) that requires a full firmware upload before it would start working, which means recording everything from the moment you plug in the device provides a lot more noise than you would expect. But most of those I dealt with had very simple interfaces, using Control transfers for out-of-band configuration, and Bulk or Interrupt¹ transfers for transmitting the actual serial interface.

With these simpler interfaces, my “analysis” scripts (if you allow me the term, I don’t think they are that complicated) can produce a “chatter” file quite easily by ignoring the whole out of band configuration. Then I can analyse those chatter files to figure out the device’s actual protocol, and for the most part it’s a matter of trying between one and five combinations of transmission protocol to figure out the right one to speak to the device — in glucometerutils I have two drivers using 9600n8 and two drivers using 38400n8. In some cases, such as the TI3420 one, I actually had to figure out the configuration packet (thanks to the Linux kernel driver and the datasheet) to figure out that it was using 19200n8 instead.

But again, for those, the “decoding” is just a matter to filtering away part of the transmission to keep the useful parts. For others it’s not as easy.

0029 <<<< 00000000: 30 12                                             0.

0031 <<<< 00000000: 05 00                                             ..

0033 <<<< 00000000: 2A 03                                             *.

0035 <<<< 00000000: 42 00                                             B.

0037 <<<< 00000000: 61 00                                             a.

0039 <<<< 00000000: 79 00                                             y.

0041 <<<< 00000000: 65 00                                             e.

0043 <<<< 00000000: 72 00                                             r.

This is an excerpt from the chatter file of a session with my Contour glucometer. What happens here is that instead of buffering the transmission and sending a single request block with a whole string, the adapter (FTDI FT232RL) sends short burts, probably to reduce latency and keep a more accurate serial protocol (which is important for device that need accurate timing, for instance some in-chip programming interfaces). This would be also easy to recompose, except it also comes with

0927 <<<< 00000000: 01 60                                             .`

0929 <<<< 00000000: 01 60                                             .`

0931 <<<< 00000000: 01 60                                             .`

which I’m somehow sceptical they come from the device itself. I have not paid enough attention yet to figure out from the kernel driver whether this data is marked as coming from the device or is some kind of keepalive or synchronisation primitive of the adapter.

In the case of the CP2110, the first session I captured starts with:

0003 <<<< 00000000: 46 0A 02                                          F..

0004 >>>> 00000000: 41 01                                             A.

0006 >>>> 00000000: 50 00 00 4B 00 00 00 03  00                       P..K.....

0008 >>>> 00000000: 01 51                                             .Q

0010 >>>> 00000000: 01 22                                             ."

0012 >>>> 00000000: 01 00                                             ..

0014 >>>> 00000000: 01 00                                             ..

0016 >>>> 00000000: 01 00                                             ..

0018 >>>> 00000000: 01 00                                             ..

and I can definitely tell you that the first three URBs are not sent to the device at all. That’s because HID (the higher-level protocol that CP2110 uses on top of USB) uses the first byte of the block to identify the “report” it sends or receives. Checking these against AN434 give me a hint of what’s going on:

report 0x46 is “Get Version Information” — CP2110 always returns 0x0A as first byte, followed by a device version, which is unspecified; probably only used to confirm that the device is right, and possibly debugging purposes;
report 0x41 is “Get/Set UART Enabled” — 0x01 just means “turn on the UART”;
report 0x50 is “Get/Set UART Config” — and this is a bit more complex to parse: the first four bytes (0x00004b00) define the baud rate, which is 19200 symbols per second; then follows one byte for parity (0x00, no parity), one for flow control (0x00, no flow control), one for the number of data bits (0x03, 8-bit per symbol), and finally one for the stop bit (0x00, short stop bit); that’s a long way to say that this is configured as 19200n8.
report 0x01 is the actual data transfer, which means the transmission to the device starts with 0x51 0x22 0x00 0x00 0x00 0x00.

This means that I need a smarter analysis script that understands this protocol (which may be as simple as just ignoring anything that does not use report 0x01) to figure out what the control software is sending.

And at the same time, it needs code to know how “talk serial” to this device. Usually the out-of-bad configuration is done by a kernel driver: you ioctl() the serial device to the transmission protocol you need, the driver sends the right request block to the USB endpoint. But in the case of the CP2110 device, there’s no kernel driver implementing this, at least per Silicon Labs design: since HID devices are usually exposed to userland, and in particular to non-privileged applications, sending and receiving the reports can be done directly from the apps. So indeed there is no COM* device exposed on Windows, even with the drivers installed.

Could someone (me?) write a Linux kernel driver that expose CP2110 as a serial, rather than HID, device? Sure. It would require fiddling around with the HID subsystem a bit to have it ignore the original device, and that means it’ll probably break any application built with Silicon Labs’ own development kit, unless someone has a suggestion on how to have both interfaces available at the same time, while it would allow accessing those devices without special userland code. But I think I’ll stick with the idea of providing a Free and Open Source implementation of the protocol, for Python. And maybe add support for it to pyserial to make it easier for me to use it.

All these terms make more sense if you have at least a bit of knowledge of USB works behind the scene, but I don’t want to delve too much into that.
^[return]

Yak Shaving: Silicon Labs CP2110 and Linux

One of my favourite passtimes in the past years has been reverse engineering glucometers for the sake of writing an utility package to export data to it. Sometimes, in the quest of just getting data out of a meter I end up embarking in yak shaves that are particularly bothersome, as they are useful only for me and no one else.

One of these yak shaves might be more useful to others, but it will have to be seen. I got my hands on a new meter, which I will review later on. This meter has software for Windows to download the readings, so it’s a good target for reverse engineering. What surprised me, though, was that once I connected the device to my Linux laptop first, it came up as an HID device, described as an “USB HID to UART adapter”: the device uses a CP2110 adapter chip by Silicon Labs, and it’s the first time I saw this particular chip (or even class of chip) in my life.

Effectively, this device piggybacks the HID interface, which allows vendor-specified protocols to be implemented in user space without needing in-kernel drivers. I’m not sure if I should be impressed by the cleverness or disgusted by the workaround. In either case, it means that you end up with a stacked protocol design: the glucometer protocol itself is serial-based, implemented on top of a serial-like software interface, which converts it to the CP2110 protocol, which is encapsulated into HID packets, which are then sent over USB…

The good thing is that, as the datasheet reports, the protocol is available: “Open access to interface specification”. And indeed in the download page for the device, there’s a big archive of just-about-everything, including a number of precompiled binary libraries and a bunch of documents, among which figures AN434, which describe the full interface of the device. Source code is also available, but having spot checked it, it appears it has no license specification and as such is to be considered proprietary, and possibly virulent.

So now I’m warming up to the idea of doing a bit more of yak shaving and for once trying not to just help myself. I need to understand this protocol for two purposes: one is obviously having the ability to communicate with the meter that uses that chip; the other is being able to understand what the software is telling the device and vice-versa.

This means I need to have generators for the host side, but parsers for both. Luckily, construct should make that part relatively painless, and make it very easy to write (if not maintain, given the amount of API breakages) such a parser/generator library. And of course this has to be in Python because that’s the language my utility is written in.

The other thing that I realized as I was toying with the idea of writing this is that, done right, it can be used together with facedancer, to implement the gadget side purely in Python. Which sounds like a fun project for those of us into that kind of thing.

But since this time this is going to be something more widely useful, and not restricted to my glucometer work, I’m now looking to release this using a different process, as that would allow me to respond to issues and codereviews from my office as well as during the (relatively little) spare time I have at home. So expect this to take quite a bit longer to be released.

At the end of the day, what I hope to have is an Apache 2 licensed Python library that can parse both host-to-controller and controller-to-host packets, and also implement it well enough on the client side (based on the hidapi library, likely) so that I can just import the module and use it for a new driver. Bonus points if I can sue this to implement a test fake framework to implement the tests for the glucometer.

In all of this, I want to make sure to thank Silicon Labs for releasing the specification of the protocol. It’s not always that you can just google up the device name to find the relevant protocol documentation, and even when you do it’s hard to figure out if it’s enough to implement a driver. The fact that this is possible surprised me pleasantly. On the other hand I wish they actually released their code with a license attached, and possibly a widely-usable one such as MIT or Apache 2, to allow users to use the code directly. But I can see why that wouldn’t be particularly high in their requirements.

Let’s just hope this time around I can do something for even more people.

Anyone working on motherboard RGB controllers?

I have been contacted by email last week by a Linux user, probably noticing my recent patch for the gpio_it87 driver in the kernel. They have been hoping my driver could extend to IT7236 chips that are used in a number of gaming motherboards for controlling RGB LEDs.

Having left the case modding world after my first and only ThermalTake chassis – my mother gave me hell for the fans noise, mostly due to the plexiglass window on the side of the case – I still don’t have any context whatsoever on what the current state of these boards is, whether someone has written generic tools to set the LEDs, or even UIs for them. But it was an interesting back and forth of looking for leads into figuring out what is needed.

The first problem is, like most of you who already know a bit about electrical engineering and electronics, that clearly the IT7236 chip is clearly not the same series as the IT87xx chips that my driver supports. And since they are not the same series, they are unlikely to share the same functionality.

The IT87xx series chips are Super I/O controllers, which mean they implement functionality such as floppy-disk controllers, serial and parallel ports and similar interfaces, generally via the LPC bus. You usually know these chip names because these need to be supported by the kernel for them to show up in sensors output. In addition to these standard devices, many controllers include at least a set of general purpose I/O (GPIO) lines. On most consumer motherboards these are not exposed in any way, but boards designed for industrial applications, or customized boards tend to expose those lines easily.

Indeed, I wrote the gpio_it87 driver (well, actually adapted and extended it from a previous driver), because the board I was working on in Los Angeles had one of those chips, and we were interested in having access to the GPIO lines to drive some extra LEDs (and possibly in future versions more external interfaces, although I don’t know if anything was made of those). At the time, I did not manage to get the driver merged; a couple of years back, LaCie manufactured a NAS using a compatible chip, and two of their engineers got my original driver (further extended) merged into the Linux kernel. Since then I only submitted one other patch to add another ID for a compatible chip, because someone managed to send me a datasheet, and I could match it to the one I originally used to implement the driver as having the same behaviour.

Back to the original topic, the IT7236 chip is clearly not a Super I/O controller. It’s also not an Environmental Control (EC) chip, as I know that series is actually IT85xx, which is what my old laptop had. Somewhat luckily though, a “Preliminary Specifications” datasheet for that exact chip is available online from a company that appears to distribute electronics component in the general sense. I’m not sure if that was intentional or not, but having the datasheet is always handy of course.

According to these specifications, the IT7236xFN chips are “Touch ASIC Cap Button Controllers”. And indeed, ITE lists them as such. Comparing this with a different model in the same series shows that probably LED driving was not their original target, but they came to be useful for that. These chips also include an MCU based on a 8051 core, similarly to their EC solution — this makes them, and in particular the datasheet I found earlier, a bit more interesting to me. Unfortunately the datasheet is clearly amended to be the shorter version, and does not include a programming interface description.

Up to this point this tells us exactly one thing only: my driver is completely useless for this chip, as it implements specifically the Super I/O bus access, and it’s unlikely to be extensible to this series of chips. So a new driver is needed and some reverse engineering is likely to be required. The user who wrote me also gave me two other ITE chip names found on the board they have: IT8792⁰ and IT8686 (which appears to be a PWN fan controller — I couldn’t find it on the ITE website at all). Since the it87 (hwmon) driver is still developed out-of-kernel on GitHub, I checked and found an issue that appears to describe a common situation for gaming motherboards: the fans are not controlled with the usual Super I/O chip, but with a separate (more accurate?) one, and that suggests that the LEDs are indeed controlled by another separate chip, which makes sense. The user ran strings on the UEFI/BIOS image and did indeed find modules named after IT8790 and IT7236 (and IT8728 for whatever reason), to confirm this.

None of this brings us any closer to supporting it though, so let’s take a loop at the datasheet, and we can see that the device has an I²C bus, instead of the LPC (or ISA) bus used by Super I/O and the fan controller. Which meant looking at i2cdev and lsi2c. Unfortunately the output can only see that there are things connected to the bus, but not what they are.

This leaves us pretty much dry. Particularly me since I don’t have hardware access. So my suggestion has been to consider looking into the Windows driver and software (that I’m sure the motherboard manufacturer provides), and possibly figure out if they can run in a virtualized environment (qemu?) where I²C traffic can be inspected. But there may be simpler, more useful or more advanced tools to do most of this already, since I have not spent any time on this particular topic before. So if you know of any of them, feel free to leave a comment on the blog, and I’ll make sure to forward them to the concerned user (since I have not asked them if I can publish their name I’m not going to out them — they can, if they want, leave a comment with their name to be reached directly!).

Diabetes management software, online apps, and my projects

So my previous post with glucometerutils news got picked up by Hackaday, and though the comments ended up mostly talking about the (more physical, less practical) note about fiddling with the glucometers hardware themselves (which would suggest me the editor should probably have avoided moving the spotlight in the post, but never mind), I ended up replying to a few comments that were actually topical, to the point that I thought I should be writing about this more extensively.

In the comments, someone brought up Tidepool, which is a no-profit in California that develops what to me appears to be its own data storage and web application for diabetics. This is not far from what Glucosio is meant to be — and you might remember that an interaction with them, had me almost leave open source development, at least for what diabetes is concerned.

The problem with both projects, and a number of others that I’ve been pointed to over the years, is that I find most of them either not practical or web-oriented, or a mixture of the two. With not practical I mean that while building an “universal glucometer” capable of using any random strip is an interesting proposal, it does nothing to improve the patients’ life, and it actually can significantly increase the risks of misreading values and thus, risk the life of the user. For this reason, plus the fact that I do not have enough of a biochemistry understanding to figure out how to evaluate the precision of the meters that are already certified, I don’t invest any time looking into these projects.

Web-based applications such as Tidepool and similar are also far from my interests. I do not have a personal problem with accessing my blood sugar readouts for the sake of research, but I do have some concerns about which actors are allowed access to them. So in particular a startup like Glucosio is not someone I’d be particularly fond of giving access to my data to. Tidepool may be a no-profit, but that does not really make me feel much better, particularly because I would expect that an US-based no-profit would not have gone through all the possible data processing requirements of EU legislation, unlike, say, Abbott. I have already written a lot about why I don’t find self-hosting a good solution so I don’t think I need to spend much time on it here.

Except, there is one extra problem with those apps that require you to set up your own instance — like some of the people who have not been waiting some time ago. While running an app for my own interest may sound like an interesting thing to do, particularly if I want to build up the expertise to run complicated web app stacks, my personal ultimate goal is to have my doctor know what my blood sugar levels are over time. This is the whole point why I started that tool, I wanted to be able to output a PDF that my doctor could see without having to jump around a number of hoops to produce it — I failed to do so, but in part because I lost interest after I started using the awesome Accu-Chek Mobile.

If I were to tell my doctor «Log in on this site here with these credentials and you can see my readouts» he might actually do it, but mostly because of novelty and because he entertains my geekery around trying different meters and solutions. If he started to get this request from dozens of his patients, not only he’d have to keep a password managers just to deal with credentials, but he’d probably just couldn’t have the time to deal with it. The LibreLink app does have the ability to share data with a few services, and he did suggest me to look into diasend, but it looks like it got merged into something else that might or might not work for now, so I gave up.

Now, here is an interesting prospect, and why such apps are not completely worthless in my opinion. If the protocols are open to be used, and the apps are open source and can be set up by anyone, there is space for the doctors to have their own instance set up so that their patients can upload their data. Unfortunately, the idea that being open source this does not involve a significant investment in time and money is patently false. Particularly for important data like this, there has to be proper security, starting from every session being encrypted with TLS, and the data encrypted at rest (it is ironic that neither Tidepool nor Glucosio, at the time of writing, use TLS for their main websites). So I still don’t expect doctors in the public sector to be using these technologies any time soon. But on the other hand, there are more and more apps for this being built by the diabetes tech companies, so maybe we’ll see something happening in the future.

Where does this leave my project? Well, to begin with it’s not a single project but two of them. glucometerutils was born as a proof of concept and is still a handy tool to have. If someone manages to implement output to HTML or to PDF of the data, that would make it a very useful piece of software that does not need to interact with any remote, online application. The protocols repository serves a distinct need: it provides a way for more people to contribute to this ecosystem without requiring each of them to invest significant time in reversing the protocols, or getting in bed with the manufacturers, which – I can only guess – involves NDAs, data-sharing agreements, and similar bureaucracy that most hobbyist developers can’t afford.

Indeed, I know of at least one app, built for iOS, proprietary and commercial (as in, you have to pay for it), that has built support for meters thanks to my repository (and the author gave back in form of corrections and improvements on the documentation!). This is perfectly in line with my reasons to even have such a repository. I don’t care if the consumers and contributors to the repository build closed-source tools, as long as they share the knowledge on how to get to the data. And after that, may the best tool win.

As I said before, smartphones are no longer a luxury and for many people they are the only way they can access the Internet. It makes sense that the same way, for many diabetics it is their only way to analyse their readouts. This is why Contour Next One comes with Bluetooth and a nice app, and why there even are standard Bluetooth specification for glucometers (GLP/GLS) and continuous monitors (CGMP/CGMS). If my work on an open-source tool brings more people the ability to manage their diabetes, even with closed-source software, I’ll consider myself satisfied.

Now, there is one more interesting bit with Tidepool, though: they actually publish a Chrome-based uploader app that is able to download data from many more glucometers than my own tool (and the intersection between the two is minimal). This is great! But, as it happens, it comes with a little bit of a downside: the drivers are not documented at all. I confirmed the reason is that the access to the various meters’ protocols is subject to NDA — so while they can publish the code that access those meters, they cannot publish the specs of the protocols themselves, and that appears to include in-code comments that would make it easy to read what’s going on.

So, one of the things I’m going to do is read through those drivers, and try to write a protocol spec for the meters. It appears that they have a driver for Contour Next meters, which may or may not work for the Contour Next One which I’ve been trying to reverse engineer — I know there is at least one other open-source implementation of accessing data from Contour Next meters, but the other one is GPL-2 and, like OpenGlucose, I’ve avoided looking too closely to the code.

Projects such as Tidepool are extremely important to provide a proper alternative to the otherwise closed-garden of proprietary cloud diabetes management software. And if they become simple, and secure enough to set up, it is possible that some of the doctors will start providing their own instances where their patients can upload the readings, and that will make them also practical. But for now, to me they are only a good source of confrontation to figure out a way forward for my own tools.

Reverse engineering notes: USB sniffing

You have probably by now read a number of the posts I wrote about reverse engineering glucometers. And while I have complained about the lack of documentation, and maintain a repository of reverse-engineered protocols, I have not really shared the tools I’m using for my work.

The first problem with this is that I’m using a closed-source USB sniffer. I’m not proud of it, but it proved itself useful and worth the price, since the alternative that Microsoft suggests (Message Analyzer) appears not to be working for me, and USBpcap is not supported on Windows 10.

The native file format of USBlyzer is a CFBF container, but it also includes the ability to export the sniff to text CSV. Originally, I had to fight quite a bit with that, because version 2.1 of the tool produced a mostly unserviceable CSV – in particular the actual packet data was in an unmarked column – but the current version (2.2) is actually decent enough.

I have been working on these CSV, parsing them into a Python structure, and then manipulating them to produce what I refer to as “chatter” files, which is the format you see in my blog posts usually. These are just hexdumps (using the hexdump module) prefixed with a direction of the packet and the packet number, to make it easier to refer to the raw trace. The scripts I’ve used for this translation have evolved quite a bit, from a set of copy-pasted CSV parsing to building a dedicated module for the parsing, to the latest version that separates the idea of reassembling the higher-level protocol packets from actually producing the “chatter” file.

All of these scripts are yet to be released, but I hope to be able to do very soon. I’m also planning to start working a way to access the original CFBF (.ulz) files without requiring the UI to convert to CSV, as that should make my life significantly easier, as it avoids the most boring step in the process, which relies on the Windows UI. Luckily, there is an olefile module that allows you to access these files and the streams in them, I just not have started looking into what the structure of the content is.

I did contact the original developers of the software, and ask them to publish the file format specifications, since they should not contain any special sauce of their business case, but they told me they won’t do that, because it would require them to set in stone the format and never change it again. I told them I disagree on the stance, but it is their decision to make. So instead, I’ll be spending some time figuring this out in the future, feel free to keep reading this blog until I get more details on it.

One of the goals I’d like to have is the ability to convert the USBlyzer traces (either from CSV or their own format) to pcap format so that I can analyze it in Wireshark. This should allow me to make queries such as “Show me all the packets that have the fourth bytes as a value higher than 2” — which is indeed something I have been doing recently, in very convoluted ways. Unfortunately when I started looking on how to do this, I found out two things that made me very unhappy.

The first is that there isn’t a single way to store USB sniff in pcap format, but two. One is the “classic” usbmon one that you can get by building and loading the usbmon module in Linux, and starting Wireshark. The other is the one used by USBpcap to save the information from Windows. These have different formats and (as far as I can tell) there is no easy way to convert between the two. I’m also not sure if the standard Wireshark dissectors apply to that.

The other problem is that the USBpcap format itself is so Windows specific that, despite documenting the whole format on its website, it relies on some constant values coming from the Windows SDK. And you can imagine that depending on the Windows SDK for a Python utility to convert between two file formats is not a great idea. Luckily for me, Wine also has a header with the same constants, but being able to copy that code into the conversion utility means there’s a bit of work to do to make sure I can publish it under the proper license — as I would like to keep every tool’s license the most liberal I can.

You could be asking why on Earth I’m not using virtual machines and just good old standard usbmon to solve my problem. And the answer is two-fold: the laptop I’m using to do the development is just not powerful enough to run Windows 10 on a virtual machine, particularly when having to run Java software such as is the case for Contour, and on the other hand I don’t have enough USB ports to dedicate one to the virtual machine for attaching the devices.

I have an alternative plan, which involves using a BeagleBone Black and USBProxy, but I have not started on that project, among other things because it requires a bit of a complicated setup with USB Ethernet devices and external chargers. So it’s planned, but not sure when I’ll get to that.

Also, speaking of Wireshark, a quick check by using usbmon with my tool and the FreeStyle Libre (because I travel with that, so it’s easier to test on the road), tells me that here is something not quite correct in the HID dissector. In particular it looks like it only recognizes the first report sent by the device as an HID packet, when the response is fragmented into multiple packets. I need to spend some time to track that problem down, and possibly figure out how difficult it would be for me to build further dissectors that can reassemble the higher-level protocol, the way TCP and SCTP sessions can be displayed in Wireshark already.

Hardware Review: ELECOM DEFT Trackball

I know that by this point it feels like I’m collecting half-done reverse engineering projects, but sometimes you need some time off from one project to figure out how another is going to behave, so I have temporarily shelved my Accu-Chek Mobile and instead took a day to look at a different problem altogether.

I like trackballs, and with exception of gaming purposes, I consider them superior to mice: no cables that get tangled, no hands to move around the desk, much less space needed to keep clear, and so on. On laptops I prefer TrackPoint™ Style Pointers, which are rare to find, but on desktop I am a happy user of trackballs. Unfortunately my happiness has been having trouble as of late, because finding good trackballs is getting harder. My favourite device would still be Logitech’s Cordless Optical Trackman, but it was discontinued years ago, and it’s effectively impossible to find to buy. Amazon has second-hand listings for hundreds of dollars! I’m still kicking myself in the face for having dropped mine on the floor (literally) while I packed to move to Dublin, completely destroying it.

The new Logitech offerings appear to be all in the realm of thumb-operated “portable” trackballs, such as the M570, which I have and don’t particularly like. An alternative manufacturer that is easy to find both online and in store is Kensington, and I do indeed own an Orbit with the scroll ring, but it’s in my opinion too avant-garde and inconvenient. So I have been mostly unhappy.

But last year a colleague, also a trackball user, suggested me to look into the ELECOM DEFT Trackball (also available wired).

ELECOM, for those who may not be familiar with it, is a Japanese hardware company, that would sell everything from input devices to ultra flat network cables. If you have not been to Japan, it may be interesting to know that there is effectively a parallel world of hardware devices that you would not find in Europe or the USA, which makes a visit to Yodobashi Camera a must-see for every self-respecting geek.

I got the DEFT last year, and loved it. I’ve been using it at work all the time, because that’s where I mostly use a non-gaming input device anyway, but recently I started working from home a bit more often (it’s a long story) and got myself a proper setup for it, with a monitor, keyboard and, for a while, the M570 I noted above. I decided then to get myself two more of the DEFT, one to use with my work from home setup, and the other to use with my personal laptop while I work on reverse engineering.

Note here: I made a huge mistake. In both cases I ordered them from eBay directly from Japan, so I had to deal with the boring customs and VAT modules on my end. Which is not terrible, since the An Post depot is about ten minutes away from my apartment and my office, but it’s still less nice than just receiving the package directly. The second order, I ended up receiving two “Certified Frustration Free” packages, so I checked and indeed these devices are available on Amazon Japan. As I found out a few weeks ago for a completely different product, there is a feature called AmazonGlobal, which is not available for all products but would have been for these. With AmazonGlobal, the customs and VAT charges are taken care by Amazon, so there is no need for me to go out of my way and pay cash to An Post. And as it happens, if you don’t want to sign up for an account with Amazon Japan (which somehow is not federated to the others), you can just look for the same product on the USA version of Amazon, and AmazonGlobal applies just the same.

The trackball has a forefinger-operated ball (although ELECOM also makes a thumb-operated trackball), the usual left/middle/right buttons, a scroll wheel that “tilts” (badly) horizontally, and three function buttons at the top of the mouse (which I’ll go back to later). It also has two switches, one on the side, that has a red or blue area showing depending on how you pull it, and one on the bottom that is marked with the power symbol, H and L. Unfortunately, the manual leaflet that comes with the device is all in Japanese, which meant I had to get the help of Google Translate with my phone’s camera.

The switch on the side selects the DPI of the ball tracking (750 for blue, 1500 for red), while the one at the bottom appear to be a “power-assist” for the ball — it warns that the H version will use more battery.

As I said before, the trackball has three function buttons (marked Fn1, Fn2, Fn3) on the top. These are, for what I could tell, configurable through the Windows and Mac application, and they were indeed not seen by Linux at all, neither through xev nor through evtest. So I set myself up to reverse whichever protocol they used to configure this — I expected something similar to the Anker/Holtek gaming mouse I’m also working on, where the software programs the special event ID in the device directly.

The software can be downloaded on ELECOM’s website, although the whole page is in Japanese. On the other hand, the software itself is (badly) translated to English, so fewer yaks to shave there. Unfortunately when I tried using the app with the USB sniffer open… I could find nothing whatsoever. Turns out the app is actually handling all of that in software, rather than programming the hardware. So why did it not work on Linux? Well, I think that may be the topic for another post, since it turned out to require a kernel patch (which I sent, but can’t currently quite find it in the archives. I think that writing a blog post about it is going to be fairly useful, given that I had to assemble documentation that I found on at least a half dozen different sites.

Other things that may be relevant to know about the ELECOM is that somehow the 2.4GHz connection is sometimes a bit unstable. At the office, I cannot connect the receiver on the USB behind the monitor, because otherwise it skips beats. At home instead I have to put it there, because if I try to connect it directly to the Anker USB-C adapter I use with my Chromebook, the same problem happens. Ironically, the Microsoft receiver has the opposite problem: if I connect it behind the monitor at home, the keyboard sometimes get stuck repeating the same key over and over again. But again, that’s a topic for another time.