CP2110 Update for 2019

Update 2021-03-18: Hello from the future! If you’re reading this blog post, and you’re interested in using the Silicon Labs CP2110 serial adapter, you may be interested to know that it’s fully supported by pyserial 3.5 and later, on Linux and Windows.

The last time I wrote about the CP2110 adapter was nearly a year ago, and because I have had a lot to keep me busy since, I have not been making much progress. But today I had some spare cycles and decided to take a deeper look starting from scratch again.

What I should have done properly since then would have been procuring myself a new serial dongle, as I was not (and still am) not entirely convinced about the quality of the CH341 adapter I’m using. I think I used that serial adapter successfully before, but maybe I didn’t and I’ve been fighting with ghosts ever since. This counts double as, silly me, I didn’t re-read my own post when I resumed working on this, and been scratching my head at nearly exactly the same problems as last time.

I have some updates first. The first of which is that I have some rough-edged code out there on this GitHub branch. It does not really have all the features it should, but it at least let me test the basic implementation. It also does not actually let you select which device to open — it looks for the device with the same USB IDs as I have, and that might not work at all for you. I’ll be happy to accept pull requests to fix more of the details, if anyone happen to need something like this too — once it’s actually in a state where it can be merged, I’ll be doing a squash commit and send a pull request upstream with the final working code.

The second is that while fighting with this, and venting on Twitter, Saleae themselves put me on the right path: when I said that Logic failed to decode the CP2110→CH341 conversation at 5V but worked when they were set at 3.3V, they pointed me at the documentation of threshold voltage, which turned out to be a very good lead.

Indeed, when connecting the CP2110 at 5V alone, Logic reports a high of 5.121V, and a low of ~-0.12V. When I tried to connect it with the CH341 through the breadboard full of connections, Logic reports a low of nearly 3V! And as far as I can tell, the ground is correctly wired together between the two serial adapters — they are even connected to the same USB HUB. I also don’t think the problem is with the wiring of the breadboard, because the behaviour is identical when just wiring the two adapters together.

So my next step has been setting up the BeagleBone Black I bought a couple of years ago and shelved into a box. I should have done that last year, and I would probably have been very close to have this working in the first place. After setting this up (which is much easier than it sounds), and figuring out from the BeagleBoard Wiki the pinout (and a bit of guesswork on the voltage) of its debug serial port, I could confirm the data was being sent to the CP2110 right — but it got all mangled on print.

The answer was that the HID buffered reads are… complicated. So instead of deriving most of the structure from the POSIX serial implementation, I lifted it from the RFC2217 driver, that uses a background thread to loop the reads. This finally allowed me to use the pySerial miniterm tool to log in and even dmesg(!) the BBB over the CP2110 adapter, which I consider a win.

Tomorrow I’ll try polishing the implementation to the point where I can send a pull request. And then I can actually set up to look back into the glucometer using it. Because I had an actual target when I started working on this, and was not just trying to get this to work for the sake of it.

Updates on Silicon Labs CP2110

One month ago I started the yak shave of supporting the Silicon Labs CP2110 with a fully opensource stack, that I can even re-use for glucometerutils.

The first step was deciding how to implement this. While the device itself supports quite a wide range of interfaces, including a GPIO one, I decided that since I’m only going to be able to test and use practically the serial interface, I would at least start with just that. So you’ll probably see the first output as a module for pyserial that implements access to CP2110 devices.

The second step was to find an easy way to test this in a more generic way. Thankfully, Martin Holzhauer, who commented on the original post, linked to an adapter by MakerSpot that uses that chip (the link to the product was lost in the migration to WordPress, sigh), which I then ordered and received a number of weeks later, since it had to come to the US and clear customs through Amazon.

All of this was the easy part, the next part was actually implementing enough of the protocol described in the specification, so that I could actually send and receive data — and that also made it clear that despite the protocol being documented, it’s not as obvious as it might sound — for instance, the specification says that the reports 0x01 to 0x3F are used to send and receive data, but it does not say why there are so many reports… except that it turns out they are actually used to specify the length of the buffer: if you send two bytes, you’ll have to use the 0x02 report, for ten bytes 0x0A, and so on, until the maximum of 63 bytes as 0x3F. This became very clear when I tried sending a long string and the output was impossible to decode.

Speaking of decoding, my original intention was to just loop together the CP2110 device with a CH341 I bought a few years ago, and have them loop data among each other to validate that they work. Somehow this plan failed: I can get data from the CH341 into the CP2110 and it decodes fine (using picocom for the CH341, and Silicon Labs own binary for the CP2110), but I can’t seem to get the CH341 to pick up the data sent through the CP2110. I thought it was a bad adapter, but then I connected the output to my Saleae Logic16 and it showed the data fine, so… no idea.

The current status is:

I know the CH341 sends out a good signal;
I know the CP2110 can receive a good signal from the CH341, with the Silicon Labs software;
I know the CP2110 can send a good signal to the Saleae Logic16, both with the Silicon Labs software and my tiny script;
I can’t get the CH341 to receive data from the CP2110.

Right now the state is still very much up in the air, and since I’ll be travelling quite a bit without a chance to bring with me the devices, there probably won’t be any news about this for another month or two.

Oh and before I forget, Rich Felker gave me another interesting idea: CUSE (Character Devices in User Space) is a kernel-supported way to “emulate” in user space devices that would usually be implemented in the kernel. And that would be another perfect application for this: if you just need to use a CP2110 as an adapter for something that needs to speak with a serial port, then you can just have a userspace daemon that implements CUSE, and provide a ttyUSB-compatible device, while not requiring short-circuiting the HID and USB-Serial subsystems.

Reverse Engineering and Serial Adapter Protocols

In the comments to my latest post on the Silicon Labs CP2110, the first comment got me more than a bit upset because it was effectively trying to mansplain to me how a serial adapter (or more properly an USB-to-UART adapter) works. Then I realized there’s one thing I can do better than complain and that is providing even more information on this for the next person who might need them. Because I wish I knew half of what I know now back when I tried to write the driver for ch314.

So first of all, what are we talking about? UART is a very wide definition for any interface that implements serial communication that can be used to transmit between a host and a device. The word “serial port” probably bring different ideas to mind depending on the background of a given person, whether it is mice and modems connected to PCs, or servers’ serial terminals, or programming interfaces for microcontrollers. For the most part, people in the “consumer world” think of serial as RS-232 but people who have experience with complex automation systems, whether it is home, industrial, or vehicle automation, have RS-485 as their main reference. None of that actually matters, since these standards mostly deal with electrical or mechanical standards.

As physical serial ports on computer stopped appearing many years ago, most of the users moved to USB adapters. These adapters are all different between each other and that’s why there’s around 40KSLOC of serial adapters drivers in the Linux kernel (according to David’s SLOCCount). And that’s without counting the remaining 1.5KSLOC for implementing CDC ACM which is the supposedly-standard approach to serial adapters.

Usually the adapters are placed either directly on the “gadget” that needs to be connected, which expose a USB connector, or on a cable used to connect to it, in which case the device usually has a TRS or similar connectors. The TRS-based serial cables appeared to become more and more popular thanks to osmocom as they are relatively inexpensive to build, both as cables and as connectors onto custom boards.

Serial interface endpoints in operating systems (/dev/tty{S,USB,ACM}* on Linux, COM* on Windows, and so on) do not only transfer data between host and device, but also provides configuration of parameters such as transmission rate and “symbol shape” — you may or may not have heard references to something like “9600n8” which is a common way to express the transmission protocol of a serial interface: 9600 symbols per second (“baud rate”), no parity, 8-bit per symbol. You can call these “out of band” parameters, as they are transmitted to the UART interface, but not to the device itself, and they are the crux of the matter of interacting with these USB-to-UART adapters.

I already wrote notes about USB sniffing, so I won’t go too much into detail there, but most of the time when you’re trying to figure out what the control software sends to a device, you start by taking a USB trace, which gives you a list of USB Request Blocks (effectively, transmission packets), and you get to figure out what’s going on there.

For those devices that use USB-to-UART adapters and actually use the OS-provided serial interface (that is, COM* under Windows, where most of the control software has to run), you could use specialised software to only intercept the communication on that interface… but I don’t know of any such modern software, while there are at least a few well-defined interface to intercept USB communication. And that would not work for software that access the USB adapter directly from userspace, which is always the case for Silicon Labs CP2110, but is also the case for some of the FTDI devices.

To be fair, for those devices that use TRS, I actually have considered just intercepting the serial protocol using the Saleae Logic Pro, but beside being overkill, it’s actually just a tiny fraction of the devices that can be intercepted that way — as the more modern ones just include the USB-to-UART chip straight onto the device, which is also the case for the meter using the CP2110 I referenced earlier.

Within the request blocks you’ll have not just the serial communication, but also all the related out-of-band information, which is usually terminated on the adapter/controller rather than being forwarded onto the device. The amount of information changes widely between adapters. Out of those I have had direct experience, I found one (TI3420) that requires a full firmware upload before it would start working, which means recording everything from the moment you plug in the device provides a lot more noise than you would expect. But most of those I dealt with had very simple interfaces, using Control transfers for out-of-band configuration, and Bulk or Interrupt¹ transfers for transmitting the actual serial interface.

With these simpler interfaces, my “analysis” scripts (if you allow me the term, I don’t think they are that complicated) can produce a “chatter” file quite easily by ignoring the whole out of band configuration. Then I can analyse those chatter files to figure out the device’s actual protocol, and for the most part it’s a matter of trying between one and five combinations of transmission protocol to figure out the right one to speak to the device — in glucometerutils I have two drivers using 9600n8 and two drivers using 38400n8. In some cases, such as the TI3420 one, I actually had to figure out the configuration packet (thanks to the Linux kernel driver and the datasheet) to figure out that it was using 19200n8 instead.

But again, for those, the “decoding” is just a matter to filtering away part of the transmission to keep the useful parts. For others it’s not as easy.

0029 <<<< 00000000: 30 12                                             0.

0031 <<<< 00000000: 05 00                                             ..

0033 <<<< 00000000: 2A 03                                             *.

0035 <<<< 00000000: 42 00                                             B.

0037 <<<< 00000000: 61 00                                             a.

0039 <<<< 00000000: 79 00                                             y.

0041 <<<< 00000000: 65 00                                             e.

0043 <<<< 00000000: 72 00                                             r.

This is an excerpt from the chatter file of a session with my Contour glucometer. What happens here is that instead of buffering the transmission and sending a single request block with a whole string, the adapter (FTDI FT232RL) sends short burts, probably to reduce latency and keep a more accurate serial protocol (which is important for device that need accurate timing, for instance some in-chip programming interfaces). This would be also easy to recompose, except it also comes with

0927 <<<< 00000000: 01 60                                             .`

0929 <<<< 00000000: 01 60                                             .`

0931 <<<< 00000000: 01 60                                             .`

which I’m somehow sceptical they come from the device itself. I have not paid enough attention yet to figure out from the kernel driver whether this data is marked as coming from the device or is some kind of keepalive or synchronisation primitive of the adapter.

In the case of the CP2110, the first session I captured starts with:

0003 <<<< 00000000: 46 0A 02                                          F..

0004 >>>> 00000000: 41 01                                             A.

0006 >>>> 00000000: 50 00 00 4B 00 00 00 03  00                       P..K.....

0008 >>>> 00000000: 01 51                                             .Q

0010 >>>> 00000000: 01 22                                             ."

0012 >>>> 00000000: 01 00                                             ..

0014 >>>> 00000000: 01 00                                             ..

0016 >>>> 00000000: 01 00                                             ..

0018 >>>> 00000000: 01 00                                             ..

and I can definitely tell you that the first three URBs are not sent to the device at all. That’s because HID (the higher-level protocol that CP2110 uses on top of USB) uses the first byte of the block to identify the “report” it sends or receives. Checking these against AN434 give me a hint of what’s going on:

report 0x46 is “Get Version Information” — CP2110 always returns 0x0A as first byte, followed by a device version, which is unspecified; probably only used to confirm that the device is right, and possibly debugging purposes;
report 0x41 is “Get/Set UART Enabled” — 0x01 just means “turn on the UART”;
report 0x50 is “Get/Set UART Config” — and this is a bit more complex to parse: the first four bytes (0x00004b00) define the baud rate, which is 19200 symbols per second; then follows one byte for parity (0x00, no parity), one for flow control (0x00, no flow control), one for the number of data bits (0x03, 8-bit per symbol), and finally one for the stop bit (0x00, short stop bit); that’s a long way to say that this is configured as 19200n8.
report 0x01 is the actual data transfer, which means the transmission to the device starts with 0x51 0x22 0x00 0x00 0x00 0x00.

This means that I need a smarter analysis script that understands this protocol (which may be as simple as just ignoring anything that does not use report 0x01) to figure out what the control software is sending.

And at the same time, it needs code to know how “talk serial” to this device. Usually the out-of-bad configuration is done by a kernel driver: you ioctl() the serial device to the transmission protocol you need, the driver sends the right request block to the USB endpoint. But in the case of the CP2110 device, there’s no kernel driver implementing this, at least per Silicon Labs design: since HID devices are usually exposed to userland, and in particular to non-privileged applications, sending and receiving the reports can be done directly from the apps. So indeed there is no COM* device exposed on Windows, even with the drivers installed.

Could someone (me?) write a Linux kernel driver that expose CP2110 as a serial, rather than HID, device? Sure. It would require fiddling around with the HID subsystem a bit to have it ignore the original device, and that means it’ll probably break any application built with Silicon Labs’ own development kit, unless someone has a suggestion on how to have both interfaces available at the same time, while it would allow accessing those devices without special userland code. But I think I’ll stick with the idea of providing a Free and Open Source implementation of the protocol, for Python. ~~And maybe add support for it to pyserial to make it easier for me to use it.~~ pyserial support CP2110 devices since commit 8b24cbb6131a97a3e91aabe5299c834a75ae8964.

¹ l these terms make more sense if you have at least a bit of knowledge of USB works behind the scene, but I don’t want to delve too much into that.

Sometimes they come back

Sometimes things from your past might come back to your present. They have been waiting in your future for a long time, just to make you wonder how the hell did you met them once again.

No, not talking about anything philosophical here. While I had some of these encounters before (for instance like meeting an ex-classmate after about ten years when doing driving practices).

Today kernel 2.6.24 was released, and during make oldconfig, I’ve seen support for the Winchiphead (WCH) CH341 USB to serial adapter. Which is the chip found on one of the four USB to serial adapters I bought from Hong-Kong last year and for which I wanted to try writing a driver (see also this).

After almost an year, I gained a new USB to serial adapter. This is not a bad thing to me.

Sheer luck, or lack of?

Okay, so today I spent the whole day following a bug for what I’m handling for my job, and when evening came, I was tired and in need of something that relaxed me.

Missing still the information to actually start taking seriously the idea of building model ships, and missing a jigsaw (I looked for some the other day at the supermarket, but I can’t find a subject that I like, most of them are paintings of the Renaisance, but I’m not really into that kind of art, I never liked studying it when I was at school, but I admit that graphic arts are not really something I’m into in general, but if it had to be a painting, I would have preferred finding something from Magritte.. I still remember seeing one of his works at the Guggenheim Musem here in Venice some time ago and being quite impressed; surrealism can be a kind of graphic art that I make myself like, it’s actually interesting), my best choice to get a relaxing night was to start looking for the W.ch USB serial converter driver.

I installed the only Windows I still have a license of (Windows 98) on a virtual machine, and set up usbmon in the kernel so that I could use it (it would be nice if the usbmon module depended on the debugfs support, as the latter is needed to use the former), and then started fighting to get the device up. After some usual Win98 driver setup (install driver, insert peripheral, remove driver, remove peripheral, re-insert peripheral, install driver, reboot, curse, remove peripheral, remove driver and so on), I’ve started having a working COM3 port with the adapter. Too bad that anything I tried to type was sent two or more times over the wires, but I knew VMware was far from perfect with USB.

Anyway, I started looking at the output of the usbmon tracing, and I was able to isolate some of the control messages used to set the parameters up (that are basically the only thing I need, as the actual I/O seems to happen just like PL2303 code (or usbserial) handles it); I’ve got the commit sequence and the okay string that I should expect from the driver, but I still have to make sense of all of them.

Unfortunately the USBmon (note the case) utility is written in Java. I say unfortunately simply because I still can’t get Sun’s JRE to work with XCB-enabled libX11, so I can’t run the utility on my box.

After some tries, which finally resulted in an hard lockup of Enterprise, which is now shut down waiting for me resuming it tomorrow, I finally decided that the next days I’ll start by writing some tools that might help me during the development of the driver, and maybe someone else in the future.

The first tool will be a serial signal generator: basically it would be a simple app that repeated the same string over and over and over on a serial port, using the configured baud rate, bits, parity and stop bits. This is useful because I can connect the adapter in look with my real serial port, leave the generator open on the serial port, and try sending different commands to the adapter till I get the signal as I need it, then I can move to try another parameter, for instance changing the stop bits, and try the commands again until I get the same parameters as configured in the generator. Right now I’ve used for this a simple C source file that sent “Flameeyes” string with the hardcoded parameters, I’d like to be able to set the parameters at runtime so you don’t have to stop and start it every time, but I’m not yet sure of which interface I could use for that.

The second tool is the one that most likely will be helpful if I can get it as good as I hope, and it would be a Ruby-based ncurses frontend for reading the data out of usbmon. First it would show the data in a more structured fashion, with the list of read packets that can be actually inspected without having to copy them around, with a ringbuffer for the read packets, and a different buffer for particular packets you want to inspect while continuing recording, and if I can with some kind of filters so that I can pinpoint the request for the single device, and then start discarding stuff like the I/O operations I don’t care about while trying to set the control stuff.

It’s far from being an easy task (especially since I never used ncurses), and I know I probably won’t be able to complete it as I want to, but at least I can try to provide a start if someone else will need to read that usbmon data in a more comprehensible shape. Right now SnoopyPro on Windows has a simpler view of the data, although it can really improve, especially to get a decent output to print and study.

But now, to actually make some sense out of the title I used for this blog, I have to tell you what happened while I was debugging. When I was ready to start messing with the parameters of the device, I wanted to be able to get no serial connection, and just start with the pristine adapter status, so I took one of the three remaining adapters (I ordered four from the same place) and connected it to Enterprise, launched picocom (that I started using as a test – you can find it in my overlay if you want it) on the ttyUSB0 device and.. found it not being there. An lsusb run later I seen that the adapter I just took was actually a PL2303-based device, and after looking at the other two adapters, I discovered that out of the four adapters I bought, only one has actually the W.ch chip on itself, and that is the one I chosen when I wanted to try them out, luck or lack of it?

But anyway I won’t stop working on the driver, I’ll probably take it more easy though, as three serial ports are enough for now, I ordered one more “just in case”, and that proven a good idea. I have my job to complete, so I’ll dedicate to the driver only my free time, compatibly with the other projects I’m working on. Not only I can’t really try to get the adapter replaced (I ordered it from an Hong Kong reseller on e-bay, the price to sending it back would probably be higher than I paid for it, besides I can’t really complain, the adapters were cheap and they never told me which chip they used), but I have no intention to: writing a driver for this adapter is an interesting challenge, it can be useful to other people, might be useful to me again in the future (I might need more serial adapters, and I’ll probably just order them as I ordered these ones, so I might find more W.ch-based devices), and it’s a nice way to start kernel hacking, as it’s a simple enough device.

What will I do next

So, quite a bit of people asked me what I’ll be doing now that I left Gentoo.

For sure, I won’t be bored. I have xine to work on (I’ve cleaned up some stuff in the past days already), I’ve Rust to complete, and a few Rust extensions to work on.

I have my job, that will take me more time now anyway because it’s coming to the end of the contract, so I was already planning into going in away from Gentoo for the time needed to complete the job.

I have the SPARC64 problem to debug, now that I put the Intel card in it the network performance is decent and I can probably start looking more into kgdb.

Today I also got something new to work on: not sure if I blogged about this before, but last month I ordered four USB to serial converters from eBay from an Hong Kong shop, they were cheap and I needed quite a bit of them, and I received them today. I hoped they were PL2303 based as they seemed identical to a similar product in an Italian shop, in Photo. Unfortunately, they are not. The chip used is from a Chinese manufacturer WCH, I have some Windows drivers for it, but that’s it. I’ve sent a mail to the company, in hope they can provide me some specs about these, they already got spec of some of their chips on the site, although in Chinese; if they can provide me that, I’ll see to write a driver for these, as, well, I need them, and it seems to be an easy task to start looking at kernel’s internals, and I might buy a few more of these, because serial ports aren’t really that common nowadays, and they are quite needed when working with embedded stuff.

As I said there is a lot of things for me to do, and even if I still have to find anything about building model ships (that is something I’d like to do, I like the results, and it’s something that would take me enough time so that I can relax myself), I’ll probably try to find something to do away from the keyboard just to relax myself.

And maybe, now that stuff seems to have started moving, in two or three months I might consider coming back. Not now for sure, and not too soon, but if the currently problems will be sorted out, I might consider it for the future. In the mean time I’ll still be around anyway, as an user contributing patches when it seems fit.

Update (2017-04-29): Looks like WCH is gone. On the other hand, someone else already contributed a ch314 driver, so I never actually implemented this myself.

Writing your own serial terminal

Today I wasn’t doing much Gentoo work, because I was at a meeting with my employer (hi Carlo :) ), where I picked up some stuff to do that requires me to use a serial console (without going in further details). Beside the fact that enterprise only have one serial port, which means I now have to choose between three different devices to connect to it, it’s time for me to use massively serial terminals during the day, so it’s worth spending some of my free time to make the work easier.

I never had a really good experience with terminal emulators, in particular minicom always was terrible to configure, and it sends the AT commands every time; this time it also does not work (does not write to the serial port), and I’ve been told it does not work with USB serial converters (that I need). Cutecom is not good either, because it’s a line-based emulator and does not allow me to send BREAK signals (for Klothos console).

So I decided to spend my free time out of job doing something that would make my job easier, funnier, and might be useful to others… writing a new serial terminal emulator, for KDE!

I took Konsole code, and started replacing PTY and processes with TTYs, the result of about an hour of hacking is a Konsole that can actually receive the data out of the serial console (not yet send though).

Update (2016-04-29): Unfortunately I cannot find the screenshot anywhere for this one, so it’s gone.

I’ve mailed Robert Knight, the new Konsole maintainer, as it’d be nice to me if Konsole would be able to work as a serial emulator, but I really doubt for 3.5 series that is going to happen; for this reason, I’ll be working on a serial terminal emulator that forks off Konsole code, and only supports serial ports. As Konsole is the German for Console, I’m gonna call it Serielle Konsole, guess what is the German for? :)

I know it’s not exactly the simplest of the projects, but luckily most of the complex code (displaying and parsing) is already written on Konsole, I just need to replace the settings, remove the bookmarks (no CWD to bookmark anyway), replace the way Sessions work (I will use different sessions for the various serial consoles, of course), and hook correctly the serial support.

It’s not that much of a difficult task actually, because I looked up most of what is needed to write a basic serial terminal emulator for my High School graduation project (it’s pretty dumb but it did the job to show my teachers what I was able to do — and it was now three or four years ago. Who said that High School is useless?)