Controlling Your TV via (Circuit)Python

This is yet another of the pre-announced projects, and possibly one of the most procrastinated ones. I own a Sony Bravia TV I bought in Ireland in 2013, and is still working very well for our needs (we don’t watch that much TV). It’s connected to a Yamaha RX-V475 receiver on one input and a cheap HDMI switch on the other input, because there’s too many devices, although we only use three or four of them most of the time: Chromecast, Portal TV, HTPC, and PlayStation 4. They are split equally between the two inputs. So far, so good.

The problem starts with the fact that sometimes if the TV is turned on by the Chromecast or the Portal, the ARC does not initialize properly, and we hear no audio. The solution is worthy of The IT Crowd: tell the TV to use the internal speakers, then tell it to use the external speakers again — turn off and on the ARC itself. It’s annoying and takes a few keypresses.

What I have been wanting for a while is a way to ask Assistant (or Alexa) to “unfuck the TV” — that is to reset the audio channel for us, recording a macro to do that ARC off/on dance. It was for this reason I bought the Feather M4 last year, but I only ended up starting to work on it just this month.

To make this possible, the first thing I needed was to know the commands sent by the remote, and have a way to replicate them back to the TV. I already had some experience with infrared receivers as, a long time ago in a different life, I maintained the LIRC patchset for Linux 2.6, for a while. I even tried writing Qt (3) bindings for LIRC. I wonder if I can find the source code anywhere. But that was not as useful as I thought.

Thankfully, Ken Shirriff wrote good documentation on the protocol, and linked to further details, and even to a full archive of all the Sony command codes. Which made my life much easier, but not completely easy. While Adafruit has an IRRemote library, it does not correctly interpret Sony SIRC commands. I considered adding the support directly into it, but it turned out to be a bit more invasive than expected, so I ended up instead writing a separate package that included both the decoder and the encoder (this was before I started the new job, so releasing it was easy — but now I’m having to wait a little bit to continue on it.)

Now, once I managed to decode the commands, I need to be able to send them. And here’s where things get interesting. What we usually refer to as commands are a bunch of bits. These are encoded, based on the protocol, as a series of pulses, which are modulated on top of a carrier wave with a certain frequency.

Unfortunately, it turns out that talking to a Sony TV is nowhere near as easy as it might sound. Let’s try to figure out what’s going on by providing a bit of vocabulary. An infrared signal as used by most remote controls usually carries a command and, in most cases, an address to specify which device should take the command (since they are usually used in place where there are multiple devices using infrared remote controls). These are encoded in bits according to the rules of the protocol, and then converted to pulses. These pulses are then applied to a carrier wave of a certain frequency, which defines the speed at which the infrared LED is “blinking” on and off. The LED itself has a wavelength which represent the “colour” of the light in the infrared spectrum.

For Sony’s SIRC, the TV expects a 40kHz carrier, and it seems a 950nm wavelength. It seems like you can use 940nm LEDs but they have worse reception, and only works if they are bright enough. The first batch of LEDs I bought, as well as the pre-soldered module from DFRobot, turned out to not be bright enough for my TV to recognize — so I decided to try with desoldering the LED from a replacement remote control from eBay, which worked fine, so I thought I needed 950nm LEDs — but no, it works with good enough 940nm LEDs, just not with the tiny ones I originally bought from Mouser.

So once I had a way to send arbitrary Sony commands to my TV, I started looking for options to trigger the ARC reset — unfortunately this is proving more complicated than I expected: there’s no command that I could send that would provide me with the ARC menu. Instead I can only bring up the Sync menu reliably — but that menu has different entries depending on whether the selected input has working HDMI CEC, which is extremely annoying.

On the other hand, I did find commands that select directly the different inputs directly, instead of showing up the input selection menu and choosing from there. Which gave me a different idea to start with: while I haven’t given up on the macro for fixing the audio, what I can do is to script input selection across the two-level mux.

I started by wondering if I could send the IR command to the HDMI switcher as well, so that I could select between those two easily — that turned out to be another pile of yaks to shave. The switcher uses the NEC protocol, which has a 38kHz carrier wave, but that turned out not to matter as much (the decoder it uses seem to accept 40kHz just as well) — instead I had a hard time to get it to receive the command because it expected a NEC “repeat signal” to seal the command. I guess that’s going to be a blog post in and by itself.

Now, my original plan was to get something running on the Feather, attach an AirLift wing to give it WiFi, and control that… somehow. I also considered re-coding this with ESP32 and ESPHome, despite it not having an obvious way to send SIRC commands while making sense — it doesn’t represent the commands the way the protocol expects, and the most reasonable way I could find was to generate the pulse sequence, and just sending that raw.

But then I thought it over and realised that, at least for the moment, it makes more sense for me to use an USB-to-GPIO bridge, and control this from a full blown computer — the reason for that is that I would want to be able to open a webpage on my phone, or my wife’s, and select the right input altogether. And while there’s a Python module for controlling the receiver, using that from CircuitPython or MicroPython would probably take some work. And even though I could just control the Feather remotely, via MQTT or USB-Serial, it would still likely be more work than driving this all from the same Flask app locally.

Unfortunately, I don’t have code to show you yet. While my new dayjob has much simpler policies than the last, I need to complete the training before I can start releasing new projects. Hopefully next update will come with code and demos.

Programming Languages are Tools

Recently, I shared a long Twitter Thread while I went through a bunch of GitHub repositories that I officially archived — they are still all available of course, but I’m not caring to take issues, or pull request; if you happen to bump into something you would care to use, feel free to fork and use it.

As these projects come from many different “eras” of my career, they all are different in style, license, and programming language used. I already have written before about my current take in licensing, so I don’t think there’s any need for me to go through more of it right now. But I didn’t really spend much time talking about languages on the blog, so I thought I would at least share my point of view with this.

I started programming with BASIC, on C64, GW-BASIC, QBasic, and eventually Visual Basic 5 CCE. I went on to learn C++ for high school, I tried (and failed) learning Pascal. I learnt PHP because that was what seemed cool, and I couldn’t get into Java at first at all. I learnt to appreciate “good old C”, and Shell. I failed at my first two attempts to get into Python, ended up picking up Ruby, and eventually C#. For a job I ended up digging deep into ActionScript (Flash’s programming language), and some proprietary language that was attached to the RDBMS that they had been using. To make my life easier, I learnt Perl (in 2012!) to extend Munin. Then I arrived at the Bubble with no knowledge of Python and was thrown in possibly the most Python-heavy team in Dublin at the time.

I like Python, but I also liked most of the other languages I worked with. But I look at languages like tools, and sometimes one tool is better than another because of the features, sometimes because it’s the one you have the most muscle memory with, and sometimes because it’s the one that is already in front of you.

As I write this post I have not started my next dayjob, and I have no idea what programming language I’ll end up using day to day. It might even be JavaScript, which I have next to no experience with. I’m not going to be picky — as long as it’s not a functional programming language, it’s just going to be a bit of work to get used to a new language: syntax, mindset, frameworks, libraries, … Not a walk in the park, but I find it part and parcel of a job.

And because I have changed my “dayjob language” a few times in the past few years, the language that my FLOSS projects were written in tended to change in sync — because that would be the language I’d have the freshest memory in. As I said, I used a ton of Python in the Bubble, and that’s why you can see me releasing a ton of Python projects now. If I had to write something in Ruby, I would have to go back and figure out how much the language changed, and what the current best practices are — this is also true for all of my old projects: if I were to pick up ruby-elf again today, it would probably take me a few days even just to figure out how Ruby evolved since seven years ago (the last time I touched it). If the project didn’t have as much code and test and complexities built in, it would probably be faster to just rewrite the whole thing into a language I have more confidence in. Which probably shows why there’s so much stuff I abandoned over the years.

Of course there are developers who focus on one language and know all of its inside out. They are the ones that make languages possible. But that is not what I do, it’s not my dayjob, and I am much more likely to just accept what the status quo for a project is, and adapt. I can only think once of having rewritten something because of its language — and ironically, that was rewriting Python in Perl. Although in that case, I think the problem was less about the language (despite my ranting back then — little did I know that a few months later I would be forced to learn Python), and more about the language providing a flat battery.

I find that the same is true from a company/employer/project owner point of view — languages are tools. And that means sometimes you stick to the tools you have, and not “buy” more tools (in the form of hiring experts or spending on training), while sometimes you don’t care which tools are used in the shop as long as the results are valid. When you have to build some software that a team has to maintain, for instance, you may not want to introduce a new language just because it’s the preferred language of one of the programmers — even if the language is perfectly fine by itself, you need to consider the cost of relying on a single person being the “expert” in that language.

As a counterexample — before the bubble I was working for a small company (not quite a startup, particularly because it didn’t have any venture capital pouring in, so it did spare expenses). The new product was being developed using the same proprietary language used for a previous breadwinner product. The language was clunky, modelled after early PHP, and hard to wire in with modern HTML, let alone JavaScript (this was 2012). Getting anyone to work on that product would require a significant amount of training, which is why the company owner was the one doing nearly all of the work on it (I did the minimum possible on it myself). Replacing the language would have required re-training the owner to some new language (since they didn’t really know any of the possible alternative languages) but since this was 2012, I kept arguing that it would be significantly cheaper to hire one or two junior developers to reimplement the web side of the product in Rails, leaving the owner working on the Flash application instead — particularly because at the time there wasn’t really much of a web part of the product and reimplementing it would have costed nearly nothing by comparison.

This does not mean that developing new languages is not useful or important. Or that the differences between languages don’t matter. I really enjoy writing Python — but if I needed to write something that is very high performance I wouldn’t go and use Python for it. I’m enjoying CircuitPython as it makes quick prototyping awesome, but I also understand it’s a limitation, as it needs more expensive components to run, and if every cent of your BOM counts, it might be a bad choice.

I also am holding hopes for Rust to become useful as a “base system language” to replace good old C in a bunch of places — if nothing else because removing entire classes of mistakes would be nice. But that is also a compromise: it might introduce new classes of mistakes, and it will have side effects on bootstrapping new architectures. I never expect any migration not to have any costs.

Don’t Ignore Windows 10 as a Development Platform for FLOSS

Important Preface: This blog post was written originally on 2020-05-12, and scheduled for later publication, inspired by this short Twitter thread. As such it well predates Microsoft’s announcement of expanding support of WSL2 to graphical apps. I considered trashing, or seriously re-editing the blog post in the light of the announcement, but I honestly lack the energy to do that now. It left a bad taste in my mouth to know that it will likely get drowned out in the noise of the new WSL2 features announcement.

Given the topic of this post I guess I need to add a preface to point out my “FLOSS creds” — because I have seen already too many attacks to people who even use Windows at all. I have been an opensource developer for over fifteen years now, and part of the reason why I left my last bubble was because it made it difficult for me to contribute to various opensource projects. I say this because I’m clearly a supporter of Free Software and Open Source, wherever possible. I also think that’s different people have different needs, and that ignoring that is a failure of the FLOSS movement as a whole.

The “Year of Linux on the Desktop” is now a meme that has been running its course to the point of being annoying. Despite what FLOSS advocates keep saying, “Linux on the Desktop” is not really moving, and while I do have some strong opinions on this, that’s for another day. Most users, and in particular newcomers to FLOSS (both as users and developers) are probably using a more “user friendly” platform — if you leave a comment with the joke on UNIX being selective with its friends, you’ll end up on a plonkfile, be warned.

About ten years ago, it seemed like the trend was for FLOSS developers to use MacBooks as their daily laptops. I did that for a while myself — an UNIX-based platform with all the tools of the trade, which allowed quite a bit of work being done without having access to a Linux platform. SSH, Emacs, GCC, Ruby, and so on. And at the same time, you had the stability of Mac OS X, with the battery life and all the hardware worked great out of the box. But then more recently, Apple’s move towards “walled gardens” seemed to be taking away from this feasibility.

But back to the main topic. Over the past many years, I’ve been using a “mixed setup” — using a Linux laptop (or more recently desktop) for development, and a Windows (7, then 10) desktop for playing games, editing photos, designing PCBs, and for logic analysis. The latter is because Saleae Logic takes a significant amount of RAM when analysing high-frequency signals, and I have been giving my gamestations as much RAM as I can just for Lightroom, so it makes sense to run it on the machine with 128GB of RAM.

But more recently I have been exploring the ability of using Windows 10 as a development platform. In part because my wife has been learning Python, and since also learning a new operating system and paradigm at the same time would have been a bloody mess, she’s doing so on Windows 10 using Visual Studio Code and Python 3 as distributed through the Microsoft Store. While helping her, I had exposure to Windows as a Python development platform, so I gave it a try when working on my hack to rename PDF files, which turned out to be quite okay for a relatively simple workflow. And the work on the Python extension keeps making it more and more interesting — I’m not afraid to say that Visual Studio Code is better integrated with Python than Emacs, and I’m a long-time user of Emacs!

In the last week I have actually stepped up further how much development I’m doing on Windows 10 itself. I have been using HyperV virtual machines for Ghidra, to make use of the bigger screen (although admittedly I’m just using RDP to connect to the VM so it doesn’t really matter that much where it’s running), and in my last dive into the Libre 2 code I felt the need to have a fast and responsive editor to go through executing part of the disassembled code to figure out what it’s trying to do — so once again, Visual Studio Code to the rescue.

Indeed, Windows 10 now comes with an SSH client, and Visual Studio Code integrates very well with it, which meant I could just edit the files saved in the virtual machine and have the IDE also build them with GCC and executing them to get myself an answer.

Then while I was trying to use packetdiag to prepare some diagrams (for a future post on the Libre 2 again), I found myself wondering how to share files between computers (to use the bigger screen for drawing)… until I realised I could just install the Python module on Windows, and do all the work there. Except for needing sed to remove an incorrect field generated in the SVG. At which point I just opened my Debian shell running in WSL, and edited the files without having to share them with anything. Uh, score?

So I have been wondering, what’s really stopping me from giving up my Linux workstation for most of the time? Well, there’s hardware access — glucometerutils wouldn’t really work on WSL unless Microsoft is planning a significant amount of compatibility interfaces to be integrated. Similar for using hardware SSH tokens — despite PC/SC being a Windows technology to begin with. Screen and tabulated shells are definitely easier to run on Linux right now, but I’ve seen tweets about modern terminals being developed by Microsoft and even released FLOSS!

Ironically, I think it’s editing this blog that is the most miserable experience for me on Windows. And not just because of the different keyboard (as I share the gamestation with my wife, the keyboard is physically a UK keyboard — even though I type US International), but also because I miss my compose key. You may have noticed already that this post is full of em-dashes and en-dashes. Yes, I have been told about WinCompose, but last time I tried using it, it didn’t work and even screwed up my keyboard altogether. I’m now trying it again, at least on one of my computers, and if it doesn’t explode in my face again, I may just give it another try later.

And of course it’s probably still not as easy to set up a build environment for things like unpaper (although at that point, you can definitely run it in WSL!), or to have a development environment for actual Windows applications. But this is all a matter of different set of compromises.

Honestly speaking, it’s very possible that I could survive with a Windows 10 laptop for my on-the-go opensource work, rather than the Linux one I’ve been using. With the added benefit of being able to play Settlers 3 without having to jump through all the hoops from the last time I tried. Which is why I decided that the pandemic lockdown is the perfect time to try this out, as I barely use my Linux laptop anyway, since I have a working Linux workstation all the time. I have indeed reinstalled my Dell XPS 9360 with Windows 10 Pro, and installed both a whole set of development tools (Visual Studio Code, Mu Editor, Git, …) and a bunch of “simple” games (Settlers, Caesar 3, Pharaoh, Age of Empires II HD); Discord ended up in the middle of both, since it’s actually what I use to interact with the Adafruit folks.

This doesn’t mean I’ll give up on Linux as an operating system — but I’m a strong supporter of “software biodiversity”, so the same way I try to keep my software working on FreeBSD, I don’t see why it shouldn’t work on Windows. And in particular, I always found that providing FLOSS software on Windows a great way to introduce new users to the concept of FLOSS — focusing more on providing FLOSS development tools means giving an even bigger chance for people to build more FLOSS tools.

So is everything ready and working fine? Far from it. There’s a lot of rough edges that I found myself, which is why I’m experimenting with developing more on Windows 10, to see what can be improved. For instance, I know that the reuse-tool has some rough edges with encoding of input arguments, since PowerShell appears to still not default to UTF-8. And I failed to use pre-commit for one of my projects — although I have not taken notice yet much of what failed, to start fixing it.

Another rough edge is in documentation. Too much of it assumes only a UNIX environment, and a lot of it, if it has any support for Windows documentation at all, assumes “old school” batch files are in use (for instance for Python virtualenv support), rather than the more modern PowerShell. This is not new — a lot of times modern documentation is only valid on bash, and if you were to use an older operating system such as Solaris you would find yourself lost with the tcsh differences. You can probably see similar concerns back in the days when bash was not standard, and maybe we’ll have to go back to that kind of deal. Or maybe we’ll end up with some “standardization” of documentation that can be translated between different shells. Who knows.

But to wrap this up, I want to give a heads’ up to all my fellow FLOSS developers that Windows 10 shouldn’t be underestimated as a development platform. And that if they intend to be widely open to contributions, they should probably give a thought of how their code works on Windows. I know I’ll have to keep this in mind for my future.

Code Validation and Reviews

During my “staycation” week I decided to spend some time looking at my various Python projects, trying to make it easier, nicer and cleaner both for contributors and for myself. I also decided to spend some time to improve some of the modules I use, based on various lessons I learned in my own tools — and that got me to learn more about those tools.

First of all, thanks to Ben, glucometerutils has been fully re-formatted with black and isort already. And he set it up with pre-commit to make sure that new changes don’t break these formattings. This is awesome.

As I was recently discussing with Samuele on Twitter, it’s not that I always agree with the formatting choices in black. It’s that it takes away the subjective matter of agreement on formatting. Black does not always make the code look like the way I would make it look like, and I could argue that I can do a better job than black. But it’s also true that it makes everybody’s code look the same, which is actually a great way to fix it.

This is something I definitely picked up in my dayjob — I have been part of the Python Readability program (you can find more about it in the Building Secure and Reliable Systems book, downloadable for free), and through that I have reviewed literally hundreds of changes, coming from all different parts of Alphabet. While my early engagement had lots of comments on code formatting, the moment when Python formatting tools became more reliable and their usage became widespread, the “personal load” of doing those reviews went down significantly, as I could finally focus on pointing out overly complex functions, mismatching code and documentation, and so on. For everything else, my answer was unchanging: “let the formatter do its job, maybe just add a comma there as a suggestion to it.”

This is why I’m happy about black — not that I think its formatting is 100% what I would do. But because it gets close enough, and removes the whole argument or uncertainty around it. The same applies to isort, as well.

While applying pretty much the same set of presubmits and formatting to python-pcapng, I also found out flake8. This is another useful tool to reduce the work needed for reviews, and it also can be configured to run as part of the pre-commit hooks, making sure that violations are identified sooner rather than later. While the tool is designed to deal with styleguide violations, it also turned out to identify a few outright mistakes in glucometerutils. I’m now going to apply it throughout, whenever I can.

There’s more checks I would actually want to integrate — today I was going through all the source files in glucometerutils to update the type annotations, since I dropped Python 3.6 support. As I went to do that I realised that one of the files created in a pull request I approved recently was actually missing licensing information. I have now added both license and copyright annotations as suggested by Matija (who is an actual lawyer, unlike me) — but would love a pre-commit check that just ensured that all the files have a license, copyright, and that they have the expected license, for instance.

There’s a few more trivial checks available in pre-commit that I may actually enable throughout: checks for trailing whitespace, and missing newlines at end of files. All of those are easily fixed, and the fixers do exactly that, which is also a great way to make the tests easier on newcomers and contributors: you don’t just get told that “it’s wrong”, but also “let me fix that for you already”.

It’s not quite as encompassing as the bubble I’m used to, but it seems to be the closest I’m getting to it right now. Maybe I should just start building all the hooks that I feel I need, and see if someone else will adopt them afterwards. It seems to be a common thing to do after all.

Anyone who has written Gentoo ebuilds, by the way, have most likely recognized similarities with repoman, the tool used to validate ebuilds before submitting them to the tree. I think that’s possibly why I’m so interested in this. Because I do agree that tools like repoman are the way to go, and have insisted myself for repoman to be extended to cover more and more cases over time, as it would stop divegence.

I honestly hope to get to a point where there’s no argument made over a code change, on whether it complies to style or not — but rather leaving the enforcement (and the fixing) to computers, whenever it is possible. And that also means helping the computers making it possible by being less picky on things that can be overlooked.

Planets, Clouds, Python

Half a year ago, I wrote some thoughts about writing a cloud-native feed aggregator. I actually started drawing some ideas of how I would design this myself since, and I even went through the (limited) trouble of having it approved for release. But I have not actually released any code, or to be honest, I have not written any code either. The repository has been sitting idle.

Now, with the Python 2 demise coming soon, and me not interested in keeping around a server nearly only to run Planet Multimedia, I started looking into this again. The first thing that I realized is that I both want to reuse as much code exist out there as I can, and I want to integrate with “modern” professional technologies such as OpenTelemetry, which I appreciate from work, even if it sounds like overkill.

But that’s where things get complicated: while going full “left-pad” of having a module for literally everything is not something you’ll find me happy about, a quick look at feedparser, probably the most common module to read feeds in Python, shows just how much code is spent trying to cover for old Python versions (before 2.7, even), or to implement minimal-viable-interfaces to avoid mandatory dependencies at all.

Thankfully, as Samuel from NewsBlur pointed out, it’s relatively trivial to just fetch the feed with requests, and then pass it down to feedparser. And since there are integration points for OpenTelemetry and requests, having an instrumented feed fetcher shouldn’t be too hard. That’s going to probably be my first focus when writing Tanuga, next weekend.

Speaking of NewsBlur, the chat with Samuel also made me realize how much of it is still tied to Python 2. Since I’ve gathered quite a bit of experience in porting to Python 3 at work, I’m trying to find some personal time to contribute smaller fixes to run this in Python 3. The biggest hurdle I’m having right now is to set it up on a VM so that I can start it up in Python 2 to begin with.

Why am I back looking at this pseudo-actively? Well, the main reason is that rawdog is still using Python 2, and that is going to be a major pain with security next year. But it’s also the last non-static website that I run on my own infrastructure, and I really would love to get rid of entirely. Once I do that, I can at least stop running my own (dedicated or virtual) servers. And that’s going to save me time (and money, but time is the most important one here too.)

My hope is that once I find a good solution to migrate Planet Multimedia to a Cloud solution, I can move the remaining static websites to other solutions, likely Netlify like I did for my photography page. And after that, I can stop the last remaining server, and be done with sysadmin work outside of my flat. Because honestly, it’s not worth my time to run all of these.

I can already hear a few folks complaining with the usual remarks of “it’s someone else’s computer!” — but the answer is that yes, it’s someone else’s computer, but a computer of someone who’s paid to do a good job with it. This is possibly the only way for me to manage to cut away some time to work on more Open Source software.

CP2110 Update for 2019

The last time I wrote about the CP2110 adapter was nearly a year ago, and because I have had a lot to keep me busy since, I have not been making much progress. But today I had some spare cycles and decided to take a deeper look starting from scratch again.

What I should have done properly since then would have been procuring myself a new serial dongle, as I was not (and still am) not entirely convinced about the quality of the CH341 adapter I’m using. I think I used that serial adapter successfully before, but maybe I didn’t and I’ve been fighting with ghosts ever since. This counts double as, silly me, I didn’t re-read my own post when I resumed working on this, and been scratching my head at nearly exactly the same problems as last time.

I have some updates first. The first of which is that I have some rough-edged code out there on this GitHub branch. It does not really have all the features it should, but it at least let me test the basic implementation. It also does not actually let you select which device to open — it looks for the device with the same USB IDs as I have, and that might not work at all for you. I’ll be happy to accept pull requests to fix more of the details, if anyone happen to need something like this too — once it’s actually in a state where it can be merged, I’ll be doing a squash commit and send a pull request upstream with the final working code.

The second is that while fighting with this, and venting on Twitter, Saleae themselves put me on the right path: when I said that Logic failed to decode the CP2110→CH341 conversation at 5V but worked when they were set at 3.3V, they pointed me at the documentation of threshold voltage, which turned out to be a very good lead.

Indeed, when connecting the CP2110 at 5V alone, Logic reports a high of 5.121V, and a low of ~-0.12V. When I tried to connect it with the CH341 through the breadboard full of connections, Logic reports a low of nearly 3V! And as far as I can tell, the ground is correctly wired together between the two serial adapters — they are even connected to the same USB HUB. I also don’t think the problem is with the wiring of the breadboard, because the behaviour is identical when just wiring the two adapters together.

So my next step has been setting up the BeagleBone Black I bought a couple of years ago and shelved into a box. I should have done that last year, and I would probably have been very close to have this working in the first place. After setting this up (which is much easier than it sounds), and figuring out from the BeagleBoard Wiki the pinout (and a bit of guesswork on the voltage) of its debug serial port, I could confirm the data was being sent to the CP2110 right — but it got all mangled on print.

The answer was that the HID buffered reads are… complicated. So instead of deriving most of the structure from the POSIX serial implementation, I lifted it from the RFC2217 driver, that uses a background thread to loop the reads. This finally allowed me to use the pySerial miniterm tool to log in and even dmesg(!) the BBB over the CP2110 adapter, which I consider a win.

Tomorrow I’ll try polishing the implementation to the point where I can send a pull request. And then I can actually set up to look back into the glucometer using it. Because I had an actual target when I started working on this, and was not just trying to get this to work for the sake of it.

Glucometerutils News: Continuous Integration, Dependencies, and Better Code

You may remember glucometerutils, my project of an open source Python tool to download glucometer data from meters that do not provide Linux support (as in, any of them).

While the tool started a few years ago, out of my personal need, this year there has been a bigger push than before, with more contributors trying the tool out, finding problem, fixing bugs. From my part, I managed to have a few fits of productivity on the tool, particularly this past week at 34C3, when I decided it was due time to start making the package shine a bit more.

So let’s see what are the more recent developments for the tool.

First of all, I decided to bring up the Python version requirement to Python 3.4 (previously, it was Python 3.2). The reason for this is that it gives access to the mock module for testing, and the enum module to write actual semantically-defined constants. While both of these could be provided as dependencies to support the older versions, I can’t think of any good reason not to upgrade from 3.2 to 3.4, and thus no need to support those versions. I mean, even Debian Stable has Python 3.5.

And while talking about Python versions, Hector pointed me at construct, which looked right away like an awesome library for dealing with binary data structures. It turned out to be a bit more rough around the edges than I’ve expected from the docs, particularly because the docs do not contain enough information to actually use it with proper dynamic objects, but it does make a huge difference compared to dealing with bytestring manually. I have started already while in Leipzig to use it to parse the basic frames of the FreeStyle protocol, and then proceeded to rewrite the other binary-based protocols, between the airports and home.

This may sound like a minor detail, but I actually found this made a huge difference, as the library already provides proper support for validating expected constants, as well as dealing with checksums — although in some cases it’s a bit heavier-handed than I expected. Also, the library supports defining bit structures too, which simplified considerably the OneTouch Ultra Easy driver, that was building its own poor developer version of the same idea. After rewriting the two binary LifeScan drivers I have (for the OneTouch Verio 2015 and Select Plus, and the Ultra Easy/Ultra Mini), the similarities between the two protocols are much easier to spot. Indeed, after porting the second driver, I also decided to refactor a bit on the first, to make the two look more alike.

This is going to be useful soon again, because two people have asked for supporting the OneTouch Verio IQ (which, despite the name, shares nothing with the normal Verio — this one uses an on-board cp210x USB-to-serial adapter), and I somehow expect that while not compatible, the protocol is likely to be similar to the other two. I ended up finding one for cheap on Amazon Germany, and I ended up ordering it — it would be the easier one to reverse engineer from my backlog, because it uses a driver I already known is easy to sniff (unlike other serial adapters that use strange framing, I’m looking at you FT232RL!), and the protocol is likely to not stray too far from the other LifeScan protocols, even though it’s not directly compatible.

I have also spent some time on the tests that are currently present. Unfortunately they don’t currently cover much of anything beside for some common internal libraries. I have though decided to improve the situation, if a bit slowly. First of all, I picked up a few of the recommendations I give my peers at work during Python reviews, and started using the parameterized module that comes from Abseil, which was recently released opensource by Google. This reduces tedious repetition when building similar tests to exercise different paths in the code. Then, I’m very thankful to Muhammad for setting up Travis for me, as that now allows the tests to show breakage, if there is any coverage at all. I’ll try to write more tests this month to make sure to exercise more drivers.

I’ve also managed to make the setup.py in the project more useful. Indeed it now correctly lists dependencies for most of the drivers as extras, and I may even be ready to make a first release on PyPI, now that I tested most of the devices I have at home and they all work. Unfortunately this is currently partly blocked on Python SCSI not having a release on PyPI itself. I’ll get back to that possibly next month at this point. For now you can install it from GitHub and it should all work fine.

As for the future, there are two in-progress pull requests/branches from contributors to add support for graphing the results, one using rrdtool and one using gnuplot. These are particularly interesting for users of FreeStyle Libre (which is the only CGM I have a driver for), and someone expressed interest in adding a Prometheus export, because why not — this is not as silly as it may sound, the output graphs of the Libre’s own software look more like my work monitoring graphs than the usual glucometer graphs. Myself, I am now toying with the idea of mimicking the HTML output that the Accu-Check Mobile generate on its own. This would be the easiest to just send by email to a doctor, and probably can be used as a basis to extend things further, integrating the other graphs output.

So onwards and upwards, the tooling will continue being built on. And I’ll do my best to make sure that Linux users who have a need to download their meters’ readings have at least some tooling that they can use, and that does not require setting up unsafe MongoDB instances on cloud providers.

glucometerutils news: many more meters, easier usage and Windows support

You probably have noticed by now that I write about glucometers quite a bit, not only reviewing them as an user, but also reverse engineering to figure out their protocols. This all started four years ago when I needed to send my glucometer readings to my doctor and I ended up having to write my own tool.

That tool started almost as a joke, particularly given I wrote it in Python, which at the time I was not an expert in at all (I have since learnt a lot more about it, and at work I got to be more of an expert than I’d ever expected to be). But I always known that it would be for the most part just a proof of concept. Not only exporting CSV is mostly useless, but the most important part in diabetes management software is the analysis and I don’t have any clue how to do analysis.

At first I thought I could reuse some of the implementation to expand Xavier’s OpenGlucose but it turned out that it’s not really easy for those meters that are using serial adapters or other USB devices beside the HID ones that he implemented already. Of course this does mean it would probably work fine for things like the FreeStyle Libre which I appear to have written the only Linux software to download from, but even in that case, things are more complicated.

Indeed, as I have noted here and there previously, we need a better format to export glucometer data, and in particular the data from continuous or mixed meters like the Libre. My current out format for it only includes the raw glucose readings from the meter that are not marked as errors; it does provide an unstructured text comment that tells you whether the reading is coming from the background sensor, an explicit scan or a blood sample, but it does not provide all the level of details of the original readings. And it does not expose ketone readings at all, despite the fact that most of the FreeStyle-line devices support them and I even documented how to get them. But this is a topic for a different post, I think.

On the other hand, over the past four years, the number of meters increased significantly, and I even have a few more that I only have partially reversed and not published yet. Currently there are 9 drivers, covering over a dozen meters (some meters share the same driver, either because they are just rebranded versions or simply because they share the same protocol). One is for the InsuLinx, which also loses a bunch of details, and is based off Xavier’s own reverse engineering — I did that mostly because all the modern FreeStyle devices appear to share the same basic protocol, and so writing new drivers for them is actually fairly trivial.

This would make the project an interesting base if someone feels like writing a proper UI for it. If I ever tried to look into that, I may end up just starting an HTTP server and provide everything over HMTL for the browser to render. After all that’s actually how OpenGlucose is doing things, except there is no server, and the browser is embedded. Alternatively one could just write an HTML report file out, the same way Accu-Chek Mobile does using data URLs and JavaScript bundles.

One of the most important usability changes I have added recently, though, is allowing the user not to specify the device path. When I started writing the tool, I started by looking at serial adapter based devices, which usually come with their own cable, and you just access it. The next driver was for the LBA-over-SCSI used int he OneTouch Verio, which I could have auto-detected but didn’t, and the following ones, mostly based off HID, I just expected to be given an hidraw path.

But all of this is difficult, and indeed I had more than a few people asking me which device are they meant to use, so over the past few months I adapter the drivers to try auto-detecting the devices. For the serial port based meters, the auto-detection targets the original manufacturer’s cable, so if you have a custom one, you should still pass them a path. For HID based devices, you also need the Python hidapi library because I couldn’t bother writing my own HID bus parsing on Linux…

… and the library also brings another important feature: it works on non-Linux operating systems. Indeed I now have not one but two confirmed users that managed to use the tool on Windows, for two separate FreeStyle devices (at the time of writing, the only ones implementing HID-based protocols, although I have another one in the pipeline.

Supposedly, all of this should work fine on macOS (I almost called it OS X), though the one person who contacted me trying to have it working there has been having trouble with it — I think the problem has something to do with the Python version available (I’m targetting Python 3 because I had a very hard time to do the right processing with Python 2.7). So if you want to give it a try feel free.

And yes, I’m very happy to receive pull request, so if you want to implement that HTML output I talked above about, you’re awesome and I’m looking forward to your pull request. I’m afraid I won’t be much help with the visualisation though.

Planets, feeds and blogs

You have probably noticed that last month I replaced Harvester with rawdog for Planet Multimedia. The reasons was easy to explain: Harvester requires libraries that only work with Ruby 1.8 — and while on one hand moving to Ruby 1.9 or 2 would mean being able to use the feedfetcher (the same one used by IFTTT), my attempts at updating the code to work with a more modern version of Ruby have been all failures.

Since I did not intend to be swamped with one more custom tool to maintain I turned to another standard tool to implement the aggregator, rawdog — holding my nose on the use of darcs for source control, and the name (please don’t google for it without safe search on, at work). The nice part about using this tool is that it’s packaged in Gentoo already, so it’s handled straight by portage with binary packages. Unfortunately, the default templates are terrible, and the settings non-obvious, but Luca was able to make the best out of it.

But more and more problems got obvious with time. The first is that the tool is does not respect the return codes at exit — it always returns zero (success) even if the processing was incomplete; it took me two weeks to figure out that the script failed when running in cron because the environment lacked the locale settings, as the cron logs said that everything was alright, and since I use fcron, it also did not send me any email, as I set it to mail me only for errors.

A couple of days ago, I got complains again that the Planet was not updating; again, no error in the cron logs, no error in my email. I ran the command manually, and I was told by it that Luca’s feed, on blogs.gentoo.org, was unreachable. Okay, sure. But then it did not solve itself when it came back up. Today I looked back into it and J-B’s and Rémi’s blogs feed were unreachable. Once again, no non-zero exit status, thus no mail, no error in the logs. This is not the way it should behave.

But that’s not enough. the other problem with rawdog is that it does not, by default, support generating a feed for the aggregation, like Harvester (and Planet/Venus) does. I found that Jonathan Riddell actually built a plugin for Planet KDE to generate the feed, but I haven’t tested it yet because I have not found the authoritative source of it, but just multiple copies of it in different websites. It also produces RSS feeds, rather than Atom feeds. And I’m sorry to say but Atom is much preferred, for me.

So where does it leave us? I’m not going to try fixing rawdog I’m afraid. Mostly because I don’t intend spending time with darcs. My options are either go back to Harvester and fix it to not use DBI and support Ruby 1.9, or try to adapt parts of NewsBlur – that already deal with aggregating feeds and producing new feeds – to make up an alternative to rawdog. If I am to do something like that, though, I’m most likely going to take my dear time and make it a web-configurable tool, rather than something that needs to be configured on the command line or with configuration files.

The reason for that is, very simply, that I’m growing fond of doing most of my work on a browser when I can, and this looks like a perfect solution to the problem. Even more so if you can give access to someone else to look into it — and if you can avoid storing passwords.

So, any takers to help me with this project?

Book Review: Learning Cython Programming

Thanks to PacktPub I got a chance to read an interesting book this week: Learning Cython Programming by Philip Herron (Amazon, Packt). I was curious because, as you probably noticed, after starting at my current place of work the past April, I ended up having to learn to use Python, which ended up with me writing my glucometer utilities in that language, contrarily to most of my other work, which has been done in Ruby. But that’s a topic for a different post.

First of all, I was curious about Cython; I heard the name before but never looked much into it, and when I saw the book’s title and I quickly checked what it was, my interest was definitely picked. If you haven’t looked into it either, at a quick summary it’s a code generator bridge between Python and good plain old C, wrapping the latter such that you can either make it run Python callbacks, or generate a shared object module that Python can load, and offload the computation-intensive code to a more performant language. And it looks a lot like a well-designed and well-implemented version of what I hoped to get in Ruby with Rust — no connection with Mozilla’s language with the same name.

The book is a quick starter, short and to the point, which is an approach I like. Together with the downloadable source code, it makes it a very good solution to learn Cython, and I recommend it if you’re interested. Not only it covers the obvious language itself, but it covers a wide range of use cases that show how to make good use of the options provided by Cython. It even goes on to show how to integrate it in a build system (although I have some reserves on the Autotools code in there, which I think I’ll send Philip a correction for).

I seriously wish I had Cython and this book when I was working on Hypnos, an Ultima OnLine «server emulator» for which I wanted to add Python-based scripting — other emulators at the time used either a very simple, almost basic-like scripting language, Perl or C#. This was before I tried to use Python for real, which turned me to hate its whitespace-based indentation. I did write some support for it but it was a long and tedious process, so I never finished it. Not only Cython would make that work much less tedious, but the book shows exactly how to add Python scripting capabilities to a bigger, C program using tmux as the example.

The book does not ignore the shortcomings of Cython of course, including the (quite clumsy) handling of exceptions when crossing the foreign language barrier. While there are still a bunch of issues to be straightened out, I think the book is fairly good at setting proper expectation for Cython. If you’re interested in the language, the book looks like the perfect fit.