You may remember glucometerutils, my project of an open source Python tool to download glucometer data from meters that do not provide Linux support (as in, any of them).
While the tool started a few years ago, out of my personal need, this year there has been a bigger push than before, with more contributors trying the tool out, finding problem, fixing bugs. From my part, I managed to have a few fits of productivity on the tool, particularly this past week at 34C3, when I decided it was due time to start making the package shine a bit more.
So let’s see what are the more recent developments for the tool.
First of all, I decided to bring up the Python version requirement to Python 3.4 (previously, it was Python 3.2). The reason for this is that it gives access to the mock module for testing, and the enum module to write actual semantically-defined constants. While both of these could be provided as dependencies to support the older versions, I can’t think of any good reason not to upgrade from 3.2 to 3.4, and thus no need to support those versions. I mean, even Debian Stable has Python 3.5.
And while talking about Python versions, Hector pointed me at construct, which looked right away like an awesome library for dealing with binary data structures. It turned out to be a bit more rough around the edges than I’ve expected from the docs, particularly because the docs do not contain enough information to actually use it with proper dynamic objects, but it does make a huge difference compared to dealing with bytestring manually. I have started already while in Leipzig to use it to parse the basic frames of the FreeStyle protocol, and then proceeded to rewrite the other binary-based protocols, between the airports and home.
This may sound like a minor detail, but I actually found this made a huge difference, as the library already provides proper support for validating expected constants, as well as dealing with checksums — although in some cases it’s a bit heavier-handed than I expected. Also, the library supports defining bit structures too, which simplified considerably the OneTouch Ultra Easy driver, that was building its own poor developer version of the same idea. After rewriting the two binary LifeScan drivers I have (for the OneTouch Verio 2015 and Select Plus, and the Ultra Easy/Ultra Mini), the similarities between the two protocols are much easier to spot. Indeed, after porting the second driver, I also decided to refactor a bit on the first, to make the two look more alike.
This is going to be useful soon again, because two people have asked for supporting the OneTouch Verio IQ (which, despite the name, shares nothing with the normal Verio — this one uses an on-board cp210x USB-to-serial adapter), and I somehow expect that while not compatible, the protocol is likely to be similar to the other two. I ended up finding one for cheap on Amazon Germany, and I ended up ordering it — it would be the easier one to reverse engineer from my backlog, because it uses a driver I already known is easy to sniff (unlike other serial adapters that use strange framing, I’m looking at you FT232RL!), and the protocol is likely to not stray too far from the other LifeScan protocols, even though it’s not directly compatible.
I have also spent some time on the tests that are currently present. Unfortunately they don’t currently cover much of anything beside for some common internal libraries. I have though decided to improve the situation, if a bit slowly. First of all, I picked up a few of the recommendations I give my peers at work during Python reviews, and started using the parameterized module that comes from Abseil, which was recently released opensource by Google. This reduces tedious repetition when building similar tests to exercise different paths in the code. Then, I’m very thankful to Muhammad for setting up Travis for me, as that now allows the tests to show breakage, if there is any coverage at all. I’ll try to write more tests this month to make sure to exercise more drivers.
I’ve also managed to make the setup.py in the project more useful. Indeed it now correctly lists dependencies for most of the drivers as extras, and I may even be ready to make a first release on PyPI, now that I tested most of the devices I have at home and they all work. Unfortunately this is currently partly blocked on Python SCSI not having a release on PyPI itself. I’ll get back to that possibly next month at this point. For now you can install it from GitHub and it should all work fine.
As for the future, there are two in-progress pull requests/branches from contributors to add support for graphing the results, one using rrdtool and one using gnuplot. These are particularly interesting for users of FreeStyle Libre (which is the only CGM I have a driver for), and someone expressed interest in adding a Prometheus export, because why not — this is not as silly as it may sound, the output graphs of the Libre’s own software look more like my work monitoring graphs than the usual glucometer graphs. Myself, I am now toying with the idea of mimicking the HTML output that the Accu-Check Mobile generate on its own. This would be the easiest to just send by email to a doctor, and probably can be used as a basis to extend things further, integrating the other graphs output.
So onwards and upwards, the tooling will continue being built on. And I’ll do my best to make sure that Linux users who have a need to download their meters’ readings have at least some tooling that they can use, and that does not require setting up unsafe MongoDB instances on cloud providers.
Another way to parse out bytestrings in Python is the built-in `struct` module… although `construct` looks a lot better polished (even if the documentation isn’t).That said, the trap people fall into with `struct` is to use `struct.pack` and `struct.unpack` with static format strings… if your format strings are static, your better option is to pass that to the constructor `struct.Struct`… and call the `pack` or `unpack` method on the `Struct` object created. *That* will yield better performance. (makes a *big* difference on ARMv5 for instance.)
Graphing is interesting, but maybe it doesn’t belong in this codebase? Maybe programs like gluco-web, gluco-pdf, gluco-gtk could consume output from glucometerutils instead. More useful for inclusion in glucometerutils would be gluco-dump-sticher for combining multiple data dumps from multiple meters into one dataset. I’m probably not the only person using more than one meter every day. But this almost takes us all the way to gluco-db that glucometerutils could dump data into.Graphing glucose data in a useful way is very difficult. Most diabetes management software focus on personal reflection on individual days. You don’t need software to realize you screwed up one day.The trouble is knowing what days are exceptional just from the glucose measurements alone. I’ve been trying to find a useful way to map this for years. I’ve tried to identify days with enough measurements throughout the day to be a useful. Then I’ve tried identifying trends like having okay measurements on Friday around 14:00 but then often get low by 17:00. Or trying to figure out the best times for measuring (e.g. periods and trends where I haven’t measured 1 hour before a high measurement). *sigh* Health data is complex and not all that fun to work with.
Yeah I agree that I don’t want to get into the whole processing of data because it’s where the art is. As I have stated in the GitHub projects, I’m more interested in outputting in a data format that can be consumed by TidePool, since they have a format that can be used.But on the other hand people want to have graphs, because graphs are pretty so who am I to tell them not to, as long as they make half sense? I’m more interesting in something I can send directly to my doctor, thus the AccuChek-like interface.
The struct module is what I was doing before, and was indeed using the struct objects. But that requires manual unpacking of pieces of the strings here and there. Using construct is significantly more code-efficient.It may include a performance loss, as some of the code is complicated, and I know for a fact that one of the drivers *is* significantly slower with the new code, but that’s because of serial port buffering. And I care less about performance than I care about correctness and readability of the code.
That is completely understandable.I don’t think anyone wants to get into that, which is why there are no/few open source tools available.I had a look at TidePool sometime back. Or, I didn’t get very far past the sign-up form as they block everything but Chrome after you’ve already given them all your information.
HAHAHAHA I didn’t even notice that :0I have not spent time on the TidePool app at all. I just though that they had a better attitude than the Glucosio folks, but they are also ridden with NDAs I don’t want to get too near to :)I plan on reversing their opensource drivers and document the protocols that they got access to, but that’s about it.