Software Defined Remote Control

A number of months ago I spoke about trying to control a number of TV features in Python. While I did manage to get some of the adapter boards that I thought I would use printed, I hadn’t had the time to go and work on the software to control this before we started looking for a new place, which meant I shelved the project until we could get to the new place, and once we got there it was a matter of getting settled down, and then, … you got the idea.

As it turns out, I had one week free at the end of November — my employer decided to give three extra days on the (US) Thanksgiving week, and since my birthday was at the end of the week, I decided to take the remaining two days off myself to make it a nice nine days contiguous off. Perfect timeframe to go and hack on some projects such as that.

Also, one thing changed significantly since the time I started thinking about this: I started using Home Assistant. And while it started mostly as a way for me to keep an eye on the temperature of the winter garden, I found that with a bit of configuration, and a pull request, changing the input on my receiver with it was actually easier than using the remote control and trying to remember which input was mapped to what.

That gave me finally the idea of how to implement my TV input switch tree: expose it as one or more media players in Home Assistant!

Bad (Hardware) Choices

Unfortunately, as soon as I went to start implementing the switching code, I found out that I had made a big mistake in my assumptions: the Adafruit FT232H breakout board does not support PWM outputs, including the general time-based pulsing (without a carrier frequency). Indeed, while the Blinka library can technically support some of the features, it seems like none of the Linux-running platforms would be able to manage that. So there goes my option of just using a computer to drive the “fake remote” outputs directly. Well, at least without rewriting it in some other language and find a different way to send that kind of signals.

I looked around for a few more options, but all of it ended up being some compromise: MicroPython doesn’t have a very usable PulseOut library as far as I could tell; Arduino-based libraries don’t seem to allow two outputs to happen at roughly the same time; and as I’m sure I already noted in passing, CircuitPython lacks a good “secondary channel” to be instructed from a computer (the serial interface is shared with the REPL control, and the HID is gadget-to-host only).

After poking around a few options and very briefly considering writing my own C version on an ATmega, I decided to just go for the path of least resistance, and go back to CircuitPython, and try to work with the serial interface and its “standard input” to the software.

The problem with doing that is that the Ctrl-C command is intended to interrupt the command, and that means you cannot send the byte 0x03 un-escaped. At the end I thought about it, and decided that CircuitPython is powerful enough that just sending the commands in ASCII wouldn’t be an issue. So I decided to write a simplistic Flask app that would take a request over HTTP and send the command via the serial port. It worked, sort of. Sometimes while debugging I would end up locking the device (a Trinket M0) in the REPL, and that meant the commands wouldn’t be sent.

The solution I came up with was to reset the board every time I started the app, by sending Ctrl-C and Ctrl-D (0x03, 0x04) to force the board to reset. It worked much better.

Not-Quite-Remote Controlled HDMI Switch

After that worked, the problem was ensuring that the commands sent actually worked. The first component I needed to send the commands to was the HDMI switch. It’s a no-brand AliExpress-special HDMI switch. It has one very nice feature for what I need to do right now. It obviously has an infrared remote control – one of those thin, plasticky domes one – but it particularly has the receiver for it on a cord, which is connected with a pretty much standard 3.5mm “audio jack”.

This is not uncommon. Randomly searching Amazon or AliExpress for “HDMI switch remote” can find you a number of different, newer switches that use the same remote receiver, or something very similar to it. I’m not sure if the receivers are compatible between each other, but the whole idea is the same: by using a separate receiver, you can stick the HDMI switch behind a TV, for instance, and just make the receiver poke from below. And most receivers appear to be just a dome-encased TSOP17xx receiver, which is a 3-pin IC, which works great for a TRS.

When trying this out, I found that what I could do would be to use a Y-cable to allow both the original receiver and my board to send signals to the switch — at which point, I can send in my own pulses, without even bothering with the carrier frequency (refer to the previous post for details on this, it’s long). The way the signal is sent, the pulses need to ground the “signal” line (that is usually at 5V); to avoid messing up the different supplies, I paired it on an opto-coupler, since they are shockingly cheap when buying them in bulk.

But now that I tried setting this up with an input selection, I found myself not able to get the switch to see my signal. This turned out to require an annoying physical debugging session with the Saleae and my TRRS-to-Saleae adapter (that I have still not released, sorry folks!), which showed I was a bit off on the timing of the NEC protocol the switch used for the remote control. This is now fixed in the pysirc pysdrc library that generates the pulses.

Once I got the input selector working for the switch with the Flask app, I turned to Home Assistant and added a custom component that exposes the switch as a “media_player” platform. In a constant state of “Idle” (since it doesn’t have a concept of on or off), it allowed me and my wife to change the input while seeing the names of the devices, without hunting for the tiny remote, and without having to dance around to be seen by the receiver. It was already a huge improvement.

But it wasn’t quite enough where I wanted it to be. In particular, when our family calls on Messenger, we would like to be able to just turn on the TV selected to the right input. While this was partially possible (Google Assistant can turn on a TV with a Chromecast), and we could have tried wiring up the Nabu Casa integration to select the input of the HDMI switch, it would have not worked right if the last thing we used the TV for was the Nintendo Switch (not to be confused with the HDMI switch) or for Kodi — those are connected via a Yamaha receiver, on a different input of the TV set!

Enter Sony

But again, this was supposed to be working — the adapter board included a connection for an infrared LED, and that should have worked to send out the Sony SIRC commands. Well, except it didn’t, and that turned out to be another wild goose chase.

First, I was afraid that when I fixed the NEC timing I broke the SIRC ones — but no. To confirm this, and to make the rest of my integration easier, I took the Feather M4 to which I hard-soldered a Sony-compatible IR LED, and wrote what is the eponymous software defined remote control: a CircuitPython program that includes a few useful commands, and abstractions, to control a Sony device. For… reasons, I have added VCR as the only option beside TV; if you happen to have a Bluray player by Sony, and you want to figure out which device ID it uses, please feel free.

It might sound silly, but I remember seeing a research paper in UX from the ’90s of using gesture recognitions on a touchpad on a remote control to allow more compact remote controls. Well, if you wanted, you could easily make this CircuitPython example into a touchscreen remote control for any Sony device, as long as you can find all the right device IDs, and hard code a bunch of additional commands.

So, once I knew that at least on the software side I was perfectly capable of control the Sony TV, I had to go and do more hardware debugging, with the Saleae, but this time with the probes directly on the breadboard, as I had no TRS cable to connect to. And that was… a lot of work, to rewire stuff and try.

The first problem was that the carrier frequency was totally off. The SIRC protocol specifies a 40kHz carrier frequency, which is supposedly easier to generate than the 38kHz used by NEC and others, but somehow the Saleae was recording it as a very variable frequency that oscillated between 37kHz and 41kHZ. So I was afraid that trying to run two PWM outputs on the Trinket M0 was a bad idea, even if one of them was set to nought hertz — as I said, the HDMI switch didn’t need a carrier frequency.

I did toy briefly with the idea of generating the 40kHz carrier wave separately, and just gating it to the same type of signal I used for the HDMI switch. Supposedly, 40kHz generators are easy, but at least for the circuits I found at first glance, it requires a part (640kHz resonator) that is nearly impossible to find in 2020. Probably fell out of use. But as it turn out it wouldn’t have helped.

Instead, I took another Feather. Since I ran out of M4, except for the one I hardwired already an IR LED to, I instead pulled up the nRF52840 that I bought and barely played with. This should have been plenty capable to give me a clean 40kHz signal and it indeed was.

At that point I noticed another problem, though: I totally screwed up the adapter board. In my Feather M4, the IR LED was connected directly between 3V and the transistor switching it. A bit out of spec, but not uncommon given that it’s flashed for very brief impulses. On the other hand when I designed the adapter, I connected it to the 5V rail. Oops, that’s not what I was meant to be doing! And I did indeed burn out the IR LED with it. So I had to solder a new one on the cable.

Once I fixed that, I found myself hitting another issue: I could now turn on and off the TV with my app, but the switch stopped responding to commands either from the app or from the original remote! Another round of Saleae (that’s probably one of my favourite tools — yes I splurged when I bought it, but it’s turning out to be an awesome tool to have around, after all), and I found that the signal line was being held low — because the output pin is stuck high…

I have not tried debugging this further yet — I can probably reproduce this without my whole TV setup, so I should do that soonish. It seems like opening both lines for PWM output causes some conflicts, and one or the other end up not actually working. What I solved this with was only allowing one command before restarting the Feather. It meant taking longer to complete the commands, but it allowed me to continue with my life without further pain.

One small note here: since I wasn’t sure how Flask concurrency would interact with accessing a serial port, I decided to try something a bit out of the ordinary, and set up the access to the Feather via an Actor using pykka. It basically means leaving one thread to have direct access to the serial port, and queue commands as messages to it. It seems to be working fine.

Wrapping It All Up

Once the app was able to send arbitrary commands to the TV via infrared, as well as changing the input of the HDMI, I extended the Home Assistant integration to include the TV as a “media_player” entity as well. The commands I implemented were Power On and Off (discrete, rather than toggle, which means I can send a “Power On” to the TV when it’s already on and not bother it), and discrete source selection for the three sources we actually use (HDMI switch, Receiver, Commodore 64). There would be a lot more commands I could theoretically send, including volume control, but I can already access those via the receiver, so there’s no good reason to.

After that it was a matter of scripting some more complicated acts: direct selection of Portal, Chromecast, Kodi, and Nintendo Switch (which are the four things we use the most). This was easy at that point: turn on the TV (whether it was on or not), select the right input on either the receiver or the switch, then select the right input ion the TV. The reason why the order seems a bit strange is that it takes a few seconds for the TV to receive commands after turning on, but by doing it this way we can switch between Chromecast and Portal, or Nintendo Switch and Kodi, in pretty much no time.

And after that worked, we decided the $5/month to Nabu Casa were worth it, because that allows us to ask Alexa or Google Assistant to select the input for us, too.

Eventually, this lead me to replace Google’s “Turn off the TV” command in our nightly routine to trigger a Home Assistant script, too. Previously, it would issue the command to the Chromecast, routing through the whole Google cloud services between the device that took the request and the Chromecast. And then the Chromecast would be sending the CEC command to power off… except that it wouldn’t reach the receiver, which would stay on for another two hours until it finally decided it was time to turn off.

With the new setup, Google is triggering the Home Assistant script, and appears to do that asynchronously. Then Home Assistant sends the request to my app, that then sends it to the Feather, that sends the power off to the TV… which is also read by the receiver. I didn’t even need to send the power off command to the receiver itself!

All in all, the setup is satisfying.

What remains to be done is to try exposing a “Media Player” to Google Home, that is not actually any of the three “media_player” entities I have, but is a composite of them. That way, I could actually just expose the different input trees as discrete inputs to Google, and include the whole play, pause, and volume control that is currently missing from the voice controls. But that can wait.

Instead, I should probably get going at designing a new board to replace the breadboard mess I’m using right now. It’s hidden away enough that it’s not in our face (unlike the Birch Books experiments), but I would still like having a more… clean setup. And speaking of that, I really would love if someone already contributed an Adafruit Feather component for EAGLE, providing the space for soldering in the headers, but keeping the design referencing the actual lines as defined in it.

Converting Unpaper to Meson

You may remember that I took over Unpaper a number of years ago, as the original author was not interested in maintaining it in the long term. And one of the first things I did was replacing the hand-rolled build system with Autotools, as that made packaging it significantly simpler. Followed by replacing the image decoding with libav first, and more recently FFmpeg.

For various reasons, I have not spent much time on Unpaper over the years I spent in my last bubble. When I joined my current employer I decided that I cared more to get Unpaper back into a project people can contribute to, and less about maintaining the copyright of my contributions, which makes it easier for me to work on it without having to carve out some special time for it.

One of the main requests over on the GitHub project, over these years, has been Windows support. And I did say that Windows 10 is becoming increasingly interesting for Free Software development. So when I was asked to rant on a bit about build systems, which I did over on Twitch, I decided to take a stab at figuring out if Meson (which supports Windows natively), would be a good fit. And I did that using Visual Studio Code and Debian/WSL!

If you haven’t seen the video yet, spoiler alert: I tried, got it working within ten minutes, and it worked like a charm. I’m not kidding you, it pretty much worked at the first attempt (well, the first session, not the first execution), and it made total sense the way it works. You can tell that the folks involved in building Meson (including Gentoo’s own Nirbheek!) knew what they were embarking to do and how to have it fit together. Even small bits like keeping large files support always enabled made me a very happy user.

I have now a branch on GitHub for the Meson build, although it’s incomplete. It doesn’t install any of the docs, and it doesn’t install the man page. Also it doesn’t build nor run the tests. For all of those I created a project to track what needs to be done: move on from the current implementation, it’s 2020!

The test system in the current Autotools version of Unpaper is leveraging the implementation of make check from Automake, together with a C program that compares the expected outputs with what is generated by the newly-built unpaper binary. It also needs to consider a threshold of difference between the two, because precision is not guaranteed, in either direction. This is incredibly fragile and indeed is currently failing for… not sure which reason. Getting a “diff” of the generated versus expected in C is fairly hard and deserves its own project. Instead, relying on Python for instrumenting and running the tests would make it much easier to maintain, as you wouldn’t need to maintain at least three integration points to keep this together.

Something similar is a problem for documentation: right now the documentation is split between some Markdown files, and a single DocBook (XML) file that is converted to a man page with xsltproc. Once the bullet of Python is bit, there’s no reason not to just use ReStructuredText and Sphinx, which already provides integration to generate man pages — and honestly nowadays I feel like the XML version is definitely not a good source because it can’t be read by itself.

What I have not done yet is making sure that the Meson build system allows a Windows build of Unpaper. The reason is relatively simple: while I have started using Visual Studio Code, and clang is available for Windows in the form of a binary, and so is FFmpeg, fitting it all together will probably take me more time, as I’m not used to this development environment. If someone is interested in making sure this works out of the box, I’m definitely happy to review pull requests.

So yeah, I guess the Autotools Mythbuster can provide a seal(ion) of approval to Meson. Great work, folks! Happy to see we have a modern build system available that compromises in the right places instead of being too dogmatic!

Service Announcement: Live Whiteboarding this Thursday

I’m breaking the usual post schedule to point out that this Thursday (2020-07-30) I’ve decided to attempt (again) an online whiteboarding session, this time with virtual whiteboard software instead of a physical one.

The plan is to have a one hour of me rambling on and ranting about some of my projects that are not formed enough to be blog posts (such as my electronics projects), but that I would love to share with the wider community sooner rather than later.

I plan on streaming this on Twitch, for the first time ever, since their Studio software appears to be the most straightforward way to stream a single window on the screen, and it should have a decent chat system as well. So if you’re interested in hearing some of my thought process, or you’re an ex colleague who misses my office rants, you’re welcome to join us there.

The set time is 8pm London time, and I plan on rant around for an hour. If there’s going to be enough interest in this, I’ll try to make this a more regular thing.

Kodi, NUC, and CEC adapters

Welcome to this year’s yearly Kodi post. The fact that I don’t write about Kodi very often should be a sign that it’s actually fairly stable, and indeed beside for the random fight with X11 over whether DPMS should or should not be enabled, my HTPC setup works fairly well, even though it’s getting used less often — the main content I have on it is stuff that we own in DVDs — either because we ripped it ourselves or… found it to make it easier on us.

In part due to my poking around my other TV-related project, and in part because my phone seems to have issues at keeping its IPv6 stable – which means I can’t control Kodi with the Kore app until I turn wifi off and on again – I decided to address the biggest missing features of Intel’s NUC: the lack of HDMI CEC support — and as far as I know that’s a missing feature even on their most recent NUCs.

With CEC, HDMI-connected devices can provide a level of control, including signaling a request to turn on or off, to change the input selection, and so on. It also allows a TV remote control to send most button presses to a connected device. This allows, for instance, to have control over Kodi’s menus with a single remote control, rather than having to set up a second remote just for that, which is what I was originally planning on doing, using a vintage SVHS remote control.

As I said, Intel doesn’t support CEC on their NUC, to this day. Instead they suggest you to buy an adapter from companies such as Pulse Eight, which thankfully still also sell the harness for my six years old model. The adapter is a microcontroller that is connected to an Intel-provided header, from which it receives the CEC signals coming from the HDMI port on one side, and exposes an USB device through the motherboard header, from which the libCEC – that Pulse Eight themselves developed – can decode it and provide information to the other software.

The Pulse-Eight installation instructions gloss over the most interesting part: which pins does it need on the Custom Solutions Header, and more interesting how do you get to them. It turns out that I had to disassemble most of the NUC to get to the connector, and only found the right pins thanks to Kingj who reviewed this kit five years ago. The thing that his review seemed to ignore, though, is that the connectors are just about as tall as the same there is to use that theader — and the cable rubs onto the WiFi antenna (which I actually never use, so I was actually tempted to remove, until I remember they do Bluetooth, too). And the suggested position to stick the small board onto the Ethernet connector bumps into the heat foam that’s meant to protect the mSATA drive. But beside those two issues, it fit all together, and with a bit of worry and fear to break stuff, and possibly one of the black pip retainers broken, I did manage to get the whole thing installed just fine.

Troubleshooting Pulse-Eight CEC Adapter Issues

After installing and booting up the NUC, I found myself nearly crying from the frustration when I booted up and saw the adapter turn up on lsusb as Atmel at90usb162 DFU bootloader. This appears to be a common bane of multiple people, and at least on Windows people suggested that a firmware update would help — the firmware update being only available as an option on Windows (and possibly Wine?), but not released on Linux. Turns out that this is a red herring.

The DFU bootloader is the Device Firmware Update — this is the mode you need a microcontroller (MCU) to be in to load it with a new program. Usually there’s two ways to enter this operation: a physical one and a logical one (sending a specific command via USB, for instance), and while I didn’t expect them to be stateful, they appear to be. At first I was afraid I grounded something I shouldn’t have (which would have put into firmware download mode), or that the device left the factory without being programmed (which would have indeed been fixed by running the firmware update).

Instead, it turns out it might just be stateful, and was left in programming mode when it left the factory — and you can fix that with a single command on Linux: dfu-tool attach. The command is provided by fwupd (LVFS), and worked like a charm on my board for it to show up as the /dev/ttyACM0 it needed to be.

The next problem was figuring out why Kodi was not acting on any of the remote button presses. According to a lot of blogs, and forum posts, and wikis, this might have had to do with remote.xml. It didn’t, at least for me. The problem is that Kodi recognizes the presence of the CEC device by looking at the USB devices connected, but it doesn’t check if it has permissions to open the device, until it tries to. And the device (/dev/ttyACM0, which is usually the easiest no-drivers-needed way to connect an USB-to-UART bridge), is usually owned by the dialout group, which my Kodi user was not part of.

Why dialout and what the heck does it mean? Given it is 2020 as I write this, it might be worth giving a bit of a history lesson, without being condescending to those who might not have heard the term before. This group is usually “in charge” of serial port devices, such as ttyS0 and ttyACM0 and ttyUSB0 — the reason for that is that back in the old days of phone-line connectivity (which for some are current days still), the main use for serial ports was to connect modems (or in some cases faxes) — and that meant that whoever could access the serial ports would be able to “dial out” — that is, make a phone call. Since phone calls could be (and possibly still are) very expensive, accessing those ports couldn’t be made too easy. This is now a totally legacy naming, but… well, it’s hard to change these things, particularly in UNIX-land.

I solved this by adding Kodi to the group. The NUC doesn’t have any other serial port, so it’s not an issue to do that. The alternative would be to have an udev rule that explicitly sets the device as owned by the user running Kodi, similar to the rules in glucometerutils.

So now the device works, Kodi can receive the commands just fine, and I don’t have to bother to get the Kore app out just to select the next episode of whatever we’re watching at any one time. The only thing that remains annoying to me is that I can’t access the CEC device from any other process. I was wondering if I could implement most of what I wanted in the project I described previously (namely, controlling TV inputs) through CEC (answer: probably), but that’s a moot point because serial devices can only be accessed by a single process at a time (the same is true of most other devices — this is the reason why Windows has device drivers, and why you end up with so many “daemons” in Linux and other *nix: you need something that “multiplexes” the account across different processes).

If this was ten years ago, you’d have me design a CEC Daemon, and proof-of-concept integrate it in Kodi. Time being what it is, I’m unlikely to be working on this any time soon. But if someone feels like it’s a worthy task, I’ll be happy to chat, discuss designs, or review code to implement it.

Planets, Clouds, Python

Half a year ago, I wrote some thoughts about writing a cloud-native feed aggregator. I actually started drawing some ideas of how I would design this myself since, and I even went through the (limited) trouble of having it approved for release. But I have not actually released any code, or to be honest, I have not written any code either. The repository has been sitting idle.

Now, with the Python 2 demise coming soon, and me not interested in keeping around a server nearly only to run Planet Multimedia, I started looking into this again. The first thing that I realized is that I both want to reuse as much code exist out there as I can, and I want to integrate with “modern” professional technologies such as OpenTelemetry, which I appreciate from work, even if it sounds like overkill.

But that’s where things get complicated: while going full “left-pad” of having a module for literally everything is not something you’ll find me happy about, a quick look at feedparser, probably the most common module to read feeds in Python, shows just how much code is spent trying to cover for old Python versions (before 2.7, even), or to implement minimal-viable-interfaces to avoid mandatory dependencies at all.

Thankfully, as Samuel from NewsBlur pointed out, it’s relatively trivial to just fetch the feed with requests, and then pass it down to feedparser. And since there are integration points for OpenTelemetry and requests, having an instrumented feed fetcher shouldn’t be too hard. That’s going to probably be my first focus when writing Tanuga, next weekend.

Speaking of NewsBlur, the chat with Samuel also made me realize how much of it is still tied to Python 2. Since I’ve gathered quite a bit of experience in porting to Python 3 at work, I’m trying to find some personal time to contribute smaller fixes to run this in Python 3. The biggest hurdle I’m having right now is to set it up on a VM so that I can start it up in Python 2 to begin with.

Why am I back looking at this pseudo-actively? Well, the main reason is that rawdog is still using Python 2, and that is going to be a major pain with security next year. But it’s also the last non-static website that I run on my own infrastructure, and I really would love to get rid of entirely. Once I do that, I can at least stop running my own (dedicated or virtual) servers. And that’s going to save me time (and money, but time is the most important one here too.)

My hope is that once I find a good solution to migrate Planet Multimedia to a Cloud solution, I can move the remaining static websites to other solutions, likely Netlify like I did for my photography page. And after that, I can stop the last remaining server, and be done with sysadmin work outside of my flat. Because honestly, it’s not worth my time to run all of these.

I can already hear a few folks complaining with the usual remarks of “it’s someone else’s computer!” — but the answer is that yes, it’s someone else’s computer, but a computer of someone who’s paid to do a good job with it. This is possibly the only way for me to manage to cut away some time to work on more Open Source software.

Boot-to-Kodi, 2019 edition

This weekend I’m oncall for work, so between me and my girlfriend we decided to take a few chores off our to-do lists. One of the things for me was to run the now episodic maintenance over the software and firmware of the devices we own at home. I call it episodic, because I no longer spend every evening looking after servers, whether at home or remote, but rather look at them when I need to.

In this case, I honestly forgot when it was the last time that I ran updates on the HTPC I use for Kodi and for the UniFi controller software. And that meant that after the full update I reached the now not uncommon situation that Kodi refused to start at boot. Or even when SSH’ing into the machine and starting the service by hand.

The error message, for ease of Googling, is:

[  2092.606] (EE) 
Fatal server error:
[  2092.606] (EE) xf86OpenConsole: Cannot open virtual console 7 (Permission denied)

What happens in this case is that the method I have been using to boot-to-Kodi was to use a systemd unit lifted from Arch Linux, that started a new session, X11, and Kodi all at once. This has stopped working now, because Xorg no longer can access the TTY, because systemd does not think it should access the console.

There supposedly are ways to convince systemd that it should let the user run X11 without so much fluff, but after an hour trying a number of different combinations I was not getting anywhere. I finally found one way to do it, and that’s what I’m documenting here: use lightdm.

I have found a number of different blog posts out there that try to describe how to do this, but none of them appear to apply directly to Gentoo.

These are the packages that would be merged, in order: 
 
Calculating dependencies... done! 
[ebuild   R    ] x11-misc/lightdm-1.26.0-r1::gentoo  USE="introspection -audit -gnome -gtk -qt5 -vala" 0 KiB

You don’t need Gtk, Qt or GNOME support for lightdm to work. But if you install it this way (which I’m surprised is allowed, even by Gentoo) it will fail to start! To configure what you need, you would have to manually write this to /etc/lightdm/lightdm.conf:

[Seat:*] 
autologin-user=xbmc 
user-session=kodi 
session-wrapper=/etc/lightdm/Xsession

In this case, my user is called xbmc (this HTPC was set up well before the rename), and this effectively turns lightdm into a bridge from systemd to Kodi. The kodi session is installed by the media-tv/kodi package, so there’s no other configuration needed. It just… worked.

I know that some people would find the ability to do this kind of customization via “simple” text files empowering. For me it’s just a huge waste of time, and I’m not sure why there isn’t just an obvious way for systemd and Kodi to get along. I would hope somebody builds one in the future, but for now I guess I’ll leave with that.

I’m told Rust is great, where are the graphics libraries?

While I’m still a bit sour that Mozilla decided to use the same name for their language as an old project of mine (which is not a new thing for Mozilla anyway, if someone remembers the days of Phoenix and Firebird), I have been looking from the sideline as the Rust language as a way forward to replace so many applications of embedded C, with a significantly safer alternative.

I have indeed been happy to see so much UEFI work happening in Rust, because it seems to me like we came far enough that we can sacrifice some of the extreme performance of C for some safety.

But one thing that I still have not seen is a good selection of graphics libraries, and that is something that I’m fairly disappointed by. Indeed, I have been told that there are Rust bindings for the classic C graphics libraries — which is pointless, as then the part that needs safety (the parsing) is still performed in C!

The reason why I’m angry about this is that I still have one project, unpaper, which I inherited as a big chunk of C and could definitely be rewritten into a safer language. But I would rather not do so in a higher level language like Python due to the already slow floating point calculations and huge memory usage.

Right now, unpaper is using libav, or ffmpeg, or something with their interface, depending on how much they fought this year. This is painful, but given that each graphic library implements interfaces in different ways, I couldn’t find a better and safe way to implement graphics processing. I was hoping that with all the focus on Rust out there, particularly from Mozilla, implementing graphics parsing libraries would be high in the list of priorities.

I think it’s librsvg that was ported to Rust — which was probably a great idea to prioritize, given it is exactly the type of format where C performs very poorly: string parsing. But I’m surprised nobody tried to make an API-compatible libpng or libtiff. It sounds to me like Rust is the perfect language for this type of work.

At any rate, if anyone finally decides to implement a generic graphic file input/output library, with at least support for TIFF, PNM and PNG, I’d love to know. And after that I would be happy to port unpaper to it — or if someone wants to take unpaper code as the basis to reimplement it as a proof of concept, that’d be awesome.

The problem for a lot of these libraries is that you have to maintain support for a long list of quirks and extensions that over time piled up on the formats. And while you can easily write tests to maintain bit-wise compatibility with the “original” C language based libraries for what concerns bitmap rendering (even for non-bitmap graphics such as JPEG and WebP), there are more things that are not obvious to implement, such as colour profiles, and metadata in general.

Actually, I think that there is a lot of space here to build up a standard set of libraries for graphics libraries and metadata, since there’s at least some overlapping between these, and having a bigger group of people working on separate, but API-similar libraries for various graphic formats would be a significant advantage for Rust over other languages.

Dear Amazon, please kill the ComiXology app

Dear Amazon, Dear Comixology,

Today, May the 4th, is Free Comic Book Day. I thought this was the right time to issue a plea to you: it’s due time you get rid of the ComiXology app, and ask your customers to just use an unified Kindle app to read their comic books.

I love comic books, and I found ComiXology an awesome service, with awesome selection and good prices. I have been a customer for many years, starting from when I just had bought an iPad for a job. For a while, I have signed up for their ComiXology Unlimited service, that for a monthly fee gave you access to an astounding amount of comics — particularly a lot of non-mainstream comics, a great way to discover some interesting independent authors.

When Amazon bought ComiXology, I was at the same time pleased and afraid — pleased because that could have (and did) boost ComiXology’s reach, afraid because there was always a significant overlap with the Kindle app, ecosystem and market. And it turned out that my fears were just as real, as I found out last year.

I don’t want to repeat the specifics here, the short version is that the ComiXology app has been broken for over a year now for any Android user that relies on microSD storage rather than the internal storage, such as mine. After multiple denial from ComiXology support, the blog post helped me get this to the attention of at least one engineer on the team, who actually sent me a reply nearly 11 months ago:

I followed up with our team and a few weeks ago we met about your report. We realized you are 100% correct, and we’re re-evaluating our decision RE adoptable storage. I don’t have news on when that answer is coming, but the topic is open internally and I want to thank you for your detailed emails and notes. Hopefully we can figure this out and get you back.

Matt, ComiXology Support, May 30, 2018

Unfortunately, months passed, and no changes were pushed to the app. The tablet got an Android OS update, ComiXology got updates every few months, but the app to this day has any way to store its content on microSD cards. The last contact I have from support is from last summer:

Our team has tracked down what’s going on and you are correct in your analysis. They are working on a solution, though we do not have an estimate for when you will be seeing it. We will keep on checking in on this and making sure things move along.

Erin, ComiXology Support, August 13th, 2018

This is not just a simple annoyance. There is a workaround, that involves using the microSD as so-called “portable storage”, and telling the app to store the comics on the SD card itself. But it has another side effect: you can’t then use the SD card to download Netflix content. The Netflix app cannot be moved to the card, either as adopted or portable storage – just like ComiXolgy – but it supports selecting an “adopted storage” microSD card for storage, and actually defaults to it. So you end up choosing between Netflix and ComiXology.

And here’s the kicker: the Kindle app, developed by a different branch of the same company, does this the right way.

And this brings me back to the topic of this post: the Kindle app is not stellr for reading comic books in my experience, ComiXology did a much better job at navigating panels. But that’s where it stops — Kindle has a better library handling, a better background download support, and clearly better support for modern Android OS. But I can’t read the content I already paid for in ComiXology on that.

I think the best value for the customers, for the people actually reading the comic books, would be if Amazon just stopped investing engineering into the ComiXology app at this point, which clearly appears understaffed and not making any forward progress anyway, and instead allowed reading of ComiXology content on Kindle apps. And maybe Kindle hardware — I would love reading my manga collection on a Kindle, even if I had to upgrade from my Paperwhite (but please, if you require me to do that, use USB-C for the next gen!)

Will you, Amazon?

“Planets” in the World of Cloud

As I have written recently, I’m trying to reduce the amount of servers I directly manage, as it’s getting annoying and, honestly, out of touch with what my peers are doing right now. I already hired another company to run the blog for me, although I do keep access to all its information at hand and can migrate where needed. I also give it a try to use Firebase Hosting for my tiny photography page, to see if it would be feasible to replace my homepage with that.

But one of the things that I still definitely need a server for is keep running Planet Multimedia, despite its tiny userbase and dwindling content (if you work in FLOSS multimedia, and you want to be added to the Planet, drop me an email!)

Right now, the Planet is maintained through rawdog, which is a Python script that works locally with no database. This is great to run on a vserver, but in a word where most of the investments and improvements go on Cloud services, that’s not really viable as an option. And to be honest, the fact that this is still using Python 2 worries me no little, particularly when the author insists that Python 3 is a different language (it isn’t).

So, I’m now in the market to replace the Planet Multimedia backend with something that is “Cloud native” — that is, designed to be run on some cloud, and possibly lightweight. I don’t really want to start dealing with Kubernetes, running my own PostgreSQL instances, or setting up Apache. I really would like something that looks more like the redirector I blogged about before, or like the stuff I deal with for a living at work. Because it is 2019.

So sketching this “on paper” very roughly, I expect such a software to be along the lines of a single binary with a configuration file, that outputs static files that are served by the web server. Kind of like rawdog, but long-running. Changing the configuration would require restarting the binary, but that’s acceptable. No database access is really needed, as caching can be maintained to process level — although that would men that permanent redirects couldn’t be rewritten in the configuration. So maybe some configuration database would help, but it seems most clouds support some simple unstructured data storage that would solve that particular problem.

From experience with work, I would expect the long running binary to be itself a webapp, so that you can either inspect (read-only) what’s going on, or make changes to the database configuration with it. And it should probably have independent parallel execution of fetchers for the various feeds, that then store the received content into a shared (in-memory only) structure, that is used by the generation routine to produce the output files. It may sounds like over-engineering the problem, but that’s a bit of a given for me, nowadays.

To be fair, the part that makes me more uneasy of all is authentication, but Identity-Aware Proxy might be a good solution for this. I have not looked into that but used something similar at work.

I’m explicitly ignoring the serving-side problem: serving static files is a problem that has mostly been solved, and I think all cloud providers have some service that allows you to do that.

I’m not sure if I will be able to work more on this, rather than just providing a sketched-out idea. If anyone knows of something like this already, or feels like giving a try to building this, I’d be happy to help (employer-permitting of course). Otherwise, if I find some time to builds stuff like this, I’ll try to get it released as open-source, to build upon.

Why do we still use Ghostscript?

Late last year, I have had a bit of a Twitter discussion on the fact that I can’t think of a good reason why Ghostscript is still a standard part of the Free Software desktop environment. The comments started from a security issue related to file-access from within a PostScript program (i.e. a .ps file), and at the time I actually started drafting some of the content that is becoming this post now. I then shelved most of it because I’ve been busy and it was not topical.

Then Tavis had to bring this back to the attention of the public, and so I’m back writing this.

To be able to answer the question I pose in the title we have to first define what Ghostscript is — and the short answer is, a PostScript renderer. Of course it’s a lot more than just that, but for the most part, that’s what it is. It deals with PostScript programs (or documents, if you prefer), and renders them into different formats. PostScript is rarely if at all use in modern desktops — not just because it’s overly complicated, but because it’s just not that useful in a world that mostly settled in PDF, which is essentially a “compiled PostScript”.

Okay not quite. There are plenty of qualifications that go around that whole paragraph, but I think it matches the practicalities of the case fairly well.

PostScript has found a number of interesting niche uses though, a lot of which focus around printing, because PostScript is the language that older (early?) printers used. I have not seen any modern printers speak PostScript though, at least after my Kyocera FS-1020, and even those who do, tend to support alternative “languages” and raster formats. On the other hand, because PostScript was a “lingua franca” for printers, CUPS and other printer-related tooling still use PostScript as an intermediate language.

In a similar fashion, quite a few software that deal with faxes (yes, faxes), tend to make use of Ghostscript itself. I would know because I wrote one, under contract, a long time ago. The reason is frankly pragmatic: if you’re on the client side, you want Windows to “print to fax”, and having a virtual PostScript printer is very easy — at that point you want to convert the document into something that can be easily shoved down the fax software throat, which ends up being TIFF (because TIFF is, as I understand it, the closest encoding to the physical faxes). And Ghostscript is very good at doing that.

Indeed, I have used (and seen used) Ghostscript in many cases to basically combine a bunch of images into a single document, usually in TIFF or PDF format. It’s very good at doing that, if you know how to use it, or you copy-paste from other people’s implementation.

Often, this is done through the command line, too, the reason for which is to be found in the licenses used by various Ghostscript implementations and versions over time. Indeed, while many people think of Ghostscript as an open source Swiss Army Knife of document processing, it actually is dual-licensed. The Wikipedia page for the project shows eight variant, with at least four different licenses over time. The current options are AGPLv3 or the commercial paid-for license — and I can tell you that a lot of people (including the folks I worked under contract for), don’t really want to pay for that license, preferring instead the “arms’ length” aggregation of calling the binary rather than linking it in. Indeed, I wrote a .NET Library to do just that. It’s optimized for (you guessed it right) TIFF files, because it was a component of an Internet Fax implementation.

So where does this leave us?

Back ten years ago or so, when effectively every Free Software desktop PDF viewer was effectively forking the XPDF source code to adapt it to whatever rendering engine they needed, it took a significant list of vulnerabilities that needed to be fixed time and time again for the Poppler project to take off, and create One PDF Rendering To Rule Them All. I think we need the same for Ghostscript. With a few differences.

The first difference is that I think we need to take a good look at what Ghostscript, and Postscript, are useful for in today’s desktops. Combining multiple images in a single document should _not_ require processing all the way to PostScript. There’s no reason to! Particularly not when the images are just JPEG files, and PDF can embed them directly. Having a tool that is good at combining multiple images into a PDF, with decent options for page size and alignment, would probably replace many of the usages of Ghostscript that I had in my own tools and scripts over the past few years.

And while rendering PostScript for either display or print are similar enough tasks, I have some doubt the same code would work right for both. PostScript and Ghostscript are often used in networked printing as well. In which case there’s a lot of processing of untrusted input — both for display and printing. Sandboxing – and possibly writing this in a language better suited to deal with untrusted input than C is – would go a long way to prevent problems there.

But there are a few other interesting topics that I want to point out on this. I can’t think of any good reason for desktops to support PostScript out of the box in 2019. While I can still think of a lot of tools, particularly from the old timers, that use PostScript as an intermediate format, most people _in the world_ would use PDF nowadays to share documents, not PostScript. It’s kind of like sharing DVI files — which I have done before, but I now wonder why. While both formats might have advantages over PDF, in 2019 they definitely lost the format war. macOS might still support both (I don’t know), but Windows and Android definitely don’t, which make them pretty useless to share knowledge with the world.

What I mean with that is that it’s probably due time that PostScript becomes an optional component of the Free Software Desktop, one that the users need to enable explicitly if they ever need it, just to limit the risks that accepting, displaying and thumbnailing full, Turing-complete programs masqueraded as documents. Even Microsoft stopped running macros in Office documents by default, when they realize the type of footgun it had become.

Of course talk is cheap, and I should probably try to help directly myself. Unfortunately I don’t have much experience with graphics formats, beside for maintaining unpaper, and that is not a particularly good result either: I tried using libav’s image loading, and it turns out it’s actually a mess. So I guess I should either invest my time in learning enough about building libraries for image processing, or poke around to see if someone wrote a good multi-format image processing library in, say, Rust.

Alternatively, if someone starts to work on this and want to have some help with either reviewing the code, or with integrating the final output in places where Ghostscript is used, I’m happy to volunteer my time. I’m fairly sure I can convince my manager to let me do some of that work as a 20% project.