Investigating Chinese Acrylic Lamps

A couple of months ago I built an insulin reminder light, roughly hacking around what I would call an acrylic lamp. The name being a reference to the transparent acrylic (or is it polycarbonate?) shape that you fit on top, and that lights up with the LEDs it’s resting on top. They are totally not a new thing, and even Techmoan looked at them three years ago. The relatively simple board inside looked fairly easy to hack around, and I thought it would make a good hack project to look more into them.

They are also not particularly expensive. You can go on AliExpress and get them for a few bucks each with so many different shape designs. There’s different “bases” to choose from, too — the one I hacked the Pikachu on was a fairly simple design with a translucent base, and no remote control, although the board clearly showed space for a TSOP-style infrared decoder. So I ended up ordering four different variants — although all of them without remotes because that part I didn’t particularly care for: one translucent base, one black base with no special features, one with two-colour shapes and LEDs, one one self-changing LEDs with mains power only.

While waiting for those to turn up, I also found a decent deal on Amazon on four bases without the acrylic shapes on them for about £6 each. I took a punt and ordered them, which turned out to be a better deal than expected.

These bases appear to use the same board design, and the same remote control (although they shipped four remotes, too!), and you can see an image of it on the right. This is pretty much the same logic on the board as the one I hacked for my insulin reminder, although it has slightly different LEDs, which are not common anode in the package, but are still wired in a common-anode configuration.

For both the boards, the schema above is as good a reversing as I managed on my own. I did it on the white board, so there might be some differences in the older green one, particularly in the number of capacitors, but all of that is not important for what I’m getting to right now. I shortened the array to just four LEDs to show, but this goes on for all of the others too. The chip is definitely not a Microchip one, but it fits the pinout, so I kept that one, similarly to what I did for the fake candle. Although in this case there’s no crystal on the board, which suggests this is a different chip.

I kind of expected that all the remaining boards would be variation on the same idea, except for the multi-color one, but I was surprised to figure out that only two of them shared the same board design (but took different approaches as to how to connect the IR decoder — oh yeah, I didn’t select any of the remote-controlled lamps, but two of them came with IR decoderes anyway!)

The first difference is due to the base itself: there’s at least two types of board that relate to where the opening for the microUSB port is in relation to the LEDs: either D-shaped (connector inline with the LEDs) or T shaped (connector perpendicular to the LEDs). Another difference is in the placement of the IR decoder: on most of the bases, it’s at 90° from the plug, but in at least one of them it’s direct opposite.

Speaking of bases, the one that was the most different was the two-colours base: it’s quite smaller in size, and round with a smooth finish, and the board was proper D shaped and… different. While the LEDs were still common-anode and appeared wired together, each appears to have its own paired resistor (or two!), and the board itself is double-sided! That was a surprise! It also is changing the basic design quite a bit more than I expected, including only having one Zener, and powering up the microcontroller directly over 4.5V instead of using a 3V regulator.

It also lacks the transistor configuration that you’d find on the other models, which shouldn’t surprise, given how it needs to drive more than the usual three channels. Which actually had me wonder: how does it drive two sets of RGB LEDs with an 8-pin microcontroller? Theoretically, if you don’t have any inputs at all, you could do it: VDD and VSS take two pins, each set of LEDs take three pins for the three colour channels. But this board is designed to take an IR decoder for a remote control, which is an input, and it comes with a “button” (or rather, a piece of metal you can ground with your finger), which is another input. That means you only have four lines you can toggle!

At first I thought that the answer was to be found on the other six-pin chip on the lift, but turns out that’s not the case. That one is marked 8223LC and that appears to correspond to a “touch controller” Shouding SD8223L and is related to the metal circlet that all of these bases use as input.

Instead, the answer became apparent when using the multimeter in continuity mode: since it provides a tiny bit of current, you can turn on LEDs by pointing them between anode and cathode of the diode. Since the RGB cathode on the single LED package are already marked on the board, that’s also not difficult to do, and while doing that I found their trick: the Blue cathods are common to all 10 LEDs, they are not separate for outer and inner groups, and more interestingly the Green cathodes are shorted to the anodes for the inner four LEDs — that means that only the outer LEDs have the full spectrum of colours available, and the only colour combination that make the two groups independent is Green/Red.

So why am I this interested in these particular lamps? Well, they seem to be a pretty decent candidate to do some “labor of love” hack – as bigclive would call it – with making them “Internet of Things” enabled: there’s enough space to fit an ESP32 inside, and with the right stuff you should be able to create a lamp that is ESPHome compatible — or run MicroPython on it, either to reimplement the insulin reminder logic, or something else entirely!

A size test print of my custom designed PCB.

Indeed, after taking a few measurement, I decided to try my hand at designing a replacement board that fits the most bases I have: a D-shaped board, with the inline microUSB, has just enough space to put an ESP32 module on it, while keeping the components on the same side of the board like in the original models. And while the ESP32 would have enough output lines to control at least the two group of LEDs without cheating, it wouldn’t have enough to address normal RGB LEDs individually… but that doesn’t need to stop a labor of love hack (or an art project): Adafruit NeoPixel are pretty much the same shape and size, and while they are a bit more expensive than the regular RGB LEDs they can be individually addressed easily.

Once I have working designs and code, I’ll be sharing, particularly in the hopes that others can improve on them. I have zero designing skills when it comes to graphics or 3D designing, but if I could, I would probably get to design my own base as well as the board: with the exception of the translucent ones, the bases are otherwise some very bland black cylinders, and they waste most of the space to allow 3×AAA batteries (which I don’t think would last for any amount of time). Instead, a 3D printed base, with hooks to hold it onto a wall (or a door) and a microUSB-charged rechargeable battery, would be a lovely replacement for the original ones. And if we have open design for the board, there’s pretty much no need to order and hope for a compatible base to arrive.

Diagonal Contributions

This is a tale that starts on my previous dayjob. My role as an SRE had been (for the most part) one of support, with teams dedicated to developing the product, and my team making sure that it would perform reliably and without waste. The relationship with “the product team” has varied over time and depending on both the product and the SRE team disposition, sometimes in not particularly healthy way either.

In one particular team, I found myself supporting (together with my team) six separate product teams, spread between Shanghai, Zurich and Mountain View. This put particular pressure on the dynamics of the team, particularly when half of the members (based in Pittsburgh) didn’t even have a chance to meet the product team of two services (based in Shanghai), as they would be, in the normal case, 12 hours apart. It’s in this team that I started formulating the idea I keep referring to as “diagonal contributions”.

You see, there’s often a distinction between horizontal and vertical contributions. Vertical referring to improving everything of a service, from the code itself, to its health checks, release, deployment, rollout, … While horizontal referring to improving something of every service, such as making every RPC based server be monitored through the same set of metrics. And there are different schools of thought on which option is valid and which one should be incentivised, and so it usually depends on your manager and their manager on which one of the two approach you’ll be rewarded to take.

When you’re supporting so many different teams directly, vertical contributions are harder on the team overall — when you go all in to identify and fix all the issues for one of the products, you end up ignoring the work needed for the others. In these cases an horizontal approach might pay off faster, from an SRE point of view, but it comes with a cost: the product teams would then have little visibility into your work, which can turn into a nasty confrontation, particularly depending on the management you find yourself dealing with (on both sides).

It’s in that situation that I came up with “diagonal contributions”: improve a pain point for all the services you own, and cover as many services you can. In a similar fashion to rake collection, this is not an easy balance to strike, and it takes experience to have it done right. You can imagine from the previous post that my success at working on this diagonal has varied considerably depending on teams, time, and management.

What did work for me, was finding some common pain points between the six products I supported, and trying to address those not with changes to the products, but with changes to the core libraries they used or the common services they relied upon. This allowed me to show actual progress to the product teams, while solving issues that were common to most of the teams in my area, or even in the company.

It’s a similar thing with rake collection for me: say there’s a process you need to follow that takes two to three days to go through, and four out of your six teams are supposed to go through it — it’s worth it to invest four to six days to reduce the process to something that takes even just a couple of hours: you need fewer net people-days even just looking at the raw numbers, which is very easy to tell, but that’s not where it stops! A process that takes more than a day adds significant risks: something can happen overnight, the person going through the process might have to take a day off, or they might have a lot of meetings the following day, adding an extra day to the total, and so on.

This is also another reason why I enjoy this kind of work — as I said before, I disagree with Randall Munroe when it comes to automation. It’s not just a matter of saving time to do something trivial that you do rarely: automation is much less likely to make one-off mistakes (it’s terrifyingly good at making repeated mistakes of course), and even if it doesn’t take less time than a human would take, it doesn’t take human time to do stuff — so a three-days-long process that is completed by automation is still a better use of time than a two-days-long process that rely on a person having two consecutive days to work on it.

So building automation or tooling, or spending time making it easier to use core libraries, are in my books a good way to make contributions that are more valuable than just to your immediate team, while not letting your supported teams feel like they are being ignored. But this only works if you know which pain points your supported teams have, and you can make a case that your work directly relates to those pain points — I’ve seen situations where a team has been working on very valuable automation… that relieved no pain from the supported team, giving them a feeling of not being taken into consideration.

In addition to a good relationship with the supported team, there’s another thing that helps. Actually I would argue that it does more than just help, and is an absolute requirement: credibility. And management support. The former, in my experience, is a tricky one to understand (or accept) for many engineers, including me — that’s because often enough credibility in this space is related to the actions of your predecessors. Even when you’re supporting a new product team, it’s likely its members have had interactions with support teams (such as SRE) in the past, and those interactions will colour the initial impression of you and your team. This is even stronger when the product team was assigned a new team — or you’re a new member of a team, or you’re part of the “new generation” of a team that went through a bit of churn.

The way I have attacked that problem is by building up my credibility, by listening, and asking questions of what the problems the team feel are causing them issues are. Principles of reliability and best practices are not going to help a team that is struggling to find the time to work even on basic monitoring because they are under pressure to deliver something on time. Sometimes, you can take some of their load away, in a way that is sustainable for your own team, in a way that gains credibility, and that further the relationship. For instance you may be able to spend some time writing the metric-exposing code, with the understanding that the product team will expand it as they introduce new features.

The other factor as I said is management — this is another of those things that might bring a feeling of unfairness. I have encountered managers who seem more concerned about immediate results than the long-term pictures, and managers who appear afraid of suggesting projects that are not strictly within the scope of reliability, even when they would increase the team’s overall credibility. For this, I unfortunately don’t have a good answer. I found myself overall lucky with the selection of managers I have reported to, on average.

So for all of you out there in a position of supporting a product team, I hope this post helped giving you ideas of how to building a more effective, more healthy relationship.

Art Projects Setbacks: Birch Books’ New Brain

You may remember that two months ago I declared my art project completed, and posted pictures of it. I have asid back then that it wasn’t quite all done and dusted, because I have been waiting for PCBs to arrive from PCBway, an online PCB ordering service that has been heavily advertising on YouTube makers’ videos for the past year or so. I ordered those back in April and, now that it’s July, they still haven’t turned up! I decided instead to follow again the advice of bigclive, and tried JLCPCB instead, which turned out to be a very good idea: ordered on Sunday, the boards arrived on Friday!

So on Friday I set myself to spend some of the time listening to training talks, and solder on the boards, which turned out to be a bit less practical than I intended — I might try again to solder microUSB connectors, but then I’ll follow Richard’s advice and try without the shielding in the back. After some screwups, I managed to get working boards, programmed a new STC89, and went on to connect it to the LEGO set — and it all turned up at once, which it wasn’t meant to!

After trying a few combinations of different Darlington arrays, and flashing a new build of the firmware, I ended up figuring that all the micros I flasked anew were completely stuck, with all the I/O ports staying at 5V with an occasional momentary pull down. At first I thought I screwed up the R/C network for the reset line, but no, I managed to check it with the Saleae Logic Pro and the reset line behaves exactly as it should have. I also popped in the original STC I used, before the ones I ordered on AliExpress arrived from China. And it worked fine, if a bit misconfigured (the new firmware has a few fixes). I also tried it and one of the “pristine” ones on the programmer, and they all work fine, but anything I program anew fails the same way.

It’s not a firmware problem either: I tried going back in time and build even the first draft version; I tried demos written by someone else; I tried four different versions of SDCC — all to no avail. Nothing changed in stcgal so it doesn’t sound like it’s a problem there… I’m at a complete loss of what’s going on there. I honestly felt desperate about that, because it worked perfectly fine two months ago, and now it suddenly stopped.

So instead, I spent Saturday working on my plan B: using CircuitPython for the main scenes logic, with an Adafruit board — in particular I decided to start working with the Feather M0, but I’m targeting the Trinket M0, which is cheaper, smaller, and still plenty powerful for what I need. This was also fun because, since I designed the actuator board to be separate from the MCU and in particular to be possible to plug it into a breadboard, which means that half of the design was already known working and didn’t need redesign.

Unfortunately what I didn’t think of was to produce a test board that would just plug into the pins on the actuator board to test it, without connecting it to the bookstore… so I ended up having to manually wire a lot of LEDs on the breadboard to turn them on. I’m going to use this as a visible example of why you should always build a fake of your service if you’re writing a client, to run integration testing with! I am addressing this by ordering an explicit testing board to connect on top of the actuator, so that I can just use it. Also fun fact: it looks like the LEDs that I have used in this pictures are… more sensible than the other ones, and Sparky the Blue Smoke Monster came to visit me again when I gave it a straight 5V.

There’s more jankiness going around with this board, though. When I was looking at expanding the I/O capabilities of the Feather, I ended up buying a few MCP23016 expanders. These are I²C chips that provide access to 16 input or output lines while only requiring two wires on the MCU. I can’t for the life of me remember or figure out why I went with this particular model, that currently sports a «Not Recommended for new designs.» warning on the top of the Microchip product page. I might as well have mistyped the MCP23017, which is the modern version of the same chip.

Beside not being (at all) pin compatible, the documentation on Adafruit’s website is designed for the later (’17) variant, and indeed the older (’16) version is not currently supported by CircuitPython (although I’m addressing this). It doesn’t stop there: 16 requires an additional R/C network, which turned out to be very unreliable: it ended up working better with a single resistor on the CLK line, and I’m not at all sure on why. So in general, the difference between MCP23016 and MCP23017 is that the latter is much nicer to use. Since I do have a few 16, and the components needed for the RC network, I’ve started writing the CircuitPython code based on that first, but also designed a separate board to fit the ’17, using a non-default address, so that I can distinguish between the two at startup.

Part of the reason for that is also that the ’16 is very finnicky, and when it receives a command it doesn’t entirely like, it decides to simply crash, and requires a full power cycle (not just a reset), which isn’t very good.

There’s another jankiness that you can possibly spot on the right side of the board: why is there a transistor, in this fairly minimalistic design? I mean okay, the pull-up resistors for I²C lines, and the RC network already ruined the idea of having the daughterboard and the expander only. But a transistor? Well, it turns out when I designed the actuator board I made a silly mistake: I didn’t consider that reset lines (RST) are not always active high (as they are on the 8051), but are sometimes actually ¬RST — and as you can imagine now, both the Feathers and the Trinket M0 are indeed active low.

And since pushing the Reset button is bridging the RST line of the actuator board to VIO (hey, at least I did realise that if I went to a different platform I would need a different “logic high”!), it wouldn’t work at all. Instead, it now activates the transistor, so that it bridges the RST line to ground. Basically, a NOT gate implemented with a single transistor — which I might have remembered thanks to Richard Feyman Lectures on Computation, which is a book I actually would recommend for those interested.

And if you’re wondering why the I/O lines are all orange, I checked which equipment wire roll I had the most of, turned out to be orange.

Also, a word of advise that I found out by complete random chance as well: the default for “quote dimensions” in Eagle is to put them in as traces at the top of the board! When I sent the generated gerber files to JLCPCB this time, they noted some dead shorts and asked me to confirm if that was intentional — so kudos to them for noticing. They allowed me to upload new, fixed gerber files after that.

Was Acronis True Image 2020 a mistake?

You may remember that a few months ago I complained about Acronis True Image 2020. I have since been mostly happy with the software, despite it being still fairly slow when uploading a sizable amount of changed files, such as after shooting a bunch of pictures at home. This would have been significantly more noticeable if we had actually left the country since I started using it, as I usually shoot at least 32GB of new pictures on a trip (and sometimes twice as much), but with lockdown and all, it didn’t really happen.

But, beside for that, the software worked well enough. Backup happened regularly, both on the external drive and the Cloud options, and I felt generally safe with using it. Until a couple of weeks ago, when suddenly it stopped working, and failed with Connection Timeout errors. They didn’t correlate with anything: I did upgrade to Windows 10 20H1, but that was a couple of weeks before, and backups went through fine until then. There was no change in network, there was no change from my ISP, and so on.

So what gives? None of the tools available from Acronis reported errors, ports were not marked as blocked, and I was running the last version of everything. I filed a ticket, was called on the phone by one of their support people who actually seemed to know what he was doing — TeamViewer at hand, he checked once again for connectivity, and once again found that everything is alright, the only thing he found to change was disabling the True Image Mounter service, which is used to get quick access to the image files, and thus is not involved in the backup process. I had to disable tha tone because, years after Microsoft introducing WSL, enabling it breaks WSL filesystem access altogether, so you can’t actually install any Linux distro, change passwords in the ones you already installed, or run apt update on Debian.

This was a week ago. In the meantime support asked me to scan the disks for errors because their system report reported one of the partitions as having issues (if I read their log correctly, that’s one of the recovery images so it’s not at all related to the backup), and the more recent one to give them a Process Monitor log while running the backup. Since they don’t actually give you a list of process to limit to, I ended up having to kill most of the other running application to take the log, as I didn’t want to leak more information that I was required to. It still provided a lot of information I’m not totally comfortable with having provided. And I still have no answer, at the time of writing.

It’s not all here — the way you provide all these details to them is a fairly clunky: you can’t just mail them, or attach them through their web support interface, as even their (compressed) system report is more than 25MB for my system. Instead what they instruct you to do is to take the compressed files and uploaded them through FTP with the username/password pair they provide to you.

Let me repeat that. You upload compressed files, that include at the very least most of the filenames you’re backing up, and possibly even more details of your computer, with FTP. Unencrypted. Not SFTP, not FTPS, not HTTPS. FTP. In 2020.

This is probably the part that makes my blood boil. Acronis has clearly figured out that the easiest way for people to get support is to use something that they can use very quickly. Indeed you can still put an FTP URL In the location bar of your Windows 10 File Explorer, and it will allow you to upload and download files over it. But it does that in a totally unencrypted, plain-text manner. I wonder how much more complicated it would be to use at least FTPS, or to have an inbound-only password-protected file upload system, like Google Drive or Dropbox, after all they are a cloud storage solution provider!

As for myself, I found a temporary workaround waiting for the support folks to figure out what they likely have screwed up on their London datacenter: I’m backing up my Lightroom pictures to the datacenter they provide in Germany. It took three days to complete, but it at least gives me peace of mind that, if something goes horribly wrong, at least the most dear part of my backup is saved somewhere else.

And honestly, using a different backup policy than the rest of the system just for the photos is probably a good idea: I set it to “continuous backup”, because generally speaking it usually stays the same all the time, until I go and prepare another set to publish, then a lot of things change quickly and then nothing until the next time I can do it.

Also, I do have the local backup — that part is still working perfectly fine. I might actually want to use it soon, as I’m of two minds between trying to copy over my main OS drive from a 1TB SSD to a 2TB SSD, and just getting a 2TB SSD, and installing everything anew onto it. If I do go that route, I also will reuse the 1TB SSD onto my NUC instead, which right now is running with half SATA and half NVMe storage.

Conclusions? Well, compared to the Amazon Glacier + FastGlacier (that has not been updated in just over two years now, and still sports a Google+ logo and +1 button!), it’s still good value for money. I’m spending a fraction of what I used to spend with Amazon, and even in the half-broken state it’s backing up more data and has significantly faster access. The fact that you can set different policies for different parts of the backup is also a significant plus. I just wish there was a way to go from a “From Folders to Cloud” backup to a tiered “From Folders to External, plus Cloud” — or maybe I’ll bite the bullet and, if it’s really this broken, just go and re-configure also the Lightroom backup to use the tiered option.

But Acronis, consider cleaning up your support act. It’s 2020, you can’t expect your customers to throw you all their information via unencrypted protocols, for safety’s sake!

Update 2020-06-30: the case is now being escalated to the “development and cloud department” — and if this is at all in the same ballpark as the companies I worked for it means that something is totally messed up in their datacenter connectivity and I’m the first one to notice enough to report to them. We’ll see.

Kodi, NUC, and CEC adapters

Welcome to this year’s yearly Kodi post. The fact that I don’t write about Kodi very often should be a sign that it’s actually fairly stable, and indeed beside for the random fight with X11 over whether DPMS should or should not be enabled, my HTPC setup works fairly well, even though it’s getting used less often — the main content I have on it is stuff that we own in DVDs — either because we ripped it ourselves or… found it to make it easier on us.

In part due to my poking around my other TV-related project, and in part because my phone seems to have issues at keeping its IPv6 stable – which means I can’t control Kodi with the Kore app until I turn wifi off and on again – I decided to address the biggest missing features of Intel’s NUC: the lack of HDMI CEC support — and as far as I know that’s a missing feature even on their most recent NUCs.

With CEC, HDMI-connected devices can provide a level of control, including signaling a request to turn on or off, to change the input selection, and so on. It also allows a TV remote control to send most button presses to a connected device. This allows, for instance, to have control over Kodi’s menus with a single remote control, rather than having to set up a second remote just for that, which is what I was originally planning on doing, using a vintage SVHS remote control.

As I said, Intel doesn’t support CEC on their NUC, to this day. Instead they suggest you to buy an adapter from companies such as Pulse Eight, which thankfully still also sell the harness for my six years old model. The adapter is a microcontroller that is connected to an Intel-provided header, from which it receives the CEC signals coming from the HDMI port on one side, and exposes an USB device through the motherboard header, from which the libCEC – that Pulse Eight themselves developed – can decode it and provide information to the other software.

The Pulse-Eight installation instructions gloss over the most interesting part: which pins does it need on the Custom Solutions Header, and more interesting how do you get to them. It turns out that I had to disassemble most of the NUC to get to the connector, and only found the right pins thanks to Kingj who reviewed this kit five years ago. The thing that his review seemed to ignore, though, is that the connectors are just about as tall as the same there is to use that theader — and the cable rubs onto the WiFi antenna (which I actually never use, so I was actually tempted to remove, until I remember they do Bluetooth, too). And the suggested position to stick the small board onto the Ethernet connector bumps into the heat foam that’s meant to protect the mSATA drive. But beside those two issues, it fit all together, and with a bit of worry and fear to break stuff, and possibly one of the black pip retainers broken, I did manage to get the whole thing installed just fine.

Troubleshooting Pulse-Eight CEC Adapter Issues

After installing and booting up the NUC, I found myself nearly crying from the frustration when I booted up and saw the adapter turn up on lsusb as Atmel at90usb162 DFU bootloader. This appears to be a common bane of multiple people, and at least on Windows people suggested that a firmware update would help — the firmware update being only available as an option on Windows (and possibly Wine?), but not released on Linux. Turns out that this is a red herring.

The DFU bootloader is the Device Firmware Update — this is the mode you need a microcontroller (MCU) to be in to load it with a new program. Usually there’s two ways to enter this operation: a physical one and a logical one (sending a specific command via USB, for instance), and while I didn’t expect them to be stateful, they appear to be. At first I was afraid I grounded something I shouldn’t have (which would have put into firmware download mode), or that the device left the factory without being programmed (which would have indeed been fixed by running the firmware update).

Instead, it turns out it might just be stateful, and was left in programming mode when it left the factory — and you can fix that with a single command on Linux: dfu-tool attach. The command is provided by fwupd (LVFS), and worked like a charm on my board for it to show up as the /dev/ttyACM0 it needed to be.

The next problem was figuring out why Kodi was not acting on any of the remote button presses. According to a lot of blogs, and forum posts, and wikis, this might have had to do with remote.xml. It didn’t, at least for me. The problem is that Kodi recognizes the presence of the CEC device by looking at the USB devices connected, but it doesn’t check if it has permissions to open the device, until it tries to. And the device (/dev/ttyACM0, which is usually the easiest no-drivers-needed way to connect an USB-to-UART bridge), is usually owned by the dialout group, which my Kodi user was not part of.

Why dialout and what the heck does it mean? Given it is 2020 as I write this, it might be worth giving a bit of a history lesson, without being condescending to those who might not have heard the term before. This group is usually “in charge” of serial port devices, such as ttyS0 and ttyACM0 and ttyUSB0 — the reason for that is that back in the old days of phone-line connectivity (which for some are current days still), the main use for serial ports was to connect modems (or in some cases faxes) — and that meant that whoever could access the serial ports would be able to “dial out” — that is, make a phone call. Since phone calls could be (and possibly still are) very expensive, accessing those ports couldn’t be made too easy. This is now a totally legacy naming, but… well, it’s hard to change these things, particularly in UNIX-land.

I solved this by adding Kodi to the group. The NUC doesn’t have any other serial port, so it’s not an issue to do that. The alternative would be to have an udev rule that explicitly sets the device as owned by the user running Kodi, similar to the rules in glucometerutils.

So now the device works, Kodi can receive the commands just fine, and I don’t have to bother to get the Kore app out just to select the next episode of whatever we’re watching at any one time. The only thing that remains annoying to me is that I can’t access the CEC device from any other process. I was wondering if I could implement most of what I wanted in the project I described previously (namely, controlling TV inputs) through CEC (answer: probably), but that’s a moot point because serial devices can only be accessed by a single process at a time (the same is true of most other devices — this is the reason why Windows has device drivers, and why you end up with so many “daemons” in Linux and other *nix: you need something that “multiplexes” the account across different processes).

If this was ten years ago, you’d have me design a CEC Daemon, and proof-of-concept integrate it in Kodi. Time being what it is, I’m unlikely to be working on this any time soon. But if someone feels like it’s a worthy task, I’ll be happy to chat, discuss designs, or review code to implement it.

Falsehoods in Tutorials: Database Schemas

It’s well possible that a number of people reading this post have already stumbled across a few of the “Falsehoods Programmers Believe…” documents. If not, there appears to be a collection of them, although I have honestly only read through the ones about names, addresses, and time. The short version of all of this, is that interfacing software with reality is complicated, and in many cases, programmers don’t know how complicated it is at all. And sometimes this turns into effectively institutional xenophobia.

I have already mused that tutorials and documentation are partially to blame, by spreading code memes and reality-hostile simplifications. But now I have some more evidence of this being the case, without me building an explicit strawman like I did last time, and that brings me to another interesting point, in regards to the raising importance of getting stuff right beforehand, as costs to correct these mistakes are raising.

You see, with lockdown giving us a lot of spare time, I spent some of it on artsy projects and electronics, while my wife spent it learning about programming, Python, and more recently databases. She found a set of tutorials on YouTube that explain the basis of what a database is, and how SQL works. And they were full of those falsehoods I just linked above.

The tutorials use what I guess is a fairly common example of using a database for employees, customers, and branches of a company. And it includes in the example the fields for first name and last name. Which frankly is a terrible mistake — with very few exception that include banks and airlines, there’s no need to distinguish between components of a name, and a simple full name field would work just as well, and don’t end up causing headaches to people from cultures that don’t split names the same way. The fact that I recently ranted about this on Twitter against VirusTotal is not totally coincidental.

It goes a bit beyond that though, by trying to explain ON DELETE triggers by attaching them to the deletion of an employee from the database. Now, I’m not a GDPR lawyer, but it’s my understanding that employee rosters are one of those things that you’re allowed to keep for essential business needs — and you most likely don’t want to ever delete employees, their commissions payment history, and tax records.

I do understand that a lot of tutorials need to be using simple examples, as setting up a proper HR-compatible database would probably take a lot more time, particularly with compartmentalizing information so that your random sales analyst don’t have access to the home phone numbers of their colleagues.

I have no experience with designing employee-related database schemas, so I don’t really want to dig myself into a hole I can’t come out of, by running with this example. I do have experience with designing database schemas for product inventory, though, so I will run with that example. I think it was a different tutorial that was talking about those, but I’ll admit I’m not sure, because I didn’t pay too much attention as I was getting annoyed at the quality.

So this other tutorial focused on products, orders and sales total — its schema was naïve and not the type of databases any real order history system would use — noticeably, it assumed that an order would just need to connect with the products, with the price attached to the product row. In truth, most databases like those would need to attach the price for which an item was sold to the order — because products change prices over time.

And at the same time, it’s fairly common to want to keep the history of price changes for an item, which include the ability to pre-approve time-limited discounts, so a table of products is fairly unlikely to have the price for each item as a column. Instead, I’ve commonly seen these database to have a prices table that references the items, and provides start and end dates for the price. This way, it’s possible to know at any time what is the “valid price” for an item. And as some of my former customers had to learn on their own, it’s also important to separate which VAT is used at which time.

Example ER diagram showing an example of a more realistic shop database.

There are five tables. * indicates the primary key.

Order (*ID, Customer_ID, Billing_Address, Shipping_Address)
Order_Products(*Order_ID, *Product_ID, Gross_Price, VAT_Rate)
Product(*ID, Name)
Product_VAT(*Product_ID, *Start_Date, End_Date, VAT_Rate)
Product_ID(*Product_ID, *Start_Date, End_Date, Gross_Price)

This is again fairly simplified. Most of the shopping systems you might encounter use what might appear redundant, particularly when you’re taught that SQL require normal form databases, but that’s just in theory — practice is different. Significantly so at times.

Among other things, if you have an online shop that caters to multiple countries within the European Union, then your table holding products’ VAT information might need to be extended to include the country for each one of them. Conversely, if you are limited to accounting for VAT in a single country you may be able to reduce this to VAT categories — but keep in mind that products can and do change VAT categories over time.

Some people might start wondering now why would you go through this much trouble for an online store, that only needs to know what the price is right now. That’s a good point, if you happen to have multiple hundreds’ megabytes of database to go through to query the current price of a product. In the example above you would probably need a query such as

SELECT Product.ID, Product.Name, Product_Price.Gross_Price, Product_VAT.VAT_Rate
FROM Product
  LEFT JOIN Product_Price ON Product_Price.Product_ID = Product.ID
  LEFT JOIN Product_VAT ON Product_VAT.Product_ID = Product.ID
WHERE
  Product.ID = '{whatever}' AND
  Product_Price.Start_Date <= TODAY() AND
  Product_Price.End_Date > TODAY() AND
  Product_VAT.Start_Date <= TODAY() AND
  Product_VAT.End_Date > TODAY();

It sounds like an expensive query, doesn’t it? And it seems silly to go and scan the price and VAT tables all the time throughout the same day. It also might be entirely incorrect, depending on its placement — I do not know the rules of billings, but it may very well be possible that an order be placed close to a VAT change boundary, in which case the customer could have to pay the gross price at the time of order, but the VAT at shipping time!

So what you do end up using in many places for online ordering is a different database. Which is not the canonical copy. Often the term used for this is ETL, which stands for Extract, Transform, Load. It basically means you can build new, read-only tables once a day, and select out of those in the web frontend. For instance the above schema could be ETL’d to include a new, disconnected WebProduct table:

The same ER diagram as before, but this time with an additional table:

WebProduct(*ID, *Date, Name, Gross_Price, VAT_Rate)

Now with this table, the query would be significantly shorter:

SELECT ID, Name, Gross_Price, VAT_Rate
FROM WebProduct
WHERE ID = '{whatever}' AND Date = TODAY();

The question that comes up with seeing this schema is “Why on Earth do you have a Date column as part of the primary key, and why do you need to query for today’s date?” I’m not suggesting that the new table is generated to include every single day in existence, but it might be useful to let an ETL pipeline generate more than just one day’s worth of data — because you almost always want to generate today’s and tomorrow’s, that way you don’t need to take down your website for maintenance around midnight. But also, if you don’t have any expectation that prices will fluctuate on a daily basis, it would be more resource-friendly to run the pipeline every few days instead of daily. It’s a compromise of course, but that’s what system designing is there for.

Note that in all of this I have ignored the issue of stock. That’s a harder problem, and one that might not actually be suited to be solved with a simple database schema — you need to come to terms with compromises around availability and the fact that you need a single source of truth for how many items you’re allowed to sell… consistency is hard.

Closing my personal rant on database design, there’s another problem I want to point a spotlight to. When I started working on Autotools Mythbuster, I explicitly wanted to be able to update the content, quickly. I have had multiple revisions of the book on the Kindle Store and Kobo, but even those lagged behind the website a few times. Indeed, I think the only reason why they are not lagging behind right now is that most of the changes on the website in the past year or two have only been cosmetics, and not applying to ePub.

Even for a project like that, which uses the same source of truth for the content, there’s a heavy difference in the time cost of updating the website rather than the “book”. When talking about real books, that’s an even bigger cost — and that’s without going into the print books realm. Producing content is hard, which is why I realised many years ago that I wouldn’t have the ability to carve out enough time to make a good author.

Even adding diagrams to this blog post has a slightly higher cost than just me ranting “on paper”. And that’s why sometimes I could add more diagrams with my ideas, but I don’t, because the cost of producing it, and keeping it current would be too high. The Glucometers Protocols site as a few rough diagrams, but they are generated with blockdiag so that they can be edited quickly.

When it comes to online tutorial, though, there’s an even bigger problem: the possibly vast majority of them are, nowadays, on YouTube, as videos shot with a person in frame, to be more like a teacher in a classroom, that can explain things. If something in the video is only minimally incorrect, it’s unlikely that those videos would be re-shot — it would be an immense cost in time. Also, you can’t just update a YouTube video like you do a Kindle book — you lose comments, likes, view-counts, and those things matter for monetization, which is what most of those tutorials out there are made for. So unless the mistakes in a video-tutorial are Earth-shattering, it’s hard to expect the creators to go and fix them.

Which is why I think that it’s incredibly important to get the small things right — Stop using first and last name fields in databases, objects, forms, and whatever else you are teaching people to make! Think a bit harder as for how a product inventory database would look like! Be explicit in pointing out that you’re simplifying to an extreme, rather than providing a real-world-capable design of a database! And maybe, just maybe, start using examples that are ridiculous enough that they don’t risk being used by a junior developer in the real world.

And let me be clear on this: you can’t blame junior developers for making mistakes such as using a naïve database schema, if that’s all they are taught! I have been saying this at previous dayjob for a while: you can’t complain about the quality of code of newbies unless you have provided them with the right information in the documentation — which is why I spent more time than average on example code, and tutorials, to fix up trimmings and make it easier to copy-paste the example code into a working change that follows best practices. In the words of a colleague wiser than me: «Example code should be exemplar.»

So save yourself some trouble in the future, by making sure the people that you’re training get the best experience, and can build your own next tool to the best of specs.

Google Maps: Spam and Scams

Last year I wrote an analysis of fishy ads on Facebook, among other things because I had no way to escalate the level of scam that I was getting, and I thought that it would be good for others to be able to follow the breadcrumbs to recognize that type of scams. Since this year I switched company, I find myself being unable to escalate the massive amount of scams and spam I see around me in Google Maps… so let me try to show you what I mean.

For the past two and a half years I’ve been living in West London — I’m not going to show exactly where, but it’s in Brentford. And when I moved into the apartment, I noticed that the various pictures I was taking at home (of food, the back of the router, and so on) kept popping up on my notifications with a suggestion by Google Maps to add them to a review of the building my apartment is in. The reason for that turned out to be that the building was marked as Lodging, which had it acting as a commercial accommodation, rather than a residential building. Oops, but why?

Well, turns out that it’s a smaller version of what’s going on in London and the rest of the world, and it became apparent a few months later, and very clear now. On our floor, there’s not one, but two “holiday rental” units — despite that being apparently against the leasehold agreement between the building and the owners. And indeed, both that company and another few appears to have tried taking over the Google Maps entities for the apartment buildings in our complex, and in a few of the nearby ones, particularly adding phone numbers for inquiries about the buildings — buildings that have no central phone number.

Let me try to show you how deep that rabbit hole is — on your right you can see “Pump House Crescent”, which is a road going around some tall apartment buildings not far from where we live. You can already notice that there’s a Green Dragon Estate that has a pink pin instead of the gray one — that’s because it’s currently marked up as a Lodge and it has reviews — probably for the same reason why I was prompted to add pictures from my own apartment. As far as I can tell, it’s actually part of the various blocks on the crescent, and instead holds residential apartments. I’ve sent an edit request.

When you zoom in a bit more, you can see interesting things. Turner House, Hyperion Tower, Cunningham House, Masson House, and Aitons House are all marked as either Apartment Building or Flat Complex, both of which are residential, and that’s correct. But then there’s Bridgeman House that is marked as Lodging — and that is likely wrong (edit sent and published as I’m typing, turns out Google Maps reviewers are very prompt to fix these issues when you report them).

But not far from it is the possible scam you can see on the left. 360 Serviced Apartments is even showing availability for a residential building — which they are not in. They provide their address as Masson House for Google Maps to put them there, but their actual company address according to their website is in London W5 — and the picture that they posted is quite a way away on the river, rather than where the pin is present. Can we call it scam? I think I will.

It doesn’t end here — there’s another similar scam across the road for another serviced apartment that even managed to put their website’s domain in the name of the point of interest! In this case there’s no picture of the outside, so it is possible that the inside pictures are actually appropriate — but again we’re talking about a residential building, in a residential area, that, as far as I can tell, is not cleared for subletting. Does it end here? Of course it does not.

In the gallery above you can see one of the most blatant scams that I could find on Maps in this area. As you can see from the first picture, down a lane from Kew Bridge station there’s an Apple Apartments Kew Bridge. Once you click on it, you can see that is reported as a 4-star hotel. With a picture of a station — except that the station is definitely not the National Rail Kew Bridge station, but rather the London Underground Hammersmith Station, which as any resident of West London would be able to tell you, is nowhere near Kew Bridge!

And as you scroll the gallery, you can see more pictures uploaded as if they were taken from the hotel, but they are clearly taken at different places. There’s pictures of Kew Bridge itself, as if you were able to see it from the place. And then you can see from the various reviews that this is not a 4-star hotel at all. Indeed, the star rating of UK hotels is defined by the AA, and for 4-star hotels they expect:

4 stars: Professional, uniformed staff respond to your needs or requests. Well-appointed public areas. The restaurant or dining room is open to residents and non-residents. Lunch is available in a designated eating area.

Let me remind you that it’s definitely a holiday let apartment that those “Apple Apartments” are.

I’m sure that all of these examples are not limited to Google Maps, as I remember being a “SuperUser” for FourSquare years ago, and having to review similar spam/scam situations, which is why I’ve been doing this on and off with Google Maps, both while working at Google and now. But at the same time, I don’t think it’s fair we, the public, end up having to clean up after this level of spam and abuse.

Anyway, if you live in an area that has lots of residential buildings, do take a look if there’s a lot of them marked Lodging, and that have phone numbers (particularly mobile numbers), and consider giving them a clear up if you can. It’s not just about removing scams for tourists, it’s also making sure that a residential area is not peppered with commercial corridors created by rule-bending (or rule-breaking) holiday let listings.

Controlling Your TV via (Circuit)Python

This is yet another of the pre-announced projects, and possibly one of the most procrastinated ones. I own a Sony Bravia TV I bought in Ireland in 2013, and is still working very well for our needs (we don’t watch that much TV). It’s connected to a Yamaha RX-V475 receiver on one input and a cheap HDMI switch on the other input, because there’s too many devices, although we only use three or four of them most of the time: Chromecast, Portal TV, HTPC, and PlayStation 4. They are split equally between the two inputs. So far, so good.

The problem starts with the fact that sometimes if the TV is turned on by the Chromecast or the Portal, the ARC does not initialize properly, and we hear no audio. The solution is worthy of The IT Crowd: tell the TV to use the internal speakers, then tell it to use the external speakers again — turn off and on the ARC itself. It’s annoying and takes a few keypresses.

What I have been wanting for a while is a way to ask Assistant (or Alexa) to “unfuck the TV” — that is to reset the audio channel for us, recording a macro to do that ARC off/on dance. It was for this reason I bought the Feather M4 last year, but I only ended up starting to work on it just this month.

To make this possible, the first thing I needed was to know the commands sent by the remote, and have a way to replicate them back to the TV. I already had some experience with infrared receivers as, a long time ago in a different life, I maintained the LIRC patchset for Linux 2.6, for a while. I even tried writing Qt (3) bindings for LIRC. I wonder if I can find the source code anywhere. But that was not as useful as I thought.

Thankfully, Ken Shirriff wrote good documentation on the protocol, and linked to further details, and even to a full archive of all the Sony command codes. Which made my life much easier, but not completely easy. While Adafruit has an IRRemote library, it does not correctly interpret Sony SIRC commands. I considered adding the support directly into it, but it turned out to be a bit more invasive than expected, so I ended up instead writing a separate package that included both the decoder and the encoder (this was before I started the new job, so releasing it was easy — but now I’m having to wait a little bit to continue on it.)

Now, once I managed to decode the commands, I need to be able to send them. And here’s where things get interesting. What we usually refer to as commands are a bunch of bits. These are encoded, based on the protocol, as a series of pulses, which are modulated on top of a carrier wave with a certain frequency.

Unfortunately, it turns out that talking to a Sony TV is nowhere near as easy as it might sound. Let’s try to figure out what’s going on by providing a bit of vocabulary. An infrared signal as used by most remote controls usually carries a command and, in most cases, an address to specify which device should take the command (since they are usually used in place where there are multiple devices using infrared remote controls). These are encoded in bits according to the rules of the protocol, and then converted to pulses. These pulses are then applied to a carrier wave of a certain frequency, which defines the speed at which the infrared LED is “blinking” on and off. The LED itself has a wavelength which represent the “colour” of the light in the infrared spectrum.

For Sony’s SIRC, the TV expects a 40kHz carrier, and it seems a 950nm wavelength. It seems like you can use 940nm LEDs but they have worse reception, and only works if they are bright enough. The first batch of LEDs I bought, as well as the pre-soldered module from DFRobot, turned out to not be bright enough for my TV to recognize — so I decided to try with desoldering the LED from a replacement remote control from eBay, which worked fine, so I thought I needed 950nm LEDs — but no, it works with good enough 940nm LEDs, just not with the tiny ones I originally bought from Mouser.

So once I had a way to send arbitrary Sony commands to my TV, I started looking for options to trigger the ARC reset — unfortunately this is proving more complicated than I expected: there’s no command that I could send that would provide me with the ARC menu. Instead I can only bring up the Sync menu reliably — but that menu has different entries depending on whether the selected input has working HDMI CEC, which is extremely annoying.

On the other hand, I did find commands that select directly the different inputs directly, instead of showing up the input selection menu and choosing from there. Which gave me a different idea to start with: while I haven’t given up on the macro for fixing the audio, what I can do is to script input selection across the two-level mux.

I started by wondering if I could send the IR command to the HDMI switcher as well, so that I could select between those two easily — that turned out to be another pile of yaks to shave. The switcher uses the NEC protocol, which has a 38kHz carrier wave, but that turned out not to matter as much (the decoder it uses seem to accept 40kHz just as well) — instead I had a hard time to get it to receive the command because it expected a NEC “repeat signal” to seal the command. I guess that’s going to be a blog post in and by itself.

Now, my original plan was to get something running on the Feather, attach an AirLift wing to give it WiFi, and control that… somehow. I also considered re-coding this with ESP32 and ESPHome, despite it not having an obvious way to send SIRC commands while making sense — it doesn’t represent the commands the way the protocol expects, and the most reasonable way I could find was to generate the pulse sequence, and just sending that raw.

But then I thought it over and realised that, at least for the moment, it makes more sense for me to use an USB-to-GPIO bridge, and control this from a full blown computer — the reason for that is that I would want to be able to open a webpage on my phone, or my wife’s, and select the right input altogether. And while there’s a Python module for controlling the receiver, using that from CircuitPython or MicroPython would probably take some work. And even though I could just control the Feather remotely, via MQTT or USB-Serial, it would still likely be more work than driving this all from the same Flask app locally.

Unfortunately, I don’t have code to show you yet. While my new dayjob has much simpler policies than the last, I need to complete the training before I can start releasing new projects. Hopefully next update will come with code and demos.

Paperless home, sorted home

You probably don’t remember, but I have been chasing the paperless office for many years. At first it was a matter of survival, as running my own business in Italy meant tons of paperwork, and sorting it all out while being able to access it was impossible. By scanning and archiving the invoices and other documents, the whole thing got much better.

I continued to follow the paperless path when I stopped running a company and just working, but by then, the world started following me and most services started insisting on paperless billing anyway, which was nice. In Dublin I received just a few pieces of paper a month, and it was easy to scan them, and then bring them to the office to dispose of in the secure shredding facilities. I kept this on after moving to London, despite the movers steaming my scanner, using a Brother ADS-1100W instead.

But since the days in Italy, my scanning process changed significantly: in Dublin I never had a Linux workstation, so the scanner ended up connected to my Gamestation using Windows — using PaperPort which was at the time marketed by Nuance. The bright side of this was that PaperPort applies most of the same post-processing as Unpaper while at the same time running OCR over the scanned image, making it searchable on Google Drive, Dropbox and so on.

Unfortunately, it seems like something changed recently, either in Windows 10, the WIA subsystem or something else altogether, and from time to time after scanning a page, PaperPort or the scanner freeze, and don’t terminate the processing, requiring a full reboot of the OS. Yes I tried powercycling the scanner, yes I tried disconnecting the USB and reconnecting, none seem to work except a full reboot, which is why I’m wondering if it might be a problem with the WIA subsystem.

The current workaround I have is to use the TWAIN system, which is the same that I used with my scanner on Windows 98, which is surprising and annoying — in particular I need to remember to turn on the scanner before I open PaperPort, otherwise it fails to scan and the process will need to be killed with the Task Manager. So I’m actually considering switching the scanning to Linux again.

My old scan2pdf command-line tool would help, but it does not include the OCR capabilities. Paperless seems more interesting, and it uses Unpaper itself. But it assumes you want the document stored on the host, as well as scanned and processed. I would have to see if it has integration with Google Drive, or otherwise figure out how to get that integration going with something like rclone. But, well, that would be quite a bit of work that I’m not sure I want to do right now.

Speaking of work, and organizing stuff — I released some hacky code which I wrote to sort through the downloaded PDF bills from various organizations. As I said on Twitter when I released it, it is not a work of engineering, or a properly-cleaned-up tool. But it works for most of the bills I care about right now, and it makes my life (and my wife’s) easier by having all of our bank statements and bills named and sorted (particularly when just downloading a bunch of PDFs from different companies once a month, and sorting them all.)

Funnily enough, writing that tool also had some surprises. You may remember that a few years ago I leaked my credit card number by tweeting a screenshot of what I thought was uninitialized memory in Dolphin. Unlike Irish credit card statements, British card statements don’t include the full PAN in any of the pages of a PDF. So you could think it’s safe to provide a downloaded PDF as proof of address to other companies. Well, turns out it isn’t, at least for Santander: there’s an invisible (but searchable and highlightable) full 16-digit PAN at the top of the first page of the document. You can tell it’s there when you run the file over pdf2text or similar tools (there’s a similar invisible number on bank statements, but that’s also provided visible: it’s the sort-code and account number).

Oh and it looks like most Italian bills don’t use easily-scrapeable layouts, which is why there’s none of them right now in the tool. If someone knows of a Python library that can extract text from pages using “Figure” objects, I’m all ears.

The Importance of Project Boxes

Given my background, you may think I’m referring to servers — after all I ran the tinderbox for a long while. But no, in this case I’m referring to actual, physical boxes. And a few virtual ones as well.

See, when I listed my electronics projects, it probably sounded like my house is now top-to-bottom filled with random electronics stuff. This might have been the case if I was doing that in Dublin, but nowadays I’m not alone, and while my wife accepts the need for my random projects, making a total mess of the apartment is probably not a great idea. Indeed the “study” (that is, the second bedroom turned into a home office) is not quite sorted out, but it’s also not a total mess right now (you can see it in the test Live whiteboarding).

The way I’m trying to keep all of this together, is through a significant number of boxes. I already bought a number of LEGO boxes back in Dublin — the reason for the choice was that those boxes are particularly designed to be used by kids, and that made them particularly sturdy. Unfortunately they are also not particularly big. That made them not particularly useful for storing more bulky items, such as my multimeter, the dremel or most of my screwdrivers’ set. I did bring a toolbox with me from Italy – which includes not one but two hammers, more screwdrivers, one older soldering iron, and other similar tools – but this ended up being the kind of toolbox where tools end up to die, because it’s just so uncomfortable to use.

Instead I now bought bigger, bulkier boxes — using one to store the tools, and another one to store my “electronics” stuff, most of which is stored in tiny boxes inside — so that I have a box of transistors, two of resistors, a couple of Dupont connectors, and so on. This makes it easier to sort through what I need at any one time. I considered using bigger boxes with more compartments, but it turns out that most of the time I only need three or four smaller boxes, and sorting between smaller boxes is easier. I have unfortunately not found a very good box to hold the soldering kit – TS-100 and so on – but I might end up looking for something with foam inserts if I ever figure out how to properly design those, after all.

The LEGO boxes are now (mostly) project boxes: one of them has all of the glucometers that I’ve either already reversed or plan to reversing soon; another one has the R205 boards that I never made much progress on, and so on. The fact that I have so many half-complete projects is indeed disheartening when I think of it. Unfortunately there’s reason for that: either lack of skills on my part, the lack of time for myself, or the lack of tooling at the right time. I can for instance tell you that I definitely would have a much easier time nowadays trying to solder those those pins, not just because I improved my soldering skills, but also because I now have tools that I didn’t even know existed back then, including the Dupont crimper.

And this is where the project boxes come into play. Being able to put everything for one particular project into a box and not worry about it for a while is a liberating experience. I had to literally do that when I started my new job — it’s not that difficult passing from one Big Tech company to another, but it is time consuming, and after spending nearly eight straight hours in front of a monitor watching videos and/or reading tutorials, the last thing I wanted was to spend some more time in front of the computer… having all the half-projects in front of me was tough, and I preferred shoving everything into a project box, and going back to it a week or so later — I have indeed gone back to one of my projects already and will likely have a blog post about it later.

But as I said, there’s also “metaphorical” project boxes — that work for me both at work and at home: there are times when working on the same project for months at a time is not feasible. It may be that what was once a priority isn’t anymore, or a higher priority comes out. Or maybe you’re just too burnt out to work on one thing, and decided that it’s time to work on something else for a while. Having the ability to set aside a project, and come back to it later, having state and notes about what something was about is something important. For most of my opensource projects, I leave my own notes in the blog, which is why there’s so many posts of old projects that have not gone anywhere. For work projects, I try to leave design docs or notes, and for other stuff, I started having proper journals dedicated to the ideas. That’s how Birch Books started as well.

The main advantage I found for doing this at work is that it allows collecting rakes without losing track of the main objective of your job. Many of the examples I gave in the previous blog post on the topic have been months-long efforts. Some of those took two years to close the reported issues! And in at least one case, the work had to literally be halted, and wait for two quarters, because one of the blockers could only be solved by deleting the whole source of a service that was being replaced.

So what I’m advocating about is the ability to keep things around, but out of sight. It’s the reason why I can’t stand certain policies in projects (at work and FLOSS alike) that pretty much disallow “long-standing bugs”: sometimes you want to keep all the information you have on a project together, and a bug/issue/task that has been opened for a number of years might just be the right way — closing it for the sake of an OKR target is unlikely to do anyone a good service.

And also allowing ourselves (all of us), to box up projects — whether it is for a while or forever. Being reminded of the stuff you have not completed is not going to make you feel any better, in my experience. Being able to say “Yeah it’s in the box, one day I might or might not get to it” is a way to breathe in, and give yourself space.