Senior Engineering: Give Them Time To Find Their Own Stretch

This blog post is brought to you by a random dream I had in December.

As I reflect more on what it means for me to be considered a senior engineer, I find myself picking up answers from my own dreams as well. In a dream this past December, I felt myself being at the office (which, given the situation, is definitely a dream state) but also in particular I found that I was hosting a number of people from my old high school.

As it turns out it’s well possible that the reason why I dreamt of my old high school is because I was talking with my wife about how I would like there to be an organization that would reach out in schools such as that one. But that’s not what the dream was about. And it probably was helped along by having seen a Boston Legal episode about high schools the night before.

In the dream, I was being asked by one of the visitors what would I would recommend them to do, so that more of their students would be successful, and work at big companies. Small aside: there were three of us from my high school, and my year, working in Google when I joined – as I understand one of them was in New York and then left, but I was never really in touch with him – the other guy was actually in Dublin the same as me for a while, and I knew him well enough that I visited him when he moved away.

What I answered was something along the lines of: just don’t fill them of homework that takes them time. They won’t need the homework. They need to find something they can run with. Give them time to find their own stretch.

This is likely part of my personal, subjective experience. I generally hated homework from the beginning, and ignored most of it most of the time, if I could get away with it (and sometimes even when I couldn’t). Except when the homework was something that I could build upon myself. So while on most of the days I would be coming home from school and jump on Ultima OnLine right after lunch, other times I would be toying with expanding on what my programming teacher gave me or, as I told already, with writing a bad 8086 emulator.

But it reminded me of the situation at work, as well — “stretch” being a common word used to refer to the work undertaken by tech employees outside of their comfort zone or level, to “reach” their next goal or level. This plays especially into big tech companies, where promotions are given to those who already act as the next level they’re meant to reach.

While thankfully I only experienced this particular problem first hand a few times, and navigated most of them away, this is not an uncommon situation. My previous employer uses quarterly OKR planning to set up the agenda for each quarter, and particularly in my organization, the resulting OKRs tended to over-promise, by overcommitting the whole team by design. That meant taking on enough hi-priority tasks (P0 as they are usually referred to) to take the whole time of many engineers just to virtually keep the lights on.

Most of this is a management problem to solve, particularly when it came to setting expectations with partner teams about the availability of the engineers who could be working on the projects, but there’s a number of things that senior engineers should do, in my opinion, to prevent this from burning junior engineers.

When there’s too many “vertical” stakeholders, I still subscribe to the concept of diagonal contributions – work on something that brings both you, and many if not all of your verticals, in a better place, even when you could bring a single vertical to a much better place ignoring all the others – but for that to work out well, you need to make the relationship explicit, having buy-in from the supported teams.

The other important point is just to make sure to “manage upwards”, as I’ve been repeatedly told to do in the previous bubble, making sure that, if you have juniors in your team, they are given the time for training, improving, collaborating. That means being able to steer your manager away from overcommitting those who need more time to ramp up. It’s not a hypothetical — with my last manager at my previous role, I had to explicitly say a few times “No, I’m not going to delegate this to Junior Person, because they are already swamped, let them breathe.”

As I said previously, I think that it’s part and parcel of a senior engineer job to let others do the work, and not being the one concentrating the effort — so how do you reconcile the two requirements? I’m not suggesting it’s easy, and it definitely varies situation by situation. In the particular case I was talking about, the team was extremely overcommitted, and while I could have left a task for Junior Person to work on for a quarter or maybe two, it would have gone to be an urgent task afterwards, as it was blocking a migration from another team. Since it would have taken me a fraction of the time, I found it more important for Junior Person to find something that inspired them, rather than something that our manager decided to assign.

Aside: a lot can be written about the dynamics of organizations where managers get to say “I know they said the deadline is Q4, but nobody believes that, we’ll work on it in Q1 next year.” The technical repercussion of this type of thinking is that migrations drag along a lot longer than intended and planned, and supporting the interim state makes them even more expensive on both the drivers and receivers of the migration itself. Personally, if I’m told that Q4 is a deadline, I always tried to work towards migrating as many of the simple cases by Q3. That allowed me identifying cases that wouldn’t be simple, making requests to the migration drivers if needed, and left Q4 available for the complex cases that needed a lot more work. If you don’t do that, you risk assuming that something is a simple case up until the deadline, and then find out it’s another quarter worth of project work to address it.

Of course, a lot of it comes down to incentives, as always. My understanding of the state of it before I left my previous company is that the engineers are not evaluated directly on their OKR completion, but managers are evaluated on their team’s. Making that clear and explicit was itself a significant life improvement I would say — particularly because this didn’t use to be the case, and I was not alone in having seen the completion rate as being a fantasy number that didn’t really do much — much has been written by others on OKR and planning, and I’m no expert at that, so I won’t go into details of how I think about this altogether.

Keeping this in mind, another factor in providing stretch to more junior engineers, is to provide them with goals that are flexible enough for them to run with something they found interesting — leaving your task as the senior person to make sure the direction is the right one. To give an example, this means that instead of suggesting one specific language for implementing a tool, you should leave it up to the engineer working on it to choose one — within the language choice the team has accept, clearly.

It also means not requiring specific solutions to problems, when the solution is not fully fleshed out already. Particularly when looking at problems that apply to heterogenous systems, with different teams maintaining components, it’s easy to fall into the trap of expecting the solution to be additive – write a new component, define a new abstraction layer, introduce a compatibility library – ignoring subtractive solutions — remove the feature that’s used once every two years, delete the old compatibility layer that is only used by one service that hasn’t been ported yet. Focusing on what the problem to be solved is, rather than how it needs to be solved, can give plenty of room to stretch.

At the very least this worked for me — one of the tasks I’ve been given in my first team was to migrate between two different implementations of a common provider. While I did do that to “stop the fire”, I also followed up by implementing a new abstraction on top of it, so I could stop copy-pasting the same the code in ten different applications. I maintained that abstraction until I left the company, and it went from something used within my team only, to the suggested solution to that class of problems.

The other part that I want to make sure people consider, is that deciding that what you were assigned is not important and that there’s better things to do, is not a bad thing. Trying to make it harder to point out “Hey, I was assigned (or picked up) this goal, but after talking with stakeholders directly there’s no need to work on it” is just going to lead to burnout.

To go back to the continuous example of my last team at the previous employer, I found myself having to justify myself for wanting to drop one of the goals from my OKR selection – complicated by the fact that OKR completion was put on my performance plan – despite the fact that the area expert already signed off on it. The reason? We were in a month already in the quarter, and it seemed to late in the game to change our mind. Except the particular goal was not the highest priority, and the first month of the quarter was pretty much dedicated full time to one that was. Once I started poking at the details of this lower priority goal, I realized it would not gain anything for anyone – it was trading technical debt in one place to technical debt in another – while we had a more compelling problem on a different project altogether.

Again, all of this does not lead to a cookie-cutter approach — even senior engineers joining a new space might need to be told what to do so that they are not just wasting air for a while. But if you’re selecting tasks for others to work on, make sure that they get an opportunity to extend on it. Don’t give them a fully-detailed “just write the code for this very detailed specification” if you can avoid it, but rather tell them “solve this problem, here’s the tools, feel free to ask for details”.

And make sure you are around to be asked questions, and to coach, and to show what they may not know. Again from personal experience, if someone comes to you saying that they’d like advice on how to write design docs that pass arbitrarily complicated review processes, the answer “Just write more of them” is not useful. Particularly when the process involves frustrating, time-consuming synchronous reviews with people wanting you to know the answer to all their “What if?” questions without having heard them before.

In short, if you want your juniors to grow and take your place – and in my opinion, you should want that – then you should make sure not to manage their time to the detail. Let them get passionate about something they had to fix, and own seeing it fixed.

Flashing ESPHome on White Label Smart Plugs

You may remember that I bought a smart plug a few years ago, for our Christmas tree, and had been wondering what the heck do you use a smart plug for. Well, a few months ago I found out another use case for it that I had not thought about to that point: the subwoofer.

When I moved to Dublin, nearly eight years ago now, I bought a new TV, and an AV receiver, and a set of speakers — and with these last one, came a subwoofer. I don’t actually like subwoofer that much, or at least I didn’t, because the low-frequency sounds give (or gave) me headaches, and so while it does sound majestic to watch a good movie with the subwoofer on, I rarely do that.

On the other hand, it was still preferable to certain noise coming from the neighbours, particularly if we ended up not being in the same room as the device itself. Which meant we started using the subwoofer more — but unlike the AV receiver, that can be controlled with a network endpoint, the subwoofer has a physical switch to turn on and off… a perfect use case for setting this up with a smart plug!

Since I didn’t want to reuse the original plug for the Christmas tree (among other things, it’s stashed away with the Christmas tree), I ended up going online and ordering a new one — and I selected a TP-Link, that I thought would work just the same as my previous one… except it didn’t. Turns out TP-Link has two separate Smart Home brands, and while they share the same account management, they use different apps and have different integration support. The Christmas tree uses a Kasa-branded plug, while the one I ordered was a Tapo-branded plug — the latter does not work with Home Assistant, which was a bit annoying, while the former worked well, when I used it.

When TP-Link then also got in the news for having “fixed” a vulnerability that allowed Home Assistant to control the Kasa plugs on the local network, I got annoyed enough that I decided to do something about it. (Although it’s good to note that nowadays Home Assistant works fine with the “fixed” Kasa plugs, too.)

I asked once again help to Srdjan who has more patience than me to look up random IoT projects out there, and he suggested me the way forward… which turned out a bit more tortuous than expected, so let me try to recount it here, showing shortcuts as much as I can at the time of writing

The Hardware

Things start a bit iffy to begin with — I say that because the hardware I ended up getting myself was Gosund SP111 from Amazon, after checking reviews that they were still shipping an older version of the hardware that can be converted with tuya-convert (I’ll get back to the software in a moment). There’s plenty of warnings about the new models, but in a very similar fashion to the problem with CGG1 sensors, there’s no way to know beforehand what version of a firmware you’re going to find.

But in particular, SP111 are not Gosund engineering’s own work. Tuya is a company that build “white label” goods for many different brands, and that’s how these plugs were made. You can find them under a number of different brand names, but they all share the same firmware and, critically, the same pairing and update keys.

One of the best part about these Tuya/Gosund plugs is that they are tiny in the sense of taking very little space. They are less than half the size of my first Kasa-branded TP-Link smart plug, and even when compared with the newer Tapo-branded one they are quite slimmer. This makes for adding them easily to places that are a bit on the constrained side.

The Firmware

I did not buy these devices to run with the original firmware. Unlike the CGG1 and pretty much everything else I’m running at home, this time I went straight for the “I want to run ESPHome on them.” The reason for that was on one side, I’m annoyed and burnt by the two TP-Link devices, and from the other, the price of Hue-compatible smart plugs was annoyingly high. That would have been my default alternative.

Of the various important things to me, Home Assistant support grown from a “meh, sure” to “yeah I want that” — and in part because I did manage to set up scripts for quite a bit of tools at home through it. One of my original concerns (wanting to still be able to control most of these features by voice) is taken care of by signing up for Nabu Casa, which provides integration for both Google Assistant and Alexa, so even with the plug running a custom and maybe janky, firmware, getting it to work with the rest of the house would be a breeze.

There’s other options for open source firmware for the SP111, as well as other smart plugs. Tasmota is one of the most commonly used for this, and there’s another called ESPurna. I have not spent time investigating them, because I already have one ESPHome device running, and it was easier to lower my cognitive load to use something I already kinda know. After all, the plan is to replace the Kasa plug once we take out the Christmas tree, which would remove not one but two integrations from Google and Amazon (and one from Home Assistant).

All of these options can be flashed the first time around with tuya-convert, even though the procedure ended up not being totally clear to me — although in part that was caused by my… somewhat difficult set up. This was actually part of the requirements. Most smart home devices appear to be built around ESP8622 or ESP32, which means you can, with enough convincing, flash them with ESPHome (I’m still convincing my acrylic lamp circuit board), but quite a few require you have physical access to the serial port. I wanted something I could flash without having to take it apart.

The way this works, in very rough terms, is that the Tuya-based devices can be tricked into connecting to a local open network, and from there with the right protocol, they can be convinced to dump their firmware and to update it with an arbitrary firmware binary. The tuya-convert repository includes pretty much all the things you need to set this up, neatly packaged in a nearly user-friendly way. I say nearly, because as it turns out there’s so much that can go wrong with it, that the frustration is real.

The Process.

Part 1: WiFi.

First of all, you need a machine that has a WiFi adapter that supports Access Point mode (AP mode/hostapd mode, those are the keywords to look for it). This is very annoying to know for sure, because manufacturers tend to use the same model number across hardware revisions (that may entirely change the chipset) and countries — after all, Matthew ended up turning to Xbox One controller adapters! (And as it turns out, he says they should actually support that mode, with a limited range.)

The usual “go-to” for this is to use a laptop which also has an Ethernet port. Unfortunately I don’t have one, and in particular, I don’t have a Linux laptop anymore. I tried using a couple of Live Distros to set this up, but for one reason or another they were a complete bust (I couldn’t even type on Ubuntu’s Terminal, don’t even ask.)

In my case, I have a NUC on top of my desk, and that’s my usual Linux machine (the one I’m typing on right now!) so I could have used that… except I did disable the WiFi in the firmware when I got it, since I wired it with a cable, and I didn’t feel the need to keep it enabled. This appears to have been a blessing in disguise for a while, or I would have been frustrated at one of the openSUSE updates in December, when a kernel bug caused the system to become unusable with the WiFi on. Which is what happened when I turned the WiFi on in the firmware.

Since I don’t like waiting, and I thought it would generally be a good idea to have at least one spare USB WiFi dongle at home (it would have turned useful at least once before), I went and ordered one on Amazon that people suggested might work. Except it probably got a hardware revision in the middle, and the one I received wasn’t suitable — or, some of the reports say that it depends on the firmware loaded on it; I really didn’t care to debug that too much once I got to that stage.

Fast forward a few weeks, the kernel bug is fixed, so I tried again. The tuya-convert script uses Docker to set up everything, so it sounded like just installing Docker and docker-compose on my openSUSE installation would be enough, right? Well, no. Somehow the interaction of Docker and KVM virtual machines had side effects on the networking, and when I tried I both lost connectivity to Home Assistant (at least over IPv6), and the tuya-convert script kept terminating by itself without providing any useful information.

So instead, I decided to make my life easier and more difficult at the same time.

Part 1.1: WiFi In A Virtual Machine

I didn’t want Docker to make a mess of my networking setup. I also wasn’t quite sure of what tuya-convert would be doing on my machine (yes, it’s open source, but hey I don’t have time to audit all of it). So instead of trying to keep running this within my normal openSUSE install, I decided to run this in a virtual machine.

Of course I need WiFi in the VM, and as I said earlier, I couldn’t just pass through the USB dongle, because it wouldn’t work with hostapd. But modern computers support PCI pass-through, when IOMMU is enabled. My NUC’s WiFi supports hostapd, and it’s sitting unused, since I connect to the network over a cable.

The annoying part was that for performance issues, IOMMU is disabled by default, at least for Intel CPUs, so you have to add intel_iommu=on for the option of passing through PCI devices to KVM virtual machines to be available. This thread has more information, but you probably don’t need all of it, as that focuses on passing through graphic cards, which is a much more complicated topic.

The next problem was what operating system to run in the VM itself. At first I tried using the LiveDVD of openSUSE — but that didn’t work out: the Docker setup that tuya-convert uses is pretty much installing a good chunk of Ubuntu, and it runs out of memory quickly, and when it does it throws a lot of I/O errors from the btrfs loaded into memory. Oops.

Missing a VM image of openSUSE Tumbleweed, I decided to try their image for JeOS instead, which is a stripped down version meant to be used in virtualized environments. This seemed to fit like a glove at first, but then my simplicity plans got foiled by not realizing that usually virtualized environments don’t care for WiFi. Although the utter lack of NetworkManager or any other WiFi tooling turned out to be handy to make sure that nothing tried to steal the WiFi away from tuya-convert.

In addition to changing the kernel package with a version that cares about WiFi drivers, you need to install the right firmware package for the card. After that, at least the first part is nearly taken care of — you will most likely need a few more tools, such as Git and your editor of choice. And of course, Docker and docker-compose.

And then, do yourself a favour, and turn off firewalld entirely in your virtual machine. Maybe I should have said earlier “Don’t let the VM be published to the Internet at large”, but I hope if you get to try to pass-through a WiFi device, you knew better than doing that anyway. The firewall is something that is not obvious is going to get in your way later when you set to run tuya-convert, and it’ll make the whole setup fail silently in the hardest way possible to debug.

Indeed, when I looked for my issues I found tons of posts, issues, blogs all complaining about the same symptom I had, which was all caused by having a firewall in place. The tuya-convert script does a lot of things to set up stuff, but it can’t easily take down a firewall, and that is a biggie.

Indeed, and I’ll repeat that later, at some point the instructions tell you to connect some other device to their network and suggest otherwise it might not be working. My impression is that this is done because if it doesn’t work, you shouldn’t try taking the next steps yet. But the problem is that there is no note anywhere to help you if it doesn’t work — and the reason for it failing is likely the firewall stopping the DHCP server from receiving the requests. Oops.

Part 2: The Firmware Blob

ESPHome configurations are… sometimes very personal. I have found one for the SP111 on the Home Assistant forums and adapted it, but… I don’t really feel like recommending that one. So I’m afraid I won’t take responsibility for how you configure your ESPHome firmware for the plug.

Also, once you have ESPHome on the device, changing the config is nearly trivial, from the Home Assistant integration, so I feel it’s important to have something working at first, and then worry about perfecting it.

I think someone will be confused here on why am I jumping on configuring the firmware blob before we got to convert the device to use it. The reason for that is that you want to have the binary file (either built locally or generated with the Home Assistant integration and downloaded), and you put it into tuya-convert/files/, you will be able to directly flash that version, without going through the intermediate step of using one of the bundled firmware just to be able to update to an arbitrary firmware. But to do that, it needs to happen before you complete the Docker setup.

So, find yourself a working config for the device on the forums (and if you find one that is maintained and templated, so that one can just drop it in and just configure the parameters, please let me know), and generate your ESPHome firmware from there.

Also note that the firmware itself identifies the specific device. This means you cannot flash more than one device with the same firmware or you’ll have quite the headache to sort them out afterwards. Not saying it isn’t possible, but I just found it easier to make the firmware for the devices I was going to flash, and then load each one. As usual, my favourite tool to remember what is what would be my label maker, so that I don’t mix up which one I flashed with which binary.

Part 3: The Conversion

Okay so here’s the inventory of what you should have by this point before we move on to the actual conversion:

  • a virtual machine with a passed-through WiFi card that is supported by hostapd;
  • an operating system in the VM with the drivers for the WiFi, Docker, docker-compose, and no firewall;
  • a checkout of tuya-convert repository;
  • one or more ESPHome firmware binary files (one per device you want to flash), in the files/ directory of the checkout.

Only at this point you can go and follow the instruction of tuya-convert: create Docker image, setup docker-compose, and run the image. The firmware files need to be added before creating the docker image, because docker-compose does not bind the external files/ directory at all.

Once the software starts, it’ll ask you to connect another device that is not the plug to the WiFi. I’m not entirely sure if it’s just for diagnostics, but in either case, you should be able to connect to the network — the password should be flashmeifyoucan, although I don’t think I’ve seen that documented anywhere except when googling around for other people having had issues with their process.

If you try this from your phone, you should be prompted to login into the WiFi network through a captive portal — the portal is just a single page telling you that the setup is completed. If your phone gets stuck in the “Obtaining IP Address” phase, like mine did, make sure you really took down the firewall. This got me stuck for a while because I thought that the Docker itself controlled the whole firewall settings — but that does not appear to be the case.

Final Thoughts

I guess that this guide is not going to be very useful, with the new versions of the SP111 not being supported by tuya-convert (and not clear if it can be supported), but since I have two plugs still unused, it helps me to have written down the process to avoid getting myself stuck again.

The plugs appear to have configurable sensors for voltage, amperage, and total wattage used — and the configuration of those is why I’m not comfortable sharing the config I’m using: I took someone’s from a forum post but I don’t quite agree with some of the choice made, some of the values appear fairly pointless to me.

Voltage monitoring would have been an useful piece of information when I was back in Italy — those who read the blog a long time ago might remember that the power company over there didn’t really have any decent power available. Over here it feels like it’s very stable, so I doubt we’ll notice anything useful with these.

Having Smart Home devices that don’t rely on cloud services is much more comfortable than otherwise. I do like the idea of being able to just ask one of the voice assistants to turn off the subwoofer while I’m playing Fallout 76, for sure — but it’s one thing to have the convenience, and another to depend on it to control it. And as I said some time ago, I disagree with the assertion that there cannot be a secure and safe IoT Smart Home (and yes, “secure” and “safe” are two separate concepts).

As for smart plugs in particular? I’m still not entirely sold, but I can see that there definitely are devices where trying to bring the smart in the device is unlikely to help. Not as many though — it’s still a problem to find something that cannot be served better by more fine-grained control. In the case of the subwoofer, most of the controls (volume, cross-over, phase) are manual knobx on the back of the device. Would it have made sense to have a “smart subwoofer” that can tweak all of those values from the Home Assistant interface? I would argue yes — but at the same time, I can see in this case an expense of £10 for a smart plug beats the idea of replacing the subwoofer entirely.

I honestly have doubts about the Christmas tree lights as well. Not that I expect to be able to control them with an app, but the “controller” for them seems to be fairly standard, so I do expect if I search AliExpress for some “smart” controller for those I will probably find something — the question is whether I would find something I can use locally without depending on an external cloud service from an unknown Chinese brand. So maybe I’ll go back to one of my oldest attempts at electronics (13 years ago!) and see what I can find.

By the way, if you’re curious what else I am currently planning to use these smart plugs on… I’m playing with the idea of changing my Birch Books to use 12V LEDs – originally meant for Gunpla and similar models – and I was thinking that instead of leaving it always-on, I can just connect it with the rest of the routines that we use to turn the “living” items on and off.

Kind Software

This post sprouts in part from a comment in my previous disclaim of support to FSFE, but it’s a standalone post, which is not related to my feelings towards FSFE (which I already covered elsewhere). It should also not be a surprise to long time followers, since I’m going to cover arguments that I have already covered, for better or worse, in the past.

I have not been very active as a Free Software developer in the past few years, for reasons I already spoke about, but that does not mean I stopped believing in the cause or turned away from it. At the same time, I have never been a fundamentalist, and so when people ask me about “Freedom 0”, I’m torn, as I don’t think I quite agree on what Freedom 0 consists of.

On the Free Software Foundation website, Freedom 0 is defined as

The freedom to run the program as you wish, for any purpose (freedom 0).

At the same time, a whole lot of fundamentalists seem to me to try their best to not allow the users to run the programs as they wish. We wouldn’t, otherwise, be having purity tests and crusade against closed-source components that users may want to actually use, and we wouldn’t have absurdist solutions for firmware, that involve showing binary blobs under the carpet, and just not letting the user ever update them.

The way in which I disagree with both formulation and interpretation of this statement, is that I think that software should, first of all, be usable for its intended purpose, and that software that isn’t… isn’t really worth discussing about.

In the case of Free Software, I think that, before any licensing and usage concern, we should be concerned about providing value to the users. As I said, not a novel idea for me. This means that software that that is built with the sole idea of showing Free Software supremacy, is not useful software for me to focus on. Operating systems, smart home solutions, hardware, … all of these fields need users to have long-term support, and those users will not be developers, or even contributors!

So with this in mind, I want to take a page out of the literal Susan Calman book, and talk about Kind Software, as an extension of Free Software. Kind Software is software that is meant for the user to use and to keep the user as its first priority. I know that a number of people would make this to be a perfect overlap and contrast, considering all Free Software as Kind Software, and all proprietary software as not Kind Software… but the truth is that it is significantly more nuanced than that.

Even keeping aside the amount of Free Software that is “dual-use” and that can be used by attackers just as much as defenders – and that might sometimes have a bit too much of a bias towards the attackers – you don’t need to look much further than the old joke about how “Unix is user friendly, it’s just very selective of who its friends are”. Kind software wouldn’t be selective — the user use-cases are paramount, any software that would be saying “You don’t do that with {software}, because it is against my philosophy” would by my definition not be Kind Software.

Although, obviously, this brings us back to the paradox of tolerance, which is why I don’t think I’d be able to lead a Kind Software movement, and why I don’t think that the solution to any of this has to do with licenses, or codes of ethics. After all, different people have different ideas of what is ethical and what isn’t, and sometimes you need to make a choice by yourself, without fighting an uphill battle so that everyone who doesn’t agree with you is labelled an enemy. (Though, if you think that nazis are okay people, you’re definitely not a friend of mine.)

What this tells me that I can define my own rules for what I consider “Kind Software”, but I doubt I can define them for the general case. And in my case, I have a mixture of Free Software and proprietary software in the list, because I would always select the tools that first get their job done, and second are flexible enough for people to adapt. Free Software makes the latter much easier, but too often is the case that the former is not the case, and the value of a software that can be easily modified, but doesn’t do what I need is… none.

There is more than that of course. I have ranted before about the ethical concerns with selling routers, and I’ve actually been vocal as a supporter for law requiring businesses to have their network equipment set up by a professional — although with a matching relaxation of the requirements to be considered a professional. So while I am a strong believer in the importance of OpenWRT I do think that trying to suggest it as a solution for general final users is unkind, at least for the moment.

On the other side of the room, Home Assistant to me looks like a great project, and a kind one to it. The way they handled the recent security issues (in January — pretty much just happened as I’m writing this) is definitely part of it: warned users wherever they could, and made sure to introduce safeties to make sure that further bugs in components that they don’t even support wouldn’t introduce this very same problem again. And most importantly, they are not there to tell you how to use your gadgets, they are there to integrate with whatever is possible to.

This is, by the way, the main part of the reason why I don’t like self-hosting solutions, and why I would categorically consider software needing to be self-hosted as unkind: it puts the burden of it not being abused on the users themselves, and unless their job is literally to look after hosted services, it’s unlikely that they will be doing a good job — and that’s without discussing the fact that they’d likely be using time that they meant to be spending on something else just to keep the system running.

And speaking of proprietary, yet kind, software — I have already spoken about Abbott’s LibreLink and the fact that my diabetes team at the hospital is able to observe my glucose levels remotely, in pretty much real-time. This is obviously a proprietary solution, and not a bug-free one at that, and I’m also upset they locked it in, but it is also a kind one: the various tools that don’t seem to care about the expiration dates, that think that they can provide a good answer without knowing the full extent of the algorithm involved, and that insist it’s okay to not wait for the science… well, they don’t sound kind to me: they not just allow access to personal data, which would be okay, but they present data that might not be right for people to take clinical decisions and… yeah that’s just scary to me.

Again, that’s a personal view on this. I know that some people are happy to try open-source medical device designs on themselves, or be part of multi-year studies for those. But I don’t think it’s kind to expect others to do the same.

Unfortunately, I don’t really have a good call to action here, except to tell Free Software developers to remember to be kind as well. And to think of the implications of the software they write. Sometimes, just because we’re able to throw something out there, doesn’t mean it’s the kind thing to do so.

My Take on What I Would Replace FSFE With

So it looks like my quick, but on-the-spot renegation of FSFE last December made the round much further than most of the blog posts I ever write. I think that, in comparison, it made a much wider range than my original FSFE support post.

So I thought it would be worth spending a little more time to point out why I decided to openly stop supporting FSFE — I did provide most of this reasoning in short form on Twitter, but I thought this is better summarised in a blog post that others can reference, and that I can point people at.

So first of all, this is not all about the allegations. It was very easy to paint my post, and all the other critical outbursts against FSFE, as a position taken on hearsay. But as I already said, this was just the “flash trigger” of me calling back the support for an organization for which my feeling cooled down significantly for years. Again, I said already in the other post that I got in touch with Matthias a few years ago already about my concerns with the organization, and it was Public Money, Public Code that kept me as a supporter since then.

The reason why I decided to write renege my support when the allegations were extended, and I even changed my posts schedule for it, is that I didn’t want my (much older) post on supporting FSFE to be used as an excuse to support an organization that was in the middle of a controversy. I have been a strong supporter and have been talking people about FSFE for years, including about their more recent REUSE initiative last year, and I wouldn’t have wanted to be used as a shield from criticism.

I had an entire draft complaining about the way FSFE made me feel most like I was supporting FSFG (Free Software Foundation Germany), and that doesn’t seem to have changed that much since I wrote it two years ago. Both the news page and the activities page at the time of writing are clearly highlighting a tight focus on German issues, including talking in very broad strokes about how the German Corona Warn App doesn’t need Google – strokes so broad that make it feel like a lot of smoke and no meat underneath – and still more focus on dealing with router lock-ins (missing a lot of nuance).

I do understand that, if most of the volunteers engaging are German they will care about German issues the most, and that if the “wins” come from Germany, obviously the news will be filled with German wins. But at the same time, an organization that wants to be European should strive to have some balance and decide not to use all the news coming from a single country. Looking at the news archive page at the time I’m writing this post, there’s seven references to «Germany», one to «France», and none to «Italy», «Ireland», «United Kingdom», «Great Britain», and so on.

And it’s not that there’s nothing happening in those other countries. COVID Tracker Ireland, to stay in the topic of Covid tracing apps, is also Free Software (licensed under MIT license), and a number of other apps have been literally built based on its code. Public Money, Public Code to its best! But nothing about it on the FSFE’s website, while there’s a number of references to the German app instead.

And again speaking of Public Money, Public Code, Italy doesn’t seem to be represented at all in their list of news, with the only reference being a two years old entry about “FSFE Italy” asking for support to the project by political parties. This despite the fact that the Italian Team Digitale and the established pagoPA company have been also releasing a lot of Free Software.

Once again, if you want to change the direction of an organization, joining directly and “walking the walk” would help. But there’s a number of reasons why that might be difficult for people. While I was working for Google – a cloud provider, very clearly – it would have been fairly difficult for me to join an organization with loud complaints about “the cloud” (which I anyway disagree with). And similarly given the amount of coverage of privacy, even when not related to Free Software directly, it would be hard for me to be an activist given my current employer.

Before you suggest that this is my problem, and that I’m not the target to such an organization, I want to point out that this is exactly why I didn’t go and say that they are terrible organization and called for a boycott. I just pointed out that I no longer support them. I did say that, out of my experience, I have no reason to disbelieve their accusation, and even after reading their response statement I don’t have any reason to change my mind about that.

But I also have been a Free Software developer and advocate for a long time. I believe in the need for more Free Software, and I agree that government-developed software should be released to the public, even if it doesn’t benefit directly the taxpayers of the government that developed it. I made that case in Italian over ten years ago (I should possibly translate that blog post or at least re-tell the tale). I would enjoy being an activist for an organization that cares about Free Software, but also cares to get more people onboard rather than fewer, and would rather then not build “purity tests” into its role.

Another big problem is with the engagement method. Because of the abovementioned purity test, FSFE appears to only be engaging with its community in a “write only” media over Twitter. If you want to be an activist for FSFE you need to use email and mailing list, or maybe you can use Mastodon. In-person meetings seemed to still be all the rage when I discussed this a few years ago, and I do wonder if with 2020 happening they manage to switch at least to Jitsi, or if they ended up just using an Asterisk server connected to a number of landlines to call into.

I’m still partially comfortable with mailing lists for discussion, but it’s honestly not much of a stretch to see how this particular communication medium is not favorable to younger people. It’s not just the lack of emoji and GIF reactions — it’s also a long-form medium, where you need to think careful about all the words you use, and that persists over time. And that counts double when you have to handle discussion with an organization that appears to have more lawyers than developers.

I joked on Twitter that for a Gen-Z person, asking to use email to partecipate is the equivalent of asking a Millennial (like me) to make a phone call. And I say that knowing full well how much time I used to spend on the phone when I ran my own company: it’s not fun to me at all.

But that means you’re cutting out two big categories of people who could have both the intentions and the means to help: younger people with time on their hand, who can actively partecipate in programs and organization, and professionals who might have the expertise and the contacts.

And speaking of the professionals — you may remember that I came to the REUSE tool (which I contributed a number of fixes to myself) after complaining about having a hard time contributing while at Google because projects, among others, often didn’t provide a proper license that I could refer to, to submit patches. At the time of writing, just like a few years ago when I first tried correcting something on the website, the FSFE Website repository does not provide a license or SPDX headers (to comply with REUSE).

What would I like is for an actual European-wide organization, focused not on government policy, but rather on making it possible to have a sustainable ecosystem of Free Software development, particularly when it comes to all of those nuances that differ from a discussion of Free Software and licensing that is always too USA-centric.

The organization I have in mind, that I would love to provide monetary contribution to (if not outright be an activist for, time constraint being a thing), would be spending time in universities and high school, showing the usefulness of Free Software to learn new things. Convincing both professors and students of the usefulness of sharing and receiving, and respecting the licensing.

This is not as obvious: back when I was in high school, BSA was the only enforcement of license compliance in schools, and Free Software advocates were explicitly undermining those efforts, as they were working for a few proprietary software manufacturers. But as I said before, undermining proprietary software licenses undermines the Free Software movement. If people are trained to ignore licensing requirements, they are going to do so for Free Software, and that’s now you end up with projects ignoring the limits of GPL and other licenses.

And this connects to the next problem: for the movement to be sustainable, you also need people to make a living off it, and that requires taking licensing seriously. It’s a topic I come back over and over: business-oriented Free Software is already lacking, and people need money to survive. When the option for more experience developers, project managers, community managers, … is basically to barely make ends meet or go work for one of the big companies that don’t focus on Free Software… well the answer is obvious for a lot more people you may imagine. Not everyone gets to be the “star” that Greg KH or Linus Torvalds are, and get paid to pretty much keep their Free Software engagement — most others either do it as a side hustle, or have a side hustle.

The Document Foundation found out the hard way that there’s need for actual business plans if you want to keep maintaining big, complex Free Software projects. And even Mozilla, once shown as a core pillar of Free Software paid development, has shown this year how hard it is to keep “running the show” without a sustainable plan on the long term.

An organization focused on sustainability in Free Software should, at least in my hopes, focus on providing this kind of support. Providing blueprints for business engagements, providing outreach on license compliance to the benefit of Free Software, but also providing the pragmatic tools for Free Software enthusiast consultants to engage with their customers and take down the market barriers that make it so hard for single developers to find customers.

FSFE has lots of public policy engagements particularly with the European Union — and some of those are extremely valuable. They are required to level the playing field between Free Software developers and big corporations with entire organizations of lawyers and marketers. But they shouldn’t be the only think that an European organization focusing on Free Software should be remembered for.

Video: unpaper with Meson — pytesting our way out

Another couple of hours spent looking at porting Unpaper to Meson, this time working on porting the tests from the horrible mix of Automake, shell, and C to a more manageable Python testing framework.

I’ll write up a more complex debrief of this series of porting videos, as there’s plenty to unpack out of all of them, and some might be good feedback for the Meson folks to see if there’s any chance to make a few things easier — or at least make it easy to find the right solution.

Also, you can see a new avatar in the corner to make the videos easier to recognize 😀 — the art is from the awesome Tamtamdi, commissioned by my wife as a birthday present last year. It’s the best present ever, and it seemed a perfect fit for the streams.

And as I said at the end, the opening getting ready photo stars Pesto from Our Super Adventure, and you probably saw it already when I posted about Sarah and Stef’s awesome comics.

As a reminder, I have been trying to stream a couple of hours of Free Software every so often on my Twitch channel — and then archiving these on YouTube. If you’re interested in being notified about these happening, I’m usually announcing them with a few hours to spare (rarely more than that due to logistics) on Twitter or Facebook.

Progress Logging and Results Logging

There is one thing that my role as software mechanic seems to get me attracted to, and that’s the importance of logging information. Logging is one of those areas that tend to bring up opinions, and with the idea of making this into a wider area of observability, it brought on entire businesses (shout out to friends Honeycomb.io). But even in smaller realities, I found myself caring about logging, setting up complex routing with metalog, or hoping to get a way to access Apache logs in a structured format.

Obviously, when talking about logging in bubbles, there’s a lot more to consider than just which software you send the logs to — even smaller companies nowadays need to be careful with PII, since GDPR makes most data toxic to handle. I can definitely tell you that some of the analysis I used to do for User-Agent filtering would not pass muster for a company at the time of GDPR — in a very similar fashion as the pizzeria CRM.

But leaving aside the whole complicated legal landscape, there’s a distinction in logs that I have not seen well understood by engineers – no matter where they are coming from – and that is the difference between what I call progress logging and results logging. I say that I call them this way, because I found a number of other different categorizations of logs, but none that matches my thoughts on the matter, and I needed to give it names.

Distinctions that I did hear people talk about are more like “debug logs” versus “request logs”, or “text logs” versus “binary logs”. But this all feels like it’s mixing media and message, in too many cases — as I said in my post about Apache request logs, I would love for structured (even binary would do) request logs, which are currently “simple” text logs.

Indeed, Apache (and any other server) request logs to me fit neatly in the category of results logging. They describe what happened when an action completed: the result of the HTTP request includes some information of the request, and some information of the response. It provides a result of what happened.

If you were to oversimplify this, you could log each full request and each full response, and call that results logging: a certain request resulted in a certain response. But I would expect that there is a lot more information available on the server, which does not otherwise make it to the response, for many different reasons (e.g. it might be information that the requestor is not meant to find out, or simply doesn’t have to know, and the response is meant to be as small as possible). In the case of an HTTP request to a server that act as a reverse proxy, the requestor should not be told about which backend the request was handled by — but it would be an useful thing to log as part of the result.

When looking at the practicality of implementing results logging, servers tend to accrue the information needed for generating the result logs in data structures that are kept around throughout the request (or whatever other process) lifetime, and then extract all of the required information from them at the time of generating the log.

This does mean that if the server terminates (either because it’s killed, the power goes off, or the request caused it to crash), and the result is not provided, then you don’t get a log about the request happening at all — this is the dirty secret of Apache request logs (and many other servers): they’re called request logs but they actually log responses. There are ways around this, by writing parts of the results logs as they are identified – this helps both in terms of persistence and in terms of memory usage (if you’re storing something in memory just because you should be logging it later) but that ends up getting much closer to the concept of tracing.

Progress logs, instead, are closer to what is often called shotgun debugging or printf() debugging. They are log statement emitted as the code goes through them, and they are usually free-form for the developer writing the code. This is what you get with libraries such as Python’s logging, and can assume more or less structured form depending on a number of factors. For instance, you can have a single formatted string with maybe the source file and line, or you may have a full backtrace of where the log event happened and what the local variables in each of the function calls were. What usually make you choose between the two is cost, and signal-to-noise ratio, of course.

For example, Apache’s mod_rewrite has a comprehensive progress log that provides a lot of details of how each rewrite is executed, but if you turn that on, it’ll fill up your server’s filesystem fairly quickly, and it will also make the webserver performance go down the drain. You do want this log, if you’re debugging an issue, but you most definitely don’t want it for every request. The same works for results logs — take for instance ModSecurity: when I used to maintain my ruleset, I wouldn’t audit-log every request, but I had a special rule that, if a certain header was provided in the request, would turn on audit-logging. This allowed me to identify problems when I was debugging a new possible rule.

Unfortunately, my experience straddling open-source development and industry bubbles means I don’t have overall good hopes for an easy way to implement logging “correctly”. Both because correctly is subjective, and because I really haven’t found a good way to do this that scales all the way from a simple tool like my pdfrename to a complex Cloud-based solution. Indeed , while the former would generally be caring less about structured logs and request tracing, a Cloud software like my planned-and-never-implemented Tanuga would get a significant benefit from using OpenTelemetry to connect feed fetching and rendering.

Flexible and configurable logging libraries, such as are available for Python, Ruby, Erlang, and many more, provide a good “starting point” but by experience they don’t scale well between in and out of an organization or unit. It’s a combination of problems similar to the schema issue and the RPC issue: within an organization you can build a convention of what you expect logs to be, and you can pay the cost of updating the configuration for all sorts of tools to do the right thing, but if you’re an end user, that’s unlikely — besides, sometimes that’s untested.

So it makes sense that, up to this day, we still have a significant reliance on “simple”, unstructured text logs. They are the one universally accessible way to provide information to users. But I would argue that we should be better off to build an ecosystem of pluggable, configurable backends, where the same tool, without being recompiled or edited, can be made to output simple text on the standard error stream, or to a more structured event log. Unfortunately, judging by how the FLOSS world took the idea of standardizing services’ behaviours with systemd, I doubt that’s going to happen any time soon in the wide world… but you can probably get away with it in big enough organizations that control what they run.

Also, for a bit of fun related tidbit, verbose (maybe excessively so) progress logging is what made my reverse engineering the OneTouch Verio so easy: on Windows the standard error is not usually readable… unless you run the application through a debugger. So once I did that, I could see every single part of the code as it processed the requests and responses for the device. Now, you could think that just hiding the logs by default, without documenting the flag to turn them on would be enough — but as it turns out, as long as the logging calls are built into a binary, it’s not too hard to understand them while reverse engineering.

What this is meant to say is that, just because easy access to logs is a great feature for open source tools, and for most internal tools in companies and other institutions, the same cannot be said for proprietary software: indeed, the ability to obfuscate logs, or even generate “encrypted” logs, is something that proprietary software (and hardware) thrive on: it makes it harder to reverse engineer. So it’s no surprise if logs are a complicated landscape with requirements that are not only divergent, but at times opposite, between different stakeholders.

Moving Notes #3: Paperwork, Service Providers, Xenophobia

This is part 3 of a series of posts talking about moving apartments, after me and my wife moved, the past October.

So, in spite of most agents we worked with, and in spite of Rightmove letting them lie out of their teeth on their pages, we eventually found a place we liked, and that we could actually rent. To be honest, we were a bit scared because we ended up renting on the higher end of our budget despite finding only a two bedroom flat, but it gave us a good impression, and in particular it didn’t involve dealing with a property management agency.

Unfortunately, it still required dealing with the real estate agency to make sure the paperwork was taken care of: referencing, credit check, and finally rental agreement. Even more so because our actual landlord is once again overseas, and they instead rely on a family member to look after the apartment.

Tenancy Agreements

As I already noted previously, this agency required us to pay a deposit when making an offer — the only one that we have found doing that. This deposit would have been refundable if our offer was not accepted, or if they proven to be “unreasonable” in their demands during the rental negotiations. If we decided to withdraw the offer, then we would have lost the deposit. But if the offer was accepted, and we completed the paperwork, that deposit would add up to the normal rental hold.

As we completed the offer, and the landlord had been reasonable throughout, we had no problem with this particular hold. It also seemed to give the whole process a bit more “propriety”, although obviously it’s also a privilege to be able to just provide the hold deposit on the spot without knowing that beforehand.

But of course, it wouldn’t just go completely smooth. After the offer was accepted we had two main objectives: completing the referencing process, and signing the contract. Neither was successful on the first try.

The Referencing Process

The referencing process was something that we partially feared, due both to our previous experience with Dexters, and because of at least one of the previous agents saying that me being on probation at work would be an issue. Unsurprising, given the current situation between lockdowns and Brexit, my wife had been left without a job at the time, and we were wondering how that would fly through — as it turns out, that was not an issue, possibly because we have been married for nearly a year already when we started the process.

What turned out to be a problem there was our “old friend” institutional xenophobia. The referencing process needs to be started directly by the agency, who types in your name, and the system does not let you amend it, but rather just note additional names you may be known by. Turns out that English agents are not very good at either typing, or splitting in different components, foreign names. Not that it it is a surprise, and that’s not something I would call xenophobia in and by itself, but you can see how it becomes an extra burden, after receiving a half filled-in form, having to go back and say “Yeah, no, that’s not my name, please write such-and-such into it.”

Where I do apply the label of xenophobia is the service provider not even considering this option, and not allowing for the referenced person to correct the spelling of their name, despite requesting them to swear they are providing truthful information. And just like I said previously about the three years of addresses in the energy supply switch group, that’s not to say that it would have been a problem for me to fill it in as “Diego Pettenò” instead of “Diego Elio Pettenò” – half of everything appears to know me as the former, and that sweet tech bubble job insulates me from most of those problems – but there will be people who get in trouble for silly stuff like that.

Thankfully, of the two sets of referencing issues, only mine had further problems, and for once they had nothing to do with my nationality — just my place of work. When they asked me to provide the address of my employer, it was through a postcode lookup…

Let me explain postcode lookup for those who are not aware of this works in the UK, as it’s not common in many other countries. Postcodes in the UK designate a very limited space — a handful of buildings in most cases. In other cases, they may designate a single building, or – like is the case for my employer’s London HQ – a single address. Which means it’s not uncommon for services to ask you for your postcode first, then show you the list of addresses that correspond to that postcode. In our flat’s case, this means around eighty different addresses for the 80 flats in the building.

… and, since the postcode in my contract only has one valid address, that should have been easy. Except, when I tried submitting that, the second postcode field – not the one for the search, but the one in the actual address – complained that it was not valid. Well, turns out that the reason is that the postcode was a non-geographical postcode, because the office used to be the Royal Mail’s headquarters, and so it was issued a custom postcode to replace their original W1A 1HQ special one.

Quite a few back-and-forth with the agency later we managed to get all of the paperwork through and we got our references, so it was time to look into signing the tenancy agreements.

Tenancy Agreement Templates

So it turns out most tenancy agreements are based on some kind of standard template, which is not surprising. Even in Ireland, the Private Rental Tenancy Board (PRTB) provides a base template for the landlords to use. But here, it looks like different agencies use significantly different templates with similar, but not identical terms.

Most of these templates have a specified section for an “additional schedule”, which include additional requests from the offer – e.g. we wanted a couple of rooms emptied out, a new mattress, and authorization for a few changes – but what is not obvious to many is that additional clauses can also be waived if you agree to it. And indeed, on the original tenancy agreement when I moved to London, the consultancy company that was helping me asked a number of clauses to be explicitly voided, including all of those that wouldn’t apply anyway, such as talking about gardens.

So when I received the agreement draft, I went through with the fine comb to figure out if any of the clauses would make sense to remove. And I did find a few interesting ones. For instance, the tenancy agreement required us to be cleaning internal and external windows every month. We live on the twelfth floor of a high raiser — there is no way for us to clean the external windows and even if there was it wouldn’t make sense for us to be responsible for that. I got the clause removed, together with a number of other ones.

Most importantly, I was looking for references to the headlease. And this is yet another thing that appears to be very UK specific and not sure if I have all the context to explain. You see, in the UK most flats are not outright owned by the landlords who rent them out, but they are on a so-called “leasehold” for often hundreds of years. Add to that a complex system of shell companies in partnership between the leaseholder of the land, the development company, the management company, and the whole set of leaseholders of the flats in the building, and in addition to the tenancy agreement you’re subject to the terms of the headlease between the … nebulous set of shell companies and your landlord.

I found this out by chance, by the way. When I moved, the consultancy company didn’t say much about it. When Dexters then eventually sent me a hot water charge bill, and I fell of the sky without having heard about anything like that before, I was informed that most likely that bill was referenced in the headlease, and as a tenant it was my right to request a copy of it, to verify it. I did request the copy — Dexters sent me a scanned PDF, except that whoever scanned it put it through the machine in the wrong angle, so it was showing a “landscape” page on screen, with only two thirds of the page visible, while the right side was blank. They had to get a new copy for it — they didn’t even check before mailing it out.

So of course the tenancy agreement had a clause that said “If this tenancy agreement is subordinated to a headlease, you will find this attached” (phrasing to the best of my knowledge). Except that no headlease was attached, so I asked for it, only to be told that no headlease was involved in the agreement. That would be surprising! Buying out a leasehold into a “freehold” is expensive, and not usually done for flats in high-raisers. But so be it, I asked then, if they are absolutely certain that no headlease would be involved, to remove the clause… which they did!

But here’s a trick a former colleague (Andrew) told me about: in the UK, you can find information about leases via the HM Land Registry, for a nominal fee (of £3 when I used this). With that, you can find out who holds the chain of lease of an apartment. While this did not answer the question of “is a headlease involved?” it did make it make clear that it was not a freehold, and at that point the agency was convinced to ask the landlord whether a headlease was involved, received it, and forwarded it over. The clause in the tenancy agreement was not re-established, though — they had already sent the lease to sign, and they didn’t want to wait the turnaround of re-signing the contract.

And just to make this clear, that doesn’t mean that we want to ignore the headlease — we just wanted to make sure we wouldn’t be surprised if requests came through that we were not aware of. Your mileage is likely to vary, but I hope that this kind of information might help someone else in the future. It’s not all of my own invention — it’s applying the lessons from Getting More: I did my homework and showed it to the agency, and instead of circling around whether something was immutable or not, we made it business.

Suppliers, Addresses, Banks

Once we signed the agreement, it was time to sort out the various bills, addresses, and stuff like that. These mainly fell into three categories: utilities, banks, and stores. There’s a few other cases beside those, but those are the big ones. The ease of updating addresses between these ranged… drastically.

When it comes to utilities in particular, it’s important to know the start and end date for the services, and most of the providers that supply a flat will handle overlaps in the supply, with more or less care.

So Energy (affiliate link because I like them, and recommend them for a simple and ease to use interface) requires at least six weeks notice to the start of the new furniture, allows overlap and will roll-up the due on the old account into the new one, but it required all the interaction to happen with a person, rather than through any automated system. This was not terrible — once I gave them enough notice and the address of the new supply, they didn’t have any hurry in providing the remaining details, since they were just accounting issues. Also they preferred email over phone, which as a millennial, is always a plus.

Thames Water also supported overlapping supply, but in their case the start date had to be no more than four weeks in the future, when requesting the move. And while the process can be kicked off from their website, their website is prone to crashing. Like So Energy, Thames Water creates different “accounts” per supply – even if you only have one online account – but unlike the other supplier, they don’t carry over the amount due, and indeed require a different direct debit to be set up, which I nearly missed.

Hyperoptic, which is an awesome Internet provider, doesn’t support overlapping supply. But on the other hand they did switch the supply from one port to the other while on the phone. Yeah the irony of the Internet provider being the one requiring me to stay on the phone is not lost on me. But the whole thing took less than half an hour start to finish. Except for one issue with the MAC address.

Indeed, the MAC address of my UniFi Security Gateway appears not to work to connect to their network — I solved the problem by changing the address on the gateway. I actually have been convinced there might have been some MAC filtering going on, though Hyperoptic has repeatedly told me they have none… until the last time I rebooted the gateway and realized it’s not that it needs the MAC that they used to have over here — it just doesn’t work with the MAC of the gateway itself. I now start wondering if it might be a routing table cache somewhere.

Most banks (and yes, between the two of us that’s… a few of them, because it’s not usually worth closing old accounts) were not eventful at all. Except for two, in both direction. M&S Bank, which is operated by HSBC, was the only bank that allowed you to provide a date from when to change the address on file. I wish I had noticed that earlier, because then I would have put it on in time to get at least one starting proof of address.

The other case was the Metro Bank, and this is worth talking a moment about. All of the other banks we had to update the account on allowed us to request the change online. Except Metro Bank. Their solution involved either going into one of their “stores” (branches), or calling them on the phone. Given that I’m still avoiding going out, I opted to call them. Unfortunately when you call, they need to confirm your identity with a bunch of questions followed by an SMS OTP. The SMS didn’t arrive within their expected timeframe, and they pretty much just said goodbye and told me to show up in store.

I’m not sure if the problem was due to coverage, or if the 4G/LTE support on my phone means I can’t receive SMS while on the phone (I heard rumors of that, but it’s outside the area of my expertise). But the end result was that the messages only reached me the moment I disconnected the call. So I had a decision to make — clearly, with the current pandemic, going to a branch to fix the address wouldn’t be a good idea.

I could have ignored the address change until the pandemic situation improved, since most of the mail would be forwarded by Royal Mail at least for the next year. But then I looked around, and found that at that point NatWest had a £125-one-off offer for new customers (even those with other NatWest group accounts, like me — I still have my Northern Irish Ulster Bank account), when requesting a current account switch (which implies, closing an old account).

And yes, I managed to open a new account – a joint account while we were at it – with NatWest without having to talk with anyone on the phone. It did take a lot longer than a simple address update would have taken, and now we have two more of the annoying EMV-CAP devices, but it also meant that the new account paid itself for the time taken — and it didn’t require us to leave the flat, or even speak with anyone on the phone.

Oh Yeah, The Stores

So I said above that we had to make sure to update the addresses for stores as well (and a few other things). Well, that’s not the hardest problem of course — most of the communication we get is electronic, very few of the stores will send us mail. So instead of proactively going out of our way to update stuff, we did what everyone does: we procrastinated. With exceptions. I wrote down on a piece of paper all of the sources of mail that sent us something in the last month at our old address, and those, together with whatever would be forwarded to the new address, would get their address updated immediately.

I say immediately of course, but it still takes time. My wife’s membership of Cats Protection was updated fairly quickly on their backend, but the mail merge source they use to send the membership letters out takes a few weeks to update. So we received a few of those letters addressed to the old place. The only annoying part is the waste of purrfectly lovely return address labels that are for the wrong address now.

After those are tackled, there’s the matter of the mail arriving for the previous occupants of the flat, who did not set up Royal Mail redirection. As it turns out, we didn’t have to do much work for that — they have a friend living in the same building (just as we have in the old building), so we were just asked to drop it off on the other mailbox. But we noticed that we kept receiving, every fortnight or so, the Harrods catalogue.

So here’s another trick for those who might not be aware: if you moved into a flat, and the previous occupier keeps receiving subscriptions, offers, discount, and similar things, it’s perfectly reasonable to contact the senders and ask them to cancel those mail. Most will ask you to prove you have access to the mail (not your proof of address, but the proof of having the other person’s mail) and will then remove the address from the file. Turning this around, if there’s something you really care about and don’t want someone else to unsubscribe you from, make sure not to throw away the address label in one piece. If you don’t have a shredder at home, at least make sure to tear up the address in half.

And to finish off the post with a note of levity, in December I also received a surprising letter from Amazon, to our old address. I say surprising, because we made a number of purchases to the new flat (even the week before moving into it), and so there’s no reason for them to reach at the old address. Even the credit card I got from them just before Prime Day had its address updated on file very quickly after receiving it.

Well, it turned out to be addressed to me, but not quite to the old address. It was addressed to the address I used on AliExpress — slightly different formatting. It was addressed to the Diego Elio Pettenò who has an account to leave review of terrible, cheap products, as part of the usual brushing scam. Once again, Amazon is unable to deactivate the account created in bad faith that happen to use my name (and possibly my profile picture, I cannot tell).

Video: unpaper with Meson — From DocBook to ReStructured Text

I’m breaking the post-on-Tuesday routine to share the YouTube-uploaded copy of the stream I had yesterday on Twitch. It’s the second part of the Unpaper conversion to Meson, which is basically me spending two hours faffing around Meson and Sphinx to update how the manual page for Unpaper is generated.

I’m considering trying to keep up with having a bit of streaming every weekend just to make sure I get myself some time to work on Free Software. If you think this is interesting do let me know, as it definitely helps with motivations, to know I’m not just spending time that would otherwise be spent playing Fallout 76.

unpaper: re-designing an interface

This is going to be part of a series of post that will appear over the next few months with plans, but likely no progress, to move unpaper forward. I have picked up unpaper many years ago, and I’ve ran with it for a while, but beside a general “code health” pass over it, and back and forth on how to parse images, I’ve not managed to move the project forward significantly at all. And in the spirit of what I wrote earlier, I would like to see someone else pick up the project. It’s the reason why I create an organization on GitHub to keep the repository in.

For those who are not aware, unpaper is a project I did not start — it was originally written by Jens Gulden, who I understand worked on its algorithms as part of university. It’s a tool that processes scanned images of text pages, to make them easier to OCR them, and it’s often used as a “processing step” for document processing tools, including my own.

While the tool works… acceptably well… it does have a number of issues that always made me feel fairly uneasy. For instance, the command line flags are far from standard, and can’t be implemented with a parser library, relying instead on a lot of custom parsing, and including a very long and complicated man page.

There’s also been a few requests of moving the implementation to a shared library that could be used directly, but I don’t feel like it’s worth the hassle, because the current implementation is not really thread-safe, and that would be a significant rework to make it so.

So I have been having a bit of a thought out about it. The first problem is that re-designing the command line interface would mean breaking all of the programmatic users, so it’s not an easy decision to take Then there’s been something else that I learnt about that made me realize I think I know how to solve this, although it’s not going to be easy.

If you’ve been working exclusively on Linux and Unix-like systems, and still shy away from learning about what Microsoft is doing (which, to me, is a mistake), you might have missed PowerShell and its structured objects. To over-simplify, PowerShell piping doesn’t just pipe text from one command to another, but structured objects that are kept structured in and out.

While PowerShell is available for Linux nowadays, I do not think that tying unpaper to it is a sensible option, so I’m not even suggesting that. But I also found out that the ip command (from iproute2) has recently received a -J option, which, instead of printing the usual complex mix of parsable and barely human readable output, generates a JSON document with the same information. This makes it much easier to extract the information you need, particularly with a tool like jq available, that allows “massaging” the data on the command line easily. I have actually used this “trick” at work recently. It’s a very similar idea to RPC, but with a discrete binary.

So with this knowledge in my head, I have a fairly clear idea of what I would like to have as an interface for a future unpaper.

First of all, it should be two separate command line tools — they may both be written in C, or the first one might be written in Python or any other language. The job of this language-flexible tool is to be the new unpaper command line executable. It should accept exactly the same command line interface of the current binary, but implement none of the algorithm or transformation logic.

The other tool should be written in C, because it should just contain all the current processing code. But instead of having to do complex parsing of the command line interface, it should instead read on the standard input a JSON document providing all of the parameters for the “job”.

Similarly, there’s some change needed to the output of the programs. Some of the information, particularly debugging, that is currently printed on the stderr stream should stay exactly where it is, but all of the standard output, well, I think it makes significantly more sense to have another JSON document from the processing tool, and convert that to human-readable form in the interface.

Now, with a proper documentation of the JSON schema, it means that the software using unpaper as a processing step can just build their job document, and skip the “human” interface. It would even make it much easier to write extensions in Python, Ruby, and any other language, as it would allow exposing a job configuration generator following the language’s own style.

Someone might wonder why I’m talking about JSON in particular — there’s dozens of different structured data formats that could be used, including protocol buffers. As I said a number of months ago, the important part in a data format is its schema, so the actual format wouldn’t be much of a choice. But on the other hand, JSON is a very flexible format that has good implementations in most languages, including C (which is important, since the unpaper algorithms are implemented in C, and – short of moving to Rust – I don’t want to change language).

But there’s something even more important than the language support, which I already noted above: jq. This is an astounding work of engineering, making it so much easier to handle JSON documents, particularly inline between programs. And that is the killer reason to use JSON for me. Because that gives even more flexibility to an interface that, for the longest time, felt too rigid to me.

So if you’re interested to contribute to an open source project, with no particular timeline pressure, and you’re comfortable with writing C — please reach out, whether it is to ask questions for clarification, or with a pull request to implement this idea altogether.

And don’t forget, there’s still the Meson conversion project which I also discussed previously. For that one, some of the tasks are not even C projects! It needs someone to take the time to rewrite the man page in Sphinx, and also someone to rewrite the testing setup to be driven by Python, rather than the current mess of Automake and custom shell scripts.

NewsBlur Review

One of the very, very common refrain I hear in my circles, probably because my circles are full of ex-users of it, and at the same time of Googlers and Xooglers, is that the Internet changed when Google Reader was shut down, and that we would never be able to come back. This is something that I don’t quite buy out right — Google Reader, like most of the similar tools, was used only by a subset of the general population, while other tools, such as social networks, started being widely used right around the same time.

But in the amount of moaning about Google Reader not existing anymore, I rarely hear enough willingness to look for alternatives. Sure there was a huge noise about options back then, which I previously called the “Google Reader Exodus“, but I rarely hear of much else. I see tweets going by of people wishing that Reader still existed, but I don’t think I have seen many willing to go out of their way to do something about it.

Important aside here: while I did work at Google when Reader was shut down in effect, the plan was announced in-between me signing my contract and my start date. And obviously it was not something that was decided there and then, but rather a long-term decision taken who knows how long before. So while I was at Google for the “funeral”, I had no saying, or knowledge, of any of it.

Well, the good news is that NewsBlur, which I have started using right before the Reader shut down, is still my favourite tool for this, it’s open source, and it has a hosted service that costs a reasonable $36/year. And it doesn’t even have a referral program, so if you had any doubt of me shilling, you can vacate it now.

So first of all, NewsBlur has enough options for layout that look so much like Google Reader “of back then” — before Google+ and before losing the “Shared Stories” feature. Indeed, it supports both its own list of followers/following, and global sharing of stories on the platform. And you don’t even need to be an user to follow what I share on it, since it also automatically creates a blurblog, which you can subscribe to with whatever you want.

I have in the past used IFTTT to integrate further features, including saving stories to Pocket, and sharing stories on Twitter, Facebook, and LinkedIn. Unfortunately while NewsBlur has great integration, IFTTT is now a $4/month service, which does not have nearly enough features for me to consider subscribing to, sorry. So for now I’m talking about direct features only.

In addition to the sharing features, NewsBlur has what is for me one of the killer features: the “Intelligence Trainer”. Which is not any type of machine learning system, but rather a way for you to tell NewsBlur to hide, or highlight, certain content. This is very similar to a feature I would have wanted twelve years ago: filtering. Indeed, this allowed me to hide my own posts from Gentoo Universe – back when I was involved in the project – and to only read Matthew’s blog posts in one of the many Planets he’s syndicated, like I wanted. But there’s much more to it.

I have used this up to this day to hide repetitive posts (e.g. status updates for certain projects being aggregated together with blogs), to stop reading authors that didn’t interest me, or wrote in languages I couldn’t read. But I also used the “highlighting” feature to know when a friend posted on another Planet, or to get information about new releases or tours from metal bands I followed, through some of the dedicated websites’ feeds.

But where this becomes extremely interesting is when you combine it with another feature that nowadays I couldn’t go without, particularly as so much content that used to be available as blogs, sites, and feeds is becoming newsletters: it’s the ability to receive email newsletters and turn them into a feed. I do this for quite a few of them: the Adafruit Python for Microcontrollers newsletter (which admittedly is also available through their blog), the new tickets alerts from a bunch of different venues (admittedly not very useful this year), Tor.com, and Patreon.

And since the intelligence trainer does not need to have tags or authors to go along, but can match a substring in the title (subject), this makes it an awesome tool to filter out certain particular messages from a newsletter. For instance, while I do support a number of creators on Patreon, a few of them share all their public videos as updates — I don’t need to see those in the Patreon feed, as I get them directly at source, so I can hide those particular series from the Patreon feed for myself. And instead, while I can wait for most of the Tor.com releases, I do want to know quickly if they are giving away a free book, or if there’s a new release from John Scalzi that I missed. And again, the highlighting helps me there: it makes a green counter appear next to the “feed”, that tells me there’s something I want to look at sooner, rather than later.

As I said the intelligence trainer doesn’t have to use tags — but it can use them if they are there at all. So for instance for this very blog, if I were to post something in Italian and you wouldn’t be able to read it, you could train NewsBlur to hide posts in Italian. Or if you think my opinions are useless, you can just hide those, too.

But this is not where it ends. Beside having an awesome implementation of HTTP, which supports all bandwidth-saving optimizations I know of, NewsBlur thinks about the user a lot more than Google Reader would have. Whenever you decide to do some spring cleaning of your subscription, NewsBlur will send you by email an OPML file with all of your subscribed feed before you made the first change (for the day, I think). That way you never risk deleting a subscription without having a way to find it agian. And it supports muting sites, so you don’t need to unsubscribe not to get a high count of unread posts of, say, a frequent flyers’ blog during a pandemic.

Plus it’s extremely tweakable and customizable — you can choose to see the stories as they appear in the feed, or load into a frame the original website linked by the story, or try to extract the story content from the linked site (the “reader mode”).

Overall, I can only suggest to those who keep complaining about Google Reader’s demies, that it’s always a good time to join NewsBlur instead.