Reverse Engineering an LG Aircon Control Panel — Fit It All Together

Since we moved to this flat, one of my wishlist item was to have a way to control the HVAC (heat, ventilation, and air conditioning) without having to get out of bed, or go to a different room. A few months ago, I started on a relatively long journey of reverse engineering the protocol that the panel used so that I could build my own controller, and I’ve been documenting pretty much every step along the way, either on Twitter, this blog, or Twitch. As I type this, while I can’t say that the project is all done and dusted, I’ve at least managed to get to the point where I’m reaping the benefits of this journey.

You may remember that at the end of the previous chapter in this saga I was looking at controlling the actual HVAC with a Python command line tool. I built that on stream, to begin with, and while I didn’t manage to make it work the way I was hoping (with a curses-based UI), I did manage to get something that worked to emulate both the panel and (to a minor point) the HVAC itself.

I actually used this “in production” (that is, to control the air conditioning in my home office) for a little over a week. The way I was using it was through a Beagle Bone Black which I had laying around for a long while – I regret not signing up for the RISC-V preview, but honestly I wouldn’t have had time – connected through an USB-to-UART adapter (a CH340 based one, because they can actually do 104bps!), and a breadboarded TLIN1206 on a SOP-8-to-DIP adapter. A haphazard and janky setup if you’d ever seen one, but it was controllable by phone… with JuiceSSH and Tailscale.

While absolutely not ergonomic, this setup still allowed me to gain the experience needed with the protocol, in order to send the PCB design to manufacture. I did that about a week in, and as usual, JLCPCB‘s turnaround is phenomenally fast, and by the following Monday I had my boards.

The design is pretty much the same as the one I spoke about in part two, with two bus transceiver blocks, a 3.3V DC-DC converter, the quad-NAND to make sure the code cannot send data to the bus if the physical switch is turned to “disable”.

While the design proven it needs a little bit more work to be optimal, it’s a great starting point because it just works fine. And while I did originally plan to have this support the original panel with the switch turning on “pass-through mode”, I decided that for the moment, this is not needed, nor desired.

My original intention was to allow the physical switch to just prevent the custom controller from talking to the HVAC engine, and let the original panel take over. Unfortunately this does not work: the panel is not entirely stateful, but there is a bit in the command packet that says “Listen to me, I’m changing configuration” — and if the configuration in the panel and the HVAC don’t match when that bit is not set, an error state is introduced.

This basically means I wouldn’t be able to seamlessly switch between the old panel and my custom controller. Instead what I’m going to be doing if I need to is to first flip the switch to disable the custom logic, and then connect the panel to the secondary bus. That way it would initialize directly on the bus. If I do need to re-design this board, I’m going to make the switch more useful, and add a power disconnection so that it can be all connected without power on at all.

There was another reason why I originally planned to support the original control panel: it reports a temperature. I thought that I would be able to use that temperature as a “default” sensor, while still allowing me to change the source of the temperature at the time of configuration. But the panel’s temperature sensor is pretty much terrible, and it’s only able to measure 16°C~30°C, plus it’s easily fooled by… a lamp. So not exactly something you can call reliable for this use, and not something I would care to add to my monitoring.

To my own astonishment, my first attempt at soldering the full board was successful — two out of three of the boards work like a charm, the third one is a bit iffy, but it might be a not perfect component into it. I’ll have to see, but also I don’t want to make a new respin now that even some of the components I need are getting harder (and more expensive) to find. Take the JST ZR connection: in the picture above you an see it white rather than cream — that’s because it’s not an official JST part but an AliExpress clone that fits the same footprint, and I could actually buy.

Custom ESPHome Climate

Once I had the board, I rigged up a quick test bed on my desk with a breadboard and another CH340 adapter (I have a few around, they are fairly cheap and fairly versatile), and started off to complete the Custom Climate component.

Since I started this project, the ESPHome Climate API actually changed a couple of times, particularly as Nabu Casa is now sponsoring the project, and its development moved to a monthly release kind of deal. Somehow the Climate components were among those that most required work.

But the end result is that the API was clearer, and actually easier to implement, by the time I had this ready. So I wrote down the code to generate the six bytes packets I needed to send, and ran it against the emulator… and it seemed to work fine. I had wired this with a test component in Home Assistant and I could easily change the mode, the temperature, and everything else just fine, and I could see in my emulator that it was getting the right data in just fine.

At that point I was electrified, and I thought it would just be a matter of putting it to the wall and see it work. You can imagine my disappointed when I called in my wife to assist me in my victory… and it didn’t work. And I was ready to detach all of it to spend the next day debugging what was going on, until I realized that I forgot to send the “settings changed” flag. I have to say that the protocol turns out to be a bit more complex than I expected it to be, and I should probably write a bit more documentation about it, not just scatter it around the (now multiple) implementations.

After that, I actually went ahead and replaced all three of the control panels with my custom ones, and connected them to Home Assistant. That turned out to be a much easier prospect than anything else we have been trying up to now: we can decide which temperature reading to use to control the room, rather than being stuck to the silly temperature sensor in the panel, and we can use the two-point set temperature in the way most modern smart thermostat can (which the panel didn’t support, despite having an “auto mode” that would turn on cooling or heating “as needed”).

The first couple of days lead to a bit of adjustments being necessary — including implementing a feature that my wife requested: when not cooling or heating, the original panel would enter “fan only” mode. Which I enjoy for myself in the office, but bothers my wife. The original panel does not have an option to turn the fan off — but I could implement that in the custom controller. This allows us to keep our thermostat on the heat-cool mode most of the time, and just make sure the range is what we actually care for.

I also made the mistake at first of not counting on hysteresis. That turned out to be a bit more annoying to implement, not in the matter of code, but in the matter of the logic behind it — but it should now be working: it means that there is more friction to change the state of the air conditioning, which means the temperature is not as constant, but it should be significantly easier to run. To be honest I was impressed by how stable the temperature was when I left it to short cycle…

Home Assistant Integration

This was probably the simplest part of the whole work! Nabu Casa is doing an awesome job at keeping the two projects integrated very well, and with the help of “packages” configuration, replicating the configuration for the three separate boards took basically no time.

The only problem I had was that I couldn’t seem to be able to flash the first ESPHome firmware onto my ESP32 devkits using the WebSerial support. I have used it multiple times in the past, particularly to update my BLE Bridge, which I just need to connect to my main workstation rather than the standalone power supply, to upgrade, but for the pristine ESP32 devkits it didn’t work out quite as well.

The UI is very similar to the one that Google Home exposes for Nest thermostats where they do support air conditioning. And indeed, with the addition of the Home Assistant Cloud service, the same UI shows up for these thermostats.

And at that point it was just a matter of configuring the expected range of operation, both for the “daytime” and for the “night scene”. Which is one of the reasons why I wanted to have thermostat that we can control with Home Assistant.

You see, some time ago I set up so that we have a routine phrase (“To the bedroom”) and Flic buttons in both the living room and the bedroom, that prepare us to get to bed: turn off the TV, subwoofer, air fresheners, lights everywhere except bedroom and bathroom, set the volume of the bedroom speaker for the relaxing night sound, and so on.

Recently, thanks to a new Dyson integration, it also been setting the humidifier to raise the humidity in the bedroom (it gets way too dry!), and turn on the night mode on the other purifiers, which has been a great way not just to make it easier not to forget things, but also to save us from leaving the humidifier running 24/7: it’s easier for us to keep it running overnight at a higher humidity, than trying to keep it up during the whole day.

Now, with the climate controls in place, we can also change the temperatures before going to bed, rather than turning it off, which is the only option we had before. And this is a big deal, because particularly for the living room, we don’t want it to get too scorching hot even if we’re not there: it’s where the food cupboards, among other things, and during the heatwave we exceeded the 30°C for a couple of hours every other day. Being able to select different ranges while we’re still sleeping gives us a bit more safety, without having to keep running the air conditioning overnight.

Features And Remote Control

Similarly, our scheduled morning routine, configured to go off together with the alarm clock, can scale back the range in my home office to something suitable for my work day (it can get warm fast with computers and stuff running, and the door closed for meetings), so I don’t have to over-run the office in the morning when having my first meeting: it starts automatically, and it welcomes me with a normal, working temperature for my first meeting.

The final point is that, since we can actually set a very wide range on the thermostats, and rely on much more accurate thermometers that are not restricted to the 16°C to 30°C range, we can leave the thermostat on, with a very wide range, when we’re not actually going to be at home. This is particularly interesting now that we might be able to travel again, at least to see families and friends — we don’t want to leave the air conditioning or heating running all the time, but we also want to have some safeguards against the temperature dropping or raising out of control. This became very clear when the CGG1 in our winter garden hit the 50°C ceiling it could report while we were out playing Pokémon Go and we couldn’t turn on the air conditioning at all — thankfully the worst result of that was that the body of one of the fake candles we have in the winter garden for ambiance melted… and now it looks even more realistic as a candle (it also damaged the moving flame part, but who cares.)

Now, the insulation of this flat is not great. In particular the home office tends to drop down close to freezing temperatures in the winter, because the window does not seal even when closed. This means I can’t easily follow Alec’s suggestion on energy storage. But having a bit more control on automation does make it easier to keep the temperature in the flat more stable. In the winter, I expect we’ll make sure that we keep a minimum temperature overnight to avoid having to force a much higher differential when we wake up, for instance.

There’s a few features that I have not yet implemented, and that I definitely should look into implementing soon. To start with, as soon as summer gives away to autumn, we’re likely going to want to be using the dehumidifier more — without turning on the heat pump, particularly in the living room as we cook and eat. Since the CGG1 provides humidity readings together with temperature, it means I can set up an automation that, if nothing else is running, turns the dehumidifier on if the humidity reaches a certain set point, for instance.

There’s also two switches that I have not implemented yet, but should probably do, soon. One turns on resistive heating – and this will make sense again if you watch Alec’s video on heat pumps – while the other has to do with the Plasma Filter.

What’s a plasma filter? That’s a good question, and one that I’m not sure I have the right answer for. I know that this is something that the original control panel suggests is present in our HVAC, although I have no way to know for sure (we don’t have the manual of the actual engines). The manual for the PQRCUDS0 says that «you can use the plasma function» but also states «If the product is not compatible with the Plasma function,
it will not do the Plasma function even though the indicator is turned on.» This suggests that unlike other features like the swirl/swing, it’s not part of the feature query that the panel sends at turn on.

When googling further to look for information about LG’s plasma filter, I did find another manual, for an actual unit, rather than the control panel. Not the unit we’ve got, I think, but at least an unit. And this one has a description for the plasma functionality:

Plasma filter is a technology developed by LG to get rid of microscopic contaminants in the intake air by generating a plasma of high charge electrons. This plasma kills and destroys the contaminants completely to provide clean and hygienic air.

This is quite a bit interesting — and next to this, a video from LG refers to this as a “Plasma/Ionizer”, which pretty much suggested me that this is one of Big Clive’s favourite toys: ozone generators. Which makes sense given that one of his favourites is a Sharp Plasmacluster.

Code And Next Steps

First of all, the code and the board designs are all available on my GitHub. I originally considered making this a normal component for ESPHome, but since it relies on a very custom board, it doesn’t feel like the right thing to do. It does mean I need to manually keep in mind all the various changes in APIs, but hopefully that will not take too much of my time.

As I said previously, I have not actually implemented the panel bus handler — the panel will enter into error mode if it does not get an expected reply from the HVAC engine, so connecting it right now would not work at all, except if you were to disable the actual ESP32 control. I’m likely going to leverage that behaviour to test some more error handling in the future.

I would like to put a box around the board — right now it’s literally stuck to the wall with some adhesive feet, in all three rooms. And while the fixed red LED is not too annoying overnight, it is noticeable if you wake up in the middle of the night. My original idea was to find someone who can help me 3D print a box that fits on the same posts the panel fits, and provide a similar set of posts for the original panel. But it also involved me finding a way to flip the switch without taking it down the wall.

But since figuring out 3D printing with no experience is going to take a lot of investment, I am not going to take a look at that option until I’m absolutely certain I’m not changing anything in the design. And I know I would want to change the design a little bit.

First of all, I want to have a physical cut-off for the connection — since the power to run the ESP32 module comes from the same cable controlling the HVAC, the only way to turn off the power is to disconnect the cable, right now. Having a physical switch that just disconnects the power and data would just make it easier to, say, replace the devkit module.

Similar, as I said, the panel is not that useful to keep running all the time. So instead I would change the switch implementation to keep the panel off most of the time, and only power it on when disabling the ESP32. This would also save some components, since there would be no need to have the second bus transceiver and its passives.

I’m also wondering if it would make sense to have some physical feedback and access, in addition to the Home Assistant integration and the voice assistant controls: in particular I’m considering having an RGB LED on the board to tell the current action being taken (optionally, I wouldn’t want to have that in the bedroom, as it would be way too bright) together with a button to at least turn the HVAC “soft off”.

Finally, there’s a couple of optimizations that could be done to make the board a bit cheaper. One of the capacitor is ceramic, but could be replaced with a polymer one for ⅓ of the price, and the TVS diodes pair (which were actually a legacy part of the design, recommended by the MCP2021, but not in the reference designs for the TLIN1027) could be replaced with a single integrated TVS diode — it would just be “a bit” harder to solder by hand in TO-236 packages.

These are all minor though — the main cost behind the board is actually the ESP32-DEVKIT-32D that it’s designed around. It would be much cheaper to use only the ESP32 WROOM module, and not have the USB support components on the board. But I have had bad experiences with trying to integrate that in my designs, so I’m feeling a bit sceptical of going down that route — it also would mean that a botched board sacrifices the whole module (I did sacrifice two or three of those already) unless you get very good at desoldering them (or you have a desoldering station).

So most likely it will take me a few months before I actually get to the point of trying building a 3D printed cover for it. With a bit of luck by then we’ll be back in an office at least part of the time, and I can get someone to teach me how to use the 3D printer there.

Also as a final note — the final BOM for the boards suggest that building one of them costs around £25 or so, without the case. As I said there’s a few cost saving measures that I could take for the next round — though it’s questionable, because it would require me to get more components I don’t currently have. Of course the actual cost of building three of them turned out to be… significantly higher.

I think this is the part that sometimes it’s hard to explain to people who have not had this type of experience: the BOM costs are only one of the problems you need to solve — you can really screw up a project by choosing the wrong components and bringing the BOM price too high… but a low BOM cost does not make for a cheap project to finish, particularly when you’re developing it from nowhere.

In this case, if I wanted to tally up the cost of building these custom thermostat panels, I would have to, at the very least, count the multiple orders from DigiKey from which I ordered the wrong components (like the two failed attempt at using Microchip’s LIN bus transceivers rather than the TLIN, or the discrete UARTs with their own set of passives). But then there’s the cost of having all of the various tools that were needed to get this all done. Thankfully most of those tools have been used (and sometimes abused) for different projects, but they are a good metaphor for the cost of R&D that many products need — so it makes sense that what you end up buying costs more than the “simple” expense of the BOM. So keep that in mind next time you see an open source hardware device costing more than what you expect it to.

Reverse Engineering an LG Aircon Control Panel — Introduction

I like reverse engineering stuff. It’s not just the fact that it’s a nice puzzle to solve, but I enjoy the thrill of “Oh, that’s how that works.” I’m sure I’m not alone, as can be clearly seen following marcan’s Asahi Linux work, or following Foone on Twitter, or Big Clive on YouTube (and many, many others).

Sometimes, a lot more rarely, my reverse engineering is actually geared towards something I want to make use of, rather than just for the sake of finding answers — this is one of those cases. If you have been following me on Twitter or decided to watch me work on this live on Twitch, you probably already know what I’m talking about. If not, be warned that this is going to be the first part of a (possibly long) series of posts on the same topic. It turned out to be very long for a single post, and I decided to split it instead.

You see, when we moved from the last apartment, we sold our Nest smart thermostat to a friend. The new apartment has an aircon system with heat pump, rather than a “classic” heating system, which is really important as the balcony can easily reach 40°C in the mornings when the sun shines. And unlike in the US, where thermostats are pretty much standardized, Europe’s landscape of thermostats is different enough that Nest gave up, and does not support aircon systems.

Aside: I do have a bit of a rant about Nest Thermostats in Europe, but some of that might be a bit tricky to phrase for me without risking breaching confidentiality with my previous employer, which I don’t want to do. So I will leave a question here for European Nest Thermostats users: can you finally enable hot water boost with the Google Home app?

To be honest, this also kind of makes sense: in a flat that is cooled and heated with an HVAC, it makes sense to have multiple thermostats so that each room can set a different required temperature. If we’re spending the evening in the living room, what’s the point of heating up the bedroom? If I’m on vacation and not spending time in the office, why would I turn on the air conditioning? And so on.

Unfortunately what we ended up with is three thermostat units from LG, model number LG-PQRCUDS0 (provided for ease of searching and finding this blog post), which are definitely not smart, and also not convenient. These are wired, non-smart control panels, that do support features like timing, but do not provide any way to control without tapping on the screen. As far as I know, these are configured to read a temperature sensor that is not on the panel itself, but on the other hand, the placement of those sensors are a bit on the unfortunate side: in particular in the bedroom it appears located in a position that is too natural to fit a wardrobe in, making it register always a higher temperature that the room actually has.

This had been particularly annoying during the winter but it was proving to be worse during the summer: as I said the temperature in the balcony can reach 40°C in the morning, as we’re facing east and it’s a all-glass external wall. That means that the temperature inside the apartment can easily reach 30°C quite suddenly. This is not good for electronics already, but it’s doubly non-good for things like food and medicine, including insulin, which I very much depend on.

While we could just try leveraging the timer mode to turn on the AC in the morning, the complication of where the sensor is makes it very hard to judge the temperature to set it at. And since, as Alec points out on the video, the thermostat’s job is only to turn something on or off (in theory, at least)… well, there has to be an easier way.

So I embarked in this quest of reverse engineering my aircon control panel, with the intent of introducing an ESPHome-compatible add-in that would allow me to control the HVAC through Home Assistant.

Inspection

The first thing to do when setting off to reverse engineer something is to figure out what it is, whether there is any documentation for it, and whether someone else already reverse engineered it. The model number, as I said, is LG-PQRCUDS0 and LG has user and installation manuals online describing it a Delux Wired Remote Controller (together with the -B and -S variants of the part number).

Reverse image search for the panel actually seemed to struck gold at first, as this Instructables post showed exactly the same UI as mine, and included a lot of information about the protocol. But also the comments pointed to a couple of different models that seemed all similar but a bit different. So instead of going ahead and trying to build the already reversed protocol I wanted to confirm how it all worked myself.

A close up of the door behind my LG aircon control panel showing a JST ZR connector, and a yellow-red-black cable going to the wall.

The first question is going to be what the electrical “protocol” it’s using. The back of the panel has a door, that hides the inbound connection from “the wall” (that is, the actual HVAC unit), which is three wires and terminates in a JST ZR connector.

With my multimeter I could confirm that the voltage would be around 12V — but I couldn’t confirm whether it would be differential data or what else, since I’m still using an older multimeter and it doesn’t have any option to indicate there’s a signal on a wire. If someone has a good suggestion for a multimeter that does that, please leave a comment ~~below the video~~ in this post as I’d love to get a good one.

Now this is a good news, overall. The fact that the plug, and the cable itself, can be bought off the shelf means I don’t have have to take risky approaches, which is great, given that we’re renting, so any reverse engineering and replacement implementation needed to be non-destructive and reversible.

So I took out my Logic Pro, a very long USB 3.0 cable, and I ordered just enough components from Digikey to debug this thing. And a bench power supply — because I didn’t have a bench power supply, and given this thing needed 12V, it sounded something handy to have for this. The end result is the following:

With this connected, I used the Logic 2 software to check the voltage levels, and figure out that the yellow wire is data, while the red wire (in the middle) is 12V supply. The data turned out to, indeed, be a 104 Bd serial connection, which would make it share a lot of the information from the previous reverse engineering…

Except that something was off: what I could see on the wire was a burst of 12 bytes in a single stream, exactly once a second, which I assumed at that point to be unidirectional from the panel to the HVAC. But when trying to verify the checksum it didn’t match what the instructions on the other project suggested: sum everything, modulo 256, and xor with 0x55 (the confusing ‘U’ in the various descriptions is actually a bit pattern). So while I could figure out that the first byte seemed to include the mode of operation, and the third one appeared to include the fan speed, I couldn’t figure out for the life of me the checksum, so I thought I wouldn’t be able to send commands to the HVAC to do what I wanted.

On the other hand, in the worst case scenario I could have just replayed the commands I could record from the panel, so I decided to try my luck at drawing and ordering a PCB that would have just enough components for me to play around with.

Drawing the PCB

I’m far from being even a passable expert on electronics, but I could at least figure out some of the things I wanted from a “smart controller” unit to attach to this aircon. So I started with a list of requirements:

Since I wanted it to use ESPHome, it should be designed around an ESP32 module. I already attempted this once with the acrylic lamps, and I have yet to get a working board out of that. On the other hand, this time I’m much less space constrained, so I decided to go for a full DEVKIT module, one of those with already the full board of regulators, USB port and serial adapters. This turned out to be a further blessing in disguise, since the current chip shortage appears to have affected the CP2104 module I used in my previous design and I wouldn’t have been able to replicate it.
While I don’t expect that the HVAC power supply has been limited in power significantly (after all there’s even more deluxe WiFi enable controllers in other versions), I didn’t want to increase the load on the 12V supply significantly. Which meant I went for the more complex, but also more efficient, route of building in a buck converter to 3.3V to power up the ESP32.
Also, I really know that relying on my code for “enjoyment-critical” use cases can be frustrating, I wanted a physical way to hard-disconnect a possibly misbehaving automation, and go back to use the old controller, without having to fidget with cables.

With these conditions, and the assumption that the twelve bytes I was seeing were being sent directly from the controller to the HVAC, I drew and manufactured the above board. Feel free to spot the error at the top of the board, if you may.

Now, since JLCPCB turnaround is usually fairly fast, I went ahead and got that manufactured while I was still fighting with figuring out the checksum. So when the boards arrived and I populated them, I was planning on just keep changing settings to find more possible combinations of bytes to see how the checksum would behave.

And that’s when I found out I was very wrong in my assumption, and it’s possible that either the reverse engineering notes I’ve seen for other are missing a big chunk of information, or LG has so many different ways to achieve roughly the same endgame. One I powered up the panel from the bench supply, then I could see that the panel was rather only sending six bytes, rather than the twelve I expected. It’s a bidirectional communication on a single wire, a bus.

That meant going back to the literal drawing board, find the right components to implement this, and start what turned out to be a much large sidequest of complicating matters.

Home Automation: Physical Contact

In the previous post on the matter, I described the setup of lighting solutions that we use in the current apartment, as well as the previous apartment, and my mother’s house. Today I want to get into a little bit more detail of how we manage to use all of these lights without relying solely on our phones, or on the voice controlled assistants.

First of all, yes, we do have both Google Assistant and Alexa at home. I only keep Assistant in the office because it’s easiest to disable the mic on it with the physical switch, but otherwise we like the convenience of asking Alexa for the weather, or let Google read Sarah Millican for us. To make things easier to integrate, we also signed up for Nabu Casa to integrate our Home Assistant automations with them.

While this works fairly decently for most default cases, sometimes you don’t want to talk, for instance because your partner is asleep or dozing off, and you still want to control the lights (or anything else) without talking. The phone (with the Home Assistant app) is a good option, but it is often inconvenient, particularly if you’re going around the flat with pocketless clothes.

As it turns out, one of the good things that smart lights and in general IoT home automation bring to the table, is the ability to add buttons, which usually do not need to be wired into anything, and that can be placed just about anywhere. These buttons also generally support more than one action connected to them (such as tap, double-tap, and hold), which should allow providing multiple controls at a single position more easily. But there are many options for buttons, and they are not generally compatible with each other, and I got myself confused for a long while.

So to save the day, Luke suggested me some time ago to look into Flic smart buttons, which were actually quite the godsend for us, particularly before we had a Home Assistant set up at all. The way these work is that they are Bluetooth LE devices, that can pair with a proprietary Bluetooth LR “hub” (or with your phone). The hub can either connect with a bunch of network services, or work with a variety of local network devices, as well as send arbitrary HTTP requests if you configure it to.

While Flics were our first foray into adding physical control to our home automation, I’m not entirely sure if I would recommend them now. While they are quite flexible at a first glance, they are less than stellar in a more complex environment. For instance, while the Flic Hub can talk directly to LIFX lights on local network (awesome, no Internet dependency), it doesn’t have as much control on the results of that: when we used the four LIFX spots in the previous flat’s office, local control was unusable, as nearly every other click would be missing one spot, making it nearly impossible to synchronise. Thankfully, LIFX is also available as a “Cloud” integration, that could handle the four lights just fine.

The Flic Hub can talk to a Hue Bridge as well, to toggle lights and select scenes, but this is still not as well integrated as an original Hue Smart Button: the latter can be configured to cycle between scenes on multiple taps, while I could only find ways to turn the light on or off, or to select one scene per action with the default Flic interface.

We also wanted to use Flic buttons to control some of the Home Assistant interactions. While buttons on the app are comfortable, and usually Google Assistant can understand when we say “to the bedroom”, there are other times when we could rather use a faster response than the full Google Assistant round-trip. Unfortunately this is an area where Flic leaves a lot to be desired.

First of all, the Flic Hub does not support IPv6 (surprise), which meant I can’t just point it at my Home Assistant hostname, I need to use the internal IPv4 address. Because of that, it also cannot validate the TLS certificate either. Second, Flic does not have an Home Assistant native integration: for both Hue and LIFX, you can configure the Hub against a Cloud account or the local bridge, then configure actions to act on the devices connected to them, but for Home Assistant there is nothing, so the options are limited to setting up manual HTTP requests.

This is where things get fairly annoying. You can either issue a bearer token to login to Home Assistant, in which case you can configure the Flic to execute a script directly, or you can use the webhook trigger to send the action to Home Assistant and handle it there. The former appears to be slightly more reliable in my experience, although I have not figured out if it’s the webhook request not being sent by the hook, or the fact that HA is taking time to execute the automations attached to it; I should spend more time debugging that, but I have not had the time. Using the Bearer Tokens is cumbersome, though. Part of the problem is that the token itself is an extremely long HTTP header, and while you can add custom headers to requests in the Flic app, the length of this header means you can’t copy it, or even remove it. If you need to replace the token you need to forget the integration with the HTTP request and create a new one altogether.

Now on the bright side, Flic has recently announced (by email, but not on their blog) that they launched a Software Development Kit that allows writing custom integrations with the Flic Hub. I have not looked deeply into it, because I have found other solutions that work for me to augment my current redundant Flics, but I would hope that it means we will have a better integration with Home Assistant one day in the future.

To explain what the better alternatives we’re using are, we need to point out the obvious one first: the native Hue smart buttons. As I said in the previous post, I did get them for my mother, so that the lights on the staircase turn on and off the same way as they did before I fixed the wiring. We considered getting them here, but it turns out that those buttons are not cheap. Indeed, the main difference between different buttons we have considered (or tried, or at are using) is to be found in the price. At the time of writing, the Hue buttons go for around £30, Flics for £20, Aqara (Xiaomi) buttons for around £8 and Ikea ones for £6 (allegedly).

So why not using cheaper options? Well, the problem is that all of these (on paper) require different bridges. Hue, Aqara and Ikea are all ZigBee based, but they don’t interoperate. They also have different specs and availability. The Aqara buttons can be easily ordered from AliExpress and they are significantly cheaper than the equivalent from Hue, but they are also bigger, and of a shape just strange enough to make it awkward to place next to the wallplates with the original switches of the apartment, unlike both Flic and Hue. The Ikea ones are the cheapest, but unless you have a chance to pop in their store, it seems like they won’t be shipping in much of a hurry. ~~As I write this blog post, it’s been nearly three weeks that I ordered them and they still have not shipped, with the original estimate for delivery of just over a month~~ — updated before even posting this: Ikea (UK) cancelled my order and the buttons are no longer available online, which meant I also didn’t get the new lights that I was waiting for. Will update if any new model becomes available. In the meantime I checked the instructions and it looks like these buttons only support a single tap action.

This is where things get more interesting thanks to Home Assistant. Electrolama sells a ZigBee stick that is compatible with Home Assistant and that can easily integrate with pretty much any ZigBee device, including the Philips Hue lights and the Aqara buttons. And even the Aqara supports tap, double-tap, and hold in the same fashion as Flic, but with a lot less delay and no lost event (again, in my experience). It turned out that at the end of the day for us the answer is to use cheaper buttons from AliExpress and configure those, rather than dealing with Flic, though at the moment we have not removed the Flics around the apartment at all, and we rather have decided to use them for slightly different purposes, for automation that can take a little bit more time to operate.

Indeed, the latency is the biggest problem of using Flic with Home Assistant right now: even when event is not going lost, it can sometimes take a few seconds before the event is fully processed, and in that time, you probably would have gotten annoyed enough that you would have asked a voice assistant, which sometimes causes the tap to be registered after the voice request, turning the light back off. Whereas, the Aqara button is pretty much instantaneous. I’m not entirely sure what’s going on there, it feels like the bridge is “asleep” and can’t send the request fast enough for Home Assistant to register it.

It is very likely that we would be replacing at least a couple of the Flics we already have set up with the Aqara buttons, when they arrive. They support the same tap/double-tap/hold patterns as the Flic, but are significantly lower latency. Although they are bigger, and they do seem to have very cheap brittle plastic, I nearly made it impossible to change the battery on my first one, because trying to open the compartment with a 20p coin completely flattened the back!

Once you have working triggers with ZigBee buttons, by the way, connecting more controls become definitely easier. I really would consider making a “ZigBee streamerdeck” to select the right inputs on the TV, to be honest. Right now we have one Flic to double-tap to turn on the Portal (useful in case one of our mothers is calling), and another one to select PS4 (tap), Switch (double-tap), or Kodi (hold).

Wiring automation, and selection of specific scenes, is the easiest thing you can do in Home Assistant, so you get a lot of power for a little investment in time, from my point of view. I’m really happy to have finally set it up just the way I want it. Although it’s now time to consider updating the setup to no longer assume that either of us is always at home at any time. You know, with events happening again, and the lockdown end in sight.

Light Nerdery: Evolving Solutions At Homes

Of all topics, I find myself surprised that I can be considered a bit of a lights nerd. Sure, not as much as Technology Connections, but then again, who is nerdier than him on this? The lightbulb moment (see what I’m doing?) was while I was preparing to write the post that is now going to be in the future, and realized that a lot of what I was about to write needed some references about my experiments with early LED lighting, and what do you know? I wrote about it in 2008! And after all, it was a post about new mirror lights that nearly ended up the last post ever of mine, being posted just a few hours before my trip to the ICU for the pancreatitis.

Basically this is going to be a “context post” or “setup post”: it likely won’t have a call to action, but it will go into explaining the details of why I made certain tradeoffs, and maybe, if you’re interested in making similar changes to your home, it can give you a bit of an idea as well.

To set the scene up, let me describe the three locations that I’ll be describing light setups in. The first is my mother’s house in Venice mainland, which is an ’80s built semi-detached on two floors. The second is the flat in London, when my now wife moved in. And the third one is the current flat we moved into together. You may notice a hole in this setup: Dublin. Despite having lived there for longer than in London, I don’t have anything to talk about when it comes to light setup, or even in general about home; looking back, it sounds like I have actually spent my time in Dublin taking it nearly as a temporary hotel room, and spent very little time in “making it mine”.

A House Of Horrible Wiring

In that 2008 post I linked above, I complained how the first LED lights I bought and set up in my bedroom would keep glowing, albeit much dimmed, when turned off. Thanks to the post and discussions had with a number of people with more clue than me at the time, I did eventually find the reason: like in many other places throughout the house, deviators were used to allow turning the light on and off in different places. In the particular case of my bedroom, a switch by the door was paired with another on a cord, to be left by the bed. But instead of interrupting the live of the mains, they were interrupting the neutral which meant that, turning the light “off” still allowed enough current to go through between live and ground that the LEDs would stay on.

This turned out to be a much, much bigger deal than just a simple matter of LED lights staying on. Interrupting the neutral is not up to regulation: you may still have phase and get shocked if touching the inside of the lampholder, even with the switch turned off, but most importantly, it looks like it causes a number of CFLs electronics to be “stressed”, with their lifespan being seriously shortened by that. This was actually something that we have noticed and complained for many years, but never realised it was connected.

Eventually, I fixed the wiring in my bedroom (and removed the deviator I didn’t quite need), but I also found another wiring disaster. The stairwell at my mother’s had two light points, one on the landing, and one on the top of the stairs, and they were controlled together by switches at the top and bottom of the staircase. As expected, these interrupted the neutral, but most importantly, each position was wired into the live of the floor it was closest to, ignoring the separation of the circuits and passing through a phase even when turning the circuit of. And that explained why no bulb ever lasted more than a year for those.

Unfortunately, fixing the deviators there turned out to be pretty much impossible, due to the way the wiring conducts were set down inside the walls. So instead, I had to make do with separating the two lights, which was not great, but was acceptable while I was still living with my mother: I would turn on the light upstairs for her (I was usually upstairs working) and she would not need the light downstairs. But I had to come up with a solution after I prepared to leave.

The first solution to this was adding a PIR (Passive Infra Red) movement sensor at the bottom of the stairs to turn on the light on the landing. That meant that just taking the first step of the stairs would illuminate enough of the staircase that you could make your way up, and then the timer would turn it off. This worked well enough for a while, to the point that even our cat learned to use it (you could observe her taking a step, wait for the light, then run all the way upstairs).

Then, when I was visiting a couple of years back, I noticed that something wasn’t quite working right with the sensor, so I instead ordered a Philips Hue lightbulb, one of those with BLE in addition to ZigBee, so that it wouldn’t require a bridge. At that point my mother could turn the light on and off with her phone, so that the sensor wasn’t needed anymore (and I removed it from the circuit).

This worked much better for her, but she complained earlier this year that she kept forgetting to turn off the light before she would get to bed, and then she’d have to go back to the staircase as her phone couldn’t reach the light otherwise. What would be a minor inconvenience for me and many of the people reading this, for my mother was a major annoyance, as she starts getting old, so I solved it by getting her a Hue Bridge and a couple of Smart Buttons: the bridge alone meant that her phone didn’t need to reach the light (the bridge would be reachable over WiFi), but the buttons restored a semblance of normality about turning the light on and off as she used to before I re-wired the lights.

This is something that Alec from Technology Connections pointed out on Twitter some time ago: smart lights are, at the very least, a way to address bad placement of lights and switches. Indeed, over ten years later, my mother now has normal-acting staircase lights that do not burn out every six months. Thank you, Internet of Things!

While we won’t be visiting my mother any time soon, due to the current pandemic and the fact she has not been called in for the vaccine yet, once we do I’ll also be replacing the lightbulbs in my bedroom. My office there is now mostly storage, sadly but we have been staying in my old bedroom that still has the same lights I wrote about in 2008. A testament to them lasting much longer now that the wiring is right, given that for the most part I don’t remember spending more than a few months without needing to replace a lightbulb somewhere, when I was living there.

The 2008 lights I chose to keep the light to a minimum before going to bed, which I still think is a good idea, but they do make it hard to clean and sort things out. Back then I was toying with the idea of building a controller to turn on only part of the lights, but nowadays the available answer is to just use smart lights and configure them with separate scenes for bedtime and cleantime. And buttons and apps mean that there is no need anymore for having a separate bedside lamp, for the most part.

Sharing A Flat Planned For One

When I moved to London, i went looking for an apartment that, as I suggested to the relocation aide, would be “soulless” — I had heard all of the jokes about real estate agents defining flats falling apart as having “characters” or big design issues considered “the soul of the building”, so I wanted to make clear I was looking mostly for a modern flat, and something that I would be able to just tweak to my own liking.

I also was looking for an apartment that was meant to be mine, with no plan on sharing. Then I met my (now) wife, and that plan needed some adjustments. One of the things that became obvious early on was that the bedroom lights were not really handy. Like pretty much all of the apartments I lived in outside of Italy, that one had GU-10 spotlight holders scattered throughout the ceiling, and a single switch for them by the door. This worked okay for me alone, as I would just turn the light off and then lay on the bed, but it becomes an awkward dance of elbows when you share a fairly cozy mattress.

So early on I decided to get an IKEA Not floorlamp, and a smart light. I settled for the LIFX Mini, because it supported colours (and I thought it would be cool), didn’t need a bridge, and also worked without Internet connection over the local network. The only fault in my plan was to have gotten a Not with two arms at first, which meant we have a few times turned the wrong light off, until I Sugru’d away the main switch on the cable.

I said “at first” there, because we then realised that this type of light is great not only in the bedroom but also in the living room. When watching TV keeping the light on would be annoying (the spots were very bright), but turning it entirely off would be tiring for the eyes. So we got a second Not lamp, and a second LIFX bulb, and reshuffled them a bit so that the two-arms one moved to the living room, as there the additional spotlight is sometimes useful).

This worked very nicely for the most part, to the point that I considered whether my home office could use some of that. This was back in the beforetimes, where the office (i.e. the second bedroom) would mostly be used to play games in the evening, by my wife when not using the laptop, or in the rare times I would be working from home (Google wouldn’t approve). That meant that in many cases having the full light would also be annoying, and in other cases having very dim light would also not work well. In particular, for a while, I was keeping company to my wife while she played by reading, and I would have wanted light on the side of my seat, but not as much on her side.

Since the office only had four spots, we decided to buy four LIFX GU-10 lights. I’m not linking to those because the price seems to have gone off the charts and they now cost more than twice what we paid for them! These are full-fledged LIFX bulbs, with WiFi and the whole colour range. Lovely devices, but also fairly bulky and I’ll get back to that later on.

Overall, this type of selective smart lighting helped significantly with the enjoyment of the flat, by reducing the small annoyances that would be presenting on a regular basis, like navigating the bedroom in the dark, or having the light too bright to watch TV. So we were looking towards replicating that after moving.

No Plan Survives Contact With The New Flat

When we moved to the new flat we knew that a number of things would have had to be changed. For instance, the new office had six rather than four spots, and the rooms layout meant that some of the choice of where to put the lamps would be not as good as we had previously.

Things got even a little more complicated than that: the LIFX GU-10 bulbs are significantly bigger than your average GU-10 spots, so they didn’t actually fit at all in the receptacle that this flat used! That meant that we couldn’t get them in, even if we decided to only keep four out of the six.

It’s in this situation that we decided to bite the bullet and order a Philips Hue Bridge for our flat, together with six White Ambiance spots: these spots are the same size as normal GU-10 spots, and do not have issues with fitting in the receptacle, so they could be used in the new office. While there is a model supporting colour changes as well as white temperature balance, in the office we never really used colour lighting enough to justify the difference in price (though we did and still do rely on the colour temperature selection).

Once you have a bridge, adding more lights becomes cheaper than buying a standalone light, too. So we also added a “reading light” to the bedroom, which is mostly for me to use if I can’t sleep and I don’t want to wake up my wife.

Another thing we would have liked to do was to replace the clicky switch for the ensuite with a soft/smart button. The reason for that was twofold: the first is that it took us months to get used to the placement of the switch in this flat, the second is that the switch is so damn noisy when clicking that it kept waking the other up when one of us went to use the toilet overnight. Unfortunately changing the switch appears not trivial: with our landlord permission we checked the switch connection, and it is not wired with the correct L/N positions or colours, and I could see a phase on three out of the four positions.

Instead, when we could confirm that the switch did not control the extraction fan, we decided to put on three Hue spots in the bathroom, this time not the temperature controlled once, but just the dimmer ones. And at that point, we could keep a single one at 1% overnight, and not need to turn anything on or off when using the restroom during the night: it gets very dim, so you don’t wake up, but you can still see plenty to use the toilet and wash your hands. This by the way was an idea that came to me after watching some of BigClive videos: the very low level light makes for a good light to have overnight to avoid waking youself up.

To explain just how useful this setup ended up being for us, the ensuite has three spots: one is pretty much in the shower box, the other two are by the door and in the middle. Overnight, we only leave the shower spot running at 1%, with the other two being off. If we need more light, for instance to brush our teeth, or for me to get my insulin, we have an “on” scene in which all three spots are at around 30%. When we’re taking a shower, we only turn on the shower spot to 100%. And when we’re cleaning the bathroom, we set all three to 100%. At maximum light, the new bulbs are brighter than the old ones were; in the “on” scene we use, they are much less bright, because we don’t really need that much light on all the time.

We also have ordered more spots, this time from IKEA, that sells an even cheaper model, although with not as good low-light performance. We could do this because I’ve recently replaced the Hue Bridge for a ZigBee stick on our Home Assistant, and I’ll go into more details about that in a future post. At the time I’m writing this the spots have not arrived yet, but we decided that, now that we’re more likely going to be going out again (we both got our first dose of vaccine, and soon to receive the second), it makes sense to have a bit more control of the light on the entrance.

In particular, the way the entrance is currently set up, we turn on six spots all the time in the T-shaped hallway. When coming back during the day, only one spot would be necessary, to take off shoes and put down keys, the rest of the flat is very bright and does not need illumination. And similarly when just going back and forth during the evening, only the two or three spots illuminating the top of the T would be needed, while the ones at the door are just waste. Again smart lights in this case are helpful to shape inflexible wiring.

Conclusion

I wrote before that I get annoyed at those who think IoT is a waste of time. You can find the IoT naming inane, you can find the whole idea of “smart” lights ridiculous, but however you label it, the ability to spread the control of lightbulbs further than the origin wiring intended is a quality of life improvement.

Indeed, in the case of my mother’s house, it’s likely that the way we’ll solve the remaining problems with wiring will be by replacing most switches with smart lights and floorlamps.

And while I have personally some questions about keeping the “off” lightbulbs connected to something, a quick back of the envelope calculation I did some months ago shows that even just the optimisation of being able to automate turning on and off lights based on presence, or the ability to run the light most of the time at lower power, can easily reduce the yearly energy usage considerably (although not quite to the level that buying all new bulbs would save you money, if you already have LED lights).

So once again, and probably not for the last time, I want to tip off my hat to Home Assistant, Electrolama, and all the other FLOSS projects that work with the wishes of people, rather than against them, to empower them to have smart home technologies they can control.

Software Defined Remote Control

A number of months ago I spoke about trying to control a number of TV features in Python. While I did manage to get some of the adapter boards that I thought I would use printed, I hadn’t had the time to go and work on the software to control this before we started looking for a new place, which meant I shelved the project until we could get to the new place, and once we got there it was a matter of getting settled down, and then, … you got the idea.

As it turns out, I had one week free at the end of November — my employer decided to give three extra days on the (US) Thanksgiving week, and since my birthday was at the end of the week, I decided to take the remaining two days off myself to make it a nice nine days contiguous off. Perfect timeframe to go and hack on some projects such as that.

Also, one thing changed significantly since the time I started thinking about this: I started using Home Assistant. And while it started mostly as a way for me to keep an eye on the temperature of the winter garden, I found that with a bit of configuration, and a pull request, changing the input on my receiver with it was actually easier than using the remote control and trying to remember which input was mapped to what.

That gave me finally the idea of how to implement my TV input switch tree: expose it as one or more media players in Home Assistant!

Bad (Hardware) Choices

Unfortunately, as soon as I went to start implementing the switching code, I found out that I had made a big mistake in my assumptions: the Adafruit FT232H breakout board does not support PWM outputs, including the general time-based pulsing (without a carrier frequency). Indeed, while the Blinka library can technically support some of the features, it seems like none of the Linux-running platforms would be able to manage that. So there goes my option of just using a computer to drive the “fake remote” outputs directly. Well, at least without rewriting it in some other language and find a different way to send that kind of signals.

I looked around for a few more options, but all of it ended up being some compromise: MicroPython doesn’t have a very usable PulseOut library as far as I could tell; Arduino-based libraries don’t seem to allow two outputs to happen at roughly the same time; and as I’m sure I already noted in passing, CircuitPython lacks a good “secondary channel” to be instructed from a computer (the serial interface is shared with the REPL control, and the HID is gadget-to-host only).

After poking around a few options and very briefly considering writing my own C version on an ATmega, I decided to just go for the path of least resistance, and go back to CircuitPython, and try to work with the serial interface and its “standard input” to the software.

The problem with doing that is that the Ctrl-C command is intended to interrupt the command, and that means you cannot send the byte 0x03 un-escaped. At the end I thought about it, and decided that CircuitPython is powerful enough that just sending the commands in ASCII wouldn’t be an issue. So I decided to write a simplistic Flask app that would take a request over HTTP and send the command via the serial port. It worked, sort of. Sometimes while debugging I would end up locking the device (a Trinket M0) in the REPL, and that meant the commands wouldn’t be sent.

The solution I came up with was to reset the board every time I started the app, by sending Ctrl-C and Ctrl-D (0x03, 0x04) to force the board to reset. It worked much better.

Not-Quite-Remote Controlled HDMI Switch

After that worked, the problem was ensuring that the commands sent actually worked. The first component I needed to send the commands to was the HDMI switch. It’s a no-brand AliExpress-special HDMI switch. It has one very nice feature for what I need to do right now. It obviously has an infrared remote control – one of those thin, plasticky domes one – but it particularly has the receiver for it on a cord, which is connected with a pretty much standard 3.5mm “audio jack”.

This is not uncommon. Randomly searching Amazon or AliExpress for “HDMI switch remote” can find you a number of different, newer switches that use the same remote receiver, or something very similar to it. I’m not sure if the receivers are compatible between each other, but the whole idea is the same: by using a separate receiver, you can stick the HDMI switch behind a TV, for instance, and just make the receiver poke from below. And most receivers appear to be just a dome-encased TSOP17xx receiver, which is a 3-pin IC, which works great for a TRS.

When trying this out, I found that what I could do would be to use a Y-cable to allow both the original receiver and my board to send signals to the switch — at which point, I can send in my own pulses, without even bothering with the carrier frequency (refer to the previous post for details on this, it’s long). The way the signal is sent, the pulses need to ground the “signal” line (that is usually at 5V); to avoid messing up the different supplies, I paired it on an opto-coupler, since they are shockingly cheap when buying them in bulk.

But now that I tried setting this up with an input selection, I found myself not able to get the switch to see my signal. This turned out to require an annoying physical debugging session with the Saleae and my TRRS-to-Saleae adapter (that I have still not released, sorry folks!), which showed I was a bit off on the timing of the NEC protocol the switch used for the remote control. This is now fixed in the ~~pysirc~~ pysdrc library that generates the pulses.

Once I got the input selector working for the switch with the Flask app, I turned to Home Assistant and added a custom component that exposes the switch as a “media_player” platform. In a constant state of “Idle” (since it doesn’t have a concept of on or off), it allowed me and my wife to change the input while seeing the names of the devices, without hunting for the tiny remote, and without having to dance around to be seen by the receiver. It was already a huge improvement.

But it wasn’t quite enough where I wanted it to be. In particular, when our family calls on Messenger, we would like to be able to just turn on the TV selected to the right input. While this was partially possible (Google Assistant can turn on a TV with a Chromecast), and we could have tried wiring up the Nabu Casa integration to select the input of the HDMI switch, it would have not worked right if the last thing we used the TV for was the Nintendo Switch (not to be confused with the HDMI switch) or for Kodi — those are connected via a Yamaha receiver, on a different input of the TV set!

Enter Sony

But again, this was supposed to be working — the adapter board included a connection for an infrared LED, and that should have worked to send out the Sony SIRC commands. Well, except it didn’t, and that turned out to be another wild goose chase.

First, I was afraid that when I fixed the NEC timing I broke the SIRC ones — but no. To confirm this, and to make the rest of my integration easier, I took the Feather M4 to which I hard-soldered a Sony-compatible IR LED, and wrote what is the eponymous software defined remote control: a CircuitPython program that includes a few useful commands, and abstractions, to control a Sony device. For… reasons, I have added VCR as the only option beside TV; if you happen to have a Bluray player by Sony, and you want to figure out which device ID it uses, please feel free.

It might sound silly, but I remember seeing a research paper in UX from the ’90s of using gesture recognitions on a touchpad on a remote control to allow more compact remote controls. Well, if you wanted, you could easily make this CircuitPython example into a touchscreen remote control for any Sony device, as long as you can find all the right device IDs, and hard code a bunch of additional commands.

So, once I knew that at least on the software side I was perfectly capable of control the Sony TV, I had to go and do more hardware debugging, with the Saleae, but this time with the probes directly on the breadboard, as I had no TRS cable to connect to. And that was… a lot of work, to rewire stuff and try.

The first problem was that the carrier frequency was totally off. The SIRC protocol specifies a 40kHz carrier frequency, which is supposedly easier to generate than the 38kHz used by NEC and others, but somehow the Saleae was recording it as a very variable frequency that oscillated between 37kHz and 41kHZ. So I was afraid that trying to run two PWM outputs on the Trinket M0 was a bad idea, even if one of them was set to nought hertz — as I said, the HDMI switch didn’t need a carrier frequency.

I did toy briefly with the idea of generating the 40kHz carrier wave separately, and just gating it to the same type of signal I used for the HDMI switch. Supposedly, 40kHz generators are easy, but at least for the circuits I found at first glance, it requires a part (640kHz resonator) that is nearly impossible to find in 2020. Probably fell out of use. But as it turn out it wouldn’t have helped.

Instead, I took another Feather. Since I ran out of M4, except for the one I hardwired already an IR LED to, I instead pulled up the nRF52840 that I bought and barely played with. This should have been plenty capable to give me a clean 40kHz signal and it indeed was.

At that point I noticed another problem, though: I totally screwed up the adapter board. In my Feather M4, the IR LED was connected directly between 3V and the transistor switching it. A bit out of spec, but not uncommon given that it’s flashed for very brief impulses. On the other hand when I designed the adapter, I connected it to the 5V rail. Oops, that’s not what I was meant to be doing! And I did indeed burn out the IR LED with it. So I had to solder a new one on the cable.

Once I fixed that, I found myself hitting another issue: I could now turn on and off the TV with my app, but the switch stopped responding to commands either from the app or from the original remote! Another round of Saleae (that’s probably one of my favourite tools — yes I splurged when I bought it, but it’s turning out to be an awesome tool to have around, after all), and I found that the signal line was being held low — because the output pin is stuck high…

I have not tried debugging this further yet — I can probably reproduce this without my whole TV setup, so I should do that soonish. It seems like opening both lines for PWM output causes some conflicts, and one or the other end up not actually working. What I solved this with was only allowing one command before restarting the Feather. It meant taking longer to complete the commands, but it allowed me to continue with my life without further pain.

One small note here: since I wasn’t sure how Flask concurrency would interact with accessing a serial port, I decided to try something a bit out of the ordinary, and set up the access to the Feather via an Actor using pykka. It basically means leaving one thread to have direct access to the serial port, and queue commands as messages to it. It seems to be working fine.

Wrapping It All Up

Once the app was able to send arbitrary commands to the TV via infrared, as well as changing the input of the HDMI, I extended the Home Assistant integration to include the TV as a “media_player” entity as well. The commands I implemented were Power On and Off (discrete, rather than toggle, which means I can send a “Power On” to the TV when it’s already on and not bother it), and discrete source selection for the three sources we actually use (HDMI switch, Receiver, Commodore 64). There would be a lot more commands I could theoretically send, including volume control, but I can already access those via the receiver, so there’s no good reason to.

After that it was a matter of scripting some more complicated acts: direct selection of Portal, Chromecast, Kodi, and Nintendo Switch (which are the four things we use the most). This was easy at that point: turn on the TV (whether it was on or not), select the right input on either the receiver or the switch, then select the right input ion the TV. The reason why the order seems a bit strange is that it takes a few seconds for the TV to receive commands after turning on, but by doing it this way we can switch between Chromecast and Portal, or Nintendo Switch and Kodi, in pretty much no time.

And after that worked, we decided the $5/month to Nabu Casa were worth it, because that allows us to ask Alexa or Google Assistant to select the input for us, too.

Eventually, this lead me to replace Google’s “Turn off the TV” command in our nightly routine to trigger a Home Assistant script, too. Previously, it would issue the command to the Chromecast, routing through the whole Google cloud services between the device that took the request and the Chromecast. And then the Chromecast would be sending the CEC command to power off… except that it wouldn’t reach the receiver, which would stay on for another two hours until it finally decided it was time to turn off.

With the new setup, Google is triggering the Home Assistant script, and appears to do that asynchronously. Then Home Assistant sends the request to my app, that then sends it to the Feather, that sends the power off to the TV… which is also read by the receiver. I didn’t even need to send the power off command to the receiver itself!

All in all, the setup is satisfying.

What remains to be done is to try exposing a “Media Player” to Google Home, that is not actually any of the three “media_player” entities I have, but is a composite of them. That way, I could actually just expose the different input trees as discrete inputs to Google, and include the whole play, pause, and volume control that is currently missing from the voice controls. But that can wait.

Instead, I should probably get going at designing a new board to replace the breadboard mess I’m using right now. It’s hidden away enough that it’s not in our face (unlike the Birch Books experiments), but I would still like having a more… clean setup. And speaking of that, I really would love if someone already contributed an Adafruit Feather component for EAGLE, providing the space for soldering in the headers, but keeping the design referencing the actual lines as defined in it.

Home Assistant and CGG1 Sensors

You may or may not know that I’m one of the few tech people who don’t seem to scream against IoT and I actually have quite a few “smart home” devices in my apartment — funnily enough, one less now that we don’t have normal heating at home and we had to give up the Nest Thermostat. One of the things that I have not set up until now was Home Assistant, simply because I didn’t really need anything from it. But when we moved to this flat, I felt the need to have some monitoring of humidity and temperature in the “winter garden” (a part of the flat that is next to the outside windows, and is not heated), since we moved up in floors and we no longer have the “repair” of being next to another building.

After talking with my friend Srdjan, who had experience with this, I settled on ordering Xiaomi-compatible “ClearGrass” CGG1 sensors from AliExpress. These are BLE sensors, which mean they can broadcast their readings without requiring active connections, and they have a nice eInk display, which wastes very little power to keep running. There’s other similar sensors, some cheaper in the short run, but these sounded like the right compromise: more expensive to buy, but cheaper to run (the batteries are supposed to last six months, and cost less than 50p).

Unfortunately, getting them to work turned out to be a bit more complicated than either of us planned at the beginning. The sensors arrived on a day off, less than two weeks after ordering (AliExpress is actually fairly reliable when ordering to London), and I spent pretty much the whole day, together with Srdjan over on videocall, to get them to work. To save this kind of pain for the next person who come across these issues, I decided to write this up.

Hardware Setup Advices

Before we dig into the actual configuration and setup, let me point out a few things about setting up these devices. The most annoying part is that the batteries are, obviously, isolated to avoid running out in shipping, but the “pull out tab” doesn’t actually pull out. It took quite a bit of force to turn the battery compartment door, open it up, and then re-set it in. Be bold in taking them out.

The other advice is not from me but from Srdjan: write the MAC address (or however it’s called in Bluetooth land) on each of the sensors. Because if you only have one sensor, it’s very easy to tell which one it is, but if you bought more (say, four like I did), then you may have issues later to identify which one is which. So it’s easier to do while you’re setting them up, by turning them on one at a time.

To tell what’s the address, you can either use an app like Beacon Simulator, and listen to the broadcast, or you can wait until you get to a point when you’re ready to listen to the broadcast, later in the process. I would recommend the former, particularly if you live in a block of flats. Not only there’s a lot of stuff that broadcast BLE beacons, but nowadays pretty much every phone is broadcasting them as well due to the Covid 19 Exposure Notifications.

And to make sure that you don’t mix them up, I definitely suggest to use a label maker — and if you are interested in the topic, make sure to check out the Museum of Curiosity Series 13, Episode 6.

Finally, there’s a bit of a spoiler of where this whole process is going to end up going to — I’m going to explicitly suggest you avoid using USB Bluetooth dongles, and instead get yourself an ESP32 kit of some kind. ESP32 devkits are less than £10 on Amazon at the time of writing, and you can find them even cheaper on AliExpress — and they will be much more useful, as I’ll go ahead and tell you.

Home Assistant Direct Integration

So first of all, we spent a lot of time mucking around with the actual Home Assistant integrations. There’s a mitemp_bt sensor integration in Home Assistant, but it’s an “active” Bluetooth implementation, that is (supposedly) more power hungry, and require associating the sensors with the host running Home Assistant. There’s also an alternative implementation, that is supposed to use passive BLE scans.

Unfortunately, even trying to install the alternative implementation turned out to be annoying and difficult — the official instructions appears to expect you install another “store-like” interface on top of Home Assistant, which appears to not be that easy to do when you use their “virtual appliance” image in the first place. I ended up hacking it up a bit, but got absolutely nothing out of it: there isn’t enough logging to know what’s going on at any time, and I couldn’t tell if any packet was even received and parsed.

There is also a clear divergence between the Home Assistant guidelines on how to build new integration, and the way the alternative implementation is written — one of the guides (which I can’t now find easily, and that might speak to the reason for this divergence) explicitly suggests not to write complex parsing logic in the integration, and instead build an external Python library to implement protocols and parsers. This is particularly useful when you want to test something outside of Home Assistant, to confirm it works first.

In this case, having a library (and maybe a command line tool for testing) would have made it easier to figure out if the problem with the sensors was that nothing was received, or that something was wrong with the received data.

This was made more annoying too by the fact that for this to work, you need a working Bluetooth adapter connected to your Home Assistant host — which in my case is a virtual machine. And the alternative implementation tells you that it might interfere with other Bluetooth integrations, so you’re suggested to keep multiple Bluetooth interfaces, one for each of the integrations.

Now this shouldn’t be too hard, but it is: the cheapest Bluetooth dongles I found on Amazon are based on Realtek chipsets, which while supported by (recent enough) Linux kernels, need firmware files. Indeed the one dongle I got requires Linux 5.8 or later, or it requests the wrong firmware file altogether. And there’s no way to install firmware files in the Home Assistant virtual appliance. I tried, quite a few times by now.

ESPHome Saves The Day

ESPHome is a project implementing firmware (or firmware building blocks, rather) for ESP8266 and ESP32 boards and devices, that integrates fairly easily with Home Assistant. And since ESP32 supports BLE, ESPHome supports Xiaomi-compatible BLE sensors, such as the CGG1. So the suggestion from Srdjan, which is what he’s been doing himself, is to basically use an ESP32 board as a BLE-to-WiFi bridge.

This was easy because I had a bunch of ESP32 boards in a box from my previous experiments with acrylic lamps, but as I said they are also the same price, if not cheaper, than Bluetooth dongles. The one I’m using is smaller than most breadboard-compatible ESP32 boards and nearly square — it was a cheap option at the time, but I can’t seem to find one available to build now. It’s working out well by size, because it also doesn’t have any pin headers soldered, so I’m just going to double-side-tape it to something and give it a USB power cable.

But it couldn’t be as easy as to follow the documentation, unfortunately. While configuring the ESPhome is easy, and I did manage to get some readings almost right away, I found that after a couple of minutes, it would stop seeing any signal whatsoever from any of the sensors.

Digging around, I found that this was not uncommon. There’s two ESPhome issues from 2019: #317 and #735 that report this kind of problems, with no good final answer on how to solve them, and unfortunately locked to collaborators, so I can’t leave breadcrumbs for the next person in there — and it’s why I am now writing this, hopefully it’ll save headaches for others.

The problem, as detailed in #735, is that the BLE scan parameters need to be adjusted to avoid missing the sensors’ broadcasts. I tried a few combinations, and at the end found that disabling the “active” scan worked — that is, letting the ESP32 passively listen to the broadcasts, without trying to actively scan the Bluetooth channels seemed to let it stay stable, now for over 24 hours. And it should also be, as far as I can tell, less battery-draining.

The final configuration looks something like this:

esp32_ble_tracker:
  scan_parameters:
    duration: 300s
    window: 48ms
    interval: 64ms
    active: False

sensor:
  - platform: xiaomi_cgg1
    mac_address: "XX:XX:XX:XX:XX:XX"
    temperature:
      name: "Far Corner Temperature"
    humidity:
      name: "Far Corner Humidity"
    battery_level:
      name: "Far Corner Battery Level"

The actual scan parameters should be, as far as I can tell, ignored when disabling the active scan. But since it works, I don’t dare to touch it yet. The code in ESPhome doesn’t make it very clear if changing those parameters when disabling active scan is entirely ignored, and I have not spent enough time going through the deep stack to figure this out for certain.

The only unfortunate option of having it set up this way, is that by default, Home Assistant will report all of the sensors in the same “room”, despite them being spread all over the apartment (okay, not all over the apartment in this case but in the winter garden).

I solved that by just disabling the default Lovelace dashboard and customizing it. Because turns out it’s much nicer to customize those dashboards than using the default, and it actually makes a lot more sense to look at it that way.

Looking To The Future

So now I have, for the first time, a good reason to use Home Assistant and to play around with it a bit. I actually have interesting “connected home” ideas to go with it — but they mostly rely on getting the pantograph windows working, as they don’t seem currently to be opening at all.

If I’m correct in my understanding, we’ll need to get the building managers to come and fix the electronic controls, in which case I’ll ask for something I can control somehow. And then it should be possible to have Home Assistant open the windows in the morning, assuming it’s not raining and the temperature is such that a bit of fresh air would be welcome (which is most of the summer, here).

It also gives me a bit more of an incentive to finish my acrylic lamps work — it would be easy, I think, to use one of those as a BLE bridge that just happens to have a NeoPixel output channel. And if I’m going “all in” to wire stuff into Home Assistant, it would also allow me to use the acrylic lamps as a way to signal for stuff.

So possibly expect a bit more noise on this front from me, either here or on Twitter.

Update 2020-12-06: The sensors I wrote about above are an older firmware CGG1 sensors that don’t use encryption. There appear to be a newer version out that does use encryption, and thus require extracting a bind key. ~~Please follow the reported issue for updates.~~

Update 2021-06-26: I forgot to update this, but ESPHome 1.17.1 and later support encrypted CGG1 out of the box as my pull request was merged.