Home Assistant and CGG1 Sensors

You may or may not know that I’m one of the few tech people who don’t seem to scream against IoT and I actually have quite a few “smart home” devices in my apartment — funnily enough, one less now that we don’t have normal heating at home and we had to give up the Nest Thermostat. One of the things that I have not set up until now was Home Assistant, simply because I didn’t really need anything from it. But when we moved to this flat, I felt the need to have some monitoring of humidity and temperature in the “winter garden” (a part of the flat that is next to the outside windows, and is not heated), since we moved up in floors and we no longer have the “repair” of being next to another building.

After talking with my friend Srdjan, who had experience with this, I settled on ordering Xiaomi-compatible “ClearGrass” CGG1 sensors from AliExpress. These are BLE sensors, which mean they can broadcast their readings without requiring active connections, and they have a nice eInk display, which wastes very little power to keep running. There’s other similar sensors, some cheaper in the short run, but these sounded like the right compromise: more expensive to buy, but cheaper to run (the batteries are supposed to last six months, and cost less than 50p).

Unfortunately, getting them to work turned out to be a bit more complicated than either of us planned at the beginning. The sensors arrived on a day off, less than two weeks after ordering (AliExpress is actually fairly reliable when ordering to London), and I spent pretty much the whole day, together with Srdjan over on videocall, to get them to work. To save this kind of pain for the next person who come across these issues, I decided to write this up.

Hardware Setup Advices

Before we dig into the actual configuration and setup, let me point out a few things about setting up these devices. The most annoying part is that the batteries are, obviously, isolated to avoid running out in shipping, but the “pull out tab” doesn’t actually pull out. It took quite a bit of force to turn the battery compartment door, open it up, and then re-set it in. Be bold in taking them out.

The other advice is not from me but from Srdjan: write the MAC address (or however it’s called in Bluetooth land) on each of the sensors. Because if you only have one sensor, it’s very easy to tell which one it is, but if you bought more (say, four like I did), then you may have issues later to identify which one is which. So it’s easier to do while you’re setting them up, by turning them on one at a time.

To tell what’s the address, you can either use an app like Beacon Simulator, and listen to the broadcast, or you can wait until you get to a point when you’re ready to listen to the broadcast, later in the process. I would recommend the former, particularly if you live in a block of flats. Not only there’s a lot of stuff that broadcast BLE beacons, but nowadays pretty much every phone is broadcasting them as well due to the Covid 19 Exposure Notifications.

And to make sure that you don’t mix them up, I definitely suggest to use a label maker — and if you are interested in the topic, make sure to check out the Museum of Curiosity Series 13, Episode 6.

Finally, there’s a bit of a spoiler of where this whole process is going to end up going to — I’m going to explicitly suggest you avoid using USB Bluetooth dongles, and instead get yourself an ESP32 kit of some kind. ESP32 devkits are less than £10 on Amazon at the time of writing, and you can find them even cheaper on AliExpress — and they will be much more useful, as I’ll go ahead and tell you.

Home Assistant Direct Integration

So first of all, we spent a lot of time mucking around with the actual Home Assistant integrations. There’s a mitemp_bt sensor integration in Home Assistant, but it’s an “active” Bluetooth implementation, that is (supposedly) more power hungry, and require associating the sensors with the host running Home Assistant. There’s also an alternative implementation, that is supposed to use passive BLE scans.

Unfortunately, even trying to install the alternative implementation turned out to be annoying and difficult — the official instructions appears to expect you install another “store-like” interface on top of Home Assistant, which appears to not be that easy to do when you use their “virtual appliance” image in the first place. I ended up hacking it up a bit, but got absolutely nothing out of it: there isn’t enough logging to know what’s going on at any time, and I couldn’t tell if any packet was even received and parsed.

There is also a clear divergence between the Home Assistant guidelines on how to build new integration, and the way the alternative implementation is written — one of the guides (which I can’t now find easily, and that might speak to the reason for this divergence) explicitly suggests not to write complex parsing logic in the integration, and instead build an external Python library to implement protocols and parsers. This is particularly useful when you want to test something outside of Home Assistant, to confirm it works first.

In this case, having a library (and maybe a command line tool for testing) would have made it easier to figure out if the problem with the sensors was that nothing was received, or that something was wrong with the received data.

This was made more annoying too by the fact that for this to work, you need a working Bluetooth adapter connected to your Home Assistant host — which in my case is a virtual machine. And the alternative implementation tells you that it might interfere with other Bluetooth integrations, so you’re suggested to keep multiple Bluetooth interfaces, one for each of the integrations.

Now this shouldn’t be too hard, but it is: the cheapest Bluetooth dongles I found on Amazon are based on Realtek chipsets, which while supported by (recent enough) Linux kernels, need firmware files. Indeed the one dongle I got requires Linux 5.8 or later, or it requests the wrong firmware file altogether. And there’s no way to install firmware files in the Home Assistant virtual appliance. I tried, quite a few times by now.

ESPHome Saves The Day

ESPHome is a project implementing firmware (or firmware building blocks, rather) for ESP8266 and ESP32 boards and devices, that integrates fairly easily with Home Assistant. And since ESP32 supports BLE, ESPHome supports Xiaomi-compatible BLE sensors, such as the CGG1. So the suggestion from Srdjan, which is what he’s been doing himself, is to basically use an ESP32 board as a BLE-to-WiFi bridge.

This was easy because I had a bunch of ESP32 boards in a box from my previous experiments with acrylic lamps, but as I said they are also the same price, if not cheaper, than Bluetooth dongles. The one I’m using is smaller than most breadboard-compatible ESP32 boards and nearly square — it was a cheap option at the time, but I can’t seem to find one available to build now. It’s working out well by size, because it also doesn’t have any pin headers soldered, so I’m just going to double-side-tape it to something and give it a USB power cable.

But it couldn’t be as easy as to follow the documentation, unfortunately. While configuring the ESPhome is easy, and I did manage to get some readings almost right away, I found that after a couple of minutes, it would stop seeing any signal whatsoever from any of the sensors.

Digging around, I found that this was not uncommon. There’s two ESPhome issues from 2019: #317 and #735 that report this kind of problems, with no good final answer on how to solve them, and unfortunately locked to collaborators, so I can’t leave breadcrumbs for the next person in there — and it’s why I am now writing this, hopefully it’ll save headaches for others.

The problem, as detailed in #735, is that the BLE scan parameters need to be adjusted to avoid missing the sensors’ broadcasts. I tried a few combinations, and at the end found that disabling the “active” scan worked — that is, letting the ESP32 passively listen to the broadcasts, without trying to actively scan the Bluetooth channels seemed to let it stay stable, now for over 24 hours. And it should also be, as far as I can tell, less battery-draining.

The final configuration looks something like this:

esp32_ble_tracker:
  scan_parameters:
    duration: 300s
    window: 48ms
    interval: 64ms
    active: False

sensor:
  - platform: xiaomi_cgg1
    mac_address: "XX:XX:XX:XX:XX:XX"
    temperature:
      name: "Far Corner Temperature"
    humidity:
      name: "Far Corner Humidity"
    battery_level:
      name: "Far Corner Battery Level"

The actual scan parameters should be, as far as I can tell, ignored when disabling the active scan. But since it works, I don’t dare to touch it yet. The code in ESPhome doesn’t make it very clear if changing those parameters when disabling active scan is entirely ignored, and I have not spent enough time going through the deep stack to figure this out for certain.

The only unfortunate option of having it set up this way, is that by default, Home Assistant will report all of the sensors in the same “room”, despite them being spread all over the apartment (okay, not all over the apartment in this case but in the winter garden).

I solved that by just disabling the default Lovelace dashboard and customizing it. Because turns out it’s much nicer to customize those dashboards than using the default, and it actually makes a lot more sense to look at it that way.

Looking To The Future

So now I have, for the first time, a good reason to use Home Assistant and to play around with it a bit. I actually have interesting “connected home” ideas to go with it — but they mostly rely on getting the pantograph windows working, as they don’t seem currently to be opening at all.

If I’m correct in my understanding, we’ll need to get the building managers to come and fix the electronic controls, in which case I’ll ask for something I can control somehow. And then it should be possible to have Home Assistant open the windows in the morning, assuming it’s not raining and the temperature is such that a bit of fresh air would be welcome (which is most of the summer, here).

It also gives me a bit more of an incentive to finish my acrylic lamps work — it would be easy, I think, to use one of those as a BLE bridge that just happens to have a NeoPixel output channel. And if I’m going “all in” to wire stuff into Home Assistant, it would also allow me to use the acrylic lamps as a way to signal for stuff.

So possibly expect a bit more noise on this front from me, either here or on Twitter.

Stop slagging off IoT users if you care about them

It’s the season for gifts (or, as some would say, consumerism), and as way too often is the case, it starts a holy war between those who enjoy gadgets, new technology, and Internet-connected appliances, and those who define themselves as security conscious and tell people that they wouldn’t connect a computer to the Internet if they didn’t have to.

Those who follow me on Twitter, probably already know which side of this divide I find myself in: I do have a few IoT devices at home, and I’m “IoT-positive”. I even got into a long Twitter discussion years ago about the fact that IoT is no longer just a random marketing buzzword, but got to actually refer to a class of devices that the public at large can identify, the same way as “white goods” would, in the British Isles.

I have a very hard time giggling Twitter posts from geek supremacists making fun of Internet-connected ovens, when the very same geeks insist they would never possibly buy something like that — despite the excited reactions of the Linux, BSD and FLOSS communities nearly fifteen years ago at the release of a NetBSD-operated toaster.

This does not mean that I’m okay with all the random stuff that’s being proposed as an Internet-enable device. I have looked briefly at Bluetooth toothbrushes and I’m still lost on what the value proposition is with them. And even last year when I got a smart plug it took me a lot of thoughts to figure out what it would be used for, and decided that, for 11 months of the years, the plug will stay in a box, and it will come out at the same time as the Christmas Tree.

Today’s musing is finding a “Smart Essential Oil Diffuser” which was funny because I was looking for something completely different (a kitchen oil bottle, it’s a long story), but I actually clicked on it out of curiosity. I have looked into this type of devices last year, while I was writing my post about smart plugs: they sounded like an interesting approach to make sure they are on for a few minutes before we arrive home, just to give a good smell to the flat without having to keep a more standard Ambipur on all the time.

Indeed, I have considered converting our Muji diffuser into a “Smart” one with an Adafruit Featherwing, but it works too good to open it up right now, and nearly everything I can see in stores like TkMaxx appears to be fairly low quality and with power supplies that look too low to be true. But the device I found over there also appears to be a fairly bad one, so I think our old-school Muji diffuser will stay around instead.

The thing is, whether you like it or not, the public at large, not just the geeks, are the driving force of manufacturers. And you won’t win anyone over by being smug and pointing at how good you are at not buying stuff that is Internet-enabled, because you don’t trust it. The public will. So instead of throwing all IoT options under a bus, and making fun of their users, I prefer Matthew’s approach of actually looking into the various lightbulbs and documenting which ones are, indeed, terrible.

Indeed, if you think that Internet-enabled aroma diffusers are pointless, useless, and nobody will want to have one… you’ll find out that someone will be making one, people will buy one, and most likely some random Chinese factory will start making a generic enough model that other companies can rebrand, and provide the least secure option out there.

I think this is also a valid metaphor for politics nowadays. It doesn’t matter that you are sure you have the right answer — if you demonize the public at large telling them they are stupid, or that they are at fault for things, you’re not likely going take your advice for long.

So if you care about the people around you, instead of telling them that IoT is terrible and you shouldn’t connect anything to a computer ever in a million years, try finding what is not terrible, while still providing them with the convenience they desire. Whether it is a smart lightbulb, a smart thermostat, or an app-enabled doorbell. And if you can’t find anything, and you still think you’re smarter than others, make it. Clearly there’s desire for those tools, can you make a secure and safe one?

Musings after buying a smart plug

I know that people will go and start ranting on using terms like “Internet of Shit” just for the title I’m using here. Despite being as wary and cynical about the subject of connected appliances as the next security-aware engineer, I want to point out that those reactions are blind and lacking empathy. So if your answer is to think that you’re smarter than the plug and me combined, there’s maybe no reason for you to stay around to read the post.

I also need to put the usual disclaimer forward: I work for Google, a company that produces “smart” appliances. I don’t have anything to do with the hardware products, have no special insight into them, and I am her talking about things as myself alone. I’m also not really talking about Google hardware beside for a few references to the Assistant here and there, and that’s simply because I happen to be using Google Home as my hub.

As I said I’m fairly cynical about smart appliances. It took quite a bit for me to even buy a single one, but I’m now a very happy user of a LIFX Mini Colour smart bulb. It was probably this year’s best gadget buy for me, and it is not just about the ability to control the light with an app on my phone — or with the Assistant. The bulb can dim, change colours, and can be set onto a dynamic schedule. It’s extremely convenient, and an improvement in my quality of life, particularly by setting it to red as I go to sleep, instead of keeping it bright white.

Of course, like always when buying a device that relies on external services to work (the infamous “cloud”), I am still worried about the risk of the company going under, or dropping support for my specific device, and letting me deal with the broken pieces. But quite honestly, if you tried to avoid all the cloud-based services and hardware nowadays, you will end up a luddite. And maybe you want that. Besides IKEA, that requires their full bridge, I don’t know of any other smart home brand that provides local-only controls — and local-only means no talking to the Assistant to turn on the light as part of the morning routine.

I’m happy enough that my LIFX can be controlled without an active Internet connection (this happened before). Maybe I’ll follow Matthew Garrett’s example and start reverse engineering it into a Python script for the rainy days.

But I digressed enough. What I wanted to talk about was rather smart plugs. Because that’s a device I’m not entirely sold on the idea of smart plugs, I started the original draft of this post because I thought they were completely useless. I changed my own mind as I was writing this, and that’s why I actually wanted to post this.

So why did I buy a smart plug if I am not sold on the idea? Well, since this is our first Christmas together, my girlfriend wants to have a proper Christmas tree at home. And since I would like to see the tree while I approach the apartment on the bus or on foot (hey, I have not had a Christmas Tree for more than a decade, I can have some fun!), I would like to have IFTTT turn it on for me.¹

I ended up buying a TP-Link Smart Plug (UK version), which comes with their own app, and integration with the various services including IFTTT and Google Assistant. Which means we’ll be able to say “Hey Google, turn on the Christmas Tree!”

There are differences between a smart bulb and a plug though. The former adds a significant amount of value add, with things like dimming, different colours, and so on. A smart plug is still only a binary operator, it’s either on, or off. You cannot do fine-grained control over that, you can only turn things on or off.

So after thinking about this, I realized there are a few requirements for something to make sense to have connected to a smart plug:

It needs to be something that cannot stay on standby the whole day. Because if it can, there’s no real advantage in having a smart plug for it, keeping it in stand by is easier, and can easily be cheaper, as the stand-by of the plug connected to WiFi might be higher consumption than the device itself.

It needs to be something that can be at least “readied” unattended. Turning on the plug for a hairdryer is not going to be very useful, if you’re not there to use it. Also if readying something unattended is too risky, it’s a bad idea to use a smart plug. This is the case for clothes irons for instance; I wouldn’t want to turn mine on if I’m not there to make sure that it’s not on top of something it shouldn’t be.

If it’s something that comes with consumables, it needs to have big enough reserves, or a way to feed itself. Going back to the clothes iron, the one I have does not have enough of a water tank. If I was to turn it on too soon, it would just waste all of it and I would go and find it empty, which is just as bad.

Given these considerations, one of the common suggestions I hear is coffee makers. At first I thought this was pointedly American, as indeed a percolator style coffee can be filled in in the evening, and then be set to turn on in the morning and make coffee for you to drink. When I spent extensive time in Los Angeles, I used the timer on a percolator to make sure I would have hot “coffee” ready immediately after waking up. But then I realized that this is very similar for Italian-style espresso machines, too: they have an internal boiler that takes a while to get to temperature and be usable, they usually have a tank big enough for a full day (or in some cases they may be connected to the water mains), and they consume enough power in standby that you wouldn’t want to keep it turned on overnight. For those who don’t drink coffee, the same can be true of automated teawakers or teamakers — I had one from Twinings back in Italy.

Another appliance that fits the bill fairly well is the electric bathroom heater, or towel rack. Heating in general is likely better suited by a smarter “whitebox” approach — indeed I have booked an appointment to install a Nest thermostat at my apartment, after getting my landlord’s permission, because I want to be able to automate hot water availability and easily tweak the temperature over the day. But in some cases, you have additional bathroom heating that has less control: I have on/off towel racks in my bathrooms in London, and my mother uses a small electric heater in Italy, after we messed up with the house’s heating plan by replacing a bulky and leaky boiler with a more modern and efficient one.

Now for both of these examples, smart plugs are not the only obvious solution. Indeed, percolators, teawakers, and espresso machines, as well as many small electric heater, often come with their own timer. This works great for the people who have a clear schedule and fixed routine. In my case that’s rarely the case: I wake up at a different time depending on what my day looks like, sometimes I oversleep because I had a bad night, sometimes I’m up earlier than average because my girlfriend is staying over and she has to go to work. A similar result exists for my mother due to different requirements: she lives alone and really doesn’t have any reason to get up a fixed time unless she’s waiting for deliveries, services, or stuff like that. And since the house is on two floors, and she has knee pain, being able to turn on the heating, get the bathroom ready, or make sure that the coffee machine is warmed up without having to get downstairs immediately, would be a very nice feature.

I can definitely see myself appreciating the idea of saying “Hey Google, Good Morning”, and know that by the time I finished listening to the BBC News headlines, the coffee is ready and still hot for me, while the bathroom is warm enough to take a shower in. Doesn’t really work for me here, because I make pour-over coffee, and the towel rack is not controlled by a normal plug, but I can dream can’t I?

By the way, Google Assistant can do that, although it’s a bit hidden: from the Home app, go into the Account tab (the last one on the right), click Settings, go to the Assistant tab, and then select Routines. From there you can set up the actions you want taken when you give it a specific hotphrase.

For most of other appliances, I would probably need more whitebox smartness. I already rely on the timer for my washing machine, but it would be nice to just put it into “standby”, loaded and locked, but not start it until I wake up, or until I’m actually leaving the apartment (I don’t get woken up by the noise of the one I have here in London, but I would have been by the one in Dublin). And something that can remind me was I get home (“Hey Google, I’m home”) that I need to unload the dishwasher.

One of the things that I actually nearly considered giving a smart plug to was the Air Wick freshner. While I would love having a fine grained intensity control that would keep a background fragrance during the day, but raise it just as I’m ready to get home, to make me feel good, just having the ability to turn it off the moment I leave and on again when I come back home, would be a very nice thing to have. On the other hand, it turns out that the plug-in device consumes significantly less power than the smart plug in stand-by, so it makes no sense as it is.

I guess using more sophisticated fragrance delivery devices, such as Yankee Candle’s Scenterpiece (that my mother has, at home) would make more sense. Alternatively, Muji has very nice oil burners, though they have a small tank for water, and candle warmers are getting more common (these are probably better than the Scenterpiece in my experience). Unfortunately these are usually table-top devices, rather than plug-in, and I don’t have the space where I would want to use it. So if someone from Air Wick or Ambi Pur is reading, consider that I would pay just as much as a smart plug to have a smart plug-in freshener that can be set to adjust the intensity over the day!

So to close it up, I’m somewhat skeptical about getting more smart plugs for myself, but I can definitely see a number of useful cases for them, as well as for smarter “whitebox” appliances. Indeed, if my mother had a decent Internet connection in 2018, I would probably set her up with quite a few of those, to make her life easier. Call them accessibility helpers, maybe.

You may remember that I have some particular attachment to Christmas lights Rube Goldberg machinery. The idea of having my own IFTTT-compatible smart Chrimast light tube did pass through my head. ↩