When I wrote about my new home theatre automation, I said I would give a more detailed explanation of what went on when I was looking at rewriting the interfacing between Home Assistant and the Feather that controlled the HDMI switch.
To understand a bit this setup, I need to take a step back and describe again what I wrote before. The code that has been running for the past few months was based on a Flask app that would then send the commands to the serial port. Originally I intended to make the Flask requests be answered one at a time, since access to the serial port can only happen synchronously. Unfortunately the nature of custom components, and the fact that I didn’t want to hardcode the mapping of sources to port in Home Assistant (as otherwise it would require a restart of HA if I changed where things would be connected) meant that I had to allow multiple requests in parallel.
To avoid spending too much time thinking about synchronization, at the time I went for Pykka to implement an actor model. Part of the reason for that is likely to be found in the fact I’ve been spending a lot of time working in Erlang over the past couple of years, which means the actor model is very fresh in my mind. Also to be honest, the whole idea of orchestrating a set of independent components is something that the actor model fits very nicely — which is why I was interested in seeing how Pykka made the actor component work for Python, though unfortunately I found it more complex and complicated than I would have liked.
So at the very least, I as hoping that switching to MQTT would mean reducing the overhead of using Pykka, since the messages are (to the best of my understanding) ordered, reducing significantly the need for synchronisation. This gut feeling did turn out to be fairly spot on, but the journey to get to the end was not linear at all.
Attempt 1: Keep Python, Throw Flask
My first attempt was the minimal change: remove Flask, but keep everything else the same, including Pykka for the time being, replacing it with an MQTT client. For this to work I needed to make sure I would get a working connection to Home Assistant’s Mosquitto Broker, then publish to the discovery topic, and finally subscribe to the command topic. It sounds simple, it turns out not to be as straightforward.
The first problem is that the choice of libraries in Python is not particularly exciting. There’s paho-mqtt which is basically the reference implementation, but to say that it’s not Pythonic is an understatement: values such as the qos parameter (which actually defines what I would call message semantics: at most once, at least once, exactly once) is left as a raw integer value of 0, 1, or 2 (and not documented anywhere) rather than wrapped in an enum class, errors are passed on in return values, rather than throwing exceptions, and the documentation is provided in the very long and windy readme rather than through clear API documentation.
The worst part for me was the callback definitions. I’m not exactly sure what I expected, but definitely I didn’t expect to just provide a callback callable as an attribute to the client:
def main():
switch = hdmi_switch.HDMISwitch()
def on_connect(client, userdata, flags, rc):
print(rc)
(result, _) = client.subscribe("$SYS/#")
print(result == mqtt_client.MQTT_ERR_SUCCESS)
(result, _) = client.subscribe(f"{NODE_NAME}/#")
print(result == mqtt_client.MQTT_ERR_SUCCESS)
HDMI_SWITCH_CONFIG = json.dumps(
{
"name": "HDMI Switch Input",
"options": list(switch.sources_list()),
"optimistic": "true",
"state_topic": HDMI_SWITCH_STATE,
"command_topic": HDMI_SWITCH_COMMAND,
}
)
result = client.publish(
HDMI_SWITCH_DISCOVERY, payload=HDMI_SWITCH_CONFIG, qos=1, retain=True
)
print(result.rc == mqtt_client.MQTT_ERR_SUCCESS)
result = client.publish(HDMI_SWITCH_STATE, payload="none", qos=1, retain=True)
print(result.rc == mqtt_client.MQTT_ERR_SUCCESS)
def on_message(client, userdata, message):
if message.topic == HDMI_SWITCH_COMMAND:
switch.select_source(message.payload.decode())
client = mqtt_client.Client(client_id="HDMI Switch")
client.on_connect = on_connect
client.on_message = on_message
client.tls_set()
client.username_pw_set("user", "password")
client.connect("homeassistant", 8883, 60)
while True:
client.loop()
Code language: Python (python)Now the code above is rough in the best of cases, so don’t expect that to be quite clean. But you can see the slight confusion on having to create a client, attach the generic callbacks, go with it. I didn’t do any error check in this code, but as far as I can tell, a connection failure would be reported as a call to on_connect with a rc value that is not zero.
As I said, there’s very little idiomatic Python in this library, and no usage of asyncio, probably because it still supports Python 2.7!
According to Wikipedia there’s at least two more Python libraries for MQTT. The first is from Adafruit — which I would probably have considered, if it wasn’t that they seem to have joined the trashfire of NFTs (not for the first time). So at this point I do not feel comfortable with approaching their projects at all, which is a shame, given how much I liked their Feathers. The second is mqttools, which starts well: Python 3.7 for asyncio, and a more idiomatic approach to block while waiting for the next message. Unfortunately, it does not support authentication (which Home Assistant requires), nor any other qos setting but “at most once”, which makes it a bit on the annoying side to integrate with Home Assistant.
So after trying a few times, I thought it would be better to try in a different way.
Attempt 2: Only Half-Jokingly, Erlang
So you may remember that I keep saying that languages are just tools. And one of the most obscure tools in the arsenal is Erlang. It’s obscure for good reasons, but it also turns out to be something that I’ve been working on heavily (though not exclusively) for the past year and a half, so it was the kind of tool that was easy for me to reach out to.
Because of Erlang’s innate Actor Model integration, I expected that what I would have to do is to write a gen_server module, with the client casting messages to it. In my mind that would be the perfect architecture, and would probably even have been the kind of example usage of Erlang that would make a whole lot of sense!
Somehow, the first couple of libraries I found to implement this didn’t feel particularly idiomatic for Erlang either, as they relied on tail-calling a loop-of-sorts, and felt like a translation to Erlang of the paho library. If I had spent a bit more time, I would probably have found emqtt, which despite not being exactly what I had in mind, comes a lot closer. Of course then I would have had to also figure out how to access a serial port from Erlang, and that does not sound like fun.
But yes, I do feel that for the part that involves interacting with MQTT, Erlang is a fairly good option. The Actor Model works fairly well for dealing with the pubsub model defined by the protocol, and if hardware access was not an issue (or if you could delegate those portions to NIFs), then it would be my tool of choice to solve this problem space in the future.
Attempt 3: What About Rust?
Notwithstanding what I wrote about programming languages as tools, there are some valid reasons to look into programming languages that are “trendy”, particularly when writing something new, as a calculated risk. From one side, you have a lot of information, and possibly a lot of examples on how to achieve whatever you need to do to pick from, from the other you may end up with very bad quality suggestions due to terrible tutorials and a lot of newcomers writing their first library.
While I have dabbled in fixing minor annoyances with Rust tools at work, I haven’t really done anything in Rust from scratch before. You may remember that I was hoping to find decent image processing libraries, but that didn’t pan out, and to this date, I don’t think there’s anything particularly interesting to me in that field. But certainly there would be a better MQTT client in Rust, right?
Well, yes. rumqttc is a fairly good client based on my brief experience with it. The documentation and the examples could use a little bit more work, and there’s some funny nested matching when you end up writing something like
for notification in connection.iter() {
match notification {
Ok(rumqttc::Event::Incoming(rumqttc::Packet::Publish(rumqttc::Publish{topic: hdmi_switch_set_topic, payload: payload, ..}))) => {
println!("Requested changing HDMI switch {:?}", payload);
match input_to_command(payload) {
command if !command.is_empty() => remote.send_command(command),
_ => ()
}
},
_ => println!("Notification = {:?}", notification),
}
}Code language: PHP (php)Now, Rust is notoriously a batteries-not-included language. So to set up TLS encryption you need to use rustls. Which is where I both nearly got ready to flip the table, and where I got a bit annoyed. The bit annoyed part is that I freaking mistype it all the time! It’s not “rutls” and it’s not “rusttls” — it’s “rustls”, with the last t in “rust” merged with the first t in “tls”. Just freaking why?
The near-tableflip comes instead from the philosophy behind the library As their documentation states (emphasis mine):
Rustls is a TLS library that aims to provide a good level of cryptographic security, requires no configuration to achieve that security, and provides no unsafe features or obsolete cryptography.
What this means is that Rustls does not let you open a TLS connection to a server with self-signed certificates. There’s no option for you to allow that, there’s no “break glass” option. The best official answer I found is suggesting that if you really don’t want to use a public PKI you should set up your own CA and use a certificate signed by that, then you can trust the CA instead.
I feel this is overkill, particularly when trying to talk to a local TLS broker, but thankfully I don’t have to worry too much, as I configured my Home Assistant for Let’s Encrypt, which means it also loaded the same cert onto the Mosquitto broker. On the other hand, this does complicate the setup if you don’t want to leak your hostname to the public Internet, by the way of Certificate Transparency.
Once solved that issue, writing the rest of the client was actually quick. I’m not entirely sure whether publishing needs to happen in a separate thread from the one that is reading events from the connection, but I followed the examples in the repository and got something that worked. My plan was to tidy up that only at the end.
Unfortunately, the next hurdle was to handle the serial port, and that’s where I did not expect things to go as sour as they ended up going. There’s a serialport crate, which is inspired by Qt’s serial port interface, and that sounds good. Unfortunately it does not look like Qt is a good fit for Rust concepts, and what you end up with is a hodgepodge of traits and builders that to me make no sense. So for instance the builder takes the various parameters of the serial port, (baudrate, symbol definition) in addition to the port name, but then the constructed object (and its trait) need to define a way to tweak all of those since you can change them after opening it.
Where things get a lot more annoying for what I can tell is that the implementation of reading and writing from the serial port is not really thought out, as they share the same object (or whatever is called in Rust). This means that when you want to read line-by-line, you need to use BufReader, which is easy enough, but it becomes a problem if you want to also write to the serial port. There is a try_clone() method to attempt dealing with this, but even with that I failed at writing enough code to drive the IR adapter.
The main issue I found was the inability to store the BufReader object in a structure. I’m sure that I’m just missing enough context to figure out how to make the dynamic type work, but it just was discouraging, particularly as it’s clear that the dynamic typing of all of this is only present due to the abstraction that is built into the library. This appears to be the case of the typical early abstraction that might not have been needed at all. I can’t imagine what other type of serial port, beside the TTYSerial it implements already, it would possible to implement with the code it has.
Now to be honest, the crate does say:
Please note that if you want a real asynchronous serial port you should look at mio-serial or tokio-serial.
Unfortunately both those crates are also providing a notice that they are not being maintained, and that didn’t bode well for using Rust to work with serial port devices, which is quite a few different things I can think of, not just my hacky smart home solutions. I guess this is the second time Rust looked promising, but misses the mark for me.
I can at least say that I tried, and that I have managed at least to get some of the work done with Rust, which is a first for me. I did like the match expressions, and it is something I got more used to by working with Erlang, and while I do miss the ability to define a literal hashmap, I guess it just means that the setup is a bit more verbose, but otherwise it would be just as fast.
I didn’t go far enough to understand the command line parsing facilities allowed by Rust. I have seen that there appear to be at least some level of porcelain to set that up, which is promising, but I doubt I’ll be dabbling much more into Rust for the moment (although I might have had done that already by the time this goes to publish, but at work, and with that I mean you won’t be reading anything about that on the blog.)
What I Didn’t Try: Go
The elephant in the room here is that I have not even attempted ot use Go for this. To be honest, I think Go would have a perfectly valid chance to work right for what I was trying to get. Go has pretty good handling of events via channels, which means the implementation of MQTT clients should be fairly straightforward.
But I found that I don’t particularly enjoy writing Go myself, and I would rather have kept that as a last resort.
MQTT and Home Assistant
There isn’t really much to see about this part. Home Assistant supports MQTT discovery out of the box, once configured. And having set that up already for zigbee2mqtt means that I just had the input select show up so by having sent the right configuration for the HDMI input switch selector, it just showed up correctly in the list of entities.
This part was basically just magic. Wiring this to work with the Universal Media Player was a bit more complicated, but that didn’t require any code on the server side, which was a significant improvement for me.
Among other reasons, this means removing the last custom component that I had not published yet. Yay!
Final Thoughts
As I said in the post about the control itself, I’m going to replace the whole shenanigan with an ESPHome based controller, which means that the programming language is only a temporary selection anyway: YAML is the language that ESPHome will require me to use.
It was a good idea to spend some time looking at alternatives. I would have been laughing maniacally if I did end up wiring Erlang in my home automation! Rust was a fun language to pick up quickly, and the fact that I did manage to get at least the MQTT client working after spending a couple of hours in pretty much the middle of the night (my wife was playing online) suggests it’s not that difficult of a language to mangle, at least.
I am honestly annoyed that the options for writing MQTT clients in Python are. Because the MQTT support for Home Assistant is absolutely sweet, and I could see a lot more options to integrate with other Python tools through it. Although honestly I don’t think I have anything else to integrate any time soon.
I guess these are two more tools (MQTT, and Rust) that will go nicely in my mental toolbox, but not going to be used much for the time being.