Cultural Diversity

I would say that in most cases, I’m the worst person to talk about diversity as, like John Scalzi said, I’m playing at the lowest difficulty setting. There is a small exception to this when the matter relates to cultural diversity in a vastly USA-based environment, which is what I would like to spend a few words on this time.

Let’s start with language. For open source developers, working with people from different countries, and for which English is not the native language, is not uncommon at all — indeed, projects such as Gentoo and VideoLan tend to have overall more people for whom English is a second (if not third) language. There is a difference, though, when it comes to work environments such as my previous bubble, where you have to speak English a lot — impromptu, on the spot.

It took adjustment when I moved to Dublin, despite having spent most of the previous year in Los Angeles: on one side, South-Californian English and Dublin English are significantly different in tone and intentions, and on the other hand, it required “checking the gain” on what people coming from other languages meant with certain words. Again, not something totally new for those who spent time in various Open Source projects, but even IRC allowed you to take a moment to type an answer back, or to re-read what the other person said. And while the voice tone and body language help, it’s still harder to process, understand, and form a reply in your second (or third) language in real-time than it would be over asynchronous medium.

London was another kettle of fish altogether — maybe because I have listened to enough Radio 4 to grasp many Londoners expression fasters than I picked them up in Dublin, or maybe because I have built up the experience there. But that doesn’t mean it wasn’t hard — indeed in my experience I found that British-born-and-raised people tend to be (unwittingly) less forgiving for mis-speaking, expecting every word used to having been carefully chosen by the speaker, including any obvious-to-them rude turn of phrase. This doesn’t appear to be my impression only — the FT’s Michael Skapinker wrote about it, more than once, and I would suggest reading his articles to both the English native speakers, and those of us “second-languagers” that find it hard to work productively with them.

Now, before somebody says that I’m painting the whole group of native English speakers with a single brush — that is certainly not the case. I already singled out the Dubliners before, and I have plenty of friends, colleagues, and ex-colleagues that have learnt not to assume that every word is perfectly weighted beforehand, particularly in spoken lines, and have asked me more than a few times if the word I used was meant to sound as aggressive as it did.

There’s another fun interaction that I learnt to appreciate: talking shop with people coming from cultures that are very direct, such as some of my Romanian friends — we can start cussing and repeat the word shit many times in a row, get excited, and maybe even disagree vehemently on concepts, solutions, and decisions, … and then go grab coffee together like nothing happened. At least once I’ve been asked not quite directly whether I have been a hypocrite — but no, it’s just that for me (us) technical disagreements among friends are just that… technical disagreements.

If all of this boggles your mind and sounds like me trying to justify my tone or behaviour, I would suggest you to read The Culture Map. The book is a fascinating read, and goes into a lot of examples of how different culture “baselines” differ — with repeated reminders that the fact that a culture baseline does not mean that everyone in that culture behaves exactly the same way, we’re not in a world of hats.

Another point that I feel should be spelled out explicitly is about general (popular) cultural references — they don’t really translate very well, under different axis. I have a friend who gets annoyed at Harry Potter references in documentation and service naming, because they didn’t actually care for the series, and so anything that feels “obvious” to a fan goes straight over their head, and I sympathise.

I know of similar complains with most other “big fandoms”: Star Wars, Star Trek, Game of Thrones, Lord of the Rings, Dragonball, Naruto, … it’s a type of gatekeeping that is subtle, but still present: if you don’t happen to have at least dabbled in most of these, things will be slightly harder for you, and give you a feeling that you’re not welcome. Also it turns out that those who read/watched some of those big names in another language might be just as annoyed, because a lot of times names and terms got translated to something different, maybe closer to the target culture, that makes the reference even harder to grasp.

Also, let me be clear that this is not only a problem within the tech spaces dominated by white male geeky engineers. A few years ago I found myself having an argument over the fact that I missed the window of time to submit intern projects ideas, because I went on to check the old site, which was handily named “iwantanintern”, rather than finding out that they switched it over a new one named “redfishbluefish”. When I pointed out it’s a very opaque name to me, I was informed by a surprised HR person that they were expecting everybody to know the book by Dr. Seuss – which turns out it doesn’t even appear to be translated in Italian according to Wikipedia – and that it fit perfectly well with the companion Waldo website, named after the North American name of the protagonist from Where is Wally? As it turns out, I at least knew Wally as a kid, but most of the other children books I was given were either Italian or Disney, or both, so Dr. Seuss never figured into my upbringing.

So what’s the point of bringing up all of this? Well, I wanted to point out that there still is quite a bit of discrimination that can be felt among those of us that are otherwise well into a privileged class — with the hope that this can both make it easier to empathise with those who are less privileged, and as an answer – that I should have provided there and then, and out loud – to those I have heard before saying that I shouldn’t complain, since I can completely pass for British if I wanted to (cannot, and will not!) and so I have nothing to fear from drunk right wingers at a pub the day after Brexit deadlines.

Diagonal Contributions

This is a tale that starts on my previous dayjob. My role as an SRE had been (for the most part) one of support, with teams dedicated to developing the product, and my team making sure that it would perform reliably and without waste. The relationship with “the product team” has varied over time and depending on both the product and the SRE team disposition, sometimes in not particularly healthy way either.

In one particular team, I found myself supporting (together with my team) six separate product teams, spread between Shanghai, Zurich and Mountain View. This put particular pressure on the dynamics of the team, particularly when half of the members (based in Pittsburgh) didn’t even have a chance to meet the product team of two services (based in Shanghai), as they would be, in the normal case, 12 hours apart. It’s in this team that I started formulating the idea I keep referring to as “diagonal contributions”.

You see, there’s often a distinction between horizontal and vertical contributions. Vertical referring to improving everything of a service, from the code itself, to its health checks, release, deployment, rollout, … While horizontal referring to improving something of every service, such as making every RPC based server be monitored through the same set of metrics. And there are different schools of thought on which option is valid and which one should be incentivised, and so it usually depends on your manager and their manager on which one of the two approach you’ll be rewarded to take.

When you’re supporting so many different teams directly, vertical contributions are harder on the team overall — when you go all in to identify and fix all the issues for one of the products, you end up ignoring the work needed for the others. In these cases an horizontal approach might pay off faster, from an SRE point of view, but it comes with a cost: the product teams would then have little visibility into your work, which can turn into a nasty confrontation, particularly depending on the management you find yourself dealing with (on both sides).

It’s in that situation that I came up with “diagonal contributions”: improve a pain point for all the services you own, and cover as many services you can. In a similar fashion to rake collection, this is not an easy balance to strike, and it takes experience to have it done right. You can imagine from the previous post that my success at working on this diagonal has varied considerably depending on teams, time, and management.

What did work for me, was finding some common pain points between the six products I supported, and trying to address those not with changes to the products, but with changes to the core libraries they used or the common services they relied upon. This allowed me to show actual progress to the product teams, while solving issues that were common to most of the teams in my area, or even in the company.

It’s a similar thing with rake collection for me: say there’s a process you need to follow that takes two to three days to go through, and four out of your six teams are supposed to go through it — it’s worth it to invest four to six days to reduce the process to something that takes even just a couple of hours: you need fewer net people-days even just looking at the raw numbers, which is very easy to tell, but that’s not where it stops! A process that takes more than a day adds significant risks: something can happen overnight, the person going through the process might have to take a day off, or they might have a lot of meetings the following day, adding an extra day to the total, and so on.

This is also another reason why I enjoy this kind of work — as I said before, I disagree with Randall Munroe when it comes to automation. It’s not just a matter of saving time to do something trivial that you do rarely: automation is much less likely to make one-off mistakes (it’s terrifyingly good at making repeated mistakes of course), and even if it doesn’t take less time than a human would take, it doesn’t take human time to do stuff — so a three-days-long process that is completed by automation is still a better use of time than a two-days-long process that rely on a person having two consecutive days to work on it.

So building automation or tooling, or spending time making it easier to use core libraries, are in my books a good way to make contributions that are more valuable than just to your immediate team, while not letting your supported teams feel like they are being ignored. But this only works if you know which pain points your supported teams have, and you can make a case that your work directly relates to those pain points — I’ve seen situations where a team has been working on very valuable automation… that relieved no pain from the supported team, giving them a feeling of not being taken into consideration.

In addition to a good relationship with the supported team, there’s another thing that helps. Actually I would argue that it does more than just help, and is an absolute requirement: credibility. And management support. The former, in my experience, is a tricky one to understand (or accept) for many engineers, including me — that’s because often enough credibility in this space is related to the actions of your predecessors. Even when you’re supporting a new product team, it’s likely its members have had interactions with support teams (such as SRE) in the past, and those interactions will colour the initial impression of you and your team. This is even stronger when the product team was assigned a new team — or you’re a new member of a team, or you’re part of the “new generation” of a team that went through a bit of churn.

The way I have attacked that problem is by building up my credibility, by listening, and asking questions of what the problems the team feel are causing them issues are. Principles of reliability and best practices are not going to help a team that is struggling to find the time to work even on basic monitoring because they are under pressure to deliver something on time. Sometimes, you can take some of their load away, in a way that is sustainable for your own team, in a way that gains credibility, and that further the relationship. For instance you may be able to spend some time writing the metric-exposing code, with the understanding that the product team will expand it as they introduce new features.

The other factor as I said is management — this is another of those things that might bring a feeling of unfairness. I have encountered managers who seem more concerned about immediate results than the long-term pictures, and managers who appear afraid of suggesting projects that are not strictly within the scope of reliability, even when they would increase the team’s overall credibility. For this, I unfortunately don’t have a good answer. I found myself overall lucky with the selection of managers I have reported to, on average.

So for all of you out there in a position of supporting a product team, I hope this post helped giving you ideas of how to building a more effective, more healthy relationship.

Working in a bubble, contributing outside of it

The holiday season is usually a great time for personal projects, particularly for people like me who don’t go back “home” with “the family” — quotes needed, since for me home is where I am (London) and where my family is (me and my wife.) Work tends to be more relaxed – even with the added pressure of completing the OKRs for the quarter, and to define those for the next – and given that there is no public transport going on, the time saved in commuting also adds up to an ideal time to work on hobbies.

Unfortunately, this year I’m feeling pretty useless on this front, and I thought this uselessness feeling is at least something I can talk about for the dozen-or-so remaining readers of this blog, in an era of social media and YouTube videos. If this sounds very dismissive, it’s probably because that is the feeling of irrelevancy that took over me, and something that I should probably aim to overcome in 2020, one way or another.

If you are reading this post, it’s likely that you noticed my FLOSS contributions waning and pretty much disappearing over the past few years, except for my work around glucometerutils, and the usbmon-tools package (that kind-of derives off it.) I have contributed the odd patch to the Linux kernel, and more recently to some of the Python typing tooling, but those are really drive-by contributions as I found time for.

Given some of the more recent Twitter threads on Google’s policies around open source contributions, you may wonder if it is related to that, and the answer is “not really”. Early on, I was granted an IARC approval for me to keep working on unpaper (which turned out possibly overkill), for the aforementioned glucometerutils, and for some code I wrote while reverse engineering my gaming mouse. More recently, I’ve leveraged the simplified patching policy, and granted approval for releasing both usbmon-tools and tanuga (although the latter is only released as a skeleton right now.)

So I have all the options, and all the opportunities, to contribute FLOSS projects while in employment of a big multinational Internet company. Why don’t I do that more, then? I think the answer is that I work in a bubble for most of the day, and when I try to contribute something on my spare time, I find myself missing the support structure that the bubble gives me.

I want to make clear here that I’m not saying that everything is better in the bubble. Just that the bubble is soft and warm enough that makes the world outside of it scary, sometimes annoying, but definitely more vast. And despite a number of sensible tools being available out there (and in many cases, better tools), it takes a significant investment in researching the right way to do something, to the point that I suffer from CBA syndrome.

The basic concepts are not generally new: people have talked out loud at conferences about the monorepo, my friend Dinah McNutt spoke and wrote at length about Rapid, the release system we use internally, and that drives the automatic releases, and so on. If you’re even more interested in the topic, this March the book Software Engineering at Google will be released by O’Reilly. I have not read it myself, but I have interacted on and off with two of the curators and I’m sure it’s going to be worth its weight in gold.

Some of the tools are also being released, even if sometimes in modified ways. But even when they are, the amount of integration you may have internally is lost when trying to use them outside. I have considered using Bazel for glucometerutils in the past — but in addition to be a fairly heavy dependency, there’s no easy way to reference most of the libraries that glucometerutils need. At the end of the day, it was not worth trying to use it, despite making my life easier by reducing the cognitive load of working on opensource projects in my personal time.

Possibly the main “support beam” of the bubble, though, is the opinionated platform, which can be seen from the outside in form of the style guides but extends further. To keep the examples related to glucometerutils, while the tests do use absl‘s parameterized class, they are written in a completely different style than I would do at work, and they feel wrong when it comes to importing the local copy of the module to test it. When I looked around to figure out what’s the best practice to write tests in Python, I could find literally dozens of blog posts, StackOverflow answers, documentation for testing frameworks, that all gave slightly different answers. In the bubble you have (pretty much) one way to write the basic test — and while people can be creative even within those guidelines, creativity is usually frown upon.

The same is true for release engineering. As I noted and linked above, all of the release grunt work is done by the Rapid tool in the bubble — and for the most part it’s automated. While there’s definitely more than one way to configure the tool, at least you know which tool to use. And while different teams have often differing opinions on those configurations, you can at least find the opinion of your team, or the closest team to you with an Opinion (with the capital O) and follow that — it might not be perfect for your use, but if it’s allowed it usually means it was reviewed and vouched for (or copy-pasted from something else that was.)

An inside joke from the Google bubble is that the documentation is always out of date and never to be trusted. Beside the unfairness of the joke to the great tech writers I had pleasure to work with, who are more than happy to make sure the documentation is not out of date (but need to know that’s the case, and most of them don’t find out until it’s too late), the truth is that at least we do have documentation for most processes and tools. The outside world has tons of documentation, and some of it is out of date, and it’s very hard to tell whether it’s still correct and valid.

Trying to figure out how to configure a CI/CD tool for a Python project on GitHub (or worse, trying to figure out how to make it release valid packages on PyPI!) still feels like going by the early 2000s HOWTOs, where you hope that the three years old description of the XFree86 configuration file is still matching the implementation (hint: it never did.) Lots of the tools are not easy to integrate, and opting into them takes energy (and sometimes money) — the end result of which is that despite me releasing usbmon-tools nearly a year ago, you still need an unreleased dependency, as the fix I needed for it is not present in any released version, and I haven’t dared bothering the author to ask for a new release yet.

It’s very possible that if I was not working in a bubble all of these issues wouldn’t be be big unknowns — probably if I spend a couple of weeks reviewing the various options for CI/CD I can come up with a good answer for setting up automated releases, and then I can go to the dependency’s author and say “Hey, can I set this up for you?” and that would solve my problem. But that is time I don’t really have, when we’re talking about hobby projects. So I end up opening up the editor in the Git repository I want to work on, add a dozen line or so of code to something I want to do, and figure out that I’m missing the tool, library, interface, opinion, document, procedure that I need, feel drained, and close the editor without having committed – let alone pushed – anything.

Can you run a brick and mortar store on Free Software?

I have written before about the CRM I wrote for a pizzeria and I am happy to see that even FSFE started looking into Free Software for SME. I also noted the needs for teams to develop healthy projects. Today I want to give an example of why I think these things are not as easy as most people expect them to be, and how many different moving parts exist that are required to align to make Free Software for SME.

As I’m no longer self-employed, and I have no intention of going back to be a MSP in my lifetime, what I’m writing here is more of a set of “homework pointers” if a community of SME-targeted Free Software projects would be formed.

I decided to focus in my thoughts on the need of a brink and mortar store (or high street store if you prefer), mostly because it has a subset of the requirements that I could think of, compared to a restaurant like the pizza place I actually worked with.

These notes are also probably a lot more scattered and incomplete than I would like, because I have only worked retail for a short while, between high school and the two miserable week of university, nearly fifteen years ago, in a bookstore to be precise.

For most of the people who have not worked retail, it might seem like the most important piece of software/hardware for a store is the till, because that is what they interact with most of the time. While the till systems (also called POS) are fairly important, as those are in direct contact with the customer, they are only the tip of the iceberg.

But let’s start with the POS: whether you plan on integrating them directly with a credit card terminal or not, right now there are a number of integrated hardware/software solution for these, that include a touchscreen to input the receipt components and a (usually thermal) printer for the receipts to be printed on, while sometimes allowing the client to be emailed the receipt instead. As far as I know, there’s no Free Software system for this. I do see an increasing number of Clover tills in Europe, and Square in the United States (but these are not the only ones).

The till software is more complicated than one would think, because in addition to the effects that the customers can see (select line items, print receipt, eventually take payment), it has to be able to keep track of the cash flow, whether it is in form of actual cash, or in the form of card payments. Knowing the cash flow is a requisite for any business, as without that information you cannot plan your budgets.

In bigger operations, this would feed into a dedicated ERP system, which would often include an inventory management software — because you need to know how much stock you have and how fast it is moving, to know when to order new stock.

There is also the need to handle invoices, which usually don’t get printed by the till (you don’t want an invoice printed on thermal paper, particularly in countries like Italy, where you’re meant to keep the original of an invoice for over ten years).

And then there is the filing of payable invoices and, well, their payment. This is part of the accounting procedures, and I know of very few systems that allow integration with a bank to the point of automating this part. PSD2 is meant to require financial institutions to provide APIs to make this possible, at least in Europe, but that has been barely received yet, and we’ll have to see what the solution will be.

Different industries have different expected standards, too. When I worked in the bookstore, there was a standard piece of software that was used to consult the online stock of books from various depots, which was required to handle orders of books for people looking for something that was not in the store. While Amazon and other online services have for the most part removed the need for many to custom order books in a store, I know still a few people who do so, simply to make sure the bookstore stays up. And I assume that very similar, yet different, software and systems exist for most other fields of endeavour, such as computer components, watches, and shoes.

Depending on the size of the store, and the amount of employees, and in general the hours of operation, there may also be need for a roster management software, so that the different workers have fair (and legal) shifts, while still being able to manage days off. I don’t know how well solutions like Workday work for small realities, but in general I feel this is likely going to be one area in which Free Software won’t make an easy dent: following all the possible legal frameworks to actually be compliant with the law is the kind of work that requires a full-time staff of people, and unless something changes drastically, I don’t expect any FLOSS project to keep up with that.

You can say that this post is not giving any answer and is just adding more questions. And that’s the case, actually. I don’t have the time or energy of working on this myself, and my job does not involve working with retailers, or even developing user-focused software. I wanted to write this as a starting point of a project if someone is interested in doing so.

In particular, I think that this would be prime territory for a multi-disciplinary university project, starting from asking questions to store owners of their need, and understanding the whole user journey. Which seems to be something that FSFE is now looking into fostering, which I’m very happy about.

Please, help the answer to the question “Can you run a brink and mortar store on Free Software?” be Yes!

The importance of teams, and teamwork

Today, on Twitter, I have received a reply with a phrase that, in its own sake and without connecting back with the original topic of the thread, I found significant of the dread I feel with working as a developer, particularly in many opensource communities nowadays.

Most things don’t work the way I think they work. That’s why I’m a programmer, so I can make them work the way I think they should work.

I’m not going to link back to the tweet, or name the author of the phrase. This is not about them in particular, and more about the feeling expressed in this phrase, which I would have agreed with many years ago, but now feels so much off key.

What I feel now is that programmers don’t make things work the way they think they should. And this is not intended as a nod to the various jokes about how bad programming actually is, given APIs and constraints. This is about something that becomes clear when you spend your time trying to change the world, or make a living alone (by running your own company): everybody needs help, in the form of a team.

A lone programmer may be able to write a whole operating system (cough Emacs), but that does not make it a success in and by itself. If you plan on changing the world, and possibly changing it for the better, you need a team that includes not only programmers, but experts in quite a lot of different things.

Whether it is a Free Software project, or a commercial product, if you want to have users, you need to know what they want — and a programmer is not always the most suitable person to go through user stories. Hands up all of us who have, at one point or another, facepalmed at an acquaintance taking a screenshot of a web page to paste it into Word, and tried to teach them how to print the page to PDF. While changing workflows so that they make sense may sound the easiest solution to most tech people, that’s not what people who are trying to just do their job care about. Particularly not if you’re trying to sell them (literally or figuratively) a new product.

And similarly to what users want to do, you need to know what the users need to do. While effectively all of Free Software comes with no warranty attached, even for it (and most definitely for commercial products), it’s important to consider the legal framework the software has to be used on. Except for the more anarchists of the developers out there, I don’t think anyone would feel particularly interested in breaching laws for the sake of breaching them, for instance by providing a ledger product that allows “black book accounting” as an encrypted parallel file. Or, to reprise my recent example, to provide a software solution that does not comply with GDPR.

This is not just about pure software products. You may remember, from last year, the teardown of Juicero. In this case the problems appeared to step by the lack of control over the BOM. While electronics is by far not my speciality, I have heard more expert friends and colleagues cringe at seeing the spec of projects that tried to actually become mainstream, with a BOM easily twice as expensive as the minimum.

Aside here, before someone starts shouting about that. Minimising the BOM for an electronic project may not always be the main target. If it’s a DIY project, making it easier to assemble could be an objective, so choosing more bulky, more expensive parts might be warranted. Similarly if it’s being done for prototyping, using more expensive but widely available components is generally a win too. I have worked on devices that used multi-GB SSDs for a firmware less than 64MB — but asking for on-board flash for the firmware would have costed more than the extremely overprovisioned SSDs.

And in my opinion, if you want to have your own company, and are in for the long run (i.e. not with startup mentality of getting VC capital and get acquired before even shipping), you definitely need someone to follow up the business plan and the accounting.

So no, I don’t think that any one programmer, or a group of sole programmers, can change the world. There’s a lot more than writing code, to build software. And a lot more than building software, to change society.

Consider this the reason why I will plonk-file any recruitment email that is looking for “rockstars” or “ninjas”. Not that I’m looking for a new gig as I type this, but I would at least give thought if someone was looking for a software mechanic (h/t @sysadmin1138).

We Need More Practical, Business-Oriented Open Source — Case Study: The Pizzeria CRM

I’m an open source developer, because I think that open source makes for safer, better software for the whole community of users. I also think that, by making more software available to a wider audience, we improve the quality, safety and security of every user out there, and as such I will always push for more, and more open, software. This is why I support the Public Money, Public Code campaign by the FSFE for opening up the software developed explicitly for public administrations.

But there is one space that I found is quite lacking when it comes with open source: business-oriented software. The first obvious thing is the lack of good accounting software, as Jonathan has written extensively about, but there is more. When I was consulting as a roaming sysadmin (or with a more buzzwordy, and marketing-friendly term, a Managed Services Provider — MSP), a number of my customers relied heavily on nearly off-the-shelf software to actually run their business. And in at least a couple of cases, they commissioned me custom-tailored software for that.

In a lot of cases, there isn’t really a good reason not to open-source this software: while it is required to run certain businesses, it is clearly not enough to run them. And yet there are very few examples of such software in the open, and that includes from me: my customers didn’t really like the idea of releasing the software to others, even after I offered a discount on the development price.

I want to show the details of an example of one such custom software, something that, to give a name to it, would be a CRM (Customer Relationship Manager), that I built for a pizzeria in Italy. I won’t be opening the source code for it (though I wish I could do so), and I won’t be showing screenshots or provide the name of the actual place, instead referring to it as Pizza Planet.

This CRM (although the name sounds more professional than what it really was), was custom-designed to suit the work environment of the pizzeria — that is to say, I did whatever they asked me, despite it disagreeing with my sense of aesthetics and engineering. The basic idea was very simple: when a customer calls, they wanted to know who the customer was even before picking up the phone — effectively inspecting the caller ID, and connecting it with the easiest database editing facility I could write, so that they could give it a name and a freeform text box to write down addresses, notes, and preferences.

The reason why they called me to write this is that they originally bought a hardware PBX (for a single room pizzeria!) just so that a laptop could connect to it and use the Address Book functionality of the vendor. Except this functionality kept crashing, and after many weeks of back-and-forth with the headquarters in Japan, the integrator could not figure out how to get it to work.

As the pizzeria was wired with ISDN (legacy technology, heh), to be able to take at least two calls at the same time, the solution I came up with was building a simple “industrial” PC, with an ISDN line card and Asterisk, get them a standard SIP phone, and write the “CRM” so that it would initiate a SIP connection to the same Asterisk server (but never answer it). Once an inbound call arrived, it would look up if there was an entry in a simple storage layer for the phone number and display it with very large fonts, to be easily readable while around the kitchen.

As things moved and changed, a second pizzeria was opened and it required a similar setup. Except that, as ISDN are legacy technology, the provider was going to charge up to the nose for connecting a new line. Instead we decided to set up a VoIP account instead, and instead of a PC to connect the software, Asterisk ran on a server (in close proximity to the VoIP provider). And since at that point the limitation of an ISDN line on open calls is limited, the scope of the project expanded.

First of all, up to four calls could be queued, “your call is very important to us”-style. We briefly discussed allowing for reserving a spot and calling back, but at the time calls to mobile phones would still be expensive enough they wanted to avoid that. Instead the calls would get a simple message telling them to wait in line to contact the pizzeria. The CRM started showing the length of the queue (in a very clunky way), although it never showed the “next call” like the customer wanted (the relationship between the customer and the VoIP provider went South, and all of us had to end up withdrawing from the engagement).

Another feature we ended up implementing was opening hours: when call would arrive outside of the advertised opening hours, an announcement would play (recorded by a paid friend, who used to act in theatre, and thus had a good pronunciation).

I’m fairly sure that none of this would actually comply with the new GDPR requirements. At the very least, the customers should be advised that their data (phone number, address) will be saved.

But why am I talking about this in the context of Open Source software? Well, while a lot of the components used in this set up were open source, or even Free Software, it still requires a lot of integration to become usable. There’s no “turnkey pizzeria setup” — you can build up the system from components, but you need not just an integrator, you need a full developer (or development team) to make sure all the components fit together.

I honestly wish I had opensourced more of this. If I was to design this again right now, I would probably make sure that there was a direct, real-time API between Asterisk and a Web-based CRM. It would definitely make it easier to secure the data for GDPR compliance. But there is more than just that: having an actual integrated, isolated system where you can make configuration changes give the user (customer) the ability to set up things without having to know how the configuration files are structured.

To set up the Asterisk, it took me a week or two reading through documentation, books on the topic, and a significant amount of experimentation with a VoIP number and a battery of testing SIM cards at home. To make the recordings work I had to fight with converting the files to G.729 beforehand, or the playback would use a significant amount of CPU.

But these are not unknown needs. There are plenty of restaurants (who don’t have to be pizza places) out there that probably need something like this. And indeed services such as Deliveroo appear to now provide a similar all-in-one solution… which is good for restaurants in cities big enough to sustain Deliveroo, but probably not grate for the smaller restaurants in smaller cities, who probably would not have much of a chance of hiring developers to make such a system themselves.

So, rambling asides, I really wish we had more ready-to-install Open Source solutions for businesses (restaurants, hotels, … — I would like to add banks to that but I know regulatory compliance is hard). I think these would actually have a very good social impact on all those towns and cities that don’t have a critical mass of tech influence, that they come with their own collection of mobile apps, for instance.

If you’re the kind of person who complains that startups only appear to want to solve problems in San Francisco, maybe think of what problems you can solve in and around your town or city.

Inside the life of an embedded developer

Now that the new ultrabook is working almost fine (I’m still tweaking the kernel to make sure it works as intended, especially for what concerns the brightness, and tweaking the power usage, for once, to give me some more power), I think it’s time for me to get used to the new keyboard, which is definitely different from the one I’m used to on the Dell, and more similar to the one I’ve used on the Mac instead. To do so, I think I’ll resume writing a lot on different things that are not strictly related to what I’m doing at the moment but talk about random topic I know about. Here’s the first of this series.

So, you probably all know me for a Gentoo Linux developer, some of you will know me for a multimedia-related developer thanks to my work on xine, libav, and a few other projects; a few would know me as a Ruby and Rails developer, but that’s not something I’m usually boasting or happy about. In general, I think that I’m doing my best not to work on Rails if I can. One of the things I’ve done over the years has been working on a few embedded projects, a couple of which has been Linux related. Despite what some people thought of me before, I’m not an electrical engineer, so I’m afraid I haven’t had the hands-on experience that many other people I know had, which I have to say, sometimes bother me a bit.

Anyway, the projects I worked on over time have never been huge project, or very well funded, or managed projects, which meant that despite Greg’s hope as expressed on the gentoo-dev mailing list, I never had a company train me in the legal landscape of Free Software licenses. Actually in most of the cases, I ended up being the one that had to explain to my manager how these licenses interact. And while I take licenses very seriously, and I do my best to make sure that they are respected, even when the task falls on the shoulder of someone else, and that someone might not actually do everything by the book.

So, while I’m not a lawyer and I really don’t want to go near that option, I always take the chance of understand licenses correctly and, when my ass is on the line, I make sure to verify the licenses of what I’m using for my job. One such example is the project I’ve been working since last March, of which I’ve prepared the firmware, based on Gentoo Linux. To make sure that all the licenses were respected properly, I had to come up with the list of packages that the firmware is composed of, and then verify the license of each. Doing this I ended up finding a problem with PulseAudio due to linking in the (GPL-only) GDBM library.

What happens is that if the company you’re working for has no experience with licenses, and/or does not want to pay to involve lawyers to review what is being done, it’s extremely easy for mistake to happen unless you are very careful. And in many cases, if you, as a developer, pay more attention to the licenses than your manager does, it’s also seen as a negative point, as that’s not how they’d like for you to employ your time. Of course you can say that you shouldn’t be working for that company then, but sometimes it’s not like you have tons of options.

But this is by far not the only problem. Sometimes, what happens is a classic 801 — that is that instead of writing custom code for the embedded platform you’re developing for, the company wants you to re-use previous code, which has a high likelihood of being written in a language that is completely unsuitable for the embedded world: C++, Java, COBOL, PHP….

Speaking of which, here’s an anecdote from my previous experiences in the field: at one point I was working on the firmware for an ARM7 device, that had to run an application written in Java. Said application was originally written to use PostgreSQL and Tomcat, with a full-fledged browser, but had to run on a tiny resistive display with SQLite. But since at the time IcedTea was nowhere to be found, and the device wouldn’t have had enough memory for it anyway, the original implementation used a slightly patched GCJ to build the application to ELF, and used JNI hooks to link to SQLite. The time (and money, when my time was involved) spent making sure the system wouldn’t run out of memory would probably have sufficed to rewrite the whole thing in C. And before you ask, the code bases between the Tomcat and the GCJ versions started drifting almost immediately, so code sharing was not a good option anyway.

Now, to finish this mostly pointless, anecdotal post of mine, I’d like to write a few words of commentary about embedded systems, systemd, and openrc. Whenever I head one or the other side saying that embedded people love the other, I think they don’t know how different embedded development can be from one company to the next, and even between good companies there are so big variation, that make them stand lightyears apart from bad companies like some of the ones I described above. Both sides have good points for the embedded world; what you choose depends vastly on what you’re doing.

If memory and timing are your highest constraints, then it’s very likely that you’re looking into systemd. If you don’t have those kind of constraints, but you’re building a re-usable or highly customizable platform, it’s likely you’re not going to choose it. The reason? While if you’re half-decent in your development cross-compilation shouldn’t be a problem, the reality is that in many place, it is. What happen then is that you want to be able to make last-minute changes, especially in the boot process, for debugging purposes, using shell scripts is vastly easier, and for some people, doing it more easily is the target, rather than doing it right (and this is far from saying that I find the whole set of systemd ideas and designs “right”).

But this is a discussion that is more of a flamebait than anything else, and if I want to go into details I should probably spend more time on it than I’m comfortable doing now. In general, the only thing I’m afraid of, is that too many people make assumption on how people do things, or take for granted that companies, big and small, care about doing things “right” — by my experience, that’s not really the case, that often.

FreeRADIUS and 802.1x authentication

Sometimes my work require me to do stuff that is so interesting I work overtime without actually telling anybody to make it work better, like I’ve done for Munin and Icinga — most of the time, though, my work is just boring, repetitive, but that’s okay, excitement goes a long way to ruin a person’s health. Especially when said excitement hits you on the jawbone with a network where Ethernet ports are authenticated with 802.1x…

You might not know it but there is a way to authenticate clients more or less the same way you do on WiFi when you use actual wired Ethernet. This is done through 802.1x and RADIUS. What is RADIUS? Well, it’s basically an authentication and accounting protocol, which I guess was originally developed for dial-up Internet access… remember those 56k modem days? No? Sigh, I feel old now. At any rate, for that reason, you’ll find the FreeRADIUS server in net-dialup/freeradius as of this moment .. it really should be moved to sys-auth/freeradius-server but I don’t want to bother with that right now.

So what happens during 802.1x is simple: the switch act as a proxy between the client and the RADIUS server (called authenticator), passing through the authentication messages, most of the case EAP-based. Until the authentication is passed, all other packets sent over the network are simply dropped. Now depending on how you set up the network and the capability of your switch, you can make it so that if authentication does not happen in a given time you go to a guest VLAN, or you just keep dropping packets waiting for authentication. Unfortunately if you go with default DHCP configuration, with the default timeouts, it’s likely that you won’t get any network, which is the problem we hit with our device (comparatively, OSX had the same issue, and you had to renew the DHCP lease after about a minute of connecting the Ethernet cable).

So I bought the cheapest 802.1x capable switch I could find on Amazon (an eight ports Cisco SG-200, if you’re interested) and started setting up to simulate an 802.1x network — this switch does not support the guest vlan as far as I can tell, which is actually what bothers me quite a bit, but on the whole it looks okay for our testing. I actually found out after the whole work was done, that it was technically possible to authenticate against a local database instead of having to deal with an external authenticator, but I’ve not been able to get it running that way, so it’s okay.

For what concerns the main topic of this discussion from the Gentoo point of view, I was quite lucky actually; there is good documentation for it on nothing less than TLDP — it’s a 2004 howto but it’s still almost perfect. The only difference in syntax for FreeRADIUS’s configuration is the way the password is designed in the users configuration file. I’ve not checked the client-side configuration of course, since that is probably completely out of date nowadays thanks to WPA Supplicant and NetworkManager.

The big hurdle was getting FreeRADIUS in a decent shape: simply emerging it and trying to start it would get it to fail silently, so I had to kick it into submission, hard. It so happened that a new version of the server was just released in September, so I decided to update to that one version and get it working as a proper ebuild. The new ebuild in tree should work quite nicely, the problem is that if you look at it, it’s hideous. The big problem is that their build system is a complete fsckup, and you have to resolve to deleting whole subdirectories to configure the features to build, and it has quite a few of them as it has over half a dozen database backends, and so many integrations that it’s really not funny to deal with those optional dependencies.

If you used the older ebuilds, or compare the new one to the old ones you can probably notice that I dropped a number of USE flags, especially those that were so specific to FreeRADIUS that they had a fr- prefix. This is because I’ve followed my usual general idea, that USE flags are great, if you’re turning on/off features that are heavy, or that have external dependencies, but if you just enable/disable codepaths that are barely noticeable, they just add to the noise. For this reason there is now only one local USE flag for pcap (which is actually a global flag with a more detailed description).

Also, when you build with SSL (which you want to do when doing 802.1x!) you need a CA to sign the users’ certificates. While you can set up your own CA relatively easily, like you already do for OpenVPN, I’ve made it much easier by wiring the originally-provided script to the --config option for the package (so you just need to run emerge --config freeradius for it to work).

As I said, the build system is extremely bad, to the point that they are actually committing to their GIT repository the whole autotools-generated files, which is not a good idea. At least this time around I was able to free up the files directory as all the patches are handled as tarballed patchsets on my devspace; if you want to see the patches in a more friendly way, I also got a copy of the repository on Gentoo’s GitHub account — you can also find a number of other projects that I patched the same way, including Munin.

Due to security issues, the new version of FreeRADIUS I put in tree is now stable on the two arches that were stable before, and all the old versions are gone, together with their patches (it cleaned up nicely) for the love for our rsyncs. Hopefully that doesn’t screw with anybody’s plans — if somebody has a problem with my changes, feel free to prod me.

License auditing your code

I’ve already said that I’m working on a new device, whose firmware is Gentoo-based, and it goes easily said that it’s a partly-closed source software. That’s just the way it is: while probably the most part of the software within the device is Free Software, the business logic is behind closed doors. People are getting used to this, and I don’t think it’s entirely a bad thing. I mean, we’re giving back to Free Software in this context in many ways: Luca is working on libav and Aravis me I’m working on Linux drivers and together we’re working on Gentoo so the environment as a whole is gaining something.

Of course, when you are dealing with this kind of devices, you have to take care of auditing the licenses of the software you’re bundling up in the firmware, which is what I’m doing today (well, yesterday for you who read me I guess). It’s not my first time at this game, and as usual my starting point is an UML package diagram.

For those not used to UML, a package diagram is a decent way to identify who makes use of what; thanks to the way the UML is specified you can use two different “stereotypes”, called import and access which show you the execution/linking boundary quite clearly. By giving each project a package, and each library within that package a subpackage, you can easily see how things are connected.

So while working through it, with the two objectives to both reduce the amount of software we had to install (I talked about that yesterday), and to verify that we don’t distribute our closed-source code linked to GPL libraries, I started noticing a few bad things; from one side the license identification in Gentoo is shabby, but that’s nothing new, as I write this, I’m fixing a few ebuilds that report the wrong license information, for instance; from the other side, we have packages like PulseAudio that does not let you understand their licensing in a very clear way.

In the case of PulseAudio, the LICENSE file tells you this:

All PulseAudio source files are licensed under the GNU Lesser General Public License. (see file LGPL for details)
However, the server side has optional GPL dependencies. These include the libsamplerate and gdbm (core libraries), LIRC (lirc module), FFTW (equalizer module) and bluez (bluetooth proximity helper program) libraries, although others may also be included in the future. If PulseAudio is compiled with these optional components, this effectively downgrades the license of the server part to GPL (see the file GPL for details), exercising section 3 of the LGPL. In such circumstances, you should treat the client library (libpulse) of PulseAudio as being LGPL licensed and the server part (libpulsecore) as being GPL licensed. Since the PulseAudio daemon, tests, various utilities/helpers and the modules link to libpulsecore and/or the afore mentioned optional GPL dependencies they are of course also GPL licensed also in this scenario.
[…]

Is this clear to you? It should be: libpulse is the library you implement a PulseAudio client with, and libpulsecore used to be the convenience library only used by the server… but in PulseAudio’s history, this hasn’t been the case for quite a while, with the result that libpulse requires libpulsecore, and that means that if you link GDBM into PulseAudio’s core library … you now have GPL’d libpulse.

This is not the case for all the libraries it uses though: for instance BlueZ is not loaded into the core library so you still only have a PulseAudio daemon GPL’d and not the libraries, as intended.

What’s the catch about this? Well, turns out that Nokia knew about this for a while, since they did contribute a “simple” database in alternative to GDBM (GPL-2) and TDB (GPL-3), which is fine for most embedded usage, if not for desktops — which is exactly what I need here.. of course the ebuilds still force GDBM enabled. I’m fixing that as well.

I’m leaving for later fixing the license specification for other USE flags, it’s a time constraint for now.

I guess that every time I do this I understand how difficult license auditing is, and why people don’t like having multi-license projects or even multiple licenses doing almost the same thing. Oh well.

How down can you strip a Gentoo system?

In my previous post I noted I’m working on a project here in Los Angeles, of which I don’t wan to get much into the details of. On the other hand, what I’m going to tell you about this, for now, is that it’s a device and it’s running Gentoo as part of its firmware.

You can also guess, if you know us, that since both me and Luca are involved, there is some kind of multimedia work going on.

I came here to strip down as much as possible the production firmware so that the image could be as small as possible, and still allow all the features we need on the device. This is not a new task for me, as I’ve done my best to strip down my own router so that it would require the least amount of space as possible, and I’ve also done some embedded firmwares based on Gentoo before.

The first obstacle you have if you want to reduce the size of Gentoo is almost certainly the set of init scripts that come with OpenRC; for a number of reasons, the init scripts for things like loadkeys and hwclock are not installed by the package that install the commands (respectively sys-apps/kbd and sys-apps/util-linux) but rather are installed by OpenRC itself. They are also both enabled by default, which is okay for a general desktop system but fails badly on embedded systems, especially when they don’t even have a clock.

Then you have to deal with the insane amount of packages that form our base system; without going into the details of having man, tar and so on as part of the base untouchable system (which luckily is much easier to override with the new Portage 2.2, even if it insists bothering you about an overridden system set), and focusing on what you’re going to use to boot the system, OpenRC is currently requiring you a mixture of packages including coreutils, which (a single command that lies in its own package … for ESR’s sake, why was it not implemented within coreutils yet?), grep, findutils, gawk and sed (seriously, four packages for these? I mean I know they are more widely used than coreutils, as they are often used on non-Linux operating systems, but do they really deserve their own package, each of them?).

The most irritating part nowadays for me I guess is the psmisc vs procps battle: with the latter now maintained by Debian, I wonder why they haven’t merged the former yet. Given that they are implementing utilities for the same areas of the system… of course one could wonder why they are not all part of util-linux anyway — yes I know that Debian is also supporting GNU/kFreeBSD on their package. At any rate there is another consideration to be made: only the newer procps allows you to drop support for the ncurses library, earlier depended on it forcefully, and the same is still true for psmisc.

For what it’s worth, what I decided to do was to replace as much as possible with just busybox, including the troublesome psmisc, so that I could drop ncurses from our firmware altogether — interestingly enough, OpenRC depends explicitly on psmisc even though it is not bringing in most of the rest of its dependencies.

Public Service Announcement: if you’re writing an init script and you’re tempted to use which, please ~~replace it with type -p~~ use command -v instead … depending on an extra program when sh already has its built-in is bad, ‘mkay?

Edit: people made me notice that type -p is not in POSIX so it does not work in Dash. I’m afraid that my only trials to run OpenRC without bash before have used BusyBox, which supports it quite fine; the option to use command -v is valid though, thanks Tim.

Oh right, of course to replace coreutils usage with BusyBox you have to be able to drop it out of the dependency tree. Sounds easy, doesn’t it? Well, the problem is that even if you’re not deal with PulseAudio (which we are), which brings in eselect-esd, as of yesterday at least every package that could use Python would bring eselect-python in! Even when you were setting USE=-python.

Fortunately, after I bitched a bit about it to Luca, he made a change which at least solves the issue at hand until the stupid eclass is gone out of the tree. And yes I’m no longer even trying to consider it sane, the code is just so hairy and so encrypted that you can’t make heads or tails of it.

There are more issues with a project like this that I can discuss, outside of those part that are “trade secret” (or rather business logic), so expect to hear a bit more about it before it’s announced full out. And many of these have to do with how easy (or not) is to use Gentoo as a basis for devices’ firmwares.