Reflecting On Free Software And Paychecks

Necessary context: this post was written well before the whole FLOSS world started talking about funding due to the log4j vulnerability earlier this month. So while it is topical to a point, it won’t be answering for some of the questions that would come out from it. I did attempt to provide some related nuance before posting this, but honestly it wasn’t a well-thought-out piece to begin with. If you want a more specific view of the issue with log4j in mind, refer to Filippo’s post, which is better written, and on topic.

The following post is going to be difficult to write, and also to read. It’s mainly a personal reflection, but because our culture makes talking about money a difficult topic, things are not as easy or straightforward as they should be when it comes to the professional life of Free Software contributors — a lot of what I’m going to be talking about may be more relevant to code contributors, but that’s because it’s the experience I have more at hand, I expect that most of the reflections would apply to other roles, as well. So please bear with me, in what is going to be a bit of a journey, and a bit rambly at times.

By now you probably know that, at the beginning of the pandemic crisis, I took on a new job as a senior engineer. Both this and my previous job were in the Big Tech Bubble with connected privileges, paychecks, and ethical dilemmas. I’m going to explore a little bit of the context around this, which obviously means what you will read is just my opinion, and does not reflect that of my employers current, past, and future.

The reason why I even went to this reflection, is noticing just how much more Free Software work I used to do in the past, but also how much harder that life was. I looked around the Tinderbox posts and I found so many, many posts in which I was basically begging for help — with mixed results. Nowadays instead I have barely any time for Free Software (though I’m not completely gone from the scene), but I’m definitely “well off.” So what changed?

Like many other Free Software maintainers, I ended up being hired in a full time position — you could say that I was hired because of my visibility as a Free Software maintainer, but my dayjob had little to nothing to do with it. Indeed, as I wrote last year, it’s harder to contribute code, often requiring projects explicitly take steps in the direction of making it easier for the people working in Big Tech… and for many that’s the absolutely last thing on their list.

Would I have been able to work full time for a company that actually cared about my Free Software contributions? Possibly, but that’s a bit of a stretch: I’m aware only of a handful of companies that cared about Gentoo Linux in a professional fashion, which means I would have rather had to be hired by a company that would be interested in my Free Software relationships, but not to the actual project. This should explain why a number of Gentoo Linux developers ended up being hired by SUSE, for instance.

This is, in my opinion, the most important key information to take away not just from my experience, but when looking around the people that spend time in the community: only a small handful of people succeed in being paid to do what they already do for free — a larger number of people stay active in the community, but might rather focus on something that aligns more clearly with someone’s business needs. For instance, at some point I was being offered to interview for a position of PulseAudio maintainer at Canonical. You can see from the blog posts, that I have spent a lot of time and energy on that project already by then, so it would have been a decent fit… but I didn’t pursue it back then. I sometimes regret it a bit, thinking that maybe I would have at least got in a place to influence paying off the technical debt that is Linux audio, but that does not appear to be likely. Things are obviously different for maintainers of project that were started while already working at an open source company — particularly if the project is meant to be addressing a need for customers, even better if paying customers.

So why do so many people end up “drained” from the Free Software community into Big Tech, either with big established brands or startups? Well the answer is obvious: money. Much as one’s opinion can be about the capitalistic world we live in, and much as “money doesn’t buy happiness”, lack thereof definitely becomes an issue. And this is where things get a bit more complicated to discuss, because in a perfect world the person writing this would be a sociologist with enough information to not just make a generalization out of anecdata but rather a proper study.

See, my impression is that the vast majority of the contributors that I have seen explicitly rejecting the offers of Big Tech, despite being offered, are people who come for a more comfortable, privileged background, or those who might have had access to less than crystal clear funding. Maybe not in the most obvious of ways, but most of the people who grew up in families “tight around the belt” would know the importance of making a stable paycheck at the end of the month. You don’t need to be properly poor for that — it might be enough to know you can’t spend ten years trying to get a university degree.

To be “successful” in the Free Software community has always required some level of privilege: not everybody can be dedicating time without being paid for it “for exposure.” Particularly before things like Patreon or GitHub Sponsors arrived, most Free Software developers were not making a dime of their work at all — as I said, I begged for help many times when I was running the Gentoo Tinderbox, and indeed I had a significant help from other members of the community… but that help alone wouldn’t have let me continue being a Free Software developer full time.

But being successful is not just about having the time, or the programming skills: the success of a Free Software project depends on a lot of factors, many of which come down to building relationships to expand the project itself beyond the single person. Building connections and relationship online is one thing, but I would argue that a significant amount of those connections always required a certain physical presence — which is why in the past two years we’ve all been missing CCC, FOSDEM, and all the various similar venues. Being able to attend these is not just a matter of having the financial means to pay for the tickets, it also requires feeling safe to attend those, which is a lot less inclusive than people expect.

There’s also a timing argument to be found here: it is true that many of the early projects ended up being successful despite this lack of collaboration. After all, Linux itself only started as an amateur project, not professional like Minix, right? But then we’re talking about specific situations that are much harder to reproduce nowadays. Being an early mover in the field obviously provides advantages — but itself is a form of privilege, in so far as not everyone will ever have the same opportunities: being online in 1993 to be able to join in the early fun of creating what became the most widely used kernel in the world is not something everyone could have done, both because Internet connections weren’t cheap, and because, let’s be honest, in 1993 I was eight years old, and the best English I could muster was barely enough to play Castle of the Winds.

And as for something that is not limited to Free Software communities, but also applies to most tech companies, there appear to be a higher bar for new members than for those who created projects earlier on. No matter the importance of the ideology behind him, nobody would accept RMS as a community leader if he showed up for the first time in 2021 — and even sticking to technical merits, a project like bash wouldn’t be widely adopted nowadays without a public source control and bug tracker! As I said that happens in companies as well — I definitely have seen tenured engineers being allowed to bend, if not breach, policies that would have put a junior engineer in trouble with HR, but that’s for another time.

What I’m trying to point out here, is that scorning community members that decide to take a stable paycheck is not nice. Indeed, a number of people might have joined the Free Software community in the first place because they hoped to “score” such an opportunity, and I wouldn’t blame them for it, but too often I have read disdainful commentary about students joining Free Software communities only to have something to show off at tech interviews. Here’s a not-so-secret background about me: I spent a long time looking at Solaris because I was actually hoping I would be able to go and work for Sun! Except that they collapsed well before I would have had the opportunity.

I also want to make clear that I’m not saying this with the cynicism needed to just absolve myself from wrongdoing, or to rationalize why my Free Software projects never went anywhere. My projects didn’t go anywhere because I lacked (and still lack) the leadership skills needed to bring people together towards a cohesive vision. I find myself having ideas that actually would turn out to work out well, but I have rarely managed to produce anything workable out of it — Gentoo/FreeBSD is probably the only project that I managed to actually push through, for a while, and even that didn’t end up blooming as much as it should have if I had the right set of skills.

But what about the random person in Nebraska that so often comes out as an argument about Big Tech funding? Well, I’m sure there’s plenty of those, both in Free Software and within companies. While I was in Google I kind-of was that. because for a number of months, if not years, I maintained one of the most used internal Python libraries, which probably more than half of the company depended on, despite it not being anywhere close to my expected duties. Attempts to make it my duty failed badly, and indeed the mere fact that I was spending time on that was a controversial point with my managers.

In Big Tech, this kind of behaviour is often seen as “heroism”, and it’s considered bad because it is hiding a broken system from the powers that can fix it. It might not sound fair, but at that scale, it’s obvious that if you need oil, you need to let the wheel squeak. But in Free Software, and in any other community, this situation is not seen as the optimal one at all. In a company, a component left unattended either works fine (in which case there’s no reason to pay for maintenance of it), or causes problems, big or small that they are. If the problem is that the component is hard to use, eventually the velocity of the engineering team will become enough reason to staff the component for maintenance or replacement. If the component causes a visible enough outage, then it will also be squeaking louder enough to get the required lubrication — or a replacement.

I keep mentioning replacement here because there’s a common misconception that the solution to all problems with understaffed projects is to staff the project as they are. If a project within a company got to the point of needing a hero to fix it, it’s usually a sign of the project itself being unhealthy: it might be using an old technology stack that most newer people don’t know how to handle, it might be built upon assumptions that no longer hold true, or it might “just” have been written in a way that is overcomplicated and nobody has the knowledge to understand it now. This makes the project a liability both ways: if you staff it and then the new maintainers don’t understand it well enough, they may break stuff in subtle ways that will be hard to rectify later on; if you rewrite it without understanding the underlying requirements you may end up having to rediscover all of them as your system starts to fail. Personally I’m a fan of shrink-and-replace by removing as many unused, or rarely used, features as possible, and migrating them to new solutions.

This approach does not work well for Free Software projects, though. First of all, there’s a lot more “ego” in a Free Software project, particularly if it’s started by one’s own need rather than through the demands of a business organization. Suggesting the maintainer to step away and “let it squeak” is unlikely to be a suggestion that would win many friends. But it also shows up in figuring out how to assign resources to these projects: if an opensource project is not needing active maintenance because it just works most of the time, there’s no incentive for users of it (whether they are end users or businesses) to go ahead and hire the maintainer full time. If the project is broken, or needs a significant rework, then it might be easier to build something that does not rely on one person to begin with.

When maintaining core libraries that people are already using successfully, things are even more complicated. Within a bubble you can find all the users of a certain library, and you can come up with plans on how to improve the library by solving real problems faced by its users. Do this enough times, and you may be able to provide your employer with reasons to keep you full time on its maintenance. With a Free Software project? Well, you may be able to see a number of projects using your library, but unless you know how to unbias the sample, you may end up focusing on something that does not quite work. Or you may only focus on those projects who can explicitly pay you to build features, which would be just as bad.

This should explain why so many projects end up being developed in a flurry of enthusiasm, and then either become “stale” with no changes happening to them for a long time, or keep creeping up in scope by adding features, building new interfaces, or chasing some recent trend. Both of these end up turning into competing project deciding to start anew, either by rewriting it in modern standards (if it was left with no changes for so long) or minimized designs (if it chased so many features that it’s now an overcomplicated ball of spaghetti code.) Neither of these are things that make for a particularly fun community discussion.

So am I suggesting that the options are to work for a big company and stop maintaining Free Software or starve? Not at all! Indeed, I think there’s plenty of examples of the opposite, either via sponsorships (like Asahi Linux), or via small(er) companies like Nabu Casa providing value-add services for projects that are otherwise Free Software. But since these rely on end-user customers paying for services or sponsorships, successful subscriptions and sponsorships tend to be limited to projects that have a direct impact on users. Hector’s work on promoting Asahi as building a smooth path for Linux on Apple Silicon is extraordinary for such a low-level (in terms of distance to hardware) problem! What I am suggesting is that we shouldn’t be buying the argument that Free Software is a vocation and there’s no need to talk about money, making a living, or generally paying the contributors!

Indeed, when I think of most Free Software maintainers that are paid to maintain it full time, I think of people working for companies that started small, maybe as consulting companies that allowed them to bill customers, rather than Big Tech companies. So why am I not doing that? These smaller companies, just like Californian startups, are often high risk, and big pay off — many struggled, and more will struggle, at their inception, making it riskier to approach them without a safety net or a nest egg to fall onto if they do fail. As it turned out, the companies I have been working with when I had my own consulting companies went through that, and I couldn’t keep up with expenses without a stable paycheck — thus why I originally accepted the Google offer.

Looking at it with hindsight, if I had not been scared, I should have done that even earlier: with my health scare, getting some more short term gains at the beginning of my career might have given me the opportunity to now dedicate myself to something I feel more passionate about. But I was scared, and so I struggled to make ends meet until I accepted a full time role in Big Tech, and now I find myself able to learn the skills I felt deep down I have been missing, as well as support my family — which makes it less likely I’ll drop out back into Free Software full time.

I don’t have any easy solutions to give out here. I’m just sharing my experience and my thoughts about why things behave the way they do. I’m not suggesting that it is fair, or that we shouldn’t attempt to improve on it. But I am suggesting that this is not a world that can be represented in extremes: the one guy in Nebraska may be a hero to you, but our descendents won’t be reading about them in the history books, not even on computer history.