Free Software Cooperatives Redux

Some years ago, I mused that what the world needed was Free Software cooperatives. The post was written on the back of BerliOS shutting down, and so the conversation that followed mostly focused on the need for hosting providers to handle source control and issue tracking, a problem that nowadays is covered by Codeberg — although I personally haven’t taken the plunge to migrate because I don’t think my workflows would be compatible with it, and I don’t have the time to look into it any time soon.

The recent kerfuffle with xz has brought this topic back to my mind, because, though a number of discussions covered the problem of funding the costs of development, I don’t think any even scratched the surface — and I think (again) that cooperatives might be a solution, but far from an easy (or affordable) one.

The first problem, if you want to discuss the matter of paying maintainers for developing open source projects, is that you’re competing with the salaries of tech companies and the like — nothing new, I wrote about it on the back of a previous large-scale vulnerability (log4shell), though I would point out that Filippo put it more eloquently (not for the first or last time!) so you should also read his point of view.

I’m sure there’s plenty of people out there that are either already well off, or in a situation not to have to care that much about the money in their bank account, that taking a role that pays a fraction of the equivalent market rate is acceptable to them — and indeed, the actual paycheck at the end of the month is not always the most important part, as we have seen even before the lockdowns, with people preferring to live close to their family and friends, or in cities or countries with lower costs of living. Most people do not choose their job based on salary alone, but it is a factor.

Realistically, if you’re trying to suggest that a project is of such importance that it is worth paying (close to) market rate for a maintainer to be dedicated to it, it is also quite possible that it is important enough to have more than one person on payroll. This should be very clear by how the whole xz backdoor played out: most of the necessary trigger changes happened while the legitimate maintainer was on a break, to the point that it was happenstance that he noticed what was happening, and came back explicitly to help clean up the mess. And there’s no way that an individual maintainer, even paid a full time employee equivalent salary, can handle security incidents 24/7, 365 days a year.

This is something I learnt the hard way when I freelanced. While I opened my VAT ID primarily to work on LScube – and even doing that I had to be woken up because at one of the ISF sessions the software failed to start due to an annoying locale problem – I quickly found out that you can make a lot more money as an MSP (Managed Service Provider), a sysadmin-for-hire that is. And indeed, during quiet and calm months, you could say that MSPs can earn quite a bit by doing next to nothing! The problem is when things are not quiet.

Back then, any time I took some time off, I had to organize cover with another contractor. Thankfully, I had friends who were in the same business that could take care of emergencies, but that didn’t always work out well. And there’s a point in which, friends or not, you’re basically introducing your customers to your competitors. But it has to happen! There’s no way that a paying customer would wait for you to be back from even a single week vacation, if their entire business is blocked by a computer issue!

While maintenance of software (rather than systems) is usually less time sensitive, it wouldn’t do if a Black Swan vulnerability such as log4shell would be exploited in the wild while the maintainer is on a round-the-world trip and is currently taking a flight from Singapore to Melbourne. You’d need at the very least a second person for each project, and make sure that they don’t take vacation at the same time. And that doesn’t really account for sleep or weekend, and let me tell you that being on-call every other weekend is not light – I have done it for work, and I’m usually responsible for relatively low-stake backend services that do not directly impact users, and even just the tension of knowing you may be called during the day is enough to not make you rest at all.

So let’s say you need three maintainers, and let’s say they’re paid ⅔ of the going market rate for a mid-level engineer in a big tech company. In London, the gross salary paid to the employee for this would probably be roughly around £75000 — I’m oversimplifying and rounding up a lot here, assuming we’re in the ballpark. There is always an undercurrent suggesting that Free Software developers shouldn’t need to “make money” as much as making a living, of course, and it is well possible that, for the sake of argument, you find three people who would rather work on Free Software maintenance full time rather than work for a big tech company, and thus would accept a much lower salary — but while this can possibly happen to begin with, if we’re talking about the sustainability of Free Software, you need to assume you can backfill the positions, which means you need to compare to similar positions in the industry, even more so because the life of a Free Software maintainer is rarely filled with welcoming, affirming feedback. And speaking of feedback, that’s why I’m targeting this hypothetical to mid-level engineers in terms of experience: I know I grew a lot with the feedback I received working in a bubble, and I believe relying too much on newcomers to maintain critical software is fraught with risks.

At this point, you may expect then that the funding for these engineers would be roughly £225000 an year — that’s not a very high number in big tech as I know quite a few senior engineers making more than that alone. But then you have to add all of the overhead to run an entity to handle the contracts, and making sure that the funding is in place and… well, there’s a lot of stuff to keep everything together, so the ballpark I’ve been taught about, when thinking of companies with salaried employees, is to roughly double the payroll at the least in administrative costs, and add on top any additional costs of doing business. With free software maintenance, there should be no or minimal additional costs so let’s say that running a full time, three person maintenance crew would cost roughly half a million pounds a year.

There is quite a lot of stuff you can do with half a million pounds a year, that is objectively more important than maintaining a single Free Software project. Which is why I keep saying that the only answer I can see is co-operatives. Instead of having a half a million pound budget for a single project, you can pool together resources (particularly, on-call for emergencies) for multiple projects under a specific umbrella. For this to work well, though, you need to have projects that have some commonality — VideoLAN is my obvious go-to example, particularly as it encompasses more than just VLC nowadays, not just by welcoming existing projects into its support, but by spinning up new projects within the boundary of the organization (dav1d anyone?) Another, more recent example, is the way Nabu Casa funded development of related projects, and the very recently announced Open Home Foundation.

Having a topical organization with a clear remit of the projects they are in the business to maintain is useful in another way, as VideoLAN shows, and that is they reduce (but not cancel) the boredom of keeping a maintainer full-time onto a project that requires no changes. Michał covered this, and linked to my old post as well, so I won’t spend much time describing the issue again, except to say that, for most projects, there’s really not much to do on a day-to-day basis that could keep occupied the type of person who could be employed as mid-level engineer in big tech — even if they were ethically opposed to the existence of those positions, a maintainer left with nothing to do but to be available to fix a once-in-ten-years security issue is going to grow bored and start focusing on something else, anything else.

Among the arguments trying to refute this, I heard that there’s infinite amount of performance improvement that can be squeezed out of the tiniest of libraries — this may be true to a point, in the sense that you could micro-optimize for all kind of different architectures that most people have never heard of, but it’s the kind of work that, in my experience, ends up bringing negative value. You don’t want to end up holding something like xz, a compression library, to have optimized assembly implementations for h8300, mipsel, and a dozen variants of ARM, just because your customers paid for it at one point — the end result is a project that can barely move and react to events, as testing all of these variants is unlikely to be easy, and also has less flexibility in replacing its maintainers as you would need experience with too wide a range of systems and implementations!

You may notice here that this is something that FFmpeg actually does have success with. But again, I’m here trying to talk about making sustainable the small, one-person-show kind of projects. Those that people keep referring to as Nebraska made, thanks to that one XKCD. FFmpeg, and cURL, are definitely not your typical projects, for more easons than I have time to get into the details of right now.

It’s a similar problem as for when people suggest big tech to just hire the maintainers of the projects they depend on — yeah it would ensure that someone is looking at the project, but is that feasible? With very few exceptions, I do not know of any engineer who gets hired into big tech and be told, effectively, to coast — or to make up a possible phrasing “look after the project, but don’t try to get promoted by finding larger scope.” Once you’re in a company, and you get bored, it’s silly not to figure out that you could have a larger impact (and larger compensation) by doing something else, at which point the project might have a maintainer, but not a full-time one!

Approaches like Tidelift sound interesting on paper, but they remind me too much of Patreon: it’s great while people (or organizations in the case of Tidelift, primarily) have money to spare, but when cash starts being expensive, those elective expenses are the first to be cut. It’s similar to contracting: if you only price yourself only at the rate that you would get as a full-time employee, you’ll eventually find that you’re running out of cash or sanity: even in Europe, where you don’t need to worry about health insurance and the like, you don’t get paid time off if you’re freelancing and contracting, so if you do need a break between clients, you need to have savings to get to. And if you happen to not be available when a past or prospective client needs you… you may be losing more than one week of a contract! That’s how my adventures in Los Angeles costed me pretty much my whole freelancing career.

I actually think this is a widespread problem among Free Software enthusiasts: unless you have experience both working as a maintainer and working in a big tech company, you don’t have enough points of reference on the costs/compensation to come up with a workable plan for the people you’re talking about. Again, remember I’m not talking here about compensating university students who happened to have written a library you like! If you want to have sustainable Free Software maintenance, you need to be able to compensate those people who could easily get a much better paid job that does not involve having to deal with self-centered, over-entitled customer support requests!

And part of what makes me generally unhappy with the state of affairs we arrived to, is that I see the same exact people keep throwing around a “clearly whatever_service was a zero-interest rate phenomenon!” as a reason for mocking commercial failures — folks, a lot of corporate-sponsored large Free Software projects existed because of the next-to-free cash as well! I have not been alone trying to explain sustainability, but many of you refused to listen, because it would require accepting not every person in the world subscribe to the same ideology as you!

But I don’t have all of the answers, heck I don’t have most of the answers — I can do a back-of-the-envelope calculation on the funding you would need to sustainably maintain projects, but I honestly have no idea where to find those funds! How many projects would a three-person cooperative need to maintain, to be able to charge enough large companies to support those budgets? I don’t know! How much work would need to happen to demonstrate that the money isn’t “wasted”? I don’t know!

But what I do know is that the path to sustainability cannot rely on Patreon or YouTube style “this month you’re loaded, next month you may have nothing.” And that’s without accounting for the fact that the same popularity contest model means that you’re almost certainly not going to solve the problem for those projects and maintainers who need it the most – those that are used everywhere because they’re so stable, a single release every ten years is fine, short of a discovered vulnerability – but rather for those who are the better known, the flashiest, the most recently exploited, or with the loudest maintainers. And honestly? It all makes me sad.