Making money with Free Software development

This post will talk about financial, real-life problems. If you don’t feel like knowing about my personal financial and work situation, feel free to skip this. I say this, because some people complained before that I make it know too often that I don’t get paid for most of my Gentoo-related work. Not sure, maybe they feel at unease because of that. At any rate, this is not conditioning my contributions, most of the time.

While I like working on Free Software and think it’s great if I were to do that just as a pass-time like I used to do five or six years ago, I wouldn’t go much far. Time passed, I grew older, and life got complicated. I don’t have a rent to pay (for now — fortunately), but I’m living alone with my (unemployed) mother since last autumn. Electricity, connectivity, water, LPG, taxes… at the end of the day, I don’t have any more the liberty of turning down many offers. (On the other hand, I had to turn down a few because I was already quite fully booked).

But at least I try my best to develop Free Software when I’m working: thanks to the fact that there is space to choose licenses, sometimes I even can write new Free Software while doing my job. Although this often requires some trickery that is not too obvious when you start the kind of freelancing/consulting work I’ve been doing the past two/three years.

The first obvious way to make money during development is improving the Free Software that is used as part of the infrastructure by the software you’re writing. This might end up getting apparent in spite of various NDAs you might have signed, so it has to be vouched carefully, but in general it’s the kind of information that you know you’ll just show the world. If somebody is following my commits to the tree lately can probably tell something’s up, with me picking up two tied packages on areas that I don’t usually follow that much. And from that you can see that there’s a better pam_pgsql out there which was needed for my task to go on.

Similarly, whenever I used Gentoo for something work-related, I ended up trying to get in the tree all the modification and improvements needed; gdbserver was one of the packages I needed to complete a past project, and it’s now in the tree for everyone to use. it’s not a bad thing. At the same time, you can count in the presence of Rudy in the tree, and thus the fixes for its dependencies that I keep sending upstream.

But there are more “subtle” ways to work on Free Software while doing paid work, and that is to divide the problem you’re working on in separate functions; all that concerns business logic will most likely have to be kept proprietary, at least with most customers, but at the same time, the smaller, general problems that don’t include any business logic in them may be left out of the secrecy. I actually find this a generally good approach for many reasons: you may get code improvements if someone else needs to solve similar problems and decide to use your codebase; or you might get better security reviews (let’s be honest, security issues will appear in your software, whether it’s open- or closed-source; but if it’s open, they are more likely to come in sooner; and sooner might mean your deployment is still limited enough you can cover your bases with upgrades without spending a fortune, or bothering your users).

Of course to be able to split your problem into open-source subprojects, you also have to make it very clear that the business logic of your customer is not at risk. This means that you have to carefully choose the license to use (and in these cases, GPL is rarely an option, unless you work at an arm’s length), and at the same time you have to make sure that the terms are clear to the customer. Telling them “I used this, so you should follow these general terms” is rarely going to help you convince them to use more Free Software when possible, and contribute back to it; it’s actually going to make them wish not to respect the license and just not do anything. Having clear statements about what they have to publish, for GPL and similar licenses is the best way to accomplish that.

And there is another thing you have to learn, and especially teach your customers: the advantage of upstreaming changes. This is something that is generally well-understood by distribution developers, packagers, and all those developers using others’ code but not developing it directly, but most of the time, corporate, proprietary-oriented customers won’t understand the importance of it. If you have to customize some software to suit your need better, whether or not you’re obligated to publish your changes should really have little bearing in whether you should send them upstream or not. Even if the license is very permissive and won’t require you to publish change, you really should send them upstream. Why? Because sooner or later you’re going to have to update your software.

Even when software update is not that welcome a task, because obviously the fewer changes, the less risk of bugs, there are situations where upgrades are inevitable. New architectures, new hardware revisions, new required functionalities, security issues. All these problems might require you to update your software. And when you update software that you customised, then the trouble starts. If your customisations are simply hacks, piled up over and over again, you’re going to have to make some sense out of the hacks, and apply them again over the new codebase, praying that they’ll apply cleanly, or work at all. If you sent your changes upstream, or at least sent a request for he feature you pushed in, there are chances that either the code is already there (and might require just little tweaking) or at least that someone else kept the patches up-to-date.

Obviously, this is very low-scale; there are entire projects with consultancy companies behind them to offer support, customisation and all the frills needed; on the other hand this is what I do for a living myself, most of the time. So please, don’t mess up expecting that all Free Software work is done by volunteers, or that it’s never paid for. Nor try to defend messy code with “nobody was paid to do that”… when you pay people to write code, most of the time, they will not make it nice; they’ll most likely try to cut as many corners as possible. Unless, of course, readability and properness is a requirement (which rarely is).

Tweaking the tinderbox

After I turned it off I got quite a few people asking what’s going to happen to the tinderbox. I finally decided to turn it back on a week or so ago, since I started having trouble with new packages on my system and they were worth the hassle. I’m stil not sure how to handle this on the long run to be honest: there are many costs tied to the tinderbox as I pondered and I’m not happy with having to keep paying for them myself.

On the other hand I have no time to either figure out how to get the Foundation to pay for tinderboxes, and even less to find a way to analyse the huge amount of logs provided by the tinderbox remotely. So unless somebody can help me with solving these problems, the tinderbox will only exist for as long as I wish to keep it going.

Putting these management problems aside, the tinderbox is now going through a rough patch mostly because EAPI=2 USE-dependencies started to find their way into the tree: even stuff that was previously ignored by built_with_use is now added as a dependency without trouble, and the results are, well, not excessively nice for what concerns the tinderbox. While it’s definitely better for the user to know of the needs and dependencies of the packages they merge before starting a long and boring build, in the case of the tinderbox, where as many packages are installed together as possible, it becomes troublesome to make all the dependencies fit.

Let’s not even talk about the = misplaced dependencies (if you need a given feature enabled on a library to have the same feature enabled, it usually doesn’t mean that the library has to have it disabled for you to have it disabled), and let’s just consider two prominent examples: Tcl’s and Perl’s support for multi-threading. Depending on whether the package makes use of threads, or is not multithread safe, the dependencies on the interpreter itself will require the USE flag to be either enabled or disabled. Since a package cannot be both multi-threaded and not, I’m forced to make a decision, which simply is to enable threads for both (it seems like they have the most dependencies this way).

Interestingly, most of the packages wanting single-threaded Perl, have optional Perl support, which means that to solve the conundrum I only have to disable the perl USE flag for those packages.

What might be of interest here, just for the news, is that just to satisfy the dependencies of USE flags, with the default profile and almost no global USE flag set (I enabled ruby globally, which is not enabled by default, and java5/java6 as I’m using Icedtea as global VM), I have 178 lines of package.use file. Plus 37 lines dedicated at enabling options for those packages that need at least one enabled (although some might be redundant right now because I filed bugs for them and they were fixed).

Further tweaks that are present in the configuration right now include having a longish list of masked packages as they need me to register (and sometimes pay as well) to get the distfiles (and they seem to be mostly scientific packages), or that need a CD-Rom (although they are now mostly solved by PROPERTIES=interactive), and finally for packages that block each other (which is baaad).

Sigh I wonder if we’ll be able to skip over all these problems at some point in the future, although I’m pretty sure that for the big part the problem lies in the way most upstream work: ignoring the fact that there is other software they don’t use out there.

New Linux-PAM and pambase

So I noticed tonight that yesterday a new Linux-PAM (in portage as a generic sys-libs/pam, even though I want to change it one day) was released: 1.1.0. I actually was scared that it was a big overhaul, it seems instead to be little more than a feature release of the previous release, so it’s fine.

All my patches from pre-1.0 (as well as the one to disable NIS support when not found) are still outstanding and they still apply cleanly. I guess I’ll have to harass upstream more about them and see that they are applied, maybe for 1.1.1 or something.

At any rate, the new release does have a few interesting changes, like a new pam_tally module that is now wordsize-independent (that is, it works the same way both in 32- and 64- bit configurations), for this reason I’m now preparing a new pambase to make use of it instead of the old one by default if present. Unfortunately I’m finding a couple of issues that make it not easy to solve.

There is another change that might be worth nothing: I have already blogged about the sha512 support that Linux-PAM 1.0.1 has provided us with. Well, with this new release, together with sha512, sha256 and md5 hashing… there’s blowfish!

But pambase is not going to provide a way to enable it. It’s not that I don’t want people to use blowfish, but rather that upstream relies on the crypt() function provided by the C library: when it supports the above algorithms, then pam_unix will support them as well, otherwise, you’re out of luck. Since Gentoo does not provide a blowfish-patched glibc, then it won’t support the blowfish method. Until that time, this post will serve as an answer for those interested in the matter, who think that by just using “blowfish” you can gain stronger hashing.

Since this is a minor version bump (compared to the previous 1.0), I’ve also decided to change a bit the ebuild too; I’ve removed the warnings for pam_userdb and other modules that have been moved in separate packages, since they were only checked for when updating from pre-1.0 versions. I’m almost tempted to remove the safechecks for pam_stack, pam_pwdb, and pam_console; the pam_timestamp checks have been dropped because that particular module, initially present in our older versions of PAM which applied RedHat-supplied patches, is now present upstream.

I’m going to try mediating with Kukuk about the patches we apply, dropping a few, merging a few more. It’s more time spent on stuff I don’t care much about (PAM is something people detest and I don’t really have a good use for it myself), but I guess it’s something that Gentoo needs. As usual, kudos and gifts are very welcome (the latter in particular because with the recent expenses I’ve had to take care of my leisure budget reduced considerably).

Time to get going to work then, I’m afraid I’ll have to bump the pambase version to 20090621, I don’t think I can fix it before night!

Problems and mitigation strategies for –as-needed

Doug and Luca tonight asked me to comment about the --as-needed by default bug. As you can read there, my assessment is that the time is ready to bring on a new stage of --as-needed testing through use of forced --as-needed on specs files. If you wish to help testing with this (and I can tell you I’m working on testing this massively), you can do it by creating your own asneeded specs.

Note, Warning, End of the World Caution!

This is not a procedure for the generic users, this is for power users, who don’t mind screwing up their system even beyond repair! But it would help me (and the rest of Gentoo) testing.

First of all you have to create your own specs file, so issue this command:

# export SPECSFILE=$(dirname "$(gcc -print-libgcc-file-name)")/asneeded.specs
# export CURRPROFILE=/etc/env.d/gcc/$(gcc-config -c)
# gcc -dumpspecs | sed -e '/link:/,+1 s:--eh-frame-hdr: --as-needed:' > "$SPECSFILE"
# sed "${CURRPROFILE}" -e '1iGCC_SPECS='$SPECSFILE > "${CURRPROFILE}-asneeded"
# gcc-config "$(basename "${CURRPROFILE}")-asneeded"
# source /etc/profile

This will create the specs file we’re going to use, it just adds —as-needed for any linking command that is not static, then it would create a new configuration file for gcc-config pointing to that file. After this, —as-needed will be forced on. Now go rebuild your system and file bugs about the problems.

A package known to break with this is xmlrpc-c, which in turn makes cmake fail; as some people lack common sense (like adding a way to not have xmlrpc-c as a dependency because you might not want your cmake to ever submit the results of tests), this can get nasty for KDE users. But maybe, just maybe, someone can look into fixing the package at this point.

But xmlrpc-c does require some reflection on how to handle --as-needed in these cases: the problem is that the error is in one package (xmlrpc-c) and the failure in another (cmake) which makes it difficult to asses whether --as-needed break something; you might have a broken package, but nothing depending on it, and never notice. And a maintainer might not notice that his package is broken because other maintainers will get the bugs first (until they get redirected properly). Indeed it’s sub-optimal.

Interestingly enough, Mandriva actually started working on trying to resolve this problem radically, they inject -Wl,--no-undefined in their build procedures so that if a library is lacking symbols, the build dies sooner rather than later. This is fine up to a certain point, because there are times when a library does have undefined symbols, for instance if it has a recursive dependency over another library (which is the case of PulseAudio’s libpulse and libpulsecore, which I discussed with Lennart some time ago). Of course you can work this around by adding a further -Wl,--undefined that then tells ld to discard the former, but it requires more work, and more coordination with upstream.

Indeed, coordination with upstream is a crucial point here, since having to maintain --as-needed fixes in Gentoo is going to be cumbersome in the future, and even more if we start to follow Mandriva’s steps (thankfully, Mandriva is submitting the issues upstream so that they get fixed). But I admit I also haven’t been entirely straight on that; I pushed just today a series of patches to ALSA packages, one of which, to alsa-tools, was for --as-needed support (the copy we have in Portage also just works around the bug rather than fixing it). Maybe we need people that starts checking the tree for patches that haven’t been pushed upstream and tries to push them (with proper credits of course).

Another thing that we have to consider is that many times we have upstream that provide broken Makefiles, and while sending the fix upstream is still possible, fixing it in the ebuild takes more time than it is worth; this is why I want to refine and submit for approval my simple build eclass idea, that at least works as a mitigation strategy.

Outdated tools

There is one interesting differnce between Linux and full operating system projects like FreeBSD, the other BSDs and OpenSolaris: Linux historically didn’t have much coordination between kernel and userland.

This becomes a problem for instance when udev and the kernel disagree on how to handle something, or on when you end up with a tool trying to use some old kernel interface.

It looks tremendously bad when you see that even strace fail (which on FreeBSD does not seem to happen, as ktrace is part of the single project). And I don’t know of any strace replacement.. and I relay a lot on that tool!

This gets further interesting when you add in the USB access through /proc (usbfs), that has been deprecated a long time ago, but that most people are probably still enabling in their kernel. The new interface, using /dev is available for quite a while, and libusb is supporting it very well. But as it turns out, VMware does not use libusb for accessing the USB devices, and it does not suppor the new interface.

I wonder how many projects have this problem. I remember net-tools being worked on, iproute2 replacing ifconfig and so on… but how many tools are actually always in sync between kernel and userland, as of now?

ALSA is also a very common problem with this as the drivers in the past often ended up out of sync between kernel and driver, causing subtle and obnoxious problems.

And even counting software tools that are well in sync between the two, how many of these tools are being audited for, for instance, performance improvements? I wonder.

I’m afraid this is a blog post without solution, but I’d like to make people think about this, maybe someone can help finding solutions ;)

Please don’t build your examples by default

I’m in the middle of an emerge -e world I started to try getting as much glibc 2.8 failure have a bug already on Gentoo’s Bugzilla so that users hitting them don’t have to report them anew.

One thing I noticed during this rebuild is the amount of time spent with no good reason to build tests and examples.

I have the test feature disabled, I only enable it for the software I maintain, and I usually don’t have so much time to look at the test of all software. Still some packages, like dbus, hal, gtk, and so on, build their tests anyway. They don’t run them, but they build tests and example during the default make all call.

I talked with Lennart about that some time ago as also libdaemon does that. He pointed out to me that these examples are often a way for the developers to test that the code still links correctly, for instance, and thus should not be disabled by default during development. I agree it can be useful that way.

For this reason, I prepared a patch for libdaemon, that you can now find here, which I should have committed and I forgot about – I’ll see to do that soonish :P – which should make it possible to have the best of both options: developers still get their examples (and/or tests), while distributions can opt-out them.

For ebuilds, this means that the src_test function should do a make -C tests or make -C examples to build tests and examples, after such a patch is applied, to explicitly build the code that most users won’t need during standard run.

I know it’s just seconds, or minutes, of time to build the examples, but if you add all those up, you’re wasting lots of time. And we should always strive to optimise time usage, no?

Update: the conditional-examples patch was a dead link, so I now link directly to Lennart’s gitweb and the diff that he used; remember please that it works for examples, but tests might be slightly different.

Update (2017-04-29): That link is now dead so I removed the example, sorry.