Tag: Distributions

Distributions are becoming irrelevant: difference was our strength and our liability

For someone that has spent the past thirteen years defining himself as a developer of a Linux distribution (whether I really am still a Gentoo Linux developer or not is up for debate I’m sure), having to write a title like this is obviously hard. But from the day I started working on open source software to now I have grown a lot, and I have realized I have been wrong about many things in the past.

One thing that I realized recently is that nowadays, distributions lost the war. As the title of this post says, difference is our strength, but at the same time, it is also the seed of our ruin. Take distributions: Gentoo, Fedora, Debian, SuSE, Archlinux, Ubuntu. They all look and act differently, focusing on different target users, and because of this they differ significantly in which software they make available, which versions are made available, and how much effort is spent on testing, both the package itself and the system integration.

While describing it this way, there is nothing that scream «Conflict!», except at this point we all know that they do conflict, and the solutions from many different communities, have been to just ignore distributions: developers of libraries for high level languages built their own packaging (Ruby Gems, PyPI, let’s not even talk about Go), business application developers started by using containers and ended up with Docker, and user application developers have now started converging onto Flatpak.

Why the conflicts? A lot of time the answer is to be found in bickering among developers of different distributions and the «We are better than them!» attitude, which often turned to «We don’t need your help!». Sometimes this went all the way to the negative side to the point of «Oh it’s a Gentoo [or other] developer complaining, it’s all their fault and their problem, ignore them.» And let’s not forget of the enmity between forks (like Gentoo, Funtoo and Exherbo), in which both sides are trying to prove being better than the other. A lot of conflict all over the place.

There were of course at least two main attempts to standardise parts of how a distribution works: the Linux Standard Base and FreeDesktop.org. The former is effectively a disaster, the latter is more or less accepted, but the problem lies there: in the more-or-less. Let’s look at these two separately.

The LSB was effectively a commercial effort, which was aimed at pleasing (effectively) only the distributors of binary packages. It really didn’t make much of an assurance of the environment you could build things in, and it never invited non-commercial entities to discuss the reasoning behind the standard. In an environment like open source, the fact that the LSB became an ISO standard is not a badge of honour, but rather a worry that it’s over-specified and over-complicated. Which I think most people agree it is. There is also quite an overreach of specifying the presence of binary libraries, rather than being a set of guidelines for distributions to follow.

And yes, although technically LSB is still out there working, the last release I could find described in Wikipedia is from 2015, and I couldn’t even find at first search whether they certified any distribution version. Also, because of the nature of certifications, it’s impossible to certify a rolling-release distribution, which as it happens are becoming much more widespread than they used to.

I think that one of the problem of LSB, both from the adoption and usefulness point of views, is that it focused entirely too much on providing a base platform for binary and commercial application. Back when it was developed, it seemed like the future of Linux (particularly on the desktop) relied entirely on the ability for proprietary software applications to be developed that could run on it, the way they do on Windows and OS X. Since many of the distributions didn’t really aim to support this particular environment, convincing them to support LSB was clearly pointless.

FreeDesktop.org is in a much better state in this regard. They point out that whatever they write is not standards, but de-facto specifications. Because of the de-facto character of these, they started by effectively writing down whatever GNOME and RedHat were doing, but then grown to be significantly more cross-desktop, thanks to KDE and other communities. Because of the nature of the open source community, FD.o specifications are much more widely adopted than the “standards”.

Again, if you compare with what I said above, FD.o provides specifications that make it easier to write, rather than run, applications. It provides you with guarantees of where you should be looking for your file and which icons should be rendering, and which interfaces are exposed. Instead of trying to provide an environment where a in-house written application will keep running for the next twenty years (which, admittedly, Windows has provided for a very long time), it provides you building blocks interfaces so that you can create whatever the heck you want and integrate with the rest of the desktop environments.

As it happens, Lennart and his systemd ended up standardizing distributions a lot more than LSB or FD.o ever did, if nothing else by taking over one of the biggest customization points of them all: the init system. Now, I have complained about this before that it probably could have been a good topic for a standard even before systemd, and independently from it, that developers should have been following, but that’s another problem. At the end of the day, there is at least some cross-distribution way to provide init system support, and developers know that if they build their daemon in a certain way, then they can provide the init system integration themselves, rather than relying on the packagers.

I feel that we should have had much more of that. When I worked on ruby-ng.eclass and fakegem.eclass, I tried getting the Debian Ruby team, who had similar complaints before, to join me on a mailing list so that we could discuss a common interface between Gems developers and Linux distributions, but once again, that did not actually happen. My afterthought is that we should have had a similar discussion for CPAN, CRAN, PyPI, Cargo and so on… and that would probably have spared us the mess that is go packaging.

The problem is not only getting the distributions to overcome their differences, both in technical direction and marketing, but that it requires sitting at a table with the people who built and use those systems, and actually figuring out what they are trying to achieve. Because in particular in the case of Gems and the other packaging systems, who you should talk with is not only your distribution’s users, but most importantly the library authors (whose main interest is shipping stuff so that people can use them) and the developers who use them (whose main interest is being able to fetch and use a library without waiting for months). The distribution users are, for most of the biggest projects, sysadmins.

This means you have a multi-faceted problem to solve, with different roles of people, and different needs of them. Finding a solution that does not compromise, and covers 100% of the needs of all the roles involved, and requires no workflow change on any one’s part is effectively impossible. What you should be doing is focusing on choosing the very important features for the roles critical to the environment (in the example above, the developers of the libraries, and the developers of the apps using those libraries), requiring the minimum amount of changes to their workflow (but convince them to change the workflow where it really is needed, as long as it’s not more cumbersome than it was before for no advantage), and figuring out what can be done to satisfy or change the requirements of the “less important” roles (distribution maintainers usually being that role).

Again going back to the example of Gems: it is clear by now that most of the developers never cared of getting their libraries to be carried onto distributions. They cared about the ability to push new releases of their code fast, seamlessly and not have to learn about distributions at all. The consumers of these libraries don’t and should not care about how to package them for their distributions or how they even interact with it, they just want to be able to deploy their application with the library versions they tested. And setting aside their trust in distributions, sysadmin only care to have a sane handling of dependencies and being able to tell which version of which library is running on their production, to upgrade them in case of a security issues. Now, the distribution maintainers can become the nexus for all these problems, and solve it once and for all… but they will have to be the ones making the biggest changes in their workflow – which is what we did with ruby-ng – otherwise they will just become irrelevant.

Indeed, Ruby Gems and Bundler, PyPI and VirtualEnv, and now Docker itself, are expressions of that: distribution themselves became a major risk and cost point, by being too different between each other and not providing an easy way to just provide one working library, and use one working library. These roles are critical to the environment: if nobody publish libraries, consumers have no library to use; if nobody consumes libraries, there is no point in publishing them. If nobody packages libraries, but there are ways to publish and consume them, the environment still stands.

What would I do if I could go back in time, be significantly more charismatic, and change the state of things? (And I’m saying this for future reference, because if it ever becomes relevant to my life again, I’ll do exactly that.)

  • I would try to convince people that even on divergence of technical direction, discussing and collaborating is a good thing to do. No idea is stupid, idiotic or any other random set of negative words. The whole point of that is that you need to make sure that even if you don’t agree on a given direction, you can agree on others, it’s not a zero-sum game!
  • Speaking of, “overly complicated” is a valid reason to not accept one direction and take another; “we always did it this way” is not a good reason. You can keep using it, but then you’ll end up with Solaris a very stagnant project.
  • Talk with the stakeholders of the projects that are bypassing distributions, and figure out why they are doing that. Provide “standard” tooling, or at least a proposal on how to do things in such a way that the distributions are still happy, without causing undue burden.
  • Most importantly, talk. Whether it is by organizing mailing lists, IRC channels, birds of a feather at conferences, or whatever else. People need to talk and discuss the issues at hand in clear, in front of the people building the tooling and making the decisions.

I have not done any of that in the past. If I ever get in front of something like this, I’ll do my best to, instead. Unfortunately, this is a position that, in the current universe we’re talking about, would have required more privilege than I had before. Not only for my personal training and experience to understand what should have been done, but because it requires actually meeting with people and organizing real life summits. And while nowadays I did become a globetrotter, I could never have afforded that before.

Ego-stroke development

I’ll write another post, expanding from a single tweet of mine:

If you release code and are not interested in helping distributions package it… don’t release code. Honestly.

It might sound harsh and mean, but I mean it, and I think it’s for the best.

You might have a number of different reasons to work on opensource projects, but you can basically find that the people who write open source projects and then don’t care about distributions and other kind of integrators, tend to be those developers that only work for their own ego. I don’t mean they write bad code, sometimes they might actually be the best programmer, but their dislike for distributors is poisonous for the whole opensource environment.

Why do I say this? Well, the problem is that if you release good opensource code, whether it’s packaged by distributions or not, it might very well happen that some other developer will find and use it, that’s what you want, when you release it opensource, isn’t it? Now when that project is no longer a library or backend tool, but rather an user facing tool that the developer wants to see used, there will be requests to package it. But once that happens, you might be stuck with some backend dependency that cannot be clearly packaged. In that situation, the whole frontend can’t be packaged, and that means that the distribution’s users will end up suffering.

This is getting even worse with gems, especially when the most insightful comment in my latest post comes from an user that seems to proclaim himself a troll (or at least the nick suggests strongly so). Indeed, trying to excuse Ruby gems developers for not caring about our needs (which usually boil down to having tests around, a clear repository, the ability to rebuild documentation and tests without bringing in a whole lot of gem-building tools – jewelwer, hoe, and so on – and most importantly code that works and tests green, which is by far not obvious when dealing with gems), is just saying that you care about open source for your own feel good, rather than for the whole environment.

This gets worrisome when it comes to Ruby stuff, especially because of the ties with Rails. I guess the majority of the gems are developed for usage with Rails, which means web-facing code; if they are developed without caring about the opensource environment and their possible users, it would be silly not to expect that at least a part of them are so insecure they are likely to cause vulnerabilities in code that is actually published for the general usage.

So I think I just decided to make two personal rules, that I wish were widespread:

  • if you do not plan on helping distributions to get your software packaged, don’t release it opensource;
  • if you plan on making your software opensource, do not use software that distributions can’t package.

Maintaining backports with GIT

I have written last week of the good feeling of merging patches upstream – even though since then I don’t think I got anything else merged … well, beside the bti fixes that I sent Greg – this week, let’s start with the opposite problem: how can you handle backports sanely, and have a quick way to check what was merged upstream? Well, the answer, at least for the software that is managed upstream with GIT, is quite easy to me.

Note: yes this is a more comprehensive rehashing of what I posted last December so if you’ve been following my blog for a long time you might not be extremely surprised by the content.

So let’s start with two ideas: branches and tags; for my system to work out properly, you need upstream to have tagged their releases properly; so if the foobar project just released version 1.2.3, we need to have a tag available that is called foobar-1.2.3, v1.2.3, or something along these lines. From that, we’ll start out a new “scratch branch”; it is important to note that it’s a scratch branch, because it means that it can be force-pushed and might require a complete new checkout to work properly. So we have something like the following:

% git clone git://git.foobar.floss/foobar.git
% cd foobar
% git checkout -b 1.2.3-gentoo origin/v1.2.3

This gives us the 1.2.3-gentoo branch as the scratch branch, and we’ll see how that behave in a moment. If upstream fails to provide tags you can also try to track down which exact commit a release corresponds to – it is tricky but not unfeasible – and replace origin/v1.2.3 with the actual SHA hash of the commit or, even better as you’ll guess by the end of the post, tag it yourself.

The idea of using a scratch branch, rather than an actual “gentoo branch” is mostly out of simplicity to me; most of the time, I make more than a couple of changes to a project if I’m packaging it – mostly because I find it easier to just fix possible autotools minor issues before they actually spread throughout the package and other packages as well – but just the actual fixes I want to apply to the packaged version; cleanups, improvements and optimisations I send upstream and wait for the next release. I didn’t always do it this way, I admit.. I changed my opinion when I started maintaining too many packages to follow all of them individually. For this reason I usually have either a personal or a “gentoo” branch where I make changes to apply to master branch, which get sent upstream and merged, and a scratch branch to handle patches. It also makes it no different to add a custom patch or a backport to a specific version (do note, I’ll try to use the word “backport” whenever possible to stress the important of getting the stuff merged upstream so that it will be present in the future, hopefully).

So we know that in the upstream repository there have been a few commits to fix corner case crashers that, incidentally, seem to always apply on Gentoo (don’t laugh, it happens more often than you can think). The commits have the shorthashes 1111111 2222222 3333333 — I have no fantasy for hashes, so sue me.

% git cherry-pick 1111111
% git cherry-pick 2222222
% git cherry-pick 3333333

Now you have a branch with three commits, cherry-picked copies (with different hashes) of the commits you need. At this point, what I usually do, is tagging the current state (and in a few paragraphs you’ll understand why), so that we can get the data out properly; at this point, the way you name the tag depends vastly on how you will release the backport, so I’ll get to that right away.

The most common way to apply patches in Gentoo, for good or bad, is adding them to the files/ subdirectory of a package; to be honest this is my least preferred way unless they are really trivial stuff, because it means that the patches will be sent down the mirrors to all users, no matter whether they use the software or not; also, given the fact that you can use GIT for patch storage and versioning, it’s also duplicating the effort. With GIT-stored patches, it’s usually the easiest to create a files/${PV}/ subdirectory and store there the patches as exported by git format-patch — easy, yes; nice nope: given that, as I’ll say, you’ll be picking the patches again when a new version is released, they’ll always have different hashes, and thus the files will always differ, even if the patch itself is the same patch. This not only wastes time, it makes it non-deduplicable and also gets around the duplicated-files check. D’oh!

A more intelligent way to handle these trivial patches is to use a single, combined patch; while patchutils has a way to combine patches, it’s not really smart; on the other hand, GIT, like most other source control managers, can provide you with diffs between arbitrary points in the repository’s history… you can thus use git diff to export a combined, complete patch in a single file (though lacking history, attribution and explanation). This helps quite a lot when you have a few, or a number, of very small patches, one or two hunks each, that would cause too much overhead in the tree. Combining this way bigger patches can also work, but you’re more likely to compress it and upload it to the mirrors, or to some storage area and add it to SRC_URI.

A third alternative, which is also requiring you to have a storage area for extra distfiles, is using a so-called “patchset tarball”; as a lot of packages already do. The downside of this is that if you have a release without any patch tarball at all, it becomes less trivial to deal with it. At any rate, you can just put in a compressed tar archive the files created, once again, by git format-patch; if you add them as a subdirectory such as patches/ you can then use the epatch function from eutils.eclass to apply them sequentially, simply pointing it at the directory. You can then use the EPATCH_EXCLUDE variable to remove one patch without re-rolling the entire tarball.

Note: epatch itself was designed to use a slightly different patchset tarball format, that included the use of a specification of the architecture, or all to apply to all architectures. This was mostly because its first users were the toolchain-related packages, where architecture-dependent patches are very common. On the other hand, using conditional patches is usually discouraged, and mostly frown upon, for the rest of the software. Reason being that’s quite more likely to make a mistake when conditionality is involved; and that’s nothing new since it was the topic of an article I wrote over five years ago.

If you export the patches as multiple files in filesdir/, you’re not really going to have to think much about naming the tag; for both other cases you have multiple options: tie the name to the ebuild release, tie it to the CVS revision indication, and so on. My personal preferred choice is that of using a single incremental, non-version-specific number for patch tarballs and patches, and mix that with the upstream release version in the tag; in the example above, it would be 1.2.3-gentoo+1. This is, though, just a personal preference.

The reason is simple to explain and I hope it makes sense for others than me; if you tie it to the release of the ebuild (i.e. ${PF}), like the Ruby team did before, you end up in trouble when you want to add a build-change-only patch – take for instance the Berkeley DB 5.0 patch; it doesn’t change what is already installed on a system built with 4.8; it only allows to build anew with 5.0; given that, bumping the release in tree is going to waste users’ time – while using the CVS revision will create quite a few jumps (if you use the revision of the ebuild, that is) as many times you change the ebuild without changing the patches. Removing the indication of the upstream version is also useful, albeit rarely, when upstream does not merge any of your patches, and you could simply reuse the same patchset tarball as previous release; it’s something that comes handy especially when security releases are done.

At this point, as a summary you can do something like this:

  • mkdir patches; pushd patches; git format-patch v1.2.3..; popd; tar jcf foobar-gentoo-1.tar.bz2 patches — gets you a patchset tarball with the patches (similarly you can prepare split patches to run add to the tree);
  • git diff v1.2.3.. > foobar-gentoo-1.patch — creates the complete patch that you can either compress, or upload to mirrors or (if very very little) put it on the tree.

Now, let’s say upstream releases version 1.2.4, and integrates one of our patches. Redoing the patches is quick with GIT as well.

% git checkout -b 1.2.4-gentoo
% git rebase v1.2.4

If there are compatible changes, the new patches will be applied just fine, and updated to not apply with fuzz; any patch that was applied already will count as “empty” and will be simply removed from the branch. At that point, you can just reiterate the export as said above.

When pushing to the repository, remember to push explicitly the various gentoo branches, and make sure to push --tags as well. If you’re a Gentoo developer, you can host such repository on git.overlays.gentoo.org (I host a few of them already; lxc, libvirt, quagga …); probably contributors, even not developers, can ask for similar repositories to be hosted there.

I hope this can help out other developers dealing with GIT-bound upstreams to ease their overweight.

Enabling –as-needed, whose task is it?

A “fellow” Gentoo developer today told me that we shouldn’t try to get --as-needed working because it’s a task for upstream (he actually used the word “distributors” to intend that neither Gentoo nor any other vendor should do that, silly him)… this developer will go unnamed because I’ll also complain right away that he suggested I give up on Gentoo when I gave one particular reason (that I’ll repeat in a moment) for us to do that instead. Just so you know, if it was up to me, that particular developer right now would have his access revoked. Luckily for him, I have no such powers.

Anyway, let me try to put in proper writing why it should fall to Gentoo to enable that by default to protect our users.

The reason I gave above (it was a twitter exchange so I couldn’t articulate it completely), is that “Fedora does it already”. To be clear, both Fedora and Suse/Novell do it already. I’m quite sure Debian doesn’t do it, and I guess Ubuntu doesn’t do that either to keep in line with Debian. And the original hints we took to start with --as-needed came from AltLinux. This alone means that there is quite a bit of consensus out there that it should be a task for distributors to look at. And it should say a lot that the problems solved by --as-needed are marginal for binary distributions like the ones I named here; they all do that simply to reduce the time needed for their binary packages to rebuild, rather than to avoid breaking users’ systems!

But I’m the first person to say that the phrase “$OtherDistribution does it, why don’t you?” is bogus and actually can cause more headaches than it solves. Although most of the time, this is meant when $Otherdistribution is Ubuntu or a relative of theirs. I seriously think that we should take a few more hints from Fedora; not clone them, but they have very strong system-level developers working on their distributions. But that’s a different point altogether by now.

So what other reasons are there for us to provide --as-needed rather than upstream? Well, actually this is the wrong question; the one you should formulate is “Why is it not upstream’s task to use --as-needed?”. While I have been pushing --as-needed support in a few of my upstream packages before, I think by now that it’s not the correct solution. It all boils down to who knows better whether it’s safe to enable --as-needed or not. There are a few things you should assess before enabling --as-needed:

  • does the linker support --as-needed? it’s easier said than done; the linker might understand the flag, but supporting is it another story; there are older versions of ld still in active use that will crash when using it; other with a too greedy --as-needed that will drop libraries that are needed, and only recently the softer implementation was added; while upstream could check for a particualr ld version, what about backports?
  • do the libraries you link to, link to all their needed dependencies? one of the original problems with --as-needed when introduced to the tree was that you’d have to rebuild one of the dependencies because it relied on transitive linking, which --as-needed disallowed (especially in its original, un-softened form); how can a given package make sure that its dependencies are all fine before enabling --as-needed?
  • do the operating system at all support --as-needed? while Gentoo/FreeBSD uses modern binutils, and (most of) the libraries are built so that all the dependencies are linked in for --as-needed support, using it is simply not possible (or wasn’t possible at least…), because gcc will not allow for linking the threading libraries in for compatibility with pre-libmap.conf FreeBSD versions; this has changed recently for Gentoo since we only support more recent versions of the loader that don’t have that limitation; even more so, how can upstream know whether the compiler will have the fix already or not?

Of course you can “solve” most of the doubts by running runtime tests; but is that what upstreams should do? Running multiple tests from multiple packages require sharing the knowledge and risks for the tests to get out-of-sync one with the other; you have redundancy of work.. when instead the distributor can simply decide on whether using --as-needed is part of their priorities or not. It definitely is for Fedora, Suse, AltLinux… it should be for Gentoo as well, especially as a source-based distribution!

Of course, you can find some case-by-case where --as-needed will not work properly; PulseAudio is unfortunately one of those, and I haven’t had the time to debug binutils to see why the softer rules don’t work well in that case. But after years working on this idea, I’m very sure that it’s a very low percentage of stuff that fails to work properly with this, and we should not be taken hostage by a handful of packages out of over ten thousands!

But, when you don’t care about the users’ experience, when you’re just lazy or you think your packages are “special” and deserve to break any possible rule, you can afford yourself to ignore --as-needed. Is that the kind of developers Gentoo should have, though? I don’t think so!

What’s wrong with release notifications?

Distributions of the like of Gentoo have one huge issue with users: they all demand their updates the same moment they are released. This is why many people, including me, have ranted before about the meme of 0day bumps. Generally speaking, we tend to know about the new release of a package we maintain, because we follow, tightly or loosely the development. Unfortunately, it’s quite possible that the new release passes into background for whatever reason, and the result is, well, a not bumped package. Note here: it’s very well possible that some developer will forget to bump his own (upstream) package; shit happens sometimes.

Most of the time, to solve this kind of problem we can use one of the many tools at our disposal to identify release notifications… unfortunately this is not all that feasible nowadays: it was better and it definitely got worse in the past months! Given that most upstream barely have a release publishing procedure, most of us preferred notifications that are not “actively handled” by the developers, but rather than happen as a by-product of the release itself: this way even sloppier release had their notification sent out.

The biggest provider of by-product release notifications was, once upon a time, SourceForge — I say “once upon a time” because they stopped that. While I can understand that a lot of the services offered by SF were redundant, and that most projects ended up setting up better, if less integrated, software anyway (such as phpBB, Mantis – as Bugzilla wouldn’t work, or various Wikis), and I can appreciate that the old File Release System was definitely overcomplex, I can’t see why they stopped allowing to subscribe to notifications. The email that they sent are now loosely replaced by the RSS feed of released files… the problem is that the feed is huge (as it lists all the files ever released for a project) and not sorted chronologically. Sure there is still freshmeat but to have notifications working there you’re asking the upstream maintainer to explicitly remember bumping the version in a different website, and that’s a bit too much, for most people.

You’d expect that other sites that took the place of SourceForge got better at handling this things, wouldn’t you? But that’s definitely not true. Strangely enough, the good example here seem to come from the Ruby community (I say “strangely enough” because you might remember that I ranted so much about missing practises, mandatory procedures and metadata and so on.

First of all, Rubyforge still keeps release notifications by mail (good!), and second, kudos to gemcutter that allows for subscribing to a (semi-)private RSS feed with the new release of just a subset of all the gems released (the Gentoo Ruby team has a common one where the gems present in portage are subscribed — if I remember to add them, that is). It works, sorta. It still requires for you to poll something, although at the end you’re switching the mail reader for the feed reader, so it’s not much of a change. The one problem with that is that you end up receiving a bit more noise than I’d like, as gems that are available in binary form for Windows and Java are listed more than once after an update. But it’s good that it actually is integrated with the gem release procedure.

On the other hand, excluding all the packages that have no hosting facility included at all (which sometimes, such as sudo’s case, have better announcement systems), there are two sites that I count as major screw up, with different degrees: Google Code and Launchpad. The former, is just partly screwed: starring a project does not subscribe to updates, but at least it has a feed of all the files released by the project. What I find definitely strange is that there is no integrated “Subscribe in Google Reader” that would have been definitely more friendly.

Launchpad, instead looks much worse. I recently picked up co-maintainership of a trio of projects and not only there is no email notification, there is no feed of either releases or files! Which means that the only way to find whether a project made a new release is to check the homepage of them. Yuppie. I opened a bug for that on launchpad, but I now lost the link: it was duped against something else, and is no longer visible through launchpad’s own interface, which is, in my book, yet another failure.

Why is it so difficult that packagers need the notifications? This gets even more silly considering that I’m sure the main argument against notification is going to be ”but users won’t have to care, as the package will be available on their distribution”.

What distributions want

Or A 101 lesson on how to ensure that your software package is available to the distribution users (which, incidentally, are the Linux users; while already the conglomerate Linux marketshare is pretty low when compared with Windows and OS X, the marketshare of not-really-distributions like Slackware or Linux from scratch is probably so trivial that you don’t have to care about them most of the time. That, and their users are usually not so much scared about installing stuff by themselves.

I’m posting this quickie because I’d like to tell one other thing to Ryan… yes, you are a drama queen. And you proven it right with your latest rant and it really upsets me that instead of trying to understand the problem your solution seems to be closing yourslef even more inside your little world. It upsets me because I can see you as a capable programmer and I’d prefer your capacity being used for something that people can benefit from, rather than wasted on stuff pointless, like FatELF.

The fact that instead of trying to understand the technical points that me and others made, and tell us why you think they are not good enough, you’re just closing yourself further. By saying that “lots of people talked about it” you’re just proving what you’re looking for: fame and glory. Without actual substantial results to back it up. Just an hint: the people who matters aren’t those who continue saying “FatELF will make distributions useless, will make it possible to develop cross-platform software, will solve the world’s hunger”; the people who matters are those that review FatELF for its technical side, and most of us already deemed it pointless; I already explained what I think about it.

Any ISV that thinks FatELF will solve their cross-distribution or cross-architecture problems have no idea what an ELF file is; they don’t really understand the whole situation at all. I’m pretty sure there are such ISV out there… but I wouldn’t really look forward for them to decide what to put inside the kernel and the other projects.

Do you want your software (your games) to be available to as many people as possible? Start working with the freaking distributions! You don’t need to have mastered all the possible package managers, you don’t even need to know about any of them directly, but you got to listen if packagers ask for some changes. If a packager asks you to unbundle a library or allow selecting between bundled or system library; do it, they have their reasons and they know how to deal with eventual incompatibilities. If a packager asks you to either change your installation structure or at least make it flexible, that’s because with a very few exceptions, distributions are fine with following the FHS.

Take a look to “Distributions-friendly packages”: part 1 part 2 and part 3 .

But no, Ryan’s solution here is again taking cheap shots to distributions and packagers, without actually noticing that, after more than ten years distributions are not going away.

Oh and the first commenter who will try to say again that FatELF is the solution, can please tell me how’s that going to ensure that the people writing the code will understand the difference between little endian and big endian? Or that the size of a pointer is not always 32-bit? Count that in as a captcha; if you cannot give me an answer to those two questions, your comment supporting FatELF as The Solution will be deleted.

Distributions *are* the strength of Linux

People disagree, some people think that no operating system has any need for distributions, with all their difference and their central repositories that aren’t as central. But one of the thing that impress most the users who switch is, in many cases (at least that I could look at myself) the presence of distributions and the ability to install almost any software by simply looking it up in the package manager.

This said, when people think that overcomplex solutions are a perfect way to solve the issues that “vendors” have with distributing their software, you’re probably missing the point quite a bit. Instead of proposing changes in all the possible layers of the operating system stack, you should try to speak with with the distributors and see what you can do to make your software behave in such a way that they can lift the “send the software to the user” problem from you.

It’s a tremendously important point I’m making here: when you develop your software coming from a Windows background to work on Linux, youŕe probably making a huge amount of mistakes; the most common one is to assume that the directory to work on is the directory the program is in, or that the current working directory is the home of the user. Both differ between Windows and Linux. Fixing these minor issues is usually trivial, if you have access to the code, and if you’re willing to bend a bit around to accommodate the requests. In the case that icculus brought up, the proper solution is, generally, splitting the data from the engine, so that you can reuse the data between different architectures, and have a different engine for each architecture; or have a single huge download with all the architectures available, if they are, say, 10% over the size of the data.

The main point here is still that you have first to remember that distributions exist and that users like to rely on them (most of the time) and second to understand that neither the Windows way nor the OS X way applies to Linux. This doesn’t make Linux right and the other wrong, or vice-versa; they are three different worlds, and each one has its own good and bad side.

The biggest mistake in misunderstanding Linux for just another Windows version is providing a setup program, even worse a graphical setup program. If your software has no drivers to install, nothing to register itself into (there is no registry on Linux, after all), you most likely should not give that as the only option. First of all such a program would rarely tell you what’s going to do, and you’d also be going to run that with root privileges to install the stuff, so why should you trust proprietary software with root on your system? Of course if you’re just a “Joe User” you won’t care, you have no clue about that, but any decently skilled user would know that it’s never a good idea to trust any software you cannot control with root privileges on your box.

The second misconception is that some people seem to think that it’s a task for upstream of a project – be it a proprietary software vendor or a free software project – to provide binaries, installer and packages. This is the main reason why that silly FatELF idea is still tickling on some people. Well, let me say it once and for all it’s the distributions’ task to provide packages to the users!

Of course the problem is that distributions rarely can provide all the possible software in the world as package, may it be because their policy is to only allow Free Software (like Debian and Fedora) or for other reasons. In any case the solution is not to say “The distributions are the problem” but rather to wonder “Why are they not packaging my software?”. Of course when the problem is policy related to the license there is little to do, so you’re forced to rely on third party repositories (like RPM Fusion ) that don’t have such problems with policies. In general, a very little leeway for the distributions can go a great deal into making your software available to users.

All kind of projects who want to reach for users should listen to the distributors: that means that if a distributor complain about the way you (don’t) release software, for instance because you only use a “live” repository for the users to use, or about the way you make use of bundled libraries, you should most likely discuss with them a way to handle the situation; failing to do that is going to drive the distributor away (and then you’d probably be complaining that you’ll have to provide binaries for that distribution yourself). Unfortunately I’m quite sure that especially icculus have problems with stuff like that, given I’ve reported more than one Gentoo policy violation for ebuilds that come from icculus.

For proprietary software, this often goes not as much into the way of changing the development of the software but rather to change some distribution details: allow the developer to redistribute your software (so don’t use strange click-through download systems, don’t require the user to go a long way to find what it has to download); give a “raw tarball” option that the distribution can use as source for their packaging, be it binary packages, or source-based packages like Gentoo’s.

Move the packaging task to the packagers, they know it better.

And if you’re developing proprietary commercial software, you might want to approach some developers, and eventually give out some free licenses for them to play with so that they can package the software, and eventually give you feedback in what they would like for you to change. Most of the time, packagers are pretty pragmatic and will not be scared off by “helping proprietary software”; for instance in my overlay you can find some packaging for the Visual Paradigm Suite for which I bought a license a few weeks ago (I needed a working UML software for a job); it’s nowhere near Gentoo-ready, but I’ve not given up on it; since the Visual Paradigm customer support is also quite ready to answer to problems and suggestions, I’ve been sending them my feedback, both as user and as packager. Hopefully I might get to the point where the package is fine with Gentoo policies and I can add it to the main tree normally.

A similar situation happens with the EntropyKey software packaging since I was interested I got two of those and packaged it up; if upstream was interested in packaging this beyond their own support (I think they already have a Debian packager as part of the staff anyway), they could have created a developer program for distributors, and I’m pretty sure almost all distributors would have supported the ekeyd software in no time.

Yes, I am seeing all this situation from a packager point of view, but that’s because I definitely like this approach and instead of resent us for “not providing the stuff you want” or attacking distributions because “you have to make dozens of different packages”, try working with them. Like I said before, Ryan should stop keep inside his own little world where he can do whatever he wants and then expect people to bend at his needs, he should listen to the needs of distributors (which aren’t really so impossible!) and so should anybody who want to enter the Linux ecosystem as it is now.

And it’s definitely not only proprietary software that still doesn’t get this, Mozilla has had a hard time to get to work with distributors, OpenOffice still has such a hard time, Avidemux is a perfect example of how a package gets to ignore all the possible distribution requests (by still shipping a modified FFmpeg for instance).

Most of the time, the reasons why developers don’t want to make accommodations for distributions, are stuff along the lines of “I don’t see what difference does it make”… which is also the very reason why they have such a hard time to get their packaging together.

I still dislike github

When github was uncovered, I was a bit concerned with the idea that it seemed to foment the idea of forking software all over the place, instead of branching, with results that are, in my opinion, quite upsetting in the way some software is handled (see also this David Welton post which is actually quite to the point – I don’t always bash Debian, you know, and at least with the Debian Ruby team I seem to be often on the same page ). I was so concerned that I even wrote an article for LWN about forking and the problems that it comes with.

Thinking about this, I should tell people to read that when they talk about the eglibc mess. And when I can find time I should see to translate my old article about MySQL from Italian to English – and maybe I should see to change the articles page to link the articles directly in HTML form rather than just PDF and DVI.

At any rate, the “fork it” button is not what I’m going to blog about today, but rather what happened yesterday when I decided to update hpricot which is now hosted strictly on github. Indeed there is no download page if not the one in github which points to the tags of the git repository to download.

It starts to get increasingly used the idea that just tagging a release is enough to get it downloaded, no testing, no packaging, nothing else. For Ruby stuff gems are prepared, but that’s it (and I think that github integrates enough logic for not even doing that). It’s cool, isn’t it? No it’s not, not for distributions and not for security.

There is one very important feature for distributions on released code and is the verifiability of the release archives, while it might be a bit too much to ask for all the upstream projects to have a verifiable GnuPG signature and sign all their release, but at least making sure that a release tarball will always be available identical to everybody who download it would be usable. I let you guess that github does not do that which is giving me headaches since it means I have to create the tarballs manually and push them to the Gentoo mirrors for them to be available (git archive makes it not too difficult, but it’s still more difficult that just fetching the release upstream).

I wonder how it might be possible to explain to the Ruby community (because here it’s not just the Rails community I’d say) that distributions are a key to proper management and not something to hinder at every turn.

Bundling libraries: the curse of the ancients

I was very upset by one comment from Ardour’s lead developer Paul Davis in a recently reported “bug” about the un-bundling of libraries from Ardour in Gentoo. I was, to be honest, angry after reading his comment, and I was tempted to answer badly for a while; but then I decided my health was more important and backed away, thought about it, then answered how I answered (which I hope is diplomatic enough). Then I thought it might be useful to address the problem in a less concise way and explain the details.

Ardour is bundling a series of libraries; like I wrote previously, there are problems related to this and we dealt with them by just unbundling the libraries, now Ardour is threatening to withdraw support from Gentoo as a whole if we don’t back away from that decision. I’ll try to address his comments in multiple parts, so that you can understand why it really upset me.

First problem: the oogie-boogie crashes<

It’s a quotation from Adam Savage from MythBusters, watch the show if you want to actually know the detail; I learnt about it from Irregular Webcomic years ago, but I have only seen it about six months ago, since in Italy it only passes on satellite pay TV, and the DVDs are not available (which is why they are in my wishlist).

Let’s see what exactly Paul said:

Many years ago (even before Gentoo existed, I think) we used to distribute Ardour without the various C++ libraries that are now included, and we wasted a ton of time tracking down wierd GUI behaviour, odd stack tracks and many other bizarre bugs that eventually were traced back to incompatibilities between the way the library/libraries had been compiled and the way Ardour was compiled.

I think that I now coined a term for my own dictionary, and will call this the syndrome of oogie-boogie bugs, for each time I hear (or I’m found muttering!) “we know of past bad behaviour”. Sorry but without documentation, these things are like unprovable myth, just like the one Adam commented upon (the “pyramid power”). I’m not saying that these things didn’t happen, far form that I’m sure they did, the problem is that they are not documented and thus are unprovable, and impossible to dissect and correct.

Also, I’m not blaming Paul or the Ardour team to be superficial, because, believe it or not, I suffer(ed, hopefully) from that syndrome myself: some time ago, I reported to Mart that I had maintainer mode-induced rebuilds on packages that patched both Makefile.am and Makefile.in, and that thus the method of patching both was not working; while I still maintain that it’s more consistent to always rebuild autotools (and I know I have to write on why is that), Mart pushed me into proving it, and together we were able to identify the problem: I was using XFS for my build directory, which has sub-second mtime precision, while he was using ext3 with mtime precise only to the second, so indeed I was experiencing difficulties he would never have been able to reproduce on his setup.

Just to show that this goes beyond this kind of problem, since I joined Gentoo, Luca told me to be wary about suggesting use of -O0 when debugging because it can cause stuff to miscompile. I never accepted his word for it because that’s just how I am, and he didn’t have any specifics to prove it. Turns out he wasn’t that wrong after all, since if you build FFmpeg with -O0 and Sun’s compiler, it cannot complete the link. The reason for this is that with older GCC, and Sun’s compiler, and others I’m sure, -O0 turns off the DCE (Dead Code Elimination) pass entirely, and cause branches like if (0) to be compiled anyway. FFmpeg relies on the DCE pass to always happen. (there is more to say about relying on the DCE pass but that’s another topic altogether).

So again, if you want to solve bugs of this kind, you have to just do like the actual Mythbusters: document, reproduce, dissect, fix (or document why you have to do something rather than just saying you have to do it). Not having the specifics of the problem, makes it an “oogie-boogie” bug and it’s impossible to deal with it.

Second problem: once upon a time

Let me repeat one particular piece of the previous quote from Paul Davis (emphasis mine): “Many years ago (even before Gentoo existed, I think)”. How many years ago is that? Well, since I don’t want to track down the data on our own site (I have to admit I found it appalling that we don’t have a “History” page), I’ll go around quoting Wikipedia. If we talk about Gentoo Linux with this very name, the 1.0 version has been released on 2002, March 31 (hey it’s almost seven years go by now). If we talk about Daniel’s project, Enoch Linux 0.75 was released in December 1999, which is more than nine years ago. I cannot seem to be able to confirm Paul’s memories since their Subversion repositories seems to have discarded the history information from when they were in CVS (it reports the first commit in 2005, which is certainly wrong if we consider that Wikipedia puts their “Initial Release” in 2004).

Is anything the same as it was at that time? Well, most likely there are still pieces of code that are older than that, but I don’t think any of those are in actual use nowadays. There have been, in particular, a lot of transitions since then. Are difficulties found at that time of any relevance nowadays? I sincerely don’t think so. Paul also don’t seem to have any documentation of newer happenings of this, and just says that they don’t want to spend more time on debugging these problems:

We simply cannot afford the time it takes to get into debugging problems with Gentoo users only to realize that its just another variation on the SYSLIBS=1 problem.

I’ll go around that statement in more details in the next problem, but for now let’s accept that there has been no documentation of new cases, and that all that it goes here is bad history. Let’s try to think about what that bad history was. We’re speaking about libraries, first of all, what does that bring us? If you’re an avid reader of my blog, you might remember what actually brought me to investigate bundled libraries in the first place: symbol collisions ! Indeed this is very likely, if you remember I did find one crash in xine due to the use of system FFmpeg, caused by symbol collisions. So it’s certainly not a far-fetched problem.

The Unix flat namespace to symbols is certainly one big issue that projects depending on many libraries have to deal with; and I admit there aren’t many tools that can deal with that. While my collision analysis work has focused up to now to identify the areas of problem, it only helps in the big scheme of things to find possible candidate to collision problems. This actually made me think that I should adapt my technique to identify problems in a much smaller scale, giving one executable in input and identifying duplicated symbols. I just added this to my TODO map.

Anyway, thinking about the amount of time passed since Gentoo’s creation (and thus what Paul think is when the problems started to happen), we can see that there is at least one big “event horizon” in GNU/Linux since then (and for once I use this term, because it’s proper to use it here): the libc5 to libc6 migration ; the HOWTO I’m linking, from Debian, was last edited in 1997, which puts it well in the timeframe that Paul described.

So it’s well possible that people at the time went to use libraries built for one C library with Ardour built with a different one, which would have created, almost certainly, subtle and difficult to identify (for a person not skilled with linkers at least) issues. And it’s certainly not the only possible cause of similar crashes, or even worse unexpected behaviour. If we look again at Paul’s comment, he speaks of “C++ libraries”; I know that Ardour is written in C++ and I think I remember some of the libraries being built being written in C++ too; I’m not sure if he’s right at calling all of them “C++ libraries” (C and C++ are two different languages, even if the foreign calling convention glue is embedded in the latter’s language), but given even a single one is as such, it can open a different Pandora’s vase.

See, if you look at GCC’s history, it wasn’t long before Enoch 0.75 release that a huge paradigm shift initiated for Free Software compilers. The GNU C Compiler, nowadays the GNU Compiler Collection, forked the Experimental/Enhanced GNU Compiler System (EGCS) in 1997, which was merged back into GCC with the historical release 2.95 in April 1999. EGCS contained a huge amount of changes, a lot related to C++. But even that wasn’t near perfection; for many, C++ support was mostly ready from prime time only after release 3 at least, so there were wild changes going on at that time. Libraries built with different versions of the compiler at the time might as well had wildly differently built symbols with the same name, and even worse, they would have been using different STL libraries. Add to the mix the infamous 2.96 release of GCC as shipped by RedHat, I think the worse faux-pas in the history of RedHat itself, with so many bugs due to backporting that a project I was working with at the time (NoX-Wizard) officially unsupported it, suggesting to use either 2.95 or 3.1. We even had an explicit #error out if the 2.96 release was used!

A smaller scale paradigm shift has happened with the release of GCC 3.4 and the change from libstdc++.so.5 to libstdc++.so.6 which is what we use nowadays. Mixing libraries using the two ABIs and the STL versions caused obvious and non-obvious crashes; we still have software using the older ABI, and that’s why we have libstdc++-v3 around; Mozilla, Sun and Blackdown hackers certainly remember that time because it was a huge mess for them. It’s one very common (and one of my favourite) arguments against the use of C++ for mission-critical system software.

Also, GCC’s backward compatibility is near non-existent: if you build something with GCC 4.3, without using static libraries, executing it on a system with GCC 4.2 will likely cause a huge amount of problems (forward compatibility is always ensured though). Which adds up to the picture I already painted. And do we want to talk about the visibility problem? (on a different note I should ask Steve for a dump of my old blog to merge here, it’s boring not remembering that one post was written on the old one).

I am thus not doubting at all of Paul’s memories regarding problems with system libraries and so on so forth. I also would stress another piece of his comment: “eventually were traced back to incompatibilities between the way the library/libraries had been compiled and the way Ardour was compiled”. I understand he might not actually just refer to the compiler (and compiler version) used in the build; so I wish to point out two particular GCC options: -f(no-)exceptions and -f(no-)rtti.

These two options enable or disable two C++ language features: exceptions handling and run-time type information. I can’t find any reference to that in the current man page, but I remember that it warned that mixing code built with and without it in the same software unit was bad. I wouldn’t expect it to be any different now sincerely. In general the problem is solved because each piece of software builds its own independent unit, in the form of executable or shared object, and the boundary between those is subject to the contract that we call ABI. Shared libraries built with and without those options are supposed to work fine together (I sincerely am not ready to bet though), but if the lower-level object files are mixed together, bad things may happen, and since we’re talking about computers, they will, in the moment you don’t want them to. It’s important to note here for all the developers not expert with linkers that static libraries (or more properly, static archives) are just a bunch of object files glued together, so linking something statically still means linking lower-level object files together.

So the relevance of Paul’s memories is, in my opinion, pretty low. Sure shit happened, and we can’t swear that it’ll never happen again (most likely it will), but we can deal with that, which brings me to the next problem:

Third problem: the knee-jerk reaction

Each time some bug happens that is difficult to pin down, it seems like any developer tries to shift the blame. Upstream. Downstream. Sidestream. Ad infinitum. As a spontaneous reflex.

This happens pretty often with distributions, especially with Gentoo that gives users “too much” freedom with their software, but most likely in general, and I think this is the most frequent reason for bundling libraries. By using system libraries developers lose what they think is “control” over their software, which in my opinion is often just sheer luck. Sometimes developers admit that their reasons are just desire to spend the less time possible working on issues, some other times they try to explicitly move the blame on the distributions or other projects, but at the end of the day the problem is just the same.

Free software is a moving target; you might developer software against a version of a library, not touch the code for a few months, it works great, and then a new version is released and your software stops working. And you blame the new release. You might be right (new bug introduced), or you might be wrong (you breached the “contract” called API, some change happened and something that was not guaranteed to work in any particular way changed the way it worked, and you relied on the old behaviour). In either case, the answer “I don’t give a damn, just use the old version” is a sign of something pretty wrong with your approach.

The Free Software spirit should be the spirit of collaboration. If a new release of a given dependency breaks your software, you should probably just contact the author and try to work out between the two project what the problem is; if it’s a bug introduced, make sure there is a testsuite, and that the testsuite includes a testcase for the particular issue you found. Writing testcases for bugs that happened in the past is exactly why testsuites are so useful. If the problem is that you relied on a behaviour that has changed, the author might know how not to rely on that and have code that work as expected, or might take steps to make sure nobody else tries that (either by improving documentation or changing the interface so that the behaviour is not exposed). Bundling the dependency citing multiple problems and giving no option is usually not the brightest step.

I’m all for giving working software to users by default, so I can understand bundling the library by default; I just think that it should either be documented why that’s the case or give a chance of not using it. Someone somewhere might actually be able to find what the problem is. Just give him a chance. In my previous encounter with Firefox’s SQLite, I received a mail from Benjamin Smedberg:

Mozilla requires a very specific version of sqlite that has specific compiled settings. We know that our builds don’t work with later or earlier versions, based on testing. This is why we don’t build against system libsqlite by design.

They know based on testings that they can’t work with anything else. What does that testing consists of, I still don’t know. Benjamin admitted he didn’t have the specifics, and relied me to Shawn Wilsher who supposedly had more details, but he never got back at me with those details. Which is quite sad since I was eager to find what the problem was because SQLite is one of the most frequent oogei-boogei sources. I even noted that the problem with SQLite seems to lie upstream, and I still maintain that in this case; while I said before that it’s a knee-jerk reaction, I also have witnessed to more than a few project having problems with SQLite, myself I had my share of headaches because of that. But this should really start make us think that maybe, just maybe, SQLite needs help.

But we’re not talking about SQLite here, and trust me that most upstreams will likely help you out to forwardport your code, fix issues and so on so forth. Even if you, for some reason I don’t want to talk about now, decided to change the upstream library after bundling, often times you can get it back to a vanilla state by pushing your changes upstream. I know it’s feasible even for the most difficult upstreams, because I have done just that with FFmpeg, with respect to xine’s copy.

But just so that we’re clear, it does not stop with libraries, the knee-jerk reaction happens with CFLAGS too; if you have many users reporting that using wild CFLAGS break your software, the most common reaction is to just disallow custom CFLAGS, while the reasoned approach would be to add a warning and then start to identify the culprit; it might be your code assuming something that is not always true, or it might be a compiler bug, in either case the solution is to fix the culprit instead of just disallowing anybody from making use of custom flags.

Solution: everybody’s share

So for now I dissected Paul’s comment into three main problems; I could probably write more about each of them, and I might if the points are not clear, but the post is already long enough (but I didn’t want to split it down because it would take too long to be available), and I wanted to reach a conclusion with a solution, which is what I already posted in my reply to the bug.

The solution to this problem is to give everybody something to do. Instead of “blacklisting Gentoo” like Paul proposed, they should just do the right thing and leave us to deal with the problems caused by our choices and our needs. I have already pointed out some of these in my three-parts article for LWN (part 1, part 2 and part 3). This means that if you get user reporting some weird behaviour, using the Gentoo ebuild, your answer should not be “Die!” but “You should report that to the Gentoo folks over at their bugzilla”. Yes I know it is a much longer phrase and that it requires much more typing, but it’s much more user friendly and actually provides us all with a way to improve the situation.

Or you could also do the humble thing and ask for help. I already said that before, but if you got problem with anything I have written about, and have a good documentation of what the problem is, you can write me; of course I don’t always have time to fix your issues, sometimes I don’t even have time to look at them in a timely fashion I’m afraid, but I never sent away someone because I didn’t like them. The problem is that most of the time I’m not asked at all.

Even if you might end up asking me some question that would be very silly if you knew the topic, I’m not offended by those; just like I’d rather not be asked to learn all about the theory behind psychoacoustic to find why libfaad is shrieking my music, I don’t pretend that Paul knows all the inside out of linking problems to find out why the system libraries cause problems. I (and others like me) have the expertise to identify relatively quickly a collision problem; I should also be able to provide tools to identify that more quickly. But if I don’t know of the problem, I cannot magically fix it; well, not always at least .

So Paul, this is an official offer; if you can give me the details of even a single crash or misbehaviour due to the use of system libraries, I’d be happy to look into it.

On patching and security issues

Jeff, I think your concerns are pretty well real. The problem here though is not that Debian users should not be suggested not to file bugs upstream, the problem is that Debian should not go out of their way to patch stuff around.

Of course this is not entirely Debian’s fault, there are a few projects for which dealing with upstream is a tremendous waste of time of cosmic proportions, as they ignore distributors, think that their needs are totally bogus and stuff like that. Now, not all projects are like that of course. Projects like Amarok are quite friendly with downstream (to the point all the patches that are in Gentoo, added by me at least, were committed at the same time on the SVN), and most of the projects that you can find not suiting any distribution are most likely not knowing what the distributors need.

I did write about this in the past, and you can find my ideas on the “Distribution-friendly Projects” article, published on LWN (part 1, part 2 and part 3). I do suggest the read of that to anybody who has an upstream project, and would like to know what distributors need.

But the problem here is that Debian is known for patching the blood out of a project to adapt it to their needs. Sometimes this is good, as they take a totally distribution-unfriendly package into a decent one, sometimes it’s very bad.

You can find a few good uses of Debian’s patches in Portage, it’s not uncommon for a patched project to be used. On the other hand, you can think of at least two failures that, at least for me, shown the way Debian can easily fail:

  • a not-so-commonly known failure in autotoolising metamail, a dependency of hylafax that I tried to run on FreeBSD before. They did use autoconf and automake, but they made them so that they only work under Linux, proving they don’t know autotools that well;
  • the EPIC FAIL of the OpenSSL security bug; where people wanted to fix a problem with Valgrind, not knowing valgrind (if you have ever looked at valgrind docs, there is a good reference about suppression files, rather than patching code you don’t understand either).

Now this of course means nothing, of course even in Gentoo there has been good patches and bad patches; I have yet to see an EPIC FAIL like the OpenSSL debacle, but you never know.

The problem lies in the fact that Debian also seem to keep an “holier than thou” attitude toward any kind of collaboration, as you can easily notice in Marco d’Itri’s statements regarding udev rules (see this LWN article). I know a few Debian developers who are really nice guys whom I love to work with (like Reinhard Tartler who packages xine and Russel Coker whose blog I love to follow, for both technical posts and “green” posts; but not limited to), but for other Debian developers to behave like d’Itri is far from unheard of, and actually not uncommon either.

I’m afraid that the good in Debian is being contaminated by people like these, and by the attitude of trusting no one but themselves in every issue. And I’m sorry to see that because Debian was my distribution of choice when I started using Linux seriously.