What’s wrong with Gentoo, anyway?

Yesterday I snapped and declared my intent to resign from Gentoo together with stopping the tinderbox and leaving the use of Gentoo either. Why did that happen? Well, it’s a huge mix of problems, all joined together by one common factor: no matter how much work I pour into getting Gentoo working like it should be, more problems are generated by sloppy work from at least one or two developers.

I’m not referring about the misunderstandings about QA rules, which happens and are naturally caused by the fact we’re humans and not being of pure logic (luckily! how boring it would be otherwise, to always behave in the most logical way!). Those can upset me but they are still after all no big deals. What I’m referring to is the situation where one or two developers can screw up the whole tree without anybody being (reasonably) able to do a thing about it. We’ve had to two (different) examples in the past few months, and while both have undeniably bothered QA, users, and developers alike, no action has been taken in any of these cases.

We thus have developer A, who decided that it’s a good idea to force all users to have Python 3 installed on their systems, because upstream released it (even when upstream consider it still experimental, something to toy with), and who kept on ignoring calls for dropping that from both users and developers (luckily, the arch teams are not mindless drones, and wouldn’t let this slide to stable as he intended in the first place). The same developer also hasn’t been able to properly address one slight problem with the new wrapper after months from the unleashing of that to the unstable users (unstable does not mean unusable).

Then we have developer B who feels like the tree’s saviour, the only person who can make Gentoo bleeding edge again… while most of if not all of the rest the developer pool is working on getting Gentoo more stable and more maintainable. So, among the things he went on doing, there was a poorly-performed Samba bump (suboptimal was the term he used — I ended up having to fix the init scripts myself because they weren’t stopping/restarting properly, as the ebuild and the init scripts went out of sync regarding paths), some strangely incomplete PostgreSQL changes, and a number of minor problems with the packages.

Of the two, I was first upset most by the former, but on the long run, the latter is the one who drove me mad. Let’s not dig too much on the stance about --as-needed (cosmetics — yeah because being able to return from a jpeg bump with less than 100 packages, rather than the whole world, is just cosmetics), and the fact that he’s ignored most of the QA issues with the packages he touched. Instead look at the behaviour with a package of mine (alas, I made the mistake of let this one slip with just a warning, I should have taken the chance to actually defer it to devrel…): vbindiff.

The package is something I added a while ago because from time to time it comes out useful. I’m in metadata.xml; I’m definitely not an unresponsive maintainer. Yet, while my last bump was on June 2008, the version in tree was not the latest one up to last September (2009). Why? A quick glance at the homepage shows that the beta4 release was mostly fixing a Win32 bug, and introducing a way to enable debug-mode. So what happens? Our mighty developer decides to go on and bump the package; without asking me; with nobody asking him; without a mail, a nod or anything. I literally notice this as emerge tries to upgrade a package I know I maintain. You’d expect for the debug support to be present in the ebuild then, and you’d find a debug USE flag if you checked now indeed, but that’s something I added myself afterwards, as the damage of pointlessly bumping something was already done.

Now, why did that happen? Well, he admitted he just went through the dev-* categories, without considering maintainers declared in metadata, and blindly bumped ebuilds when the latest version available on the site was higher than the one in tree. Case in point he had to open the vbindiff site and thus the release notes regarding Win32 and --enable-debug would have been clearly visible, if he cared to even read part of them. Whoever tried doing serious ebuild business should know that most of the time even the upstream-provided release notes are not something to go on by… Interestingly enough, his bleeding-edge hunger didn’t make him ask for a new stable, and we currently have a very old one.

So there we have your developer B, the super-hero, the last good hope of the bleeding edge, who bumps packages without consulting the guy who maintain them (and is around almost 247) and without even caring to use them at all. Why did I let it slip? Because I was most focused on trying to stop developer A at the time is probably the right answer. I did issue a reprimand reminding him to not touch someone else’s packages, and to learn using package.mask for things like Samba. I was hoping he would listen. Oh boy, was I ever so wrong.

Speaking a second again about Samba, did I tell you yet that the split into multiple packages was done, straight to ~arch, without any plan to follow-up to convert dependencies? Wonder why the whole thing is now stalemated again. Maybe the arch teams don’t see it all too well to have the same kind of dependency breakage in stable as there was/is on unstable right now.

First-hand information about our developer B wants him to be inlined with a zealot point of view regarding the Mono project — you’d then guess that dotnet stuff would be the last thing he’d be touching, but instead, without any questioning, ignoring the fact I stated at FOSDEM that I was going to look into that as soon as I had time, the fact that I stated before multiple times that I was already working on un-splitting the gtk-sharp packages, and the fact that I took contact with the Mono developers (again at FOSDEM) to try following upstream more closely. Oh and the one thing that pissed me off about that bump? Beside the fact that tomboy now refuses to work? Remember this patch? It was dropped; without even mailing me if I had or could make a version for the latest version. It was dropped in unstable (or, how it should be called if this kind of stuff is allowed to continue, unusable).

And the cherry on top? As I said, this developer touched Samba, PostgreSQL, now Mono… there are three aliases for these things (samba, pgsql-bugs and dotnet), who the bugs are assigned to… he’s on none of them! And before somebody tries to argue that, I’m pretty confident he’s not following the aliases on the Bugzilla (plus, given he also argued that the problem was with leaving security-vulnerable stuff in the tree – which by the way means having working, complete, safe ebuilds to be able to mark stable, and he doesn’t seem to be able to come up with any of those – the most important security bugs don’t get sent to watchers). How does he suppose to see the bugs coming? Oh but by wrangling the bug himself! Yeah, after all developers don’t file bugs themselves assigning them straight to the maintainers by procedure, do they? (fun fact: Bugzilla queries report at most 5K bugs, so that list is a very much limited result from what I was hoping to get); nor do other developers ever wrangle it would be silly, and there is no Arch Tester to speak of, right?

You can now see most of the pictures, and why I’m mostly upset with developer B. What made me snap yesterday were remarks that insisted that I was just “whining” and “not doing enough” as bugs kept piling up. What the heck? I constantly had over 1000 bugs (over 1300 today) for the past year or so, I know very well that bugs keep piling up! And I’ve been doing all I can do outside of my work hours (while I have to thank some people, including Paul, David, Simon, Andrew and Bela for their contributions, I’m not paid to do Gentoo work; and while I do get to use it, and thus contribute back to, for some of the jobs I take, it’s definitely not the same as working on Gentoo), including the whole RubyNG porting and improvement trying to make sure we can actually get to a point where unmasking Ruby 1.9 will not break any user whatsoever. Am I really doing too little? ”Not enough”?

Okay so the proper way to handle this, with the current procedures, would be to take this up to the Developers’ Relations so that they could act on it; QA can only ask infra to restrict commit access if we’re expecting a grave and dangerous breaking of the tree, or misuse of commit rights. So why didn’t I bring this up to devrel? Well, the main reason is that devrel nowadays, as far as I can tell, is exactly three people: Petteri, Denis and Jorge, and of the three the only one who’s for preventive suspension of commit rights is Denis (this has been proven with the case about developer A above); one out of three does not really sound much of a chance for this to improve the situation. And if – again as happened with developer A – DevRel then decided that the right action would be to issue a reprimand, that would amount to scolding the developer and asking to work more with others… well, it wouldn’t change a thing.

The whole QA system has to change! We’ve got to write down guidelines, rules, and laws, and be conservative in applying them. You shouldn’t go around breaching them and then appealing when QA finds you out of line, you should talk with QA if you feel the rule is misapplied to your case in any way.

So here you go, in a nutshell, why my preservation instinct right now is telling me to flee. I’m not sure yet if I’ll outright flee or just give it time for the situation is addressed and then decide. The reason is: I still like the Gentoo system, and since I rely on it for my work I cannot leave it alone; if I were to move to anything else I would have to spend (waste?) even more time to fix the same issues anyway, and I’d much rather get Gentoo working right. But I cannot do this alone, I cannot do this especially if I have support neither from developers nor users. So please voice your concern.

If you feel like Gentoo needs the better QA, if you feel like we shouldn’t be translating unstable to unusable, then please ask for it. I’m not saying that we should become stale like Debian stable, but if it takes a few months to get something straight, then it should take its time and not be forced through (that’s what the Ruby team has been doing all this time to work with Ruby 1.9 and Ruby EE and other implementations as well!). If you use Twitter, identi.ca, Digg, Reddit, Slashdot, whatever, get this post running. Maybe I’m subverting the process, but to quote BBC’s NewsQuiz, “Trial by media is the most efficient form of justice” (this was in reference to the British MP expenses scandal last year), and right now my only concern is effectiveness.

89 thoughts on “What’s wrong with Gentoo, anyway?

  1. Since I’m using pgsql for important stuff I decided to go see who “Developer B” was… and I’m not surprised at all.Maybe it’s time to ditch this sinking ship for Exherbo. I’d personally rather deal with assholes who know what they’re doing and have low tolerance for BS — than ones who don’t and are full of it.

    Like

  2. I’ve been on Gentoo since 2003, it’s really been a while and many things have changed since.My opinion is also that unstable should not be unusable. It used to be bearable back in the days because packages would quickly move to stable and one just had to be a bit patient. Nowadays though, the stable branch seems almost out-of-date, probably because there are so many packages and so many few devs (I won’t pretend that adding devs will solve that problem though) and because the other distros are providing them so quickly too. So if one wants to keep his system up-to-date and doesn’t have 3 hours a week to spend on managing the package.keywords file, one needs a usable ~arch tree.It’s not the right place to talk about that but just like other people who commented before, I do think it would be a good idea for Gentoo to *officially* provide some sort of bleeding-edge tree (for the strong-of-the-heart).I’d like to go on and on about how the political system of Gentoo seems to be leading us nowhere but that’s not my place to rant, that’s yours! Please just don’t do something reckless and think about it calmly before really resigning. Gentoo users need people like you, and as much as you might hate that, they might also need you to push for a change because there will undoubtedly people behind you if you choose so.

    Like

  3. Oh, my. You really released something here, didn’t you? The comments are pretty depressing I must say, some trolls, some bullshit (arch being stale so you must run ~arch? yeah, right…), and some standard FUD (yeah, I rather use Gentoo on an production machine then some other distro that does not even support themselves and/or does not know how to handle licenses).That said, I hope you stay with us. I feel the QA and other work you do is really needed.

    Like

  4. Hi Diego, I have only one comment :Your work with Gentoo is awesome and I really appreciate what you have done in the past and hopefully what you will do for Gentoo :PYou really make a diff IMOO.

    Like

  5. Update: As I feared: OOM – I’d need at least 4 GiB physical memory I think, and I have only 3. I’ll see what I can do about that (and in the meantime convert the individual cvs modules).

    Like

  6. Frankly, while I agree with samba situation being bad (they’re moths old bugs with patchesjust waiting for a review and being applied), I would regret dropping split ebuilds.At very least, if we get back to the merge build, it should provide similar functionality, that isability to build only the libs, as some of us need just that from samba.

    Like

  7. Wow! I think it really put on me the pretended worries.. Well, let’s say I can motivate an user group to sign a request for something, to be sent to the right people.1) what something it could be? Ask the problematic developers to be shredded from the planet surface? Ask a judgement process to be started? Ask the very suspension of the commit rights for these foes?2) who wold be the right people for whom that possible pettiton should be addressed?

    Like

  8. I am a software developer who’s been using gentoo since 2003.I always had to ~arch some packages, follow some overlays and bump some ebuilds here and there as needed by my developments but it never was a big issue.I have always been amazed by the fact that at any point in time it’s easy to quite effortlessly upgrade gentoo systems installed years before, without ever getting in the need of reinstalling from scratch or of following some major release schedule.I really appreciate the fact that gentoo is flexible *and* stable *and* easy to use *and* well-documented.Losing even a single one of those features for the sake of a version bump is nonsense and seriously harms gentoo.The procedure should be:* people needing bleeding edge stuff should file bugs, preferably with new ebuilds* the dev team should review and consider such contributions for inclusion in the tree* the QA process should be improved and strengthened, also by means of proper and *full* dependency tracking, as often proposed by Diego* discussion should be fluent and thorough, arrogance should be banned* developers abusing their commit rights in a careless and clueless way should be kicked out of the dev teamPlease Diego reconsider your decision, I am fully convinced you are right and I really appreciate your hard work and insightful posts.Please dev team don’t cut corners for the sake of a blind version-bump race with other distros.Please users (including myself) support more our beloved gentoo developers, help QA unstable stuff and submit support for bleeding edge stuff we’d like to see included in the tree soon.Please all gentoo stakeholders let’s keep on learning to work well as a team, that’s always been our strength in the past and that’s where our strength will come from even more so in the future.

    Like

  9. From the perspective of “solving” these issues I think that the way a maintainer is listed in a textfile rather than enforced through a set of rules is like asking to complications. A suitable process here might be to allow any dev to work with any package, but systematically hold on the commits so that the maintainer(s) get to approve or deny any commits that touch their work. That way changes won’t come as a surprise and although it’s an artificially slowing process, it is so to prevent damages.Just my 2 cents on the “how do we make this situation better”.

    Like

  10. So that is why everything completely borked this time i tried to update things. Let the bleeding edge guys live eat and breathe autounmask. I would like to have a nice mix bleeding edge and ease of install/use with my ~86. And I am fairly certain that stable users would want things to work and be rock solid, at the expense of the latest greatest on their systems.

    Like

  11. You sound a bit overloaded. Don’t feel as though you have to be a one-man-band, and be responsible for everything.My advice is to recruit more developers, and delegate some of your work to them. Then you’d have more time free to deal with organizational and structural issues like this.The Gentoo organization needs to grow, and I think we need you to contribute to leading it.

    Like

  12. Yay, so…. didn’t take much time to prove Diego’s point. Bug closed yet again as duplicate, all is fine, an editor (Gentoo’s default one) that only prints junk on your screen is a feature and there’s no need to bother with documenting the behaviour of USE=debug there, it’s much more important to have USE=justify described like this: “Toggle the justify option” in metadata.xml. So much for useless usage of metadata.xml by certain developers.

    Like

  13. I would hate so see you leave..Gentoo is afaik unique in havingthe possibility to have a stable platform with a few added (unstable extra’s) and the ability to add yourown without too much trouble.I do value that. And yes i have been surprised in the passed a few times and rebuild a system.The one name that all those year struck me as a sign of quality and drive to make things better without leaving a wake full of wrecks is ‘flameeyes’ .My main worklife is OpenVMS based (quite a different beast) after that i have some off-line life as a life-guard after that i fill my time (left) with doing odd things with computers, like maintaining my private & (one man company systems) based on gentoo. After other distro’s broke a lot after having to add other packages not supported through the “official channels” Gentoo is the best fit I found. I would like to see your name mentioned in the future too together with gentoo…Now I might be of some help, I do issue the odd bugreports i might be able to help a bit. Yet i cannot commit to become a developer some how i think i would take too much on.

    Like

  14. Just wanted to let you know your efforts are very much appreciated and that it would a real shame for gentoo and its users if you left,

    Like

  15. Gentoo seems to have been suffering from bugs and quarrels for a long time. Therefore it seems odd that so many Gentoo users and developers have been loyal to it despite all that. Gentoo may be ok, but, still, why don’t more of those people interested in source-based distributions try some of the many source-based alternatives to Gentoo, like Source Mage GNU/Linux? Many things in SMGL infrastructure seem more sensible to me, it’s technically simpler, easier to maintain, yet has a good and Debian-like social contract and a friendly atmostphere AFAIK.

    Like

  16. now, reading through your post and the comments, I really don’t think I need to write it all down again. I agree with Bryan Jacobs (and others :D ). Also I know, you are having a lot of work, so if you decide to stay, I’m willing to help. I’ve been learning about the core system, and maybe I finally can get around to help someone around here. Well you got my mail.Well good luck and I hope that no matter which decision you will make, will brighten your days.

    Like

  17. I feel your pain.. I recently did a long-overdue emerge -uDN world and got blind-sided with both Python 3 and the bloody jpeg library issues at once. And lets not start on the whole xcb catastrophe a while back. I’m just glad I happened to have temporarily removed samba a couple months ago.I’m certainly no developer, but I’ve been with Gentoo for years and I’ve always hated seeing these (stupid) issues that end up driving off good developers and supporting users. All the best to you, whether you stay with us or not.

    Like

  18. Well. I guess that my point of view is dead simple.As a developer for any(!) product/software/driver/or anything else, you’ve got to take care. You’re responsible. And if you’re forced to break something, do it so noone else is hit by your changes. That’s what VCSs are made for…If you’ll break something in the development phase, it will hit every user that uses the devel-tree.However… Breaking things up due to not testing, ignoring common rules and not being able to take responsibility for this is in my opinion simply (censored, I won’t speak this out, too many children could read this…)Diego, I think that you’ve done good work, like to read your blog. Giving up would be the wrong way. If too many children stop playing in the kindergarten, but do it in real world and noone stops them, it will be a real big loose for Gentoo and Gentoo will really be unusable.Thanks a lot!

    Like

  19. Hey Flameeyes, keep it up! I always appreciated your good work and enjoyed your visionary posts as well. Gentoo needs devs like you.

    Like

  20. Hey Diego, no matter where you go, what you do, there will be A’s and B’s. Things will never be as good as you are able to make them, simply because of the fact that there aren’t many developers as good as you are. The A’s and B’s won’t even get what you’re talking about half the time, and usually they’ll care more about how they look than about the work itself.So given that this sort of crap will always be out there, the trick is not to let it stop you from enjoying yourself. Focus your efforts in the place that makes you happiest. Avoid the temptation to fix things that bug you just because you can. Let the rest of it take care of itself, including letting them stuff up badly.It would be a huge loss for Gentoo if you resigned. But maybe it would be good for you, have a break, think about what you want to do. When you work with the same group or at one workplace for a while responsibilities can pile up just because you’re capable, and things can get way out of control. Sometimes the only way of dealing with that is to step away. You can always come back.Sorry for the lecturing tone, I don’t mean to tell your what to do, nor am I encouraging you to leave Gentoo; that would be a great loss. I’ve really enjoyed your work and have learned heaps from it (thanks particularly for your autotools intro). I hope you see this as a message of support, because that’s what it’s supposed to be. Take a load off, mate.Cheers,Gid

    Like

  21. Sorry to hear you are leaving Gentoo. The community is losing a really valuable contributor.But if you are looking for a new distro, try Fedora. It’s a very nice distro, especially for people who want to contribute back. There’s an active community of people surrounding it, easy to help out with development and everything is very transparent and open.

    Like

  22. I don’t know the full story, so i’m going to refrain from commenting on the issues you brought up. However I do know you’ve always been around the past years, so it saddens me to see you go. Hence I can only hope this gets resolved over time.can somebody magically fix all these dev-issues please, ktnxby.

    Like

  23. I’ve been using gentoo as primary OS since 2002 on everything I have. During that time frame I have seen ~arch go from broken to usable with bugs. I agree fully that ~arch should be buggy but not broken, and it has taken quite a few years for that to happen. For one developer to want “really” bleeding edge, perhaps another version of ~arch should exist for the completely untested builds, something like $arch? I do want at least somebody to have tried to build an ebuild before tossing it into the mainline tree as ~arch!

    Like

  24. I just hope you stay. I have read your blog posts for almost a year now and I really like them. I also appreciate your tinderbox efforts to make gentoo a more stable system; nobody likes a broken system. On the other hand I understand the desire of those developers A and B to have most recent system. But of course their means were kind of suboptimal. Just running around the portage tree and altering everything in cowboy style is not the way to go. I totally agree that unstable should mean “not stable” and not unusable.

    Like

  25. Good idea would be to talk directly to A and B and end up in one or other diplomatic solution. Remember that both you and they believe that they make better Gentoo. All those common desires are failing just because misunderstanding of each other: so sad. :(

    Like

  26. I’ve being using linux based systems since 1998 and have tried most of the popular distributions (SuSE, Red Hat, Mandriva, Debian, Slack + derivateves). Since 2005 I’m periodically trying to install Gentoo on my PC seduced by the tales of its “supreme efficiency”. Never have I succeeded for more than 2 consecutive months (last 5 months ago) to keep such a system because after weeks of compilations and configurations a change is coming and the system is becoming unusable. Nobody expects Gentoo to be as stable as Debian but the current state is out of any limits. You are right that changes MUST be done and the changes must be in the organisation of work. What is the meaning of very efficient/bleeding edge system when it is working only half of the time and the other half it is not functioning?

    Like

  27. I’m using Gentoo since 2003 and I’m totally with you on this matter. I always prefer stable over bleeding edge, specially when I use it on servers and downtime and failure is not tolerable.

    Like

  28. I see no trouble to using Python 3 and any other fresh code. But using ONLY Python 3 is bad idea, until python 3 not supporting crosscompiling (in september was so) – it may break Gentoo cross-development logic.

    Like

  29. These dudes who make Python 3 the only Python available are just purely ignorant of the fact that a very large chunk of Python development still happens in Python 2, and this is not going to change any time soon.

    Like

  30. On issues like this there always has to be a level attained that is some where in between the two extremes. The line drawing is difficult to say the least. However, it is not acceptable to establish a known set of rules that allows breakage ..any breakage at all. The level of QA that needs to be in place should _garantee_ that no changes to the production tree breaks anything else in the tree. If that can not be done, then the whole gentoo way needs to be simplified so that it _can_ be done.

    Like

  31. I was one who spoke out against python3 as well. I felt like I was talking to a fence post. Sometimes I wonder if speaking up on the dev mailing lists is even worth the effort.

    Like

  32. This is the main reason I switched from gentoo to archlinux, although I did it quite some time ago (I switched over for similar reasons, however in this case it was due to KDE4 being released)Gentoo was great 5-10 years ago? However it has its own practical problems (building everything from source is very time consuming and having to find out all the appropriate flags). I do however respect that certain people either need to or wish to use such a system.However the other thing that is killing gentoo is the politics going on behind the scenes, and the problems that it is causing (and it seems to be still happening by the likes of it).Archlinux basially did everything what gentoo used to do, bleeding edge, lots of packages (through AUR) and a great development distro. Only difference is arch is binary based (although it has a very easy way of building through source using ABS) and Gentoo is source based.I guess the other difference is, Archlinux works almost all of the time (and if problems arise they are easy to fix), Gentoo is another ballgame

    Like

  33. All our servers and Desktops are Gentoo, actually was Gentoo. Since the rc converting and KDE upgrade problems… we can’t solve the problems in a timely matter – so we moved to Kubuntu with all Desktops. Now we facing emerge issues with our servers. Lately Gentoo becomes a pain where I loved so much and now what?

    Like

  34. I just installed calculate linux 15.12 xfce. From what i can tell people are really excited about it I’m very impressed how easy it was to install. And I’ve finally got a working gentoo based system that I am happy with that i needed no help with and was entirely worthwhile(timewise). I had already decided that I wanted to make a website using ruby within gentoo and no matter what it takes i will achieve that goal :) Please don’t give in everyone who loves gentoo and what it stands for really loves it. As far as I’m concerned gentoo is linux and if you give up on gentoo then i give up on linux. Just the way i feel.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s