Broken every other week

I’m seriously disappointed in Gentoo; even after my last post on the topic the Gentoo quality seems to be on a downward spiral, rather than climbing up. The problem is social: too many developers don’t give a f♥♥k.

Let me be clear: I’m not pretending that we should be never wrong, and that mistakes can’t happen. We are, they happen. This is why we have the unstable/testing support. But at the same time, we should still try to think twice and commit once.

Hey, I screwed up Friday night. I launched a double-commit without checking its status, committing virtual/ruby-ssl and www-servers/thin… but the former’s commit was rejected by repoman and I didn’t even notice. Thankfully Mr Bones was around to notify me, and Luca was able to commit a workaround for me until I got home (I was at my sister’s for her birthday dinner).

So what kind of problems upset, demoralise and disappoint me? It’s the most pernicious ones; like those we’ve been warning about for years, those that we could see coming from a mile away, and those that are caused by sheer lack of testing for important packages.

Say you’re a developer who’s already been warned to doublecheck what you do, and asked to use package.mask before committing big changes regarding the language the official package manager is written in. When you’re just disregarding this, trying to force a new version of the language altogether; ignoring the whole (known) problem of dependencies on multiple implementations, … you probably are screaming as loud as you can “I’m going to break the tree sooner or later”.

Say you’re an old-time developer that was known to do a relatively clean work albeit scarcely documented, and you start making mistakes like forgetting to update your own wrapper when bumping autoconf (sure it’s masked, but it means you’ve not even barely tested the thing); say you actually commit half-broken ebuilds for a while; I can’t stop wondering what is wrong with you when you actually blame me for not covering your (scarcely documented at all) use case with NFS when I actually fix it to work for Kerberos (before my involvement, it didn’t work for either case, after my original involvement, it covered one case fully). Do you still think you should be the one voicing opinions about the Quality Assurance team?

And if you say that “the new GLIBC works for me”, are you saying that the package itself builds or if it’s actually integrated correctly? Because, you know, I used to rebuild the whole system whenever I made a change to basic system packages when I maintained Gentoo/FreeBSD, and saying that it’s ready for ~arch when you haven’t even rebuilt the system (and you haven’t, or you would have noticed that m4 was broken) is definitely something I’d define as reckless and I’d venture to say you’re not good material to work on the quality assurance status.

I’m just tired, very very tired, of repeating the same post over and over again. Gentoo needs stronger QA. Full stop.

25 thoughts on “Broken every other week

  1. When your most powerful team, aka QA, is keeping her lips sealed on every Gentoo screw up, something is seriously wrong. You are not the only one who gets demotivated from time to time. Each one of us feel the same, because QA, as an entity, is unable ( or perhaps unwilling) to kick some asses that don’t seems to follow pretty standard QA rules.

    Like

  2. Right now as I see we are a step (basically more hardware involved and possibly more people) to get a _really_ wonderful set of Q/A checks. (Yes I’m one of those that looking at an 1/2 empty glass think about the bottle next to it).Helping usually takes as much time as complaining and is more rewarding even if takes more mental energy, I know. Right now your system is able to catch quite easily lots of bugs, I know we should be _more_ careful overall and I wholeheartedly agree.Still the solution to the problem, beside having more paid time for Gentoo (cue here the axant experiment and your “for hire” flags), is having more people able to devote their free time helping on those tasks and perfect the tools that might let use use this time more effectively.

    Like

  3. To answer the question of whether something not only appears to work but is “integrated correctly” most of the time takes some very specialized knowledge that I bet most devs don’t have about most of the packages they maintain. It’s just impossible for you to know the internals of every package to the degree required to start talking about “correctness”. IMO Gentoo is unmaintainable for that reason.

    Like

  4. @smorg, gentoo works pretty well for many people. Diego’s efforts are toward making it better preventing occasional slips, no matter how seldom they are.

    Like

  5. I think gentoo needs to be more modular. It’s supposed to be about choice and flexibility, but as soon as I try and use clang instead of GCC, or replace any part of the core system, it becomes clear that the gentoo devs still have a very narrow view of the final product. As soon as Daniel Robbins left, it all went to hell.

    Like

  6. smorg → “correctness” in the case of the system C library would be “it a t least leaves the system set building and running”; glibc 2.12 does not work this way.Duck → sorry? do you know _when_ Daniel Robbins left? And can you see funtoo actually growing a clue on how to do things? “Choice and flexibility” require things to work. Half the packages won’t build with clang, full stop; it doesn’t mean that we won’t ever support doing that, but unless you actually start working on them, you can’t pretend Gentoo makes it magic to support it.

    Like

  7. Yes, weak QA is very disappointing. It’s just one of the many problems Gentoo has. I was told the QA team has got more powers now. So why don’t you use them?As I see it, this is just one more symptom that shows that Gentoo is, slowly but surely, dying…

    Like

  8. As a Gentoo user for more than 6 years this is a very depressing thread. Now I see why the number of updated that have actually broken my installation has increased over the last year and a half.At this moment my computer will not complete boot-up. Why. Well in emerge -uD world, it the a msg. that ivman is being dropped and to start using halevt. So I did. After such a emerge I check elogviewer to see if there are any special instruction that need to be implemented to insure a stable system. There where none so I rebooted the computer and there it is.I don’t expect anyone here to fix my problem. I’m competent enough to fix this myself. But this just the kind of thing that has become more frequent in Gentoo.So I get what Diego is trying to say.May developers should not have the power to submit there own changes. Maybe all changes should go through a peer review process? Maybe you need to support fewer packages in portage so that the remaining packages get more attention from the developers?I’ve not plans to leave Gentoo, but breaking your distribution is really an unforgivable offence and should be so rare that few ever happen.

    Like

  9. Ben, the powers have been given to the *QA lead*… you might not like my ideas of QA, but have you _ever_ see the current lead do something at all?

    Like

  10. I’ve been using gentoo on a few production servers for something like seven years and on my laptop/home pcs for even more time. I have to admit that reading your blog has saved my (professional) life in more than one occasion (I’m joking, but… not far from reality) and that you’re causing me to grow a really paranoid attitude in emerge upgrading the world. When I noticed the version bump on glibc I couldn’t resist masking it. Now I know that my spider senses where right! :)

    Like

  11. Why not to fork the whole Gentoo project?I mean, if too many people don’t give a f… ,then the solution is to get rid of them… …but since the people that don’t give a f… are in control, the only thing you can do is to fork the project ,or leave (don’t do that!!! :-) )

    Like

  12. This thread and comments just confirm one thing to me: It is easy to complain about the negative (even I do it too). Just like all the news stations only report about the horrible deaths and accidents.

    Like

  13. My suggestion about forking Gentoo was maybe “too strong”, but acting like everything is all right, will not not help either.In my opinion Gentoo is at the moment somewhere between debian testing and unstable (quality wise).The big problem is that, you can’t set up gentoo as run & forget server (e.g. it does not have automatic updates as debian/ubuntu/fedora/openSuse). You always have to watch your servers for something to break during updates etc… (and thats not what I call “stable”, and thats what should be solved in the firts place)

    Like

  14. Perhaps an adaptation of the kernels commit to Gentoo commits. Each commit must be signed off.Of course that will slow stabling packages.How does BSD address this?

    Like

  15. Jeremy, do you think we should just be saying “lalalalalalalaaaaa Gentoo is fineeeee”?I’ll be clear once again: we can do all our heck of a good job but ONE person doing it wrong is going to make a mess out of things. Take Python 3; how many people take to break it? And how many to get it right again? Answers: one and three (me noticing, Brian getting the fix ready, Petteri blocking the account).You can get 1000 good commits a day for a week, and then have a bad one at the weekend; next week, Gentoo will be broken. It sucks, but so it is :/

    Like

  16. Diego, heads up! ATM a negative point of view doesnt solve things and only adds stress to you. There are too many users following your blog and agreeing with your ideas with respect to Gentoo’s QA processes. In the end, we really do depend on you.I might have asked the question already, but is there a way for non-devs (meaning gentoo users) to voice some support to a particular cause, such as a stronger qa, up to the council?

    Like

  17. Regarding python it was my/our fault for not complaining loudly enough and not stick a p.mask for python-3 and defend that.About the Q/A lead and Q/A powers… we should try to get it right….

    Like

  18. Flameeyes: Only the QA lead? That is disappointing. I thought to QA as a team. And I agree with nico, it sounds to me like it’s time to elect a new QA team lead. It needs to be someone who is very active in the distro, like you, scarabeus or hwoarang.And I may not always agree with you, but in general I do like your ideas about QA. At the least they are a big improvement over the current mess.Havis: actually, a fork may be the only way out. There currently are some people who really want to turn Gentoo around. Some of them are in the new council. So there is hope. But there is also a lot of inertia, so it is a difficult task. Maybe learning our lessons and starting over is not such a bad idea.Jeremy: so see it as a call to action. Identify problems, propose solutions and implement them.Pavel: anyone can bring a topic to the council for discussion. You can react to the announcements made for council meetings on the gentoo-project mailing list.lu_zero: this is one of the problems with what I have called the lack of leadership in Gentoo. One pushy and uncooperative dev can cause a lot of damage, and nobody steps in until it is too late. I actually had the same idea at the time about masking python-3, but I didn’t want to get into a commit war. Especially since I’m not a member of the python team.

    Like

  19. so it seems ‘static’ at least is reported broken with glibc-2.12.1 In addition if your user has updated and now tries to revert to an earlier glibc via package.mask emerge fails on sanity checks without asking after warning the user. nice portage ;-/ I should now have to revert to a locally installed source since portage decides I am too stupid to downgrade though only m4 was updated afterward.

    Like

  20. There are queite a few things I strongy agree with. But I switched to gentoo about three years ago and never ever regrettet that step.For a few problems that appeared in the past, there were various nice ideas for improvments to portage/system packages. Not all of them were adopted.Maybe instead of a ‘louder’ QA team we need better motivation for those taking care of such things?For my taste, I also do not see enough http://flattr.com/ buttons :P

    Like

  21. Why does this sound like we need, among others, a new method for submitting ebuilds; one that automatically adds the submitted ebuild to package.mask.Of course not for everyone, but for all the people who are known to slip up too often… It would be a very *welcome* safeguard, to say the least.

    Like

  22. I’ve just come back to this thread. I see there has have been many comments. Again I’m just a user though I’ve been using it since 93′ or was it 96′. I’ve got no experience with the dev world and I can not pretend that I know the difficulties and compromises it involves.I do know that I do not support the idea of forking gentoo. My idea of gentoo is that it was alway a bit cutting edge. That always has its inherent risks. But careless and stupid breakages the are unforgivable.There is saying in he business world that many managers get promoted one level too high. I guess this is true of for devs and team leaders.As far as addressing those that at present are in control of thing at gentoo. Clearly there is a communication problem. It the job of the people who want change to convince those in charge that it is necessary. I do not mean this to be an absolute, but it you can’t convince the people you need to convince that your new way of doing things is better or more sound. Don’t blame them. It is your job to convince this with sound rational arguments. Address the issue/s without pointing fingers, recrimination or denigration the individuals involved. Your arguments will need to be clear, concise and find a way of bring the leadership into your process. If you have to pretend that you are at a company you have been told that you have one chance to make a presentation that will save the jobs of yourself and 50,000 others in your division. The company has no finances for buy-outs or golden handshakes, that the coffers for pensions are empty.As previously stated, complaining is easy, but to attempt the need for change as stated above is infinitively harder.So it begs the question. Who in the dev community is up to the challenge?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s