The five-minutes-fix myth

One of the complains I often get for my QA work (which I have to say is vocally not appreciated even by the other devs), is that I could just go on and fix the issues I find rather than opening but for them because “it just takes five minutes from bug to commit” to fix the problem.

No it does not take five minutes to fix something, I can assure you!

Of course people will continue to say that it just takes a few minutes to find the problem and come up with a patch; the problem is that most of the time, for the kind of bugs I report myself, to fix them properly takes much, much more.

Most of the time that some developer decides that some single problem does not warrant to remove a package, even if it doesn’t have anybody looking after it, the same packages re-enter my radar at the next round of tinderboxing, or in the best of cases, a few months later.

The problem is, when a package has a maintainer, that maintainer is supposed to keep the package in line with the development policies; if you don’t maintain a package but commit a five-minutes-fix you’re not ensuring it complies with the policy but you’re fixing the single symptom I reported, which most likely is just the one that I hit first.

When I (and I’m not boasting here, but just stating how it is) fix a package, I do things like removing older versions, making sure it complies with all the policies, check all the bugs the package has open, check for things that there aren’t bugs for, like CFLAGS and LDFLAGS being respected, the files not being pre-stripped, features that might be missing, version bumps requirement, correct dependencies, and so on so forth. It’s not a five-minutes work! It’s often a multiple hours work instead!

What is upsetting me, to return to the fact that Gentoo’s QA problem is with developers is that some developers think it’s fine to remove a package from the QA removal queue just because they fixed the last bug I filed. I’m sincerely considering the idea of making it policy that if you wish to save a package from QA you got to fix all the problems with it, and take maintainership of it if it breaks again. And for those ignoring these, we should definitely enforce some kind of penalty in form of not being able to “save” a package from removal any longer.

8 thoughts on “The five-minutes-fix myth

  1. Have noticed the QA flags for insecure permissions on installed files. Seen no mention of in your blog. Worth adding to tinderbox checks?


  2. I agree with you, workarounds must not land in the portage tree or, at least, *not as a definitively solution*, but your proposal “we should definitely enforce some kind of penalty in form of not being able to save a package from removal any longer” is not viable, because it leads to slackerism: in the end devels will tend to ignore *all* the last rites problems.Instead to impose a kind of severe penalty, I’ll prefer to just fix the “developer problem” once and for all:1. blocking all the command-line commits into the cvs repos and forcing them through the repoman tool (it can commit into the repos only if the tool does not report any kind of problems/warnings);2- improving repoman and making it more strict and evilish when dealing with QA issues;3- forcing the usage of specific portage hooks for the duties of the developer (forcing repoman to commit only if you are using a valid developer profile) just to help and simplify the devel life;4- forcing all the ebuilds to stay ~arch until there are not pending bug reports and *until the maintainer have written or documented everything he/she MUST document* (so the problem of missing docs or “unwritten rules” is gone forever);the sentence “*if you wish to save a package from QA you got to fix all the problems*” must be changed into: “if you wish to commit a package into portage, you got to fix all the problems”. simple and effective, then a new era of real QA for portage can begin to spread.p.s.: as I have told you several time in the past, it’s time to write a kind of official Gentoo guides about “*Best practises and QA Policies for ebuild maintainers*” and “*Rules of common sense for the Gentoo bugzilla*”, or the internal discussions between developers and end-users will never end.


  3. While the good is sometimes the enemy of the best, perhaps sometimes the best is the enemy of the good.I think we need to be careful about a standard of “perfection or nothing” – we might just end up with the latter.Also, while it is good to try to fix upstream build problems (not respecting CFLAGS, etc), sometimes this isn’t always practical. Honestly, for the packages that I maintain if my only choices were fix every possible issue like this or take it out of portage, I’d just move it to my own overlay. I personally get just as much benefit either way – it is just everybody else who loses out.equilibrium suggests that was should enforce that all devs should use a valid developer profile. Doing some googling I see no documentation at all on g.o about these profiles. I do have some idea of what they do, but I think that if we’re going to enforce a mandatory policy, we ought to actually write it down somewhere. If we’re going to use technical tools to enforce policy, we ought to document what they actually do.We actually have a Gentoo developer handbook on the website, which doesn’t list most of the recently enforced QA policies. In fact, it actually contradicts one I was flagged on (it states that live cvs builds must be masked ~arch, when the policy seems to be that it should be in package.mask).Also – the all-or-nothing approach to commits also makes it impossible to make incremental improvements. Suppose a package has 10 problems with it, and I have a fix for 5 of them. Such a policy suggests that I cannot fix those 5, unless I’m willing to fix all 10.I’ve seen these kinds of policies in action at work – stuff subject to them just goes completely stale as nobody wants to touch it for fear of becoming the “owner” who has to clean up years of cruft. We end up living with stuff with 47 problems because if somebody stepped up to fix 42 of the problems they would get yelled at for not fixing the other 5, rather than being thanked for what they have done.


  4. Rich, if *you* are maintaining a package, I have *somebody* to scream at if it fails in multiple times; the fifth time a maintainer is ignoring all possible problems with a package, then I’m quite entitled to be angry with the developer (as a QA member), am I not?Also, I should remind you that you *have* to respect CFLAGS or both QA and Toolchain teams get upset, as well as most likely users who might desire their packages to be built with particular configurations. While we _do_ tend to leave people breath if the advantage/disadvantage ratio is too low, *the problems need to be fixed*.And in this case I’m not pretending to be perfectionist, but if a package has been *ignored for years* you cannot do a “five minutes fix“, because most likely it’ll be wasting _my_ time again when I find a different failure or QA breach with it in the next round, and _your_ time again for the next “five minutes fix”. Rinse and repeat.I don’t have much of a problem with *actively maintained packages having smaller issues* but a package that has no maintainer, nobody interested in being its maintainer, it hasn’t been touched in years and hasn’t released a new version in an even longer time, a five minutes fix will just not do. *Saving those packages is just stupid*.


  5. I agree with most of your points – there is definitely a balance and I agree that if a package is generally unmaintained and has serious QA concerns it should be put on the remove list. If somebody wants to step up and rescue it that is wonderful.We should have a little patience when that happens, but I do agree that a commitment to maintain a package does mean more than revising it once and then dumping it again.I have gotten one or two tinderbox bugs and I appreciate the instructions you include with them which are very helpful.


  6. I am not familiar with Gentoo development procedures. However this seems like a lack of process definitions. Document the development process and make sure developers know it. Embedd checkpoints within this process to make sure that the process is correctly followed. Checkpoints are sign-offs, okays etc. that one has to obtain from certain people (roles) before continueing in the process. They can include technical barriers that prevent progress until the checkpoint is passed. A checkpoint can be anything from an informal “OK” in IRC to very formal (web or paper form). Also look at how the linux kernel has implemented QA: every patch needs to be signed off by a subsystem maintainer at least. It’s a “checkpoint” system.


  7. @Odi I doubt that making it harder to contribute to Gentoo is going to make it easier to solve the problems. I *heavily* doubt that.@Rich, as I said I have no problem if somebody wants to pick up a package, and I’m fine when that happens. On the other hand, what I don’t want to see/read is: complaints about me not fixing the bugs I report (after around 3K bugs, even if they were all “five minutes fixes” I would have poured over 10 full days of work, 24 hours, for that); people complaining about masking their package (with a 60 days allowance) after over six months ignoring build failures and especially people doing _exactly one fix_ over a package that is deeply broken.My penalty system rambling there is mostly related to the last point: there are more than a couple of developers who like to just commit fixes for various packages they don’t maintain when they see patches ready or “quick fixes” to make. This in theory works fine since it fixes more packages, but it really doesn’t: the packages will likely have more problems in the future, which will cause the same trouble again. Sometimes, people don’t even test if the binaries from the package starts up after changing them.While making a mistake, or underestimating the problems of a package, should not really be much of a concern, repeatedly, over time, making that mistake is definitely something that we should avoid because it forces more burden over the quality assurance development.As for documenting the QA policies and practises, I wanted to tackle that problem these holidays, but the git repository for the devmanual isn’t ready yet and I have a visceral hate for Subversion ;)


  8. As someone with IT management experience, there are a number of “truisms” which come to mind. I’m not looking to be preachy nor negative. Rather, I want to try to be constructive here.The first one is that “nobody will do as good a job as you would do yourself.” This was one of the most difficult things for me to learn and to deal with as a manager. And it really is true. The corollary is that “If you want a job done right, do it yourself”. As a perfectionist or merely as someone who continually desires to see that a job is done ‘right’, it is damn difficult to cope with “good enough” and especially to cope with “well … at least is works somewhat.”Good QA demands a perfectionist attitude. Point of fact: a perfectionist attitude was always the primary requirement for anyone I considered for a QA position. However, “getting everything {or at least most things} done” requires a fine balancing of and acceptance between “good enough”, “just enough” and “doing it right”. Because it is a fine balancing and everyone’s idea of “good enough” is different, you can’t ever please everyone or even please yourself.My general suggestions for these type of issues are basically a multi-prong approach. The reality is there is no simple approach to getting the most benefit and results.The first thing I try to do is not set “gold standards”. Once you get hooked on the concept of the best way to do something, you immediately run into the issue that nothing gets done because no one has time to do it the best way. Instead, what needs to happen is that “minimum standards” need to be set. There needs to be consensus on what constitutes the minimum accepted standards that most people will adhere to. I can’t stress enough the word “consensus”. When a consensus happens, anyone who deviates will feel the need to justify that deviation to their peers. This implies there must be good conscious reasons for deviation beyond, “I was to lazy to do it to meet minimum standards”.As an aside, I don’t believe in the 80/20 rule. I think it’s stupid to quantify things this way. Rather, I like to view things like standards in terms of “What are those items most people will agree on and actively follow and, of those, which are most useful?” Everything else, regardless of how one personally feels about it, is fluff. Nice to have fluff. But still only fluff.I expressly use the word “fluff” because “time” is the one commodity that can never be replaced. Because time is such a prime resource, anything which requires more time than the effort to meet minimal standards become fluff by definition.The second thing after agreeing on minimal standards are to look at ways to help people follow the standards. In other words, people will always follow standards which don’t require effort on their part. I’m not saying that ‘no effort’ is a requirement. It is perfectly fine to require some “de minus” effort on their part. The objective is to provide whatever tools / methods / habits of thought / attitudes needed as part of the development environment resulting in natural adherence to consensus standards.Since I’m not a dev, I can’t speak to specifics for Gentoo devs. But perhaps if you step back a little bit and think in terms of 1) what would be a good list of minimum standards and 2) what kinds of tools/infrastructure would promote easy standards adherence, you might be able to initiate a more positive ownership attitude by devs towards the ebuilds they commit to maintaining long term. Further, a better work approach towards the {for want of a better term} one-shot ebuilds where they take an un-owned package and bring it up to date.Just some not so random thoughts. I hope you find them helpful in your own thinking. If these spark additional ideas so much the better. If you’d like to bounce any ideas, list of standards or anything like that, feel free to email me. I’ll be happy to be a sounding board.Best regards


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s