I’m partly sad, but for the most part this is a weight that goes away from my shoulders, so I can’t say I’m not at least in part joyful of it, even though the context in which this is happening is not exactly what I expected.
I turned off the Gentoo tinderbox, never to come back. The S3 storage of logs is still running, but I’ve asked Ian to see if he can attach everything at his pace, so I can turn off the account and be done with it.
Why did this happen? Well, it’s a long story. I already stopped running it for a few months because I got tired of Mike behaving like a child, like I already reported in 2012 by closing my bugs because the logs are linked (from S3) rather than attached. I already made my position clear that it’s a silly distinction as the logs will not disappear in the middle of nowhere (indeed I’ll keep the S3 bucket for them running until they are all attached to Bugzilla), but as he keeps insisting that it’s “trivial” to change the behaviour of the whole pipeline, I decided to give up.
Yes, it’s only one developer, and yes, lots of other developers took my side (thanks guys!), but it’s still aggravating to have somebody who can do whatever he likes without reporting to anybody, ignoring Council resolutions, QA (when I was the lead) and essentially using Gentoo as his personal playground. And the fact that only two people (Michał and Julian) have been pushing for a proper resolution is a bit disappointing.
I know it might feel like I’m taking my toys and going home — well, that’s what I’m doing. The tinderbox has been draining on my time (little) and my money (quite more), but those I was willing to part with — draining my motivation due to assholes in the project was not in the plans.
In the past six years that I’ve been working on this particular project, things evolved:
- Originally, it was a simple chroot with a looping
emerge, inspected withgrepand Emacs, running on my desktop and intended to catch--as-neededfailures. It went through lots of disks, and got me off XFS for good due to kernel panics. - It was moved to LXC, which is why the package entered the Gentoo tree, together with the OpenRC support and the first few crude hacks.
- When I started spending time in Los Angeles for a customer, Yamato under my desk got replaced with Excelsior which was crowdfounded and hosted, for two years straight, by my customer at the time.
- This is where the rewrite happened, from attaching logs (which I could earlier do with more or less ease, thanks to NFS) to store them away and linking instead. This had to do mostly with the ability to remote-manage the tinderbox.
- This year, since I no longer work for the company in Los Angeles, and instead I work in Dublin for a completely different company, I decided Excelsior was better off on a personal space, and rented a full 42 unit cabinet with Hurricane Electric in Fremont, where the server is still running as I type this.
You can see that it’s not that ’m trying to avoid spending time to engineer solutions. It’s just that I feel that what Mike is asking is unreasonable, and the way he’s asking it makes it unbearable. Especially when he feigns to care about my expenses — as I noted in the previously linked post, S3 is dirty cheap, and indeed it now comes down to $1/month given to Amazon for the logs storage and access, compared to $600/month to rent the cabinet at Hurricane.
Yes, it’s true that the server is not doing only tinderboxing – it also is running some fate instances, and I have been using it as a development server for my own projects, mostly open-source ones – but that’s the original use for it, and if it wasn’t for it I wouldn’t be paying so much to rent a cabinet, I’d be renting a single dedicated server off, say, Hetzner.
So here we go, the end of the era of my tinderbox. Patrick and Michael are still continuing their efforts so it’s not like Gentoo is left without integration test, but I’m afraid it’ll be harder for at least some of the maintainers who leveraged the tinderbox heavily in the past. My contract with Hurricane expires in April; at that point I’ll get the hardware out of the cabinet, and will decide what to do with it — it’s possible I’ll donate the server (minus harddrives) to Gentoo Foundation or someone else who can use it.
My involvement in Gentoo might also suffer from this; I hopefully will be dropping one of the servers I maintain off the net pretty soon, which will be one less system to build packages for, but I still have a few to take care of. For the moment I’m taking a break: I’ll soon send an email that it’s open season on my packages; I locked my bugzilla account already to avoid providing harsher responses in the bug linked at the top of this post.
Feel very sorry for Gentoo. I am a heavy Gentoo user and remember the times before tinderbox very clearly where it were multiple build breakages and my inability to finish upgrades as a result. Seem like it is coming back, soon or later.
Thats a Shame, really. Did you propose Mike himself to write and host a bot to automatically attach these logs?
Sad to see it go! Was definitely useful for all us Gentoo users.
Thanks for ages of good work!
I agree, thank you and oh do I not want the days of constant breaks to come back… 🙁
Thank you for your numerous contributions to Gentoo.As a Gentoo user I have benefited from your efforts.
Thanks for all your hard work and your many insightful posts.
Please note that I have *not* yet decided what I’m going to do with Gentoo as a whole. For the moment I’m taking a break, but it’s well possible (likely, even) that I’ll be back doing development work, just not running the tinderbox ever again.And will probably either shy away from dealing with Mike in the future; if I’d have to, then I’d probably just get out of the game.
I’d like for people in Gentoo to realize how important a tinderbox is. At NixOS all packages are built using the build farm http://hydra.nixos.org/jobs… to ensure at least that they all build.It’s also sad for Gentoo community to repeat mistakes that were learned the hard way in the past.Hope you change your mind nevertheless.
This sure took a lot of effort to run over the years and I can only join everyone in thanking you for this.The worst part of this negative story is the personal morale downside side a technical and pointless matter can have.As a fellow Gentoo dev, I am truly sorry you feel that way and hope you’ll not loose your faith and enthusiasm for too long.
Why isn’t Mike removed from Gentoo?
I’m wondering if it would be worthwile talking to Google? IIRC they use(d?) portage to build their ChromeOS and they surely wouldn’t mind having one or a few more boxes, but it could be a nice way for them to give something back, especially with their new security team hunting for bugs.Just throwing ideas around, I’m not to knowledgeable about the whole Tinderbox situation, but I agree with @Domen that this is bad for all of Gentoo and we should know better by now, damn egos 🙁
Go and do what’s right for you. If you really can’t stop yourself working on QA stuff, do it for Funtoo or some other project that won’t burn you out. At least do it for a group more deserving of your efforts than us, because there’s many out there these days.Trying to co-exist with Internet Fuckwads like him doesn’t help in the long run; it just teaches new recruits that his kind of attitude is acceptable. And even at the distance I’m observing at (well away from any actual mailing lists), that’s what started happening in practice. I won’t name names, but Gentoo is in dire need of some hard, harsh quality control applied to the developers themselves… before all the ones with common sense walk out.
Please donate it. Several of us helped pay for it after all.I haven’t touched Gentoo in six months now. I hate to say it but I don’t miss it. I just wish there were better options.
Ryan, the main idea is to donate it, Gentoo Infra is not sure about power consumption, will try to get a number for that. Do note that by the end of the contract with HE I would have paid almost twice as much to keep it running, and that’s just for one year, so its TCO is not exactly low, I wouldn’t blame them for turning it down.
Hi,it was nice to hear that somebody works for community, when somebody else just sucks from it 🙂 :-)I quit contributing when original wiki had gone because of economy problems around 2009 (??) and now fully understand rationales for your decision.Good luck with another, better projects.That said, I’m still with Gentoo somehow, I even don’t know why and what for 😉
Your tinderbox runs have been invaluable. To me personally, to our users and to the FLOSS community as a whole. I hope someday we’ll be able to handle those things better, both technically and humanly. At this point I can only hope for the best.Thanks again for the time and effort you put into this. Here’s to a beer or two if we ever cross paths 🙂
It’s sad to read this. I think that tinderbox was really important piece to squash Gentoo bugs.I can only say, thanks for your time and effort.
I am perhaps a little confused. What is tinderbox and what happened?