You might not be familiar with the term “run control” even though you use openrc; guess what the rc stands for?
This post might not be of any interest to you; it’s going to delineate some requirements for my tinderbox to be expanded and streamlined. So if you don’t care to help me or know not of development, you can skip it and wait for the next one tomorrow.
As the title leave you to guess, I’m looking for a better way to handle the execution of the tinderbox itself. Right now as I shown you, I’m just using a simple xargs call that then takes care of launching a number of emerge requests to build the list of packages one by one. Unfortunately this approach has quite a few problems, the first of which is that I have no way to check if the tinderbox is proceeding properly without connecting to its console. Which is quite taxing to do especially when I’m not at home.
I could write a shell script to handle that; but on the other hand I’d rather have something slightly more sophisticated, and more powerful. Unfortunately because the whole design of the tinderbox relies so heavily on the Portage internals, I guess the language of choice for something like this should probably be Python, so, for instance it can call the tinderbox script via function call rather than forking.
What I’d be looking for, right now, would be a daemon: have this daemon run in the background started automatically by the init system inside the container, with two ports open for remote control: one, with the command console that allows for starting and suspending the execution, or aborting the current run, and one for the logging console that shows what emerge is doing; the whole thing would look a lot like the ssh/@screen@/@xargs@ execution I’m doing right now, just, less locally-tied. Bonus points if the whole system only allows for SSL connection using known client certificates.
My reasons to wanting a separate run control than just ssh to the console is to allow for other developers eventually to access the tinderbox, for instance to prioritize one particular package over another, or if needed to change the settings (flags and similar) for a particular execution. In the case the client authentication was too much to implement, it could probably be easily solved by creating a script using nc6 to talk to the console, and using that as a shell, leaving access through SSH (with non-root accounts).
Another reason for this is to better handle the cascading failure of dependencies. If I’m going to merge package C that requires A and B, but B fails, I should be masking that for the time of the current run. This way when I’m asked to install D that also requires B, it’ll be skipped over (instead of insisting rebuilding the same package over and over). At the end of the run (which would mean at the next --sync), those masking can be dropped entirely.
This means that at the end of an emerge request, you need to find if it completed fine or not, and if not, on which package it failed.
Other features for the run control would sprout once this basis is done, so if anybody is interested in helping out with the tinderbox and wants to start with writing this type of code, it’s definitely welcome.
What you may be looking for is what gearman is doing. Its basically a message queue to perform tasks. It can give you the status of currently running tasks, it can parallelise whatever you throw at it, it has priorities, etc. Upon success or fail nothing stops you from requeuing the task, sending notifications by email or whatever …There are bindings to scripting languages so its quite easy to write a write a a web frontend with any access policies you wish for. So, basically, webserver+scripting language, gearman, scripts for gearman to execute and portage should be what you need.
I’d venture it is not much further to actaully turn this into a distributed project and off load the light bill and equipment for other work. Just make it a job gentoo users could run in the background. ‘tinderbox project’ make it an emerge and ‘nice’ it in the background.Tout the uses and bugs it keeps users from seeing. Add to it.
Unfortunately, there is no way for something like the tinderbox (controlled known environment) to work on user systems; what we can do on the other hand is to find a way to easily report those bugs found on user systems.
Another option would be running it from buildbot. It’s written entirely in python, I used to use it for my tinderbot stuff, though pkgcore based. But starting a simple emerge shouldn’t be that hard actually