Finally Excelsior is doing what it has been bought to do: tinderboxing. Well, not really to full capacity but I’ve got enough pieces of the puzzle in place that it should soon start building regularly.
What the tinderbox set up is doing now is actually a limited run: it’s building the reverse dependencies of virtual/ffmpeg to find out which packages actually require libpostproc, so that we can fix their dependencies with libav — to follow up on Tomáš’s post I’m also strongly attached to libav, being, together with Luca, part of the development team of the project.
Speaking about libav, one of the things that since a couple of days ago Excelsior is doing, is running FATE so to both cover AMD Bulldozer’s architecture and Gentoo Hardened. For those who do not know, FATE is an automated testing environment that configures, builds, and run the tests for libav and then send the result to a central server for analysis. The whole build takes three minutes on Excelsior. Thanks to libav’s highly-parallelized makefiles.
There is only one problem with the Tinderbox, and is once again the logs analysis. To be honest this time I have some idea on how to proceed: the first step is to replace the grep command I have used before with a script that produces an HTML output of the log file. Yes many people said before that HTML is not a good idea for this kind of thing: since nobody else has helped me writing a better log analysis tool, that’s going to be enough.
So what has to happen is reading line by line the input log, then create an HTML file with (numbered) lines, marked with a specific (CSS) file that makes the row red. As I said before it would be nice to have some kind of javascript to jump from one error line to the other. Until this is all set, though, just creating the HTML file would be enough.
The next step would probably be getting the HTML output on S3 (for easy access), and write it down on a database that actually give you an index of the error logs — for those who wonder, using s3fs as PORT_LOGDIR is not a good idea at all. I guess tomorrow I might try to find some time to work on this. I could find some time on the weekend, given that my original idea (biking around with Luca) is disabled due to me falling and (lightly) injuring myself last night.
Now that you’re a libav developer, will you be able to push those symbol visibility patches that were rejected years ago? Or was that for libxine…
Symbol visibility since a long time is decided via name prefixes in both FFmpeg and libav.
I would suggest that you parse the logs and store the result in a relational database and then queries that database to produce the HTML files rather than parsing HTML files to store in to the database.The more information you parse out and store in the database, even if not used for the HTML files, the better.Do that little extra work now, and save yourself a lot of time later on.Alternatively you can email me, jonas@faceways.se, with an example log file and then give me 1-3 days and I’ll reply with a database structure, a script to parse the example log file, a script to display it as HTML and a Javascript to jump from one point of interest to another within the page.
Jonas, I don’t think that filling the reports straight into the database is that good an idea, mostly because fitting the log into the database is a no-go, and cutting it is also bad. What I have currently in mind (and partially on hand) is this: * a Ruby script I already wrote can work more or less like a @grep@ clone by searching line by line for a match in a regexp; when it finds it; each line is output as a li-tag in HTML, and if there is a match it sets the approriate CSS class; this already works and even an 8MB HTML file is rendered fine by Chromium, as long as it can receive the data; * results of the scripts are uploaded into S3 for storage (helps because you can tie an expiration date to the files so that stale data is cleaned out automatically), and are at the same time written down in a database table with a very quick preview so that the output matches what I used to see before.I’ll forgo authentication/authorization for the moment simply because Apache can take care of that for the most part, especially as long as I’m doing this alone. The only problem is who should take care of the analysis? The same hardware host is okay, but not the same container that takes care of the tinderbox…
I don’t see the issue regarding the database but on the other hand I’m not here to question your good work, so: to each one his own.
Jonas to give you an idea, “here is a log”:https://www.flameeyes.eu/tm… passed through the first part of the script. It’s not the biggest one I’ve seen but it’s not a small one either. The result is definitely good enough to track down the issues I’m going to find.There are still a few things to take care of, in particular there has to be a way to remove the escape sequences (see at the end of the log), but for now this would do.While generating the single output is not that taxing for a script, but on the other hand, having to re-create this many times doesn’t sound like a good idea…
You are tinderboxing quite a lot! Maybe I should get you a match? *ducking and running because of that good joke*