Unpaper fork, part 2

Last month I posted a call to action hoping for help with cleaning up the unpaper code, as the original author has not been updating it since 2007, and it had a few issues. While I have seen some interest in said fork and cleanup, nobody stepped up with help, so it is proceeding, albeit slowly.

What is available now in my GitHub repository is mostly cleaned up, although still not extremely more optimised than the original — I actually removed one of the “optimisations” I added since the fork: the usage of sincosf() function. As Freddie pointed out in the other post’s comments, the compiler has a better chance of optimising this itself; indeed both GCC and PathScale’s compiler optimise two sin and cos calls with the same argument into a single sincos call, which is good. And using two separate calls allows declaring the temporary used to store the results as constant.

And indeed today I started rewriting the functions so that temporaries are declared as constant as possible, and with the most limited scope as it’s applicable to theme. This was important to me for one reason: I want to try making use of OpenMP to improve its performance on modern multicore systems. Since most of the processing is applied independently to each pixel, it should be possible for many iterative cycles to be executed in parallel.

It would also be a major win in my book if the processing of input pages was feasible in parallel as well: my current scan script has to process the scanned sheets in parallel itself, calling many copies of unpaper, just to process the sheets faster (sometimes I scan tens of sheets, such as bank contracts and similar). I just wonder if it makes sense to simply start as many threads as possible, each one handling one sheet, or if that would risk to hog the scheduler.

Finally there is the problem of testing. Freddie also pointed me at the software I remembered to check the differences between two image files: pdiff — which is used by the ChromiumOS build process, by the way. Unfortunately I then remembered why I didn’t like it: it uses the FreeImage library, which bundles a number of other image format libraries, and upstream refuses to apply sane development to it.

What would be nice for this would be to either modify pdiff to use a different library – such as libav! – to access the image data, or to find or write something similar that does not require such stupidly-designed libraries.

Speaking about image formats, it would be interesting to get unpaper to support other image format beside PNM; this way you wouldn’t have to keep converting from and to the other formats when processing. One idea that Luca gave me was to make use of libav itself to handle that part: it already supports PNM, PNG, JPEG and TIFF, so it would provide most of the features it’d be needing.

~~In the mean time, please let me know if you like how this is doing — and remember that this blog, the post and me are Flattr enabled!~~

Forking unpaper, call for action

You might or might not know the tool by the name of unpaper that has been in Gentoo’s main tree for a while. If you don’t know it and you scan a lot, please go look at it now, it is sweet.

But sweet or not, the tool itself had quite a few shortcomings; one of these was recently brought to my attention as unsafe use of sprintf() that was fixed by upstream after 0.3 release, but which never arrived to a release.

When looking at fixing that one issue, I ended up deciding for a slightly more drastic approach: I forked the code, imported it to github and started hacking at it. This both because the package lacked a build system, and because the tarball provided didn’t correspond with the sources on CVS (nor with those on SVN for what it’s worth).

For those who wonder why I got involved in this while this is obviously outside my usual interest area, I’m using unpaper almost daily on my paperless quest that is actually paying its fruits (my accountant is pleasantly surprised by how little time it takes to me to find the paperwork he needs). And if I can shave even fractions of seconds from a single unpaper process it can improve my workflow considerably.

What I have now in my repository is an almost identical version that has passed through some improvements: the build system is autotools (properly written), that works quite fine even for a single-source package, as it can find a couple of features that would otherwise be ignored. The code does not have the allocation waste that it did before, as I’ve removed a number of pointers to characters with preprocessor macros, and I started looking at a few strange things in the code.

For instance, it now no longer opens the file, seek to the end, then rewind to the start to find the file’s size, which was especially unhelpful since the variable where the file’s size was saved was never read from but the stdio calls have side effects, so the compiler couldn’t drop them by itself.

And when it is present, it will use sincosf() rather than calling sin() and cos() separately.

I also stopped the code from copying a string from a prebuilt table, and parse it at runtime to get the represented float value.. multiple times. This was mostly tied with the page size parsing, which I have basically rewritten, also avoiding looping twice over the two sizes with two separate loops. Duh!

I also originally overlooked the fact that the repository had some pre-defined self-tests that were never packaged and thus couldn’t be used for custom builds before; this is also fixed now, and make check runs the tests just fine. Unfortunately what this does not do is comparing the output with some known-good output, I need an image compare tool to do so; for now it only ensures that unpaper behaves as expected with the commandline it is provided, better than nothing.

At any rate this is obviously only the beginning: there are bugs open on the berlios project page that I should probably look into fixing, and I have already started writing a list of TODO tasks that should be taken care of at some point or another. If you’re interested in helping out, please clone the repository and see what you can do. Testing is also very much appreciated.

I haven’t decided when to do a release, for now I’m hoping that Jens will join the fork and publish the releases on berlios based on the autotools build system. There’s a live ebuild in main tree for testing (app-text/unpaper-9999), so I’d be grateful if you could try it on a few different systems. Please enable FEATURES=test for it so that if something breaks we’ll know son enough. If you’re a maintainer packaging unpaper on other distributions, feel free to get in touch with me and tell me if you’ve other patches to provide (I should mail the Debian maintainer at this point I guess).

Revisiting my opinion of GitHub — How do you branch a readonly repository?

I have expressed before quite a bit of discontent of GitHub before, regarding the way they continue suggesting people to “fork” projects. I’d like for once to state that while I find the fork word the wrong one the idea is not too far from ideal.

Fast, reliable and free distributed SCMs have defined a new landscape in the world of Free Software, as many probably noted already; but without the help of GitHub, Gitorious and BitBucket I wouldn’t expect them to have made such an impact. Why? Because hosting a repository is generally not an easy task for non-sysadmins, and finding where the original code is supposed to be is also not that easy, a lot of times.

Of course, you cannot “rely on the cloud” to be the sole host for your content, especially if you’re a complex project like Gentoo, but it doesn’t hurt to be able to tell users “You want to make changes? Simply branch it here and we’ll take care of the management”, which is what those services enabled.

Why am I writing about this now? Well, it happened more than once before that I was in need to publish a branch of a package whose original repository was hosted on SourceForge, or the GNOME source , where you cannot branch it to make any change. To solve that problem I set up a system on my server to clone and mirror the repositories over to Gitorious; the automirror user is now tracking twelve repositories and copying them over to gitorious every six hours.

As it happens, last night I was hacking a bit at ALSA — mostly I was looking into applying what I wrote yesterday regarding hidden symbols on the ALSA plugins, as my script found duplicated symbols between the two Pulse plugins (one is the actual sound plugin the other is the control one), but ended up doing general fixing of their build system as it was slightly broken. I sent the patches to the mailing lists, but I wanted to have them available as a branch as well.

Well, now you got most of the ALSA project available on Gitorious for all your branching and editing needs. Isn’t Git so lovely?

Before finishing though I’d like to point out that there is one thing I’m not going to change my opinion on: the idea of “forking” projects is, in my opinion, very bad, as I wrote on the article at the top of the page. I actually like Gitorious’s “clones” better, as that’s what it should be: clones with branches, not forks. Unfortunately as I wrote some other time, Gitorious is not that great when it comes to licensing constraints, so for some projects of mine I’ve been using GitHub instead. Both the services are, in my opinion, equally valuable for the Free Software community, and not only.

Update (2017-04-22): as you may know, Gitorious was acquired by GitLab in 2015 and turned down the service. This means that all the references are totally useless now. Sorry.

Sharing more of my infrastructure

While my repeated tries to release FSWS haven’t sorted out yet, mostly because I’m waiting on the Free Software Foundation to tell me whether I can re-use the wording of their autoconf exception in terms of AGPL3, I decided to publish something else for everybody to see: the ModSecurity rules I’ve been using on this blog for the longest time.

You can now find my RuleSet on GitHub — it includes most of the general rules I’ve been using and at least part of the comment antispam I’ve applied. I’m now trying to clean up my own rules so that they can actually be pushed further to other users as well.

You can see that most of the rules weren’t made to be flexible or generic enough, so I’m still going through a bit of trial and error to get them published, but it shouldn’t take much to have them cleaned up properly.

All my rules will be released under a very permissive license: CC-BY-SA; which is to say I don’t care what you do with them, as long as you make them further available and credit me for the original in case. If you need a different license to ship it with commercial products or to merge it to other rule sets, simply contact me, I’ll most likely agree to any term as long as it’s reasonable.

Rubygems… UNHACKED!

I have written yesterday about the difficulty of removing the Rubygems hacks we’ve been using — well, today I got a good news: I was able to mostly remove them. I say mostly because there are still a couple of things that need to be fixed, with the help of upstream (all my changes are available on my Rubygems fork in the gentoo branch):

there is no way in the original sources to change the location of the system configuration file; for Windows it’s found on the register, for everyone else it’s /etc; I’ve had to change the sources to allow for overriding that with the operating system defaults;
there is one test that actually only works if there is no alternate default set installed, as it checks that the binary directory of the gems is the same as the one for Ruby; that is no longer the case for us;
JRuby … pretty much fails a whole bunch of tests; some it’s not really its fault, for instance it lacks mkmf/@extconf@ since that there are no C-compiled extensions; others are caused by different problems, such as tests’ ordering or the huge difference in handling of threading between the original implementations and JRuby;
I had to patch up a bit the Rakefile so that it can be used without Rubyforge support, which was a requirement for Ruby Enterprise (well at least for my system; the problem is that it still does not build with OpenSSL 1.0, so I have a non-OpenSSL Ruby Enterprise install… and that means that Rubyforge can’t load, and even so the Rubyforge plugin for Hoe);
documentation fails to build, not sure on why or how, but it does, when using the rake docs command; I’ll have to check out why and see if I can get it fixed.

But now back to the important good news: you can now safely use the gem command as root! Gems installed with sudo gem install foo will install in /usr/local rather than directly in /usr, no longer colliding or clashing with the Portage-installed packages (which are officially supported). Obviously, both are searched, but the local variant would take precedence, and the user’s home gems get even higher priority for search. This also means that if you don’t care about Gentoo support for the Ruby extensions, you can simply use gem and be done with it.

Now moving a bit back to not-too-good news and next steps. The not-too-good news is that since this dropped all the previously-present hacks, including a few needed to get gem from within Portage, this new version is not compatible with the old gems.eclass. Well, it’s relatively not good news, and partially good news; this is one of the final nails in that eclass’s coffin; we have now a very good reason to get rid of all the remaining packages; soon.

*Note: I have said before that we still lack a way to properly build bindings that are part of bigger packages; luckily none of those require gems.eclass, they rather use the old ruby.eclass which does not require rubygems at all. So it’s fine to deprecate and get rid of the uses of that right now even though we might have more work to do before ruby-ng takes over everything.*

Next steps for what concerns the rubygems package itself, if it was up to me, would be to drop the four gem18, gem19, gemee18 and jgem: all the ruby-fakegem.eclass binary wrappers right now install a single script, which by default uses the currently-configured Ruby interpreter, and you simply have to use ruby19 -S command to start it with a different interpreter. But gem itself is not treated this way and we rather have four copies of it with different names, and different shebangs, which sounds a waste. To change this, among other things, we need to change the eselect-ruby module (which I sincerely would avoid touching if possible).

Further step: supporting multiple Ruby implementation with the current system is mostly easy… but we have no way to change the current interpreter on a per-user or per-session basis; this is something that, as far as I can tell, we could actually take out of the Python book (or maybe the Java book, both have something like that, but the Python failures actually teach us one very important thing: we cannot use a script, we have to use a binary to do that job, even if it’s a very stupid one, as older Linux and other non-Linux systems will fail if you chain interpreters). Again, an eselect-ruby task… I would probably wait for Alex, unless there is enough interest for me to work on it.

Then let’s see, we’ve got to fully migrate out of the old-style virtual/ruby dependencies into =dev-lang/ruby-1.8* so that we can actually implement a new-style virtual that proxies the ssl USE flag, and then start using that; we’ve got to implement a general way to handle one-out-of-multiple target handling for packages that use Ruby as an embedded interpreter rather than build libraries for it; then there is again the problem of finding a way to build bindings within larger packages (or move some of them to build the Ruby extensions by themselves — I remember obexftp could use something like that), there is the fake-specification-generation to be fixed so that it works with bundler, and there are some (corner?) cases where Ruby 1.9 complain about missing fields in our generated files. So as you can see there is a long way still, but we’re going up there, step by step.

And before I leave, I’d like to thank Zeno Davatz for funding the un-hacking of rubygems — if that wasn’t the case, I’d probably have had to post this next week, or month. And if there are other people wanting Ruby 1.9 available faster (or improved 1.8 support), I’m still available to be hired, you can contact me and provide me with an idea of what you’re looking for. I’m available both to implement particular feature pieces, porting of the web of dependencies of given gems, or even just on a retainer basis so that you can have “priority” support for what concerns the general extension packaging in Gentoo.

Ruby-NG: Too Frequently Asked Questions

Okay I’d sincerely would like to stop blogging about Ruby and actually working on making it better, but it seems like that one particular user is trying to start up fires all around the Ruby Team to push for his agenda… without seeming to care at all about the results. So here comes a list of common questions and mostly-official answers.

I forgot one of the most important questions! Added afterwards!

Why is Ruby 1.9 masked? It’s stable upstream!!!! The exclamation marks are there because that’s how we receive them oftentimes. Sure, Ruby upstream seems to declare that Ruby 1.9 branch is “stable”, by their standards already, which I’ll have to tell you all, are not very solid. To be precise, they mark 1.9.1 stable, while 1.9.2 is still “development” and is currently a Release Candidate. While the code for 1.9 sounds more solid, the changes between 1.9.1 and 1.9.2 are not trivial, and wouldn’t have been considered for most other projects worth of “minor versions”. To make a comparison, they really look like the difference that there would be between GCC 4.5 and GCC 4.6, while the changes between 1.8.7 and 1.9.1 are like those between GCC 4 and GCC 5.

What we’re actually targeting for unmasking at some point in the hopefully not-so-distant future, is 1.9.2, not 1.9.1, so no, our target is not considered stable upstream yet.

Just unmask 1.9.1 then! Unfortunately, that’s not feasible. The reasons are long and are (partly) on our side: we’ve not had a proper way to handle extensions’ packaging until I created Ruby-NG in May 2009 – all the previous ebuilds used various series of hacks and dirty tricks to allow side-by-side installation of Ruby packages, but none actually worked properly. Even worse for the gem-based packages. From May 2009, we’ve been refining techniques, eclasses and porting packages, but the work is still not complete. There will be much more to deal with before unmasking. And that means that our most likely candidate is going to be 1.9.2 rather than 1.9.1.

This does not mean that we’ve not tried targeting 1.9.1; I actually did a lot of porting to 1.9.1… and now that we’ve turned to look at 1.9.2 I had to do further porting.. wasting twice the time to identify and fix issues. Trust me it’s not funny.

Why doing the work? Leave it to Rubygems! Haha, very funny. I’ve written a long time ago about the reasons why we don’t consider the Rubygems package manager apt to production use. Even though a number of people find it just perfect, we’ve got our reserves, and feel like Gentoo should aim to provide higher-quality packaging to our users, since we actually can.

In particular you can manage a remote system with binary packages without having to keep around the development tools thanks to our packaging, as you’d be building the binary gems together with normal Gentoo packages, and abiding to the settings you provide to Portage.

Of course you can use the standard gem command, but if you do so, particularly as root, you “void the warranty” and the Gentoo Ruby team won’t respond for any problem you might experience.

Why do I see so many tests failing? I didn’t see them at all before! You broke something! Not really; by default gems don’t execute tests; sure there is an option in the Rubygems package manager that provides support for running tests, but the number of gems that actually specify their test dependencies, and the way to run the tests, is risible. And with our previous packaging system, the tests simply weren’t executed. So all the failures you might experience with the new ebuilds based on Ruby-NG, are for the vast part coming directly from upstream.

To be fair to most upstreams, some tests are either designed to work on older Ruby versions, on other operating systems (such as Mac OS X that is case-insensitive — bundler had a problem related to that!), or with a particular series of installed gems. Which means that they might not have seen the failure at all. And some failures are actually coming from our environment; in particular lately the tinderbox is reporting me a spree of failures happening when test-unit 2.1.1 is installed for Ruby 1.8 (and would for Ruby EE if I was testing that).

We’re doing our best to actually fix the test failures, sending them upstream, as you can see from my own GitHub page that has forks for a number of Ruby projects for which I sent one or two patches to fix tests. This, though, is often a non-trivial yet boring work and we’re just four guys trying hard to get results.

Why is package #{foo} not available for 1.9? Upstream says it works! This usually has Rails (2.3) in place of #{foo}… yes upstream sometimes says their package works with 1.9, and most times they are right, but some other times, they are definitely not. Rails is a good example of the latter. It “works” in the sense that you can use it for some of its features, it does not work as in “everything is fine and smooth”. Even its own tests, for what concerns Rails, are totally bogus on 1.9. It gets worse for binary extensions (bindings).

There is a forked gem/patch to get #{bar} binary extension to build with 1.9, why you insist it’s not working on 1.9? This gets fun, and I actually written explicitly about it — sometimes the patched versions do exactly what it’s written on the box: they build on 1.9, but that says nothing about them working at all. Since a lot of symbols (functions and macros) have gone from 1.9, even fixing the extension to build might still entail using symbols that are no longer available, through implicit declarations. The original Ruby 1.8 and 1.9 build systems leave that be, and depending on the settings of the system this will lead to either non-loadable extensions, or runtime suicide of the Ruby process, because of missing symbols.

Personally, I don’t like having timebombs in my packages, so under the assumption that I’d rather have a build-time failure – or no package at all – than one that can kill my software stack at runtime, I’ve made our Ruby packages inject -Wl,--no-undefined so that similar problems are caught at build-time already. At least two packages that patched for Ruby 1.9 support in the overlay before failed this very test and were thrown out of the Ruby 1.9 compatibility.

Why can’t I merge the packages directly if I have FEATURES=test (or USE=doc) enabled? Since we’re actually running tests, we’ve got to depend on the packages that are used by the tests themselves; to be on the safe side, when there are optional tests we try to push all of them to be run during the test phase. Unfortunately because of the nature of the gems themselves this way too often lead to circular dependencies; this is why I call it a dependency web rather than a dependency tree, nowadays.

When merging Ruby 1.9, it’s telling me that Rake and Rubygems are being dropped, why is that? Ruby 1.9 comes with a bundled copy of Rake and Rubygems, but those are quite older, and a number of packages required much newer versions of either, or both. We could allow for an override for those, but it would be much harder than simply removing them and adding the dependencies over the normal Rake and Rubygems that install for 1.9 as well. Actually, the same destiny is there for JRuby that bundles even more packages.

Portage is reporting that the package has been built without respecting LDFLAGS, why’s that? Well, while I’m the first person who’d want packages to always respect flags, Ruby packages are very tricky. Sometimes we can rewrite CFLAGS/LDFLAGS directly on the emake command line, but there are many cases where that is not possible, so the actual LDFLAGS used for rebuilding Ruby itself are used. This is one of the things that we could use some help, or some more dedicated time.

What about JRuby? You started so well on it and is now slowing down, why? Mostly because a number of issues upstream required newer JRuby versions altogether; then they fractured support in an even higher number of further packages with mixed Java and Ruby code. I’m no Java expert to begin with, and keeping up with the development is not something I have very much time to tackle. The other members of the team seems highly uninterested in JRuby so far, thus you get what there is time for.

The package #{fnord} is installing itself as a gem, but it’s using a GitHub tarball rather than the gem file, why? Once again it has to do with the fact that gems are not required to provide or execute tests; while a few gems feel safe enough to let users run their tests, there are many that simply do not provide tests with their gem files. To solve this, we go around it, and download the code directly from GitHub, and use that for the packaging. Thankfully, GitHub got their shit together and is now feasible for us to keep using their download service. That is, if the upstream developers actually bother tagging their releases. Sigh!

How can I help to get Ruby 1.9 support in? Again there is a dedicated blog post from me that can be summarized as “we need more testing of the packages, reporting bugs upstream where it make sense, and where it makes sense more tests .”

There are a number of issues that need to be tackled, you can find most on Bugzilla — simply open the page and look through the bugs. Make sure that the test failures get fixed upstream, and not simply blacklisted, if possible.

I need Ruby 1.9 available right now as I’m using it in production! Tough luck, because “right now” is definitely not going to happen. Soon, maybe; right now, no. As I said there are a number of things to kink out; on the other hand, you can either help as stated before, or find someone who can be paid to help. ~~I’m up for hire~~ — and I’m repeating this because I’m growing tired of the few people who actually make a living out of this expecting the four of us to abide to their agenda.

I’m using Funtoo and… Stop there! The Gentoo Ruby team does not and will never support Funtoo users. Daniel decided to go on with Ruby 1.9 at a point where neither the upstream projects nor the eclasses were designed to. Whatever he did after that, we have no intention to sort out.

Ruby and Automagic Dependencies

I have written last week about one real-world case of automagic dependencies screwup (as well as having started the automagic fixing guide a long time ago). One would expect that wi Ould rather not see automagic dependencies anywhere, but that wouldn’t be quite correct, since the whole topic is quite trickier than this.

Indeed, in Ruby (as well as Perl and Python and almost all dynamic languages), automagic dependencies are well part of the game: you load code and features conditionally to the presence of the needed libraries on the system). There is, though, one very important distinction to make here: automagic dependencies for dynamic languages happen directly at runtime, not at build time. This means that it varies on the running system’s configuration rather than the build system’s.

This distinction is particularly important; when you need to rely, in an ebuild, on a feature that is enabled automagically at build-time, you have nearly no way to do so: you have to check for presence and abort in case, which is bad. With runtime-automagic dependencies, you just need to add as a dependency the library needed to provide the function, and you’re done.

Let’s see a common example in Ruby packaging. The echoe gem provide some common packaging tasks, wrapping around features like rcov coverage and rspec testing. You usually need Echoe at build time when you need to access the Rakefile (so for doc generation, some testsuites, and build of native extensions); if you need to execute the RSpec-based testsuite, you’d obviously need RSpec as well, you depend on both, and Echoe “enables” the conditional support for running that testsuite. Done, you don’t need to reinstall Echoe, you, thus, don’t need an USE flag on Echoe. And if you just needed to rebuild the documentation, well, you won’t be needing RSpec anyway!

_Note this is a simplification; Echoe up to the 4.3 release didn’t automagically depend on RSpec, but simply required it as soon as the package using Echoe had spec files available; I’ve fixed this for Gentoo and is now upstreamed since 4.3.1 so that we don’t have to depend on RSpec even for documentation building. And, actually, we’ve started using RSpec directly rather than Rake to run the easiest testsuites because sometimes the Rakefiles are just full of shitty dependencies that it’s easier to ignore them than fix them. If I do so, I usually also patch the Rakefile upstream so that in the future we can migrate out of that custom code and back into generic Rake calling._

Another example: the Storable gem allows the use of JSON data structures, via either the JSON gem or the YAJL-Ruby library. By default, it needs either, and won’t support the JSON data; if you’ve got a package that needs the JSON support in Storable, rather than having an USE flag on Storable you should just depend on both Storable and either of the two JSON libraries. Easy as pie, even though it might sound a bit cumbersome.

The reason why I stress that these things shouldn’t have USE flags is that we want USE flags to be, generally, absolute: if I built Storable with USE=-json I’d expect for it not to support JSON. But as it is, it might very well support it, if either JSON or YAJL-Ruby were installed as dependency of another package. Patching the code to disable it forcefully would be silly, and blocking the packages that could be used, would just increase complexity.

So for most Ruby libraries, we have to accept automagic dependencies as part of the game. But is that alright? Does it not add problems? I actually know it does, and it’s one of the main reasons why Ruby-NG ebuilds have, unfortunately, varied widely since I’ve started working on them last year.

The first problem arrives when you look carefully at the automagic Rakefiles: so many of them enable testsuites only if RSpec is present, documentation building only if a given documentation system is present (RDoc2, Hanna, …), GemSpec building if Jeweler is present and so on so forth. This is good because it means we don’t have to depend on, say, the documentation libraries when we’re not building documentation, but it also creates situations in which we can’t be sure that our ebuild works: what if a gem we haven’t installed is installed on the user’s system? It might even not be in our tree, but only in some overlay. What instead if something is missing and we forgot to add a dependency? Does the documentation building switch off clearly, if the needed libraries are not found?

I guess the commonest situation up to now is related to gems using Jeweler; like Echoe, Jeweler is a middleware used for packaging purposes; unlike Echoe, we don’t usually need it at build time because it only takes care of the proper Gems tasks (and we, generally, don’t care about the gem metadata). We’ve been able, up to now, to stay away from packaging Jeweler at all, which is good given that it’s quite complex to package. I guess we have to thank that the examples of usage for Jeweler shown it how to make it properly optional.

Unfortunately, even though rake -D will complete properly when you don’t have Jeweler installed, and documentation building usually work just as fine, it fails as soon as you try to run the test or spec task (the testsuites). Why? Because the Rakefile adds a prerequisite task to the testsuite: check_dependencies, provided by Jeweler, which is used to make sure that the gem is generated with the correct dependency information. It’s usually easy to fix: in Gentoo we can just sed away the line (or run the testsutie manually), and upstream we move the prerequisite declaration to the conditional block added when Jeweler is properly found. But easy or not, it’s bothersome.

Where it’s less of a hassle but more of an open question on how to handle it is with optional dependencies in testsuites. Whenever the package have automagic dependencies, you may have two opposite methods to handle testsuites: you might request that all the optional dependencies are available, so that they all get tested, or you might exclude the tests for those packages that couldn’t be found. The former option does not leave us packagers with an choice, the latter leave up to us to decide whether we want to test the deepest or stay shallow.

The obvious choice for extra safety is to always test the deepest; but this is not safe from problems. What happens if the packages are not available? The automagic, conditional code might not work well in the opposite situation (which is just as bad as the code failing). What if the packages are not available for a given implementation? While we can (and do, for some packages) add dependencies just for one or two implementations, it’s cumbersome. And most importantly, adding all the optional packages to the dependencies makes the dependency tree bigger and much more complex. For instance, rack would use during its test phase almost all the available Ruby-based webservers. Do you think it’d be okay to add all of them as dependencies? (And this is without saying that some of those depend on Rack!)

I hope you can now see why it is still quite a problem for us to properly package Ruby libraries, and why even if it’s just one or two packages, I often end up working a whole day on Ruby ebuilds to make sure they work as intended… it should also show given the number of repositories I have on GitHub (none of which I started; they are all “forks” of packages I fixed for Gentoo).

Ruby-NG: Package in a Bottle (or, learn how to write a new Ruby ebuild)

I have to say that in the months we’ve been working on the new eclasses, I never went on describing properly how to use them. My hope was to write this documentation straight into the next-generation development manual for Gentoo, but since that project is far from coming, I’ll just rely on my blog for a little while more.

As described in my blog posts the idea behind the “new” (they are in tree for a few months already by now) eclasses is to be able to both handle “proper” Gentoo phases for packaging gems, and at the same time manage dependency and support tracking for multiple Ruby implementations (namely, Ruby 1.8, Ruby 1.9 and JRuby right now). How can we achieve this? Well, with two not-too-distinct operations; first of all we avoid using RubyGems as a package manager – we still use, in some cases, the gem format, and we always use the loader when it makes sense – and then we leverage the EAPI=2 USE-based dependencies.

Why should we not use RubyGems package management for our objective? With the old gems.eclass we used to encapsulate the install operation from RubyGems inside our ebuilds, but it was all done at once, directly into the install phase of the ebuild. We couldn’t have phases (and related triggers) such as prepare, compile, test and install. In particular we had no way to run tests for the packages at install time, which is one of the most useful features of Gentoo as a basis for solid systems. There are also other problems related to the way the packages are handled by RubyGems, including dependencies that we might want to ignore (like runtime dependencies injected by build-time tools), and others that are missing in the specification. All in all, Portage does the job better.

For what concerns the USE-based dependencies, when we merge a package for a set of implementations (one, two, three or any other number), we need its dependencies (at least, the non-optional ones) installed for the same set of implementations, otherwise it cannot work (this is a rehashing of the same-ABI, any-ABI dependencies problem I wrote about one and a half years ago). To solve this problem, our solution is to transforms the implementation into USE flags (actually, they are RUBY_TARGETS flags, but we handle them exactly like USE flags thanks to USE_EXPAND), at that point, when one is enabled for a package, the dependencies need to have the same flag enabled (we don’t care if a dependency has a flag enabled that is not enabled in the first package, though).

This actually creates a bit of a problem though, as you end up having two sets of dependencies: those that are used through Ruby itself (same-ABI dependencies) and those that are not (any-ABI dependencies), such as the C libraries that are being wrapped around, the tools used at runtime by system calls, and so on so forth. To handle this, we ended up adding extra functions that handle the dependencies: ruby_add_bdepend and ruby_add_rdepend, both of which “split the atoms” (yeah this phrase sounds nerdy enough), appending the USE-based dependencies to each. They also have a second interface, in which the first parameter is now a space-separated (quoted) list of USE flags the dependency is conditional to.

This is not the only deviation from the standard syntax that ruby-ng.eclass causes: the other is definitely more substantial: instead of using the standard src_(unpack|prepare|compile|test|install) functions, we have two sets of new functions to define: each_ruby_$phase and all_ruby_$phase. This ties into the idea of supporting multiple implementations, as there are actions that you want to take in almost the same way for all the supported implementations (such as calling up the tests), and others that you want to execute just once (for instance generating, and installing, the documentation). So you get one each and one for all function for each phase.

There are more subtle dependencies of course; in the call to the each type of functions you get ${RUBY} to be the command to call the current implementation, while in the all functions it’s set to the first-available implementation (this is important as we might not support the default implementation of the system). The end result is that you cannot call neither scripts, nor commands, directly; you should, instead, use the ${RUBY} -S ${command} format (for the commands in the search path, like rake, at least), so that the correct implementation gets called.

Oh and of course you cannot share the working directory between multiple implementations, most of the time, especially for the compiled extensions (those written in C). To solve this problem, at the end of the prepare phase, we create an implementation-private copy of the source directory, and we use that in the various each functions; to be on the safe side, we also keep a different source directory for the all functions, so that the results from one build won’t cause problems in the others. To avoid hitting performance too much here, we actually do exactly two tricks: the first is to use hardlinks when copying the source directories (this way, the actual content of the files is shared among the directories, and only the inodes and metadata is duplicated); the second is to invert the order of the all/@each@ calls on the prepare phase.

While in all other cases all is executed after the implementation-specific functions, the all phase is executed before the other prepare functions… which are preceded by the copying, of course. This means that the changes applied during the all_ruby_prepare function are done over the single generic directory and then is copied (hardlinked) to the others.

So this covers most of the functionality of the ruby-ng.eclass, but we had another tightly-related eclass added at the same time: ruby-fakegem.eclass. Like the name let you guess, this is the core of our ditching RubyGems as a package manager entirely. Not only it gives us support for unpacking the (newer) .gem files, but it also provides default actions to deal with testing, documentation and installation; and of course, it provides the basic tools to create fake RubyGems specifications, as well as wrapping of gem-provided binaries. An interesting note here: all the modern .gem files are non-compressed tarballs, that include a compressed metadata YAML file, and a compressed tarball with the actual source files; in the past, there has been a few gems that used instead a base64/mime encoding for sticking the two component files together. For ease of maintaining it, and for sanity, we’ve decided to only support the tarball format; the older gems can be either fixed, worked around or replaced.

The boilerplate code for ruby-fakegem assumes that most gems will have their documentation generation, and tests, handled through means of rake; this is indeed the most common situation, even though it’s definitely not the same situation among different projects. As I said before, Ruby’s motto is definitely “there are many ways to skin a cat”, and there are so many different testing frameworks, with different task names, that it’s not possible to have the same exact code to work for all the gems unless you actually parametrise it. The same goes for the documentation building, even when the framework is almost always the same (RDoc; although there are quite a few packages using YARD nowadays, and a few that are using Hanna — which we don’t have in tree, nor will support, as it requires a specific version of the RDoc gem. an older one). The result is that we have two variables to deal with that: RUBY_FAKEGEM_TASK_TEST and RUBY_FAKEGEM_TASK_DOC which you can set in the ebuild (before inheriting the eclass) to call the correct task.

Now, admittedly this goes a bit beyond the normal ebuild syntax, but we found it much easier to deal with common parameters through variables set before the inherit step, rather than having to write the same boilerplate code over and over… or have to deduce get it directly from the source code (which would have definitely wasted much more time). Together with the two variables above we have two more to handle documentation: RUBY_FAKEGEM_DOCDIR that is used to tell the eclass where the generated documentation is placed, so that it can be properly installed by the ebuild, and RUBY_FAKEGEM_EXTRADOC that provides a quick way to install “Read Me”, ”Change logs” and similar standalone documentation files.

Finally, there are two more variables that are used to handle more installation details. RUBY_FAKEGEM_EXTRAINSTALL is used to install particular files or directories from the sources to the system; this is useful when you have things like Rails or Rudy wanting to use some of the example or template files they are shipped with, at runtime; they are simply installed in the tree like they were part of the gem itself. RUBY_FAKEGEM_BINWRAP is the sole glob-expanded variable in the eclass, and tells it to call the “binary wrapper” (not really binary, but rather scripts wrapper; the name is due to the fact that it refers to the bin/ directory) for the given files, defaulting to all the files in the bin/ directory of the gem; it’s here to be tweaked because in some cases, like most of the Rudy dependencies, the files in the bin/ directory are not really scripts that are useful to be installed, but rather examples and other things that we don’t want to push in the system’s paths. It also comes useful when you might want to rename the default scripts for whatever reason (like, they are actually slotted).

What I have written here is obviously only part of the process that goes into making ebuilds for the new eclasses, but should give enough details for now for other interested parties to start working on them, or porting them even. Just one note before I leave you to re-read this long and boring post: for a lot of packages, the gem does not provide documentation, or a way to generate it, or tests, or part of the datafiles needed for tests to run. In those cases you really need to use a tarball, which might come out of GitHub directly, if the repository is tagged, or might require you toy with commit IDs to find the correct commit. Yup, it’s that fun!

Ruby-NG: The Ruby’s eye (or, sometimes it’s a positive day)

I have ranted and ranted and ranted about Ruby packages not being good enough for packaging, I also have ranted about upstream developers not even getting their own testsuite cleared up, or being difficult to work with. I have complained about GitHub because of the way it allows to “fork” packages too easily. I’m not going to retract those notes, but… sometimes things do turn out pretty well.

In the past days I’ve been working toward adding Rudy in tree — as I don’t want to keep on building my own slow, hard, and boring scripts to deal with EC2, and I’m spending more time understanding how to get EC2 working than writing the code I’m paid to write. As I wrote before, this is another of those compound projects that is split in a high number of small projects (some literally one source files per gem!). It worried me to begin with, but on the other hand, the result is altogether not bad.

Not only the first fixes I had to apply to amazon-ec2 were applied, and a new version released, the very night I sent them upstream (and added them to Gentoo, now gone already), but also Delano (author of Rudy – and thus of lots of its dependencies) applied quickly most of my changes to get rid of the mandatory requirement for hanna, even on some packages I didn’t send them for yet, and released them again. Of course the job is far from finished, as I haven’t reached Rudy itself yet, but the outcome start to look much nicer.

I also have good words for GitHub right now: since it makes it very easy and quick to take the code from another project, patch it up and send it to the original author to be merged (and re-released hopefully). This also works fine with patches coming from other contributors, like Thomas Enebo from JRuby who sent me a fix (or “workaround” if you prefer, but it’s still a way to achieve the wanted result in a compatible way) to make newer matchy work properly with JRuby. On the whole, I have to say I’m getting quite positive about GitHub, but I’d very much like they allowed me to reply to the messages I receive by mail, rather than having to log-in on the system. I positively hate multiple mail systems, Facebook’s as well as GitHub’s, as well as most forums’.

And for the shameless plug and trivia time, I have more repositories in my GitHub page than items in my wishlist…

Anyway, back to work now!

The Ruby packager blues

Okay after ranting and implementing it’s time for crying. Because this whole day of work depressed me in a way that I wouldn’t have expected to be possible for Free Software.

Indeed, even though I added support for using the gems as base for the installation of packages in Gentoo, this does not solve one of my most important issues: tests that weren’t run and that’s bothering me.

It bothers me because for instance, running gruff’s tests, I found two bugs that are now in github (but not yet merged in the original master). It bothers me because it would take very little extra steps to get the things working properly, and nobody follows it.

If I started listing all the problems I found and I needed to cope with or workaround, I’d probably be feeling even more depressed. So I’ll just try to repeat some of the rules-of-thumb that you should follow:

please try to provide tgz as well as gems: while even Gentoo can deal with gems at this point, it still requires a double-extraction;
provide the tests and the framework to run them; without those, it’s difficult to understand whether the code works as intended or not;
always good, but especially if you don’t release the tgz: tag your releases! This allows to go look at eventually missing files (in tgz or gem) for executing tests, or to look for further changes that might have fixed things; if you use github, providing proper tags also provides automatic tarball downloads, so you don’t have to do anything else than provide the gem and tags;
try your own tests before releasing so that you’ll avoid releasing code that does not pass the tests in the first place; this is true also when the bugs are in the testsuite rather than in the code logic.

Unfortunately, the more I try to package software for Ruby, the more I see that these rules are blatantly ignored and people just release out whatever crap works for them. I’m seriously surprised that this stuff can be used in production, most of the times!

And if you feel like my work is worth something, and you’d like to say “thank you”, you can always look at my wishlist which also has useful technical elements; for instance I’d really like to read that OpenMP book so that I could learn some new tricks for making software behave better with multicore systems.