Ruby-NG: Up the JRuby Ladder

With the introduction of the new (but nowadays not so new) Ruby eclasses, Gentoo started supporting JRuby as a proper option. This means, among other things, that the libraries installed for JRuby and Ruby 1.8 (MRI18 in the future of this post and others) are now separated (before JRuby “sucked in” the libraries from MRI18), and that you can decide whether to install for MRI18, JRuby, or both of them. It’s also a nice testbed for the future working with MRI19.

Now, supporting JRuby is, to be honest, not an easy task, and I’m doing it mostly alone. Why is it not so easy? The main reason is that I’m striving for a higher ground than we had before: I want what we install for JRuby to be sure of working! And this means, mostly, that I have to make sure that tests work. This gets tricky for not just a couple of reasons; among them, Rails tests require you have the SQlite3 bindings for ActiveRecord, which in turn rely on SQLite3-Ruby. This is a problem because, simply put, SQLite3 is broken beyond repair, I’m afraid.

The problem with SQLite3-Ruby is that it relies on SQLite3 itself, which is a native library. To access it you have two options: you can use a native Ruby extension that provides MRI18 and MRI19 bindings for SQLite3, or you can use the dl-based driver that uses the dl.rb library to open the library and access the functions. Not only the tests currently fail on MRI18 and MRI19 with the native extension, but they fail to work at all with the dl-based driver! Let’s not even go to wonder how it works with JRuby.

Other problems with JRuby involve both bugs in JRuby itself (fakefs, and thus rspec’s testsuite, is kept down by that), and by incompatible design (racc is giving headius a bit of trouble for instance), like using fork (which does not work on JRuby at all, nor it seems on Windows). Thankfully, headius (Charles Nutter) is helping me out by checking the code of various extension, and coming up with possible workarounds.

Now, as I said I’m doing the support for JRuby mostly alone in Gentoo, as Hans is focused on just Ruby 1.8 and Alex looks at both 1.8 and 1.9. I try to focus more on Ruby 1.8 and JRuby. My interest in JRuby is vested: for my Ruby-Elf project I could really use threading, and Ruby 1.9 is not really providing anything useful to me (it only allows thread to work in parallel if they are not holding the global interpreter lock, which basically reduces to only having sense when calling into a blocking operation or to an extension outside of the Ruby language, like an extension — this is probably good for network-based software but not for CPU-intensive processing like Ruby-Elf).

Anyway, to understand how the current porting is taking place, I’ll add two graphs for now; I know pie chart are bad yadda yadda yadda, but since I haven’t sampled this yet (I’ll do my best to sample it daily for a while and then chart it properly with an area graph), it’ll have to do. It also shows well one thing: we have to get rid of the old eclasses usage and that’s going to take a while still. The second chart instead shows how many ebuilds are available for a given target; I have to note that for now there are exactly two ebuilds that are not available to Ruby 1.8 implementation: test-unit and jruby-openssl. In the future this is likely going to increase, especially for JRuby, as I’ll probably add a couple more JRuby-specific extensions that I’m interested directly or indirectly about. As for the other graph, I’ll start sampling the data from now on, to see how these values increase in the future (and if they’ll ever decrease).

A huge thank-you has to go to Zac for providing me with a script to be able to get this data (ironically, in Python):

#!/usr/bin/python

import sys
import os
import portage

def main():
    bins = ('ruby', 'ruby-ng', 'ruby_targets_ruby18', 'ruby_targets_ruby19', 'ruby_targets_jruby')
    stats = {}
    for k in bins:
        stats[k] = set()

    portdb = portage.portdb
    portdb.porttrees = [portdb.porttree_root] # exclude overlays
    for cp in portdb.cp_all():
        slot_dict = {}
        for cpv in portdb.cp_list(cp):
            slot, iuse, inherited = portdb.aux_get(cpv, ['SLOT', 'IUSE', 'INHERITED'])
            slot_dict.setdefault(slot, {})[cpv] = (iuse, inherited)
        for slot, cpv_dict in slot_dict.items():
            cpv = portage.best(list(cpv_dict))
            if cpv:
                iuse, inherited = cpv_dict[cpv]
                iuse = set(iuse.split())
                inherited = set(inherited.split())
                if 'ruby' in inherited:
                    stats['ruby'].add(cpv)
                if 'ruby-ng' in inherited:
                    stats['ruby-ng'].add(cpv)
                    if 'ruby_targets_ruby18' in iuse:
                        stats['ruby_targets_ruby18'].add(cpv)
                    if 'ruby_targets_ruby19' in iuse:
                        stats['ruby_targets_ruby19'].add(cpv)
                    if 'ruby_targets_jruby' in iuse:
                        stats['ruby_targets_jruby'].add(cpv)

    for k in sorted(stats):
        sys.stdout.write("%st%sn" % (k, len(stats[k])))

if __name__ == '__main__':
    main()