JRuby in Gentoo, a bumpy road

So as you might remember one of the reasons for my restructuring of Ruby eclasses has been the chance of getting “proper” support for JRuby in Gentoo. With proper I mean that the packages are declared to support specifically JRuby, and at the same time get tested so that they actually work with it. For this to work out properly it also requires that the testsuites for the packages were written in such a way to work fine when run through JRuby. But I already ranted about that so I won’t go into those details again now.

Instead of writing about the Ruby-NG side of the fence, I’d like now to talk about the (to me, much more unknown) Java side of the fence. Now, I know near to nothing about Java packaging. I know my way around the Java language, sort of, but not much about the packaging side of it: ant is mostly black magic, and the whole mess with Jar files, manifests, and dependencies are huge question marks in my view.

So the first problem, is that while JRuby has been in tree for a while, there was near to no way to make sure that it worked as intended, before. This resulted in pretty erratic behaviour with it (like the FFI-based libraries not working at all); with the root causes hidden by sub-projects as well (in this case, jffi was being installed without the needed JNI library). This is also often tied with the way JRuby is developed: it’s modularised, but the projects are sometimes only used by JRuby itself, so without it working, it’s almost impossible to tell if the rest is working as intended either.

There is also the problem of packaging what was not available before; for instance I found out that JRuby now needs a Java implementation of a YAML parser, yecht written by a JRUby developer Ola Bini, which is bundled by JRuby itself, but we should have been packaging ourselves. Thanks to Fabiano “elbryan”, I got an ebuild for yecht itself… the problem is that the version shipped with JRuby also contains the classes needed for the JRuby bindings, which aren’t available when building the Jar standalone. D’uh!

Other problems are really tied to the projects standalone: as I said above, we had some trouble with JFFI and the fact that we didn’t install the JNI library (an ELF shared object) that it needs to work. Now we do, but… every time we start jruby, whether we’re going to use FFI or not, we’ve got to call into java-config (which is a bash call into a Python script), to ask it where the jffi library is so that we can set a property… it doesn’t sound right! And the same applies to dev-java/jna (which seem to provide similar features, by the way). The problem with both is that they really should have a way to tell at build time were their native libraries should be found, and then leave the property as a way to override it… Unfortunately it seems like there is no way to do that. I reported a feature request for JFFI and we’ll see how it’s going to turn out.

Unfortunately, the problems don’t stop here. Another problem happens with the OpenSSL-compatibility extension provided by dev-ruby/jruby-openssl; the idea is to provide the same interface of Ruby with the OpenSSL extension, but using Java libraries instead. Since the standard JRE does not provide all the needed interface, this extension uses BouncyCastle to do the work, and that does not work fine here at all: not only the extension bundles a copy of bcprov and bcmail (even in the very git repository!), which by the way don’t seem to work right for me, maybe because they are the JDK 1.4 versions, while we use at least Java5, if not 6 altogether, but more importantly our already-present bcprov/bcmail packages fail badly. The issue here is that bcprov can be loaded as a JCE provider; JCE providers need to be signed, and obviously our rebuilt Jar file is not signed. This is a huge problem that the Java team should most likely look into and something I really cannot look into myself.

On the bright side, though, tonight I’m going to commit a new JRuby ebuild with a much simplified launcher script (and no more symlinks around!). This basically cuts all the conditionals for cygwin, as well as for dynamic discovery of JAVA_HOME path and replace them with assumptions about the Gentoo setup (JAVA_HOME is filled in during install phase) and calls into java-config. Hopefully, this should reduce both possibility of mistakes, and reduce the time needed to process the script. Unfortunately, the script as it is, uses bash, that we all know is far from the fastest shell out there. Porting it to use pure sh is definitely not possible — although one could argue that Java’s own startup is definitely going to take up more time than using bash).

Oh well, we’ll see how it goes from here.

6 thoughts on “JRuby in Gentoo, a bumpy road

  1. Sorry if it was my fault that the JNI library wasn’t being installed. I can’t remember too well, to be honest, but I know my understanding of the subject was pretty weak back then. As it happens, I’ve been doing a lot of work with JNI recently and understand it a hell of a lot better now. If you need me to look at the problem, let me know. I can’t speak for JFFI specifically but I can tell you that if you use System.loadLibrary instead of System.load, it’ll look in standard places like /usr/lib and LD_LIBRARY_PATH. You can also pass other locations through java.library.path.

    Like

  2. Actually I think it may be a moot point. java-config isn’t just needed to find the native library location, it’s also used to find all the required JARs. This is how most Java stuff on Gentoo is launched because there is no standard way to locate JARs. It’s a little bit like how shared libraries are located. I’m not sure how other distributions handle this but I think it’s quite a nice system.The bash script could probably be rewritten for sh. In fact, it looks like no work is needed. I just changed the shebang for one application launcher and it worked fine.Other than that, the speed impact is negligible compared to Java’s own startup time. Hardcoding JAR locations would probably make updates break things more often and I don’t think anyone wants to rewrite java-config in C.Java takes a long time to start, even without all the JARs JRuby requires. That’s why things like Nailgun (http://martiansoftware.com/… have been developed. Have you noticed that JRuby includes it? I never tried to get it working under Gentoo.

    Like

  3. James don’t worry about JFFI, having scarce documentation definitely *does not* help us the least — and upstream seem to also be gone for a while.As for @java-config@, I know the wrapper’s impact is negligible, but it still sounds a bit fishy in my view… and the JRuby script cannot be moved to standard @sh@ as it’s definitely using bash functions. At least for now.For what concerns NailGun, that was one of the reasons why I undertook the cleaning up of the script, together with JFFI: it wasn’t working as it assumed to find @ng@ on jruby’s binary directory, rather than in @/usr/bin@. In 1.4.0-r2 and later it should be all working fine :)

    Like

  4. I actually meant the bash scripts that the Java eclasses generate. The launchers look something like this…#!/bin/bashgjl_package=audiverisgjl_main=”omr.Main”source /usr/share/java-config-2/launcher/launcher.bashNo problem there obviously. And /usr/share/java-config-2/launcher/launcher.bash is a bit longer but I don’t think there’s anything there that would cause a problem. I’ll ask about it in #gentoo-java.

    Like

  5. Actually, before I do that, can you explain why sh is faster than bash when /bin/sh is just a symlink to bash? Evidently it works a little differently because a different prompt appears but do you essentially get a subset of bash’s functionality and hence it runs faster? I know that Ubuntu symlinks it to dash, does that make it even faster still?

    Like

  6. Work has been underway for a while for Gentoo to move @/bin/sh@ to a true POSIX-reduced shell rather than the (bloated, and objectively slower) bash… I’m not sure whether bash is smart enough to disable its extensions when launched through @/bin/sh@ and thus gain some speed.But on Gentoo/FreeBSD for instance, @/bin/sh@ is *already* not bash…

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s