The hard return to ruby-hunspell and rust

As you can easily imagine from what I wrote the past few days, I’ve been busy trying to cleanup after myself in old projects that are near to abandoned. I wrote about my resolution to spend more time working starting new year, to save some money for getting my driving license and a car, and in the past days I cleaned up both the rbot bugzilla plugin (as well as rbot’s ebuild) and then nxhtml today, so it was quite obvious that I had also to take a look to long ignored projects like ruby-hunspell and rust (and of course rubytag++).

I started with ruby-hunspell as with that I can also fix the hunspell plugin for rbot (and thus put back in the only feature left over from ServoFlame). The first problem I wanted to tackle down was to remove the cmake dependency. As I said yesterday, I’ve started feeling the power in GNU make, and I also have enough reasons not to use cmake that if I could convert the build of the extensions (they are quite simple after all) to simple GNU make, I would do it gladly.

Indeed switching the build system to simple GNU make with some tricks around was not difficult at all, and the result is good enough to me. It’s not (yet) perfect, but it’s nicer. I also hope to generalise it a bit so that I can reuse it for rubytag++ too, and hide some of the dirty tricks I use.

Thankfully there is a good note about it, in the five releases between the previous time I worked on ruby-hunspell and today (1.1.4 then, 1.1.9 today), hunspell added support for pkg-config files, making the build system quite nicer. Also thanks to git improvements, making the tarball is quite easier. And thanks to the power of GNU make, instead of having a script, it’s now simply make tarball (although I will probably switch to make dist, I thought about this just now).

The problems weren’t laying too far though. First, I changed something on rust some time ago it seems, and now the ruby to C type function changed name, so I had to update the ruby-hunspell description script to suit that change. Then there is the problem that Hunspell now hides some of the functions being experimental (and by the way do they have any consideration for the object ABI? Dropping some functions on a preprocessor conditionals inside a C++ class isn’t the most sound of the ideas, I’m afraid…), so I had to comment those. The biggest problem came with the parsers extension, that used to provide bindings for the libparsers library installed by hunspell.

The libparsers library is not installed only in static form, and its headers are not installed at all. This is probably half intentional, as in they probably consider the libparsers library an internal library that other projects shouldn’t use, so they removed the header files, the problem is that they still install the library at this point, making its possible use a bit ambiguous. At any rate, for now I disabled the parsers extension, it wasn’t very much hunspell related anyway, so I will certainly prefer if they dropped it from being installed entirely. That extension was also the only one that had a testunit, I should write a testsuite for ruby-hunspell and the hunspell extension too, so that at least I have something to test with.

There is one big problem though, to release a new ruby-hunspell, which is a requirement for rbot-hunspell, I need to do a release of rust, too, but I don’t remember much of rust details, it has been almost an year since I last worked on it :( Additionally, my /tmp is noexec now, it wasn’t when I prepared the testsuite, so the tests fail as the shared object built in /tmp can’t be loaded in memory. I’ll have to test tomorrow if TMPDIR environment variable is respected, in which case I’d be using /var/tmp. I’ll also add a make dist target to rust so that I don’t need extra stuff to prepare the packages.

Finally, there is the problem of the git repositories: for some reason pushing to the remote repository accessed through this trick fails like there was nothing to push. Considering I now have my own (v)server, I’ll probably just move rust and ruby-hunspell back together with the other git repositories I export. This will also simplify thins when I’ll put gitarella back too.

Tomorrow will be dedicated to work for most of the time, but if I can squeeze some time for this I’ll try to address the issues, and I promise this time I’ll write more comments and documentation.

Serving community

I’m not dead yet (if I continue using this phrase, when I’ll be dead, someone will have to write on my tombstone “he’s dead, finally”), but I’ve been quite swamped out with my job. It’s quite nice when you work on two months on a thing, without test data, and when you get test data and you get to produce the test suite you found that everything you’ve being doing was based on a wrong assumption… nice, really, not.

Anyway, since I left Gentoo, I’ve decided to change a few other things in my style. Although I did report a few bugs on the bugzilla for a couple of issues I had, most of my work lately has been upstream. Yesterday accounted for a few fixes in xine, three patches for Valgrind (that you can find in my overlay, if you want to look at them), and one for FFmpeg (that will pass unnoticed as usual). As working downstream works nice only if you can actually be helpful to users by taking action immediately, it’s less a priority for me to work on that side, although it’s obviously not entirely gone, so you can see my work on an init.d script for rbot (always in my overlay), that I needed so that ServoFlame doesn’t need to be started by hand – and with the new connection handling code in SVN, it’s actually coping nicely even with network failures – and on an updated xine-ui snapshot that is now in portage, solving the obnoxious bug with the doubleclick to go fullscreen. In all this, I stopped writing down the timetable for stable for the packages though.

Today instead I tried working on the Rust project a bit more, mainly by writing a new page (not yet linked on the site, so I won’t link it here just yet) with some step by step guide on how to set up a developer’s repository for Rust, complete with CIA-on-push and mail-on-push so that the rest of the project (users and other devs) would be able to review the change (this was suggested in a nice video from Google’s TechTalks — you can watch the video with your favourite media player, that I hope will be xine, by downloading it for the PSP/iPod; the mp4 H264/AAC file played nice for me on xine even with the green blobs.. see later on this entry); unfortunately the default update script for git installed in the hooks directory does not mail the actual diffs as far as I can see, and it doesn’t really provide a decent subject, so I’ll have to work on it a bit tomorrow.

I wish to thank David who started working on Rust with me, he’s probably the first person who was ever interested in working in a project of mine :) This is why ost of my projects ended up dead, being the only one working on them, I often felt liek they are of no interest to anyone else.

Anyway, let’s move on a bit on the green blobs I named above. If you ever tried to play an H264 stream with xine, you might have experienced this problem: green rectangles touching hte left border of the video appearing through all the stream. It doesn’t happen on all streams, but it does on some. With Valgrind’s help I was able to find the «leaf cause» of this problem, as it reports an “invalid read of size 4” in a section of the MMX code, and I can report that by disabling MMX code in FFmpeg itself the problem is solved, but why this happens it’s still a mystery to me, someone else should look into the issue… Mike that would be you, possibly :)

Anyway, tonight I was finally able to stop worrying about Windows’s crazy behaviour when a symlink is stored in a 7zip file (the extracted file is only a few bytes long, but being named .exe is enough for Windows to try running it… luckily it only produced an Invalid Instruction kind of error, rather than killing the system altogether), so tomorrow a new segment of work will start for me, and that might give me more time between one battery of tests and the other.

One of the things I am considering to relax, is to replace my current setup of xmodmap, xbindkeys and xvkbd by writing a new application that binds mouse buttons press with a keysym press, so that I can finally stop patching evdev with Olivier’s patch to use my LX700 keyboard, and finally close that odissey. With XCB the task actually seems way easier than with Xlib, I probably can write an xbindkeys workalike in very little time, but there is more involved in this as I want to replace three programs and not just one.

Right now I have evdev patched so that some of the “buttons” of the keyboard (the ones that are actually on the keyboard rather than on the mouse) are reported as keys again, then I xmodmap their keycode to a keysym (that is also bound on KDE’s shortcuts) and I can use those; for the forward/next buttons on my mouse, I instead use xbindkeys to bind the mouse press to a call to xvkbd to send the keysym, this work with a fork() call which is far from being lightweight. What I want to do is to use a vanilla evdev, no xmodmap, no xbindkeys and no keysym, but a xmouse2sym background process that reads a simple file containing a series of pairs «button = keysym» (similarly to .xmodmaprc), find if the keysym is already registered, in which case the proper keycode would be used, for the keysym that are not already registered, it would take a free slice of unused cods to allocate the key.

I’ve looked up the XCB methods to do this, and it’s mostly straightforward, and it should also be quite fast to implement with XCB compared to Xlib, but the big problem is that the string form of the keysym cannot be used internally, you need to use the number form, which usually is returned by XStringToKeysym() function, client-side… this function is not present in XCB itself, so I have either to use Xlib to get that function, or I’ll need to write my own function to get this data. Such a function would be nice in xcb-util so I might look into it… maybe I’ll be able to finally learn lexers and parsers ^_^

Okay so it’s now quite late and I should be sleeping.. it’s just that I don’t really feel that nice lately after sleeping, probably an aftermath of leaving Gentoo, or just my life that’s trying to tell me I’m not immortal, and I should employ it better, who knows…

Virtualising C in Ruby

As I’ve now left Gentoo, one of my major projects is Rust, that if I can actually complete will be a nice piece of software for ruby developers out there, as it will allow to write Ruby extensions based on C libraries without the need to write all the boilerplate code that can easily be copy pasted wrong, or that, if flawed, would need to be fixed for N instance of the same piece of code.

As I said before, Rust is not yet to the same level of C++ bindings generation as my previous generator, but I’m still quite satisfied with what it’s coming to be because the code is way cleaner that the generator, and way more flexible.

One of the features that my previous generator had and Rust didn’t up to tonight were the «virtual variables». What is a virtual variable? Well, when you write a C++ class, you don’t usually make variables public, or the user (of the library, not the final user of course) can change these to values that are not supposed to be set, and does not allow the code of the class to take changes into account; for this reason the usual method to allow the class’s users to actually change the values of the internal variables you use the so-called getter and setter methods, usually named foo and setFoo or getFoo and setFoo (where foo is the actual name of the variable with a recognisable meaning, and m_foo – or variation thereof – is the name of the private variable). You can do the same in Ruby of course, but there is also the option of naming the methods foo and foo=, and then use the variable foo as a generic variable for the instance, both getter and setter methods will be called automatically.

What a virtual variable do in Rust, is wrapping getters and setters so that they are visible as a variable, with foo and foo= as Ruby function names (and at the same time leave the true name of the methods available for library compatibility). It’s a nice way to wrap around a lot of getter/setter methods pair, and RubyTag++ had quite a lot of them.

Even on the previous generator, the setter and getter pairs were simply expanded in functions, but with the new architecture, this is made even simpler, as I just have to create the two method objects, and then yield them so that the user (of Rust) can provide the extra support needed, like providing the presence of an instance parameter for the C class wrappers (that now I think I should have called virtual classes).

Now, the other feature that is needed for RubyTag++ (although I’m not yet sure if that is the last feature) is an easy way to convert from Ruby integers (be them literals or constants) to C/C++ enum values. While the generator didn’t really carry the values of the enum together with the enum itself, I’ve been now thinking that it would be nice to check if the value coming from Ruby is actually in the list of supported values for the enum, and throw an exception otherwise, instead of passing a maybe incompatible parameter to the C/C++ code.

From this I also considered another useful thing to do with enums: virtual enums. Basically, many C libraries provide some list of #define constants that are progressive, and are used to indicate one out of a series of possible values for a function parameter (for instance). Xine does that, for visual types. I can leave the parameter for the xine_open_video_port() function to accept an integer, and then hope that xine-lib handles it properly, or I can make Rust smart and create a virtual enum that would check for the value sent by the Ruby layer, and again throw an exception if an incompatible parameter is passed.

And again, I’ve been thinking a way to handle the usual situation where a function returns 0 for a success and then a positive or negative value with the error number, or when they return 1 for success, 0 for failure and require calling another function to get the errno (this method is often used in PulseAudio, Avahi and other libraries too). What I’m thinking here is doing sorta like Ruby itself does with syscall errors (Errno:: module) and provide virtual exceptions, converting the integer errno values in proper Ruby exceptions.

What I’m trying to do now is to make Rust be not just a way to produce the C code needed to bind 1:1 the C++ classes or C function sets into Ruby, but also a way to produce an extension that uses Ruby coding standards, as much as possible (this is also why camelCase names for methods are replaced by underlined_names, like QtRuby/Korundum do already).

I’ll try to write more in the next days about the limits of Rust (there are some obvious and some less obvious), and why I’m writing it using C++ features; hopefully by dumping here my thoughts, I’ll be able to extract some documentation to ship with Rust itself.

Okay, I exaggerated

When I previously wrote that Rust is to the same level as my previous Ruby extensions generator. It was and is, yes, ready to build Ruby-Hunspell, to the point that the current GIT head of this one is currently only based on Rust and thus depends n it, but I did forget one thing. When I converted ruby-hunspell from a hand-crafted C source file to a generated extension, I indeed extended the generator to add a few more cases that were needed for hat particular task, but RubyTag++ was already covering a lot more tasks, that I still haven’t covered in Rust.

But this is not a bad news anyway, the work is starting being less messy and more comfortable, now that a pattern starts to appear. While the original generator was a mostly top-down approach, slightly object oriented, but with a lot of if and case conditions to abstract the various differences between a method and a constructor and between parameters, this time I’ve started to use a more object oriented approach, as it should happen with Ruby.

Just tonight I found a pretty good abstraction: everything that is described in the interface description file (beside the bindings object itself) is an Element, that can contain more children Elements; every Element provides to the bindings a declaration (that also contains the declarations of its children), a definition (that also contains the definitions of its children) and an initialization (that also contains the initializations of its children), these are the three main components for Ruby bindings, the first two are simply instances of the respective C/C++ concepts, while the latter is simply the content of the initialisation function for the extension itself, where modules, classes and so on are registered with Ruby.

So while I’m developing Rust, and adding more complex features, I start actually reducing the code and making it more easily readable. By moving the C/C++ code on separate files, also, I’ve made that also more readable, even if that means wasting some space to contain them.

What is missing right now to convert RubyTag++ to Rust? Well, quite a lot of code, constants I just added tonight, but then attributes have to be added too, with their set/get structure, and inheritance has to be tweaked, as right now is likely not working, and I have to get a better support for sub-modules, and classes within classes.

But I also implemented a few things that were missing or pretty much unusable in the previous generator of course, for instance you can now describe custom methods inline in the description without having YAML for describing a template, and you can just make some differences between the methods if they are slight: the description file is in Ruby, so you can use variables and treat them how you prefer. And I started adding support for classes wrapping around C interfaces, together with the start of ruby-xine bindings (that will be my real-world scenario to keep in a working state).

When I started, I decided I wanted to write as many test cases as I could to make sure that I didn’t introduce regressions between the versions; unfortunately I haven’t been able to write more than a couple of testcases up to now, but I’ve being using ruby-hunspell and ruby-xine as my test suite… this is not going to work well on the long run, so tomorrow I’ll see to add some more test cases describing the features I added tonight, hopefully this way I can just run Rust’s test directory (that uses a pretty old-style Makefile to run the tests, but I haven’t being able to understand Rakefile correctly.. unless it is just a way to invoke system from ruby, and thus has no advantage over make other than not requiring make installed) rather than building two extensions and run their testsuites (actually ruby-xine still hasn’t had a testsuite, but it will have tomorrow, it will also be quite easy, as I just need, as a “testsuite”, a single unit with a few tests, at least for now).

I’ll remember again for who’s interested, the rust site is at RubyForge, there’s a rust-devel mailing list, you can subscribe to ask questions or propose patches, enhancements and similar, there’s also a CIA project page that I just set up (thanks Micah for your site that is improving a lot lately!) where you can follow the updates; as soon as I ge something working with ruby-xine, you’ll find that also as a GIT repository under Rust.

And now the questions are: will I be able to write an user (as in person using it, not the kind of final user one expects) documentation for Rust? And, how can I focus some attention on Rust so that there will be someone else but me interested in it? :)
I know that by starting ruby-xine I’m likely to get the attention of Luis (metalgod), but that’s hardly enough, logically.

Update (2017-04-22): as you may know, Rubyforge was shut down in 2014. Unfortunately that means that most of the Rust documentation and repository are probably gone (I may have backups but I have not uploaded them anywhere).

Rust is almost ready

Okay, so today I restarted eating almost normally, I probably got the flu that seems to have voyaged through Europe in the last days, listening to what a few people already told me. I wasn’t sure if I could sustain my usual workflow today though, because I was still missing force due to the two days I passed without eating at all, so I sent the day working half time on job stuff and half time on Rust. This latter one came almost to completion in the mean time.

if you forgot about it, Rust is my bindings generator for Ruby, evolution of the bindings generator I used for RubyTag++ and Ruby-Hunspell up to now, that allows you to generate a working Ruby extension binding a C++ library by just describing its interface. While the original bindings generator used a YAML description of the interface, his wasn’t as extensible as I needed, so this time it is using Ruby itself, kinda like ActiveRecord and rails. I’ll show you all the syntax tomorrow when I upload the code, as I’ll need to write some documentation for it anyway.

So what it is working in Rust and what will still require work? First of all, Rust is now up to the same level my previous generator was, which means it’s far from being complete as I want it to be (the target is to be able to write just as easily bindings for C libraries that use OOP interfaces, like Avahi PulseAudio or xine), but it is important to me that it reached this point because this way I won’t have to maintain both Rust and my other generator, and I can test its working state by using Ruby-Hunspell and RubyTag++ as regression tests until I finish the regression tests themselves (I only wrote two up to now).

I’ve now asked hosting on Rubyforge, if all goes well in the next few days I’ll put up a draft of a site on there, and then start pushing my changes to the GIT repository there (sshfusefs is quite slow, but it works nicely for what I need to do). I’ll need a logo as most of the Ruby projects have an appealing one; if anybody have an idea or a proposal, it is welcome, my graphics skills don’t exist at all).

Hopefully, it will be possible through Rust to have bindings for the libraries I named above without too much work. I still wonder if it makes sense to have them as separate projects (as Ruby-Hunspell currently is), or if it would be simpler to leave them all live under Rust itself; but for that there will be time to decide.

For tonight, I can feel happy, and work on a few more testcases. I’d like to be able to watch some Anime too, but this depends on a series of factors, like in which bed I’ll sleep, tonight (while I wasn’t feeling well I took possess of my mother’s bed as it is more stable than mine, and I had enough nausea without adding the mattress deforming every time I moved, and here I don’t have a power socket to connect the laptop to while I watch Anime).

Oh well, it’s rusty

Yes, I am running out of blog titles, especially lately.

After the whole concern about the destiny of Gentoo/FreeBSD, and a lot of other things that happened in the last months, I’ve decide that it might be the moment for me to retire from Gentoo… or at least it will be the moment, in a few months.

I’ve sent a detailed mail on gentoo-core yesterday, from which I can report I’m not going to leave now. I do have a lot of stuff in my TODO list that I want to complete first, like ZFS support for Gentoo/FreeBSD, finishing clearing the licensing, completing the Gentoo/FreeBSD/SPARC64 stage, finding the cause of the misalignment there, and so on. I won’t take new complex tasks though, as if I continue adding stuff to that list, not only I won’t ever be able to leave Gentoo, but I won’t be able to live myself.

Many colleagues suggested me to take a vacation of a month or two and then come back stable; I doubt this would change anything about what happened in the past months, but I have to admit I won’t be able to finish much from the TODO list if I continue this way, I need to relieve part of my stress.

So I decided, I’ll take a week or two mostly off, then jump on again, finishing my TODO list, and when I’ll come at the end o the list, I’ll depart, unless something else happens that makes me change idea. It will be a few months as I said, so there is space for me to reconsider, but now I’m ready to go.

After this, that is just an introduction, let me write about the topic of this entry. One of the things I planned to do during my break (yes of course, I know I should try to relax during a vacation, but I’m unable to stand doing nothing :/) is to finally write some xine Ruby bindings. I talked before of my work on a script that generates Ruby bindings for C++ libraries easily, but that code wasn’t adaptable for C bindings too… so I decided to rewrite it entirely.

Why rewriting it rather than extending it or writing a different script? Well the code was mostly non maintainable as I wrote it and changed it as I needed; writing testcases for it was basically impossible, and the code was shared already between two repositories (ruby-hunspell and rubytag++) that didn’t diverge most of the time, but were difficult to make keep safely in sync. Also, I started working on Rails, and ActiveRecord way to define tables caught on me a lot.

So I started thinking a bit more about the generator script, and I decided I wanted something that would allow defining the code to bind via Ruby itself, and that would be installable in a system so I could release it on its own package, as build-time dependency for the bindings. but one big problem I had to cope with for sure: could I call it ruby-bindings-generator? Of course not, the name sucks and it doesn’t keep the mood with many other Ruby packages like Rake Rails or Gems.

But I do know someone who has a true passion for names, Markey from Amarok! And as soon as I proposed him a possible “ruby dust” name, he suggested instead the shorter “Rust”… why not? It doesn’t really sound that bad and it allows a lot of different word plays, and could easily provide a graphical logo (I’m currently thinking of some rusty jewellery piece with a Ruby in it), if the project were to actually get some work.

I have yet to publish the GIT repository with Rust sources, as it’s not really ready for prime time yet, but I’ll do so in a couple of days, or maybe even tomorrow. The code might be a bit slower than before because I use replacement over code templates rather than direct string output from methods, but it should be easy to maintain. I’ll also have to write some documentation for it so that it can easily be read, it’s time for me to learn rdoc (you’ll probably see some new Ruby links to the feed on the right of my blog while I do search for stuff and work on this project), as well as Rake.

And of course, after all the problems with Licenses lately, I’ve decided that even if the previous generator was licensed under GPL2, Rust will be released under a way less troublesome MIT license, as it’s after all not something anybody would be changing secretly to get any advantage (while before the generator was always copied in the sources, Rust would just be installed).