rbot and Gentoo

One nice thing about rbot is certainly, to me, the fact that installing it is mostly just an emerge away. For quite a while there has been a live ebuild of rbot that fetched first from Subversion and now from GIT.

The only problem was that the ebuild fetched the sources, then made a gem, installed it, and then removed some files. Not exactly the shortest way around.

Thanks to the guys in #rbot, this has been solved. Now rbot installs itself without using gems, but by simple setup.rb. This means not only that you won’t need rake and zip to install it, but also that the installed files have decent path, rather than being buried inside the rubygems directories.

An additional advantage comes from the size of the installed files; when you install a gem, a copy of the sources is left in the rubygems tree; this can be quite a waste. The size of the rbot binpkg has halved:

-rw-r--r-- 1 root root 959426 May 24 12:08 /usr/portage-packages/All/rbot-9999-r8.tbz2
-rw-r--r-- 1 root root 425034 Jun 23 20:18 /usr/portage-packages/All/rbot-9999-r9.tbz2

Add to this that the ruby-gettext dependency is gone too, now you got a nls USE flag that allows you to choose whether you want localised bots or not, and if you don’t want them, it won’t ask you for ruby-gettext at all. Nice, uh?

But it’s not finished here. I was hacking at the ebuild today for a different reason. rbot has a figlet plugin, that executes the figlet command and copy the output on IRC. The ebuild, though, didn’t have an option to pull in figlet, and I wanted to fix that.

So together with the removal of rubygems dependency, the new ebuild also has USE flags to enable the figlet, cal, host and fortune plugins. So you either get your dependencies pulled in, or the bot has the plugins disabled out of the ebuild. Cool!

And talking about the fortune plugin, listing the categories didn’t work on Gentoo, as we install the database in a different directory than rbot expected them to be. This is fixed in rbot upstream repository by yours truly.

Always a pleasure to fix rbot on Gentoo ;) Kudos welcome as well as bribes.

About Ohloh’s “Journals”

Although I’ve been an user for a while, I haven’t talked at all about Ohloh Open Hub before. The whole idea seems quite fun, as Lennart said, being able to actually receive “kudos” for the work done for the Free Software can be a real turn on for some volunteers at least.

And thanks to the fact that a lot of projects are using git nowadays, it makes it possible for one-time contributors with no commit rights to have their patches widely acknowledged :) Too bad that it doesn’t support Mercurial yet, so xine-lib is not tracked :/

One thing I actually see being kinda useful, though, is the new journal feature. SourceForge and other *Forge-based systems provide “Diary and notes”, but it’s pretty much useless if you’ve got a “real” blog. What the journals seems to be replacing, rather than a blog, is Twitter.

Indeed, I’ve used twitter before to write down notes about what I’ve been doing on projects, not full blown status updates like I do for the blog, just a few notes here and then through Kopete or Adium.

Considering I haven’t even seen the twitter bot online in the past weeks, this is a very welcome feature to me.

So if you are interested to follow minutiae of what I’m doing for various projects, feel free to just watch it on the Open Hub page. Too bad friendfeed does not support Ohloh yet.

On a more interesting note, the rbot-bugzilla plugin, that allows to access bugs information through rbot (Ruby Bot), is getting in great shape. Thanks to Robin (robbat2), the plugin can now announce new bugs in channels (it still lacks filtering, I’m working on that), and it has an “archstats” command like jeeves did (just, not limited to one bugzilla installation).

Stay tuned for future development, I really hope to be able to do something useful again soon :)

Announcing bugs

You might or might not know that I started working some time ago on a bugzilla plugin for rbot. The idea started from my previous try to get a bot to access bugs, which was in turn due to GenBot, an old bot that used to be around in #gentoo-* channels, disappearing.

What the current plugin can do is to fetch the XML with the data about the bug, and parse it to show a summary. Up until yesterday, you had to specify which bug tracker to get the data from, but now I implemented per-channel defaults.

The interface is still quite a bit rough and probably altogether slow, but I hope to improve it in the next days. Right now you can choose a default bug tracker to get the bugs from if a tracker wasn’t specified on the request. It also does listen to all the messages to see if someone is talking about a bug, kinda like jeeves does.

What I’m hoping to implement in the next days also is announcing bugs, again, kinda like jeeves does. Thanks to Robin, I now know how to deal with this using just the HTTP interface of Bugzilla, so in the next days I should be able to make ServoFlame join #xine@oftc and announce new bugs as they are reported.

My current problem is to avoid iterating through multiple arrays at once to get the announcement on all channels, I suppose I’ll have to parse the settings a bit so to index them per zilla. It’s more an implementation detail though.

More interesting is the refactor I started on the code, trying to document it a bit more with comments, and adding Exceptions where needed rather than hardcoding error conditions. I have to say I like the code better now.

Anyway, stay tuned for more rbot hacking in the next future, I just hope to find a way to contact markey or tango outside Freenode, as I’m steering clear of that network now (as I’ll steer clear of any network enlisting people with no trace of maturity).

Yet again rbot

Yes, I know I’m boring when I start talking for a few days one after the other about a given topic. At the moment I’m boring with rbot, on which I worked a bit today too.

First of all, the news is that no more dependencies are needed or added to the ebuild, good. I also fixed the time plugin so that it’s disabled if timezone USE is not present. This should let the build stay quiet for a while.

Then I cleaned up the bugzilla plugin a bit, re-enabled it in ServoFlame, added the xine bugtracker and… prepared to release it.

I started versioning it on git, and I put a tarball on my site so that it’s more easily found. I also changed the license, it was MIT, but it’s now Affero GPL 3.

Why Affero? Well, plugins for a bot is something you often end up editing to improve; by forcing users to actually make available their changes it should make it more easy to improve the code on the long run. Also, I find it interesting to see if actually it would be used and the license properly applied (note that the easiest way would be to send the patches upstream so I actually make the modified plugin available to anyone).

If there is request, I’ll probably put back the old MIT-licensed version; actually at the moment even ServoFlame is using that, or rather a modified version of that, rather than the AGPL3 version (which is more advanced); this is because of the other thing I’ve done after preparing a tarball of the plugin: I wrote an ebuild for it.

So if you have my overlay, just emerging net-irc/rbot-bugzilla will give you a rbot with my bugzilla plugin enabled, the httpclient package installed (as it’s a dependency), and the default configuration already with the bugzillas ServoFlame access.

For now the ebuild is in my overlay, I’m still debating with myself if I should add it to portage (and then actually use portage to maintain the bugzilla plugin used by ServoFlame, too), or wait.

Thinking nice of bugzilla again (and some rbot news)

Bugzilla always gave me the idea of something slow, heavy, and an absolute waste of resources.

I start to think better of it. While certainly it will kill easily a server when used with big projects like Gentoo or KDE, it seems to be quite decent for a small project like xine on a small vserver.

The other reason why I thought badly of Bugzilla was the amount of perl modules one had to install to get it to work; luckily this problem is gone thanks to Portage, that takes care of everything for me (or almost everything).

The final reason why I thought Bugzilla was not feasible is the interface. God knows the default interface of Bugzilla is horrible. it’s strange considering the the developers are Mozilla related, and it’s also strange that it uses so many tables around for the same reason. I thought tables for non-tabular data were quite out of standard nowadays, I don’t even use them myself when I’m paid to make a site!

Anyway, the template engine used by Bugzilla is quite impressive, and it can be used to actually get Bugzilla a decent interface. There are also some new skins available around that gives it a very interesting effect, although I haven’t used any of those yet. For now I stopped at cleaning up the interface a bit, removing the redundant actions listed at the bottom (they are the same as listed at the top), moving the logout action to the right of the screen like in other webapps, and so the search box.

I also removed a lot of information and links on the index, mostly redundant stuff or stuff like the infamous sidebar, that I never seen anyone using. It’s lot as clean as GNOME’s bugzilla, nor as shiny, but it’s not bad either. I hope to actually improve it in the future too. What I’d really like is to show the most recent reported bugs on the homepage, so that users would see more easily what might be their problem (if it’s a problem that came up recently like the infamous build failure of the internal libcdio copy with the new linux-headers).

Anyway tonight I switched the xine bug tracker to Bugzilla, importing by hand the open bugs from Flyspray (tomorrow I’ll import the closed ones too). I’d like to make the interface more shiny with some icons, but I don’t know which icons would look good with the icy look of the current skin (which is derived from the xine logo and the xine site).

I hope that mailing works, it’s the first time I set up procmail, and I’m not sure if it works properly because I see some errors in the mail queue, connection failures to the mail relay (which is internal to the farm where the vserver is located). Tomorrow I’ll see if it’s easier to just get it to send the mails directly.

If you read about my problem of having data filtered so that the password change requests don’t leak on the mailing list, the solution was quite easy, with procmail and a xine-project user (which is where flyspray was installed, so it existed already).

Also, thanks to the fact that bugzilla is, well, bugzilla, now ServoFlame on Freenode will answer to requests about the xine’s tracker too.

Talking about ServoFlame, today I also worked a bit on its bugzilla plugin, now it uses httpclient, rather than http-access2 (httpclient is the new name of http-access2; I also asked for marking httpclient stable today), and reuses the client instances for the same bugzilla, I haven’t checked if that means that it uses permanent connections yet, but I’d sincerely hope so, as that should improve multiple requests to the same bugzilla, especially for those trckers using SSL (like FreeDesktop). You may now use ServoFlame in all the channels where it’s present to request bugs about the listed projects, to know the listed projects, just ask him to “zilla list”. I finally implemented proper authorization support so that zilla list is available to anybody, but just I can add or remove trackers to the list.

For what concerns the generic rbot live ebuild (which stays where it is because upstream is now suggesting not to use rbot 0.9.10, the last release), I added an nls USE flag the other day, but now I force ruby-gettext on everybody; this is mostly because I want the gem to be generated as much as possible identical to the one that’s going to be released, which means that the locale files needs to be there; anyway it’s not a runtime dependency, you only need to install it to install rbot then it can go.

I also added a dict USE flag; this flag controls the dictclient plugin, a client for the RFC 2229 protocol (that is, dict), that uses the ruby-dict extension. As it needs the dependency or it fails to load, leaving it enabled by default wasn’t really a good idea in my opinion; now if the USE flag is disabled, the plugin is also disabled (by default, you might enable it manually, but then rebuilding with the USE flag enabled takes less hassle); as ruby-dict was not in portage, I added it, this makes it the third package added to portage just to get rbot plugins working (the other two were dev-ruby/tzinfo, for the time plugin, and dev-ruby/shorten for the shorturl plugin).

Unfortunately i already have disabled almost all the rbot’s plugin on my ServoFlame so I don’t see the failure to load most of the plugins when the extensions they need are not found. Tomorrow I’ll see to get an install of rbot on my workstation and with all the plugins enabled I’ll see what does not run correctly.

Okay now it’s very much late, and I should be sleeping already….

Some more changes on rbot

As you may or might not know, I maintain the net-irc/rbot package in Gentoo. In particular, I’ve been focusing on the 9999 live SVN ebuild, as upstream still hasn’t released some new versions since 0.9.10 which is quite old.

I’ve also been using the 9999 ebuild on the production server, and that’s where ServoFlame stays up from, so it’s usually quite tested.

Today I added two new USE flag in a new revision of the ebuild: translator and shorturl. These are present so that the dependency on dev-ruby/mechanize and dev-ruby/shorturl that the two plugins need.

What I’m looking forward to in rbot is to allow users to choose which plugins, needing an external package, should be installed, and which not, just by setting the proper USE flags.

This should make rbot administration even simpler for Gentoo users, isn’t that great? Well that and the fact that you need not to check which gems need to be installed, as Portage will take care of everything for you.

A nice productive day

I might be sick, or just crazy, or both of them, but I still think I’m quite more productive when I have fever, or the days around that time. Yesterday I had fever, and I was knock out till late afternoon, but then I started feeling better, and I started producing.

First of all, rbot’s init script in my overlay has been updated: Subversion trunk will now create a rbot.pid file inside the bot’s directory without need for start-stop-daemon to create it itself, which means I can finally let rbot fork on its own rather than forcing it to background (which is not really a good thing to do, if it can be avoided). Thanks to tango and jbn for looking after my raw patch.

Then I decided to finish the work with apcupsd; again in my overlay you can find a new ebuild for 3.14.0 based on the one found on bugzilla, but with a new apccontrol file and a totally renewed init script. This script can be multiplied, which means you can have a /etc/init.d/apcupsd.foo link, which would then start apcupsd looking for /etc/apcupsd/foo.conf configuration file. Thanks to apcupsd authors for implementing this feature in 3.14!

I’ve also attached the two UPSes to Farragut instead of Enterprise, as the latter is not a server and might as well be offline when Farragut is still up (for instance this is the case most of the times I’m outside for the whole day, or if there is noone at home); apcupsd on FreeBSD works nicely, and doesn’t require any fiddling with configuration, neither kernel side (the ugen support is built by default) nor with permissions (as the default is to run as root, this might change in the future, but as it is it’s fine to me; I’ll be working on a better handling of permissions on device nodes for Gentoo/FreeBSD, but it’s not in my priority list at the moment). Also here, they work perfectly fine.

I was also able to fix a bug in xine-lib, with mp4/mov files playback that used version 1 rather than version 0 of the media header atom, such as files generated by FFmpeg. The bug was reported on sourceforge already but I wasn’t sure what it actually meant and where to find a sample file; when I generated the same condition by chance here, I decided to take a deeper look; unfortunately MultimediaWiki doesn’t provide much information about that, but I asked Mike to give me an account there so I can try to write something useful, maybe next time someone else needs a mdhd atom description they won’t have to look at the sources of FFmpeg to see how it’s read and generated.

Then tonight I wanted to resume my work on implementing audio conversions inside the audio output loop instead of doing it for every decoder; it’s an hard work as it probably will require rewriting a good deal of code, but it should be rewarding once it’s done. Right now there are a bunch more of flag values for capabilities, so for instance I can say if a drivers supports integer or float 32-bit samples, 64-bit samples, and if it can accept streams in a different endianness. This is important because there’s little point in doing the job of the output plugin, that might handle that transparently, for instance a big-endian stream might be decoded on a little-endian machine, then sent through PulseAudio to a big-endian machine where it will be reproduced: in this setup, xine’s endian reversal of the stream (from big to little endian) would have been superfluous, as PulseAudio would have accepted the big-endian samples, then sent them to the other machine that needed not to reverse them to reproduce them.

Anyway, right now the code is quite fragile, there’s no conversion being done, there are mostly only things that are totally broken out, there are asserts 1 == 0 used to mark the code that needs to be rewritten. But something works: I was able to remove a lot of code from the musepack decoder, as libmpcdec always produces 32-bit native-endian (or maybe little-endian, I’m not yet sure) floating point samples; previously the decoder converted all the samples back to 16-bit format, and then gave it to the audio output loop to handle… now instead it sends them directly to the output, and as PulseAudio supports 32-bit float samples, they are not converted and play back fine.

Tomorrow I’ll see to work a way to handle upsamping and downsampling of streams, the problem is that it’s not trivial to decide what to do: if a plugin supports 32-bit integer samples, but not 24-bit integer samples, it should probably upscale the 24-bit to 32-bit to avoid losing precision; if it doesn’t it might upscale it to 32-bit float, or maybe downscale it to 16-bit integers. The same applies to channel mode, if the driver doesn’t support stereo output, should it be updated to 4.0 or should it be downgraded to mono?

For sure this time I’m very happy of being working on branches: leaving the code broken for weeks, maybe months, is not something you want to do on the main development branch. And I mean it, because with the changes I’m doing, not only I’ll be changing the ABI of the library itself (well, actually not much, just a couple of structures), but more importantly I’ll be changing the audio output plugins API, as I need to feed them a sample format rather than a bits-per-sample size.

Anyway, this is not going to be something easy to complete, but it will be a noticeable improvement for Amarok users once done, especially because I want to make sure that the capabilities for “mixer” volume and “PCM” volume are cleared up, probably by deprecating one of them, so that Amarok can be changed not to use xine’s software amplification (which also sucks and I also need to rewrite in good part in this branch) if the output plugin actually supports a per-stream volume (like PulseAudio).

Sponsoring, bribing, and comfort words are welcome, as xine-lib’s audio_out code is giving me creeps.

Today is day of releases

As I’ve stated on my site.

First of all, I’ve released gitarella 0.003, after fixing a load of bugs and display issues that now should make gitarella way more solid when compared with the previous versions. I wanted to do this release because I’ll be probably working on SCGI support in the next days. SCGI seems to be an interesting technique, although the Simple part is really not related to its implementation (you actually need to implement more stuff on web-app side); it’s more interesting for the ability of restarting a single webapp without taking the whole webserver down while updating stuff.

Also, I finally released the hunspell rbot plugin that I already blogged about. Grab it while it’s new ;) Please take in mind that it requires ruby-hunspell, that in turn requires the Gentoo-patched hunspell (for now, the patch is merged upstream, or at least it should be at this point), and that it clashes with rbot’s own spell.rb plugin, that should then be deleted or disabled for this to work.

I don’t count on this to be useful to anyone, but it would be good if there was someone interested in it :)

Oh, I’ve also updated typo once again, now that the development seems to be actually going somewhere :) The theme for the admin interface is entirely changed… somehow, I preferred the old one. I hope the default theme will remain available still, as I don’t want to change it with another different default, but I’m not good enough to create my own theme.

And for who’s wondering: to let typo us system’s rails, you just need to remove vendor/rails and remove the definition from vendor/’s svn:externals property. At that point, typo won’t find its own copy of rails and will use system’s one.

Released ruby-hunspell

So today I wanted to spend those few minutes to release ruby-hunspell, prepare an ebuild for it (in my overlay for now) and write the hunspell plugin for rbot, and I’ve done the release.

If you want to try it out, you can grab it from the project page. There you can also find the link to the gitarella to browse the sources and the address to get the GIT repository.

The packaging is done with a custom tarball script that’s derived from the one I used for gitarella.. probably I should clean it up and just share it between the two projects. The build system is CMake as I said, that is incidentally also the one chosen for KDE 4. It has a lot of troubles upstream I’m afraid, and the syntax could have been better (starting from avoiding that damn all-case script this time, instead of making it worse), but it’s not that bad to use after all. I have to replace the FindRuby module with my own (that I wrote already for TagLib) as the one provided by original CMake is pure and simple garbage (hardcodes i386-linux path), but now it should be just fine. The good part of this all is that hunspell and ruby are available for Windows, so ruby-hunspell should work there, too.

The ebuild was easy to write, in all the defects of CMake, there’s not the one of being unable to use it decently from an ebuild. In contrast to qmake (that is an Hell of its own) and scons (that is stupid of its own and difficult to use in an ebuild), cmake requires just a generation of the makefiles, comparable to configure script, and then classic make/make install. It respects CC/CXX/LD variables and CFLAGS and LDFLAGS as-is without doing any edit. On that, it’s fantastic.

The ebuild as I said is in my overlay, currently marked ~amd64 and ~x86-fbsd. I’ll add it to portage soon, most likely, but first I want to release the hunspell.rb script for rbot. Right now it works just fine, but hardcodes the path of en_GB dictionary and affix files in myspell-en, I want to make it configurable before release.

Oh and of course… Grandi Azzurri!, even if I don’t like soccer at all, yesterday’s game was fun to watch, once again. It was quite a while that here in Italy soccer was more played like politics rather than a sport, and they deserved to win after playing again.

Learning Ruby

Okay, lately I thought as I was being a bit “stuck” on my usual programming languages, C, C++, Python, …
Discarded Java as one alternative for me, I though I should have started learning a new language. My choice got to Ruby.. a bit because I heard a lot of good impressions about it, a bit because I know it has good Qt/KDE bindings.

So I started one of the things I planned some time ago but wasn’t going to do at all.. as Ruby is good at strings parsing, I’m trying to get a little analyzer to find out the ebuilds that uses features not compatible with Gentoo/FreeBSD.

Unfortunately, also if it started good, I’m now stuck during ebuild parsing as I don’t really know how to get a good way to search for such cases and report them.
Oh well, at least I started learning how Ruby works.

Now, the syntax is almost as messy as perl’s (that I don’t like at all), but being object oriented makes it more interesting for me. The string parsing is really lovely, but I found a couple of regexp able to crash ruby interpreter, and at one time it tried an illegal operation, too.
I think I’ll try in the next days to improve my knowledge of it to get to a point where I could make use of it in case I need it.

I also though of rewriting flamebot (that you find on #gentoo-dev and other #gentoo channels on freenode as ServoFlame, the backup bot in case jeeves and GenBot takes holiday :)), and I found rbot project as a framework for IRC bots, but.. FlameBot started as something scratchy, I wrote it on a couple of days when I wasn’t in the mood of doing anything, and it does its work (parsing bugzilla’s reports and so on), I think I’ll just add a couple more things (like being able to intercept urls for certain bugzillas – useful to display data from KDE or GNOME or FreeDesktop’s bugzillas but not for Gentoo’s as there’s already GenBot doing so – or being able to show the supported bugzillas).

Oh by the way, my personal life is more or less getting more stable, although it seems strange also to me ;)