Refreshing Gentoo Work

After a few months spent mostly working on lscube, I’ve been ignoring most of the non-basic Gentoo work for a while. Between last night (before going to sleep) and this morning, though, I started the catch-up work.

First of all, Tim released a new version of libarchive that required some testsuite fixing (and I haven’t noticed the first time around that it now wrongly uses -Werror since I have -Wno-error in my CFLAGS to avoid time wastes). Thankfully, Tim is a dream upstream to work with and the most important fix is already upstreamed.

Then I have been active in the Ruby area since I both needed to work on the new Typo and a few more packages are bundled with Typo’s code (you’re going to find a git branch with no third party code bundled in my git repositories when I’m done), and got some new tasks to work on.

The gems problem, which is hopefully going to be solved after the Summer of Code, is for now just being sidestepped; indeed, I’ve ended up adding the will_paginate library with a fake gemspec which actually works pretty nicely, without having the usual side effects of gems (no object files installed, no extra dependency on rake, no installed testsuite) and with the obvious advantages from the tarballs, including working testsuites (and tested), documentation built on request and installed, as well as examples. This, and probably a few more before end of the month, package will be tested directly here on the blog if you’re interested on the outcome.

I still have a few things that I’m supposed to have done in the past month among which figures updating calibre (I’ve been using an old version on OSX up to now), figuring out why libcdio-0.81 freezes down during install, and stuff like that. Hopefully I’ll also be able to find time for those now that my job is a bit more safe than it was before.

User Services

A little context for those reading me; I’m writing this post a Friday night when I was planning to meet with friends (why I didn’t is a long story and not important now). After I accepted that I wouldn’t have a friendly night I decided to finish a job-related task, but unfortunately I’ve had some issues with my system. Somehow the latest radeon driver is unstable (well it’s an experimental driver after all), and it messes up compiz; in turn after a while either X crashes or I’m forced to restart it. This wouldn’t be a problem if the emacs daemon worked as expected. Since it doesn’t, I lose my workspace, with the open files, and everything related to that. It’s obnoxious. Since this happened four times already today I decided to take the night off, but I wasn’t in the mood for playing, so I settled for watching Pirates of the Carribean 2 in Blu-Ray, and write out some notes regarding the topics I wanted to write about for quite a while.

The choice of topic was related to the actual context I’ve just written above. As I said GNU Emacs is acting badly when it comes to the daemon. While the idea of the daemon would be to share buffers (open files) between ttys, network and graphical session, and to actually allow restarting those sessions without losing your settings, your data, and your open files, it’s pretty badly implemented.

A few months ago I reported that as soon as X was killed by anything (or even closed properly), the whole emacs daemon went down. After some debugging it turned out to be a problem with the handling of message logging. When the clients closed they sent a message to be logged by the emacs daemon, but since it had no way to actually write it to a TTY session, it died. That problem have been solved.

Now the problem appear to be just the same mirrored around: after X dies, the emacs daemon process is still running, but as soon as I open a new client, it dies. I guess it’s still trying to logging. As of today the problem still happens with the CVS version.

So anyway, this reminded me of a problem I already wanted to discuss with a blog: user-tied services. Classically, you had user-level software that i s started by an user and services that are started by the init system when the system starts up. With time, software became less straightforward. We have hotplugged services, that start up when you connect hardware like, for instance, a bluetooth dongle, and we have session software that is started when you login and is stopped once you exit.

Now, most of the session-related software is started when you log into X, and is stopped when you exit, sometimes, though, you want processes to persist between sessions. This is the case of emacs, but also my use case for PulseAudio since I want for it to keep going from before I login to before I shut down the system straight. There are more cases of similar issues but let’s start with this for now.

So how do we handle these issues? Well for PulseAudio we have an init script for the systemwide daemon. It works, but it’s not the suggested method to handle PulseAudio (on the other hand is probably the only way to have a multi-user setup with more than one user able to play sound, but that’s for another day too). For emacs, we have a a multiplexed init script that provides one service per user; a similar method is available for other service. Indeed, in my list of things to work on regarding PulseAudio there is to add a similar multiplexed init script to run per-user sessions of PulseAudio without using the system wide instance (should solve a bit of problems).

So the issue should be solved with he multiplexed per-user init script, no? Unfortunately, no. To be able to add the init scripts to the runlevels to be started, you need to have the root privileges. To start, stop and restart the services, you also need root privileges. While you can use sudo to allow users to run the start/stop commands to the init script, this is far from being the proper solution.

What I’d like to have one day is a way to have user-linked services, in three type of runlevels: always running (start when the machine starts up, stop when the system shuts down), after the first login (the services are started at the first login and never stop till shutdown), and while logged in (the services start at the first login, and stop when the last session logs out).

At that point it would be then possible to provide init scripts capable of per-user multiplexing for stuff like mpd too, so that users could actually have the flexibility f choosing how to run any software in the tree.

Unfortunately I don’t have any idea on how to implement this right now, but I guess I could just throw this in for the Summer of Code ideas.

Shadow casting

For those of you who’d like to follow Google Summer of Code progress (I’d very much like to have the blogs of the student available on Universe by the way), today (or rather yesterday) can be counted in as a very important date :)

Seraphim prepared a patch for shadow (the package containing all the basic utilities for users management in Linux) to work with OpenPAM, rather than depending strictly on Linux-PAM. I committed it in tree now for 4.1.2.1 and upstream applied it to trunk already, so now it can be built even when using OpenPAM rather than Linux-PAM. Cool, eh?

This patch (and the related bug) are important not only because they fix a very important part of the functionality of a Linux system with OpenPAM, but also because it will act as a base reference to fix other software to use OpenPAM too.

Indeed, one of the most obnoxious problems with OpenPAM is that a lot of packages instead of writing their own conversation functions rely on misc_conv. Seraphim prepared a patch that can be applied to almost any other package relying on that to make it OpenPAM-compatible. It’s very good.

Unfortunately, as Seraphim also blogged there is one catch with being able to provide OpenPAM support, especially for the future. The problem is that although mostly API compatible, OpenPAM and Linux-PAM are not ABI compatible. Although in a very subtle way, because, as Seraphim learnt, you can have a system built against Linux-PAM run against OpenPAM just fine, up to a point.

The problem is that ABI does not only refer to the name of the functions, or the type of their parameters, but also to the meaning of flag values. In this case, Linux-PAM and OpenPAM give different flags different meanings, so modules built against OpenPAM will not work properly with software built against Linux-PAM.

This is going to be tricky, especially once we’ll allow users to switch from one to the other and vice-versa, because it means all the software will have to be rebuilt to continue functioning as it’s supposed to. And no preserved-rebuild will help us there.

Oh well, there’s time to think of that!

Google Summer of Code deadline extension

As you can see on Multimedia Mike’s blog the deadline for Google Summer of Code has been extended to April 7th.

I’ve updated the Gentoo Calendar at Google for the new deadline, too.

I sincerely count as one failure that FreeBSD had an official news item and Gentoo didn’t. We really should learn from these things so I’m pointing it out in public.

I wish xine could have gone into SoC too, but I think there are a few things that has to be cleared out before that could ever happen. I’ll see if I can get myself to that in the next year or so.

On other notes, SoC this year seems to have some interesting projects, and I didn’t see any Gentoo dev submitting applications to Gentoo up to now, which makes me quite happy :) I hope the ones needing the money applied to other projects though.

Now I have to go back to my cabling, but expect me to return to full-fledged status in the next week or so. I’ve been bumping PulseAudio yesterday, and Lennart helped me to find the reason why the M-Audio Audiophile 2496 soundcard is rejected by Pulse, I’ll try to document a bit how to use that with PulseAudio and other audio software as it seems to me like it’s not an easy task, and ALSA is not by default good with it.

By the way, anybody tried Timeshift on PlayStation 3? I tried the demo and the graphic looks seriously cool, but the joypad is not exactly what I’d consider the best input for a FPS. I’ve heard that Unreal Tournament 3 can use the keyboard as a controller, so that makes it more likely for me to buy, but I would have liked to know about other games so that I can choose the most interesting one :)

How’s this year Summer of Code coming?

I hope Joshua won’t get mad at me, but I have to write about this, maybe it will act as a good way to get the mistake noticed.

I’m afraid this year’s SoC is going to follow the path that the previous two instances took already. What makes me afraid of this is that there is little to no coordination between parts.

First off, the announcement for SoC was pretty late, GMN didn’t talk about that at all, which is already a negative bonus. Considering the short timeframe that applicants have to submit their ideas, it isn’t a very nice idea at all. For what it’s worth, it wasn’t even listed in the LWN announcements.

The official SoC ideas page got some new additions, but they came pretty late, not soon enough to give time to the students to start thinking of what to do, and maybe discussing it with the contacts.

There is also a shortage of mentors. I’m afraid this had to be foreseen, there is little to no incentive for mentors to actually do their work, there is little project spirit around lately, and I do understand it. Finding a way to actually get more mentors next year is not going to be an easy thing to solve, so I think we should start looking into that already.

And even with the very few mentors that are around, I can’t see much coordination. I’m not on IRC at the moment as I’m from the laptop, but I got Jabber and my mail client open, neither gave me any information about being accepted as a mentor or about the URL of the mentor’s dashboard to see the applications!

I don’t see any soc@gentoo.org alias or anything like that, and that is also a bad thing: I got a few users contacting me for some ideas, because I actually blog (and care) about Summer of Code. I had to refer them to other developers because I can’t handle them, not my area, or just not something I’d feel comfortable to mentor. Having a single alias that users could write to would allow all the developers interested in SoC to answer as they see fit. Yeah sure there is the mailing list, but you can guess that most people wouldn’t like to make their application’s details public, after all, they are not public even after SoC closes.

The deadlines, short as they are, were not posted on the recently created Gentoo Calendar (at Google of course); while just recently born, it would be a nice addition for this kind of stuff.

Up to now I listed the problems that should have been avoided by the SoC team itself (note to self: try to cut away more time next year so you can be part of the team and make the changes), but the biggest problem of all I wanted to leave last.

I think that both me and Donnie tried to make this point before, but Gentoo developers should really try to blog more. In today’s status of Free Software, blogs are often used to share and bounce around ideas, and to make projects and subprojects more advertised. Try to compare Planet Gnome with Planet Gentoo, and let me know.

In particular, there is just no material on Summer of Code in Planet Gentoo! Just me, Luca and Joshua blogged about it, as far as I can see. I’ve been trying Google Reader in the past weeks (which turned out to be quite good now that I don’t have my Akregator at hand), and I’ve started tagging all the posts I seen (not even read fully!) who wrote about Summer of Code. The result is right now 45 items, and please be known that I started on March 19th, with the exception of one post I was interested in and decided to look up afterward. The vast majority of the posts come from Planet Gnome, which I named before, but there are many posts from Planet KDE too.

I’m sure there are way more posts about Summer of Code around, I just probably don’t follow a lot of blogs of other projects involved, but the fact that Gentoo is not so much on that list is not something I like.

This entry will add to the list, though I’m not happy with this. I really really really hope next year we can avoid these mistakes.. at least I can say I tried though.

Summer of Code, Gentoo and other projects

So it seems we got accepted as organisation for Google Summer of Code 2008! And so were FFmpeg and FreeBSD (at least, I heard about those on a few blog).

I wish to remember the users who’re interested in Gentoo/FreeBSD that the main way to improve Gentoo/FreeBSD (once I find time to take back the project again, and make FreeBSD 7.0 ebuilds) is to improve FreeBSD itself. So if you want, you can apply for their SoC and still help the Gentoo project!

And FFmpeg is also an important project for me to have new people working on it: xine is based on that, so it’s a very important project. If you feel you can actually work on that area, join that SoC!

But of course make sure to check out Gentoo ideas, and feel free to contact me if you want further information on the project I proposed myself.

And if you still don’t know where to apply, check out the ideas for the rest of Summer of Code 2008!

Summer of Code ideas for other projects

I know I already filled the Gentoo SoC project page with ideas, but I still got a few more to propose for organisation which I’m not even sure will be on SoC itself. Think of this post just as a braindump of stuff I’d like from other projects and which I would see well suited for Summer of Code.

  • for lighttpd, a PAM-based authentication module, so that, for instance, I could allow all the xine developers to access the server where xine bugzilla and also access a private HTTP directory on it with a single user and password database (the system);
  • for libarchive (FreeBSD), built-in support for lzma (de-)compression algorithm, so that it could handle GNU’s .tar.lzma files on its own;
  • for glib, a confuse-like configuration file parser, so that I could get rid of that dependency on unieject;

Learning ADA and extending ELF analysis?

It seems like my motivation, since I left the hospital, is always falling down. Unless I’m doing something new and interesting, I’m unable to keep myself focused.

This is my reason to start the work on ruby-elf and the whole analysis thing. Unfortunately, doing the analysis that way does not seem to be the easiest way at all.

Add to that the missed challenge with C#. When I was first told I had to develop in .NET with C#, beside a first understandable visceral reaction to that, I was excited to the idea of learning a new language. It has been quite some time since I learn my last useful language. While I tried to learn LISP (ELISP to be exact), that is quite a bit jump for me, as I’m way too used to non-functional programming languages.

I’ve been wanting to look at ADA for quite a while, and after the last In Our Time podcast (I’m podcast-addicted lately), I decided it was the right time to at least start looking at the thing. It does sound quite interesting after reading a bit about it, so I’ll be trying to read about it in the next week in my spare time. It might come handy the next time I get a job to work on embedded stuff.

I admit I’m not sure how ADA support for SQL databases is, but if there is any kind of support, I’m tempted to rewrite part of my elf analysis code in ADA (and even if there is any, maybe I can do that to cowstat at least). The intrinsic support to multi-threading is what I’m more fascinated from, especially for things like cowstats that could easily analyse multiple file at once, rather than doing it sequentially.

I am really afraid of what the pancreatitis did to me on a spiritual/mental level, more than physical, lately. Not like the physical damage is nothing, it’s actually quite a lot; luckily I didn’t smoke or drink before, as now I can’t do it for sure (well, it wouldn’t have been good to do even if I didn’t have the pancreatitis, but who had similar experience knows what I mean ;) ). But the spiritual damage seems to be more than just fear to me. I really am thinking a lot of how much time I’m left, and how much I wasted my time before. I really wanted one day to find the right person, have a family, children, … and while the idea itself was already quite faint before (I’m too geeky to find a girl who can tolerate me), now it seems to be impossible altogether.

But nevermind this depressing thoughts, I sincerely think Summer of Code will give me at least something new to work on, with the students to mentor.. or at least I hope so ;) So please start working already on your applications!

Private temporary directories

One more addition to pambase has been the mktemp USE flag. With that flag enabled, the default system-auth stack features pam_mktemp module, again from OpenWall.

All this module does is setting up a per-user private temporary directory inside /tmp, and set TMP and TMPDIR environment variables so that the programs (at least those respecting them) use the new private temporary directory rather than the globally-accessible one.

This is useful as a mitigation strategy for temporary file attacks (race issues and symlink attacks); while it does not magically fix them, and their seriousness is not to be ignored, it might make it more difficult to exploit them. You can think of it to something alike to PaX, hardened and SElinux.

There are, though, a few thing that need to be improved for it to be totally useful; and until those are completed, and the setup widely tested, it will not be enabled by default.

To begin with, not all applications make use of TMPDIR, I already blogged about sqlite ignoring TMPDIR entirely, but there are other cases like that, for instance, Emacs’s server support hardcodes /tmp as directory (I’ve mailed emacs-devel and they are now discussing about using TMPDIR as every other application), as well as gpg-agent and ssh-agent. I wonder how many applications will have broken designs in that regard; I know ESounD requires /tmp as path for its socket, so PulseAudio will not respect TMPDIR by default; Xorg also seems to use /tmp for locks (although I would have expected them to be in /var/lock), and PostgreSQL uses /tmp for its sockets (why not /var/run?).

Another issue is that pam_mktemp seems to handle in specific ways ext2/ext3, with its “appendable” flag, but the same does not hold true for other filesystems (I admit I don’t know how that works, and if other filesystems support them; I’m quite sure BSDs support that though – too bad they didn’t port pam_mktemp as well as pam_passwdqc). Some porting of this would probably be useful, I suppose OpenWall has a default setup where it does not make sense to check for more than ext3. SoC, anyone? ;)

Then there is the issue to make sure that setuid and setgid programs all work fine, it probably will require quite a bit of work though. I suppose this could get well along with file-based capabilities support, as that would limit a lot the amount of setgid and setuid programs, they’d just need capabilities to access the minimum privileges they need to.

But that’s a story for a different day…

A different way to achieve this is to use pam_namespace (provided by Linux-PAM), which allows to create per-user instances of any directory on the filesystem. This would then create a different /tmp directory without fiddling with TMPDIR. While this is quite interesting as it allows to workaround the broken applications hardcoding /tmp into their source, it would probably break Xorg, PostgreSQL and ESounD as they’ll get a different per-user directory too.

On the other hand, if we’re able to get all the software fixed in its design, so that global data (locks and sockets) goes to /var, then it might work fine in the future. It would also allow for a more graceful handling of FreeDesktop-defined cache directories (XDG_DATA_DIRS) as you could set the environment variable for all the users to, say, /var/cache/xdg and then instantiate a different version of that per-user.

Right now, though, I prefer the pam_mktemp solution, even if it will require more fixes around so that /tmp is not used.

Reducing the drift between Linux and FreeBSD (in Gentoo)

One of my projects is now unleashed in the tree: pambase is now being keyworded by all arch teams so that all ~arch users can make use of it.

I’m also preparing some documentation about it, but it might take a bit more time before it is available, unfortunately the time is what it is, and tomorrow I have to go to the hospital again.

One thing that pambase is supposed to do is to make it easier for me to handle the different PAM configuration files for Linux and FreeBSD, as I can just make stuff conditional in a single configuration file rather than in multiple ones sparse on the tree.

Today I hit the first two issues where this is really important.

The first was with pam_nologin: the Linux-PAM implementation supports running it in the account chain (as needed by OpenSSH’s sshd), the OpenPAM one supports it only on auth chain. So it has to be conditional. To have the conditional inside net-misc/openssh it would be an overhead.

The second was to finally add pam_passwdqc to the default system auth stack. As it is now, by enabling the passwdqc USE flag, you add to the default passwd chain the pam_passwdqc module developed by OpenWall, which is a different analysis module from the one that you most likely got by default (pam_cracklib). It is used by default in FreeBSD, but it wasn’t present in Gentoo/FreeBSD up to now. With pambase, it will soon be present by default in Gentoo/FreeBSD, reducing a bit the difference between FreeBSD and Gentoo/FreeBSD.

You should get used to a lot of USE flags in pambase as that is what I thought it for, you can tweak most of the default configuration simply by switching around the USE flags. More will follow in the future.

Talking about PAM, I also added one more possible project to the SoC page (as well as adding a column for the contact). The new project is:

Gentoo supports already two PAM implementations, but they are tied to one operating system only: Linux-PAM (sys-libs/pam) on Linux and OpenPAM (sy-auth/openpam, plus sys-freebsd/freebsd-pam-modules) on FreeBSD. Being able to choose between either implementation on either operating system is one of the long term goals of the PAM team. The first step would be to produce a set of single PAM modules compatible with both Linux and OpenPAM that could replace Linux-PAM and its default set of PAM modules.

As it says, I’d like to be able to provide users with a choice, on whether to go with Linux-PAM or OpenPAM. OpenPAM is more lightweight. And by splitting the modules up in a number of packages, it should be more flexible for different configurations.

Other than adding modules, it would be nice to implement libpam_misc in a OpenPAM-compatible way, so that all the software that use libpam_misc could finally be used without Linux-PAM. An alternative would be to port the software that uses it not to use libpam_misc.

So if you’re interested in learning how Linux (and other Unix) authentication support works, and you’d like to be part of something entirely new, feel free to contact me to flesh out details for your application with SoC!