Qingy and ConsoleKit

I promised to explain our difficulties with Qingy and so I am. First of all, I guess it should be time to introduce Qingy, since I’m pretty sure that the average user is unlikely to ever have heard of it. Qingy is a login manager that is designed to be a replacement for both the various desktop managers (GDM, XDM, Slim, …) and the usual agetty+login combination. What Qingy uses is mostly DirectFB, or X; it has some pretty interesting features, such as console sessions support, or saving the last logged in user. It obviously users PAM to do the job.

The agetty+login combination is what you use for classic login on serial ports and tty; agetty is the one who opens and sets up the tty, but it’s login (part of the shadow package) that takes care of logging users in. In the case of Qingy, there are still two components, but they are all part of the same package.

There are a few shortcomings with their current approach to PAM to be honest, such as the fact that they still take care of looking at users’ mail and lastlog without relying on the designed PAM modules (which is some of the reason why PAM make sense: you don’t have to reimplement the same code to do the same task on a number of packages; you just have to call into the modules that are implemented already. But alas, that’s not too common a design. But one of these misdesigns in particular causes ConsoleKit not to be able to provide the user logging in with Qingy to get a proper ConsoleKit session, and not just not getting a local one, but none at all.

At any rate, let me recap how the PAM ConsoleKit Connector (pam_ck_connector module) works. The design in by itself is actually quite simple: within a session chain, during the opening sequence, the connector opens a connection (sorry) to the ConsoleKit daemon, via D-Bus. As long as this connection is open, then there is an active session. Further work is done to register the session as well as setting up the environment variables to allow the session to be recognized, but that’s about it. The connection is obviously closed when the session is closed, but the session gets de-registered as soon as the connection is closed.

Now it’s time to look at how shadow’s login works.

simplified activity diagram for login(1)

I’m simplifying a lot here since there are a number of preparative tests that need to be taken before going on with this part; but on the other hand we’re now only focusing on the sequence, rather than the actual code.

First of all as you can see there is a fork in there, which means that the login process spawns a child; this is needed because both opening and closing the PAM session is a task that should be done with super-user privileges, for creating and removing directories for instance, or updating the lastlogs. Since the user shell has to be executed with reduced privileges instead, you obviously need two processes: one to keep super-user privileges and one to drop them to the user.

In the case of login as yo can see, the session is opened before the call to fork(), so the ConsoleKit connection is duplicated in the child; this doesn’t leak further because the descriptor is marked with O_CLOEXEC; a flag that states that the descriptor is not to be passed over when calling functions like execve() (which is exactly what the child process does, after dropping privileges to the needed user. The original ConsoleKit connection is kept open on the parent process, and is closed only when the process is killed, or properly completes its life of waiting for the user shell to terminate.

simplified activity diagram for qingy

The activity diagram for Qingy is quite similar, actually; the only difference is that the opening and closing of the session is split between the two processes (parent and child); unfortunately this has one serious implication: since the session is opened in the child process, once execve() is called, the connection is closed, and there is not a copy of it that is kept open. This means that a session is opened, but it’s immediately dropped by executing the user shell itself. The only solution to this is to implement the same sequence that login uses, and execute both the opening and closing of the session in the parent process.

Interestingly, I’m seriously wondering if it is allowed at all by PAM to execute the open and close actions on two different processes, even though the handle structure is copied over. It gets more complicated by the fact that any change added to the handle on session opening is not copied over to the parent’s process handle that is used to close the session. And I’m ready to bet ConsoleKit is far from the only module that has trouble with the way Qingy currently works.

And maybe you can see why we’re taking this long with Samuli to tackle all the issues. We’ve been trying to push as fast as possible those changes that can easily be changed, but now we’re in a situation where we need a much deeper understanding of the software we’re dealing with, and it’s a definite huge amount of software involved: just in the case of Qingy, to actually gather a clue of what’s going on, I had to look at Qingy, ConsoleKit, shadow, Linux-PAM and D-Bus, and this is with me knowing at least how fork() works. Can you really blame us for taking this much time to get it right, now that we know that what was done two years ago either wasn’t enough or it obsoleted over time?

For those who wonders, the two above UML diagrams have been drawn with Visual Paradigm for UML software — it’s not Free, but since I needed something quite complete for some work projects last year, I had to get something working, Free or not. On the other hand I think it can still be of help to me and to the Free Software movement as it allows me to clarify points, like in this post here.

It’s getting boring I know; more ConsoleKit braindumps

Since people insist that I don’t make myself enough verbose on the matter of how pambase gets implemented, here comes another post detailing some of the trouble me and Samuli are going through to make Gentoo a viable modern desktop.

First of all let’s make it clear what the target is: ConsoleKit is the modern implementation of the common necessity to provide different authorisation to users depending on whether they are connected on the local system or connecting from a remote system. The reason why this is useful on desktop systems is that you obviously don’t want somebody from a remote system to access your USB devices or cdroms, or soundcard while not being root.

Interestingly, the idea of this authorisation has always been tied tightly with PAM; the first try to get this working was the RedHat-generated patch that added the infamous pam_console module to Linux-PAM; one of my first problems to solve with Linux-PAM, and one of the first patches to be killed was indeed that which added this module. Unfortunately, the method used by pam_console was a very bad one: it changed the owners of device nodes in /dev, but if a session was abruptly interrupted, then it wouldn’t set them back correctly. Besides, it didn’t work that well when multiple users were logged in on different TTYs.

ConsoleKit handles this in a much different way, even though it still uses (mostly) PAM to do the work. Basically, when you open a new login session (and just login, not authentication for services), the PAM connector connects (sorry) via D-Bus to the daemon to tell it that there is now an open session; as long as that connection is open, the session is listed, and all the processes started from there are authorised. What’s the catch here? Well, to know which session a process belongs to, you have two obvious ways; neither of which is very reliable: you can check the parent process ID (which is not going to be fun when the process detaches into background), or you can check if it has the same open tty (which is not going to work when the program makes not use of ttys at all)., or you can check for the presence of an environment variable, but you will guess it’s a very bland way to solve the problem this latter one.

Luckily, Linux already provides a decent way to identify a session: the /proc/self/sessionid virtual file — but that one needs the kernel to support the Audit framework features. This is the reason why ConsoleKit is now asking you to enable that stuff in the kernel, it’s not just a mood swing, it’s rather a requirement to have proper session isolation. Unfortunately for this to work you also need that each session is created (and destroyed) with login — and we have PAM for that; indeed the module responsible for taking care of that is pam_loginuid that has been added in the latest (non-M4) version of pambase.

The bad side is, though, that it doesn’t help if pambase is fixed, if any other maintainer of a package using PAM is not listening to me (or insist that I shouldn’t dictate the PAM policies even though I’m the only one who seem to have a clue on how things have been asked to be supported for quite a while). This means that we have to fix all the graphical login managers (gdm, kdm, wdm, xdm, entrance, slim, qingy, …) to include the correct service file (system-graphical-login), so that the proper sessions are set up. Not just that, but as we were able to debug with Samuli, we need to fix both baselayout and openrc to use the system-services stack (so finally abiding to pam_mktemp as it was supposed to do since a very long time ago), since otherwise a service started not by the rc system but rather from a running logged-in session would inherit the root’s login session id, which is obviously a bad thing.

This was designed to work correctly with system-services, but as it turns out, unless I actually pay attention to each and every package that should use them, even when I actually coordinate integration of the new services into pambase with them, or I do the original changes myself, the end result is simply bad. This is why an audit is necessary as I said before.

I was intending to end this post with a description of the incompatibilities between Qingy and ConsoleKit, but after writing a few paragraph, I decided that without enough diagrams of what’s going on, there is no way I can actually provide you with a description of it that can be understood by those who haven’t studied PAM and login themselves. So for now I’ll close on this page, and will post during the week the notes on qingy.

But I’ll also have to warn you that for the next weeks I’m quite swamped with finishing the documentation of one job and starting the work on a new one, so the normal Gentoo stuff is going to stay hidden for a while; if you wish to help with testing or motivation it’s certainly appreciated. I’ll see to provide a testing pambase ebuild on the repository itself so that those of you who feel daring enough can get to test it. As a homework for anyone who’s interested, feel free to open bugs about desktop managers in tree and official overlays – with pam-bugs@g.o in CC – if they should be migrated to system-graphical-login once the new pambase is released.

Hey host, give me a new login please!

One of the reasons why I’ve been spending all this time rewriting and extending pambase lately has been that Samuli asked for my help to sort out the ConsoleKit problems he’s been seeing for the past months. Authorisation problems with both XFCE and (less often) GNOME are mostly due to a misconfigured or misbehaving ConsoleKit, and the main cause of that seems to be PAM-related.

Indeed, after a bit of poking around I came to the idea that PAM needs a whole audit and that the new pambase was really important to work on. As it happens, most of what I wrote yesterday tied with the need for the graphical manager and the text login to differentiate their session chains — there are bonus points because we can solve a few more problems with noisy modules at the same time, but that’s beside the point now.

So there has been two main issues here:

the nox11 option given to pam_ck_connector need to be dropped for X logins;
all the desktop managers need to move to use the system-login (now system-graphical-login) service, rather than using system-auth directly.

As messed up as this might be, now that I reached the end of the alley, I guess the right questions I had to ask were:

why none of the managers used the system-local-login stack as it was intended with pambase? the answer is actually what I wrote in the previous post: the noisy modules (motd/mail/lastlog) caused hiccups and noise on login with GDM and the others; which is now solved by using a different service for them;
why was nox11 added to the system-login service? Because when I was suggested to set it up, whoever gave me the line (I don’t remember who was, I didn’t know enough about ConsoleKit to come up with it myself — and I still don’t know much about it) only considered it to be used with GDM; since GDM takes care of creating the session itself, there is no need for the PAM module to do so.

Unfortunately, as it happens, GNOME is no longer the only environment to use ConsoleKit, and thus GDM not the only desktop manager used by users that should be authorized by it. The other desktop managers, though, are not going to integrate ConsoleKit themselves (heck, PAM was designed with this need in mind) so they should be using the module without the nox11 option. So what happens if we disable it? Well, GDM will create its own session after the one from the PAM module is created; the session coming from PAM will result unused, but won’t hinder much. It’s an acceptable compromise as well. A more solid situation you’d have if you simply disable ConsoleKit by USE flags on gdm, so that only the PAM version is applied — at the time of writing this, the ebuild does not allow you to do this though. For whatever reason they decided it was a good idea to use a “same” dependency between the ConsoleKit support in GDM and pambase even though it would rather be a “either or” dependency: either in GDM or in PAM.

So this should solve the problems (once applied of course) for GDM, KDM and XDM. Unfortunately Slim, which is the manager Samuli suggested me to begin testing with, still doesn’t work. Oh, ConsoleKit creates the session, all right, but it is not marked as local at all. Why is that?

Well, among the attributes PAM applications can provide to the modules to test against is one called PAM_RHOST that stands for remote-host; XDM, GDM and KDM only set this if the DISPLAY variable does not start with a colon, which mean that we’re displaying on a remote server; ConsoleKit decides that the session isn’t local as soon as it finds a rhost value; on the other hand Slim sets the rhost to localhost in all cases (and so does the sample pam_env configuration file that we don’t actually execute anyway). This basically means that even once the PAM issues are all sorted out, Slim-initiated sessions will result non-local.

Now, where was I? Ah yes, I have to just make sure that all the ebuilds in portage call the right service. Piece of cake, right?

Anybody hiring me for PAM?

This post might sound like a nasty plug, but I’m really doing this because it seems like the only solution up to this point.

In the past few days some trouble came up on the PAM side again. Let me try to put this into prospective: if nowadays I can actually find some use in knowing how PAM works, I joined the PAM team four years ago while working on Gentoo/FreeBSD because I needed configuration files migrated to a format that worked there as well as Linux. Since then, Azarah went missing and the whole of PAM was shoved on my back. Nowadays, I maintain the Linux-PAM package, a bunch of random PAM modules, and should oversee over the general PAM configuration in Gentoo.

Unfortunately, this requires also a lot of coordinating skils, and time to do the coordination: maintainers of other PAM modules, and maintainers of packages that use PAM themselves, should talk with me about the default configurations and the like; instead I’m usually reactive on that matter. And that is, as you might guess, not the best of the experience, nor the easiest of the tasks.

I have written before about the need of a new pambase and this is now obvious to actually implement proper support for multiple authentication methods like Kerberos, LDAP, PKCS#11, YubiKey, … I have a few ideas on how to solve this, namely changing the current situation with a few predefined, hidden chains (.gentoo-session-minimal, .gentoo-session-console .gentoo-session-graphical) wrapped on the system-* series of chains, all generated with M4 rather than the current C preprocessor (that lacks any kind of arithmetic capabilities).

But even more than fixing pambase, there is the need to review the packages that use PAM. A few days ago, Samuli complained to me that ConsoleKit was not being executed properly on login(1) — turned out the problem was that /etc/pam.d/login was not calling back into system-local-login as it was expected to. Root cause was that the modified PAM chain file was replaced with the previous ones (which didn’t use that chain) after the major bump of sys-apps/shadow when it was picked up by Debian. Dated 24 Feb 2008. Over two and a half years ago.

While I cannot get rid of the fault of missing the revert; why did I miss it? Simple enough: Portage’s confmem feature never told me that /etc/pam.d/login was changed from the one I had before. It assumed that my local version was a modified one and thus accepted that one as the good one.

Now this makes the second revision bump and second stable request that I have to take care of to fix PAM-connected trouble; the previous one, back in July (for the bump) and last month (for stable) related to the chpasswd chain that had been broken for, well, almost the same time as this one.

In fixing another ConsoleKit problem, bug #342345 I found that the GDM and KDM chains are not compatible with pambase, and they both need more fine-grained control over the sessions (console and graphical sessions have different needs, in the latter case we have to skip motd/mail/lastlog modules).

A quick check around on the tinderbox told me that there are a number of PAM chain files that should be cleaned up, reduced, optimised and so on. And that the number of files there does not correspond with the number of files installed.

Basically, what we need now is an audit of all the PAM-using packages in the tree, beside the improvement of pambase as stated above. We’re talking of a month or two worth of work. Not something I’d do myself in spare time, not something I can do as it is during work time. It’s not simply a matter of writing what I did for the original pambase; it’s a work more in line with the Ruby NG situation, and that one we haven’t completed just yet with three people working on it, including a bunch of work time thrown in by me for a few jobs I took during the year.

So here’s the catch: nobody has helped me with PAM in years, and while Constanze is being ascended to developer status, I know she’s also pretty busy with her thesis so I cannot easily ask her to commit enough time to lift me from enough work. This means that we can either keep the current status-quo of just band-aiding through enough troubles so that we can keep it running, or somebody got to help me either with work or with funding. As I said, I’m already losing money with the tinderbox and I don’t want to lose time, sleep and (possible) money on working on something as ungrateful as PAM.

Don’t get me wrong, I’m not asking for donations here; I’m asking to be paid to do a job, and that job is the auditing and review of (a part of) the PAM-using ebuilds. If you’re using Gentoo (and PAM) in production, you might be interested in hiring me to get this out of the way. I don’t even have much to pretend: €1500/month, one to three months time (depending on the deepness of the work you’d want to fund), you can provide the agreement details and give me a list of priority programs to work on (those that you use in your organisation). On the same terms, I’m willing to help you package new software that is not currently in portage in the spare time.

Let me know by mail if you’re interested. Extra points if you use GPG-encrypted email because that stuff doesn’t get sent to spam.

All your pambase are belong to us

And with this I finally given up on this omnipresent meme. But I couldn’t find a better title to present to the world my new PAM infrastructure package: sys-auth/pambase.

As I suggested yesterday I wanted to improve the way that the basic PAM configuration is handled in Gentoo to make it simpler to add caps support. Last night I decided to start working on it, and it was a quite easy target actually.

The new sys-auth/pambase package installs the basic PAM configuration files for all implementations. This means that the same basic file configuration used by Gentoo Linux is shared with Gentoo FreeBSD, and changing one will also change the other. No more duplication of the files in sys-freebsd/freebsd-pam-modules and sys-libs/pam.

Of course different implementations (Linux-PAM and OpenPAM, FreeBSD modules and NetBSD modules) implement different modules and different parameters. So instead of having the final file in the pambase package, it contains some pre-parsing files, that cpp (yes, the C Pre Processor proper) expands to the final file. Through a few directives, I can then enable or disable feature on a per-implementation basis.

In addition to the system-auth PAM configuration file (that is referred to by almost all other services to provide system-integrated configuration), pambase installs a system-login PAM configuration file. What is the difference? This configuration file should be used for services like login (from shadow), xdm, gdm, kdm, entrance, … I suppose that any of you who ever tried to get stuff like ConsoleKit working would be able to grasp what is so interesting in this.

A single consolekit USE flag on pambase could allow to switch on and off ConsoleKit authentication on all major console logins without user intervention (beside enabling the USE flag and running etc-update).

For now this is a testing framework, as I’m considering if I should have system-login plus system-console-login, so that sshd for instance can use system-login (to enable stuff like motd, userfiles and similar), but still not use ConsoleKit and similar, and it still has to be tested on Gentoo FreeBSD.

I hope to coordinate with GNOME and KDE teams later today to have gdm and kdm to use the new capability, still without any keyword, and be able to put this in production in a few months, also replacing virtual/pam step by step (as it does take care of properly depending on the right PAM implementation with its eventual modules).

One nice thing is that it will probably have a huge lot of USE flags, as it would then allow to just specify a flag (say, caps) to enable the correspondent PAM module into the main system configuration, depending on the right ebuild. While most complex configurations wouldn’t work well that way, basic stuff is likely going to make it easier to manage system login for users.

Anyway, as you’ll guess this will be my project in the next few weeks (I’ll also return on linking problems, but it’s better for my sanity if I change topic from time to time for a while).

Some more PAM plans

So, I’m still working on the PAM documentation; in particular, I’m working on splitting up what initially was a single guide, because I found it was coming too complex and too long, and it was deemed to scare the devs away. Plus, there are still some concepts that are good for users as well as developers, so it would make more sense to move it in their own “shared” guides.

I’ll probably commit some of the documentation later today, and then continue working on the rest, leaving Josh to clean the grammar up :)

But, while trying to reorganise documentation and packages, I also started wondering about a few other things, for instance I’m certainly going, in the next weeks, to work on a proof of concept of a pambase package, carrying system-auth, login, and xdm configuration files. The idea of this is that if PAM changes syntax, I can just change one package to handle it.

At the same time, this would allow to provide simpler management of common optional PAM modules; for instance adding consolekit support for login, GDM, XDM and KDM would just need an USE flag to pambase, and then the default installed configuration would be updated.

Of course this transition won’t happen anytime soon, considering that PAM 0.99 is scheduled to be marked stable for September, I doubt this could go in use before next year, but it’s something that might for the future.

Comments, as usual, welcome, as well as bribes and sponsors :)