PAM, logging in and changing passwords

I’ve been spending the past ten days/two weeks handling two full-time job at once; one was Windows-related so it won’t have any direct effect in what I’d be posting on the blog, the other involved Amazon EC2, so you’ll be seeing more ~~rants~~ sorry I meant posts on the topic soon. But first, …

Thanks to Constanze who became a full-fledged developer (congratulations!), I’ve been able to breath a bit more widely for what concerns PAM; another positive note comes from Eray becoming developer as well, which means I can get someone looking at pam_krb5 package. Which means I can get back to work on the M4-powered pambase package so that hopefully before end of the year we’re going to get it in testing at least. Additionally, user prometheanfire on #gentoo-hardened provided me with a sample configuration for LDAP that should make it much easier to implement it on pambase.

But the situation starts to become much more complicated; for instance, the ConsoleKit situation is so intricated that making it behave as intended is actually quite difficult: the invocation of the module is different whether we’re going to authenticate a text login or an X11 login session; some time ago we also found the hard way that some graphical login managers fail badly when you print too much information on the PAM output channel (such as Messages of the Day, the last login data, and mail status). This all results in having to have different sessions for local text and local graphical logins. I have a huge headache already when I start to think about XDMCP already.

This turn of events also makes me think that I should simply drop the system-login service that I’ve used in the previous iterations. The reason to use and include this service was to avoid duplication, but with M4, duplication is avoided during build time, not after install. This should make available only the three “leaf” services: system-remote-login (with optional ABL support), system-local-login (not renamed for compatibility reasons) with text-based login, and (by default) mail/motd/lastlogin modules; system-graphical-login with support for X11-based ConsoleKit sessions as well as without the extra verbose modules.

A note here: somebody asked me why of the minimal USE flag for pambase; the reason is relatively simple: even though the output of those can easily be discarded, they will be kept loaded in memory by processes such as sshd and fcron; dropping the modules from the services mean also reducing the memory usage of those process, minimally, but it does.

After the login process is sorted out there is another problem here and it has to do with changing passwords; I’ve said that before, but I’ll repeat it here. When the new pambase will be put in place, software that is able to change password will have to be updated to use a different service to do so; this will hinder the changing of password through sshd that was noted in the comments of one of my previous posts, but it is necessary if we want to have proper restriction among login methods.

The problem is that with PAM design, for what concern changing passwords, you end up with either you have to know all the currently in-use authentication methods or you have to know only one of the authentication methods and then you change all the authentication method to the new value or you change only one authentication method to the new value.

The end result is that I can’t think of any way to do what would make sense: change the token only for the systems that actually use the current password provided. Lacking this the situation is that we cannot have a single tool to do everything, so we’re going to have to stick with many different password-changing tools: passwd, chpasswd and their cousins will only require the Unix password and will only change the Unix password. You’re going to use separate tools for Kerberos, LDAP, SSH keys, PKCS#11 tokens, …

While it might sound as suboptimal it’s a compromise that actually make pambase manageable without having to resort to actual custom Linux-PAM implementations. I hope you can all agree on that.

Anyway, this only acts as a braindump; I hope I’ll be able to set up real documentation about the pambase system at one point or another, including some simple drawing to show how the authentication flow actually happens. Unfortunately if you remember, I noted that OpenOffice is the only decent software I can find to write flowcharts; unfortunately that is both cumbersome to add to a GIT repository, cumbersome to auto-produce results (when what it exports is what you wanted), and finally quite expensive in term of dependencies. I should probably try Inkscape back, possibly tied with rsvg (now that gdk-pixbuf works without X) would be a decent choice.

Gentoo PAM developments

here I am blogging once again bout PAM, which seems to be my main issue nowadays. First of all I have to say I’m still looking for somebody to hire me so that the complete audit can take place, especially since, as I’ll be expressing in a moment, the situation is worse than I had anticipated.

First of all, I wanted to finalise what I started over an year ago with the pam_pkcs11 support in pambase. To do so I needed to be able to connect Gilles’s token to a virtual machine (since I didn’t want to experiment on Yamato itself). Doing so I found not one, but two libvirt bugs.

The first was a problem with passing the device bus and number; libvirt sent them prefixed with zeroes to form 3-digit numbers; but then QEmu interpreted them as octal numbers, so 001:016 became 1.14. Easy fix by swapping two sprintf() calls. The second was nastier and I was able to complete the fix just yesterday: when the kernel has support for CGROUP, libvirt uses it as a security measure, to ensure that the virtual machines can’t allocate more memory than they are supposed to, or access devices they are not supposed to. Unfortunately, if you asked libvirt to connect an USB device to QEmu, its device pair wasn’t added to the list, so QEmu was unable to use it. The first is fixed in 0.8.5; the latter in the r1 backports in Gentoo, and sent upstream to be fixed there as well.

Beside dealing with the bugs in libvirt, I also made some changes to the new pambase branch using M4, which actually works as intended now. Thanks to the comments on the previous article the situation is improving actually. In particular thanks to MK_FG, I tried again the substack/sufficient method and it works quite fine. Using simply sufficient will create a problem if you don’t want to have a stop-stack feature at the end of the system-auth (which would create other problems to other services, as I’ve learnt the hard way before), so this should be much better.

Indeed, in the new branch there is implemented support for pam_pkcs11, pam_ssh, pam_krb5 and pam_unix all together! Also, for the password-changing service now supports running both pam_passwdqc and pam_cracklib (before, only one would on the service). It doesn’t, though, work for changing the PIN of smartcards or the Kerberos password. I’m going to implement pam_p11 support soon enough.

While working on this though, and having a number of stable requests going on to fix various things (like the shadow problems and ConsoleKit), I also found that two days ago a new Linux-PAM version was released, 1.1.3, with a few security fixes that will likely require a quick stable. But more than security there is another reason why this version is notable.

You might remember that last time I stated that only two patches were applied on version 1.1.2. Well, this time around no patches are applied over the released Linux-PAM! This makes it the first version in five years that Gentoo is shipping without custom patches at all, and thus without needing re-building autotools. It is indeed a milestone for us.

Dropping the old 1.1.0 version also meant removing four extra patches from the tree; once 1.1.3 will be stabled on all arches, I’ll be removing the remaining patches, which account for about 14KiB of the tree as it is.

After all these good news, there are bad news as well; as it happens, while I’m the only person in the PAM team, the one that is following Linux-PAM, pambase and the like (soon to be joined by Constanze luckily!), there are a number of other people who add PAM modules. Lately, two fingerprint-based PAM modules were added to the tree, and both have multiple mistakes in them. Am I happy about that? Not really.

Beside, there are still problems with symbol collisions; for some packages they are easier to fix than others…

New pambase choices

So while I’m still hoping somebody will hire me to finish do a complete audit I’ve at least started working on the new pambase code. To do so I had to make a few more choices than simply maintaining the current status-quo in running state.

First of all, I changed the backend language used to describe the rules. Up to now I abused the C preprocessor with the C macro language; this allows for arithmetic comparison of (properly formatted) version numbers but doesn’t allow for increments and decrements, and it’s not extremely flexible. The new pambase will make use instead of GNU M4 (the same language used by autoconf). M4 is designed to be a macro language by itself, which makes it very simple to implement the kind of copy-and-paste of rules that pambase needs. Not only that but it’s already part of the system set both because of autoconf and because it is one of the standard POSIX utilities.

The second decision to make is a hard one and that is to actually stop proactively support OpenPAM and the FreeBSD operating system. It’s not something I’m doing lighthearted, and I’ll make sure not to force the requirement for Linux-PAM more than it is needed, but right now there is just not enough help to support both implementations. Plus while it made totally sense to support OpenPAM when I first added support for it in Portage (with Linux-PAM series 0.78), with the most recent releases, in particular 1.1 series, Linux-PAM not only doubled up the featureset of OpenPAM, but it also provides a clean interface and very polished code. By focusing more on Linux-PAM (staying, though, as independent as possible from the operating system) it’s quite possible to handle multiple authentication schemes.

Speaking about authentication schemes, when I first implemented Kerberos support in pambase there has been a few problems to be polished with it. For once, chaining a number of authentication schemes is not easy: you cannot use the required option, obviously, because you authenticate usually only against one of the authentication schemes at once; you cannot use optional because otherwise you might login even if all the schemes fail; you cannot use sufficient because that stops the chain at the first authentication that works, and you might have further restrictions in chained services.

The only solution I could find was to move further the solution I applied to Kerberos: using Linux-PAM”s advanced result specification, if any authentication succeed, then instead of proceeding with the rest of method specifications, it jumps directly to the end of the current chain, where a pam_permit.so entry will let authentication succeed. if none of the authentication methods succeed, then there is a pam_deny call that ensures that login fails.

Another problem related to multiple authentication schemes is how to handle password changing, which is another problem that we have faced with shadow. Right now we have a lot of configuration files specifying password method chains. A lot of those have likely be added due to misunderstanding that service class as “check against the password” (which is not the case, that’s auth!). For instance, sshd by default provides a password class chain, but OpenSSH does not allow you to change your password in any way.

While cleaning up all the configuration files to ensure that they only list the services they support is something that requires the full audit that I wrote about, at least I will predispose the new pambase to handle that correctly. This means that system-auth will no longer provide password chains; instead a system-password chain will be added that will take care of that and will be used by the very few packages that allow for changing passwords (such as shadow). interestingly enough, the situation here is going to be quite different from what we have now. Many of the alternative authentication methods (PKCS#11, OTP, Yubikey) will not allow to change the authentication password, so they shouldn’t be listed there; some others have different tools to change password, such as Kerberos (kadmin) and pam_ssh, and would most likely not have to be listed there. But for those that have to be listed, including Gnome-Keyring, changing password should act on all of them, not just one, so the skip system described above cannot apply there.

Unfortunately, not only this require quite some changes on the pambase package, but it has to be coordinated with a number of other packages, such as shadow, sshd and so on. Given this, don’t expect it until mid-to-end of November. Probably later if I find some other job to follow. Once again, if somebody is interested in having better PAM support in Gentoo, it can be done faster, but not in my spare time. It’s not something a single volunteer can deal with in spare time.

Kerberos and libvirt

Do you remember my latest libvirt ranting and the recent post about Kerberos and NFSv4 don’t you? Well, let’s tie the two up and consider a couple of good and bad things related to both.

First of all, as Daniel Berrange pointed out, QEmu does support IPv6; unfortunately it doesn’t seem to work just as he supposed it to: even though my hostname resolves to both IPv4 and IPv6, QEmu by default only listens to v4. The same goes if you don’t provide a listening socket (such as “”), and again the identical same happens with the default setting provided by libvirt (0.0.0.0). You can force it to listen to v6 by either providing a v6-only hostname, a v6 IP address or the v6 catch-all [::] which makes it work on both v6 and v4, lovely, isn’t it?

Then, about libvirt-remote, as many pointed out it is possible to use it with SSH as user, but there are two catches there: the first is that with the way the arguments are passed down from virt-manager down to libvirt, to ssh and zsh on the other side, something goes funky; it works fine with bash because it splits the parameters again, but with zsh as login shell for my user it tries to call a binary called nc -U ... which as you might have guessed is not correct. The second problem is that even if you set the unix socket access for your user, it won’t let it work if you are using SSH and the system is configured with PolicyKit. I guess this was designed to work in two distinct configuration (desktop and server) and trying to mix the two creates a bit of trouble.

This does not solve two problems though: the dangling connections that are kept alive even after closing virt-manager and its inability to provide diagnostic more human-readable than the Python exceptions. This became tremendously obvious today as I went to consider the idea of using Kerberos for the authentication of libvirt itself, given that it can do that via SASL. It would make more sense, since I’ll be having a Kerberos install anyway at this point, to use the Kerberos credentials for more than a couple of services.

Using Kerberos for libvirt actually makes quite a bit of sense: you can set up properly TLS support for the connection and have an user-based authentication (rather than the whole host-based authentication that is supported with the TLS-only login). Setting up libvirt itself is not difficult, if it wasn’t for the single problem that most of the documentation tells you to use /etc/libvirt/krb5.keytab while it’ll be looking only at /etc/krb5.keytab by default — maybe it’s worth for Gentoo to change the init script so that it searches for the one documented. After that, I can properly login on the libvirt-remote access with virt-manager and Kerberos…. but I still am having trouble with QEmu and VNC this time around.

Now a little note regarding pambase: as I’ve been brought to note the default configuration used by pambase with the kerberos USE flag enabled might not be well suited for all the sites using Kerberos right now. I know that, but Gentoo never pretended to give perfect defaults, or defaults that suit everybody; on the other hand I think it’s important to give a default for Kerberos in our packaging. I’ll have to talk with Robin or someone else for integrating a default regarding pam_ldap as well, since the LDAP guide we provide is hinting at the wrong solution for the PAM configuration, if the system also want to be a desktop.

Having found a decent way to provide multiple optional login systems for users is actually finally paving the way to provide token-based login that I talked about last year.

Gentoo, a three-headed dog, and me — Kerberos coming to PAM

I’ve been fighting the past few days with finding a solution to strengthen the internal security of my network. I’m doing this for two main reasons; from one side, having working IPv6 on the network means that I have to either set up a front-end firewall on the router, or I have to add firewalls on all the devices, and that’s not really so nice to do; on the other side, I’d like to give access to either containers (such as the tinderbox) or other virtual machines to other people, developers for Gentoo or other projects.

I’m not ready to just give access to them as the network is because some of the containers and VMs have still password-login, and from there, well, there would be access to some stuff that is better kept private. Even though I might trust some of the people I’m thinking to give access to, I won’t trust anybody else’s security practice with accessing my system. And this is even more critical since I have/had NFS-writeable directories around the system, including the distfiles cache that the tinderbox works with.

Unfortunately, most of the alternatives I know for this only work with a single user ID, and that means among other things that I can’t use them with Portage. So I decided to give a try to using NFSv4 and Kerberos. I’m not sure if I’ll stick with that sincerely, since it makes the whole system a whole lot more complex and, as I’ll show in a moment, it’s also not really solving my problem at its root, so it’s of little use to me.

The first problem is that the NFSv4 client support in Gentoo seems to have been totally broken up to now, bug #293593 was causing one of the necessary services to simply kill itself rather than running properly, and it was fun to debug. There is a library (libgssglue) that is used to select one out of a series of GSS API providers (either Kerberos or others); interestingly enough, this is yet another workaround for Linux missing libmap.conf and the problem with multiple implementations of the same interface. This library provides symbols that are also provided by the MIT-KRB5 GSS API library (libkrb5_gssapi); when linking the GSS client daemon (rpc.gssd) it has to link both, explicitly causing symbol collisions, sigh. Unfortunately this failed for two reasons again: .la files for libtirpc (don’t ask) caused libtool to actually reorder the linking of libraries, getting the wrong symbols in (bad, and shows again why we should be dropping those damn files), plus there was a stupid typo in the configure.ac file for nfs-utils where instead of setting empty enable_nfsv41 variable they set enable_nfsv4, which in turn caused libgssglue from not being searched for.

The second problem is that right now, as I know way too well, we have no support for Kerberos in the PAM configuration for Gentoo, this is one of the reason why I was considering more complex PAM configurations — main problem is that most of the configurations you find in tutorials, and those that I was proposed, make use of pam_deny to allow using either pam_unix or pam_krb5 at the same time; this in turn breaks the proper login chain used by the GNOME Keyring for instance. So I actually spent some time to find a possible solution to this. Later today when I have some extra time I’ll be publishing a new pambase package with Kerberos support. Note: this, and probably a lot more features of pambase, will require Linux-PAM. This is because the OpenPAM syntax is just basic, while Linux-PAM allows much more flexibility. Somebody will have to make sure that it can work properly on FreeBSD!

There is also a request for pam_ccreds to cache the credentials when running offline, I’m curious about it but upstream does not seem to be working on it as much as it should, so I’m not sure if it’s a good solution.

Unfortunately, as I said, NFSv4 does not seem so much of a good solution; beside the still lack of IPv6 support (which would have been nice to have, but it’s not required for me), if I export the distfiles over NFSv4 (with or without Kerberos), the ebuild fetch operation remain stuck in D-state for the process (blocked on I/O wait). And if I try to force the unmount of the mounted, blocked filesystem, I get the laptop to kernel panic entirely. Now, to make the thing easier to me I’m re-using a Gentoo virtual machine (which I last used for writing a patch for the SCTP support in the kernel) to see if I can reproduce the problem there, and get to fix it, in one way or another.

Unfortunately I’ve spent the whole night working and trying to get this working, so now I’ll try to get some rest at least (it’s 9.30am, sigh!). All the other fixes will wait for tomorrow. On the other hand, I’d welcome thank yous if you find the help on Kerberos appreciated; organisations who would like to have even better Gentoo support for Kerberos are welcome to contact me as well…