This is a blog post I would have definitely preferred not to write — it’s a topic that honestly does not touch me that much, for a few reasons I’ll explore in a moment, and at the same time it’s one that is quite controversial as it has quite a few meanings layered one on top of the other. Since I’m writing this, I would though first to make sure that the readers know who I am and why I’m probably going to just delete posts that tell me that I don’t care about compatibility with older systems and other operating systems.
My first project within Gentoo has been Gentoo/FreeBSD — I have a (sometimes insane) interest in portability with operating systems that are by far not mainstream. I’m a supporter of what I define “software biodiversity”, and I think that even crazy experiments have the right to exist, if anything to learn tricks and issues to avoid. So please don’t give me that kind of crap I noted above.
So, let’s see — I generally have little interest in keeping things around just for the sake of it, and as I wrote a long time ago I don’t use a separate /boot in most cases. I also generally dislike legacies for the sake of legacies. I guess it’s thus a good idea to start looking at which legacies bring us to the point of discussing whether /usr should be split. If it’s not to be split, there’s no point debating supporting split /usr, no?
The first legacy, which is specific to Gentoo is tied to the fact that our default portage tree is set to /usr/portage, and that both the ebuilds’ tree itself and the source files (distfiles), as well as the built binary packages, are stored there. This particular tree is hungry in both disk space itself and even more so in inodes. Since both the tree, and in general the open source projects we package, keep growing, the amount of these two resources we need increases as well, and since they are by default on /usr, it’s entirely possible that if this tree’s resources are allocated statically when partitioning, it’ll reach a point where there won’t be enough space, or inodes, to allocate anything in it. If /usr/portage resides in the root filesystem, it’s also very possible, if not even very likely, that the system would stop entirely to work because there is not enough space available on it.
One solution to this problem is to allocate /usr/portage on its own partition — I still don’t like it that much as an option, because /usr is supposed to be, standing to the FHS/LSB, for read-only data. Most other distributions you’ll find using subdirectories in /var as that’s what it’s designed to be used for. So why are we using /usr? Well, it turns out that this is something that was inspired from FreeBSD, where /usr is used for just about everything including temporary directories and other similar uses. Indeed, /usr/portage finds its peer in /usr/ports which is where Daniel seems to have been inspired to write Portage in the first place. It should be an easy legacy to overcome, but probably migrating it is going to be tricky enough that nobody has done so yet. Too bad.
Before somebody ask, yes, for a while ago splitting the whole /var – which is general considered much more sensible – was a pain in the neck, among other things because things were using /var/run and other similar paths before the partition could be mounted. The situation is now much better thanks to the fact that /run is now available much earlier in the boot process — this is not yet properly handled by all the init scripts out there but we’re reaching that point, slowly.
Okay so to the next issue: when do you want to split /usr at all? Well, this all depends on a number of factors, but I guess the first question is whether you’re installing a new system or maintaining an old one. If you install a new one, I really can’t think of any good reason to split /usr out — the only one that comes passing in my mind is if you want to have it in LVM and keep the rootfs as a standalone partition — and I don’t see why. I’d rather, at that point, put the rootfs in LVM as well, and just use an initrd to accomplish that — if it’s too difficult, well, that’s a reason to fix the way initrd or LVM are handled, not to keep insisting to split /usr! Interestingly enough, such a situation calls for the same /boot split I resented five years ago. I still use LVM without having the rootfs in it, and without needing to split /usr at all.
Speaking of which, most ready-to-install distributions only offer the option of using LVM — it makes sense, as you need to cater for as many systems as possible at once. This is why Gentoo Linux is generally disconnected to the rest: the power of doing things for what you want to use it for, makes it generally possible to skip the overgeneralization, and that’s why we’re virtually the only distribution out there able to work without an initrd.
Another point that came up often is with a system where the space in rootfs was badly allocated, and /usr is being split because there is not enough space. I’m sorry that this is a common issue, and I do know that it’s a pain to re-partition such a system as it involves at least a minimal downtime. But this is why we have workarounds, including the whole initrd thing. I mean, it’s not that difficult to manage, with the initrd, and yes I can understand that it’s more work than just having the whole system boot without /usr — but it’s a sensible way to handle it, in my opinion. It’s work or work, work for everybody under the sun to get split /usr working properly, or work for those who got the estimate wrong and now need the split /usr and you can guess who I prefer doing the work anyway (hint: like everybody in this line of business, I’m lazy).
Some people have said that /usr is often provided on NFS, and a very simple, lightweight rootfs is used in these circumstances — I understand this need, but the current solution to support split /usr is causing the rootfs to not be as simple and lightweight as before — the initrd route in that sense is probably the best option: you just get an initrd to be able to mount the root through NFS, and you’re done. The only problem with this solution is handling if /etc needs to be different from one system to the next, but I’m pretty sure it’s something that can be more easily fixed as well.
I have to be honest, there is one part of /usr that I end up splitting away very often: /usr/lib/debug — the reason is simple: it keeps increasing with the size of the sources, rather than with the size of the compiled code, and with the new versions of the compilers, which add more debug information. I got to a point where the debug files occupied four/five times the size of the rest of the rootfs. But this is quite the exception.
But why would it have to be that much of a problem to keep a split /usr? Well, it’s mostly a matter of what you’re supposed to be able to use without /usr being mounted. For many cases, udev was and is the only problem, as they really don’t want much in the matter of early-boot environment beside being able to start lvm and mount /usr, but the big problem happen if you want to be able to have even a single login with /usr not mounted — because the PAM chain has quite a few dependencies that wouldn’t be available until it’s mounted. Moving PAM itself is not much of an option, and this gets worse, because start-stop-daemon can technically also use chains that, partially, need /usr to be available, and if that happens, no init script using s-s-d would be able to run. And that’s bad.
So, do I like the collapsing of everything in /usr? Maybe not that much because it’s a lot of work to support multiple locations, and to migrate configurations. But at the same time I’m not going to bother, I’ll just keep the rootfs and /usr in the same partition for the time being, and if I have to split something out, it’ll be /var.
“and if I have to split something out, it’ll be /var” ..that’s what I was thinking when I started to read your article, as I have never split /usr myself (or anything at all, honestly), but I _have_ run into problems with stuff in /var taking all space.But I really don’t get what everyone’s so worked up about. Reading what you’ve wrote, I might be misinterpreting it, but the claim that “a seperate /usr is broken by default” or “break silently” is nonsense, it’s like saying scissors are weapons of destruction because people have clearly been cutting themselves, that’s a fact and you can’t deny it. You stated it’s possible in Gentoo, for example, which makes me think it’s not /usr’s fault, it’s the distro maintainers fault, allowing breaking things by the way he implemented things (where in Gentoo’s case, the user sort-of is the maintainer).Mounting /usr over NFS has always had my interest and I’ve given it some thoughts, even though I’ve never been trying to do it so far. Mainly because, quite some applications are putting their stuff in /usr which do need to be available before it’s mounted.But is that /usr’s fault? I’d say it’s the application at fault. Iirc, that’s what /sbin (and maybe even /bin) was for.. but the reason for having an /sbin seems to be slipping quite some minds, unfortunately.
I just wanted a bit of clarification; I mainly use FreeBSD, and almost all systems have / ( and thereby /usr ) mounted ro, and nothing ever complains, so which tempfiles are stored there on FreeBSD.
One thing that nobody ever touches on this splitting of mountpoints to different partitions is “filesystems”. Let’s say you don’t want fsck your rootfs, that one goes to xfs or similar. Let’s say you want a filesystem that doesn’t has a problem with insane amount of inodes for /usr/portage? Let’s say you want a filesystem with copy-on-write goodness for /usr? What about some “zomg I delete/create files *really* fast” filesystem for /var/tmp/{,portage}?It’s part of the Gentoo freedom/power of choice, to be able to pick the proper filesystem for the proper task.Disclaimer: I do use an initrd ^_^.
I guess it’s a bit silly argument, but a split /usr without initrd at least forces people to think about what is really required for boot and its dependencies.With a initrd you can always say “oh, depending in this is just a few 100 kB more, not a problem” and suddenly the initrd is larger than a full system in the past – and lots of data/libraries being dulicated instead of shared with the “main” system.I have never used a separate /usr or anything (it’s only desktop systems for me) but the initrd solution of basically having to maintain two systems always seemed horrendous to me, and more and more libraries being able to break booting isn’t a comfortable thought to me either.
I love it that I can have a separate /usr and no initrd; it’s part of the freedom I love about Gentoo. Not only do the initrd’s have to be generated, Reimer points out that they can grow to be awfully big.I have /boot, /etc, /bin, and /sbin plus the usual empty mountpoint directories (including /run) on the root partition. I get my LVM mounts in time before the init process gets to the daemons that need /usr and /var. One of the best things about not using the initrd is that the specification for what gets mounted and started is in the standard places in /etc, not split between whatever is in the initrd and /etc.As he points out, though, needing fancier startup possibilities like NFS would need the Gentoo user to know what needs to go into the root file system. A suggestion I made last summer was to be able to signal to Portage (say through USE flags) that a given package should be installed in / rather than /usr.No, I’m not hung up on being able to log into the root partition without PAM. If I want that kind of repair, there’s SysRescueCD.I don’t know if my suggestion is a non-starter; maybe there are many more very-well-known paths like /bin/bash and /usr/bin/env. In any event, I sure like being able to skip using an initrd.
I am an old fart when it comes to Unix/Linux. Back in the days when 256 MB drives were the size of washing machines old. Back then, separate /, /usr, and other file systems (/var wasn’t even a dream yet) were needed because you had to stuff more disk space into the filesystem tree. With the advent SCSI drives that were bigger than 4GB, it stopped mattering for small individual workstations if /usr was a separate partition or not. Larger machines still broke out filesystems out of reflex, and even now that is slowly dying off. Circumstances change things though. Having a lot of filesystems available in the kernel, each with special capabilities (and large numbers of disk controllers to keep the I/O separate) lets you eke out more and more performance if it is needed. Losing the capability of having a separate /usr filesystem is not a good idea, if it doesn’t cost too much to keep. Just as a for instance, I absolutely hate using initrds. I never use them myself if I can avoid them. I try to keep the startup of my own systems very simple, and don’t start complicated things that require stuff in /usr until it is mounted. I don’t really care if my workstation system comes up quickly or not, because they get rebooted so infrequently. My laptop is a huge difference though. That only has one drive, and it is an SSD now. I *Still* don’t use an initrd, but I don’t have a separate /usr, though I could if I wanted to.
ZFS solves the fixed-size partition problem. You can replace partitions with datasets and then space will be dynamically allocated from the pool on an as-needed basis. There is no need to reformat. Unfortunately, it currently requires an initramfs. Hopefully that will change in the near future.
I agree with Daniel; there are good reasons to keep the ability to break out /usr. SSDs are a perfect example, because of the limited write cycle lifetime. A read-only /usr, with a high read speed filesystem is perfect for that kind of hardware.Another classic example, already alluded to, is a shared NFS /usr. Even with cheap storage, it might still make sense to share parts of the filesystem across your development group, to ensure everyone is using the same versions of various tools, and such.
While I do understand the reluctance on using initrd, there’s also another way around that does not require using it and that’s Mike’s sep-usr.But James seems to have missed Daniel’s point: on an SSD you _don’t_ generally split /usr. And if you want to go with shared filesystem on NFS, you want a WHOLE filesystem, not just @/usr@, even more so if things would need to be moved on the rootfs, as if the two get out of sync, you’re in deep shit.Let me try to repeat: any advantage in not having to maintain a separate initrd, or slimming down the rootfs, will be negated, as long as more things will be needed for booting — and on a _desktop_ system that starts to be a lot more things than you expect, or need, on a server.If anything, Reimar is right that it should make us think on boot dependencies, but that’s a different story altogether.
Going right back to the /usr/portage point, it seems to work just fine to bind-mount /var/tmp/portage-tree (or any other location you like the sound of) as /usr/portage in /etc/fstab.This moves the write load (and consequent performance drops) off the read-only part of the file system, and lets you put them somewhere logical. Not ideal (changing the configuration in make.conf would be logical) but it works.
> But James seems to have missed Daniel’s point: on an SSD you don’t generally split /usr. And if you want to go with shared filesystem on NFS, you want a WHOLE filesystem, not just /usr, even more so if things would need to be moved on the rootfs, as if the two get out of sync, you’re in deep shit.No I don’t. I want to have only /usr via NFS, not /etc or something else.
make.conf(5) states that the PORTDIR variable can specify where the main portage tree should be looked for.So are you saying that this is not true and some parts of portage actually don’t respect PORTDIR and hacks like binding are necessary?As for splitting stuff, I agree that a desktop user generally won’t need to split anything at all, unless he requires features like LVM, disc encryption, etc.However, even though I don’t encrypt my disk on my notebook, I still split it into /boot / and /home (+swap), because this offers some protection from filesystem corruption spreading too far or in the unlikely event of the kernel going ballistic and hosing the rootfs (that actually happened on Arch some months ago, they compiled it against old coreutils or what).For advanced setups, having the opportunity to split stuff is often necessary, e.g. I’m setting up a new server with several disks in RAID, with a SSD for rootfs so libraries from /usr and configuration in /etc could be read faster, but I need to put write-intesive dirs on the RAID array, like /var and /usr/portage + /tmp.As for booting with initrd, who cares if you boot ten seconds longer? If it’s a server, boot speed is of no concern and on a notebook/desktop supend or hibernate is the way to go. There are quite a few tools to hanle initrd and initrd can be helpful with busybox if something fails. What real-life advantage does a non-initrd system have? Some kind of “purity” ?To conclude, I simply think the main problem with stuff like splitting /usr and initrd is that people forget that by generalizing and choosing one approach/solution in favor of the common denominator (i.e. the average user) we loose what we all love about *NIX systems and the OSS ecosystem: the diversity and multitude of opportunities and solutions. Having one file tree with arbitrary mount points is one of the greatest inventions in computing. Why give it up when there are still use cases for it?
> if it’s too difficult, well, that’s a reason to fix the way initrd or LVM are handled, not to keep insisting to split /usr!Though, as long as it still IS too difficult, removing the option to use a split /usr would be a regression.
I always try to respect the FHS. It’s arrangement and rules make sense. I was rather disappointed when the likes of systemd decided to make developers violate it by putting /run on the root instead of /var/run where it belongs…Now Archlinux is totally violating the FHS by making /lib and /lib64 symlinks to /usr/lib and /usr/lib64.If you have to use an initrd/initramfs/initcpio to mount a seperate /usr partition, the system setup is *broken* and needs to be fixed, the system needs to be able to be maintained assuming ONLY the / partition is available and there’s NOTHING in /usr.