flame@yamato ~ % touch /var/tmp/portage/test
touch: cannot touch `/var/tmp/portage/test’: No space left on device
flame@yamato ~ % df -h | grep /var/tmp
32G 7.2G 23G 25% /var/tmp
flame@yamato ~ % df -i | grep /var/tmp
2097152 419433 1677719 21% /var/tmp
Now, a mount cycle later it worked fine, but it’s still not too nice since it caused all the emerge running to fail, just like XFS did, but without leaving trace on the kernel log, which makes it obnoxious since it’s hard to debug. I hope 2.6.28 is going to be better, certainly the tinderboxing is a nice way to stress-test filesystems.
I start to consider the idea of OpenSolaris, NFS, and InfiniBand…
Infiniband is nice ;)I’m not sure if the last sentence is sarcasm, but here are my thoughts:OSOL will work great for fileserver. It’s a bit harder to live with for a desktop.Have you considerd btrfs? Thats supposed to have some design and functionality in common with ZFS…If you want ZFS you could also go with *BSD, the ZFS is getting really good.
When someone shouts NFS I still tend to shiver… I don’t know if the code quality is bad, or that they just lack time to make it a proper network filesystem, but everytime I use it I run into some kind of issue (albeit fixable). I think iSCSI is the way to go for network storage, or maybe even AoE.Either way, the steps in your (tinderbox) quest are nice to follow, so keep us informed of what works for you.
Andreas, I actually started considering using InfiniBand to move the data to a fileserver, running either FreeBSD or Solaris, but the problem is that I have no idea where to _find_ Infiniband hardware :/Michael, well, I looked into NFS code; I admit it’s not the nicest, and there’s a lot of copy-paste that could be avoided, but all in all it works decently well. It has the advantage of not being dependent on the filesystem behind it, which is the problem here; while I could use iSCSI (I already use iSCSI here to share data with the laptop), it would have the same problem as local drives: filesystems that under the heavy load of tinderboxing tend to fail.I wonder if there is a way to produce filesystem statistics, like number of files and directories created and deleted, average size of file and so on, to see at which point they tend to actually fail.If you’re interested in the tinderboxing experience, I’m going to blog more about it in the next days since it had some interesting developments lately.
Check your inode count on it, on many systems like this you have to up it or you can run out of inodes before you run out of space.( very definitely true for ext 2 and 3, happened to me too when I was tinderboxing Gentoo in the old days )
2Gbit fibre channel is pretty cheap used these days. All you need are some qlogic 22xx or 23xx cards and SCST on Linux to make your own high end SAN. The latency is much lower than Ethernet – you really will be able to push around 200MB/s in both directions, and you can run IPoFC.Comstar will do the same on Solaris but I think you need the more expensive 4gbit FC cards so far.In any case I think ext4 in 2.6.28 had a lot of patching done and is the first to be considered stable, it is working great on my laptop.
Spider, the @df -i@ command reports inode count, and I posted it to show that wasn’t the problem. Although I admit XFS and its 64-bit inode numbers really spoiled me, I still remember to check inode Count ;)Kevin I’m not sure about FC. It seems to be pretty expensive. Quotations at hand, I could get p2p infiniband for about €500/600 which, if I can get a stable job is not extreme. Do you have quotation for FC on my side of the pond?As for the kernel, I’m longing for .28 to be able to use the stable code, if it works well enough I’ll be converting a few more partitions. Anyway, this ext4 failure is still better than the xfs one.
Hi Diego,did you consider trying reiserfs (v3.6)? Or do I miss this test? Would be nice to know if it fails too.
Diego, a quick check on ebay.it for “qlogic” I saw 2gbit FC adapters for 30-50 euros. You can do P2P with FC as well (see my note about SCST and Comstar, or run IPoFC). All you need are a pair of adapters and some LC-LC cables. You can also use Emulex cards under Linux (well supported) for initiators but no SCST target driver. ~20 euro on ebay.it. All qlogic 22xx, 23xx and 24xx cards should work with SCST.Most people throw this stuff out because they don’t know what it is or how to work with it. I have two disk shelves, an IBM SCSI FC and a noname SATA FC hooked up to a 12 port switch for under $400USD. It can be tricky to set up but it is rock solid once it is.I would like to get some IB gear for testing but it is much less common and therefore expensive even in the US.Also, for small things like this it probably isn’t bad for overseas shipping, or at least cross EU?http://scst.sourceforge.net…http://opensolaris.org/os/p…