Since my original post from two years ago didn’t reach yet all the users, and some of the developers as well, I would like to reiterate that you should not be enabling ccache unconditionally.
It seems like our own (Gentoo’s) documentation is still reporting that using ccache makes build “10 to 5 times faster”. I’ll call this statement for what it is: bullshit. The rebuild of the same package might have such a hit, but not the normal emerge process of a standard user with Gentoo. If anything at all, the use of ccache will slow your build down, and even add further failure cases and make it difficult to identify errors.
Now, since the approach last time might not have been clear enough, let me try a different one, by describing the steps it takes when you call it:
- it has to parse the commandline to make sure you’re calling it for a single compile, it won’t do any good if you’re using it to link, or to build multiple source files at once (you can, especially if you use
-fwhole-program, but that’s for another day to write about), so in those cases, the command is passed through to the compiler itself; - once it knows that it’s doing a single compile, it changes the call to the compiler so that instead it simply preprocess the file, and stores the result in a temporary area;
- now it’s time to hash the data, with md4 (the parent of MD5), that as the man page suggests is a strong hash; this has good reasons to be strong, but it also means that it takes some time to hash the content; we’re not talking about the source files themselves, that are usually very small and thus quick to hash, but rather of the preprocessed file, which includes all the headers used… a quick example on my system, by just including eight common header files, produces a 120KB output (with
-O2and_FORTIFY_SOURCE… it goes down to 93KB if-O0is used); to that add the extra information thatccachehas to save (check the man pages for those); - now it has to search the filesystem, within its cache directory, if there is a file with the same md4; if there is, it gets either copied (or experimentally hardlinked, but let’s not go there for now), otherwise the preprocessed file is compiled and copied in the cache instead; in either case, it involves copying the object file from one side to the other.
Now, we can identify three main time-consuming operations: preprocessing, hashing and copying; all of them are executed whether this is a hit or a miss; if it’s a miss you add to that the actual build. How do they fare about the kind of resources used? Hashing, just like compiling, is a CPU-intensive operation; preprocessing is mixed (you got to read the header files from around the disk); copying is I/O-intensive. Given that nowadays most systems have multiple CPU and find themselves slowing down on I/O (the tinderbox taught me that the hard way), the copying of files around is going to slow down the build quite a bit. Even more so when the hit-to-miss ration is high. The tinderbox, when rebuilding the same failing packages over and over again (before I started masking the packages that failed at any given time), had a 40% hit-to-miss ratio and was slowed down by using ccache.
Now, as I already wrote, there is no reason to expect that the same exact code is going to be rebuilt so often on a normal Gentoo system… even if minor updates to the same package were to share most of the source code (without touching the internal header files), for ccache to work you’d have to leave untouched compiler, flags, and all the headers of all the dependent libraries… and this often includes the system header files from linux-headers. And even if all these conditions were to hold true, you’d have to have rebuilt object files for a total size smaller than the cache size, in-between, or the objects would have had expired. If you think that 2GB is a lot, think again, if you were to use -ggdb especially.
Okay now there are some cases where you might care about ccache because you are rebuilding the same package; that includes patch-testing and live ebuilds. In these cases you should not simply set FEATURES=ccache, but you can instead make use of the per-package environment files. You can then choose two options: you can do what Portage does (setting PATH so that the ccache wrappers are found before the compilers themselves) or you can simply re-set the CC variable, such as export CC="ccache gcc". Just set it in /etc/portage/env/$CATEGORY/$PN and you’re done.
Now it would be nice if our terrific Documentation team – instead of deciding once again (the last time was with respect to alsa-drivers) that they know better what the developers should support – would understand that stating in the handbook that ccache somehow magically makes normal updates “5 to 10 times faster” is foolish and should be avoided. Unfortunately upon my request the answer hasn’t been what you’d expect from logic.
Sorry but I can’t access your request:Access DeniedYou are not authorized to access bug #327945. To see this bug, you must first log in to an account with the appropriate permissions.
You are not authorized to access bug #327945.
Yes, yes I fixed that again, sounds like somebody were afraid to let public their response and mine as well.Maybe I’m expecting too much when I expect collaboration in avoiding users to shoot their feet.
Say an application has 1000 files that should be compiled.An update to the application introduced changes in 100 of those.Wouldn’t ccache in this case help by not recompiling most of the remaining files or would they have to be recompiled anyway?// Not a “compilation programmer”
Arc, realistically, at least one of the header files is going to get touched, be it to add one prototype, change one, etc etc etc. If that’s the case, _all_ the translation units (source-object files) that include that header will be invalidated and will count as a miss.Without limiting to the one-package-rebuild case, the tinderbox is the *perfect* use case for ccache and yet even there, it gets slowed down a huge lot..
EVERY gentoo host I’ve built (or rebuilt) has always had ccache in it, and I’ve done so somewhat blindly because ‘the handbook told me to.’ Now I figured that it wouldn’t cause too much trouble, but after the LIBPNG mess, I found ccache caused several build failures further throwing a wrench into the problem. I’m shocked at the mess you got through with the docs people, and future users not blinded by that same recommendation won’t know to thank you for your efforts, but theirs will be better for it.
Nice post. I too blindly put ccache into make.conf the last word on the line.Whenever theres an issue I comment out ccache, that usually clears up the issue. If not removing distcc will resolve even more ( besides the point i know)
Thanks George, I always try my best even though sometimes it is frustrating indeed..By the way, I just had an epyphany on why @ccache@ might be causing so much errors… the man page for it states that it’ll be hashing “the real compilers size and modification time” … the problem is that it’ll do that with the compiler files that it’ll be lunching and … I’ll leave strace and ls to talk for me:<typo:code>flame@yamato mytmpfs % strace -e stat,lstat,execve ccache x86_64-pc-linux-gnu-gcc test.c -o test.o |& grep x86_64-pc-linux-gnu-gccexecve(“/usr/bin/ccache”, [“ccache”, “x86_64-pc-linux-gnu-gcc”, “test.c”, “-o”, “test.o”], [/* 88 vars */]) = 0lstat(“/usr/bin/x86_64-pc-linux-gnu-gcc”, {st_mode=S_IFREG|0755, st_size=14552, …}) = 0stat(“/usr/bin/x86_64-pc-linux-gnu-gcc”, {st_mode=S_IFREG|0755, st_size=14552, …}) = 0execve(“/usr/bin/x86_64-pc-linux-gnu-gcc”, [“/usr/bin/x86_64-pc-linux-gnu-gcc”, “test.c”, “-o”, “test.o”], [/* 89 vars */]) = 0stat(“/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.0/x86_64-pc-linux-gnu-gcc”, {st_mode=S_IFREG|0755, st_size=267752, …}) = 0execve(“/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.0/x86_64-pc-linux-gnu-gcc”, [“/usr/x86_64-pc-linux-gnu/gcc-bin”…, “test.c”, “-o”, “test.o”], [/* 89 vars */]) = 0lstat(“/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.0/x86_64-pc-linux-gnu-gcc”, {st_mode=S_IFREG|0755, st_size=267752, …}) = 0lstat(“/usr/x86_64-pc-linux-gnu/gcc-bin/4.5.0/x86_64-pc-linux-gnu-gcc”, {st_mode=S_IFREG|0755, st_size=267752, …}) = 0flame@yamato mytmpfs % md5sum /usr/bin/x86_64-pc-linux-gnu-gcc /usr/lib64/misc/gcc-configa6fa7f35ddb14f6dcb3a62fc1a98fc90 /usr/bin/x86_64-pc-linux-gnu-gcca6fa7f35ddb14f6dcb3a62fc1a98fc90 /usr/lib64/misc/gcc-configflame@yamato mytmpfs % ls -l /usr/bin/x86_64-pc-linux-gnu-gcc /usr/lib64/misc/gcc-config-rwxr-xr-x 1 root root 14552 5 lug 15.30 /usr/bin/x86_64-pc-linux-gnu-gcc-rwxr-xr-x 1 root root 14552 25 apr 03.43 /usr/lib64/misc/gcc-config</typo:code>As you should be able to see now, the size and mtime that @ccache@ is saving up is _not_ that of the real compiler, but that of the @gcc-config@ wrapper.. so each time you run @gcc-config@ choosing a new profile (re-issuing the same won’t do) it will invalidate the cache. What’s the deal? Well, it *won’t* invalidate the cache if you update GCC, say, from 4.5.0 to 4.5.0-r1 since the gcc-config won’t be re-created. Fun times.
Hmmm, I have FEATURES=ccache commented out in my make.conf. I guess I must have tried it once, but seemed too mysterious/magical for me to understand how it would help with anything.Unless it’s standard practice for packages to compile the same file over and over again, I agree the documentation should be changed…
i never knew we can use ccache per-package and not globally.now i can use it when i am sure it will be useful – mostly scm packages i rebuild weekly.i guess it’s back to documentation for me.
I tried to use ccache for scm packages I rebuilt every week. Looking on statistics it wasn’t so good (more files were misses then hits). However it was about 45%/55% so the speed increased.The problem is that even small change of header changes checksum of all dependent files.
I personally experienced a tremendous speedup when rebuilding cxx code using ccache. Far beyond 5-10 times.
Thanks Diego, great post once again. I think I have ‘ccache’ enabled as well, just because the “docs told me to”. Reminds me of an The Haunted album, “The Haunted made me do it” =)I think that the bug discussion was somewhat strange as well. You did write a to-the-point conclusion of the issue (even though you didn’t supply the same amount of information as here or you previous post) and asked them to fix it, but they (Josh) seemed to become extremely defensive about it. I don’t understand Josh’s first comment at all, it’s completely off the mark. I think the proper response (and what you probably expected) would have been “Ah, recommending it to all users seems like a bad thing to do. We’ll fix it, but could you also add some more information to the man pages please?”.However, I do think that your tone became quite aggressive (and arrogant) very quickly. I guess you just feel very strongly about the subject and have become frustrated about it for a long time. It is never good building up tension! =)I think Jorge’s meddling was good, but I guess I’m just a diplomat as well, hehehe.Good work, once again! Btw, could you provide me with a link (or a post) as how to enable –as-needed? I have a couple of questions about it:1. You seem to state that it is ready for regular users (I’ve been using Gentoo since 2002, so I’m not exactly new). Is it, and do you recommend it today?2. How do I enable it, and if I do, should I do it globally?I’ve read many of your posts about it, just never really the post about _how to do it_. Thanks!
@FlameeyesOh… that sucks. Thanks for the info though.A second question, if I may. Would ccache help when recompiling after config enabling/disabling USE-flag changes? If so I’ll probably stick with it for now, as I experiment with changing USE-flags relatively often.@Maciej PiechotkaI get a 38% hit/miss ratio here.@AzPIt’s 2 years old, but there are some info on gentoo.org.http://www.gentoo.org/proj/…
I used to have ccache in FEATURES because of the ‘handbook’ that was a long while back. Running into the same issue after complaining about it over a year ago must be extremely frustrating.I think there gets to be a bit of a ‘territorial’ attitude among some developers. People tend to hate being wrong. I know I do. 😉 (not that that ever really happens mind you) The oriental concept of allowing others to ‘save face’ might be a good thing to keep in mind when dealing with others. Not apologizing for Josh; just a suggestion.I noted the ‘splitdebug’ flag mentioned. As a long time Gentoo user I like to contribute where I am able. Are there flags in FEATURES that could allow me/users to give MORE or BETTER feedback?Keep up the good work.
I see you never posted a follow-up: you failed to mention that even as you wrote this original post and published it, I was working on the handbooks to fix it. And the related documentation that referenced ccache that you completely missed when you posted the bug.Yeah, I fixed the handbook and every other ccache reference I found in the regular documentation. There was never a question that I didn’t want to fix it; I just wanted other teams to do their part and get the same fix in elsewhere, which Zac graciously did.Also, what’s the point in referencing the alsa-driver issue 3 years ago? Here, I’ll link it so everyone can see how offensive, accusatory, and demanding your tone was in every single comment:http://bugs.gentoo.org/show…Diego, people don’t always do exactly as you want them to do, every single time you want them to do it. That’s the give and take nature of what we do in Gentoo. Asking for help in the rest of the Portage stack to make sure that there’s not a mismatch between man make.conf and the handbook is not somehow “illogical” as you seem to believe.Your temper always gets the better of you. You seem to believe that you can bully people into doing whatever you want to be done, and that this is okay. This makes it very hard to work with you sometimes, as all of our fellow developers can attest (for example, the kernel team). That’s why jmbsvicetto stepped in when he did: he saw (as I did) that you had a good suggestion for the documentation, but HOW you presented it was completely out of line.
Jorge didn’t step up on his own, _you_ called devrel into the bug. My original presentation of the bug was definitely _not_ out of line, your original closing the bug with a RESO/REMIND was definitely out of line instead, you asked for ccache to be removed from Portage before you were to _consider_ updating the documentation.Yes I have a short fuse for your refusal to change documentation, when it goes into messing with people’s work: at the time of the ALSA guide change you _absolutely ignored_ the request from the ALSA maintainer to stand on the side of kernel team… do I have to ask who actually helped to lose me as ALSA maintainer? I think Fabio has a good idea of how well ALSA is maintained nowadays.So yes, before we went _again_ through the “Docs team has the last word about what goes into there, even after the people who do the work”, I was ready to escalate it already. Pardon me if I’m tired of having crap shoved down my throat by those who won’t even notice a change from one way or the other.
Hey Flameeyes,Once again, hats off to your tireless efforts for Gentoo Q/A!However, I also agree with nightmorph here. Your attacks on the Gentoo Documentation team went a little too far.Did they close the bug report prematurely? Perhaps. They did that to me too.But does that call for a full-scale rebuttal with swear words and whatever other strong language supposedly required? No.Rather, it calls for calm inter-developer discussion about the right thing to do. If the right thing isn’t being done, bring it to the next best forum (gentoo-dev?) for further discussion.nightmorph is one of the few folks who is working on the documentation. In some ways, his closing bugs might be him burning out. This is all purely speculation, and may or may not be true, but the point is people don’t always have bad intentions, and attacking them like they do only makes everyone mad, especially in cases like this where more than one explanation is likely.Thank you guys, flameeyes and nightmorph, as well as portage devs who started working on patches within hours of the bug being posted, for hitting this bug head-on. I’m sure the 2G extra free space on my hard-drive appreciates it too. 😉
Although I usually value your comments, I think that this time you are rather wrong. My experience on many system is quite the opposite, and the reason is trivial to explain: The overhead of ccache is much smaller than you want to admit.Speaking about command line parsing as a time-consuming task simply makes no sense, since we are only talking about one call for each source file – if this sums up to 1 second for a whole package, this package must be extremely huge. Moreover, for hashing several GB we are talking about some 10-30 seconds processor time even on a rather slow processor (and again, this must be a package with lots of files and each lots of includes). (Moreover, the direct mode of ccache-3 avoids most of these hashes, but that’s a different story). There remains the copy overhead which only slows down things if you are really on the IO-limit instead of the processor limit; for most of the slowly compiling projects (practically everything with C++), even for my fastest machine (admittedly “only” a dual core), the limiting factor is CPU, and not only by a few percent.On the other hand, these huge C++ projects (e.g. kdelibs, xulrunner, qt-*) compile 10-60 minutes, so the total overhead is about 1-2 percent; only a few hits already save this overhead! And usually there are much more than a few hits, since changes in -rxx are usually not so seldom and usually do not involve many files.As I understand, your tinderbox is almost the worst case example for ccache: Compiling all packages and then starting over is almost a guarantee for few hits – even if you exclude the compiler (or its wrapper, as you observed) from recompiling (but why should you do this?) it is almost sure that the old files are meanwhile removed from the ccache unless you reserve for it a tremendous amount of space. I would claim that a typical gentoo user has not more than 1500 packages installed, and the packages which are upgraded regularly are even less. So the number of hits is much higher than with your tinderbox.Of course, I agree that to name some number (like 5-10 times faster) makes no sense, since this number depends on the system, the update frequency of the user, the packages he has installed etc. However, I practically never experienced a slow-down with ccache on any of my systems in the mean, and it usually saves quite some time.I also conjectured that ccache hashes only the wrapper and not the compiler itself: Indeed, how could it be able to access the “real” compiler file? This is really a serious problem in gentoo. It might be reasonable either to patch ccache, or to force a cleaning of the ccache in the gcc ebuild… of course, I a more generic solution would be better, but I cannot think of some.
When the tinderbox was using ccache, it was re-building the same package over and over if it failed.E.g.: mplayer feailed to build, then a package depending on mplayer was queued, and mplayer tried to build again, and so on so forth.Plus, on package updates, it would rebuild those first, rather than (as it happens now) last.Obviously 2GB of cache in this case were not enough, so I had to increase it to have a decent hit-to-miss ratio. But even so, the result was a huge slow-down.On modern CPUs, as you said, hashing is quite quick… on modern CPUs with enough RAM, compiling even complex C++ code takes just a moment (if you exclude some ludicrous examples such as greycstoration)… but the copy operation takes quite a bit. I/O in most cases is the bottleneck, that’s why so many people nowadays build with tmpfs.
I am going to submit here that bugzilla is not a polite interface. That the whole process of reporting bugs though it has recently been improved is a problem. Some recognition of it’s shortcomings may help all of us.Bug reports should not be considerd an accusation. And closing a bug should not be considered a rebuff of someone by users just trying to help. but I believe it sometimes is. Curt replies. It is impossible to convey all the emotions a user whether it be the bug reporter or the developer to know all they feel. Being polite and explanatory with even a standard ‘thank you for your report’ Might allay some emotion that people feel when swatted so summarily.Now if I was a dev and we had discussed a bug over a year ago and it was agreed it should be removed and then another dev closes the bug with a curt comment I would be angry. It is not unreasonable to expect someone to feel this way.Especially if it is causing me more work and my load is enough already.And nightstorm if you know he is like that why didn’t you discuss your viewpoint before closing?
The IO overheads might be a bit less if you have your ccache store in a tmpfs.I also find the benefit of the ccache can dwindle a bit as it gets too old/large, so it being in a tmpfs that is flushed every reboot seems nice too.But that said, I stopped using ccache a while ago, and the difference is so minuscule I couldn’t tell you weather or not I was using ccache without having to look at the config.Take that as a testimony for not using ccache if you will.
I have proposed to Robin some time ago that it would be a nice idea to have ccache store the data with memcache rather than on filesystem, that would probably make it *much* nicer to use when it comes down to time-and-time again testing of a package.
I’m happy that finally the Gentoo Handbook has been fixed about the *ccache dogma*, but I have a simple question: as the ccache and split-debug features are “development features”, why aren’t they inside the “Developer Handbook”?If we need to fix this type of problems once for all it’s better to move all the features for development into a proper Handbook, surely not inside the installation Handbook for standard users.p.s.: as always, the mentioned problem has been fixed with a workaround instead of a proper solution.
I also experienced a big speedup when rebuilding cxx code using ccache.
_Note: I just deleted a bunch of comments that regarded an old idea that *should not be rehashed again*. It has no basis, it won’t work, it does not work, it has been tried, and the only reason why people seem not to forget that is wishful thinking that it can be fixed. Please google for it on this very blog if you want to know why it can’t work. I’m tired of rehashing that crap._
I don’t know, but I experience some speed improvement with updates, especially if rX moved to rX+1, because it usullay not modifies too much the package (e.g. some minor bugfixes can be there).An another good thing, where ccache can help the circular dependency resolving. E.g. cyrus-sasl depends on courier-authlib, and vice-versa. The rebuil of cyrus-sasl was done much faster than first build.Summary, I think ccache is a good stuff, but you must know its limitations, and be careful about it.
I’m not a Gentoo user, so I don’t have a gut feeling of what the common build scenario is for Gentoo, but I agree that it seems sane to not enable or recommend ccache blindly.The discussion about CPU contra I/O is interesting. I have made some measurements that you may find interesting here:http://ccache.samba.org/per…As can been seen, the overhead on a cache miss is typically 5-20%, so it’s indeed not negligible if the hit rate is low.Regarding the gcc-config wrapper: Yes, that’s bad. The first argument to ccache should always be the real compiler. If another “normal” wrapper (like distcc) is wanted, setting CCACHE_PREFIX is the preferred way. A workaround for gcc-config may be to set CCACHE_EXTRAFILES (available in ccache 3.0 and later) to some file that changes when the compiler changes, but I don’t know enough about gcc-config to say if that’s feasible. On my TODO list, there’s an item saying that it would be nice to optionally hash the output from something like “compiler -dumpspecs” to detect compiler changes. I guess that would solve the gcc-config problem nicely.Finally, I’m a bit surprised that Gentoo users are seeing so many troubles with ccache since I’ve seen no bug reports or mailing list posts about it. If someone finds a reproducible failure, I hope that s/he will report it upstream so that the problem may be fixed.– Joel (ccache maintainer)
Joel, I’ve never had problems with ccache. I think it’s an exaggeration that there are countless ccache related bugs.
Nico seriously do you think *you* have better idea of the number of ccache-related problems in Gentoo than the devs who are on the receiving end of the bugs?
Considering the handbook used to recommend using ccache, I assume a significant portion of Gentoo’s users are using it at least a couple of times a week. There would be thousands of bug reports if there were so many problems with it.
Remember that ccache is a pass-through (and won’t have any bad or good effect) when it misses, so no, “at least a couple of times a week” don’t cut it.Plus, having five bugs in two days all because of ccache are not good anyway, nor most are actually diagnosed because people will simply try again, sometimes disabling ccache, some other times changing the flags so that the hash will miss anyway.Really if you can’t believe *the whole QA team wanting ccache turned down from the general documentation* you’ve lost complete contact with reality.
After reading your post, I disabled ccache, then did my weekly eix-sync && emerge -auvND world update. I don’t have numbers, but I was surprised how much faster the display was scrolling. I’d guess maybe twice as fast.
Tony0945: I doubt that it was twice as fast. The highest slowdown I have been able to measure is 20%, and that’s for a *compilation only task*. Normally, other things than compilation take a non-negligible part of the build time, which means that the percental slowdown due to cache misses will be less than that. If you have found a case where there’s such a severe slowdown as you imply, please report a bug (on ccache, not Gentoo). Note that I’m not saying there’s no slowdown on cache misses — there is indeed, see http://ccache.samba.org/per… — but I think we should stick to real measurements.
Great work getting the docs updated. All my systems had CCACHE enabled. Recently I was building a new system and saw the updated CCACHE docs. I promptly removed it from the make.conf files on all my other systems too.
Joel, it was just a guess. Possibly it was 20%, possibly faster as I have a Reiser3 system with some fragmentation and it could (again, a guess) take longer to look up disk results than just recompiling.In any case, after reading Diego’s analysis, I concluded that almost all cache lookups would be misses.
Joel, Diego, caching mtime of wrapper is not a problem for Gentoo. As our toolchain guy told in http://bugs.gentoo.org/3131…“We take pains to make the mtime of the wrapper match the mtime of the compiler.”and thus ccache should just work.Diego, here is just an my personal opinion: this post exaggerates the problem.1. Performance. On normal Gentoo system (as robbat2 stated and I have very similar statistics here) ccache hits are about 10%-20% on normal Gentoo system and this really saves time. 2. Problems: if you check our bugzilla you’ll find really few ccache related issues. Personally I have only _one_ reproducible issue with ccache http://bugs.gentoo.org/2949… , but that’s not a big problem and I haven’t tried to reproduce it with ccache-3 yet (probably it is fixed there). So there are no known big problems with ccache.I agree that previous handbook suggestion “10 to 5 times faster” was also exaggeration, but even with better numbers ccache is still rather useful tool for most users.
Don’t know why, but setting the export in /etc/portage/env/${CATEGORY}/${PN} doesn’t work for me.
Got it working with /etc/portage/package.env.