New Note 20
To this day, we still get --as-needed failures for packages in Gentoo; both for new packages and bumps. To this day, checking the list of reverse dependencies of libpng is not enough to ensure that all the packages build fine with libpng-1.4 (as Samuli found out the hard way). One common problem in both is represented by missing dependencies, which in a big part are caused by transitional transitive dependencies.
Transitional Transitive dependencies are those caused by indirect linking; since I don’t want to bore you all repeating myself you can read about it in this post and this one and finally another one — yes I wrote a lot about the matter.
How do transitional transitive, indirect dependencies, cause trouble with both --as-needed and with upgrade verification? Well it depends on a number of factors actually:
- the build might work on the developers’ systems because the libraries linked against indirectly bring in the actually needed libraries, either by
DT_NEEDEDor by libtool archives, but the former libraries aren’t used directly, thus--as-neededbreaks the link — misguided link - the build might work properly because some of the used (and linked to) libraries optionally use (and link to) the otherwise missing libraries; this work just as long as they are not built without that support; for instance you might use OpenSSL and Curl, separately, then link to Curl only, expecting it to bring in OpenSSL… but Curl might be built against GnuTLS or NSS, or neither;
- the build might work depending on the versions of libraries used, because one of the linked libraries might replace one library for another, dropping a needed library from the final link.
The same rules generally apply to the DEPEND and RDEPEND variables; relying on another package to bring in your own dependencies is a bad idea; even if you use GTK+ it doesn’t mean that you can avoid listing libpng as a package used, if you use it directly. On the other hand, listing libpng because it’s present in the final link (especially when not using --as-needed) is a bad idea which you definitely should avoid.
By ignoring transitional transitive dependencies, you invalidate the dependency-tree, which means we cannot rely on it when we’re trying to avoid huge fuckups if an important package changed API and ABI. This is why I have (wrongly) snapped back at Samuli for closing the libpng-1.4 tracker before I had the chance of running it through the tinderbox.
Bottomline, please always depend on what you use directly, both in linking and in ebuilds. Please!
Thanks to Odi for letting me know that I used (consistently) the wrong word in the article. This goes to show that I either should stop writing drafts at 3am or I should proofread them before posting.
When you say that we shouldn’t rely on another package to bring in dependencies, do you mean directly used or even indirectly used dependencies?For example, “dev-lang/ghc”:http://sources.gentoo.org/c… depends upon dev-libs/gmp (for the infinite precision Integer type); should all Haskell libraries and binaries then have an explicit dependency on gmp?
I always mean directly used; direct deps for direct use; indirect deps for indirect use.
I think you mean “transitive” instead of “transitional” (which means temporary)?
Wouldn’t it be best to make those packages on which half of the system depends to make them multislot so that the next time those less careful users will see that they need to rebuild half of the system and at least won’t end up hitting their head against the wall because they just upgraded very important package which breaks half of their system?
Sure, you go ahead and make it possible so that all the important packages that are known to break ABI are updated to install side-by-side, and that software using that can actually find the right version…For what it’s worth, Berkeley DB does just that, in just half-hacky way (because all distributions make it slotted, even though upstream does not really care to do so themselves), but even that “has quite a bit of problems”:https://blog.flameeyes.eu/2… as there isn’t a consistent way for developers to check for that.Thinking about libpng, it would have been almost trivial to support installation of 1.2 and 1.4 at the same time, given they are technically different libraries with different pkg-config files… but how much software uses the @libpng12.pc@ or @libpng14.pc@ file, or @-lpng12@/@-lpng14@ instead of using the too generic @libpng.pc@ and @-lpng@?
Maybe the 1.2 version eats puppies while 1.4 is less evil and eats only hamsters ;)??I was thinking hmmm, what about using somehow fatelf to cram both versions in one generic file/library?? I am a complete green horn in those things but that might solve at least those problems with API/ABI changes?
Repeat after me: _FatELF solves no problem; FatELF is only pointless ego stroke. FatELF is useless._There are a number of possible solution, gnome _usually_ get it right; on the other hand, there is no 100% effective solution. Some breakage, in a distribution like Gentoo, has to be expected, as you cannot really do an atomic update.I could write up some bits about possible solutions but they require quite a bit of work and somehow “corrupt” the current Gentoo situation, but for now you have to accept these breakages are unavoidable and they should just be dealt with so that they don’t break too much.
Repeat after me: _FatELF solves no problem; FatELF is only pointless ego stroke. FatELF is useless._There are a number of possible solution, gnome _usually_ get it right; on the other hand, there is no 100% effective solution. Some breakage, in a distribution like Gentoo, has to be expected, as you cannot really do an atomic update.I could write up some bits about possible solutions but they require quite a bit of work and somehow “corrupt” the current Gentoo situation, but for now you have to accept these breakages are unavoidable and they should just be dealt with so that they don’t break too much.
I know one universal solution :)Inform users through news what they are up against 🙂 If i knew for example after emerge –sync that upgrading to png 1.4 will break half of my system i wouldn’t do that unless some free time to recompile and unless it was completelt necessary (like some new voodoo capabilities of png 1.4) but since everything was working JUST FINE with the old version this must have been some failed upgrade test case or bad communication between developers 🙂
Could you explain why FatElf is not a solution? From my lamers point of view and reading all those use cases it seems to me it in some cases great thing.
I have “written about it”:blog.flameeyes.eu/tag/fatelf since day one… the whole idea is totally pointless, bogus, and solves *nothing at all*. It only increases complexity for no good reason. And makes Ryan’s ego feel ten times bigger because he implemented something that Apple had.
I must say that “NixOS”:http://nixos.org/nixos/ takes an interesting spin on this approach: AFAICT they basically slot the entire system. For example, after upgrading to libpng-1.4, it would keep the current 1.2 install with all the packages that depend upon it and then if you then wanted to build anything that needed 1.4 it would build it and new versions of all its dependencies that are currently built with 1.2. The advantage of this is that you can rollback any changes in case of breakage (at the cost of extra disk space being used).You can explicitly do the equivalent of a revdep-rebuild that will rebuild packages explicitly against the new version of a library but you then don’t get the ability to rollback.
That’s why i very welcome btrfs and it’s easy snapshots implementation. It could be somehow incorporated into emerge which would for example make a snapshot BEFORE merging new packages.