Publishing Documentation

I have been repeating for years that blogs are not documentation out of themselves. While I have spent a lot of time over the years to make sure that my blog’s links are not broken, I also know that many of my old blog posts are no longer relevant at all. The links out of the blog can be broken, and it’s not particularly easy to identify them. What might have been true in 2009 might not be true in 2020. The best option for implementing something has likely changed significantly, given how ten years ago, Cloud Computing was barely a thing on the horizon, and LXC was considered an experiment.

This is the reason why Autotools Mythbuster is the way it is: it’s a “living book” — I can update and improve it, but at the same time it can be used as a stable reference of best practices: when they change it gets updated, but the link is still a pointer to the good practice.

At work, I pretty much got used to “Radically Simple Documentation” – thanks to Riona and her team. Which pretty much means I only needed to care about the content of the documentation, rather than dealing with how it would render, either in terms of pipeline or style.

And just like other problems with the bubble, when I try to do the same outside of it, I get thoroughly lost. The Glucometer Protocols site had been hosted as GitHub pages for a few years by now — but I now wanted to add some diagrams, as more modern protocols (as well as some older, but messier, protocols) would be much simpler to explain with UML Sequence Diagrams to go with.

The first problem was of course to find a way to generate sequence diagrams out of code that can be checked-in and reviewed, rather than as binary blobs — and thankfully there are a few options. I settled for blockdiag because it’s the easiest to set up in a hurry. But it turned out that integrating it is far from easy as it would seem.

While GitHub pages uses Jekyll, it uses such an old version that reproducing that on Netlify is pretty much impossible. Most of the themes that are available out there are mostly dedicated to personal sites, or ecommerce, or blogs — and even when I found one that seemed suitable for this kind of reference, I couldn’t figure out how to to get the whole thing to work. And it didn’t help that Jekyll appears to be very scant on debug logging.

I tried a number of different static site generators, including a few in JavaScript (which I find particularly annoying), but the end result was almost always that they seemed more geared towards “marketing” sites (in a very loose sense) than references. To this moment, I miss the simplicity of g3doc.

I ended up settling for Foliant, which appears to be more geared towards writing actual books than reference documentation, but wraps around MkDocs, and it provides a plugin that integrates with Blockdiag (although I still have a pending pull request to support more diagram types). And with a bit of play around it, I managed to get Netlify to build this properly and serve it. Which is what you get now.

But of course, since MkDocs (and a number of other Python-based tools I found) appear to rely on the same Markdown library, they are not even completely compatible with the Markdown as written for Jekyll and GitHub pages: the Python implementation is much stricter when it comes to indentation, and misses some of the feature. Most of those appear to have been at some point works in progress, but there doesn’t seem to be much movement on the library itself.

Again, these are relatively simple features I came to expect for documentation. And I know that some of my (soon-to-be-former) colleagues have been working on improving the state of opensource documentation frameworks, including Lisa working on Docsy, which looks awesome — but relies on Hugo, which I still dislike, and seems to have taken a direction which is going further and further away from me (the latest when I was trying to set this up is that to use Hugo on Linux they now seem to require you to install Homebrew, because clearly having something easy for Linux packagers to work with is not worth it, sigh).

I might reconsider that, if Hugo finds a way to implement building images out of other tools, but I don’t have strong expectations that the needs for documentation reference would be considered for future updates to Hugo, given how it was previously socialized as a static blog engine, only to pivot to needs that would make it more “marketable”.

I even miss GuideXML, to a point. This was Gentoo’s documentation format back in the days before the Wiki. It was complex, and probably more complicated than it should have been, but at least the pipeline to generate the documentation was well defined.

Anyhow, if anyone out there has experience in setting up reference documentation sites, and wants to make it easier to maintain a repository of information on glucometers, I’ll welcome help, suggestions, pull requests, and links to documentation and tools.

Reverse Engineering is just the first step

Last year I said that reverse engineering obsolete systems is useful giving as an example adding Coreboot support for very old motherboards that are simpler and whose components are more likely to have been described somewhere already. One thing that I realized I didn’t make very clear in that post is that there is an important step on reverse engineering: documenting. As you can imagine from this blog, I think that documenting the reverse engineering processes and results are important, but I found out that this is definitely not the case for everybody.

On the particularly good side, going to 33c3 had a positive impression on me. Talks such as The Ultimate GameBoy Talk were excellent: Michael Steil did an awesome job at describing a lot of the unknown details of Nintendo’s most popular handheld. He also did a great job at showing practical matters, such as which tricks did various games use to implement things that at first sight would look impossible. And this is only one of his talks, he has a series that is going on year after year, I’ve watched his talk about the Commodore 64, and the only reason why it’s less enjoyable to watch is that the recording quality suffers from the ages.

In other posts I already referenced Micah’s videos. These have also been extremely nice to start watching, as she does a great job at explaining complex concepts, and even the “stream of consciousness” streams are very interesting and a good time to learn new tricks. What attracted me to her content, though, is the following video:

I have been using Wacom tablets for years, and I had no idea how they really worked behind the scene. Not only she does a great explanation of the technology in general, but the teardown of the mouse was also awesome with full schematics and explanation of the small components. No wonder I have signed up for her Patreon right away: she deserve to be better known and have a bigger following. And if funding her means spreading more knowledge around, well, then I’m happy to do my bit.

For the free software, open source and hacking community, reverse engineering is only half the process. The endgame is not for one person to know exactly how something works, but rather for the collectivity to gain more insight on things, so that more people have access to the information and can even improve on it. The community needs not only to help with that but also to prioritise projects that share information. And that does not just mean writing blogs about things. I said this before: blogs don’t replace documentation. You can see blogs as Micah’s shop-streaming videos, while documentation is more like her video about the tablets: they synthesize documentation in actually usable form, rather than just throwing information around.

I have a similar problem of course: my blog posts are not usually a bit of a stream of consciousness and they do not serve an useful purpose to capture the factual state of information. Take for example my post about reverse engineering the OneTouch Verio and its rambling on, then compare it with the proper protocol documentation. The latter is the actual important product, compared to my ramblings, and that is the one I can be proud of. I would also argue that documenting these things in a easily consumable form is more important than writing tools implementing them as those only cover part of the protocol and in particular can only leverage my skills, that do not involve statistical, pharmaceutical or data visualisation skills.

Unfortunately there are obstacles to these idea of course. Sometimes, reverse engineering documentation is attacked by manufacturer even more than code implementing the same information. So for instance while I have some information I still haven’t posted about a certain gaming mouse, I already know that the libratbag people do not want documentation of the protocols in their repository or wiki, because it causes them more headaches than the code. And then of course there is the problem of hosting this documentation somewhere.

I have been pushing my documentation on GitHub, hoping nobody causes a stink, but the good thing about using git rather than Wiki or similar tools is exactly that you can just move it around without losing information. This is not always the case: a lot of documentation is still nowadays only available either as part of code itself, or on various people’s homepages. And there are at least two things that can happen with that, the first is the most obvious and morbid one: the author of the documentation dies, and the documentation disappears once their domain registration expires, or whatever else, or if the homepage is at a given university or other academic endeavour, it may very well be that the homepage gets to disappear before the person anyway.

I know a few other alternatives to store this kind of data have been suggested, including common wiki akin to Wikipedia, but allowing for original research, but I am still uncertain that is going to be very helpful. The most obvious thing I can think of, is making sure these information can actually be published in books. And I think that at least No Starch Press has been doing a lot for this, publishing extremely interesting books including Designing BSD Rootkits and more recently Rootkits and Bootkits which is still in Early Access. A big kudos to Bill for this.

From my side, I promise I’ll try to organize my findings of anything I’ll work on in the best of my ability, and possibly organize it in a different form than just a blog, because the community deserves better.

Testing stable; stable testing

It might not be obvious but my tinderbox is testing the “unstable” ~x86 keyword of Gentoo. This choice was originally due to the kind of testing (the latest and greatest version of packages for which we don’t know the averse effects of), and I think it has helped tremendously, up to now, to catch things that could have otherwise bit people months, if not years later, especially for the least-used packages. Unfortunately that decision also meant ignoring for the most part the other, now more common architecture (amd64 — although I wonder if the new, experimental amd32 architecture would take its place), as well as ignoring the stable branch of Gentoo, which is luckily tracked by other people from time to time.

Lacking continuous testing though, what is actually considered stable, sometimes it’s not very much so. This problem is further increased by the fact that sometimes the stable requests aren’t proper by themselves: it can be an user asking to stable a package, and the arch teams being called before they should have, it might be an overseen issue that the maintainer didn’t think of, or it might simply be that multiple maintainers had different feelings about stabilisation, which happens pretty often in team-maintained packages.

Whatever the case, once a stable request has been filed, it is quite rare that issues are brought up that are severe enough the stable is denied: it might be that the current stable is in a much worse shape, or maybe it’s a security stable request, or a required dependency of something following one of those destinies. I find this a bit unfortunate, I’m always happy when issues are brought up that delay a stable, even if that sometimes means having to skip a whole generation of a package, just as long as they mean having a nicer stable package.

Testing a package is obviously a pretty taxing task: we have numbers of combination of USE flags, compiler and linker flags, features, and so on so forth. For some packages, such as PHP, testing every and each combination of USE flags is not only infeasible, but just impossible within this universe’s time. to make the work of the arch team developers more bearable, years ago, starting with the AMD64 team, the concept of Arch Testers was invented: so-called “power users” whose task is to test the package and give green light for stable-marking.

At first, all of the ATs were as technically capable as a full-fledged developer, to the point that most of the original ones (me included) “graduated” to full devship. With time this requirement seems to have relaxed, probably because AMD64 became usable to the average user, and no longer just to those with enough knowledge to workaround the landmines originally present when trying to use a 64-bit system as a desktop — I still remember seeing Firefox fail to build because of integer and pointer variable swaps, once that became an error-worthy mistake in GCC 3.4. Unfortunately this also meant that a number of non-apparent issues became even less apparent.

Most of these issues often end up being caused by one simple fault: lack of direction in testing. For most packages, and that includes, unfortunately, a lot of my packages, the ebuild’s selftests are nowhere near comprehensive enough to tell whether the package works or not, and unless the tester actually uses the package, there is little chance that he knows really how to test it. Sure that covers most of the common desktop packages and a huge number of server packages, but that’s far from the perfect solution since it’s the less-common packages that require more eyes on the issues.

What the problem is, with most other software, is that it might require specific hardware, software or configuration to actually be tested. And most of the time, it requires a few procedures to be applied to ensure that it is actually tested properly. At the same time, I only know of Java and Emacs team publishing proper testing procedures for at least a subset of their packages. Most packages could use such a documentation, but it’s not something that maintainers, me included, are used to work on. I think that one of the most important tasks here, is to remind developers when asking stable to come up with a testing plan. Maybe after asking a few times, we’ll get to work and write up that documentation, once and for all.

PAM, logging in and changing passwords

I’ve been spending the past ten days/two weeks handling two full-time job at once; one was Windows-related so it won’t have any direct effect in what I’d be posting on the blog, the other involved Amazon EC2, so you’ll be seeing more rants sorry I meant posts on the topic soon. But first, …

Thanks to Constanze who became a full-fledged developer (congratulations!), I’ve been able to breath a bit more widely for what concerns PAM; another positive note comes from Eray becoming developer as well, which means I can get someone looking at pam_krb5 package. Which means I can get back to work on the M4-powered pambase package so that hopefully before end of the year we’re going to get it in testing at least. Additionally, user prometheanfire on #gentoo-hardened provided me with a sample configuration for LDAP that should make it much easier to implement it on pambase.

But the situation starts to become much more complicated; for instance, the ConsoleKit situation is so intricated that making it behave as intended is actually quite difficult: the invocation of the module is different whether we’re going to authenticate a text login or an X11 login session; some time ago we also found the hard way that some graphical login managers fail badly when you print too much information on the PAM output channel (such as Messages of the Day, the last login data, and mail status). This all results in having to have different sessions for local text and local graphical logins. I have a huge headache already when I start to think about XDMCP already.

This turn of events also makes me think that I should simply drop the system-login service that I’ve used in the previous iterations. The reason to use and include this service was to avoid duplication, but with M4, duplication is avoided during build time, not after install. This should make available only the three “leaf” services: system-remote-login (with optional ABL support), system-local-login (not renamed for compatibility reasons) with text-based login, and (by default) mail/motd/lastlogin modules; system-graphical-login with support for X11-based ConsoleKit sessions as well as without the extra verbose modules.

A note here: somebody asked me why of the minimal USE flag for pambase; the reason is relatively simple: even though the output of those can easily be discarded, they will be kept loaded in memory by processes such as sshd and fcron; dropping the modules from the services mean also reducing the memory usage of those process, minimally, but it does.

After the login process is sorted out there is another problem here and it has to do with changing passwords; I’ve said that before, but I’ll repeat it here. When the new pambase will be put in place, software that is able to change password will have to be updated to use a different service to do so; this will hinder the changing of password through sshd that was noted in the comments of one of my previous posts, but it is necessary if we want to have proper restriction among login methods.

The problem is that with PAM design, for what concern changing passwords, you end up with either you have to know all the currently in-use authentication methods or you have to know only one of the authentication methods and then you change all the authentication method to the new value or you change only one authentication method to the new value.

The end result is that I can’t think of any way to do what would make sense: change the token only for the systems that actually use the current password provided. Lacking this the situation is that we cannot have a single tool to do everything, so we’re going to have to stick with many different password-changing tools: passwd, chpasswd and their cousins will only require the Unix password and will only change the Unix password. You’re going to use separate tools for Kerberos, LDAP, SSH keys, PKCS#11 tokens, …

While it might sound as suboptimal it’s a compromise that actually make pambase manageable without having to resort to actual custom Linux-PAM implementations. I hope you can all agree on that.

Anyway, this only acts as a braindump; I hope I’ll be able to set up real documentation about the pambase system at one point or another, including some simple drawing to show how the authentication flow actually happens. Unfortunately if you remember, I noted that OpenOffice is the only decent software I can find to write flowcharts; unfortunately that is both cumbersome to add to a GIT repository, cumbersome to auto-produce results (when what it exports is what you wanted), and finally quite expensive in term of dependencies. I should probably try Inkscape back, possibly tied with rsvg (now that gdk-pixbuf works without X) would be a decent choice.

Debunking ccache myths redux

Since my original post from two years ago didn’t reach yet all the users, and some of the developers as well, I would like to reiterate that you should not be enabling ccache unconditionally.

It seems like our own (Gentoo’s) documentation is still reporting that using ccache makes build “10 to 5 times faster”. I’ll call this statement for what it is: bullshit. The rebuild of the same package might have such a hit, but not the normal emerge process of a standard user with Gentoo. If anything at all, the use of ccache will slow your build down, and even add further failure cases and make it difficult to identify errors.

Now, since the approach last time might not have been clear enough, let me try a different one, by describing the steps it takes when you call it:

  • it has to parse the commandline to make sure you’re calling it for a single compile, it won’t do any good if you’re using it to link, or to build multiple source files at once (you can, especially if you use -fwhole-program, but that’s for another day to write about), so in those cases, the command is passed through to the compiler itself;
  • once it knows that it’s doing a single compile, it changes the call to the compiler so that instead it simply preprocess the file, and stores the result in a temporary area;
  • now it’s time to hash the data, with md4 (the parent of MD5), that as the man page suggests is a strong hash; this has good reasons to be strong, but it also means that it takes some time to hash the content; we’re not talking about the source files themselves, that are usually very small and thus quick to hash, but rather of the preprocessed file, which includes all the headers used… a quick example on my system, by just including eight common header files, produces a 120KB output (with -O2 and _FORTIFY_SOURCE… it goes down to 93KB if -O0 is used); to that add the extra information that ccache has to save (check the man pages for those);
  • now it has to search the filesystem, within its cache directory, if there is a file with the same md4; if there is, it gets either copied (or experimentally hardlinked, but let’s not go there for now), otherwise the preprocessed file is compiled and copied in the cache instead; in either case, it involves copying the object file from one side to the other.

Now, we can identify three main time-consuming operations: preprocessing, hashing and copying; all of them are executed whether this is a hit or a miss; if it’s a miss you add to that the actual build. How do they fare about the kind of resources used? Hashing, just like compiling, is a CPU-intensive operation; preprocessing is mixed (you got to read the header files from around the disk); copying is I/O-intensive. Given that nowadays most systems have multiple CPU and find themselves slowing down on I/O (the tinderbox taught me that the hard way), the copying of files around is going to slow down the build quite a bit. Even more so when the hit-to-miss ration is high. The tinderbox, when rebuilding the same failing packages over and over again (before I started masking the packages that failed at any given time), had a 40% hit-to-miss ratio and was slowed down by using ccache.

Now, as I already wrote, there is no reason to expect that the same exact code is going to be rebuilt so often on a normal Gentoo system… even if minor updates to the same package were to share most of the source code (without touching the internal header files), for ccache to work you’d have to leave untouched compiler, flags, and all the headers of all the dependent libraries… and this often includes the system header files from linux-headers. And even if all these conditions were to hold true, you’d have to have rebuilt object files for a total size smaller than the cache size, in-between, or the objects would have had expired. If you think that 2GB is a lot, think again, if you were to use -ggdb especially.

Okay now there are some cases where you might care about ccache because you are rebuilding the same package; that includes patch-testing and live ebuilds. In these cases you should not simply set FEATURES=ccache, but you can instead make use of the per-package environment files. You can then choose two options: you can do what Portage does (setting PATH so that the ccache wrappers are found before the compilers themselves) or you can simply re-set the CC variable, such as export CC="ccache gcc". Just set it in /etc/portage/env/$CATEGORY/$PN and you’re done.

Now it would be nice if our terrific Documentation team – instead of deciding once again (the last time was with respect to alsa-drivers) that they know better what the developers should support – would understand that stating in the handbook that ccache somehow magically makes normal updates “5 to 10 times faster” is foolish and should be avoided. Unfortunately upon my request the answer hasn’t been what you’d expect from logic.

Autotools Mythbuster: Indexed!

Since there has been talking about Autotools today, and at least on the Reddit comments, my Autotools Guide got linked, I decided to take a few minutes of my time and extended the guide a bit further. I was already doing so to document the automake options (I was actually aiming at documenting the flavors and in particular the foreign mode so I would stop finding 0-sized NEWS files around), but this time I tried to make it a bit more searchable…

So right now there is a new page with the terms index and I shortened the table of concepts so that it more easily flow in the browser. The titles should be quite explaining of where to end up to. Right now I only added a single index for terms, even though I considered splitting them down per macro or variable, similarly to how it’s done in the official documentation, but for now this should do. I did add a “commons error” primary term though, as that should make it easier to find the common errors that the various tools report which I covered.

Now, these are the good news, here come the bad news though. Quite a while after first publication, the guide still is lacking a lot and my style hasn’t particularly improved. I’m not sure how good it can become by this pace. On the other hand, I’m still open to receiving requests and answering them there (thanks to Fabio asking about it, there’s now a whole section about pkg-config although it does not cover the -uninstalled variant that I use(d) on lscube so much).

Contributions in both corrections, general improvements or even just ideas are very welcome; so are donations or, more interestingly nowadays, flattr clicks (thanks to Sebastian for giving me an invite!). There is a flattr button at the bottom of the Autotools Mythbuster pages… if it is going to help you, a flattr, little as it can be, is going to show your appreciation in a way that reminds me why I started working on it.

There are going to be more news related to the guide in the future anyway, and a few more related to autotools in general, sweet news for some of you, slightly less sweet for me… so keep yourself seated, the journey is still on!

QA by disagreement

A few months ago I criticised Qt team for not following QA indications regarding the installation of documentation. Now, I wish to apologize and thank them: they were tremendously useful in identifying one area of Gentoo that needs fixes in both the QA policy and the actual use of it.

The problem: the default QA rules, and the policies encoded in both the Ebuild HOWTO (that should be deprecated!) and Portage itself, is to install the documentation of packages into /usr/share/doc/${PF}, a path that changes between different revisions of the same package. Some packages currently don’t respect that; some because it wasn’t thought about; some because they were mistakenly bound to ${P} on a zero-revision ebuild; some because they need not respect that paths.

When I started reporting for wrongly-installed documentation, I wasn’t expecting any in the latter category, it turns out to find so many examples of the latter; and a further number of examples and use cases that would call to change that policy altogether:

  • package foo requires to know where package bar installs its documentation, so that it can load it up, for whatever reason that is; this requires bar to either symlink its documentation somewhere or break the current policy, or otherwise you’d have to rebuild foo each time to find the correct path, which is not feasible;
  • package baz requires to know where its own documentation is installed to be able to access it at runtime; this either requires it to be hardcoded in the sources or to write it in a configuration file requiring semi-manual merge through etc-config; this is the case of Postfix for instance;
  • probably most important for users, API documentation bookmarks, right now, cannot be made stale unless you use symlinks; this is very annoying for the people who use those packages to develop (and you might guess that my main target here would be Ruby gems).

The solution: not sure if I can say I have a solution, but Samuli and Ulrich proposed a number of possible alternatives to solve the problem; from their suggestions I’d say we have to encode exactly three informations. The category of the package, the package name, and the slot of the package itself — the category is needed because there are a number of packages with the same name and different categories… sometimes even with the same version (dev-php5 and the old dev-php4 categories are a good example of those, and they were systematically breaking the policy stated above).

One solution I was proposed was /usr/share/doc/${CATEGORY}_${PN}-${SLOT} which wouldn’t be bad… but it would have a -0 appended to most of the directories; my preferred solution there would be to do something like omitting -${SLOT} if it’s 0. You’d have stable API documentation links, most of the intra-package and inter-package paths would be stable, and all in all you could drop the need for the document symlinking feature we currently have.

Unfortunately I’m expecting this to either require an EAPI bump or it’ll take a number of years before this can be properly implemented; I’ll probably have to either author myself – or find someone to author it for me – a GLEP to suggest changing dodoc. Contextually we should consider finding a better solution for compression, which is another problem we hit. Right now only the documentation installed with dodoc is getting compressed with the chosen compression program, which might be gzip, bzip2 or lzma, same as the man pages. While man pages gets processed after install and before binpkg/livefs merge steps, documentation is not.

But not all documentation needs to be compressed in the first place: HTML files (API documentation first of all); PDF files; code examples need to be accessible without compression; while we have a dohtml command to installing the web pages without compressing them, there is no equivalent for the other, and we have to rely on insinto/@doins@ pairs. Further on, with more and more autotools-based packages moving to autoconf 2.6x and supporting the --docdir option, we’re going to install more and more documentation directly into the directory, be it with the current ${PF} or other form; these won’t be compresses as they are, right now.

So, again thanks to Ben for actually challenging the status-quo; his insights here were the spark that made me think about this for a long time.

Blog posts are no replacement for documentation

Since I was asked in a previous post I’d like to make some notes about why I “document by blog post” in so many occasions.

I know perfectly well that my blog posts are no replacement for proper documentation; code, procedures and policies need to be properly documented, and tied to the project they are supposed to document. Documentation by blog post is difficult to write, manage and search, and can be indeed useless for the most art.

So why do I write it? Well, most of the time I start a blog post with some ideas in mind, write down it, and then depending on the feedback I either continue the topic or drop it entirely. I guess the most prominent counter-example is the For A Parallel World (which I know I haven’t updated in a while).

Writing proper documentation is important, and I know that pretty well, I have written and ranted about that before as well. And it’s knowing that, that I started the Autotools Mythbuster project which, to be honest, has given me mixed feedback, and satisfaction. The problem is: writing a blog takes just a modicum of effort, because I don’t have any obligation about form, or grammar, or language; I might soft-swear from time to time in a post, I might rant, I might have some smaller mistakes around, both in grammar and content, and so on. I don’t go updating blog posts to fix grammar and style and so on. Writing complex and organized documentation requires a lot more work, and when I say a lot I mean quite a lot more. Of course the result is also of much higher quality, because of that.

I have tried finding alternative routes to get the good results out without having to just apply that much effort in my (unpaid) free time; the first option was LWN, which actually helped me paying for a good part of Yamato’s hardware. Unfortunately LWN is not a perfect solution for me; partly because my topics tend to be quite low-level, too low-level for the LWN readers I’m afraid, and too distant from the Kernel as well (which is probably the only low-level area that LWN really writes a lot about); the other problem is that LWN is still something similar to a magazine, a journal, and thus does not allow an easy way to organised documentation; like autotools-mythbuster is. It would still be a puzzle of entries; of higher quality than a blog, but still a puzzle.

The classical form for organised documentation is that of a book; in today’s age, ebooks are also quite often used, to avoid the whole mass-production and distribution trouble for topics that might not be of enough interest (interestingly enough, that’s not true still for a lot of books, so lately I actually had to by more paper books because I couldn’t find PDFs of them to use with the Reader). Now, this also have troubles; as you might remember I already tried looking for a publisher for Autotools Mythbuster, before going with the open project it’s now.

The idea behind that would have been putting as much effort as possible into that single piece of documentation, complete it as much as possible and get it out in some complete form. There you go: high-quality results, paid effort, and organised up. Unfortunately, finding a publisher is never an easy task, and for that topic in particular, I ended up hitting a stone wall: O’Reilly already had somebody working on the topic, and the book is out now I think (I haven’t read it). This actually was ignoring a problem with classical books: they cannot easily be updated; and documentation often has to be, to correct mistakes, grammar, style, and especially to be kept up to date with what they document. For instance, Autotools Mythbuster has a specific section on forward porting (which I’ll probably keep updating for the future versions as well).

So the final option was making it an open book; again, the effort is not ignorable, so my first solution was to write on it on a donation basis: would have covered the effort I needed to put into it, and would still have been able to be there for everybody. I didn’t count in the fact that the topic is too developer-oriented to actually be of any use to people who would be donating. Indeed, I wish to thank the last two donors (in terms of time), Thomas Egger (who sent me a good mouse to replace the stupid Mighty Mouse, you’ll soon see results about that, by the way), and Joseph Booker (who sent me some books, I started with The Brief Wondrous Life of Oscar Wao because I was meaning to read it for almost two years now, but the useful one will soon prove useful, I’m sure). But they, like most others, never explicitly named the guide. And so I’m trying to find more time for the general postings than that in particular.

Just a note before you start wondering about the guide; yes I haven’t updated it in a while. Why? Because I sincerely feel like it’s not useful any more. As I said it requires a positive amount of effort to be extended; there is, true, some interest on it, but not enough to actually have moved anyone to ever try funding its extension. With O’Reilly now publishing a complete book on the matter, I don’t think it’s worth my time keeping it up. I might still extend it if I have to correct some build system, or if I discover something new, but not going to keep extending it by my own will without such a need.

Bottom-line: I could probably write more extensive, organised, and precise documentation about lots of stuff, especially the stuff I write about on the blog from time to time, but the problem is always the same: it requires time and effort; and both are precious commodity; most of my time is already committed to paid work nowadays, and Gentoo is getting more and more to the third place (first is work, second health). Documenting what I can with the blog is, in my opinion, still better than nothing, so I’ll keep doing that.

This is crazy, man!

And here, the term “man” refers to the manual page software.

I already noted that I was working on fixing man-db to work with heirloom-doctools but I didn’t go in much details about the problems I faced. So let me try to explain it a bit in more detail, if you’re interested.

The first problem is, we have two different man implementations in portage: sys-apps/man and sys-apps/man-db. The former is the “classical” one used by default, while the latter is a newer implementation, which is supposedly active and maintained. I say supposedly because it seems to me like it’s not really actively maintained, nor tested, at all.

The “classic” implementation is often despised because, among other things, it’s not designed with UTF-8 in mind at all, and even its own output, for locales where ASCII is not enough, is broken. For instance, in Italian, it outputs a latin1 bytestreams even when the locale is set to use UTF-8. Confusion ensures. The new implementation should also improve the caching by not using flat-files for the already-processed man pages, but rather using db files (berkeley or gdbm).

So what’s the problem with man-db, the new implementation? It supports UTF-8 natively, so that’s good, but the other problem is that it’s only ever tested coupled with groff (GNU implementation of the (n)roff utility), and with nothing else. How should this matter? Well, there are a few features in groff that are not found anywhere else, this includes, for instance, the HTML output (eh? no I don’t get it either, I can tell that the nroff format isn’t exactly nice, and I guess that having the man pages available as HTML makes them more readable for users, but why adding that to groff, and especially to the man command? no clue), as well as some X-specific output. Both those features are not available in heirloom, but as far as I can see they are so rarely used that it really makes little sense to depend on groff just because of those; and the way man-db is designed, when a nroff implementation different from groff is found, the relative options are not even parsed or considered; which is good to reduce the size of the code, but also requires to at least compile-test the software with a non-groff implementation, something that upstream is not currently doing it seems.

More to the point, not only the code uses broken (as in, incomplete) groff-conditionals, but the build system does depend on groff as well: the code to build the manual called groff directly, instead of using whatever the configure script found, and the testsuite relied on the “warnings” feature that is only enabled with groff (there was one more point to that: previously it returned error when the --warnings option was passed, so I had to fix it so that it is, instead, ignored).

But it doesn’t go much better from here on: not only man-db tries to use two output terminals that are no defined by the heirloom nroff (so I hacked that around temporarily on heirloom itself, the proper fix would be having a configuration option for man-db), but it also has problems handling line and page lengths. This is quite interesting actually, since it took me a long time (and some understanding of how nroff works, which is nowhere near nice, or human).

Both nroff and groff (obviously) are designed to work with paged documents (although troff is the one that is usually used to print, since that produces PostScript documents); for this reason, by default, their output is restricted width-wise, and has pagebreaks, with footers and headers, every given number of lines. Since the man page specification (command, man section, source, …) are present in the header and footer, the page breaks can be seen as spurious man page data in-between paragraphs of documentation. This is what you read on most operating systems, included Solaris, where the man program and the nroff program are not well combined. To avoid this problem, both the man implementations set line and page breaks depending on the terminal: the line break is set to something less than the width of the terminal, to fill it with content; the page break is set as high as possible to allow smooth scrolling in the text.

Unfortunately, we got another “groff versus the world” situation here: while the classic nroff implementation sets the two values with the .ll and .pl roff commands (no, really you don’t want to learn about roff commands unless you’re definitely masochist!), groff uses the LL register (again… roff… no thanks), and thus can be set with the command-line parameter -rLL=$width; this syntax is not compatible with heirloom’s nroff, but I’m pretty sure you already guessed that.

The classic man implementation, then, uses both the command and the register approach, but without option switches; instead it prepends to the manpage sources the roff commands (.ll and .nr LL that sets the register from within the roff language), and then gets the whole output processed; this makes it well compatible with the heirloom tools. On the other hand, man-db uses the command-line approach which makes it less compatible; indeed when using man-db with the heirloom-doctools, you’re left with a very different man output, which looks like an heirloom in itself, badly formatted and not filling the terminal.

Now we’re in quite a strange situation: from one side, the classic man implementation sucks at UTF-8 (in its own right), but it works mostly fine with heirloom-doctools (that supports UTF-8 natively); from the other side we got a newer, cleaner (in theory) implementation, that should support UTF-8 natively (but requiring a Debian-patched groff), but has lock-ins on groff (which definitely sucks with UTF-8). The obvious solution here would be to cross-pollinate the two implementations to get a single pair of programs that work well together and has native UTF-8; but it really is not easy.

And at the end… I really wish the roff language for man pages could die in flames, it really shows that it was designed for a long-forgotten technology. I have no difficulty to understand why GNU wanted to find a new approach to documentation with info pages (okay the info program suck; the documentation itself doesn’t suck as much, you can use either pinfo or emacs to browse it decently). Myself, though, I would have gone to plain and simple HTML: using docbook or some other format, it’s easy to generate html documentation (actually, docbook supports features that allows for the same documentation page to become both a roff-written man page, and an HTML page), and with lynx, links, and graphical viewers, the documentation would be quite accessible.

Autotools Mythbuster: updated guide

While preparing for my first vacation ever next week, I’ve been trying to write up more content on my guide so that at least my fellow developers in LScube have a references of what I’ve been doing, and Gentoo developers as well, as lately I’ve been asked quite a few interesting questions (and not just them).

So, first of all, thanks to David (user99) who cleaned up the introduction chapter, and to Gilles (eva) who gave me the new stylesheet (so that it doesn’t look as rough as it did before, it also narrows the lines so that it reads better. It might not be the final style, but it really is an improvement now.

As for my changes, I’ve been trying to change slightly the whole take of the guide, trying to write up complete working examples for the readers to use, that are listed in the main page. At the same time, I’m trying to cover the most important, or less known, topics, with particular attention to what people asked me, or what I’ve been using on projects which is not very well known. The list of topics added include:

  • using AC_CHECK_HEADERS to get one out of a prioritised list of headers;
  • using AC_ARG_* macros (enable, with and environment variables);
  • using AS_HELP_STRING to document the arguments;
  • using AC_ARG_WITH to set up automatic but not automagic dependencies;
  • using automake with non-recursive makefiles (including some of the catches);
  • improved automake silent-rules documentation;
  • using external macro files with autoconf (work in progress, for now only autoconf with no extras is documented).

I’m considering the idea of merging in some of the For A Paralllel World articles, at least those dealing with automake, to complete the documentation. The alternative would be to document all these kind of problems and writing something along the lines of “A Distributor’s Bible”… the problem with that idea is that almost surely somebody will complain if I use the name “Bible” (warning: I’m not a Catholic, I’m an atheist!)… and if I am to call it “The Sacred Book of Distributors” I’ll just be having to dig up all the possible mocks and puns over various religions, ‘cause I’ll be almost surely writing the ten commandments for upstream projects (“Thou shall not ignore flags”, “Thou shall version your packages”), and that also will enter a politically correctness problem.

Oh well, remember that I do accept gifts (and I tried not putting there the stuff that I’ll be buying next week… I already told my friends not to let me enter too many shops, but I’ll be following them and that’s not going to be a totally safe tour anyway…).