Publishing Documentation

I have been repeating for years that blogs are not documentation out of themselves. While I have spent a lot of time over the years to make sure that my blog’s links are not broken, I also know that many of my old blog posts are no longer relevant at all. The links out of the blog can be broken, and it’s not particularly easy to identify them. What might have been true in 2009 might not be true in 2020. The best option for implementing something has likely changed significantly, given how ten years ago, Cloud Computing was barely a thing on the horizon, and LXC was considered an experiment.

This is the reason why Autotools Mythbuster is the way it is: it’s a “living book” — I can update and improve it, but at the same time it can be used as a stable reference of best practices: when they change it gets updated, but the link is still a pointer to the good practice.

At work, I pretty much got used to “Radically Simple Documentation” – thanks to Riona and her team. Which pretty much means I only needed to care about the content of the documentation, rather than dealing with how it would render, either in terms of pipeline or style.

And just like other problems with the bubble, when I try to do the same outside of it, I get thoroughly lost. The Glucometer Protocols site had been hosted as GitHub pages for a few years by now — but I now wanted to add some diagrams, as more modern protocols (as well as some older, but messier, protocols) would be much simpler to explain with UML Sequence Diagrams to go with.

The first problem was of course to find a way to generate sequence diagrams out of code that can be checked-in and reviewed, rather than as binary blobs — and thankfully there are a few options. I settled for blockdiag because it’s the easiest to set up in a hurry. But it turned out that integrating it is far from easy as it would seem.

While GitHub pages uses Jekyll, it uses such an old version that reproducing that on Netlify is pretty much impossible. Most of the themes that are available out there are mostly dedicated to personal sites, or ecommerce, or blogs — and even when I found one that seemed suitable for this kind of reference, I couldn’t figure out how to to get the whole thing to work. And it didn’t help that Jekyll appears to be very scant on debug logging.

I tried a number of different static site generators, including a few in JavaScript (which I find particularly annoying), but the end result was almost always that they seemed more geared towards “marketing” sites (in a very loose sense) than references. To this moment, I miss the simplicity of g3doc.

I ended up settling for Foliant, which appears to be more geared towards writing actual books than reference documentation, but wraps around MkDocs, and it provides a plugin that integrates with Blockdiag (although I still have a pending pull request to support more diagram types). And with a bit of play around it, I managed to get Netlify to build this properly and serve it. Which is what you get now.

But of course, since MkDocs (and a number of other Python-based tools I found) appear to rely on the same Markdown library, they are not even completely compatible with the Markdown as written for Jekyll and GitHub pages: the Python implementation is much stricter when it comes to indentation, and misses some of the feature. Most of those appear to have been at some point works in progress, but there doesn’t seem to be much movement on the library itself.

Again, these are relatively simple features I came to expect for documentation. And I know that some of my (soon-to-be-former) colleagues have been working on improving the state of opensource documentation frameworks, including Lisa working on Docsy, which looks awesome — but relies on Hugo, which I still dislike, and seems to have taken a direction which is going further and further away from me (the latest when I was trying to set this up is that to use Hugo on Linux they now seem to require you to install Homebrew, because clearly having something easy for Linux packagers to work with is not worth it, sigh).

I might reconsider that, if Hugo finds a way to implement building images out of other tools, but I don’t have strong expectations that the needs for documentation reference would be considered for future updates to Hugo, given how it was previously socialized as a static blog engine, only to pivot to needs that would make it more “marketable”.

I even miss GuideXML, to a point. This was Gentoo’s documentation format back in the days before the Wiki. It was complex, and probably more complicated than it should have been, but at least the pipeline to generate the documentation was well defined.

Anyhow, if anyone out there has experience in setting up reference documentation sites, and wants to make it easier to maintain a repository of information on glucometers, I’ll welcome help, suggestions, pull requests, and links to documentation and tools.

WordPress, really?

If you’re reading this blog post, particularly directly on my website, you probably noticed that it’s running on WordPress and that it’s on a new domain, no longer referencing my pride in Europe, after ten years of using it as my domain. Wow that’s a long time!

I had three reasons for the domain change: the first is that I didn’t want to keep the full chain of redirects of extremely old link onto whichever new blogging platform I would select. And the second is it that it made it significantly easier to set up a WordPress.com copy of the blog while I tweaked and set it up, rather than messing up with the domain at once. The third one will come with a separate rant very soon, but it’s related to the worrying statement from the European Commission regarding the usage of dot-EU domains in the future. But as I said, that’s a separate rant.

I have had a few people surprised when I was talking over Twitter about the issues I faced on the migration. I want to give some more context on why I went this way.

As you remember, last year I complained about Hugo – to the point that a lot of the referrers to this blog are still coming from the Hacker News thread about that – and I started looking for alternatives. And when I looked at WordPress I found that setting it up properly would take me forever, so I kept my mouth shut and doubled-down on Hugo.

Except, because of the way it is set up, it meant not having an easy way to write blog posts, or correct blog posts, from a computer that is not my normal Linux laptop with the SSH token and everything else. Which was too much of a pain to keep working with. While Hector and others suggested flows that involved GIT-based web editors, it all felt too Rube Goldberg to me… and since moving to London my time is significantly limited compared to before, so I either spend time on setting everything up, or I can work on writing more content, which can hopefully be more useful.

I ended up deciding to pay for the personal tier of WordPress.com services, since I don’t care about monetization of this content, and even the few affiliate links I’ve been using with Amazon are not really that useful at the end of the day, so I gave up on setting up OneLink and the likes here. It also turned out that Amazon’s image-and-text links (which use JavaScript and iframes) are not supported by WordPress.com even with the higher tiers, so those were deleted too.

Nobody seems to have published an easy migration guide from Hugo to WordPress, as most of the search queries produced results for the other way around. I will spend some time later on trying to refine the janky template I used and possibly release it. I also want to release the tool I wrote to “annotate” the generated WRX file with the Disqus archive… oh yes, the new blog has all the comments of the old one, and does not rely on Disqus, as I promised.

On the other hand, there are a few things that did get lost in the transition: while JetPack Plugin gives you the ability to write posts in Markdown (otherwise I wouldn’t have even considered WordPress), it doesn’t seem like the importer knows at all how to import Markdown content. So all the old posts have been pre-rendered — a shame, but honestly that doesn’t happen very often that I need to go through old posts. Particularly now that I merged in the content from all my older blogs into Hugo first, and now this one massive blog.

Hopefully expect more posts from me very soon now, and not just rants (although probably just mostly rants).

And as a closing aside, if you’re curious about the picture in the header, I have once again used one of my own. This one was taken at the maat in Lisbon. The white balance on this shot was totally off, but I liked the result. And if you’re visiting Lisbon and you’re an electronics or industrial geek you definitely have to visit the maat!

Why I do not like Hugo

Not even a year ago, I decided to start using Hugo as the engine for this blog. This has mostly served me well, except for the fact that it relies on me having some kind of access to a console, a text editor, and my Bitbucket account, which made posting stuff while travelling a bit harder, so I opted instead for writing drafts, and then staggering their posts — which is why you now see that for the most part I post something once every three days, except for the free ideas.

Hugo was sold to me as a static generator for blogs, and indeed when I looked into it, that’s what it was clearly aiming on being. Sure the support for arbitrary taxonomies make it possible to use it in slightly different setups for a blog, but it was at that point seriously focusing on blog, and a few other similar site types. The integration with Disqus was pretty good from the start, as much as I’m not really happy about that choice, and the conversion proceeded mostly smoothly, although it took me weeks to make sure the articles were converted correctly, and even months in I dedicated a few nights a month just to go through the posts and make sure their formatting was right, or through the tags to collapse duplicates.

All in all, while imperfect, it was not as horrible as having to maintain my own Typo fork. Until last week.

I finally decided that maintaining a separate website for the projects is a bad idea. Not just for the style being out of sync between the two, but most importantly because I barely ever update that content, as most of my projects are dead or have their own website already (like Autotools Mythbuster) or they effectively are just using their GitHub repository as the main description, even though it pains me. So the best option I found is to just build the pages I care about into Hugo, particularly using a custom taxonomy for the projects, and be done with it. Except.

Except that to be able to do what I had in mind, I needed a feature that was committed after the version of Hugo I froze myself at, so I had to update. Updates with Typo were always extremely painful because of new dependencies, and new features, and changes to the database layout, and all those kind of problems. Certainly Hugo won’t have these problems! Except it decided not to be able to render the theme I was using, as one function got renamed from RSSlink to RSSLink.

That was an easy fix; a bit less easy at first was figuring out that someone decided that RSS feeds should include, unconditionally, the summary of the article, not the full text, because, and I quote: «This is a somewhat breaking change, but is what most people expect from their RSS feeds.»

I’m not sure what these “most people” are. And I’d say that if you want to change such as default, maybe you want it to be an option, but that does not seem to be Hugo’s style, as I’ll show later. But this is not why I’m angry. I’m angry because changing the RSS from full content to summary is a very clear change in impression.

An RSS feed that has full article content, is an RSS feed for a blog (or other site) that wants to be read. You can use this feed to syndicate on Planets (yes they still exist), read it on services like Feedly, or NewsBlur (no they did not all disappear with the death of Google Reader), and have it at hand on offline readers on your mobile devices, too.

RSS feeds that only carry summaries, are there to drive traffic to a site. And this is where the nasty smell around SEOs and similar titles come back in from below the door. I totally understand if one is trying to make a living off their website they want to be able to bring in traffic, which include ads views and the like. I have spoken about ads before, and though I recently removed it from the blog altogether for lack of any useful profit, I totally empathise with those who actually can make a profit and want people to see their ads.

But the fact that the tools decide to switch to this mode make me feel angry and sick, because they are no longer empowering people to make their view visible, they are empowering them to trick users into opening a website, to either get served ads, or (if they are geek enough to use Brave) give bitcoin to the author.

As it turns out, it’s not the only thing that happen to have changed with Hugo, and they all sound like someone decided to follow the path of WordPress, that went from a blogging engine to a total solution for managing websites — which is kind of what Typo did when becoming Publify. Except that instead of going to a general website solution, they decided to one-up all of them. From the same release notes of the version that changed the RSS feed defaults:

Hugo 0.20 introduces the powerful and long sought after feature Custom Output Formats; Hugo isn’t just that “static HTML with an added RSS feed” anymore. Say hello to calendars, e-book formats, Google AMP, and JSON search indexes, to name a few ( #2828 ).

Why would you want to build e-book formats and calendars with the same tool you used to build a blog with? Sure, if it actually was practical I could possibly make Autotools Mythbuster use this, but I somehow doubt it would have enough support for what I want to get out of the output, so I don’t even want to consider that for now. But all in all, it looks like widening a little too much the target field.

Anyway, I went and reverted the changes for my local build of Hugo. I ended up giving up on that by the way, and just applied a local template replacement instead, since that way I could also re-introduce another fix I needed for the RSS that was not merged upstream (the ability to put the taxonomy data into the feed, so you can use NewsBlur’s intelligence trainer to filter out some of my blog’s content). Of course maintaining a forked copy of the builtin template also means that it can break when I update if they decided that it should be FeedLink next time around.

Then I pushed the new version, including the updated donations page – which is ~~not redirected from the old one yet, still working on that~~ gone – and stopped looking too much onto it. I did this (purposefully) in the 3-days break between two posts, so that if something broke I would have time to fix it, but it looked everything was alright.

Until I noticed that I somehow flooded Planet Gentoo with a bunch of posts dating back up to 2006! And someone pinged me on Hangouts for the same reason. So I rolled back to the old version (that did not solve the flooding unfortunately), regenerated, and started digging what happened.

In the version of Hugo I used originally, the RSS feeds were fixed to 15 items. This is a perfectly reasonable debug for a blog, as I didn’t go anywhere near it even at at the time I was spending more time blogging than sleeping. But since Hugo is no longer targeting to be a blog engine, that’s not enough. “News” sites (and I use it in quote, because too many of those are actually either aggregators of other things, or just outright scammers, or fake news sites) would have many more than that per day, so 15 is clearly not a good option for them. So in Hugo 0.19 (the version before the one that changed to use summary), this change can be found:

Make RSS item limit configurable #3035

This is reasonable. The default is kept to 15, but now you can change it in the configuration file to whatever you want it to be, be it 20, 50, or 100.

What I did not notice at that point, was from the following version:

Raise the default rssLimit #3145

That sounds still good, no? It raises the limit. To what?

hugolib: Default rssLimit to unlimited

Of course this is perfectly fine for small websites that have a hundred or two pages. But this blog counts over 2400 articles, written over the span of 12 years (as I have recovered a number of blog posts from previous platforms, and I’m still always looking to see if I can find the backups with the posts of my 15 years old self). It ended up generating a 12MB RSS feed with every single page published up to them.

So what am I doing now? That, unfortunately, I’m not sure. This is the kind of bad upgrade path that frustrated the heck out of me with Typo. Unfortunately the only serious alternative I know to this is WordPress, and that still does not support Markdown unless you use a strange combinations of plugins and I don’t even want to get into that.

I am tempted to see what’s out there for Ruby-based blog engines, although at this point I’m ready to pay for a solution that works native on ~~AWS~~ EC2¹, to avoid having to manage it myself. I would like to be able to edit posts without requiring me a console and git client, and I would like to have an integrated view of the comments, instead of relying on Disqus², which at least a few people hate, and I don’t particularly enjoy.

For now, I guess I’ll have to be extra careful if I want to update Hugo. But at least I should be able to not break this again so easily as I’ll be checking the output before and after the change.

Update (2017-08-14): it looks like this blog post got picked up by Internet’s own peanut gallery that not only don’t seem to understand why I’m complaining here (how many SEOs there?), but also appear to have started suggesting with more or less care a number of other options. I say “more or less” because some are repeats or aimed at solving different problems than mine. There are a few interesting ones I may spend some time looking into, either this week or the next while on the road.

Since this post is over a month old by now, I have not been idle and started instead trying out WordPress, with not particularly stunning results either. I am though still leaning towards that option, because WordPress has the best upgrade guarantees (as long as you’re not using all kind of plugins) and it solves the reliance on Disqus by having proper commenting support.

¹ Update (2017-08-14) I mean running the stack, rather than pushing and storing to S3. If there’s a pre-canned EC2 instance I can install and run, that’ll be about enough for me.

² don’t even try suggesting isso. Maybe my comments are not big data, but at 2400+ blog posts, I don’t really want to have something single-threaded that access a SQLite file!

New blog, powered by Hugo

If you’re seeing this post, it means I have flipped the switch and started using Hugo as the engine for my blog. You may remember I almost turned off the blog, and I have indeed made the blog read-only in June, as Typo was misbehaving and I could not be bothered to fix it.

A few months ago I decided to make the blog read-only and disabled comments, it actually meant replacing the whole app with the output of the caching layer in Rails. Not great but it worked out for a while. In the meantime I have been thinking of what else I could do to make it easier to maintain the blog. At the moment, the best option I can think of is this: a static blog engine with Disqus for the commenting, which is not my favourite but could not find a better alternative.

Hugo works out fairly well, and I’ve settled on a modified version of the Strata theme. Some of the improvements I’ve made (and I’m making) I’ll send upstream, others might not so easily make it. I’m a bit surprised that Hugo does not, by default, try minimizing or merging the Javascript resources, so I might invest some time on that at some later point.

While I made sure that all the permalinks would match with what I had before, there are obvious things that changed place, such as the RSS feeds, and the monthly archives don’t exist anymore (did anybody ever use them?) plus I’m missing a CSE to search the blog. On the other hand I’m also missing ads, so your mileage may vary on whether this is an improvement or not.

As for comments, I think this is important to state outright, because I know someone will start complaining about the fact I settled on a closed, non-free service such as Disqus. I don’t like it either but it’s the best option I found in the short run. I do hope I’ll manage to get a better replacement at some point in the future, but the open-source alternatives I found appear to be the trend-du-jour.

Isso appears to be the most mature alternative, but it uses SQLite to store the comments, because «comments are not bigdata» completely ignoring the fact that SQLite is horrible for parallel accesses, which means the app is likely not going to keep up with a spam attack. The alternative from Hugo’s author relies instead on MongoDB and NodeJS, which is for a different reason horrible. I did see one written in PHP, but using XML files written directly, with the default instructions suggesting changing a directory to 777.

I particularly don’t like the fact that Disqus only allows getting a backup copy of the comments out up to a limit that they don’t disclose, with their instructions essentially suggesting that you should try requesting a backup, and if nothing mails you back twice, then you’re too big. Not a great trust I can put on that.

So anyway hopefully I’ll manage to write more interesting stuff soon, for now I hope I didn’t break too many links. There are also a few posts that likely have broken text due to having to convert all the two thousand posts written in Textile to Markdown with Pandoc and it not being completely accurate. If you see anything wrong, please just leave a comment and I’ll pick it up from there.

About the background picture: it might be worth noting, the picture I’m using is mine and I took it from my Air France flight to Shanghai, in September. Yes, the guy who was so afraid of flying is now traveling around the globe for work. Ironic?