Project Memory

For a series of events that I’m not entirely sure the start of, I started fixing some links on my old blog posts, fixing some links that got broken, most of which I ended up having to find on the Wayback Archive. While looking at that, I ended up finding some content for my very old blog. A one that was hosted on Blogspot, written in Italian only, and seriously written by an annoying brat that needed to learn something about life. Which I did, of course. The story of that blog is for a different time and post, for now I’ll actually focus on a different topic.

When I started looking at this, I ended up going through a lot of my blogposts and updated a number of broken links, either by linking to the Wayback machine or by removing the link. I focused on those links that can easily be grepped for, which turns out to be a very good side effect of having migrated to Hugo.

This meant, among other things, removing references to identi.ca (which came to my mind because of all the hype I hear about Mastodon nowadays), removing links to my old “Hire me!” page, and so on. And that’s where things started showing a pattern.

I ended up updating or removing links to Berlios, Rubyforge, Gitorious, Google Code, Gemcutter, and so on.

For many of these, turned out I don’t even have a local copy (at hand at least) of some of my smaller projects (mostly the early Ruby stuff I’ve done). But I almost certainly have some of that data in my backups, some of which I actually have in Dublin and want to start digging into at some point soon. Again, this is a story for a different time.

The importance to the links to those project management websites is that for many projects, those pages were all that you had about them. And for some of those, all the important information was captured by those services.

Back when I started contributing to free software projects, SourceForge was the badge of honor of being an actual project: it would give you space to host a website, as well as the ability to have source code repositories and websites. And this was the era before Git, before Mercurial, and the other DVCS, which means either you had SourceForge, or you likely had no source control at all. But SourceForge admins also reviewed (or at least alleged to) every project that was created, and so creating a project on the platform was not straightforward, you would do that only if you really had the time to invest on the project.

A few projects were big enough to have their own servers, and a few projects were hosted in some other “random” project management sites, that for a while appeared to sprout out because the Forge software used by SourceForge was (for a while at least) released as free software itself. Some of those websites were specific in nature, others more general. Over time, BerliOS appeared to become the anti-SourceForge, with a streamlined application process, and most importantly, with Subversion years before SF would gain support for it.

Things got a bit more interesting later, when things like Bazaar, Mercurial, GIT and so on started appearing on the horizon, because at that point having proper source control could be achieved without needing special servers (without publishing it at least, although there were ways around that. Which at the same time made some project management website redundant, and others more appealable.

But, let’s take a look at the list of project management websites I have used and are now completely or partly gone, with or without history:

  • The aforementioned BerliOS, which has been teetering back and forth a couple of times. I had a couple of projects over there, which I ended up importing to GitHub, and I also forked unpaper. The service and the hosting were taken down in 2014, but (all?) the projects hosted on the platform were mirrored on SourceForge. As far as I can tell they were mirrored read-only, so for instance I can’t de-duplicate the unieject projects since I originally wrote it on SourceForge and then migrated to BerliOS.

  • The Danish SunSITE, which hosted a number of open-source projects for reasons that I’m not entirely clear on. NoX-Wizard, an open-source Ultima OnLine server emulator was hosted there, for reasons that are even murkier to me. The site got renamed to dotsrc.org, but they dropped all the hosting services in 2009. I can’t seem to find an archive of their data; Nox-Wizard was migrated during my time to SourceForge, so that’s okay by me.

  • RubyForge used that same Forge app as SourceForge, and was focused on Ruby module development. It was abruptly terminated in 2014, and as it turns out I made the mistake here of not importing my few modules explicitly. I should have them in my backups if I start looking for them, I just haven’t done so yet.

  • Gitorious set itself up as being an open, free software software to compete with GitHub. Unfortunately it clearly was not profitable enough and it got acquired, twice. The second time by competing service GitLab, that had no interest in running the software. A brittle mirror of the project repositories only (no user pages) is still online, thanks to Archive Team. I originally used Gitorious for my repositories rather than GitHub, but I came around to it and moved everything over before they shut the service down, well almost everything, as it turns out some of the LScube repos were not saved, because they were only mirrors… except that the domain for that project expired so we lost access to the main website and GIT repository, too.

  • Google Code was Google’s project hosting service, that started by offering Subversion repositories, downloads, issue trackers and so on. Very few of the projects I tracked used Google Code to begin with, and it was finally turned down in 2015, by making all projects read-only except for setting up a redirection to a new homepage. The biggest project I followed from Google Code was libarchive and they migrated it fully to GitHub, including migrating the issues.

  • Gemcutter used to be a repository for Ruby gems packages. I actually forgot why it was started, but it was for a while the alternative repository where a lot of cool kids stored their libraries. Gemcutter got merged back into rubygems.org and the old links now appear to redirect to the right pages. Yay!

With such a list of project hosting websites going the way of the dodo, an obvious conclusion to take is that hosting things on your own servers is the way to go. I would still argue otherwise. Despite the amount of hosting websites going away, it feels to me like the vast majority of the information we lost over the past 13 years is for blogs and personal websites badly used for documentation. With the exception of RubyForge, all the above examples were properly archived one way or the other, so at least the majority of the historical memory is not gone at all.

Not using project hosting websites is obviously an option. Unfortunately it comes with the usual problems and with even higher risks of losing data. Even GitLab’s snafu had higher chances to be fixed than whatever your one-person-project has when the owner gets tired, runs out of money, graduate from university, or even dies.

So what can we do to make things more resilient to disappearing? Let me suggest a few points of actions, which I think are relevant and possible right now to make things better for everybody.

First of all, let’s all make sure that the Internet Archive by donating. I set up a €5/month donation which gets matched by my employer. The Archive not only provides the Wayback Machine, which is how I can still fetch some of the content both from my past blogs and from blogs of people who deleted or moved them, or even passed on. Internet is our history, we can’t let it disappear without effort.

Then for what concerns the projects themselves, it may be a bit less clear cut. The first thing I’ll be much more wary about in the future is relying on the support sites when writing comments or commit messages. Issue trackers get lost, or renumbered, and so the references to those may be broken too easily. Be verbose in your commit messages, and if needed provide a quoted issue, instead of just “Fix issue #1123”.

Even mailing lists are not safe. While Gmane is currently supposedly still online, most of the gmane links from my own blog are broken, and I need to find replacements for them.

This brings me to the following problem: documentation. Wikis made documenting things significantly cheaper as you don’t need o learn lots neither in form of syntax nor in form of process. Unfortunately, backing up wikis is not easy because a database is involved, and it’s very hard, when taking over a project whose maintainers are unresponsive, to find a good way to import the wiki. GitHub makes thing easier thanks to their GitHub pages, and it’s at least a starting point. Unfortunately it makes the process a little more messy than the wiki, but we can survive that, I’m sure.

Myself, I decided to use a hybrid approach. Given that some of my projects such as unieject managed to migrate from SourceForge to BerliOS, to Gitorious, to GitHub, I have now set up a number of redirects on my website, so that their official websites will read https://www.flameeyes.eu/p/glucometerutils and it’ll redirect to wherever I’m hosting them at the time.

What’s wrong with release notifications?

Distributions of the like of Gentoo have one huge issue with users: they all demand their updates the same moment they are released. This is why many people, including me, have ranted before about the meme of 0day bumps. Generally speaking, we tend to know about the new release of a package we maintain, because we follow, tightly or loosely the development. Unfortunately, it’s quite possible that the new release passes into background for whatever reason, and the result is, well, a not bumped package. Note here: it’s very well possible that some developer will forget to bump his own (upstream) package; shit happens sometimes.

Most of the time, to solve this kind of problem we can use one of the many tools at our disposal to identify release notifications… unfortunately this is not all that feasible nowadays: it was better and it definitely got worse in the past months! Given that most upstream barely have a release publishing procedure, most of us preferred notifications that are not “actively handled” by the developers, but rather than happen as a by-product of the release itself: this way even sloppier release had their notification sent out.

The biggest provider of by-product release notifications was, once upon a time, SourceForge — I say “once upon a time” because they stopped that. While I can understand that a lot of the services offered by SF were redundant, and that most projects ended up setting up better, if less integrated, software anyway (such as phpBB, Mantis – as Bugzilla wouldn’t work, or various Wikis), and I can appreciate that the old File Release System was definitely overcomplex, I can’t see why they stopped allowing to subscribe to notifications. The email that they sent are now loosely replaced by the RSS feed of released files… the problem is that the feed is huge (as it lists all the files ever released for a project) and not sorted chronologically. Sure there is still freshmeat but to have notifications working there you’re asking the upstream maintainer to explicitly remember bumping the version in a different website, and that’s a bit too much, for most people.

You’d expect that other sites that took the place of SourceForge got better at handling this things, wouldn’t you? But that’s definitely not true. Strangely enough, the good example here seem to come from the Ruby community (I say “strangely enough” because you might remember that I ranted so much about missing practises, mandatory procedures and metadata and so on.

First of all, Rubyforge still keeps release notifications by mail (good!), and second, kudos to gemcutter that allows for subscribing to a (semi-)private RSS feed with the new release of just a subset of all the gems released (the Gentoo Ruby team has a common one where the gems present in portage are subscribed — if I remember to add them, that is). It works, sorta. It still requires for you to poll something, although at the end you’re switching the mail reader for the feed reader, so it’s not much of a change. The one problem with that is that you end up receiving a bit more noise than I’d like, as gems that are available in binary form for Windows and Java are listed more than once after an update. But it’s good that it actually is integrated with the gem release procedure.

On the other hand, excluding all the packages that have no hosting facility included at all (which sometimes, such as sudo’s case, have better announcement systems), there are two sites that I count as major screw up, with different degrees: Google Code and Launchpad. The former, is just partly screwed: starring a project does not subscribe to updates, but at least it has a feed of all the files released by the project. What I find definitely strange is that there is no integrated “Subscribe in Google Reader” that would have been definitely more friendly.

Launchpad, instead looks much worse. I recently picked up co-maintainership of a trio of projects and not only there is no email notification, there is no feed of either releases or files! Which means that the only way to find whether a project made a new release is to check the homepage of them. Yuppie. I opened a bug for that on launchpad, but I now lost the link: it was duped against something else, and is no longer visible through launchpad’s own interface, which is, in my book, yet another failure.

Why is it so difficult that packagers need the notifications? This gets even more silly considering that I’m sure the main argument against notification is going to be ”but users won’t have to care, as the package will be available on their distribution”.