On Rewriting And Programming Languages

In my previous post about the life of a maintainer, I noted how more often than not, you move from blazing the path to slumber, until someone decides you are a dinosaur and they try to catch up to with new ideas, design, and standards. One more variables for these challengers is the implementation language — while this is by far not uncommon over the years, there is a significant uptick of this happening recently due to new challenger languages such as Go and Rust, taking spotlight as replacements for C++ (and, to point, for C) and bringing to light a number of projects focused solely on rewriting existing projects that have otherwise very little to add in term of updates.

To be clear, rewriting software in a different language is not a prerogative of FLOSS. Big tech companies do that all the time as well, although the motivations behind those rewrites are slightly different — at least in my experience the number one reason for software being rewritten in Big Tech is that the original authors of a specific piece of software left the team or the company, and the whole understanding of the project and its requirement left with them.

Many of you, who haven’t experienced the particular environment of Big Tech might be surprised how often my reverse engineering skills come useful in my day job. Just because I have the sources, and the source control, does not mean that I don’t need to fit pieces together based on understanding of the interaction with different systems. At least I have never had to resolve to patch a binary directly.

But, is the rewrite of software in a different language in and by itself a worth y project? Personally I do not believe so. Which does not mean that I think it shouldn’t be pursued. Particularly when the rewrite is paired with architectural redesign, simplification, or resolution of long outstanding problems. On the other hand, if the release announcement of a project is only concerned with the new language the software is written in, my professional opinion is that it is a failed project.

Now, let me be clear, this is a simplification. There are usually plenty of reasons to be found in replacing one language with another, if you go looking. The new code might be more testable, performant, or documented. The point is, if even after looking the only thing you can say is «Yeah this is the same tool as we had last week but it’s now written in D!», well…

In the case of Rust, the primary reasons that are presented are somewhat vague “security” features. Rust design does, indeed, remove entire classes of programming mistakes that could lead to security vulnerability. To be fair though, this is the case for most programming languages: Python, Perl, JavaScript, and even C#. So if you’re looking to take a new programming language without knowing what you’re trying to prevent with it, you may want to think about it for a few more moments.

I find more interesting the thread by Asahi Lina sharing their experience in developing a driver (doing tons of buffer management) in Rust. Having had built buffer management for feng once upon a time, I think I would have appreciated having something like Rust available — an idea if I ever find a time machine! So any similar reason would be pretty good reason for a rewrite.

Another good possible reason is to me use of a specific library. Back when Jürgen wrote gmaillabelpurge for me, a tool to delete old emails from specific Gmail labels, he chose to do so in Python — I ended up re-writing it in Perl later to make use of Mail::IMAPClient since that library, unlike the one present in Python’s standard library, exposed all of the features of the extended IMAP protocol Google used. Indeed, this type of standard library is nowadays referred to as a dead battery.

Of course, when rewriting software, no matter whether in the same or different language, there is a risk (or rather, a certainty) of introducing bugs and compatibility issues. This risk can be reduced or at least managed if there is a full behavioural test suite, but I have never heard of a rewrite that worked at first try.

So, personally, what would I rewrite? Well, as I did say a number of years ago, Unpaper would be a good target for a higher level, high-performance language like Rust. I even consider rewriting it in Python at some point, with Cython as wrapping the CPU-bound “unpaper” algorithm, and leveraging Pillow for image I/O. To this day, I do not think there is a good graphics library for Rust, or I would attempt the rewrite to Rust myself, particularly as FFmpeg is proving a terrible choice for still image processing.

Otherwise, I would probably consider looking into the network-facing utilities of a project such as OpenWRT. When I last had OpenWRT-based router, I found that a significant number of their tools are underdocumented and overcomplicated. This may have changed in intervening six years, but the general feeling of it is that they seemed prime targets for a rewrite focused on improving the readability of the code.