Interns in SRE and FLOSS

In addition to the usual disclaimer, that what I’m posting here is my opinions and my opinions only, not those of my employers, teammates, or anyone else, I want to start with an additional disclaimer: I’m neither an intern, a hiring manager, or a business owner. This means that I’m talking from my limited personal experience that might not match someone else’s. I have no definite answers, I just happen to have opinions.

Also, the important acknowledgement: this post comes from a short chat on Twitter with Micah. If you don’t know her, and you’re reading my blog, what are you doing? Go and watcher her videos!

You might remember a long time ago I wrote (complaining) of how people were viewing Google Summer of Code as a way to get cash rather than a way to find and nurture new contributors for the project. As hindsight is 2020 (or at least 2019 soon), I can definitely see how my complaint sounded not just negative, but outright insulting for many. I would probably be more mellow about it nowadays, but from the point of view of an organisation I stand from my original idea.

If anything I have solidified my idea further with the past five and a half years working for a big company with interns around me almost all the time. I even hosted two trainees for the Summer Trainee Engineering Program a few years ago, and I was excitedly impressed with their skill — which admittedly is something they shared with nearly all the interns I’ve ever interacted with.

I have not hosted interns since, but not because of bad experiences. It had more to do with me changing team much more often than the average Google engineer — not always by my request. That’s a topic for another day. Most of the teams I have been in, including now, had at least an intern working for them. For some teams, I’ve been involved in brainstorming to find ideas for interns to work on the next year.

Due to my “team migration”, and the fact that I insist on not moving to the USA, I often end up in those brainstorms with new intern hosts. And because of that I have over time noticed a few trends and patterns.

The one that luckily appears to be actively suppressed by managers and previous hosts is that of thinking of interns as the go-to option to work on tasks that we would define “grungy” — that’s a terrible experience for interns, and it shouldn’t be ever encouraged. Indeed, my first manager made it clear that if you come up with a grungy task to be worked on, what you want is a new hire, not an intern.

Why? There are multiple reasons for that. Start with the limited time an intern has, to complete a project: even if the grungy task is useful to learn how a certain system works, does an intern really need to get comfortable with it that way? For a new hire, instead, time is much less limited, so giving them a bit more boring tasks while they go through whatever other training they need to go through is fine.

But that’s only part of the reason. The _much more important_ part is understanding where the value of an intern is for the organisation. And that is _not_ in their output!

As I said at the start, I’m not a hiring manager and I’m not a business person, but I used to have my own company, and have been working in a big org for long enough that I can tell a few patterns here and there. So for a start, it becomes obvious that an intern’s output (as in the code they write, the services they implement, the designs they write) are not their strongest value proposition, from the organisation point of view: while usually interns are paid less than the full-time engineers, hosting an intern takes _a lot_ of time away from the intern host, which means the _cost_ of the intern is not just how much they get paid, but also a part of what the host get paid (it’s not by chance that Google Summer of Code reimburses the hosting project and not just the student).

Also, given interns need to be trained, and they will likely have less experience in the environment they would be working, it’s usually the case that letting a full-time engineer provide the same output would take significantly less time (and thus, less money).

So no, the output is not the value of an intern. Instead an internship is an opportunity both for the organisation and for the interns themselves. For the organisation, it’s almost like an extended interview: they get to gauge the interns’ abilities over a period of time, and not just with nearly-trick questions that can be learnt by heart — it includes a lot more than just their coding skills, but also their “culture fit” (I don’t like this concept), and their ability to work in a team — and I can tell you that myself, at the age of most of the interns I worked with, I would have been a _terrible_ team player!

And let’s not forget that if the intern is hired afterwards, it’s a streamlined training schedule, since they already know their way around the company.

For the intern, it’s the experience of working in a team, and figuring out if it’s what they want to do. I know of one brilliant intern (who I still miss having around, because they were quite the friendly company to sit behind, as well as a skilled engineer) who decided that Dublin was not for them, after all.

This has another side effect for the hosting teams, that I think really needs to be considered. An internship is a teaching opportunity, so whatever project is provided to an intern should be _meaningful_ to them. It should be realistic, it shouldn’t be just a toy idea. At the same time, there’s usually the intention to have an intern work on something of value for the team. This is great in the general sense, but it goes down to two further problems.

The first is that if you _really_ need something, assigning it as a task to an intern is a big risk: they may not deliver, or underdeliver. If you _need_ something, you should really assign it to an engineer; as I said it would also be cheaper.

The second is that the intern is usually still learning. Their code quality is likely to not be at the level you want your production code to be. And that’s _okay_. Any improvement in the code quality of the intern over their internship is of value for them, so helping them to improve is good… but it might not be the primary target.

Because of that, my usual statement during the brainstorms is “Do you have two weeks to put the finishing polish on your intern’s work, after they are gone?” — because if not, the code is unlikely to be made into production. There are plenty of things that need to be done after a project is “complete” to make it long-lasting, whether they are integration testing and releasing, or “dotting the is and crossing the ts” on the code.

And when you don’t do those things, you end up with “mostly done” code, that feels unowned (because the original author left by that point), and that can’t be easily integrated into production. I have deleted those kind of projects from codebases (not just at Google) too many times already.

So yes, please, if you have a chance, take interns. Mentor them, teach them, show them around on what their opportunities could be. Make sure that they find a connection with the people as well as the code. Make sure that they learn things like “Asking your colleagues when you’re not sure is okay”. But don’t expect that getting an intern to work on something means that they’ll finish off a polished product or service that can be used without a further investment of time. And the same applies to GSoC students.

Ads, spying, and my personal opinion

In the past year or so, I have seen multiple articles, even by authors who I thought would have more rational sense to them, over the impression that people get about being spied upon by technology and technology companies. I never got particularly bothered to talk about them, among other things because the company I work for (Google) is one that is often at the receiving end of those articles, and it would be disingenuous for me to “defend” it, even though I work in Site Realiability, which gives me much less insight in how tracking is done than, say, my friends who work in media at other companies.

But something happened a few weeks ago gave me an insight on one of the possible reasons why people think this, and I thought I would share my opinion on this. Before I start let me make clear that what I’m going to write about is something that is pieced together with public information only. As you’ll see soon, the commentary is not even involving my company’s products, and because of that I had access to no private information whatsoever.

As I said in other previous posts, I have had one huge change in my personal life over the past few months: I’m in a committed relationship. This means that there’s one other person beside me that spends time in the apartment, using the same WiFi. This is going to be an important consideration as we move on later.

Some weeks ago, my girlfriend commented on a recent tourism advertisement campaign by Lithuania (her country) on Facebook. A few hours later, I received that very advertisement on my stream. Was Facebook spying on us? Did they figure out that we have been talking a lot more together and thus thought that I should visit her country?

I didn’t overthink it too much because I know it can be an absolute coincidence.

Then a few weeks later, we were sitting on the sofa watching Hanayamata on Crunchyroll. I took a bathroom break between episodes (because Cruncyroll’s binge mode doesn’t work on Chromecast), and as I came back she showed me that Instagram started showing her Crunchyroll ads — “Why?!” We were using my phone to watch the anime, as I have the account. She’s not particularly into anime, this was almost a first as the material interested her. So why the ads?

I had to think a moment to give her an answer. I had to make a hypothesis because obviously I don’t have access to either Crunchyroll or Instagram ads tracking, but I think I’m likely to have hit close to the bullseye and when I realized what I was thinking of, I considered the implications with the previous Facebook ads, and the whole lot of articles about spying.

One more important aspect that I have not revealed yet, is that I requested my ISP to give me a static, public IPv4 address instead of the default CGNAT one. I fell for the wet dream, despite not really having used the feature since. It’s handy, don’t get me wrong, if I was to use it. But the truth is that I probably could have not done so and I wouldn’t have noticed a difference.

Except for the ads of course. Because here’s how I can imagine these two cases to have happened.

My girlfriend reads Lithuanian news from her phone, which is connected to my WiFi when she’s here. And we both use Facebook on the same network. It’s not terribly far-fetched to expect that some of the trackers on the Lithuanian news sites she visits are causing the apartment’s stable, static, public IP address to be added to a list of people possibly interested in the country.

Similarly, when we were watching Crunchyroll, we were doing so from the same IP address she was checking Instagram. Connect the two dots and now you have the reason why Instagram thought she’d be a good candidate for seeing an advert for Crunchyroll. Which honestly would make more sense if they intended to exclude those who do have an account, in which case I would not have them trying to convince me to… give them the money I already give them.

Why do I expect this to be IP tracking? Because it’s the only thing that makes sense. We haven’t used Facebook or Messenger to chat in months, so they can’t get signal from that. She does not have the Assistant turned on on her phone, and while I do, I’m reasonably sure that even if it was used for advertisement (and as far as I know, it isn’t), it would not be for Facebook and Instagram.

IP-based tracking is the oldest trick in the book. I would argue that it’s the first tracking that was done, and probably one of the least effective. But at the same time it’s mostly a passive tracking system, which means it’s much easier to accomplish under the current limits and regulations, including but not limited to GDPR.

This obviously has side effects that are even more annoying. If the advertisers start to target IP address indiscriminately, it would be impossible for me or my girlfriend to search for surprises for each other. Just to be on the safe side, I ordered flowers for our half-year anniversary from the office, in the off-chance that the site would put me on a targeting list for flower ads and she could guess about it.

This is probably a lot less effective for people who have not set up static IP addresses, since there should be a daily or so rotation of IP addresses that confuses the tracking enough. But I can definitely see how this can also go very wrong when a household dynamic are pathological, if the previous holder of the address managed to get the IP on targeted lists for unexpected announces.

I have to say that in these cases I do prefer when ads are at least correctly targeted. You can check your Ads preferences for Google and Facebook if you want to actually figure out if they know anything about you that you don’t want them to. I have yet to find out how to stop the dozens of “{Buzzword} {Category} Crowdfunding Videos” pages that keep spamming me on Facebook though.

Software systems and institutional xenophobia

I don’t usually write about politics, because there are people with more sophisticated opinions and knowledge out there, compared to me, playing at the easiest level, to quote John Scalzi, and rarely having to fear for my future (except for when it comes to health problems). But today I need to point out something that worries me a lot.

We live in a society that, for good or bad (and I think it’s mostly for good), is more and more tied to computer systems. This makes it very easy for computer experts of one kind or another (like me!) to find a job, particularly a good paying job. But at the same time it should give us responsibilities for what we do with our jobs.

I complained on Twitter how most of the credit card application forms here in the UK are effectively saying “F**k you, immigrant scum” by not allowing you to complete the application process if you have less than three years’ addresses in the UK. In the case of a form I tried today, even though the form allows you to specify an “Overseas address” as previous address, which allows you to select Ireland as a country, it still verifies the provided post code to UK standards, and refuses you to continue the process without it.

This is not the first such form. Indeed, I ended up getting an American Express credit card because they were the only financial institution that could be convinced to take me on as a customer, with just two months living in this country, and a full history of addresses for the previous five years and more. And even for them, it was a bit of an issue to find an online form that did indeed allow me to type that in.

Yet another of the credit card companies rejected my request because “[my] file is too thin” — despite being able to prove to them I’m currently employed full time with a very well paying company, and not expecting to change any time soon. This is nearly as bad as the NatWest employee that wanted my employer’s HR representative to tell them how long they expected me to live in the UK.

But it’s not just financial institutions, it’s just at any place where you provide information, and you may end up putting up limitations that, though obviously fine for your information might not be for someone else. Sign-up forms where putting a space in a name or surname field is an error. Data processing that expects all names to only have 7-bit ASCII encoding. Electoral registries where names are read either as Latin 1 or Latin 2.

All of these might be considered smaller data issues of nearsighted developers, but they also show how these can easily turn into real discrimination.

When systems that have no reason to discard your request on the basis of the previous address have a mistake that causes the postcode validation to trigger on the wrong format, you’re causing a disservice and possible harm to someone who might really just need a credit card to be able to travel safely.

When you force people to discard part of their name, you’re going to cause them disservice and harm when they will need a full history of what they did — I had that problem in Ireland, applying for a driving learner permit, not realising that the bills for Bord Gáis Energy wrote down my name wrong (using Elio as my surname).

The fact that my council appears to think that they need to use Latin-2 to encode names, suggests they may expect that their residents are all either English or Eastern European, which in turn leads to the idea of some level of segregation of them away from Italian, French or Irish, all of which depend on Latin-1 encodings instead.

The “funnies” in Ireland was a certain bank allowing you to sign up online with no problems… as long as you had a PPS (tax ID) issued before 2013 — after that year, a new format for the number was in use, and their website didn’t consider it valid. Of course, it’s effectively only immigrants who, in 2014, would be trying to open a bank account with such codes.

Could all of these situation be considered problems with incompetence? Possibly yes. Lots of people are incompetents, in our field. But it also means that there was no coverage for these not-so-corner cases in the validation. So it’s not just an incompetent programmer, it’s an incompetent programmer paired with an incompetent QA engineer. And an incompetent product manager. And an incompetent UX designer… that’s a lot of incompetence put together for a product.

Or the alternative is that there is a level of institutional xenophobia when it comes to software development. In the UK just as in Ireland, Italy and in the United States. The idea that the only information that are being tested are those that are explicitly known to the person doing the development is so minimalist as to be useless. You may as well not validate anything.

Not having anyone from the stakeholders to the developers and testers consider “Should a person from a different culture with different naming, addressing, or {whatever else} norms be able to use this?” (or worse, consider it and answering themselves “no”), is something I consider xenophobia¹.

I keep hearing calls to pledge ethics in the field of machine learning (“AI”) and data collection. But I have a feeling that those fields have much less impact on the “median” part of the population. Which is not to say you shouldn’t have ethical consideration in them at all. But rather than we should start with teaching ethics in everyday’s data processing too.

And if you’re looking for some harsh laugh after this mood-killing post, I recommend this article from The Register.

Yes I’m explicitly not using the word “racism” here, because then people will focus on that, rather than the problem. A form does not look at the colour of your skin, but does look at whether you comply with its creators idea of what’s “right”. ↩

Two words about my personal policy on GitHub

I was not planning on posting on the blog until next week, trying to stick on a weekly schedule, but today’s announcement of Microsoft acquiring GitHub is forcing my hand a bit.

So, Microsoft is acquiring GitHub, and a number of Open Source developers are losing their mind, in all possible ways. A significant proportion of comments on this that I have seen on my social media is sounding doomsday, as if this spells the end of GitHub, because Microsoft is going to ruin it all for them.

Myself, I think that if it spells the end of anything, is the end of the one-stop-shop to work on any project out there, not because of anything Microsoft did or is going to do, but because a number of developers are now leaving the platform in protest (protest of what? One company buying another?)

Most likely, it’ll be the fundamentalists that will drop their projects away to GitHub. And depending on what they decide to do with their projects, it might even not show on anybody’s radar. A lot of people are pushing for GitLab, which is both an open-core self-hosted platform, and a PaaS offering.

That is not bad. Self-hosted GitLab instances already exist for VideoLAN and GNOME. Big, strong communities are in my opinion in the perfect position to dedicate people to support core infrastructure to make open source software development easier. In particular because it’s easier for a community of dozens, if not hundreds of people, to find dedicated people to work on it. For one-person projects, that’s overhead, distracting, and destructive as well, as fragmenting into micro-instances will cause pain to fork projects — and at the same time, allowing any user who just registered to fork the code in any instance is prone to abuse and a recipe for disaster…

But this is all going to be a topic for another time. Let me try to go back to my personal opinions on the matter (to be perfectly clear that these are not the opinions of my employer and yadda yadda).

As of today, what we know is that Microsoft acquired GitHub, and they are putting Nat Friedman of Xamarin fame (the company that stood behind the Mono project after Novell) in charge of it. This choice makes me particularly optimistic about the future, because Nat’s a good guy and I have the utmost respect for him.

This means I have no intention to move any of my public repositories away from GitHub, except if doing so would bring a substantial advantage. For instance, if there was a strong community built around medical devices software, I would consider moving glucometerutils. But this is not the case right now.

And because I still root most of my projects around my own domain, if I did move that, the canonical URL would still be valid. This is a scheme I devised after getting tired of fixing up where unieject ended up with.

Microsoft has not done anything wrong with GitHub yet. I will give them the benefit of the doubt, and not rush out of the door. It would and will be different if they were to change their policies.

So here's a more serious discussion of what 'GitHub' is, and why Microsoft's acquisition of it is so important: Chi… twitter.com/i/web/status/1…
—
Robᵉʳᵗ Graham 🤔 (@ErrataRob) June 03, 2018

Rob’s point is valid, and it would be a disgrace if various governments would push Microsoft to a corner requiring it to purge content that the smaller, independent GitHub would have left alone. But unless that happens, we’re debating hypothetical at the same level of “If I was elected supreme leader of Italy”.

So, as of today, 2018-06-04, I have no intention of moving any of my repositories to other services. I’ll also use a link to this blog with no accompanying comment to anyone who will suggest I should do so without any benefit for my projects.

The importance of teams, and teamwork

Today, on Twitter, I have received a reply with a phrase that, in its own sake and without connecting back with the original topic of the thread, I found significant of the dread I feel with working as a developer, particularly in many opensource communities nowadays.

Most things don’t work the way I think they work. That’s why I’m a programmer, so I can make them work the way I think they should work.

I’m not going to link back to the tweet, or name the author of the phrase. This is not about them in particular, and more about the feeling expressed in this phrase, which I would have agreed with many years ago, but now feels so much off key.

What I feel now is that programmers don’t make things work the way they think they should. And this is not intended as a nod to the various jokes about how bad programming actually is, given APIs and constraints. This is about something that becomes clear when you spend your time trying to change the world, or make a living alone (by running your own company): everybody needs help, in the form of a team.

A lone programmer may be able to write a whole operating system (cough Emacs), but that does not make it a success in and by itself. If you plan on changing the world, and possibly changing it for the better, you need a team that includes not only programmers, but experts in quite a lot of different things.

Whether it is a Free Software project, or a commercial product, if you want to have users, you need to know what they want — and a programmer is not always the most suitable person to go through user stories. Hands up all of us who have, at one point or another, facepalmed at an acquaintance taking a screenshot of a web page to paste it into Word, and tried to teach them how to print the page to PDF. While changing workflows so that they make sense may sound the easiest solution to most tech people, that’s not what people who are trying to just do their job care about. Particularly not if you’re trying to sell them (literally or figuratively) a new product.

And similarly to what users want to do, you need to know what the users need to do. While effectively all of Free Software comes with no warranty attached, even for it (and most definitely for commercial products), it’s important to consider the legal framework the software has to be used on. Except for the more anarchists of the developers out there, I don’t think anyone would feel particularly interested in breaching laws for the sake of breaching them, for instance by providing a ledger product that allows “black book accounting” as an encrypted parallel file. Or, to reprise my recent example, to provide a software solution that does not comply with GDPR.

This is not just about pure software products. You may remember, from last year, the teardown of Juicero. In this case the problems appeared to step by the lack of control over the BOM. While electronics is by far not my speciality, I have heard more expert friends and colleagues cringe at seeing the spec of projects that tried to actually become mainstream, with a BOM easily twice as expensive as the minimum.

Aside here, before someone starts shouting about that. Minimising the BOM for an electronic project may not always be the main target. If it’s a DIY project, making it easier to assemble could be an objective, so choosing more bulky, more expensive parts might be warranted. Similarly if it’s being done for prototyping, using more expensive but widely available components is generally a win too. I have worked on devices that used multi-GB SSDs for a firmware less than 64MB — but asking for on-board flash for the firmware would have costed more than the extremely overprovisioned SSDs.

And in my opinion, if you want to have your own company, and are in for the long run (i.e. not with startup mentality of getting VC capital and get acquired before even shipping), you definitely need someone to follow up the business plan and the accounting.

So no, I don’t think that any one programmer, or a group of sole programmers, can change the world. There’s a lot more than writing code, to build software. And a lot more than building software, to change society.

Consider this the reason why I will plonk-file any recruitment email that is looking for “rockstars” or “ninjas”. Not that I’m looking for a new gig as I type this, but I would at least give thought if someone was looking for a software mechanic (h/t @sysadmin1138).

Europe and USA: my personal market comparison

While I have already announced I’m moving to London, I don’t want to give the idea that I don’t trust Europe. One of my acquaintances, an eurosceptic, thought it was apt o congratulate me for dropping out of Europe when I announced my move, but that couldn’t be farthest from my intention. As I said already repeatedly now, my decision is all to be found in my lack of a social circle in Dublin, and the feelings of loneliness that really need to be taken care of.

Indeed, I’m more than an Europeist, I’m a Globalist, in so far as I don’t see any reason why we should have borders, or limitations on travel. So my hope is not just for Europe to become a bigger, more common block. Having run a business for a number of years in Italy, where business rules are overly complicated, and the tax system assumes that you’re a cheater by default, and fines you if you don’t invoice enough, I would have seriously loved the option to have an “European business” rather than an “Italian business” — since a good chunk of my customers were based outside of Italy anyway.

This concept of “European business”, unfortunately, does not exist. Even VAT handling in Europe is not unified, and even though we should have at least a common VAT ID registration, back when I set up my business, it required an explicit registration at the Finance Ministry to be able to make use of the ID outside of Italy. At the time, at least, I think Spain also opted out to registering their VAT IDs on the European system by default. Indeed that was the reason why Amazon used to the run separate processes for most European business customers, and for Italian and Spanish customers.

Speaking of Amazon, those of you reading me from outside Europe may be surprised to know that there is no such thing as “Amazon Europe”, – heck, we don’t even have Amazon Ireland! – at least as a consumer website. Each country has its own Amazon website, with similar, but not identical listings, prices and “rules of engagement” (what can be shipped where and for which prices). For the customers this has quite a few detrimental effects: the prices may be lower in a country that they may not usually look at the store of, or they may have to weight the options based on price, shipping restrictions and shipping costs.

Since, as I said, there is no Amazon Ireland, living in Dublin also is an interesting exercise with Amazon: you may want to order things from Amazon UK, either because of language reasons, or simply because it requires a power plug and Ireland has the same British plug as the UK. And most of the shipping costs are lower, either by themselves, or because there are re-mailers from Northern Ireland to Dublin, if you are okay with waiting an extra day. But at the same time, you’re forced to pay in GBP rather than Euro (well, okay not forced, but at least strongly advised to — Amazon currency conversion has a significantly worse exchange rate than any of my cards, especially Revolut) and some of the sellers will actually refuse to send to Ireland, for no specific reason. Sometimes, you can actually buy the same device from Amazon Germany, which will then ship from a UK-based storehouse anyway, despite the item not being available to send to Ireland from Amazon UK. And sometimes Amazon Italy may be a good 15% cheaper (on a multiple-hundreds euro item) than Amazon UK.

So why does Amazon not run a global European website? Or why doesn’t an European-native alternative appears? It looks to me like the European Union and its various bodies and people keep hoping to find European-native alternatives to the big American names all the times, at least on the papers, probably in the hope of not being tied to the destiny of American with what comes down in the future, particularly given how things have gone with the current politics on all sides. But in all their words, there does not appear to be any option of opening up opportunities for creating cross-Europe collaboration on corporations.

The current situation of the countries that make up Europe and the States that make up the USA, is that you are just not allowed to do certain types, or levels of business in all the countries without registering and operating as a company in that country. That is the case for instance of phone operators, that get licenses per country, and so each operates independent units. This becomes sometimes ludicrous because you then have Vodafone providing services in about half of Europe, but with such independent units that their level of competence for instance on security and privacy is extremely variable. In particular it looks like Vodafone Italy still has not learnt how to set up HTTPS correctly, and despite logging you in a TLS-encrypted connection, it does not set the cookie as secure, so a downgrade is enough to steal authentication cookies. In 2017.

If you remember, when I complained about the half-baked roaming directive results, I have suggested that one of my favourite options would be to have a “European number” — just give me a special “country code” that can be replaced by any one member’s code, and the phone number is still valid, and appears local. This is important because, despite the roaming directive allowing me to keep my regular Irish (for now) SIM card on my phone while travelling to either UK or Finland, it prevents me from getting a local phone number. And since signing up for some local services, including sometimes free WiFi hotspots from various cafes and establishment, relies on being able to receive a local SMS, it is sometimes more of an hindrance than a favour.

Both Revolut and Transferwise, as well as other similar “FinTech” companies have started providing users with what they call “borderless” accounts: Euro, Sterling and Dollar accounts all into one system. Unfortunately this is only half of the battle. Indeed, while I welcome in particular Revolut’s option of using a single balance that can provide all the currencies in a single card is a great option. But this only works to a point, because these accounts are “special” — in particular the Revolut Euro account is provided with a Lithuanian IBAN, but a UK BIC code, which makes a few system that still expect both throw up. And this is not even going into how SEPA Direct Debit just does not work: my Italian services can only debit an Italian bank, my Irish services can only charge an Irish bank, and my one French service can only charge a French bank. Using credit cards via VISA has actually better success rate for me, even though at least Vodafone Italy can only charge a specific one of my credit cards, rather than any of them. Oh yeah and let’s not forget the fact that you just can’t get your salary paid into a non-Irish bank account in Ireland.

Banks in Europe end up operating as country-wide silos, to the point that even Ulster Bank Republic of Ireland cannot (at least, can no longer) provide me with an Ulster Bank Northern Ireland bank account — or to be precise, cannot act on my already-existing foreigner bank account that is open in Northern Ireland. And because of all these things happening, the moment I will actually move to London I’ll have to figure out how to get a proper account there. I’m having trouble right now opening an account there already not because I don’t have the local tax ID but because they need proof of employment from a UK company, while I’m still employed by the Irish company. Of the same multinational. Oh my.

You could say that banks and telcos are special cases. They are partial monopolies and there are good reasons why they should be administered on a country-by-country basis. But the reality is that in the United States, these things are mostly solved — plenty of telco stuff is still pretty much local, but that’s because of network access and antitrust requirements, as well, to a point, the need of building and servicing local infrastructure (a solution to this is effectively splitting the operation of the telco from the provider of physical infrastructure, but that comes with its own problems). But at the very least, banking in the US is not something that people have to deal with when changing State, or having to work with companies of other states.

These silos are also visible to consumers in other forms, that may not be quite obvious. TV, movie and similar rights are also a problem the same way. Netflix for instance will only show a subset of the programming they have access to depending on the country you’re currently located in. This is because, except for the content they produce themselves, they have to acquire rights from different companies holding them in different countries, because different TV networks would already have secured rights and not want to let them broadcast in their place.

I brought up this part last, despite being probably the one most consumers know or even care about, because it shows the other problem that companies trying to build up support for Europe, or even to be started as Europe-native companies, have to deal with. TV networks are significantly more fragmented than in the USA. There is no HBO, despite Sky being present in a number of different countries. There is nothing akin to CNN. There are a number of 24-hours news channels that are reachable over more-or-less freeview means, but the truth is that if you want to watch TV in Europe, you need a local company to provide you with it. And the reason is not one that is easy to solve: different countries just speak different languages, sometimes more than once.

It’s not just a matter of providing a second channel in a different language: content needs to be translated, sometimes adapted. This is very clear in video games, where some countries (cough Germany cough) require cutting content explicitly, to avoid upsetting something or someone. Indeed, video games releases for many platforms, in the past at least including PC, but luckily it appears not the case nowadays, end up distributing games only in a subset of European languages at a time. Which is why I loathed playing Skyrim on the PlayStation 3, as the disk only includes Italian, French and German, but no English, which would be my default option (okay, nowadays I would probably play it in French to freshen up my understanding of it).

For American start-ups – but this is true also for open source project, and authors of media such as books, TV series or movies – internationalization or localization are problems that can be easily shelved for the “after we’re famous” pile. First make the fame, or the money, then export and care about other languages. In Europe that cannot possibly be the case. Even for English, that in the computer world is still for now the lingua franca (pun intended), I wouldn’t expect there would be a majority of users happy to use a non-localized software, particularly when you consider as part of that localization the differences in date handling. I mean, I started using “English (UK)” rather than the default American for my Google account years ago because I wanted a sane date format in Gmail!

All of this makes the fragmented European market harder for most projects, companies, and even ideas to move as fast as the American or (I presume, but have not enough detail about it) the Chinese market, in which a much wider audience can be gained without spending so much effort to deal with cross-border bureaucracy and cross-culture porting. But let me be clear, I do not think that the solution is to normalize Europe onto a single language. We can’t even do that for countries, and I don’t think it would be fair to anyone to even consider this. What we need is to remove as many other roadblocks as it’s feasible to remove, and then try to come up with an easier way to fund translation and localization processes, or an easier way to access rights at a Union level rather than on a country-by-country basis.

Unfortunately, I do not expect that this is going to happen in my lifetime. I still wish we’ll end up with a United Federation of Planets, at the end of the day, though.

In defence of ads over subscriptions

This is a third draft trying to focus my opinion on one facet of a very complicated, multi-faceted discussion about Internet, content and monetization. Hopefully, I’ll be able to keep my mind focused enough on the problem I want to talk about to come out with a concise and interesting post.

You may have read my opinion on Internet ads and the more recent commentary on my own “monetization”. If you haven’t, the short version of the story is that I do not think that ads are implicitly evil, but at the same time I don’t think that they are worth the attrition for most sites, particularly if very small. My blog, and Autotools Mythbuster, are not exactly unknown sites, but they are still small enough that dealing with Ads is barely worth it for most people, and clearly not worth it for me.

For big, global and media sites, ads are still (for the most part) the way to get money, and to keep running. And that is true whether those sites started as “blogosphere” Internet-native sites, like Ars Technica, or as online versions of establishment news papers like The New York Times. And particularly as you go big, and you want to have curated content, secure content and (extremely important here!) good and verified content, you have costs that add up and that require some level of income to maintain.

You can see that with LWN.net, whose content is very well curated, and is for the most part paid contributions — I can tell you that the folks over there, starting from Jonathan, would not be able to afford such a great edited content if they had no income on it. It takes time to do so, and you can’t run it as a side project or moonlighting job. Or for a more mainstream example, take what’s happening with Snopes.

One of a common talking point I hear from many companies that are involved in advertising on the Internet, including the company I work for, is that we “enable free content publishing on a global scale”. It may sounds as a platitude written this way, and sound hollow. But once you think it twice or thrice over, it starts to ring true. Because all that content would not be able to pay for the wages of the people involved in charitable donations only. Ads do really help with that.

But then I hear the extremists that decide that all ads are bad, install uBlock, and refuse to even consider that the vast majority of both sites and advertisers are trying to make a honest living, and they are neither clickfarms nor malvertisers, even though these two problems are real. And I’m not fond of SEO tactics either, but that’s also something that is not evil by itself. What is the solution, according to some of those people I engaged with at many times? For some of them the answer is to just not expect money for your online content. This works fine for me, nowadays, but it wouldn’t have worked for me before. Also, the people I heard suggesting this before are usually people who have not tried producing high-quality content before. As I wrote years ago the amount of effort spent writing a blog post is minuscule compared to that needed for me to write a simple article for LWN. And indeed, I got paid for my effort for those — $100 sounds like a lot for writing, but turns out it’s well below minimum wage by the amount of time involved.

The alternative is of course saying that the answer is to sell ad-free subscriptions instead. Which is cool by me in general, but that can only work if those who can’t afford the subscriptions are indeed not using adblockers for everything. But that is clearly not the case, and it can’t be the case as long as malvertising is indeed a risk for most people. And my reason to have this opinion is that by relying on a subscription-only model, you end up either significantly limiting the gamut of content that is available on the Internet, or its availability to the public.

We have for instance all cursed at the bother of paywalls when trying to get to content that may be interesting, or even vital, to us. But the paywall is probably the only way to make sure that the content is paid for, with a subscription. This way you don’t risk the “cheaters” that will not subscribe but also won’t offer their attention so that advertisers can fill in for those subscriptions. Paywalls are nasty and I would not advocate for them. They limit access to information, dividing people into those who can afford to be informed, and those who can’t.

There is the other option of this of course, which is the “tip-jar” option: you provide free access to the information, with no advertisement, but you have a public support campaign that suggests people to provide money for the content, either by direct subscription or by providing fundraising campaigns, such as Patreon. The problem with this system is that it’s skewed towards the content that is of interests to the people with more disposable income. It’s the same problem as the prototypes coming in on Kickstarter and similar sites: since I’m a well-paid tech employee, I can afford to finance hobby projects and content that interests me, which means the creator will try to pander to my taste by providing projects and content that appeal to well paid tech employees. And then there is the occasional situation in which those tech employees grow a heart, a passion, or more properly just want to be some sane, decent, human beings and decide to pay for content, projects, services or charities that they have no stakes in.

But if you write content that is targeted to a group of people with no disposable income, you probably would feel better if you knew that honest companies with an advertising budget underpin your wages, rather than feeling at the mercy of the magnanimity of some people who are not your target audience, and may one day just disagree with you, your message or your method, and just cut your funds. Although of course, this is also a double-edged sword, as we’ve seen that hateful content can be cut from ads revenue but persist through the deep pocket of the people wanting the message to stay out.

Where does this leave us? I’m not sure. I just think that the subscription model is not the nice good solution that I hear some people suggest. I also find it quite ironic that a number of people who suggested this option – and told me explicitly that they think content should either be able to make people pay for their content or not get money out of it – are further to the left of me, and appear disgusted at the suggestion that the free market is here to stay. Well, guess what they just described for content? And yeah it’s not perfect, far from it.

Honestly, I really would like for super-partes agencies to be able to distribute funds, fairly gathered, into diverse content creators. But that is very hard to accomplish. The obvious answer would be to let the government collect taxes and redistribute it in funds for the art. But we also know that providing the government with the task of redistributing money for content leads to the government publishing only content that looks favourable to those in charge. And I’m not really sure there is much of a better option nowadays.

I should probably write another post talking about funding sources like the old Flattr and the Brave browser, but that’s a topic for another time.

Fake “Free” Software hurts us all

Brian Krebs, the famous information security reporter, posted today (well, at the time of writing) an article on security issues with gSOAP library. I found this particularly interesting to me because I remembered seeing the name before. Indeed, Krebs links to the project’s SourceForge page, which is something to see. It has a multi-screen long list of “features”, including a significant number of variant and options for the protocol, which ends in the following:

Licenses: GPLv2, gSOAP public license (for engine and plugins), commercial non-GPL license available upon request (software is 100% in-house developed, no third-party GPL contributions included)

Ah, there we go. “Free” Software.

You may remember my post just a few days ago about the fact that Free Software to actually make a difference in society the way Stallman prophesizes needs a mediating agency, and at least right now that agency is companies and the free market. I argued that making your software usable by companies that provide services or devices is good, as it makes for healthy, usable and used projects, and increase competition and reduce costs of developing new solutions. So is gSOAP the counterexample? I really don’t think so. gSOAP is the perfect example for me that matches my previous rant about startups and fake communities.

The project at first look like the poster child for FSF-compatible software, since it’s licensed under GPL-2, and it clearly has no CLA (Contributor License Agreement), though the company provides a “way out” from GPL-2 obligations by buying a commercial license. This is not, generally speaking, a bad thing. I have seen many examples, including in the open multimedia cliques, of using this trick to foster the development of open solutions while making money to pay salary or build new Free Software companies.

But generally, those projects allow third party contributions with a CLA or similar agreement that allows the “main” copyright holders to still issue proprietary licenses, or enforce their own rights. You can for instance see what Google’s open source projects do about it. Among other things, this contribution method also disallows re-licensing the software, as that requires agreement from all copyright holders. In the case of gSOAP, that is not the case: as they say their software is «100% in-house developed».

They are very well proud of this situation, because it gives them all the power: if you want to use gSOAP without paying, you’re tied to the GPL, which may or may not become a compliance problem. And if you happen to violate the license, they have all the copyright to sue you or just ask you to settle. It’s a perfect situation for copyright trolls.

But, because of this, even though the software is on paper “Free Software” according to FSF, it’s a piece of proprietary software. Sure you can fork the library and build your own GPL-2 instead, as you have the freedom of fork, but that does not make it a community, or a real Free Software project. And it also means you can’t contribute patches to it to make it more secure, safer, or better for society. You could report bugs, including security bugs, but what’s the likeliness that you would actually care to do so, given that one of the first thing they make clear on their “project” page is that they are not interested in your contributions? And we clearly can see that the particular library could have used some care from the community, given its widespread adoption.

What does this mean to me, is that gSOAP is a clear example that just releasing something under GPL-2 is not enough to make it Free Software, and that even “Free” Software released under GPL-2 can be detrimental to society. And it also touches on the other topic I brought up recently, that is that you need to strike a balance between making code usable to companies (because they will use, and thus very likely help you extend or support your project) and keeping it as a real community or a real project. Clearly in this case the balance was totally off. If gSOAP was available with a more liberal license, even LGPL-2, they would probably lose a lot in license fees, as for most companies, just using this as a shared library would be enough. But it would then allow developers, both hobbyists and working for companies, to contribute fixes so that they trickle down on everybody’s device.

Since I do not know what the proprietary license that the company behind gSOAP requires their customers to agree with, I cannot say whether there is any incentive in said companies to provide fixes back to the project, but I assume if they were to do so, they wouldn’t be contributing them under GPL-2, clearly. What I can say is that for the companies I worked for in the past, actually receiving the library under GPL-2 and being able to contribute the fixes back would have been a better deal. The main reason is that as much as a library like this can be critical to connected devices, it does not by itself contain any of the business logic. And there are ways around linking GPL-2 code into the business logic application, that usually involve some kind of RPC between the logic and the frontend. And being able to contribute the changes back to the open source project would allow them to not have to maintain a long set of patches to sync release after release — I had the unfortunate experience of having to deal with something in that space before.

My advice, is once again, to try figuring out what you want to achieve by releasing a piece of software. Are you trying to make a statement, sticking it to the man, by forcing companies not to use your software? Are you trying to make money by forcing companies interested in using your work to buy a special license from you? Or are you contributing to Free Software because you want to improve society? In the latter case, I would suggest you consider a liberal license, to make sure that your code (that can be better than proprietary, closed-source software!) is actually available for those mediating agencies that transform geeky code into usable gadgets and services.

I know, it would be oh so nicer, if by just releasing something awesome under GPL-2 you’d force every company to release all their firmware as Free Software as well, but that’s not going to happen. Instead, if they feel they have to, they will look for worse alternatives, or build their own (even worse) alternatives, and keep them to themselves, and we’ll all be the poorer for it. So if in doubt, consider MIT or Apache 2 licenses. The latter in particular appears to be gaining more and more traction as both Google and Microsoft appear to be fond of the license, and Facebook’s alternative is tricky.

Some of you may consider me a bit of a hypocrite since I have released a number of projects under more restrictive Free licenses (including AGPL-3!) before I came to the conclusion that’s actually bad for society. Or at least before I came back to that idea, as I was convinced of that back in 2004, when I wrote (In Italian) of why I think MySQL is bad for Free Software (I should probably translate that post, just for reference). But what I decided is that I’ll try now to do my best to re-license the projects for which I own the full copyright under Apache 2 licenses. This may take a little while until I figure out all the right steps, but I feel is the right thing to do.

Who reaps the benefits of Free Software?

I feel silly to have to start this post by having to boast about my own achievements, but my previous post has stirred up a lot of comments (outside my blog’s own, that is) and a number of those could be summed up with “I don’t know this guy, he’s of course new, has no idea what he’s talking about”. So let’s start with the fact that I’ve been involved in Free Software for just about half of my life at this point.

And while I have not tied my name to any particular project, I have contributed to a significant amount of them by now. I’m not an “ideas man” so you can count on me to help figuring out problems and fixing bugs, but I would probably start hiding in a corner if I built up a cult of personality; unfortunately that appears to be what many other contributors to Free Software have done over the years, and this brings weight to their own name. I don’t have such weight, but you probably better off googling my name around before thinking I have no stake in the fire.

Introduction over, let’s get to the meat of this post.

If you’re reading this post, it is likely that you’re a Free Software supporter, creator, user, or a combination of these. Even those people that I know fiercely criticized Free Software can’t say that they don’t use any nowadays: at the very least the two major mobile operating systems have a number of Free Software components they rely upon, which means they are all users of Free Software, whether they want it or not. Which means I have no reason to try to convince you that Free Software is good. But good for who?

RMS notoriously titled his essay anthology Free Software, Free Society, which implies that Free Software is good for everybody — I can agree to a point, but at the same time I don’t think it’s as clear cut as he wants to make it sound.

I wrote about this before, but I want to write it down again. For the majority of people, Free Software is not important. You can argue that we should make it clear to them that it is important, but that sounds like a dogma, and I really don’t like dogmas.

Free Software supporters and users, for the most part, are geeks who are able to make use of available source code of software to do… something about it. Sometimes it’s not a matter of making changes to the code, because even making changes is not enough, but for instance you can reverse engineer an undocumented protocol in such a way that can be reimplemented properly.

But what about my brother in law? He can’t code to save his life, and he has no interest in reverse engineering protocols. How is Free Software important to him and benefiting him? Truthfully, the answer is “not at all, directly”. He’s a Free Software user, because he has an Android phone and an iPad, but neither are entirely Free Software, and even all the more “libre” projects are making his life any easier. I’ll go back to that later.

In the post I already referenced, I pointed out how the availability of open-source online diabetes management software would make possible for doctors to set up their own instances of these apps to give their patients access. But that only works in theory — in practice, no doctor would be able to set this up by themselves safely, and data protection laws would likely require them to hire an external contractor to set it up and maintain it. And in that case, what would be the difference between that and hiring a company that developed their own closed-source application, and maybe provide it as a cloud service?

Here’s the somewhat harsh truth: for most people who are not into IT and geekery, there is no direct advantage to Free Software, but there are multiple indirect ones, almost all of which rotate around one “simple” concept: Free Software makes Free Market. And oh my, is this term loaded, and ready to explode a flame just by me using it. Particularly with a significant amount of Free Software activists nowadays being pretty angry with capitalism as a concept, and that’s without going into the geek supremacists ready to poison the well for their own ego.

When Free Software is released, it empowers companies, freelancer, and people alike to re-use and improve it – I will not go into the discussion of which license allows what; I’ll just hand-wave this problem for now – which increase competition, which is generally good. Re-using the example of online diabetes management software, thanks to open-source it’s no longer only the handful of big companies that spent decades working on diabetes that can provide software to the doctors, but any other company that wants to… that is if they have the means to comply with data protection laws and similar hurdles.

Home routers are another area in which Free Software has clearly left a mark. From the WRT54G, which was effectively the easiest hackable router of its time, we made significant progress with both OpenWRT and LEDE, to the point that using a “pure” (to a point) Free Software home router is not only possible but feasible, and there even are routers that you can buy with Free Software firmware installed by default. But even here you can notice an important distinction: you have to buy the router with the firmware you want. You can’t just build it yourself for obvious reasons.

And this is the important part for me. There is this geek myth that everyone should be a “maker” — I don’t agree with this, because not everyone wants to be one, so they should not be required to become one. I am totally sold that everybody should have the chance and access to information to become one if they so want, and that’s why I also advocate reverse engineering and documenting whatever is not already freely accessible.

But for Free Software to be consumable by users, to improve society, you need a mediating agency, and that agency lies in the companies that provide valuable services to users. And with “valuable services” I do not mean solely services aimed at the elites, or even just at that part of the population living in big metropolises like SF or NYC. Not all the “ubers of” companies that try to provide services with which you can interact online or through apps are soulless entities. Not all the people wanting to use these services are privileged (though I am).

And let me be clear, that I don’t mean that Free Software has to be subject to companies and startups to make sense. I have indeed already complained about startups camouflaged by communities. In a healthy environment, the fact that some Free Software project is suitable to make a startup thrive is not the same as the startup needing Free Software contributions to be alive. The distinction is hard to put down on paper, but I think most of you have seen how that turns out for projects like ownCloud.

Since I have already complained about anti-corporatism in Free Software years ago, well before joining the current “big corporation” I work for, why am I going back to the topic, particularly as I can be seen as a controversial character due to that? Well, there are a few things that made me think. Only partially, this relates to the fact that I’ve been attacked a time or two for working for said corporation, and some of it is because I stopped contributing to some projects — although all but one of those cases, the reason was simply my own energy to keep contributing, rather than a work specific problem.

I have seen other colleagues still maintaining enough energy to keep contributing to open source while working at the same office; I have seen some dedicating enough of their work time to that as well. While of course this kind of jobs do limit the amount of time you can put to Free Software, I also think that a number of contributors who end up burning themselves out due to the hardships of paying the bills would probably be glad to exchange full-time Free Software work with part-time one if they were so lucky. So in this, I still count myself particularly privileged, but embrace it, because if I can contribute less time but for a longer time, I think it’s worth it.

But while I do my best to keep improving Free Software, and contribute to the public good, including by documenting glucometer protocols, I hear people criticizing how the only open-source GSM stack is funded, even though Harald Welte is dedicating a lot of his personal time, and doing a lot of ungrateful work, while certain “freedom fighters” decide to cut corners and break licenses.

At the same time, despite not being my personal favourite company, particularly after the most recent allegations of its conduct, GitHub’s Open Source Friday is a neat idea to convince companies that rely on Free Software to do something — sometimes the something may just as well be writing documentation for software, because that’s possibly more important than coding! Given that some of the reasons I’ve read for attacking them is that they are not “pure enough”, because they do not open their core business application, I feel it’s a bit of a cheap shot, given that they are probably the company that most empowered Free Software since the original SourceForge.

So what is that I am suggesting (given people appear to expect me to have answers in additions to opinions)? If I have to give a suggestion to all Free Software contributors out there, that is to always consider what they can do to make sure that their contributions can be consumed at all. That includes for instance not using joke licenses and not discriminating against requests from companies, because those companies might have the resources to make your software successful too.

Which does not mean that you should just bend to whatever the first company passing by request you, nor that you should provide them with free labour. Again that’s a game of balance: you can’t have a successful project that nobody uses, but you’re not there to work for free either. The right way is to provide something that is useful and used. And to make this compromise work, one of the best suggestion I can give to Free Software developers, is to learn a bit about the theories of business.

Unfortunately, I have also seen way too many Free Software supporters (luckily, less so contributors) keep believing that words like “business” and “marketing” are the devil’s own, and do not even stop thinking of what they actually mean — and that is a bad thing. Even when you don’t like some philosophy, or even more so when you don’t like some philosophy, the best way to fight it, is to know it. So if you really think marketing is that evil, you may want to go and read a book about marketing: you’ll understand how it works, and how to defend yourself from its tactics.

More on IPv6 feasibility

I didn’t think I would have to go back to the topic of IPv6, particularly after my last rant on the topic. But of course it’s the kind of topic that leads itself to harsh discussions over Twitter, so here I am back again (sigh).

As a possibly usual heads’ up, and to make sure people understand where I’m coming from, it is correct I do not have a network background, and I do not know all the details of IPv6 and the related protocol, but I do know quite a bit about it, have been using it for years, and so my opinion is not one of the lazy sysadmin that sees a task to be done and wants to say there’s no point. Among other things, because I do not like that class of sysadmins any more (I used to). I also seem to have given some people the impression that I am a hater of IPv6. That’s not me, that’s Todd. I have been using IPv6 for a long time, I have IPv6 at home, I set up IPv6 tunnels back in the days of having my own office and contracting out, and I have a number of IPv6-only services (including the Tinderbox).

So with all this on the table, why am I complaining about IPv6 so much? The main reason is that, like I said in the other post, geeks all over the place appear to think that IPv6 is great right now and you can throw away everything else and be done with it right this moment. And I disagree. I think there’s a long way to go, and I also think that this particular attitude will make the way even longer.

I have already covered in the linked post the particular issue of IPv6 having originally be designed for globally identifiable, static IPv6 addresses, and the fact that there have been at least two major RFCs to work around this particular problem. If you have missed that, please go back to the post and read it there because I won’t repeat myself here.

I want instead to focus on why I think IPv6 only is currently infeasible for your average end user, and why NAT (including carrier-grade NAT) is not going away any year now.

First of all, let’s define what an average end user is, because that is often lost to geeks. An average end user does not care what a router does, they barely care what a router is, and a good chunk of them probably still just call them modem, as their only interest is referring to “the device that the ISP gives you to connect to the Internet”. An average user does not care what an IP address is, nor cares how DNS resolution happens. And the reason for all of this is because the end user cares about what they are trying to do. And what they are trying to do is browse the Internet, whether it is the Web as a whole, Facebook, Twitter, YouTube or whatever else. They read and write their mail, they watch Netflix, NowTV and HBO GO. They play games they buy on Steam or EA Origin. They may or may not have a job, and if they do they may or may not care to use a VPN to connect to whatever internal resources they need.

I won’t make any stupid or sexist stereotype example for this, because I have combined my whole family and some of my friends in that definition, and they are all different. They all don’t care about IPv6, IPv4, DSL technologies and the like. They just want an Internet connection, and one that works and is fast. And with “that works” I mean “where they can reach the services they need to complete their task, whichever that task is”.

Right now that is not an IPv6 only network. It may be, in the future, but I won’t hold my breath for a number of reasons, that this is going to happen in the next 10 years, despite the increasing pressure and the increasing growth of IPv6 deployment to end users.

The reason why I say this is that right now, there are plenty of services that can only be reached over IPv4, some of which are “critical” (for some definition of critical of course) to end users, such as Steam. Since the Steam servers are not available over IPv6, the only way you can reach them is either through IPv4 (which will involve some NAT) or NAT64. While the speed of the latter, at least on closed-source proprietary hardware solutions, is getting good enough to be usable, I don’t expect it being widely deployed any time now, as it has the drawback of not working with IP literals. We all hate IP literals, but if you think no company ever issue their VPN instructions with an IP literal in them, you are probably going to be disappointed once you ask around.

There could be an interesting post about this level of “security by obscurity”, but I’ll leave that for later.

No ISP wants to receive calls from their customers that access to a given service is not working for them, particularly when you’re talking about end users that do not want to care about tcpdump and traceroute, and customer support that wouldn’t know how to start debugging that NowTV will send a request to an IPv4 address (literal) before opening the stream, and then continue the streaming over IPv4. Or that Netflix refuse to play any stream if the DNS resolution happens over IPv4 and the stream tries to connect over IPv6.

Which I thought Netflix finally fixed until…

Netflix & Oh God It Thinks My IPv6 Tunnel Is A Proxy Fine Let's Write An Unbound python-script To Block AAAAs https://t.co/YmItS14g4N pic.twitter.com/WPgB2xvSZL
— Filippo Valsorda 🍵 #36c3 (@FiloSottile) June 25, 2017

Now, to be fair, it is true that if you’re using an IPv6 tunnel you are indeed proxying. Before I had DS-Lite at home I was using TunnelBroker and it appeared like I was connecting from Scotland rather than Ireland, and so for a while I unintentionally (but gladly) sidestepped country restrictions. But this also happened a few times on DS-Lite, simply because the GeoIP and WhoIs records didn’t match between the CGNAT and the IPv6 blocks. I can tell you it’s not fun to debug.

The end result is that most customer ISPs will choose to provide a service in such a way that their users feel the least possible inconvenience. Right now that means DS-Lite, which involves a carrier-grade NAT, which is not great, as it is not cheap to run, and it still can cause problems, particularly when users use Torrent or P2P heavily, in which case they can very quickly exhaust the 200-ports forwarding blocks that are allocated for CGNAT. Of course DS-Lite also takes away your public IPv4, which is why I heard a number of geeks complaining loudly about DS-Lite as a deployment option.

Now there is another particular end user, in addition to geeks, that may care about IP addresses: gamers. In particular online gamers (rather than, say, Fallout 4 or Skyrim fans like me). The reason for that is that most of the online games use some level of P2P traffic, and so require you to have a way to receive inbound packets. While it is technically possible to set up IGD-based redirection all the way from the CGNAT address to your PC or console, I do not know how many ISPs implement that correctly. Also, NAT in general introduces risks for latency, and requires more resources on the passing routers, and that is indeed a topic that is close to the heart of gamers. Of course, gamers are not your average end user.

An aside: back in the early days of ADSL in Italy, it was a gaming website, building its own ISP infrastructure, that first introduced Fastpath to the country. Other ISPs did indeed follow, but NGI (the ISP noted above) stayed for a long while a niche ISP focused on the need of gamers over other concerns, including price.

There is one caveat that I have not described yet, but I really should, because it’s one of the first objections I receive every time I speak about the infeasibility of IPv6 only end user connections: the mobile world. T-Mobile in the US, in particular, is known for having deployed IPv6 only 3G/4G mobile networks. There is a big asterisk to put here, though. In the US, and in Italy, and a number of other countries to my knowledge, mobile networks have classically been CGNAT before being v6-only, and with a large amount of filtering in what you can actually connect to, even without considering tethering – this is not always the case for specialised contracts that allow tethering or for “mobile DSL” as they marked it in Italy back in the days – and as such, most of the problems you could face with VPN, v4 literals and other similar limitations of v6-only with NAT64 (or proxies) already applied.

Up to now I have described a number of complexities related to how end users (generalising) don’t care about IPv6. But ISPs do, or they wouldn’t be deploying DS-Lite either. And so do a number of other “actors” in the world. As Thomas pointed out over Twitter, not having to bother with TCP keepalive for making sure a certain connection is being tracked by a router makes mobile devices faster and use less power, as they don’t have to wake up for no reason. Certain ISPs are also facing problems with the scarcity of IPv4 blocks, particularly as they grow. And of course everybody involved in the industry hates pushing around the technical debt of the legacy IPv4 year after year.

So why are we not there yet? In my opinion and experience, it is because the technical debt, albeit massive, is spread around too much: ISPs, application developers, server/service developers, hosting companies, network operators, etc. Very few of them feel enough pain from v4 being around that they want to push hard for IPv6.

A group of companies that did feel a significant amount of that pain organized the World IPv6 Day. In 2011. That’s six years ago. Why was this even needed? The answer is that there were too many unknowns. Because of the way IPv6 is deployed in dual-stack configurations, and the fact that a lot of systems have to deal with addresses, it seems obvious that there is a need to try things out. And while opt-ins are useful, they clearly didn’t stress test enough of the usage surface of end users. Indeed, I stumbled across one such problem back then: when my hosting provider (which was boasting IPv6 readiness) sent me to their bank infrastructure to make a payment, the IP addresses of the two requests didn’t match, and the payment session failed. Interactions are always hard.

A year after the test day, the “Launch” happened, normalizing the idea that services should be available over IPv6. Even though that the case, it took quite a longer while for many services to be normally available over IPv6, and I think, despite being one of the biggest proponents and pushers of IPv6, Microsoft update servers only started providing v6 support by default in the last year or so. Things improved significantly over the past five years, and thanks to the forced push of mobile providers such as T-Mobile, it’s a minority of the connections of my mobile phones that still connect to the v4 world, though there are still enough not to be able to be ignored.

What are the excuse for those? Once upon a time, the answer was “nobody is using IPv6, so we’re not bothering supporting it”. This is getting less and less valid. You can see the Google IPv6 statistics that show an exponential growth of connections coming from IPv6 addresses. My gut feeling is that the wider acceptance of DS-Lite as a bridge solution is driving that – full disclosure: I work for Google, but I have no visibility in that information, so I’m only guessing this out of personal experience and experience gathered before I joined the company – and it’s making that particular excuse pointless.

Unfortunately, there are still “good” excuses. Or at least reasons that is hard to argue with. Sometimes, you cannot enable IPv6 for your web service, even though you have done all your work, because of dependencies that you do not control, for instance external support services such as the payment system in the OVH example above. Sometimes, the problem is to be found in another piece of infrastructure that your service shares with others and that requires to be adapted, as it may have code expecting a valid IPv4 address at some particular place, and an IPv6 would make it choke, say in some log analysis pipeline. Or you may rely on hardware for the network layer that just still does not understand IPv6, and you don’t want to upgrade because you still have not found enough of an upside to you to make the switch.

Or you may be using an hosting provider that insists that giving you a single routable IPv6 is giving you a “/64” (it isn’t — they are giving you a single address in a /64 they control). Any reference to a certain German ISP I had to deal with in the past is not casual at all.

And here is why I think that the debt is currently too spread around. Yes, it is true that mobile phones batteries can be improved thanks to IPv6. But that’s not something your ISP or the random website owner care about – I mean, there are websites so bad that they take megabytes to load a page, that would be even better! – and of course a pure IPv6 without CGNAT is a dream of ISPs all over the world, but it is very unlikely that Steam would care about them.

If we all acted “for the greater good”, we’d all be investing more energy to make sure that v6-only becomes a feasible reality. But in truth, outside of controlled environments, I don’t see that happening any year now as I said. Controlled environments in this case can refer not only to your own personal network, but to situations like T-Mobile’s mobile data network, or an office’s network — after all, it’s unlikely that an office, outside of Sky’s own, would care whether they can connect to NowTV or Steam. Right now, I feel v6-only network (without NAT64 even) are the realm of backend networks. Because you do not need v4 for connecting between backends you control, such as your database or API provider, and if you push your software images over the same backend network, there is no reason why you would even have to hit the public Internet.

I’m not asking to give a pass to anyone who’s not implementing v6 access now, but as I said when commenting on the FOSDEM network, it is not by bothering the end users that you’ll get better v6 support, is by asking the services to be reachable.

To finish off, here’s a few personal musings on the topic, that did not quite fit into the discourse of the post:

Some ISPs appear to not have as much IPv4 pressure as others; Telecom Italia still appears to not have reassigned or rerouted the /29 network I used to have routed to my home in Mestre. Indeed, whois information for those IPs still has my old physical address as well as an old (now disconnected) phone number.
A number of protocols that provide out-of-band signalling, such as RTSP and RTMP, required changes to be used in IPv6 environments. This means that just rebuilding the applications using them against a C library capable of understanding IPv6 would not be enough.
I have read at least one Debian developer in the past giving up on running IPv6 on their web server, because their hosting provider was sometimes unreliable and they had no way to make sure the site was actually correctly reachable at all time; this may sound like a minimal problem, but there is a long tail of websites that are not actually hosted on big service providers.
Possibility is not feasibility. Things may be possible, but not really feasible. It’s a subtle distinction but an important one.