Service Providers, Business Sustainability, And Society

I’m not sure how many people think consciously about the business plans of the service providers that they start using, or at least relying upon. I don’t do it terribly often, but I do sometimes, and I thought I would share some words on the topic, because I do think that the world would be a better place if we did think about the effect of our choices to the wide world.

I have for instance wondered aloud, over the years, on Twitter and even on this blog, about the fintech company Curve. Their services are actually interesting and valid… but they do not offer any interesting value at a premium — I would never pay for their “Curve Metal” tiers, because it just doesn’t add up. I couldn’t just figure out how they were expecting to keep running, given I expect the majority of their “customers” are consumer end-users that would follow the same procedure I followed: sign-up for the service, using the £5 offer, use it for the 90 days of free cashback, possibly use it a time or two while traveling, and otherwise just keeping it as a backup card. Before the pandemic I also wrote how they were giving out more free money, but more recently they decided to start crowdfunding. And they became very loud when they did. Which is what reminded me I had a half-drafted post on the topic (this post), which I should probably resurrect (which I did, since you’re reading this now).

Before going back to Curve and their crowdfunding, I want to point at two sayings that you’ll keep finding around you, when you discuss businesses, products, Internet, and privacy: «If it sounds too good to be true, it probably is» and «If you are not paying for it, you’re not the customer, you’re the product». These are good places to start a discussion, although I think there is a lot of nuance that is lost when trying to (over) simplify with these.

Full disclosure: I work for a big company that, primarily, offers services to end users free with ads, though the product I personally work on not only does not relate to ads, but does not even have ads in it. And while my previous employer was another big company that is part of the “AdTech” business (and in there I did work on ads systems for a long while), I have discussed ads in the past, and I even ran ads on this blog (back in the days before stable employment), so you can imagine that what I’m going to be writing about is my personal opinion and does not represent that of my current, past, or future employers.

So why do I think it’s important to figure out the sustainability of service providers? Well, because it becomes a problem for the whole of society when a fraudulent or scammy provider gets a certain about of market share, even if sometimes not evenly and not in a way that most people would be able to connect together. For instance you can take the examples of Enron, Bernie Madoff, Theranos, or Wirecard — organizations that promised too-good-to-be-true services and profits, and ended up bust, with different blast radiuses. For the last one we don’t even quite know yet what the blast radius will be once things settle down: the German financial services environment is likely going to be reshaped by last year’s scandal, and so it appears will be EY.

Though this is clearly not limited to financial services (VPN providers seem to be pretty much in the same position), it does look like the likes of Curve and Revolut are easily the most visible cases where a company apparently lacked a sustainable business plan, and decided to turn to a crowdfunding campaign — Curve just had one this past May, and it was so noisy that I know a couple of people who went on to find a way to delete their account (and the app) simply because they got tired of their pushy notifications (not a typo).

Now, that might sound not too bad — after all, crowdfunding for the most part just means someone is willingly going to pay to subsidise other people’s “free money”. But the next thing that Curve did after that was to increase referral bonuses for new users to £20, from the previous £5, and that smells to me even more fishy — because that sounds like trying to bring in a mass of users hoping that enough of them can’t figure out that the premium options of Curve are not worth the money.

On the other side of the tracks, Revolut has been pushing more and more for cryptocurrencies, which I’m not going to even pretend is a neutral thing. I care enough about the environment that their consumption alone makes me angry, but even more so, I find that the amount of scams related to cryptocurrencies at this point are wide enough to show that the whole concept is hostile to society. I do not support nor recommend Revolut to new users unless they live in countries like Ireland where there is no other option – in London, using Revolut feels like subsidizing scammers by lending respectability to cryptocurrencies.

But at this point, neither of those appear to have reached the full consumer scams, so it should be fine, right?

Well, let’s take a different example with the VPN market. I have complained over on Twitter a few times how I blame us geeks for the amount of VPN scams that are out there. Privacy maximalists tend to scare people with the idea that your ISP, the Starbucks, or the airport lounge you’re using can see everything you do — and while it is definitely the case that there’s a lot more data going around than you may think, ads such as those ExpressVPN pays the otherwise excellent No Such Thing As A Fish podcast to air, that suggests that your ISP would be able to tell what you’re Googling are not just falsehoods, but proper FUD.

But even accepting that ExpressVPN has no ulterior motive and are totally legit – I have no idea about that, I only used them before while in China – and leaving aside the fact that VPNs have huge targets painted on their backs, there’s still the matter that you need to trust your VPN provider. Which may or may not be more trustworthy than your home ISP — the two of of them having pretty much the same power. Most of the review websites seem to be talking more about commissions than trust, because the worst part is that there is nothing that allows you to verify their statement that they are not logging your traffic in the same way they keep insisting your ISP is doing.

So how do you trust a VPN provider? Well, for a start you may want to start considering who their founders are and how they get their money. And you’d be surprised how many dots you can connect this way. For instance, last year CNET wrote about Kape Technologies, a company that bought a Romanian VPN service called CyberGhots. In that article, they also noted that in addition to CyberGhost, the same parent company bought two more VPNs:

After buying CyberGhost, Kape then bought VPN ZenMate in 2018 and more recently Private Internet Access, a US-based VPN, in a move which Erlichman said in a press release would allow Kape to “aggressively expand our footprint in North America.”

Now, the problem is less about a single parent company owning multiple VPN services as they were different brands — this happen all the time in many other fields. Just look at the relationship between Tesco Mobile and O2, or banks such as Halifax and Lloyds. But the rest of the article does make for a good build up for why the whole situation is a bit suspect.

But more importantly, you may have heard of Private Internet Access before — they are the company that started heavily sponsoring Freenode a few years ago. And if you have been paying attention to Free Software projects’ communication in the past few months, you probably know by now that Freenode is a trash fire now. So given those connections, would you trust anything that has connections to these organizations and people? I clearly wouldn’t.

This same problem with trust and business sense applies to other businesses. With the exception of B Corporations, most companies out there are intended to make money. If nothing else, they need to make money so that they can pay the wages of the people working there. So I don’t generally trust companies that appear to be giving everything away — and rather prefer those that, if they are making money with ads, say so out right.

In the case of Fintech services — Wise (formerly known as TransferWise) is my example of choice for a company that is transparent when it comes to the cost associated with their services, and makes a good case for why they charge you, and how much so. I really wish more of them did the same because it would make it easier for people to choose how much trust to put in a company. Unfortunately it appears that the current trend in the market is to push as much grown as possible for companies to grab a captive audience before turning on the monetization screw.

Important note: this blog post was written before Wise announced they intend to go public (it was previously rumored, but I didn’t spot that). I guess I should now disclose that I will most likely consider buying some stock of the company, though probably not on the IPO day. We shall see. As I said, I do like their business sense.

Going back to a moment to that «if you’re not paying for it, you’re the product» as well — well, I don’t agree in full, but this is something that people do need to be keyed in to look out for. In particular, I don’t think that ad-supported businesses should disappear, and that everything should be hidden behind a paywall, because I do think that having wider access to information without making it costly is a good thing. But also I do think that there are services that are often crossing the line into being creepily interested in your data rather than “trade it” for useful information.

But I also think the scrutiny is often placed more on the big, established companies rather than the “scrappy” startups, or the more consulting-like companies. Heck, a few of you reading this are probably already ready to complain that both my current and past employers are seen as data hungry — but I can tell you that both companies, at least during my tenure, would never have someone state on a stage that collecting data from IoT sensors and just throwing it to a ML pipeline to gather unexpected insights, as it would go against every one of the privacy and data handling trainings and commitments…

And yet John Roese from Dell EMC stated that in his opening thoughts for LISA 16 (go to minute 44 in the open access video) in what sounds terribly like an advice to startups. To be honest, that’s not the only cringey thing in that opening talk — from a technical point of view, his insisting that persistent memory means you can’t just reboot a computer to reset the state of memory (as if re-loading the data in memory from scratch wouldn’t happen on request, whether this is persistent or not) is probably a worst phrase.

What I’m trying to say is that you need to be sure who your friends are, and it’s not as easy as to expect that all small players are ethical and all big ones are not. And asking yourself “how are they making money, if at all?” is not just allowed — it should sometimes be considered a necessity.

Fishy Facebook Ads: Earthly Citizens, Shutter & Contrast, and many more

(If you prefer this in form of a Twitter thread, see this one.)

Let’s start with the usual disclaimer that despite me working for a company that sells advertisement, this post is my own personal opinion, not my employer’s. I have written about Internet ads for years, well before I joined the company, and so it’s nothing new. To the usual disclaimer I’m going to add a few words to point out that there will be a few company names used in this post — I’ll be very clear when I think they are involved in something fishy, and when I think they are not involved at all.

This all starts with me deciding to get myself a new camera. While I’m very happy about the photos that my usual camera produce, I wanted something lighter that I could go around town more often with. But I also have been having issues with my shoulder, and I’ve been looking out for a good “handy” backpack to keep my stuff in. This is all relevant information.

Indeed, if you follow me on Twitter you may have seen me asking around for suggestions on backpacks. And this is also relevant: since I’m actually not minding ads for relevant content for myself, I have not hidden my looking for a new bag, I spoke about it on social media, and I have searched for backpacks and bags on my normal Google session. This is, again, all relevant information.

Because of my Google searches, I have been seeing a lot of ads related to photography. Including the one for the chain of photography stores that convinced me to go and grab my new camera from them. Very few of those ads are useful to me, but that one in particular have been.

Then the other day, on Instagram, I saw the ads for a backpack from a never-heard-before company advertising as Earthly Citizens. I’m not going to link directly to their website, although I’m choosing to explicitly name them here so that people who may be looking for them on Google and other search engines have a landing page helping them. The backpack that they advertised is this one (archived link) and it actually looks very nice in theory, on offer at £87.75 compared to a RRP of £159.61. To compare, my trusty Think Tank Airport Essentials is £147.04, and that’s one hell of a good bag.

The amount of red flags on that advertisement was high: unknown brand, no branding on the actual bag, unrealistic “flash sale” with no dates on it, and so on. So I didn’t really pay much attention. Then of course, since I have looked at the ad, I started seeing the same bag on Facebook — together with nearly 900 positive comments. I decided to do a minimum amount of digging into it, and found out that the website that the ad points to is a standard Shopify instance, which means that digging into it with IP addresses or WhoIs information is useless. And since there’s no address provided for the company even on privacy pages, there’s not much to go by. I walked away.

A day later, another set of ads start appearing on my Facebook stream, and they are for a backpack that is stunningly similar, or rather identical. But from a different page that has a more “photography” feel to it, called “Shutter & Contrast”. And that piqued my interest a little bit, because it sounded like another one of those cloned bags that I have seen aplenty on Instagram, and I would actually like to find the source at that point.

Just like Earthly Citizens, Shutter & Contrast don’t seem to be very well reviewed. Searching the web for the name and combination of reviews, backpack and scam don’t bring up anything useful. They also have a Shopify site, although their page for the same backpack (archived, again) is a bit more somber and “professional-looking”.

Funnily enough, it looks like they have blocked copy-paste and right-click, so that you can’t quickly reverse-image-search their photos. It didn’t surprise me, as I remembered a BuzzFeed article on fake fashion stores outright stealing real designers’ photos, so stopping the quickest reverse image search option would obviously be high in their intentions. Of course it’s actually easy to work this around, with any of the browsers’ developer tools.

Another interesting part from the Shutter & Contrast shop page is that they actually have an address in their Privacy Page: 11923 NE Sumner St, STE 813872, Portland, Oregon, 97220, USA. Again I’m repeating it here for sake of those looking any information on this company, because if you look up the address, you’ll probably find a Yelp page for a closed location called My Trail Gear, although it has a different “STE” number. The reviews, calling this a scam and pointing out that there is at least two more companies using the address, called “Bear and Tees” and “Shark and Tees”.

Checking the address on StreetView shows a smallish warehouse. My best guess is that there’s a service at that address that is similar to Ireland’s Parcel Motel and Parcel Wizard: companies that allow you to receive and send goods from that address, and then forward it somewhere else. The different “STE” numbers are used to route the parcels to the right customer. This means that despite the bad reviews on Yelp, Shutter & Contrast might be legit.

So I decided to take a closer look at the first one again. Earthly Citizen has a fairly active Facebook page, and if you read their About section, it says:

Our goal is to source all the best travel related documents from all around the world and bring them directly to your doorstep
Earthly Citizens Facebook Page

They don’t seem to be doing anything like that. Instead they seem to mostly re-post Instagram pictures by other people. At least it appears they are crediting the photographers — but it’s clear that they are using someone else’s pictures for their own marketing (so that they get people to follow their account). This should be worrisome enough, but it doesn’t stop there.

If you look at what they sell, they appear to be selling a lot of random stuff that you would find in those trinkets/gadgets shop in big malls, without brands, rhyme, or reason. So it does not look like they are the “source” of that bag to begin with. But is Shutter & Contrast then?

Earthly Citizens say that there are “too many fake websites that steal content”. They would know since they seem to be one.

A very quick reverse image search finds the same exact image appears on AliExpress (not archived because they seem to defeat it), the Chinese shopping website. There are multiple sellers for it there as well, and most of them have the same images — the same images that both Earthly Citizens and Shutter & Contrast used on their website.

It might very well be that these are the bag equivalent of Gongkai, as there are a few stores that sell them, and the fact that they come from Guangdong does not mean they are not good. I have a lovely tripod I bought at the Shanghai Xing Guang Photography Market, it’s a Chinese brand, it’s proper carbon fiber, and I paid for it half the price that you would pay in store in Europe, taxes included. If that is the case, the markups that Earthly Citizens and Shutter & Contrast are applying are thievery: they price it at $110 and $83 respectively, while AliExpress’s most expensive seller has it at $52.

But there is one thing that I forgot about during my Twitter rant, and that my girlfriend pointed out: what about the pictures of people in the advertising? Neither AliExpress nor Earthly Citizens appear to have a picture of the backpack with a person. There are people with cameras, but nobody with the actual backpack that you can reverse image search for. There is a video on Earthly Citizens’s Facebook page, which is the same used by the Instagram ad, and that suggests that the bag physically exist, but it’s heavily watermarked that makes it hard to find the source on. Shutter &Contrast has a video unlisted on YouTube, on a white background with no logos shown, and just re-captioned to fit their marketing of it. It appears uploaded in February 2019.

More useful, Shutter & Contrast appear to also have a still picture of someone wearing what looks like the backpack they are selling, and that’s the first time in this adventure I managed to find that. Reverse image search brings us to yet another Shopify instance under the name ConnectedTechPacks (archived), which can also be found as BestGearPack. Their website is a bit more well made, and it appears to only sell that single backpack. Are they the source? I doubt so, since both websites were registered in April this year, and we know that the backpack existed in February. But they also have a couple of different people with the same backpack, and another angle of the same guy.

Another reverse image search later finds yet another Shopify instance with the same backpack, a set of GIF animations that are also heavily watermarked, but are the same as Earthly Citizens’s version.

So where did all this investigation bring us? Not really anywhere. I can’t find any trustworthy brand selling the backpack, and while I may be willing to risk my £40 on the AliExpress version – rather than twice as much with any of the other Shopify instances that I found – I don’t hold my breath for it to look at all like they show it, or have the build quality that I would trust my cameras with.

It does show just how easy it is to fool people nowadays. It’s easy to set up a “storefront” without needing an actual space anymore. It’s easy to “gain trust” by having people follow your page with no original content, just by re-posting content that professionals provided.

What about the 900 positive comments that the ad received? Well it’s possible that they are actual real satisfied customers who didn’t realize they got charged probably twice as much as they should have for the same bag you can get from AliExpress. Or they may be “bought engagement”. Or just a bunch of bots that have harvested someone else’s name and pictures to create fake profile to sell the stuff.

You know all the panic around politics and elections and fake profiles? It’s not just the elections. Fake profiles sell scams. And that can hurt people just as much as political elections. I remember when it was just the artists complaining about pages re-posting their content… we should have paid attention then. Now the same pages and the same techniques are used for more nefarious purposes and we all pay the price, sooner or later.

Ads, spying, and my personal opinion

In the past year or so, I have seen multiple articles, even by authors who I thought would have more rational sense to them, over the impression that people get about being spied upon by technology and technology companies. I never got particularly bothered to talk about them, among other things because the company I work for (Google) is one that is often at the receiving end of those articles, and it would be disingenuous for me to “defend” it, even though I work in Site Realiability, which gives me much less insight in how tracking is done than, say, my friends who work in media at other companies.

But something happened a few weeks ago gave me an insight on one of the possible reasons why people think this, and I thought I would share my opinion on this. Before I start let me make clear that what I’m going to write about is something that is pieced together with public information only. As you’ll see soon, the commentary is not even involving my company’s products, and because of that I had access to no private information whatsoever.

As I said in other previous posts, I have had one huge change in my personal life over the past few months: I’m in a committed relationship. This means that there’s one other person beside me that spends time in the apartment, using the same WiFi. This is going to be an important consideration as we move on later.

Some weeks ago, my girlfriend commented on a recent tourism advertisement campaign by Lithuania (her country) on Facebook. A few hours later, I received that very advertisement on my stream. Was Facebook spying on us? Did they figure out that we have been talking a lot more together and thus thought that I should visit her country?

I didn’t overthink it too much because I know it can be an absolute coincidence.

Then a few weeks later, we were sitting on the sofa watching Hanayamata on Crunchyroll. I took a bathroom break between episodes (because Cruncyroll’s binge mode doesn’t work on Chromecast), and as I came back she showed me that Instagram started showing her Crunchyroll ads — “Why?!” We were using my phone to watch the anime, as I have the account. She’s not particularly into anime, this was almost a first as the material interested her. So why the ads?

I had to think a moment to give her an answer. I had to make a hypothesis because obviously I don’t have access to either Crunchyroll or Instagram ads tracking, but I think I’m likely to have hit close to the bullseye and when I realized what I was thinking of, I considered the implications with the previous Facebook ads, and the whole lot of articles about spying.

One more important aspect that I have not revealed yet, is that I requested my ISP to give me a static, public IPv4 address instead of the default CGNAT one. I fell for the wet dream, despite not really having used the feature since. It’s handy, don’t get me wrong, if I was to use it. But the truth is that I probably could have not done so and I wouldn’t have noticed a difference.

Except for the ads of course. Because here’s how I can imagine these two cases to have happened.

My girlfriend reads Lithuanian news from her phone, which is connected to my WiFi when she’s here. And we both use Facebook on the same network. It’s not terribly far-fetched to expect that some of the trackers on the Lithuanian news sites she visits are causing the apartment’s stable, static, public IP address to be added to a list of people possibly interested in the country.

Similarly, when we were watching Crunchyroll, we were doing so from the same IP address she was checking Instagram. Connect the two dots and now you have the reason why Instagram thought she’d be a good candidate for seeing an advert for Crunchyroll. Which honestly would make more sense if they intended to exclude those who do have an account, in which case I would not have them trying to convince me to… give them the money I already give them.

Why do I expect this to be IP tracking? Because it’s the only thing that makes sense. We haven’t used Facebook or Messenger to chat in months, so they can’t get signal from that. She does not have the Assistant turned on on her phone, and while I do, I’m reasonably sure that even if it was used for advertisement (and as far as I know, it isn’t), it would not be for Facebook and Instagram.

IP-based tracking is the oldest trick in the book. I would argue that it’s the first tracking that was done, and probably one of the least effective. But at the same time it’s mostly a passive tracking system, which means it’s much easier to accomplish under the current limits and regulations, including but not limited to GDPR.

This obviously has side effects that are even more annoying. If the advertisers start to target IP address indiscriminately, it would be impossible for me or my girlfriend to search for surprises for each other. Just to be on the safe side, I ordered flowers for our half-year anniversary from the office, in the off-chance that the site would put me on a targeting list for flower ads and she could guess about it.

This is probably a lot less effective for people who have not set up static IP addresses, since there should be a daily or so rotation of IP addresses that confuses the tracking enough. But I can definitely see how this can also go very wrong when a household dynamic are pathological, if the previous holder of the address managed to get the IP on targeted lists for unexpected announces.

I have to say that in these cases I do prefer when ads are at least correctly targeted. You can check your Ads preferences for Google and Facebook if you want to actually figure out if they know anything about you that you don’t want them to. I have yet to find out how to stop the dozens of “{Buzzword} {Category} Crowdfunding Videos” pages that keep spamming me on Facebook though.

In defence of ads over subscriptions

This is a third draft trying to focus my opinion on one facet of a very complicated, multi-faceted discussion about Internet, content and monetization. Hopefully, I’ll be able to keep my mind focused enough on the problem I want to talk about to come out with a concise and interesting post.

You may have read my opinion on Internet ads and the more recent commentary on my own “monetization”. If you haven’t, the short version of the story is that I do not think that ads are implicitly evil, but at the same time I don’t think that they are worth the attrition for most sites, particularly if very small. My blog, and Autotools Mythbuster, are not exactly unknown sites, but they are still small enough that dealing with Ads is barely worth it for most people, and clearly not worth it for me.

For big, global and media sites, ads are still (for the most part) the way to get money, and to keep running. And that is true whether those sites started as “blogosphere” Internet-native sites, like Ars Technica, or as online versions of establishment news papers like The New York Times. And particularly as you go big, and you want to have curated content, secure content and (extremely important here!) good and verified content, you have costs that add up and that require some level of income to maintain.

You can see that with LWN.net, whose content is very well curated, and is for the most part paid contributions — I can tell you that the folks over there, starting from Jonathan, would not be able to afford such a great edited content if they had no income on it. It takes time to do so, and you can’t run it as a side project or moonlighting job. Or for a more mainstream example, take what’s happening with Snopes.

One of a common talking point I hear from many companies that are involved in advertising on the Internet, including the company I work for, is that we “enable free content publishing on a global scale”. It may sounds as a platitude written this way, and sound hollow. But once you think it twice or thrice over, it starts to ring true. Because all that content would not be able to pay for the wages of the people involved in charitable donations only. Ads do really help with that.

But then I hear the extremists that decide that all ads are bad, install uBlock, and refuse to even consider that the vast majority of both sites and advertisers are trying to make a honest living, and they are neither clickfarms nor malvertisers, even though these two problems are real. And I’m not fond of SEO tactics either, but that’s also something that is not evil by itself. What is the solution, according to some of those people I engaged with at many times? For some of them the answer is to just not expect money for your online content. This works fine for me, nowadays, but it wouldn’t have worked for me before. Also, the people I heard suggesting this before are usually people who have not tried producing high-quality content before. As I wrote years ago the amount of effort spent writing a blog post is minuscule compared to that needed for me to write a simple article for LWN. And indeed, I got paid for my effort for those — $100 sounds like a lot for writing, but turns out it’s well below minimum wage by the amount of time involved.

The alternative is of course saying that the answer is to sell ad-free subscriptions instead. Which is cool by me in general, but that can only work if those who can’t afford the subscriptions are indeed not using adblockers for everything. But that is clearly not the case, and it can’t be the case as long as malvertising is indeed a risk for most people. And my reason to have this opinion is that by relying on a subscription-only model, you end up either significantly limiting the gamut of content that is available on the Internet, or its availability to the public.

We have for instance all cursed at the bother of paywalls when trying to get to content that may be interesting, or even vital, to us. But the paywall is probably the only way to make sure that the content is paid for, with a subscription. This way you don’t risk the “cheaters” that will not subscribe but also won’t offer their attention so that advertisers can fill in for those subscriptions. Paywalls are nasty and I would not advocate for them. They limit access to information, dividing people into those who can afford to be informed, and those who can’t.

There is the other option of this of course, which is the “tip-jar” option: you provide free access to the information, with no advertisement, but you have a public support campaign that suggests people to provide money for the content, either by direct subscription or by providing fundraising campaigns, such as Patreon. The problem with this system is that it’s skewed towards the content that is of interests to the people with more disposable income. It’s the same problem as the prototypes coming in on Kickstarter and similar sites: since I’m a well-paid tech employee, I can afford to finance hobby projects and content that interests me, which means the creator will try to pander to my taste by providing projects and content that appeal to well paid tech employees. And then there is the occasional situation in which those tech employees grow a heart, a passion, or more properly just want to be some sane, decent, human beings and decide to pay for content, projects, services or charities that they have no stakes in.

But if you write content that is targeted to a group of people with no disposable income, you probably would feel better if you knew that honest companies with an advertising budget underpin your wages, rather than feeling at the mercy of the magnanimity of some people who are not your target audience, and may one day just disagree with you, your message or your method, and just cut your funds. Although of course, this is also a double-edged sword, as we’ve seen that hateful content can be cut from ads revenue but persist through the deep pocket of the people wanting the message to stay out.

Where does this leave us? I’m not sure. I just think that the subscription model is not the nice good solution that I hear some people suggest. I also find it quite ironic that a number of people who suggested this option – and told me explicitly that they think content should either be able to make people pay for their content or not get money out of it – are further to the left of me, and appear disgusted at the suggestion that the free market is here to stay. Well, guess what they just described for content? And yeah it’s not perfect, far from it.

Honestly, I really would like for super-partes agencies to be able to distribute funds, fairly gathered, into diverse content creators. But that is very hard to accomplish. The obvious answer would be to let the government collect taxes and redistribute it in funds for the art. But we also know that providing the government with the task of redistributing money for content leads to the government publishing only content that looks favourable to those in charge. And I’m not really sure there is much of a better option nowadays.

I should probably write another post talking about funding sources like the old Flattr and the Brave browser, but that’s a topic for another time.

Let’s have a serious talk about Ads

I have already expressed my opinion on Internet Ads months ago, so I would suggest you all to start reading from that, as I don’t want to have to repeat myself on this particular topic. What I want to talk right now is whether Ads actually work at all for things like my blog, or Autotools Mythbuster.

I’ll start by describing the “monetization” options that I use, and then talk a bit about how much they make, look into the costs and then take a short tour of what else I’m still using.

Right now, there are two sources of ads that I use on this blog: Google AdSense and Amazon Native Ads. Autotools Mythbuster only has AdSense, because the Amazon ads don’t really fit it well. On mobile platform, the only thing you really see is AdSense, as the Native Ads are all the way to the bottom (they don’t do page-level ads as far as I can tell), on desktop you only get to see the Amazon ads.

AdSense pays you both for clicks and for views of the ads on your site, although of course the former gives you significantly higher revenue. Amazon Native Ads only pays you for the items people actually buy, after clicking on the Ads on your site, as it is part of the Amazon Affiliate program. I have started using the Amazon Native Ads as an experiment over April and May, mostly out of curiosity of how they would perform.

The reason why I was curious of the performance is that AdSense, while it used to mostly make decision on which ads to show based on the content of the page, it has been mostly doing remarketing, which appears to creep some people app (I will make no further comments on this), while the idea that Amazon could show ads for content relevant to what I talked about appealed to me. It also turned out to have been an interesting way to find a bug with Amazon’s acapbot, because of course crawlers are hard. As it turns out, the amount of clicks coming from Amazon Native Ads is near zero, and the buying rate is even lower, but it still stacks fairly against AdSense.

To understand what I mean I need to give you some numbers, which is something people don’t seem to be very comfortable with in general. Google AdSense, overall, brings in a gross between €3 and €8 a month, with a very few rare cases in which it went all the way up to a staggering €12. Amazon Affiliates (which as I’ll get to does not only include Native Ads) varies very widely month after month, as it even reaches $50. Do note that all of this is still pre-tax, so you have to just about cut it in half to estimate (it’s actually closer to 35% but that’s a longer story).

I would say that between the two sources, over the past year I probably got around €200 before tax, so call it €120 net. I would have considered that not bad when I was self-employed, but nowadays I have different expectations, too. Getting the actual numbers of how much the domains cost me per year is a bit complicated, as some of those, including flameeyes.eu, are renewed for a block of years at the same time, but I can give you makefile.am as a point of reference (and yes that is an alias for Autotools Mythbuster) as €65.19 a year. The two servers (one storing configuration for easy re-deployment, and the other actually being the server you read the blog from) cost me €7.36/month (for the two of them), and the server I use for actually building stuff costs me €49/month. This already exceeds the gross revenue of the two advertising platforms. Oops.

Of course there is another consideration to make. Leaving aside my personal preferences on lifestyle, and thus where I spend my budget for things like entertainment and food, there is one expense I’m okay with sharing, and that is my current standing donations. My employer not only makes it possible to match donations, but it also makes it very easy to just set up a standing donation that gets taken directly at payroll time. Thanks to making this very simple, I have a standing €90/month donation, spread between Diabetes Ireland, EFF and Internet Archive, and a couple others, that I rotate every few months. And then there are Patreons I subscribe to.

This means that even if I were to just put all the revenue from those ads into donations, it would barely make an impact. Which is why by the time you read this post, my blog will have no ads left on (Autotools Mythbuster will continue for a month or two just so that the next payment is processed and not left in the system). They would be okay to be left there even if they make effectively no money, except that they still require paperwork to be filed for taxes, and this is why I have considered using Amazon Native Ads.

As I said, Amazon Native Ads are part of their Affiliate program, and while you can see in the reports how much revenue is coming from ads versus links, the payments, and thus the tax paperwork, is merged with the rest of the affiliate program. And I have been using affiliate links in a number of places, not just my blog, because in that case there is no drawback: Amazon is not tracking you any more or less than before, and it’s not getting in your way at all. The only places in which I actually let Amazon count the impression (view) is when I’m actually reviewing a product (book, game, hardware device), and even that is fairly minimal, and not any different from me just providing the image and a link to it — except I don’t have to deal with the images and the link breakage connected with that.

There is another reason why I am keeping the affiliates: while they require people to actually spend money to get me anything, they give you a percentages of the selling price of what was sold. Not what you linked to specifically, but what is sold in the session that the user initiated when they clicked on your link. This makes it interesting and ironic when people click on the link to buy Autotools Mythbuster and end up buying instead Autotools by Calcote.

Do I think this experience is universal or generally applicable? I doubt so. My blog does not get that many views anyway, and it went significantly down since I stopped blogging daily, and in particular it went down since I no longer talk about Gentoo that much. I guess part of the problem with that is that beside for people looking for particular information finding me on Google, the vast majority of the people end up on my blog either because they read it already, or follow me on various social media. I have an IFTTT recipe to post most of my entries on Twitter and LinkedIn (not Google+ because there is no way to do that automatically), and I used to have it auto-post new entries that would go to Planet Gentoo on /r/gentoo (much as I hate Reddit myself).

There is also the target audience problem: as most of the people reading this blog are geeks, it is very likely that they will have an adblocker installed, and they do not want to see the ads. I think uBlock may even make the affiliate links broken, while at it. They do block things like Skymiles Shooping and similar affiliate aggregators, because “privacy” (though there is not really any privacy problem there).

So at the end of the day, there is no gain for me to keep ads running, and there is some discomfort for my readers, thus I took them down. If I could, I would love to be able to just have ads for charities only, with no money going to me at all, but reminding people of the importance of donating, even a little bit, to organizations such as Internet Archive, which saved my bacon multiple times as I fixed the links from this blog to other sites, that in the mean time moved without redirects, or just got turned down. But that’s a topic for another day, I think.

My opinion on internet ads

You may or may not remember that I did post about my (controversial) privacy policy and some of my thoughts on threat models. A related, though should probably be separate, topic is how to handle internet advertisers, and tools like AdBlock, so I thought I would write down my personal preference and how I work.

First of all, I should point out the obvious elephants in the room: not only I work for a company that sells internet ads, but I also use ads on both this blog and Autotools Mythbuster — mostly to try reducing the cost of running these operations, which are mostly a personal whim. On the other hand, the opinions I express in this post are all personal, and are not being influenced by this. They have been forged over time and experience, and some of said experience may have been related to these, though.

Once this is clarified, I should describe my current setup, since that will spark the rest of the content of the post. I (still) use AdBlock Plus extension for Chrome — even with all the possibly shady behaviour that the current owners are behind, I have not found a good replacement; uBlock Origin is not a replacement, as I’ll get to later. I’ve set the extension to behave as an opt-in, rather than opt-out: ads are not blocked anywhere until I ask it to. Chrome for Android does not have AdBlock or similar, so I have nothing really there, on the other hand it’s less of an issue there because I usually just look at the same dozen websites most of the time.

To make ads generally less annoying, I signed up for Google Contributor which allows me to declare a target monthly contribution to use to replace Google Ads with whatever set of images (or nothing at all.) I set it to show me cats, including my own.

As I said above, I set my AdBlock to not block ads by default, so when do I decide to turn it on? Well, to start with I run it on my own websites (except when I’m testing them), since otherwise it’s a bit of a mess with the Terms of Service of AdSense, so this is easier. Other than that, I usually turn it on for various sites when I land on a page and I find it “scammy.” The definition of scammy is of course up to debate, so let me try to explain where I come from.

Also, I need to make this point here, so that if you completely disagree with my idea here, you can probably stop reading (and please don’t comment either): I don’t believe that advertising and marketing are inherently evil. I know plenty of privacy extremists take an issue with the statement, so if you do feel free to move on and read something else altogether.

Not all internet ads are created equal, I think this is obvious to essentially anybody who has been browsing the Internet for more than a few months. Ads may be more or less intrusive, they may be more or less relevant to your interests and they may or may not always be legal. While no supplier is immune, most of the big names thrive hard to avoid ads that outright lie, or that try to pass off for something else. The results are usually mixed as everybody knows already.

On the other hand, there are suppliers that explicitly go for the scams, and some website operators accept them quite willingly. The reason is usually monetary: these networks pay off much better, as the “advertisers” are happy to pay premium to get their (frequently) malware advertised. To give you a bit of an idea, I suggest you read or watch this presentation from the USENIX Security conference.

This is not all, of course. There are also the self-defined “content discovery networks”, that purport to point people at other content they should be interested in, mixing content from the same site with “sponsored links.” Even I tried it once before I noticed how useless it ended up being. Nowadays a lot of those kind of links are coming from two networks: Taboola and Outbrain; in my experience, the latter actually provides kind-of relevant content, the former has lots of almost definite scams that I do not appreciate.

To give you an idea, if I’m reading an article about Brexit, I find it perfectly reasonable to get links to articles suggesting cheap vacation to the UK, an ad for Transferwise and an ad for ig.com (which is, as far as I know, a totally legit tradit website I have no affiliation with, but just seem to spend lots of money in advertisement, as I see it on every other website.) If, on the other hand, a different article on the same topic proposes me links such as “This one trick hated by doctors to lose weight” and similar, then I think there is more than a little bit of a problem.

But you can get worse than this! Some months ago I was traveling to London, and an acquaintance of mine shared on Facebook an article he wrote for an Italian newspaper (since he’s still living around where I’m from.) Since I was curious about the topic, I looked at it and … well, you can see it by yourself:

An example of sponsored content on an Italian local newspaper. Half of the links are borderline scams like "How to make $367/Day Without A Job".

Two things are kind of obvious when looking at it: “Make ¤NNN a day” scams are freaking common not only in comment-spam, and people really seem to believe you can look 30 years younger by buying something. Out of eight “links”, only half actually point back to the newspaper, two point to possibly fake cosmetics (from two “different” sites — which are clearly the same), and two points to outright scams that suggest you can make money without doing anything (these reporting the same site name at least.) It’s also apparent that those two sets are auto-generated by taking a set of stock images, a set of stock headline templates, and throwing different currency symbols, numbers and country names.

Now you may ask why a newspaper – one for which a friend of mine even writes! – would use such a blatantly scammy ad network. The answer is that they did not realize it was a scammy network until I showed him the screenshot. Indeed, from within Italy their ads are useless, but at least legit; it isn’t until you’re visiting from the outside that they start providing you with scam. This is, by the way, why sometimes you may find spam that simply links to a blog post of a newspaper or other site in a non-English language: they still want you to “see” these ads, if they are the only thing you understand in the page, that’s still okay. If you don’t know better, you may still fall for it.

There are more cases, but these are the major ones. So if I see any of these scammy ads, I just go and enable AdBlock for the whole domain. Usually, I also try to stay away from that website altogether, but sometimes it’s not as easy. For instance Wikia – yes, headed by the same Jimmy Wales that keeps insisting he doesn’t want ads on Wikipedia by putting a 50%-height banner of his face on it from time to time – uses the medium-grade scammy Taboola — it’s not quite outright illegal activity, but clearly it’s not something I care to see. So there goes AdBlock.

In addition to the actual scammy, I enable AdBlock plus if I see other ads that, whether legit or not, are just an active pain in the arse. For instance, some sites, particularly I noted around hardware reviews, use ad networks that hook on-hover ads to words. So if you’re like Randall and me and go on selecting text to remember where you were reading if you’re distracted, you may end up playing one of their stupid (sometimes scammy, sometimes not) ads. Bam. Auto-playing video ads with audio gets the AdBlock hammer too. Bam. And so do those sites that just get my CPU to spin though it’s not obvious there is any ad playing already. Bam.

So with all this explained, let me go back to uBlock Origin, which seems to be the only alternative to AdBlock Plus that is ever suggested. This extension is clearly written by privacy extremists. I already had a couple of times people replying to my complaints about it on twitter trying to be funny with “well, that’s intended” or “I don’t see a problem” — that does not make you smart, that makes you completely tone-deaf.

The extension does not only block ads, but it keeps insisting it wants to block all the client-side tracking. As I said before there is still plenty of space for server-side tracking, particularly for malicious purposes; client-side tracking is usually done for marketing purposes, and so I don’t really mind it.

It goes beyond that. The rulesets in uBlock Origin are designed to block based on regular expressions; some of these expressions are of significantly wide reach, for instance when I tried it I couldn’t even go and check my own AdSense console. Or even access SourceForge! — as much as I really disliked SourceForge’s turning to bundling malware last year, marking the whole site off-limits is crazy.

More bothersome for me, was the way the extension decided that any of the tracking-click from Skymiles Shopping were ads and so just decided it was a good thing to block them. For those who don’t know Skymiles Shopping, or one of its many other incarnation for hotels, airlines and other loyalty programs, it’s essentially a way to bridge the referral system of various online shopping venues with your own interests, pretty much the same as Socialvest used to do. When you click on a given offer from the portal, they ask you for your loyalty identifier (in my case a Delta SkyMiles frequent flyer number), then send you to the shopping site with a personalized tracker. After you order from the site, they get a referral commission, and credit you with something — in the case of Socialvest back in the days, you could donate that to non-profits, or get it added to your Flattr wallet, in the case of Skymiles Shopping, they give you a number of Delta rewards miles.

Am I trading part of my privacy away for some benefit? Yes. I’m okay with that, as I said. And so is, very likely, the majority of people out there. So without providing an option to disable this behaviour, and insisting that it’s the correct one, the only way they can read it is that the extension is not for them, and they will fallback to either the (possibly shady) AdBlock Plus, or to no extension whatsoever — and with badvertising being an actual problem, that’s not good either.

For you it might be that your privacy is just that valuable, but there are indeed enough people for which these cash-back, custom tailored offers, or generally legit, non-scammy ads are important. It’s not far from the toilet paper problem.

Indeed, this kind of tone-deaf response from many privacy and Free Software activists is what turned me significantly away from the movement over the past few months. I plan on writing more of it, but I thought this would be a good place to start.

Tinderbox and expenses

I’ve promised some insight into how much running the tinderbox actually costed me. And since today marks two months from Google AdSense’s crazy blacklisting of my website, I guess it’s a good a time as any other.

SO let’s start with the obvious first expense: the hardware itself. My original Tinderbox was running on the box I called Yamato, which costed me some €1700 and change, without the harddrives, this was back in 2008 — and about half the cost was paid with donation from users. Over time, Yamato had to have its disks replaced a couple of times (and sometimes the cost came out of donations). That computer has been used for other purposes, including as my primary desktop for a long time, so I can’t really complain about the parts that I had to pay myself. Other devices, and connectivity, and all those things, ended up being shared between my tinderbox efforts and my freelancing job, so I also don’t complain about those in the least.

The new Tinderbox host is Excelsior, which has been bought with the Pledgie which got me paying only some $1200 of my pocket, the rest coming in from the contributors. The space, power and bandwidth, have been offered by my employer which solved quite a few problems. Since now I don’t have t pay for the power, and last time I went back to Italy (in June) I turned off, and got rid of, most of my hardware (the router was already having some trouble; Yamato’s motherboard was having trouble anyway, I saved the harddrive to decide what to do, and sold the NAS to a friend of mine), I can assess how much I was spending on the power bill for that answer.

My usual power bill was somewhere around €270 — which obviously includes all the usual house power consumption as well as my hardware and, due to the way the power is billed in Italy, an advance on the next bill. The bill for the months between July and September, the first one where I was fully out of my house, was for -€67 and no, it’s not a typo, it was a negative bill! Calculator at hand, he actual difference between between the previous bills and the new is around €50 month — assuming that only a third of that was about the tinderbox hardware, that makes it around €17 per month spent on the power bill. It’s not much but it adds up. Connectivity — that’s hard to assess, so I’d rather not even go there.

With the current setup, there is of course one expense that wasn’t there before: AWS. The logs that the tinderbox generates are stored on S3, since they need to be accessible, and they are lots. And one of the reasons why Mike is behaving like a child about me just linking the build logs instead of attaching them, is that he expects me to delete them because they are too expensive to keep indefinitely. So, how much does the S3 storage cost me? Right now, it costs me a whopping $0.90 a month. Yes you got it right, it costs me less than one dollar a month for all the storage. I guess the reason is because they are not stored for high reliability or high speed access, and they are highly compressible (even though they are not compressed by default).

You can probably guess at this point that I’m not going to clear out the logs from AWS for a very long time at this point. Although I would like for some logs not to be so big for nothing — like the sdlmame one that used to use the -v switch to GCC which causes all the calls to print a long bunch of internal data that is rarely useful on a default log output.

Luckily for me (and for the users relying on the tinderbox output!) those expenses are well covered with the Flattr revenue from my blog’s posts — and thank to Socialvest I no longer have to have doubts on whether I should keep the money or use it to flattr others — I currently have over €100 ready for the next six/seven months worth of flattrs) Before this, between my freelancer’s jobs, Flattr, and the ads on the blog, I would also be able to cover at least the cost of the server (and barely the cost of the domains — but that’s partly my fault for having.. a number).

Unfortunately, as I said at the top of the post, there no longer are ads served by Google on my blog. Why? Well, a month and a half ago I received a complain from Google, saying that one post of mine in which I namechecked a famous adult website, in the context of a (at the time) recent perceived security issue, is adult material, and that it goes against the AdSense policies to have ads served on a website with adult content. I would still argue that just namechecking a website shouldn’t be considered adult content, but while I did submit an appeal to Google, a month and a half later I have no response at hand. They didn’t blacklist the whole domain though, they only blacklisted my blog, so the ads are still showed on Autotools Mythbuster (which I count to resume working almost full time pretty soon) but the result is bleak: I went down from €12-€16 a month to a low €2 a month due to this, and that is no longer able to cover for the serve expense by itself.

This does not mean that anything will change in the future, immediate or not. This blog for me has more value than the money that I can get back from it, as it’s a way for me to showcase my ability and, to a point, get employment — but you can understand that it still upsets me a liiiittle bit the way they handled that particular issue.

A story of a Registry, an advertiser, and an unused domain

This is a post that relates to one of my dayjobs, and has nothing to do with Free Software, yet it is technical. If you’re not interested in non-Free Software related posts, you’re suggested to skip this altogether. If you still care about technical matters, read on!

Do you remember that customer of mine that almost never pays me in time, for which I work basically free of charge, and yet gives me huge headaches from time to time with requests that make little to no sense? Okay you probably remember by now, or you simply don’t care.

Two years or so ago, that customer calls me up one morning asking me to register a couple of second-level domains in as many TLDs as I thought it made sense to, so that they could set up a new web-end to the business. Said project still hasn’t delivered, mostly because the original estimate I sent the customer was considered unreasonably expensive, and taking “too much time” — like they haven’t spent about the same already, and my nine months estimate sounds positively short when you compare it with the over two years gestation the project is lingering on. At any rate, this is of no importance to what I want to focus on here.

Since that day, one set of domains was left to expire as it wasn’t as catchy as it sounded at first, and only the second set was kept registered. I have been paid for the registration of course, while the domains have been left parked for the time being (no they decided not to forward them to the main domain of the business where the address, email and phone number are).

The other day I was trying to find a way to recover a bit more money out of this customer and, incidentally, this blog, and I decided to register to AdSense again, this time with my VAT ID as I have to declare eventual profits coming from that venue. One of the nice features of AdSense allows to “monetize” (gosh how much I hate that word!) parked domains. Since these are by all means parked domains, I gave it a chance.

Four are the domains parked this way: .net, .com, .eu and .it. All registered with OVH – which incidentally has fixed its IPv6 troubles – and up to now all pointing to a blackhole redirect. How do you assign a parked domain to Google’s AdSense service? Well, it’s actually easy: you just have to point the nameservers for the domain to the four provided by Google, and you’re set. On three out of four of the TLDs I had to deal with.

After setting it up on Friday, as of Monday, Google still wouldn’t verify the .it domain; OVH was showing the task alternatively as “processing” and “completed” depending on whether I looked at the NS settings (they knew they had a request to change them) or at the task’s status page (as it’ll be apparent in a moment, it was indeed cloesd). I called them — reason I like OVH: I can get somebody on the phone to eat least listen to me.

What happens? Well, looks like Registro.it – already NIC-IT, the Italian Registration Authority – is once again quite strict in what it accepts. It was just two years ago that they stopped requiring you to fax an agreement to actually be able to register a .it domain, and as of last year you still had to do the same when transferring the domain. Luckily they stopped requiring both, and this year I was able to transfer a domain in the matter of a week or so. But what about this time?

Well, it turns out that the NIC validates the new nameservers when you want to change them, to make sure that the new servers list the domain, and configure it properly. This is common procedure, and both the OVH staff and me were aware of this. What we weren’t aware of (OVH staffers had no clue about this either, they had to call NIC-IT to see what the trouble was, they weren’t informed properly either) is the method they do that: using dig +ANY.

Okay, it’s nothing surprising actually, dig +ANY is the standard way to check for a domain’s zone at a name server… but turns out that ns1.googleghs.com and its brothers – the nameservers you need to point a domain to, for use with AdSense – do not support said queries, making them invalid in the eyes of NIC-IT. Ain’t that lovely? The OVH staffer I spoke with said they’ll inform NIC-IT about the situation, but they don’t count on them changing their ways and … I actually have to say that I can’t blame them. Indeed I don’t see the reason why Google’s DNS might ignore ANY queries.

For my part, I told them that I would try to open a support request with Google to see if they intend to rectify the situation. The issue here is that, as much as I spent trying to find that out, I can’t seem to find a place where to open a ticket for the Google AdSense staff to read. I tried tweeting to their account, but it seems like it didn’t make much sense.

Luckily there is an alternative when you can’t simply set up the domain to point to Google’s DNS, and that is to create a custom zone, which is what I’ve done now. It’s not much of a problem, but it’s still bothersome that one of Google’s most prominent services is incompatible with a first-world Registration Authority such as NIC-IT.

Oh well.