Oh Gosh, Trying to Find a New Email Provider in 2020

In the year 2020, I decided to move out of my GSuite account (née Google Apps for Business), which allowed me to use Gmail for my personal domain, and that I have used for the past ten years or so. It’s not that I have a problem with Gmail (I worked nearly seven years for Google now, why would it be a problem?) or that I think the service is not up to scratch (as this experience is proving me, I’d argue that it’s still the best service you can rely upon for small and medium businesses — which is the area I focused on when I ran my own company). It’s just that I’m not a business and the features that GSuite provides over the free Gmail no longer make up for the services I’m missing.

But I still wanted to be able to use my own domain for my mail, rather than going back to the standard Gmail domain. So I decided to look around, and migrate my mail to another paid, reliable, and possibly better solution. Alas, the results after a week of looking and playing around are not particularly impressive to me.

First of all I discarded, without even looking at it, the option of self-hosting my mail. I don’t have the time, nor the experience, nor the will to have to deal with my own email hosting. It’s a landmine of issues and risks and I don’t intend to accept them. So if you’re about to suggest this, feel free to not comment. I’m not going to entertain those suggestions anyway.

I ended up looking at what people have been suggesting on Twitter a few times and evaluated two options: ProtonMail and FastMail. I ended up finding both lacking. And I think I’m a bit more upset with the former than the latter, for reasons I’ll get to in this (much longer than usual) blog post.

My requirements for a replacement solution were to have a reliable webmail interface, with desktop notifications. A working Android app. And security at login. I was not particularly interested in ProtonMail’s encrypt-and-sign everything approach, but I could live with that. But I wanted something that wouldn’t risk letting everyone in with just a password, so 2FA was a must for me. I was also hoping to find something that would make it easy to deal with git send-email, but I ended up accepting right away that nothing would be anywhere close to the solution that we found with Gmail and GSuite (more on that later.)

Bad 2FA Options For All

So I started by looking at the 2nd Factor Authentication options for the two providers. Google being the earliest adopter of the U2F standard means of course that this is what I’ve been using, and would love to keep using once I replace it. But of the two providers I was considering, only FastMail stated explicitly it supported U2F. I was told that ProtonMail expects to add support for it this year, but I couldn’t even tell that from their website.

So I tried first FastMail, which has a 30 days free trial. To set up the U2F device, you need to provide a phone number as a recovery option — which gets used for SMS OTP. I don’t like SMS OTP because it’s not really secure (in some countries taking over a phone number is easier than taking over an email address), and because it’s not reliable the moment you don’t have mobile network services. It’s easy to mistake the “no access to mobile network” with “no access to Internet” and say that it doesn’t really matter, but there are plenty of places where I would be able to reach the Internet and not receive SMS: planes, tube platforms, the office when I arrived in London, …

But surely U2F is enough, why am I even bothering complaining about SMS OTP, given that you can disable it once the U2F security key is added? Well, turns out that when I tried to login on the Android app, I was just sent an SMS with the OTP to log myself in. Indeed, after I removed the phone number backup option, the Android app threw me a lovely error of «U2F is your only two-step verification method, but this is not supported here.» On Android, which can act as an U2F token.

As I found out afterwards, you can add a TOTP app as well, which solves the issue of logging in on Android without mobile network service, but by that point I had already started looking at ProtonMail, because it was not the best first impression to start with.

ProtonMail and the Bridge of Destiny

ProtonMail does not provide standard IMAP/SMTP access, because encryption (that’s the best reason I can get from the documentation, I’m not sure at all what this was all about, but honestly, that’s as far as I care to look into it). If you want to use a “normal” mail agent like Thunderbird, you need to use a software, accessible to paying customers only, that acts as “bridge”. As far as I can tell after using it, it appears to be mostly a way to handle the authentication rather than the encryption per se. Indeed, you log into the Bridge software with username, password and OTP, and then it provides localhost-only endpoints for IMAP4 and SMTP, with a generated local password. Neat.

Except it’s only available in Beta for Linux, so instead I ended up running it on Windows at first.

This is an interesting approach. Gmail implemented, many years ago, a new extension to IMAP (and SMTP) that allows using OAuth 2 for IMAP logins. This effectively delegates the login action to a browser, rather than executing it inline in the protocol, and as such it allows to request OTPs, or even supporting U2F. Thunderbird on Windows does work very well with this and even supports U2F out of the box.

Sidenote: Thunderbird seems to have something silly going on. When you add a new account to it, it has a drop-down box to let you select the authentication method (or go for “Autodetect”). Unfortunately, the drop-down does not have the OAuth2 option at all. Even if you select imap.gmail.com as the server — I know hardcoding is bad, but not allowing it at all sounds worse. But if you cheat and give it 12345 as password, and select password authentication just to go through with adding the account, then you can select OAuth 2 as authentication type and it all works out.

Anyway, neither ProtonMail nor FastMail appear to have implemented this authentication method, despite the fact that, if I understood that correctly, it’s supported out of the box on Thunderbird, Apple’s Mail, and a bunch of other mail clients. Indeed, if you want to use IMAP/SMTP with FastMail, they only appear to give you the option to use application-specific passwords, which are a shame.

So why did I need IMAP access to begin with? Well, I wanted to import all my mail from Gmail into ProtonMail, and I though the easier way to do so was going to be through Thunderbird and manually copy the folders I needed. That turned out to be a mistake: Thunderbird crashed while trying to copy some of the content over, and I effectively was spending more time while waiting for it to index anything than instructing it on what to do.

Luckily there’s alternative options for this.

Important Importing Tooling

ProtonMail provides another piece of software, in addition to the Bridge, to paying customers: an Import Tool. This allows you to login to another IMAP server, and copy over the content. I decided to use that to copy over my Gmail content to ProtonMail.

First of all, the tool does not support OAuth2 authentication. To be able to access Gmail or GSuite mailboxes, it needs to use an Application-Specific Password. Annoying but not a dealbreaker for me, since I’m not enrolled in the Advanced Protection Program, which among other things disable “less-secure apps” (i.e. those apps using Application-Specific Passwords). I generated one, logged in, and selected the labels I wanted to copy over, then went to bed, a little, but not much, concerned over the 52 and counting messages that it said it was failing to import.

I woke up to the tool reporting only 32% of around fifty thousands messages imported. I paused, then resumed, the import hoping to getting it unstuck, and left to play Pokémon with my wife, coming back to a computer stuck exactly at the same point. I tried stopping and closing the Import Tool, but that didn’t work, it got stuck. I tried rebooting Windows and it refused to, because my C: drive was full. Huh?

When I went to look into it, I found a 436GB text file, that’s the log from the software. Since the file was too big to open with nearly anything on my computer, I used good old type, and beside the initial part possibly containing useful information, most of the file repeated the same error message about not being able to parse a mime type, with no message ID or subject attached. Not useful. I had to delete the file, since my system was rejecting writes because of the drive being full, but it also does not bode well for the way the importer is written: clearly there’s no retry limit on some action, no log coalescing, and no security feature to go “Hang on, am I DoSing the local system?”

I went looking for tools I could use to sync IMAP servers manually. I found isync/mbsync, which as a slightly annoyance is written in C and needs to be built, so not easy to run on Windows where I do have the ProtonMail bridge, but not something I can’t overcome. When I was looking at the website, it said to check the README for workarounds needed with certain servers. Unfortunately at the time of writing the document, in the Compatibility section, refers to “M$ Exchange” — which in 2020 is a very silly, juvenile, and annoying way to refer to what is possibly still the largest enterprise mail server out there. Yes, I am judging a project by its README the way you judge a book by its cover, but I would expect that a project unable to call Microsoft by its name in this day and age is unlikely to have added support for OAuth2 authentication or any of the many extensions that Gmail provides for efficient listing of messages.

I turned to FastMail to see how they are implementing it: importing Gmail or GSuite content can be done directly on their server side: they require you to provide OAuth2 access to all your email (but then again, if you’re planning to use them as your service provider, you kind of are already doing that). It does not allow you to choose which labels you want to import: it’ll clone everything, even your trash/bin folder. So at the time of writing it’s importing 180k messages. It took a while, and it showed the funny result of saying «175,784 of 172,368 messages imported.» Bonus point to FastMail for actually sending the completion note as an email, so that it can be fetched accordingly.

A side effect of FastMail doing the imports server side is that there’s no way for you to transfer ProtonMail boxes to FastMail, or any other equivalent server with server-side import: the Bridge needs to run on your local system for you to authenticate. It’s effectively an additional lock-in.

Instead of insisting on self-hosting options, I honestly feel that the FLOSS activists should maybe invest a little more thought and time on providing ways for average users with average needs to migrate their content, avoiding the lock-in. Because even if the perfect self-hosting email solution is out there, right now trying to migrate to it would be an absolute nightmare and nobody will bother, preferring to stick to their perfectly-working locked-in cloud provider.

Missing Features Mayhem

At that point I was a bit annoyed, but I had no urgency to move the old email away, for now at least. So instead I went on to check how ProtonMail worked as primary mail interface. I changed MX around, set up the various verification methods, and waited. One of the nice things of migrating the mail provider is that you end up realizing just how many mailing lists and stuff you keep receiving, that you previously just filed away with filters.

I removed a bunch of subscriptions to open source mailing lists for projects I am no longer directly involved in, and unlikely to go back to, and then I started looking at other newsletters and promotions. For at least one of them, I thought I would probably be better served by NewsBlur‘s newsletter-to-RSS interface. As documented in the service itself, the recommended way to use this is to create a filter that takes the input newsletter and forwards them to your newsblur alias.

And here’s the first ProtonMail feature that I’m missing: there’s no way to set up forwarding filters. This is more than a bit annoying: there was mail coming to my address that I used to forward to my mother (mostly bills related to her house, before I set up a separate domain with multiple aliases that point at our two addresses), and there still are a few messages that come to me only, that I forward to my wife, where using our other alias addresses is not feasible for various reasons.

But it’s not just a matter of forwards that is missing. When I looked into the filter system of ProtonMail I found it very lacking. You can’t filter based on an arbitrary header. You cannot filter based on a list-id! Despite the webmail being able to tell that an email came through from a mailing list, and providing an explicit Unsubscribe button, based on the headers, it neither has a “Filter messages like these” like Gmail has, nor a way to select this manually. And that is a lot more annoying.

FastMail, by comparison, provides a much more detailed rules support, including the ability to provide them directly in Sieve language, and it allows forward-and-delete of email as well, which is exactly what the NewsBlur integration needs (although to note, while you can see the interface for do that, trial accounts can’t set up forwarding rules!) And yes, the “Add Rule from Message” flow defaults to the list identifier for the messages. Also, to one-up even Gmail on this, you can set those rules from the mobile app as well — and if you think this is not that big of a deal, just think of much more likely you are to have spare time to do this kind of boring tasks while waiting for your train (if you commute by train, that is).

In terms of features, it seems like FastMail has the clear upper hand. Even ignoring the calendar provided, it supports the modern “Snooze” concept, letting mail show up later in the day or the week (which is great when, say, you don’t want to keep the unread email about your job interviews to show up on your mail inbox at the office), and it even has the ability to permanently delete messages in certain folders on after a certain amount of days — just like gmaillabelpurge! I think this last feature is the one that made me realize I really just need to use FastMail.

Sending It All Out

As I said earlier, even before trying to decide which one of the two providers to try, I gave up on the idea of being able to use either of them with git send-email to send kernel patches and similar. Neither of them supports OAuth2 authentication, and I was told there’s no way to set up a “send-only” environment.

My solution to this was to bite the bullet and deal with a real(ish) sendmail implementation again, by using a script that would connect over SSH to one of my servers, and use the postfix instance there (noting that I’m trying to cut down on having to run my own servers). I briefly considered using my HTPC for that, but then I realized that it would require me to put my home IP addresses in the SPF records for my domain, and I didn’t really want to publicise those as much.

But it turned out the information I found was incorrect. FastMail does support SMTP-only Application Specific Passwords! This is an awesomely secure feature that not even Gmail has right now, and it makes it a breeze to configure Git for it, and the worst that can happen is that someone can spoof your email address, until you figure it out. That does not mean that it’s safe to share that password around, but it does make it much less risky to keep the password on, say, your laptop.

I would even venture that this is even safer than the sendgmail approach that I linked above, as the other one requires full mail access with the token, which can easily be abused by an attacker.

Conclusion

So at the end of this whole odyssey, I decided to stick with FastMail.

ProtonMail sounds good on paper, but it give me the impression that it’s overengineered in implementation, and not thought out enough in feature design. I cannot otherwise see how many basic features (forwarding filters, send-only protocol support — C-S-c to add a CC line) would otherwise be missing. And I’m very surprised about the security angle for the whole service.

FastMail does have some rough edges, particularly on their webapp. Small things, like being able to right-click to get a context menu would be nice. U2F support is clearly lacking: having it work on their Android app for me would be a huge step forward. And I should point out that FastMail has a much friendlier way to test its service, as the 30 days free option includes nearly all of the functionality and enough space to test an import of the data from a 10 years old Gmail.

I’ll stick with Thunderbird still

Even though it hasn’t been an year yet that I moved to KDE, after spending a long time with GNOME 2, XFCE and then Cinnamon, over the past month or so I looked at how much of non-KDE software I could ditch this time around.

The first software I ditched was Pidgin — while the default use of GnuTLS caused some trouble KTP works quite decently. Okay some features are not fully implemented, but the basic chat works, and that’s enough for me — it’s not like I used much more than that on Pidgin either.

Unfortunately, when yesterday I decided to check whether it was possible to ditch Thunderbird for KMail, things didn’t turn out as nice. Yes, the client improved a truckload since what we had at KDE 3 time — but no, it didn’t improve enough for make it usable for me.

The obvious problem zeroth is the dependencies: to install KMail you need to build (but don’t need to enable) the “semantic desktop” — that is, Nepomuk and the content indexing. In particular it brings in Soprano and Virtuoso that have been among the least usable components when KDE4 was launched (at least Strigi is gone with 4.10; we’ll see what the future brings us). So after a night rebuilding part of the system to make sure that the flags are enabled and the packages in place, today I could try KMail.

First problem — at the first run it suggested importing data from Thunderbird — unfortunately it completely stuck there, and after over half an hour it went nowhere. No logs, no diagnostic, just stuck. I decided to ignore it and create the account manually. While KMail tried to find automatically which mail servers to use, it failed badly – I guess it tried to look for some _imap._srv.flameeyes.eu or something, which does not exist – even though Thunderbird can correctly guess that my mail servers are Google’s.

Second problem — the wizard does not make it easy to set up a new identity, which makes it tempting to add the accounts manually, but since you got three different entries that you have to add (Identity, Sending account, Receiving account), adding them in the wrong order gets you to revisit the settings quite a few times. For the curious, the order is sending, identity, receiving.

Third problem — KMail does not implement the Special Folder extension defined in RFC 6154 which GMail makes good use of (it actually implements it both with the standard extension and their own). This means that KMail will store all messages locally (drafts, sent, trash, …) unless you manually set them up. Unlike what somebody have told me, this means that the extension is completely unimplemented, not implemented only partially. I’m not surprised that it’s not implemented, by the way, due to the fact that the folders are declared in two different settings (the identity and the account).

Fourth problem — speaking about GMail, there is no direct way to handle the “archive” action, which is almost a necessity if you want to use it. While this started with GMail and as an almost exclusive to that particular service, nowadays many other services, including standalone software such as Kerio, provide the same workflow; the folder used for archiving is, once again, provided with the special-use notes discussed earlier. Even though the developers do not use GMail themselves, it feels wrong that it’s not implemented.

Fifth problem — while at it, let’s talk a moment about the IDLE command implementation (one of the extensions needed for Push IMAP). As Wikipedia says, KMail implements support for it since version 4.7 — unfortunately, it’s not using it in every case, but only if you disable the “check every X minutes” option — if that is enabled, then the IDLE command is not used. Don’t tell me it’s obvious, because even though it makes sense under some point of views, I wasn’t the only one that was tricked by that. Especially since I read that setting first as “disable if you only want manual check for new mail” — Thunderbird indeed uses IDLE even if you set the scheduled check every few minutes.

Sixth problem — there is no whitelist for remote content on HTML emails. GMail, both web and on the clients, Android and iOS, supports a complete whitelist, separate from everything else. Thunderbird supports a whitelist by adding the sender to the contacts’ list (which is honestly bothersome when adding mailing lists, like in my case). As far as I could tell, there is no way to have such a whitelist on KMail. You either got the protection enabled, or you got it disabled.

The last problem is the trickiest, and it’s hard to tell if it’s a problem at all. When I went to configure the OpenPGP key to use, it wouldn’t show me anything to select at all. I tried for the good part of an hour trying to get it to select my key, and it failed badly. When I installed Kleopatra it worked just fine; on the other hand, Pesa and other devs pointed out that it works for them just fine without Kleopatra installed.

So, what is the resolution at this point, for me? Well, I guess I’ll have to open a few bug feature requests on KDE’s Bugzilla, if I feel like it, and then I might hope for version 4.11 or 4.12 to have something that is more usable than Thunderbird. As it is, that’s not the case.

There are a bunch of minor nuisance and other things that require me to get used to them, such as the (in my view too big) folder icons (even if you change the size of the font, the size of the icon does not change), and the placement of the buttons which required me to get used to it on Thunderbird as well. But these are only minor annoyances.

What I really need for KMail to become my client is a tighter integration with GMail. It might not suit the ideals as much as one might prefer, but it is one of the most used email providers in the world nowadays, and it would go a long way for user friendliness to work nicely with it instead of making it difficult.

I think I’ll keep away from Python still

Last night I ended up in Bizarro World, hacking at Jürgen’s gmaillabelpurge (which he actually wrote on my request, thanks once more Jürgen!). Why? Well, the first reason was that I found out that it hasn’t been running for the past two and a half months, because, for whatever reason, the default Python interpreter on the system where it was running was changed from 2.7 to 3.2.

So I tried first to get it to work with Python 3 keeping it working with Python 2 at the same time; some of the syntax changes ever so slightly and was easy to fix, but the 2to3 script that it comes with is completely bogus. Among other things, it adds parenthesis on all the print calls… which would be correct if it checked that said parenthesis wouldn’t be there already. In a script link the one aforementioned, the noise on the output is so high that there is really no signal worth reading.

You might be asking how comes I didn’t notice this before. The answer is because I’m an idiot! I found out only yesterday that my firewall configuration was such that postfix was not reachable from the containers within Excelsior, which meant I never got the fcron notifications that the job was failing.

While I wasn’t able to fix the Python 3 compatibility, I was able to at least understand the code a little by reading it, and after remembering something about the IMAP4 specs I read a long time ago, I was able to optimize its execution quite a bit, more than halving the runtime on big folders, like most of the ones I have here, by using batch operations, and peeking, instead of “seeing” the headers. At the end, I spent some three hours on the script, give or take.

But at the same time, I ended up having to workaround limitations in Python’s imaplib (which is still nice to have by default), such as reporting fetched data as an array, where each odd entry is a pair of strings (tag and unparsed headers) and each even entry is a string with a closed parenthesis (coming from the tag). Since I wasn’t able to sleep, at 3.30am I started re-writing the script in Perl (which at this point I know much better than I’ll ever know Python, even if I’m a newbie in it); by 5am I had all the features of the original one, and I was supporting non-English locales for GMail — remember my old complain about natural language interfaces? Well, it turns out that the solution is to use the Special-Use Extension for IMAP folders; I don’t remember this explanation page when we first worked on that script.

But this entry is about Python and not the script per-se (you can find on my fork the Perl version if you want). I have said before I dislike Python, and my feeling is still unchanged at this point. It is true that the script in Python required no extra dependency, as the standard library already covered all the bases … but at the same time that’s about it: it is basics that it has; for something more complex you still need some new modules. Perl modules are generally easier to find, easier to install, and less error-prone — don’t try to argue this; I’ve got a tinderbox that reports Python tests errors more often than even Ruby’s (which are lots), and most of the time for the same reasons, such as the damn unicode errors “because LC_ALL=C is not supported”.

I also still hate the fact that Python forces me to indent code to have blocks. Yes I agree that indented code is much better than non-indented one, but why on earth should the indentation mandate the blocks rather than the other way around? What I usually do in Emacs when I’m getting stuff in and out of loops (which is what I had to do a lot on the script, as I was replacing per-message operations with bulk operations), is basically adding the curly brackets in different place, then select the region, and C-M- it — which means that it’s re-indented following my brackets’ placement. If I see an indent I don’t expect, it means I made a mistake with the blocks and I’m quick to fix it.

With Python, I end up having to manage the space to have it behave as I want, and it’s quite more bothersome, even with the C-c < and C-c > shortcuts in Emacs. I find the whole thing obnoxious. The other problem is that, while Python does provide basics access to a lot more functionality than Perl, its documentation is .. spotty at best. In the case of imaplib, for instance, the only real way to know what’s going to give you, is to print the returned value and check with the RFC — and it does not seem to have a half-decent way to return the UIDs without having to parse them. This is simply.. wrong.

The obvious question for people who know would be “why did you not write it in Ruby?” — well… recently I’ve started second-guessing my choice of Ruby at least for simple one-off scripts. For instance, the deptree2dot tool that I wrote for OpenRC – available here – was originally written as a Ruby script … then I converted it a Perl script half the size and twice the speed. Part of it I’m sure it’s just a matter of age (Perl has been optimized over a long time, much more than Ruby), part of it is due to be different tools for different targets: Ruby is nowadays mostly a long-running software language (due to webapps and so on), and it’s much more object oriented, while Perl is streamlined, top-down execution style…

I do expect to find the time to convert even my scan2pdf script to Perl (funnily enough, gscan2pdf which inspired it is written in Perl), although I have no idea yet when… in the mean time though, I doubt I’ll write many more Ruby scripts for this kind of processing..

Productivity improvement weekend

This weekend I’m going to try my best to improve my own productivity. Why do I say that? Well there are quite a few reasons for it. The first is that I spent the last week working full time on feng, rewriting the older code to replace it with simpler, more tested and especially well documented code. This is not an easy task especially because you often end up rewriting other parts to play nicely with the new parts; indeed to replace bufferpool, between me and Luca we rewrote almost entirely the networking code.

Then there is the fact that I finally got a possible price to replace the logic board of my MacBook Pro that broke a couple of weeks ago: €1K! That’s almost as much as a new laptop; sure not the same class, but still. In the mean time I bought an iMac; I needed access to QuickTime, even more than I knew before, because we currently don’t have a proper RTSP client; MPlayer does not support seeking, FFplay is broken for a few problems, and VLC also does not behave in a very standard compliant way. QuickTime is, instead, quite well mannered. But this means I have spent money to go on with the job, which is, well, not exactly the nicest thing you can do if you need to pay some older debts too.

So it means I have to work more; not only I have to continue my work on lscube at full time, but I’m going to have to get more jobs to the side; I got asked for a few projects already, but most seem to require me to learn new frameworks or even new programming languages, which means they require a quite big effort. I need the money so I’ll probably pick them but it’s far from optimal. I’ve also put on nearly-permanent hold the idea of writing an autotools guide, either as an open book or a real book; the former has shown no interest among readers of my blog, the latter has shown no interest among publisher. I start to feel like an endangered species regarding autotools, alas.

But since at least for lscube I need to have access to the FFmpeg mailing list, and I need access to the PulseAudio mailing list for another project and so on so forth, I need to solve one problem I already wrote about, purging GMail labels out of older messages. I really have a need for this to be solved, but I’m still not totally in luck. Thanks to identi.ca, I was able to get the name of a script that is designed to solve the same problem: imap-purge . Unfortunately there is a problem with one GMail quirk: deleting a message from a “folder” (actually a GMail label) does not delete the message from the server, it only detach the label from that message; to delete a message from the server you’ve got to move it to the Trash folder (and either empty it or wait for 30 days so that it gets deleted). I tried modifying imap-purge to do that, but my Perl is nearly non-existent and I couldn’t even grok the documentation of Mail-IMAPClient regarding the move function.

So this weekend either I find someone to patch imap-purge for me or I’ll have to write my own script based on its ideas in Ruby or something like that. Waste of time from one side, but should allow me to save time further on.

I also need to get synergy up to speed in Gentoo, there have been a few bugs opened regarding crashes and other problems and requests for startup scripts and SVN snapshots; I’ll do my best to work on that so that I can actually use a single keyboard and mouse pair between Yamato and the iMac (which I called, with a little pun, USS Merrimac (okay I’m a geek). Last time I tried this, I had sme problems with synergy deciding to map/unmap keys to compensate the keyboard difference between X11 and OSX; I hope I can get this solved this time because one thing I hate is having different key layout between the two.

I also have to find a decent way to have my documents available on both OS X and Linux at the same time, either by rsyncing them in the background or sharing them on NFS. It’s easier if I got them available everywhere at once.

The tinderbox is currently not running, because I wouldn’t have time to review the build logs, in the past eight days I turned on the PlayStation 3 exactly twice, one earlier today to try relaxing with Street Fighter IV (I wasn’t able to), and the other time just to try one thing about UPnP and HD content. I was barely able to watch last week’s Bill Maher episode, and not much more. I seriously lack the precious resource that time is. And this is after I show the thing called “real life” almost entirely out of the door.

I sincerely feel absolutely energy-deprived; I guess it’s also because I didn’t have my after-lunch coffee, but there are currently two salesman boring my mother with some vacuum cleaner downstairs and I’d rather not go meet them. Sigh. I wish life were easy, at least once an year.

Questing for the guide

I was playing some Oblivion while on the phone with a friend, when something came up to my mind, related to my recent idea of an autotools guide . The idea came up in my mind by mixing Oblivion with something that Jürgen was saying this evening.

In the game you can acquire the most important magical items in three ways: you can find them around (rarely), you can build them yourselves (by hunting creatures’ souls), you can pay for them with gold, or you can get them during quests. The latter are usually the most powerful but it’s not always true. At any rate, the “gold” option is rarely the one used because it’s a somewhat scarce resource. You might start to wonder what this has to do with the autotools guide that I’ve made public yesterday, but you might also have already seen where I’m going.

Since I’m the first one to know that money, especially lately, is a scarce resource, and that, me first, I’m the kind of person who’s glad to put in an effort with a market value three/four times more than whatever money I could afford to repay a favour, it would be reasonable for me to provide a way of “payment” through use of technical skills and effort.

So here is my alternative proposal: if you can get me a piece of code that I failed to find and I don’t have time to write, releasing it under a FOSS license (GPLv2+ is very well suggested; compatibility with GPL is very important anyway), and maintaining it until it’s almost “perfect”, I’ll exchange that for a comparable effort in extending the guide.

I’ll post these “quests” from time to time on the blog so you can see them and see whether you think you can complete them; I’ll have to find a way to index them though, for now it’s just a proposal so I don’t think I need to do this right away. But I can drop two ideas if somebody has time and is willing to work on them; both of them relate to IMAP and e-mail messages, so you’ve been warned. I’m also quite picky when it comes to requirements.

The first, is what Jürgen was looking at earlier: I need a way to delete the old messages from some GMail label every day. The idea is that I’d like to use GMail for my mailing lists needs (so I have my messages always with me and so on), but since keeping the whole archive is both pointless (there is gmane, google groups, and the relative archives) and expensive (in term of space used in the GMail IMAP account and of bandwidth needed to sync “All Mail”, via UMTS), I’d like to just always keep the last 3 weeks of e-mail messages. What I need, though, is something slightly more elaborated than just deleting the old messages. It has to be a script that I can run on a cron job locally, and connects to the IMAP server. It has to allow deleting the messages completely from GMail, which means dropping them in the Trash folder (just deleting them is not enough, you just remove the label), and emptying it too; it also has to be configurable on a per-label basis of time to keep the messages (I would empty the label with the release notifications every week rather than every three weeks), and hopefully be able to specify to keep unread messages longer, and consider flagged messages as protected. I don’t care much about implementation language but I’d frown up at things “exotic” like ocaml, smalltalk and similar since it would require me to install their environment. Perl, Python and Ruby all are fine, and Java is too since the thing would run just once a day and is not much of a slowdown to start the JVM for that. No X connection though.

The second is slightly simpler and could be coupled with the one before: I send my database backups from the server to my GMail e-mail address, encrypted with GPG and compressed with BZip2, and then split in message-sized chunks. I need a way to download all the messages and reassemble the backups, once a week, and store it on a flash card, using tar directly on it like it was a tape (no need for a filesystem should reduce the erase count). The email messages have the number of the chunk, the series of the backup (typo or bugzilla) and the date of backup all encoded in the subject. More points if it can do something like Apple’s Time Machine to keep backups each day for a week, each week for a month (or two) and then a backup a month up to two years.

So if somebody has the skill to complete these tasks and would be interested in seeing the guide expanded, well, just go for it!

The mailing lists problem

For a while I’ve been using GMane for almost all the mailing list I’m following. This made it much easier to deal with it because it mean that I didn’t have to download a local copy of all the messages and it also allowed me for a much cleaner access to archives. Unfortunately this has a downside, as it requires me to use an NNTP client, which is something that hasn’t been much cool to do lately.

In the last few few months I’ve used gnus as my client, but the problem with that is that it still needs a separate software from Evolution (which is my mail client0, and it had the nasty issue of not saving the read messages when emacs closed unexpectedly, say when X died, and because of a bug, emacs daemon died with it.

Now of course I could use mailing lists like most of the other people in the world do, by receiving them on my mail account, but I’d rather not fill my GMail “All Mail” folder with all the messages coming from mailing list, especially for when I have to access that data from an UMTS connection where I pay the traffic.

What other solutions are there for me? I’m considering the idea of doing what I did when there was a limit of 20MB on an email account from my provider, but no limit on the number of accounts, having one account per mailing list. Now the limit is 1GB but I haven’t been using those accounts in quite a long time. Even if they are available in IMAP when I’m on their connection they are only available through webmail from outside; not like that’s too much of an issue for me, since I need mailing list mostly when I’m not around hospitals or stuff like that, so I could just use that.

For this to work, though, I need a few tools working on IMAP, for instance I’d need a script that could expunge old archives, when the threads haven’t been updated in the last few months; I’ll also need a script that would automatically filter the mailing list as they arrive in Inbox (waiting with the IDLE command), since my provider does not allow me to filter the messages server-side. And such a script will have to run on Yamato since it has to be on their network.

Then there is the problem that Evolution saves its cache in my home directory, which is under RAID6; there is no need for the cache to be on my home, when XDG_CACHE_DIR is pointing to a private subdirectory in /var/cache. This includes the 200MB of SQLite database that is currently using. Does anybody know if there is a way to get Evolution to respect the XDG_CACHE_DIR variable?

GMail and Viruses

I’m an almost happy user of GMail for quite a while, almost because it still doesn’t work properly with my browser of choice. On the other hand I can use it through IMAP so I’m happier than with my own ISP’s mailserver which is reachable only from inside their network.

One thing I can’t really get, though, is how comes that I receive tons of viruses on GMail. Don’t get me wrong, I do like the spam filter that they provide and it’s really nice, even though I do receive huge amount of spam every day (when I used to use GMail through POP and a local IMAP server, I received a good 100 e-mail messages a day that were unfiltered by GMail and detected by SpamAssassin).

From time to time a new rush of spam arrives, and GMail is not prepared to filter it, but probably because of the huge pool of users marking the stuff as spam, it is usually taken care of quickly. But viruses don’t seem to get filtered that easily.

Since last week I received a number of viruses – or at least I assume they are viruses – with “Hi” or “Re: Hi” or similar subjects, a few random lines and then an attachment. They still arrive even today.

The funny thing is that when this kind of problems arise on a mailbox I fetch with GMail from a POP server, they are able to filter it out, although they then send me a mail telling me the message was left there for that reason, which is just as annoying.

I’d like to know what is up with GMail and viruses, maybe it’s just a method to remember that you should be using a decent operating system and/or pay money for an antivirus (up to you decide what is better ;)).

Claws and IMAP

So what is the cool thing of the week for those who weren’t waiting Mac OS X Leopard already? Of course is Google providing IMAP support for GMail.

Indeed it was a feature I also was waiting for a long time: I’m running my own IMAP server for years now, but that’s a bit of a hassle as I need to keep Enterprise online to get it to work and so on. Well, now I can move everything on GMail and just use that, right? Well, somewhat.

While GMail’s IMAP server works perfectly fine from the Nokia E61 and from Apple’s Mail on my MacBook Pro, it does not work correctly with Claws Mail, and I have no idea why yet.

Sniffing the traffic to know what’s going on is a bit difficult considering that the whole conversation is over SSL, so I will have to wait for Ticho to respond to my ping so I can ask him how to debug it.

On the other hand, I have to say I’m quite full of spam on my GMail account, even if their antispam filter is quite good on its own. Unfortunately up to now I used GMail mostly with POP3 and fetchmail (beside the time when Enterprise was down of course), so even when the spam passed through the filter, it was often filtered out by SpamAssassin, and trying to clean it up from GMail’s All Mail was a long and boring task (I did it a few times for the first 20 pages or so then I stopped because it was too boring and too long). Marking stuff as spam with Apple’s Mail and IMAP instead sounds easier: I can just change the order of the mails so that the ones with priority set are grouped, or group them by sender or subject, then I can select all of them and just move them on the Spam folder. Quick and easy, mostly. The problem is that I gave order to Apple’s Mail to move more than two thousands messages, and it’s taking some time to give the orders to the server.

And today is a long day..

Self-signed certificates on the E61: failed.

I’m not sure if you remember, but last Christmas I blogged about my need to find a way to allow my smartphone (a Nokia E61) to automatically accept the certificate file for my IMAP server, to avoid having to accept it every time when I want to check my mails.

Well, I originally thought it was just a problem of format, as a lot of places on the net talked about the need to convert the PEM certificate to a DER certificate, and then download it with the internal browser from a properly-configured webserver (properly meaning the certificate is served with the mime-type application/x-509-ca-cert), but then it wasn’t enough, I thought it was a problem of size of the certificate (over 1024 bits it doesn’t seem to be supported) but it wasn’t the case either, and the last idea I had was that the problem was with the firmware version.

Unfortunately last month when I asked a friend of mine to help me updating the firmware on the phone, the download process on his laptop (with Windows XP) took more than three hours, then we shut everything down, as it was coming late, and the download wasn’t complete yet. I tried upgrading through a (demo) version of Parallels Desktop for Mac OS X, but it didn’t help either, today I tried on vmware-server, and there it worked fine, without the problems with disconnection and reconnection of the USB device that Parallels suffered from (probably while the device was being reconfigured to provide a different interface to check the phone model, serial and firmware). The firmware upgrade process went fine and easy, in about thirty minutes just like Nokia wrote on the site, I’m not sure what was wrong with my friend’s laptop, nor I care much at this point, if I have to be blunt.

But, not even the firmware update helped me.. so I decided to investigate further: the posts on the web about the configuring the E61 (or the USA-marketed E62, which should just be a crippled version of mine) mixes information about the addition of extra Certificate Authority certificates with requests for help with self-signed certificate. As soon as I considered this, light shed into my mind and I found the problem: the Nokia E61 does not support self-signed certificates; this is an absolute, and there doesn’t seem to be a way to get around this. What you can do instead is create your own Certificate Authority, load that certificate on the smartphone, and then use it to generate your own SSL certificate; this solved the problem for me entirely.

If you need an easy way to build a certificate authority certificate, and to create a new certificate signed with that, you might want to look in the openvpn’s sources’ tarball, that contains an easy-rsa directory with a series of scripts that really helps on that matter.

By the way, the reason why I updated firmware was also to enable the VoIP features, so that I can have a landline-like number routed on my phone while I’m at home (or anywhere where a WLAN can be found) allowing me to receive calls directly without passing through my family’s number.

Server di posta locale con Fetchmail e Dovecot

Introduzione

Sempre più spesso i provider cominciano a permettere l’accesso alle
proprie caselle di posta elettronica tramite i server POP3 ed IMAP solo
quando si è connessi tramite loro, permettendo l’accesso dall’esterno
solo tramite la spesso scomoda WebMail.

Per poter aggirare queste restrizioni, sono stati ideati progetti
quali FreePOPs che si
occupa del parsing delle pagine della webmail simulando un server POP3
locale. Per quanto questa soluzione in genere funzioni, non è
sicuramente la più performante, e in alcuni casi può essere abbastanza
inutile. Per esempio quando effettivamente si utilizza una connessione
casalinga col provider in oggetto, ma si ha a disposizione un
portatile con cui ci si collega anche dall’esterno.

Questo articolo vuole proporre una soluzione a questo problema,
utilizzando software libero, senza andare ad “abusare” dei
servizi forniti dai provider. Il risultato sarà un accesso illimitato
alla propria posta da ovunque si vorrà.

Storia

Questo articolo deriva dal mio originale “Server di posta locale con
Fetchmail e Courier IMAP”, ma utilizza, anziché Courier IMAP, il server
di posta Dovecot.

La ragione di questa modifica è che io stesso sono passato dall’usare
Courier ad usare Dovecot, principalmente perché mi ero stufato dei
problemi che Courier continuava a darmi con Apple’s Mail e dei
rallentamenti al caricamento di una cartella con KMail.

Il cambiamento ha anche avuto l’effetto secondario di permettermi di
avere una migliore organizzazione delle cartelle di posta, che ora non
sono più tutte sottocartelle della Inbox, cosa che ha reso più semplice
usare Mail e KMail.

La differenza tra “Fetchmail e Courier IMAP” e “Fetchmail e Dovecot” non
sta solo nella sezione di configurazione del server IMAP. Ho anche
modificato le ragioni per la scelta del server IMAP, e migliorata la
descrizione per il setup della maildir.

Protocolli di posta

Senza voler scendere nel dettaglio tecnico, possiamo dire che esistono
principalmente tre protocolli che si occupano di posta elettronica:
SMTP, POP3 e IMAP4.

Mentre il primo si occupa dell’invio e del transito della posta fino al
server del provider di destinazione, gli ultimi due sono stati creati
per permettere la lettura della posta elettronica con dei client di
posta, e come si può notare dalla data di pubblicazione, sono molto più
recenti del primo. Vedremo più avanti il motivo di questo distacco.

L’invio e il transito della posta

Quando si invia un messaggio di posta elettronica tramite un client,
solitamente questo si connette al server “Posta in uscita”
del proprio provider, ovvero al server SMTP. Tale server è solitamente
chiamato mail.provider.tld, e sempre più spesso tale nome viene in
realtà definito come un round-robin1 di server.

In ascolto sul server del provider c’è un MTA (Mail Transfer Agent)
ovvero un software che verifica a quale server il messaggio va inoltrato
(verificando il dominio nell’indirizzo del destinatario). Se il server
di destinazione è il server stesso (quindi si è inviata una mail ad un
utente dello stesso provider), il messaggio passa alla seconda fase,
quella di consegna (illustrata nel prossimo paragrafo).

Se invece il messaggio deve essere consegnato ad un altro server, l’MTA
contatta il server definito nel record DNS del dominio di destinazione
(il cosiddetto mail exchanger) e inoltrerà il messaggio all’MTA del
nuovo server, che svolgerà la stessa operazione appena vista.

La consegna della posta

Una volta che il messaggio è stato ricevuto dall’MTA del server del
destinatario, il suo tragitto sugli MTA è (solitamente) finito. A questo
punto il software si occupa di passare il messaggio ad un nuovo
programma, chiamato MDA (Mail Delivery Agent).

Alcuni MTA implementano anche l’MDA nello stesso programma, ma, come
ogni utente di sistemi Unix-Like sa, un software specializzato spesso
può fornire molte più opzioni, quindi praticamente ogni MTA moderno
permette di sostituire il proprio MDA con un altro software. Nel caso di
Courier-MTA, l’MDA fornito è MailDrop, disponibile anche come software a
se stante, ma molto spesso questo viene accoppiato con procmail.

Formati di consegna

Sui sistemi Unix classici le mail venivano salvate all’interno di file
in formato mbox, ovvero un unico file di testo (per ogni utente) dove le
mail erano separate da una riga specifica.

Tale formato però soffre di gravi problemi: il file deve essere posto
sotto lock per mantenerne l’integrità (evitando che una
lettura/scrittura concorrente faccia perdere dei messaggi), aumenta
considerevolmente di dimensioni quando arrivano molti messaggi, non
permette un’organizzazione gerarchica (in cartelle) della posta, e
utilizzando un solo file, nel caso in cui questo sia danneggiato da un
fault del filesystem tutti i messaggi andrebbero persi.

Per evitare ciò, D. J. Bernstein, l’autore di QMail, ha introdotto il
formato Maildir, in cui ogni messaggio viene salvato in
un file separato all’interno di una struttura di directories che
permette anche l’archiviazione gerarchica. Tale formato viene ora
utilizzato da molti altri software, sia server che client, come Courier,
Procmail, KMail ed Evolution.

Directory di consegna

Anche il posto dove le mail vengono consegnate non è sempre lo stesso.
In genere sono due le directory dove la posta viene salvata per la
consegna: nei sistemi in cui viene gestita la posta solo per gli utenti
del sistema, questa viene solitamente consegnata all’interno delle home
directories degli utenti stessi, in un file mbox oppure nella
directory Maildir.

Mentre nei server dei provider, dove vengono gestite utenze email che
non corrispondono alle utenze del sistema (oppure vengono gestiti più
domini sullo stesso server, tramite virtual domains), vengono
solitamente salvati in una struttura di directory su /var/spool/mail.

La lettura della posta

Nei primi anni della grande rete, quando la posta elettronica veniva
solitamente usata dagli utenti locali dei computer, per leggere la posta
bastava utilizzare un client che andasse a leggere il proprio mbox,
sulla macchina locale (o eventualmente dallo share NFS del server di
posta nel caso di terminali, per esempio nelle universtità).

Col passare degli anni però la posta elettronica è diventata un servizio
offerto al pubblico da grandi provider, che ovviamente non potevano
fornire utenze all’interno dei propri server per poter leggere la posta
direttamente. Per risolvere questo problema, venne ideato il protocollo
POP (Post Office Protocol), la cui prima incarnazione (RFC 918) vede la
luce nell’ottobre del 1984. A questa seguirono altre due versioni, fino
a giungere all’attuale versione 3.

Parallelamente al protocollo POP, nacque un protocollo appositamente
pensato per i client moderni, l’IMAP (originariamete Internet Mail
Access Protocol
, attualmente Interactive Message Access Protocol),
che fornisce diverse funzionalità non presenti nel protocollo POP.

Post Office Protocol

Il protocollo POP3 è attualmente il protocollo più diffuso per l’accesso
alla posta elettronica. Tramite questo protocollo, il client si connette
al server e scarica tutti o una parte dei messaggi disponibili per la
consultazione fuori linea. Può inoltre cancellare un messaggio o
semplicemente dire al server che tale messaggio è già letto (per evitare
di riscaricarlo).

Quando si utilizza questo protocollo e si possiede più di un computer,
il metodo più semplice per poter leggere le email da entrambi è
impostare uno dei client per cancellare tutti i messaggi, e gli altri
per non farlo. A questo punto la posta deve però essere scaricata da
tutti i client in un ben preciso ordine se si vuole che la posta sia
disponibile su tutti i client. Allo stesso tempo, se un messaggio
arrivasse mentre è in corso la sequenza di download della posta, non
sarebbe disponibile su alcune macchine.

Interactive Message Access Protocol

Per poter risolvere i problemi del protocollo POP, è stato ideato il
protocollo IMAP, che cambia sostanzialmente l’approccio che i client di
posta hanno rispetto al server.

In un server IMAP, il server fa più che fornire la possibilità di
scaricare i messaggi ai client, ma si occupa di tenere archiviati i
messaggi, organizzandoli gerarchicamente (in cartelle), similmente a
come i client moderni gestiscono la posta salvata localmente.

Tra i servizi aggiuntivi che i server IMAP possono fornire, c’è la
possibilità di ordinare i messaggi per thread (discussione) server-side,
l’ordinamento per un determinato parametro sempre server-side, la
ricerca server-side, e la richiesta di una singola parte di un messaggio
(per esempio si possono scaricare tutti gli header di una cartella, poi
decidere di quali si vuole scaricare il corpo del messaggio, e intanto
eliminare i messaggi che già si vedono non desiderati, per finire si
possono scaricare solo gli allegati che ci interessano).

Questo protocollo, tenendo archiviati i messaggi lato server, è
sicuramente la scelta più comoda per poter accedere alla posta da più di
un computer, o da più di un client, ma soffre di un grave problema: i
messaggi restando sul server continuano ad occupare spazio sulla
mailbox, che finirebbe per riempirsi in breve tempo.

Si tratta alla fine di un metodo simile a quello utilizzato dalle
webmail che, anzi, molto spesso utilizzano proprio IMAP4 per l’accesso
ai messaggi.

Il problema e la soluzione

Come abbiamo visto, uno dei problemi maggiori che si possono avere
quando si possiede più di un computer, ed uno è un portatile o è per
esempio nel proprio ufficio (o in ogni caso in una rete diversa da
quella del proprio provider) è quello di accedere alla posta che si
trova nei server del provider.

Un altro problema è il fatto che utilizzando POP3 non è affatto semplice
avere la posta a disposizione su tutti i client per poterla leggere con
calma, mentre con IMAP4 si finisce col consumare la propria quota nella
mailbox. Inoltre poiché raramente i provider forniscono supporto per
connessioni sicure al server IMAP, utilizzare questo può in qualche modo
creare un problema di privacy.

Per risolvere questo, ho ideato una soluzione elaborata nell’insieme, ma
semplice da creare, che permette di avere sempre a disposizione la
propria posta senza problemi di reti, privacy o complicazioni varie.
Tale soluzione richiede veramente poche risorse, e le normali capacità
tecniche necessarie per la configurazione di un server di rete, ed
utilizza solamente software libero.

Richieste hardware e software

Per poter applicare con successo questa soluzione c’è sicuramente una
necessità hardware da soddisfare: una macchina collegata ad Internet 24
ore su 24 (o quasi, in ogni caso collegata nel momento in cui si vuole
controllare la propria posta dall’esterno).

Le richieste software sono abbastanza vaghe in realtà: il computer che
farà da server deve disporre di un sistema operativo Unix-like, come può
essere Linux, ma anche OpenBSD, FreeBSD, Solaris e chi più ne ha più ne
metta possono funzionare (io ho avuto esperienza diretta usando Gentoo
Linux e OpenBSD).

È poi necessario avere i seguenti pacchetti: fetchmail, maildrop e
dovecot. Opzionalmente è possibile utilizzare bogofilter o spamassassin.

Inoltre per poter accedere alla vostra macchina dall’esterno senza
conoscerne di volta in volta l’IP, vi consiglio di dotarvi di un
hostname dinamico come quelli forniti da dyndns.org, e di un client che
si occupi dall’aggiornamento automatico dello stesso.

Scelte pratiche

Entriamo dunque nel pieno dell’implementazione e partiamo con
un’illustrazione delle scelte che ho fatto quando sono partito con
questa mia idea.

Software Server

La scelta del software a lato server è stata sicuramente quella che mi
ha fatto impiegare più tempo. A parte la scelta del sistema operativo,
che va oltre ai fini di questo articolo, ho impiegato un po’ di tempo
nella scelta del server IMAP da utilizzare.

Scartai inizialmente QMail non tanto per la sua licenza particolare, ma
per il fatto che la sua configurazione è abbastanza complessa,
probabilmente ciò è dovuto al fatto che è stato pensato per server su
vasta scala che non sono quelli di cui abbiamo bisogno noi in questo
momento.

Scartai anche cyrus-imapd semplicemente perché non riuscivo a capire
come funzionasse. Courier, la mia prima scelta effettiva, è stata poi
scartata per problemi di compatibilità con Apple’s Mail e altre cose.
Dovecot, invece, è più compatibile, semplice e funzionale.

Come MDA ho appunto scelto lo stesso fornito con la suite Courier,
Maildrop, che è abbastanza potente e non ha grossi problemi conosciuti.
Inoltre è disponibile come pacchetto separato, per l’uso con server
diversi da Courier.

Ora, come ho già detto, non necessitiamo di un MTA, perché per
recuperare la posta (come vedremo) utilizzeremo fetchmail, che parlerà
direttamente con l’MDA senza altri intermediari.

Filtri per la posta

Visto l’aumentare vertiginoso dello spam che riceviamo sulle nostre
caselle email, è molto probabile che vogliate impostare un filtro sulla
posta in ingresso. Per farlo, possiamo dire all’MDA di passare le email
attraverso un programma esterno che ne determini o meno la natura di
spam.

Inizialmente avevo configurato sul mio server un filtro usando
bogofilter, mentre sull’attuale server ho configurato (anche se non
funziona per qualche problema con perl) spamassassin. La configurazione
è molto semplice.

Volendo è anche possibile effettuare una scansione antivirus usando
clamav, nel caso in cui si disponga di macchine che possono essere
attaccate da virus (Windows).

Connessione sicura

Sarò paranoico, ma da un po’ di tempo sono abbastanza preoccupato che il
traffico tra le mie macchine ed Internet sia sniffato da intermediari.
Forse è dovuto al fatto che perlopiù mi collego tramite la rete della
mia (ex-)scuola.

Per questo, una delle cose a cui ho fatto attenzione quando ho
configurato il mio sistema di posta è stata quella di utilizzare la
connessione cifrata usando SSL. Per fortuna Dovecot supporta
tranquillamente IMAP-over-SSL.

Webmail

Poiché non sempre ho a disposizione una connessione di rete per il
portatile quando posso navigare ad Internet (molto spesso navigo per
esempio dall’università), ho voluto installarmi anche una webmail che si
colleghi al nuovo server per poter leggere le mie email con comodità.

Anche per questa funzione ho scelto un software libero, SquirrelMail,
un’applicazione PHP, avviata su un Apache configurato come server HTTP
sicuro (SSL).

Ho deciso di non utilizzare sqwebmail perché non mi piacciono i CGI e
preferisco fidarmi di PHP quando è possibile. Inoltre, SquirrelMail
supporta qualsiasi server IMAP, anziché solo courier.

Raccogliere la posta

Il primo problema da affrontare per risolvere la questione è come
raccogliere la posta. Abbiamo visto che gli MTA si occupano di far
arrivare la posta fino al server del nostro provider, e poi tocca a noi
riuscire a prenderla. Solitamente se ne occuperebbe un client (grafico o
testuale), ma a noi occorre che ad occuparsene sia un qualcosa che ci
permetta di farla avere poi al nostro server IMAP.

Per risolvere questo esiste un bellissimo programma chiamato fetchmail,
che si occupa di scaricare la posta da server POP3 o IMAP4 e di
reimmetterla poi localmente tramite un MTA oppure un MDA. Procediamo
quindi a creare un file /etc/fetchmailrc (con permesso di lettura solo
a root), la cui sintassi generale è questa:

poll pop3.provider.tld with proto POP3
    user 'utente@provider.tld' there with password 'segreto'
    is 'localuser' here options fetchall

In questo caso si dice a fetchmail che deve raccogliere la posta dal
server usando il protocollo POP3, e che la posta dell’utente sul server,
accessibile con la password segreto, deve essere inviata all’utente
locale localuser, scaricando anche la posta segnata come già letta.

Questo però farebbe sì che fetchmail prenda e tenti di collegarsi ad un
MTA locale, usando SMTP, e tenti di inviare i messaggi per l’utente
localuser. Non è quello che vogliamo però, perché in tal caso dovremmo
impostare anche un MTA ed è un problema da non sottovalutare.

In soccorso ci arriva l’opzione mda di fetchmail, cambiamo quindi la
riga di configurazione generale con questa:

poll pop3.provider.tld with proto POP3
    user 'utente@provider.tld' there with password 'segreto'
    is 'localuser' here options fetchall
    mda "/usr/bin/maildrop -d localuser"

in questo caso fetchmail invece di contattare l’MTA locale, manderà la
mail direttamente all’MDA (maildrop) dicendogli di consegnarla
all’utente .

Nel caso si abbia più di un utente locale a cui si vuole inoltrare la
posta basta semplicemente cambiare localuser nella configurazione di
fetchmail con i nomi dei vari utenti.

A questo punto è necessario impostare un cronjob che avvii ripetutamente
fetchmail, ma prima di fare ciò è bene completare l’impostazione del
server IMAP e dell’MDA.

Prepariamo il server IMAP

Il motivo per cui prima di configurare l’MDA consiglio di preparare il
server IMAP è semplicemente una comodità per creare le sottocartelle in
cui incasellare la posta.

Iniziamo dunque i preparativi. Per prima cosa bisogna installare il
server, per farlo io ho utilizzato portage in Gentoo e pkg_add in
OpenBSD, quindi spero che la distribuzione Linux (o il sistema operativo
che state utilizzando) abbia i precompilati da installare, altrimenti vi
lascio a seguire la documentazione per la compilazione di Dovecot, che
esce dal seminato di questo articolo.

Una volta installato Dovecot, dovrete configurarlo, attraverso
/etc/dovecot.conf.

La documentazione del file di configurazione è abbastanza esplicita, e
quindi c’è poco da dire perlopiù. Le cose importanti saranno qui
elencate per sicurezza.

protocols = imaps: questa impostazione è necessaria per utilizzare
IMAP-over-SSL.

default_mail_env = maildir:%h/Maildir: questa impostazione dice a
Dovecot di cercare una directory di nome Maildir all’interno della home
dell’utente ($%$h indica proprio la home directory).

Se volete diminuire il carico di processore della macchina localmente,
potete configurare anche il server IMAP senza SSl, in modo da
utilizzarlo quando ci si connette dalla rete interna. Per fare ciò basta
aggiungere imap alla lista di protocolli alla voce protocols.

A questo punto bisogna creare la maildir in cui salvare i dati. Per
farlo, maildrop mette a disposizione il comando maildirmake che crea
la struttura completa della Maildir:

$ cd ~
$ maildirmake Maildir

Ricordatevi solo di lanciarlo dall’utente da cui volete leggere la
posta, e di sostituire Maildir con la directory che avete impostato sul
file di configurazione di imap.

Per finire avviate il server IMAP (come fare questo dipende dal sistema
operativo che state usando) e configurate un client per connettervisi,
mi raccomando perché è necessario per il prossimo passo.

Incasellare la posta

Arriviamo dunque alla parte più intricata della configurazione: lo
script per maildrop.

Dico questo perché maildrop è sicuramente un software molto avanzato, ma
allo stesso tempo ha un file di configurazione molto intricato, e mal
documentato. Per poterne trarre qualcosa di semplice, ho dovuto fare
abbastanza ricerche su Google, e quindi in questa sezione tento di
semplificare le cose al massimo.

Creiamo dunque un file  /.mailfilter . Iniziamo in realtà dal fondo, o
meglio dal minimo necessario, e poi costruiamo uno script più
complesso.

Come ultima riga del file, poniamo questo:

to "Maildir"

che indica a maildrop di mettere dentro la directory Maildir tutti i
messaggi che non sono stati trattati da regole precedenti.

Nota: Maildrop può utilizzare sia Maildir che mbox, quindi fate
attenzione perché se gli passate un nome che non specifica una directory
come argomento al to, lui salverà i messaggi in formato mbox, non
utilizzabile da Dovecot.

Per facilitare la comprensione delle regole indicate più avanti, quando
si inserisce una stringa tra due slash (/) nello script di maildrop,
ci si riferisce ad una regular expression, quindi per indicare
“qualsiasi cosa” basta inserire .*.

Inserire le mailing list in diverse cartelle

Sicuramente però ci sarà della posta che volete inserire automaticamente
dentro una cartella diversa dalla inbox, per esempio se siete iscritti a
qualche mailing list. Creiamo dunque una cartella chiamata ’wine-devel’
col client di posta, e vediamo che viene creata una nuova maildir sul
server, dentro la directory definita sul file di configurazione,
chiamata .wine-devel, che è quella in cui vogliamo inserire i
messaggi.

Per conoscere il nome che viene assegnato alla directory che contiene i
messaggi della cartella IMAP basta ricordarsi che tutte sono
sotto-directory della maildir impostata nel file di configurazione, e il
loro nome inizia sempre per ., e che il separatore per le sotto
directory è sempre . (per esempio se la cartella IMAP fosse stata
chiamata devel e fosse stata a sua volta dentro ad una cartella
chiamata Wine il nome della maildir sarebbe stato .Wine.devel).

Fate attenzione perché nello script di maildrop dovremo utilizzare il
nome delle maildir, non delle cartelle IMAP.

A questo punto, resta solo da inserire prima dell’ultima riga una regola
che dica a maildrop di spostare le email che arrivano dalla lista
wine-devel dentro quella cartella. Per farlo però bisogna avere
un’identificazione unica di tali messaggi. Per fortuna le mailing list
gestite con MailMan inseriscono un header X-BeenThere che indica
l’indirizzo della mailing list. Il nostro script per maildrop si
presenta ora così:

if ( /X-BeenThere: wine-devel@winehq.org/ )
    to "Maildir/.wine-devel"

to "Maildir"

Se non trovate un identificatore che sia sempre unico per la mailing
list, ma dovete per esempio verificare l’header To:, cercandoci una
sottostringa, potete usare una regular expression del tipo
/To:.*mailinglist@dominio.tld.*/.

Cancellare lo spam conosciuto

Spesso riceviamo spam “conosciuto” ovvero spam di cui
riusciamo ad identificare in un modo o nell’altro la provenienza, perché
per esempio proviene dal nostro stesso provider. Per liberarsi da tale
spam, basta aggiungere qualche semplice regola al nostro script di
maildrop.

Prendiamo per esempio caso di voler cancellare tutto il traffico in
arrivo da un determinato indirizzo email, poniamo caso sia .
Trasformiamo quindi il nostro script in questo:

if ( /From:.*newsletter@provider.tld.*/ )
    exit

if ( /X-BeenThere: wine-devel@winehq.org/ )
    to "Maildir/.wine-devel"

to "Maildir"

A questo punto tutti i messaggi ricevuti da quell’indirizzo saranno
semplicemente ignorati (exit dice a maildrop di non processare il
messaggio, e non arrivando a nessun to, semplicemente viene
cancellato).

Come vedete le regole le sto inserendo una sopra l’altra, facendo in
modo che quelle che “sfoltiscono” i messaggi siamo poste
prima delle altre, accorciando la quantità di test che devono essere
effettuati prima di consegnare il messaggio.

Tenete bene in testa questa cosa, perché il prossimo passo è l’inserire
un filtro che permette di classificare i messaggi come “probabile
spam”.

Filtrare il possibile spam

Visto che non è mai possibile azzerare lo spam con delle semplici regole
statiche, molti client di posta hanno aggiunto recentente delle
funzionalità per la classificazione della posta come spam. Ne è un
esempio Mozilla Thunderbird.

Possiamo a questo punto inserire anche noi un controllo per il possibile
spam sul nostro server locale. Per farlo io ho provato almeno due
filtri, bogofilter (un filtro bayesiano simile a quello usato da
Thunderbird) e spamassassin (più complesso e più conosciuto), anche se
in realtà ho avuto solo la prova del primo, a causa del fatto che perl
non funziona sulla macchina su cui ho il server ora.

Qualsiasi sia il filtro che vogliate utilizzare, è sconsigliabile
cancellare le email completamente, perché non sono rari i falsi positivi
(ovvero mail che vengono catalogate come spam anche se non lo sono), ed
è quindi meglio spostare le email in una cartella diversa, da
controllare e svuotare di tanto in tanto.

Creiamo quindi una cartella Junk Mail dentro la Inbox, la sua maildir
sarà Maildir/.Junk Mail

Bogofilter

Bogofilter è un filtro mail che
classifica i messaggi come spam o non-spam attraverso un’analisi
statistica dell’intestazione e del contenuto dei messaggi. Il
programma è capace di imparare dalle classificazioni e correzioni
dell’utente.

Si tratta in pratica di un programma che “impara” a capire
quando un messaggio è spazzatura o meno tramite l’intervento
dell’utente. Ho trovato comodo utilizzarlo quando il server di posta era
il mio desktop, e usavo KMail per far sapere a bogofilter la
classificazione. Non è molto comodo quando il server e il client di
posta primario non sono la stessa macchina perché richiede l’esecuzione
di un comando sul server.

Per effettuare il filtering tramite bogofilter, subito dopo le regole
per le mailing list, e prima dell’ultima to, inseriamo questa regola:

xfilter "bogofilter -u -e -p"
if ( /^X-Bogosity: Yes, tests=bogofilter/)
       to "Maildir/.Junk Mail"

La prima riga si occupa di lanciare il comando bogofilter, chiedendogli
di leggere dallo standard input il messaggio e riscriverlo sullo
standard input inserendo degli header che ritornino una probabilità
(compresa tra 0 e 1) del fatto che la mail sia spam. Inoltre se tale
percentuale supera 0.5, valuta la mail come sicuro spam, inserendo
X-Bogosity: Yes tra gli header.

La seconda riga invece cerca tale header, e se lo trova, inserisce la
mail dentro la cartella che abbiamo predisposto per il probabile spam.

SpamAssassin

SpamAssassin è uno dei più conosciuti antispam. Non sono molto pratico
della sua configurazione, che ho lasciato ai default, e non saprei
proprio come spiegarlo, quindi mi limito ad inserire la regola da
utilizzare, similmente a Bogofilter in precedenza, prima del to
finale:

xfilter "/usr/local/bin/spamassassin"
if (/^X-Spam-Flag: *YES/)
{
        to "Maildir/.Junk Mail"
}

Reinoltrare una copia delle email

In certi casi si vuole fa avere una copia di tutte o di una parte delle
email ad un altro indirizzo. Per esempio c’è chi utilizza servizi di
webmail con una alta quota per effettuare una copia di backup della
propria posta.

In altri si vuole semplicemente che delle email con determinati soggetti
o comunque che sottostanno a determinati check (come quelli visti prima
per le mailing list o lo spam), siano inviati ad un altro indirizzo
email (per esempio quello di un collega, oppure di una mailing list di
sviluppo nel caso riguardino dei programmi di cui si è sviluppatori).

Per inviare una copia di tutte le email (che hanno passato le regole
fino a quel punto), basta inserire una riga di questo tipo:

cc "| /usr/sbin/sendmail -- altroindirizzo@altrodominio.tld"

Questa riga non fa altro che inviare tutte le email che riescono a
giungere ad essa nello script all’indirizzo indicato utilizzando
sendmail (si suppone che il comando sendmail nel sistema funzioni
correttamente).

Solitamente inserisco questa regola dopo le regole statiche per lo spam
conosciuto e prima di quelle delle mailing list, per poter ricevere
tutta la posta che potrebbe non essere spam.

Completare l’installazione di fetchmail

Ora che abbiamo configurato il server IMAP e l’MDA, possiamo finalmente
inserire un cron job che si occupi di lanciare ripetutamente fetchmail
per andare a scaricare la posta e immetterla localmente. Per fare
questo, inseriamo questa linea in /etc/crontab :

*/5  *  * * *  root  /usr/local/bin/fetchmail -d0 -f /etc/fetchmailrc 
    -s --syslog > /dev/null 2>&1

e il gioco è fatto.

Ringraziamenti

Il più grande ringraziamento per questo documento va sicuramente a Fabio
“FVZ” per i chiarimenti riguardo al funzionamento e alla
terminologia dei servizi di posta elettronica.

Desidero inoltre ringraziare Bernardo “inquis” per avermi
spinto a scrivere questo articolo.

Per la nuova edizione, voglio inoltre ringraziare Roy
“Uberlord” per il suo consiglio riguardo Dovecot.


  1. Un hostname che viene risolto di volta in volta con un diverso IP
    scelto tra una lista di server paralleli che svolgono la stessa
    funzione.

    [return]