This entry is not about anything related to Microsoft nor any of its products. I work for a medium size company as dba, but there are also other things that I have to handle in a daily basis. One of these things is the e-mail server. It does not matter which product/vendor we use, the problem more general than that. We all know what spam is, there are lots of protecting schemes out there: SPF, DKIM, SenderID, and many more, but... what happens when the email that is consuming your bandwidth is generated from the inside, from your own users?
Of course it is a good idea to have and let your employees know that your company has a Employee Internet Usage Guidelines, or Acceptable Usage Policy. That policy document might start with This policy explains what is classified as the acceptable and unacceptable use of the Internet and e-mail systems..., but how do you enforce that? That is the main question: How do you enforce your e-mail policy?
If your company host email servers on its own servers, your bandwidth will be an issue. Download bandwidth is relatively cheap, but not so the upload. If an employee receives a chain email, with a video attachment (those are becoming more and more popular) of 2.5Mb average, that will have some associated costs, but the big cost (from the point of view of IT staff) will raise afterwards, then that employee decides to forward that amusing Videos / Jokes / PowerPoint presentations / Documents / Images to his/her 15 friends and 5 relatives and the dog. When your email server tries to upload the forwarded email to 20 recipients (fortunately his/her dog does not have an email account yet), your SMTP server queue will grow in 50Mb, and your upload bandwidth will collapse for some/several minutes.
If the Internet connection in your company is shared by other users/services, i.e. web servers, dns, VPNs, VoIP, file replication between branch/main offices, etc. all your users/services will suffer some kind of delay during those peak minutes due to a single user violating (at least) the netiquette.
We have been suffering for a while because of this problem and some problematic employees. No matter if you warn them, after some months, they (or others) will collapse again the server up to the point that any other user comes/calls to the IT department and asks if there is any problem with the Internet connection, because it sucks, or that a annoyed customer has called because he has not received the email that someone promised that it was delivered 2 hours ago. Then we open the monitoring graphs, see the upload peak in the email server, go and see the queue and voila!, there are tons of pending legitimate emails waiting to be delivered because, at some point, the outbound queue has 30 email (of non work related) summing 80Mb to be delivered.
There is an article (dated 2003) that states that 35% of corporate e-mail is non-work related. And I wonder... 35% of emails means 35% of the bandwidth used or 35% of the number of emails? If 35% of the number of emails, that would easily make 40-50% of the bandwidth due to the sizes of the attachments. And the figures are growing day by day... as you can see in another article dated March 2007 which states that Half of corporate web traffic not work related. Companies are failing to enforce their Acceptable Usage Policies, basically because (in case of email, at least), there are no embedded tools to help on this subject.
ISV and MTA are fighting against spam thinking that the enemy is out there, that the bad guys are at the other side of our firewall, but that is only partly true. Legit users inside your companies are are the other part of the problem.
I've been thinking a while on this subject and found a common pattern with those chain e-mails with huge attachments:
- A file with relatively big size attachment(s) comes into our email server.
- It comes from a legit user outside our company and the To: list has several recipients.
- After a short amount of time (from minutes to 72 hours), the same attachment(s) are send out of the company
- In this case the receipts are different, but there are also more than 3 in most cases.
Having this pattern in mind, we have created a filter that does the following:
For incoming emails
- If the message has attachments and the size of the message exceeds certain threshold, an MD5 hash is computed for every attachment.
- A daily log file is created to store MD5 hash, file size, the file name of the attachment, To and From fields. This file stores ONLY incoming content.
For outgoing emails
- If the message has attachments, the number of recipients is above certain number and the size of the message multiplied by the number of recipients is above a given threshold, MD5 hashes are calculated for every attachment in the email.
- Every MD5 hash is checked against the known MD5 hashes in the last n days log files.
- If a match is found, the filter notifies the MTA (.exe program exits with ErrorCode 1) and the outgoing email can be quarantined/deleted/moved to a lower priority queue (the action depends of the MTA filter options).
We found that this approach is smarter than just forbidding certain extensions to be delivered through your email server. Many email servers have options to block certain types of attachments, i.e. block .pps files, or .mp3 files, but sooner or later you will face a case in which a legit user needs to send a legit content of these blocked file types. This is a different approach and relies on the idea of forbidding to send out any content that is not original/originated/created at your site (self-made content), fine-tuned with certain parameters such as n retention days, number of recipients, attachment sizes, out queue threshold, etc. This filter is content-agnostic, it does not matter which file name or content type the attachment is, and, as long as it has the same MD5 hash and had been previously received within the defined time-span, the email will be pinpointed.
Keywords:
bandwidth waste, chain emails, SMTP, queue, upload, productivity losses, filter, quarantine, enforce internet usage policy, free, algorithm, procedure, method
