0x80074E24 FWX_E_CONNECTION_KILLED in ISA Server 2004 / 2006

We have recently installed a Samsung SHR-5040 digital video recorder (DVR) in our remote workshop in order to monitor activities and the evolution of customers' orders. We have our branch (workshop) and central offices connected by using a VPN, maintained by two ISA Servers. We thought that making the live output of the DVR available in our central office would be straight away but it was not so simple as it seemed at first.

After having the device set up and running, we checked that the web administration page was available from the sub-net at the branch site. It was Ok. Then, we tried to access the device web page from a computer located at central offices sub-net and, even though the http content (page) was shown, the device was not available using RTSP (Real Time Streaming Protocol) and no live image was shown, only a big black screen and a message saying 'no response'.

Since we had no idea where the problem was (which one of the ISAs, at the branch or at the central site) we started to do the checks step by step, and the first step was the ISA at the branch site (workshop). We started a live log filtering by destination IP being the IP assigned to the DVR and we saw a Connection denied error for RTSP on port 554 (destination) and TCP port 58907 (at the source): 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED.

After a little research I found that error is raised whenever a new TCP session is attempted without a SYN flag that must be set on the first packet coming from the host trying to start the connection. If ISA receives a packet that does not belong to an existing connection, it must have SYN flag enabled or it will be rejected.

Right before that Connection denied error (and with the same timestamp) we had another log entry, thought not an error, it was quite interesting: Connection close for RTSP on port 554 (destination) and also TCP port 58907 (at the source) 0x80074e24 FWX_E_CONNECTION_KILLED. It was not marked with red color as being an error but it was really weird that the source port for both log entries be the same, at the same time, and first, ISA kills the connection and next ISA says that a new connection (coming from the same host and the same port) without SYN is received and, hence, rejected.

The real problem was not the 0xc0040017 error due to un-synched packed, but the previous entry, the 0x80074e24 that kills the connection unilaterally.

We even added a new access rule to ISA server allowing all protocols from internal network to local host without any luck; still the same behavior was shown: first the connection is killed and then the rest of packets are rejected since they do not belong to an existing connection and do not have sync flag set.

After several days of research and trial and errors we found a post asking for help with the same symptoms that we had: SMTP server rule breaks incoming connections then rejects packets. The only difference was that we were trying to set a DVR instead of a mail server. At the end of the entry we could read:

Unbinding the SMTP application filter from the SMTP server firewall rule changes the "connection killed" message to "abortive shutdown", but the behavior seems otherwise the same.

We also found another old post RE: RTSP - ISA 2006 were we could read:

After upgrading from ISA 2004 to ISA 2006, my streaming media server
stopped working through the firewall. I then noticed in the logs that my RTSP
Server Publishing rule was showing the code for connection killed. I then went
and removed the RTSP filter from the RTSP Server protocol definition, and my 
streaming media started working again. Since I don't see any way of configuring the RTSP filter, and since it worked fine in ISA 2004, I'm guessing that this is a bug in ISA 2006?

This last post was dated on Oct 2006 and we thought that, in case of a bug, it might have been solved already, but we wanted to give it a try.

We went to ISA Server 2006 admin console, browsed though the protocols until we found RTSP, right click, properties, parameters and simply removed RTSP filter from application filters. We then accepted and applied the changes and everything worked like a charm.

Keywords:
ISA Server 2004, ISA Server 2006, streaming, RTSP, RTSP filter, 554, 0x80074e24, FWX_E_CONNECTION_KILLED, 0xc0040017, FWX_E_TCP_NOT_SYN_PACKET_DROPPED, digital video recorder, DVR, Samsung SHR-5040.

Illegal characters in path deploying to SharePoint

I recently started doing tests with SharePoint and I reached the point where I needed doing my own WebPart and debug it. I read about setting a development environment for WSS 3.0 in The .NET Addict's Blog: Setting up your SharePoint 2007 Development Environment. Summing up (changing slightly his suggestions), I needed a spare computer or virtual machine to:

• Install Server 2003 R2, with IIS, SP2 and updates
• Install an instance of SQL Server 2005 Standard Edition
• Install Microsoft .NET Framework 3.0 Redistributable Package
• Install Microsoft Windows SharePoint Services 3.0 with Service Pack 1 (WSS3.0SP1) not in standalone mode.
• Install Visual Studio 2005 and Service Pack 1
• Install Microsoft Windows SharePoint Services 3.0: Software Development Kit (SDK)
• Install Microsoft Windows SharePoint Services 3.0 Tools Visual Studio 2005 Extensions
• Install Microsoft Visual Studio 2005 extensions for .NET Framework 3.0 (Windows Workflow Foundation)

After a couple of days I had the virtual environment ready and tried to create my very first Web Part using the Web Part template installed in the previous steps: I just removed a comment so that the Web Part returned "Output HTML":

public class Web_Part1 : System.Web.UI.WebControls.WebParts.WebPart
{
    public Web_Part1()
    {
        this.ExportMode = WebPartExportMode.All;
    }

    protected override void Render(HtmlTextWriter writer)
    {
        // TODO: add custom rendering code here.
        writer.Write("Output HTML");
    }
}

Compiled, tried to deploy... and the following error appeared:

Creating solution ... 
System.ArgumentException 
Illegal characters in path.

No more information, no trace in the EventLog, nor any other log. What path? What character(s)?

I googled for this problem and found a a page that finally drove me in the right direction:

In System.ArgumentException: Illegal characters in path, they talk about Latin characters (my native Operating System, VS2005 and WSS3.0SP1 are Spanish too). They suggest using encoding iso-8859-1 instead of UTF-8 in XML files... but since those files are created by the tools extension add-in (not mine) I could not manage to do any changes there. But they were right in pinpointing that character outside the standard ASCII character set range of 0x00 - 0x7F might be causing the problem.

Even thought his problems were related to a WebService, the idea of the extended characters was interesting. However, none of my files had any of those. I even re-created the simple project and solution right beneath the root C:\ and the problem still appeared: Illegal characters in path.

After some hours I changed the perspective. If VS2005 cannot tell me which file/path has illegal characters, let's do other tool tell me: Process Monitor. For those of you that do not know this tool, Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity.

I started Process Monitor, filtered by process name devenv.exe (Visual Studio) and tried to do a deploy. Of course it failed again, but then I changed to Process Monitor window and started to scroll and read.

After some minutes I found it! VS2005 was trying to create some files at a temporary directory, pointed by both TMP and TEMP environment variables. They were set to: C:\Documents and settings\myusername\Configuración local\Temp

There was the extended character ó in Configuración. This directory in English systems is: C:\Documents and settings\myusername\Local settings\Temp

After knowing that, all I had to do is changing the user environment variables TMP (and TEMP, just in case) to any other directory not containing any special characters, C:\TEMP for instance. You can do it right clicking on My PC, Properties, Advanced options, Environment variables button, and modify...

I then restarted VS2005 and everything was compiled and deployed without any problem.

Keywords: Visual Studio 2005, VS2005, deploy, WSS3.0, Sharepoint Services, Web Part, development, error, solution, illegal characters in path, caracteres no válidos en la ruta de acceso, implementar

Chain e-mail traffic forwarded by your own internal users... is this spam?

This entry is not about anything related to Microsoft nor any of its products. I work for a medium size company as dba, but there are also other things that I have to handle in a daily basis. One of these things is the e-mail server. It does not matter which product/vendor we use, the problem more general than that. We all know what spam is, there are lots of protecting schemes out there: SPF, DKIM, SenderID, and many more, but... what happens when the email that is consuming your bandwidth is generated from the inside, from your own users?

Of course it is a good idea to have and let your employees know that your company has a Employee Internet Usage Guidelines, or Acceptable Usage Policy. That policy document might start with This policy explains what is classified as the acceptable and unacceptable use of the Internet and e-mail systems..., but how do you enforce that? That is the main question: How do you enforce your e-mail policy?

If your company host email servers on its own servers, your bandwidth will be an issue. Download bandwidth is relatively cheap, but not so the upload. If an employee receives a chain email, with a video attachment (those are becoming more and more popular) of 2.5Mb average, that will have some associated costs, but the big cost (from the point of view of IT staff) will raise afterwards, then that employee decides to forward that amusing Videos / Jokes / PowerPoint presentations / Documents / Images to his/her 15 friends and 5 relatives and the dog. When your email server tries to upload the forwarded email to 20 recipients (fortunately his/her dog does not have an email account yet), your SMTP server queue will grow in 50Mb, and your upload bandwidth will collapse for some/several minutes.

If the Internet connection in your company is shared by other users/services, i.e. web servers, dns, VPNs, VoIP, file replication between branch/main offices, etc. all your users/services will suffer some kind of delay during those peak minutes due to a single user violating (at least) the netiquette.

We have been suffering for a while because of this problem and some problematic employees. No matter if you warn them, after some months, they (or others) will collapse again the server up to the point that any other user comes/calls to the IT department and asks if there is any problem with the Internet connection, because it sucks, or that a annoyed customer has called because he has not received the email that someone promised that it was delivered 2 hours ago. Then we open the monitoring graphs, see the upload peak in the email server, go and see the queue and voila!, there are tons of pending legitimate emails waiting to be delivered because, at some point, the outbound queue has 30 email (of non work related) summing 80Mb to be delivered.

There is an article (dated 2003) that states that 35% of corporate e-mail is non-work related. And I wonder... 35% of emails means 35% of the bandwidth used or 35% of the number of emails? If 35% of the number of emails, that would easily make 40-50% of the bandwidth due to the sizes of the attachments. And the figures are growing day by day... as you can see in another article dated March 2007 which states that Half of corporate web traffic not work related. Companies are failing to enforce their Acceptable Usage Policies, basically because (in case of email, at least), there are no embedded tools to help on this subject.

ISV and MTA are fighting against spam thinking that the enemy is out there, that the bad guys are at the other side of our firewall, but that is only partly true. Legit users inside your companies are are the other part of the problem.

I've been thinking a while on this subject and found a common pattern with those chain e-mails with huge attachments:

1. A file with relatively big size attachment(s) comes into our email server.
2. It comes from a legit user outside our company and the To: list has several recipients.
3. After a short amount of time (from minutes to 72 hours), the same attachment(s) are send out of the company
4. In this case the receipts are different, but there are also more than 3 in most cases.

Having this pattern in mind, we have created a filter that does the following:

For incoming emails

1. If the message has attachments and the size of the message exceeds certain threshold, an MD5 hash is computed for every attachment.
2. A daily log file is created to store MD5 hash, file size, the file name of the attachment, To and From fields. This file stores ONLY incoming content.

For outgoing emails

1. If the message has attachments, the number of recipients is above certain number and the size of the message multiplied by the number of recipients is above a given threshold, MD5 hashes are calculated for every attachment in the email.
2. Every MD5 hash is checked against the known MD5 hashes in the last n days log files.
3. If a match is found, the filter notifies the MTA (.exe program exits with ErrorCode 1) and the outgoing email can be quarantined/deleted/moved to a lower priority queue (the action depends of the MTA filter options).

We found that this approach is smarter than just forbidding certain extensions to be delivered through your email server. Many email servers have options to block certain types of attachments, i.e. block .pps files, or .mp3 files, but sooner or later you will face a case in which a legit user needs to send a legit content of these blocked file types. This is a different approach and relies on the idea of forbidding to send out any content that is not original/originated/created at your site (self-made content), fine-tuned with certain parameters such as n retention days, number of recipients, attachment sizes, out queue threshold, etc. This filter is content-agnostic, it does not matter which file name or content type the attachment is, and, as long as it has the same MD5 hash and had been previously received within the defined time-span, the email will be pinpointed.

Keywords:
bandwidth waste, chain emails, SMTP, queue, upload, productivity losses, filter, quarantine, enforce internet usage policy, free, algorithm, procedure, method