0x80074E24 FWX_E_CONNECTION_KILLED in ISA Server 2004 / 2006

We have recently installed a Samsung SHR-5040 digital video recorder (DVR) in our remote workshop in order to monitor activities and the evolution of customers' orders. We have our branch (workshop) and central offices connected by using a VPN, maintained by two ISA Servers. We thought that making the live output of the DVR available in our central office would be straight away but it was not so simple as it seemed at first.

After having the device set up and running, we checked that the web administration page was available from the sub-net at the branch site. It was Ok. Then, we tried to access the device web page from a computer located at central offices sub-net and, even though the http content (page) was shown, the device was not available using RTSP (Real Time Streaming Protocol) and no live image was shown, only a big black screen and a message saying 'no response'.

Since we had no idea where the problem was (which one of the ISAs, at the branch or at the central site) we started to do the checks step by step, and the first step was the ISA at the branch site (workshop). We started a live log filtering by destination IP being the IP assigned to the DVR and we saw a Connection denied error for RTSP on port 554 (destination) and TCP port 58907 (at the source): 0xc0040017 FWX_E_TCP_NOT_SYN_PACKET_DROPPED.

After a little research I found that error is raised whenever a new TCP session is attempted without a SYN flag that must be set on the first packet coming from the host trying to start the connection. If ISA receives a packet that does not belong to an existing connection, it must have SYN flag enabled or it will be rejected.

Right before that Connection denied error (and with the same timestamp) we had another log entry, thought not an error, it was quite interesting: Connection close for RTSP on port 554 (destination) and also TCP port 58907 (at the source) 0x80074e24 FWX_E_CONNECTION_KILLED. It was not marked with red color as being an error but it was really weird that the source port for both log entries be the same, at the same time, and first, ISA kills the connection and next ISA says that a new connection (coming from the same host and the same port) without SYN is received and, hence, rejected.

The real problem was not the 0xc0040017 error due to un-synched packed, but the previous entry, the 0x80074e24 that kills the connection unilaterally.

We even added a new access rule to ISA server allowing all protocols from internal network to local host without any luck; still the same behavior was shown: first the connection is killed and then the rest of packets are rejected since they do not belong to an existing connection and do not have sync flag set.

After several days of research and trial and errors we found a post asking for help with the same symptoms that we had: SMTP server rule breaks incoming connections then rejects packets. The only difference was that we were trying to set a DVR instead of a mail server. At the end of the entry we could read:

Unbinding the SMTP application filter from the SMTP server firewall rule changes the "connection killed" message to "abortive shutdown", but the behavior seems otherwise the same.

We also found another old post RE: RTSP - ISA 2006 were we could read:

After upgrading from ISA 2004 to ISA 2006, my streaming media server
stopped working through the firewall. I then noticed in the logs that my RTSP
Server Publishing rule was showing the code for connection killed. I then went
and removed the RTSP filter from the RTSP Server protocol definition, and my 
streaming media started working again. Since I don't see any way of configuring the RTSP filter, and since it worked fine in ISA 2004, I'm guessing that this is a bug in ISA 2006?

This last post was dated on Oct 2006 and we thought that, in case of a bug, it might have been solved already, but we wanted to give it a try.

We went to ISA Server 2006 admin console, browsed though the protocols until we found RTSP, right click, properties, parameters and simply removed RTSP filter from application filters. We then accepted and applied the changes and everything worked like a charm.

Keywords:
ISA Server 2004, ISA Server 2006, streaming, RTSP, RTSP filter, 554, 0x80074e24, FWX_E_CONNECTION_KILLED, 0xc0040017, FWX_E_TCP_NOT_SYN_PACKET_DROPPED, digital video recorder, DVR, Samsung SHR-5040.