Footnote: This is fast becoming a series of posts on the woes of using the .NET Compact Framework. In fact, I've added a subcategory for this exact purpose; you can access the category and its RSS feed from the Categories list on the right-hand side of the screen.

Here's another one of those "bang your head repeatedly against the wall and look for a sharp object" issues: closing a stream derived from HttpWebResponse.GetResponseStream takes an abnormally long time. In fact, the (blocking) call to Stream.Close takes approximately the same amount of time as it would to read to the end of the stream. I've only encountered one other reference to this in a newsgroup post from 2004, which (according to Todd, the original author of the post) was never actually answered.

Seeing as I was trying to use HttpWebRequest to create a minimalistic download manager for one of our applications, I had to come up with a solution: one option was to close the stream asynchronously and just let the download run its course, wasting valuable memory, CPU time and bandwidth; another was to never close the stream and make do with a resource leak. Not content with these solutions I decided to dig into the framework code with the ever-useful Reflector. The first hurdle was locating the assemblies; they're not trivial to find, but if you look hard enough you can find actually useful versions in your Visual Studio 2005 installation folder, under SmartDevices\SDK\CompactFramework\2.0\v2.0\Debugger\BCL. These are regular .NET Assembly PEs so it's trivial to load them up in Reflector.

Some digging into the BCL sources proved that HttpWebRequest does, in fact, read until the end of the stream when closing; this is the relevant code excerpt, from HttpWebRequest.doClose:

if (!this.m_doneReading)
{
      byte[] buffer1 = new byte[0x100];
      while (this.ReadInternal(buffer1, 0, buffer1.Length, true) != 0)
      {
      }
}

It only started to make sense when I did some reading on HTTP (a protocol I'm not deeply familiar with). Apparently, HTTP 1.1 connections are persistent by default, which means the connection is maintained even after the request is completed, and further requests are served from the same connection. Technically, this means that the KeepAlive property of HttpWebRequest is true by default, and a Connection="Keep-alive" header is added to the HTTP request. I can only surmise that, with persistent connections, the response must be read in full in order to allow future requests (if the response was cut off, some concurrency issues may apply). Unfortunately, setting the KeepAlive property to false did not resolve the issue and the connection was maintained until closed.

Since I could find no way to resolve the problem, I decided to hack around it:

stream.GetType().BaseType
    .GetField( "m_doneReading", BindingFlags.Instance | BindingFlags.NonPublic )
    .SetValue( stream, true );

Screwing around with BCL internals is obviously NOT a recommended practise and it's something I tried very hard to avoid, however this problem had to be resolved in one way or another. I traced the code a bit deeper to see if this would actually have the expected results; apparently HttpWebRequest keeps an internal "reference counter" (actually a bitfield, but same principle) on the connection. According to the implementation for HttpWebRequest.connectionRemoveRef, if the reference counter reaches 0 the private m_connection field of the request becomes null, and the connection is released. I started out by testing this:

Before and after - you can see that the m_connection field becomes null

Unfortunately I have no idea how to test whether ot not the connection is actually disposed (conceptually, if KeepAlive is set to false then the connection should be immediately closed); when I find the time I'll try and track the HTTP session using Wireshark, but currently this horrible hack will just have to do.