With my project finally nearing completion, it's nigh time for Microsoft to release yet another update to the .NET Compact Framework 2.0. Service Pack 2 ought to bring it to about "beta 2" level.

Check out these gems:

By far one of the most annoying aspects of the .NET Compact Framework is how heavily it relies on P/Invoke to fill in the gaps. The framework itself is missing huge pieces of functionality, and with the lack of C++/CLI for the Compact Framework a developer is often time left with no choice but to hack around the missing functionality with P/Invoke (or COM interop, if you have the patience to muck about with ATL).

The problem is that, on occasion, a P/Invoke call would result in a MissingMethodException (in fact, if you're really unlucky, the type loader will throw the same exception on loading the method actually making the P/Invoke call). Although a lot of the scenarios have been thoroughly ironed out by now and can be resolved through a Google search or some hacking on the developer's part, there is one scario that is esoteric enough that I couldn't find any references to it on the internet: you can get a MissingMethodException when you are out of memory.

We're working on an extremely large (in mobile proportions) and complex project, involving massive amounts of .NET logic combined with a very large, performance-concious and memory-hungry native codebase. We've had to hack around a lot of missing capabilities in the .NET Compact Framework (as well as some bugs and/or shortcomings in other parts of the OS), and one of our native calls would inconsistently throw a MisingMethodException; having reearched the problem for a day or two I was convinced that the problem was an incorrect function prototype for the exported function and added explicit calling convention declarations. This seemed to have resolved and I was content for a couple of days, until the problem resurfaced.

The exception content itself is next-to-useless, and since the same P/Invoke call would work intermittently I was hoping that enabling the loader log might supply some further information. Alas, all the log would provide was the following line:

Failed to find/load [SomeDll.dll] (even in [\Program Files\Somewhere\])

Not good. I then dug up another article on the subject and proceeded to enable the interop log, which provided some additional information:

JIT ERROR FOR PINVOKE METHOD (Managed -> Native):
[pinvokeimpl][preservesig]
int Workarounds::GetSinkWrapper(out IImageSink , IImageEncoder );
int (I4_VAL) GetSinkWrapper(IImageSink& ** (INTF_REF) , IImageEncoder *(INTF_VAL) );

I was originally interested in the bizarre native signature generation (specifically the IImageSink& ** parameter - where'd the reference come from?), but upon reading some valid log files for working methods I was convinced that it's a dead duck. I then set my attention to the JIT message: what has the JIT to do with a native function? I theorized that the JIT is responsible for the native signature generation for native functions and kept working under that assumption. That narrowed the question down to, "what went wrong with the JIT compiler?"

Eventually it occured to me that this might be yet another manifestation of the memory limitation issue (processes under Windows CE 5.0 are limited to a 32MB address space). The P/Invoke call is made fairly late into the application; I added a dummy function to the library which I called on application initialization. This forced the JIT to take care of the native library before anything was sucking up memory, and the issue was resolved.

The moral of the story? If you get MissingMethodExceptions on your P/Invoke calls, fisrt make sure your DLL is actually deployed; then check the DllImport signature (you can find a lot of useful resources on this FAQ). Finally, make sure you're not out of memory.

Footnote: This is fast becoming a series of posts on the woes of using the .NET Compact Framework. In fact, I've added a subcategory for this exact purpose; you can access the category and its RSS feed from the Categories list on the right-hand side of the screen.

Here's another one of those "bang your head repeatedly against the wall and look for a sharp object" issues: closing a stream derived from HttpWebResponse.GetResponseStream takes an abnormally long time. In fact, the (blocking) call to Stream.Close takes approximately the same amount of time as it would to read to the end of the stream. I've only encountered one other reference to this in a newsgroup post from 2004, which (according to Todd, the original author of the post) was never actually answered.

Seeing as I was trying to use HttpWebRequest to create a minimalistic download manager for one of our applications, I had to come up with a solution: one option was to close the stream asynchronously and just let the download run its course, wasting valuable memory, CPU time and bandwidth; another was to never close the stream and make do with a resource leak. Not content with these solutions I decided to dig into the framework code with the ever-useful Reflector. The first hurdle was locating the assemblies; they're not trivial to find, but if you look hard enough you can find actually useful versions in your Visual Studio 2005 installation folder, under SmartDevices\SDK\CompactFramework\2.0\v2.0\Debugger\BCL. These are regular .NET Assembly PEs so it's trivial to load them up in Reflector.

Some digging into the BCL sources proved that HttpWebRequest does, in fact, read until the end of the stream when closing; this is the relevant code excerpt, from HttpWebRequest.doClose:

if (!this.m_doneReading)
{
      byte[] buffer1 = new byte[0x100];
      while (this.ReadInternal(buffer1, 0, buffer1.Length, true) != 0)
      {
      }
}

It only started to make sense when I did some reading on HTTP (a protocol I'm not deeply familiar with). Apparently, HTTP 1.1 connections are persistent by default, which means the connection is maintained even after the request is completed, and further requests are served from the same connection. Technically, this means that the KeepAlive property of HttpWebRequest is true by default, and a Connection="Keep-alive" header is added to the HTTP request. I can only surmise that, with persistent connections, the response must be read in full in order to allow future requests (if the response was cut off, some concurrency issues may apply). Unfortunately, setting the KeepAlive property to false did not resolve the issue and the connection was maintained until closed.

Since I could find no way to resolve the problem, I decided to hack around it:

stream.GetType().BaseType
    .GetField( "m_doneReading", BindingFlags.Instance | BindingFlags.NonPublic )
    .SetValue( stream, true );

Screwing around with BCL internals is obviously NOT a recommended practise and it's something I tried very hard to avoid, however this problem had to be resolved in one way or another. I traced the code a bit deeper to see if this would actually have the expected results; apparently HttpWebRequest keeps an internal "reference counter" (actually a bitfield, but same principle) on the connection. According to the implementation for HttpWebRequest.connectionRemoveRef, if the reference counter reaches 0 the private m_connection field of the request becomes null, and the connection is released. I started out by testing this:

Before and after - you can see that the m_connection field becomes null

Unfortunately I have no idea how to test whether ot not the connection is actually disposed (conceptually, if KeepAlive is set to false then the connection should be immediately closed); when I find the time I'll try and track the HTTP session using Wireshark, but currently this horrible hack will just have to do.

I've been hacking away quite a bit at an application that contains large managed and unmanaged portions, and a necessarily complex interop layer in between. It appears that interop marshalling behaves somewhat differently between the full and Compact framework. Here's a somewhat laconic list of the issues I've encountered and how to resolve them:

1. You may encounter NotSupportedException on calls to Marshall.SizeOf. Although the documentation does not specify this as a possible exception, experience shows that this is a result of wrong marshalling attributes: for example, although trying to marshal a bool to UnmanagedType.I4 makes sense from a C programmer's perspective, it results in the behavior described above. The article describing this is called "Using the MarshalAsAttribute Attribute," but the version in my locally installed copy of MSDN Library (August 2006) does not contain this information.
2. Annoyingly, the default marshalling, or even an explicit UnmanagedType.Bool, results in corrupt values (probably some minor framework bug). I worked around this by defining the member as int and manually giving it the value 0 or -1.
3. It's not obvious, but you can't use UnmanagedType.LPArray from within structures - it only works on P/Invoke method declaration parameters. The only way to do this is to manually call Marshal.StructureToPtr and do some pointer arithmetic with IntPtr (annoying, but at least it's safe code).
4. The marshaller always frees up memory; although this makes a lot of sense from the .NET perspective, it probably means that you'll have to code in some sort of deep copying mechanism in your native code if you want any of that information in your state. This also has performance repercussions you should consider when architecting your interop layer.

I'll post updates to this as I come across more issues.

For some reason the Visual Studio 2005 debugger (on my native Smart Device project) flat out refused to halt on Data Abort errors. Instead it would just show the error information on the debug trace and crash the application: 

Data Abort: Thread=81e27040 Proc=804c68c0 'Client.exe'
AKY=00000041 PC=00eb1cc0(emnative.dll+0x00001cc0) RA=00eb6d6c(emnative.dll+0x00006d6c) BVA=0e537510 FSR=00000007

For obvious reasons, I wasn't particularly happy with this. Digging into Google and the documentation didn't help; after some serious headscratching I figured that the Windows CE kernel must be catching these exceptions at some point, so from the Debug->Exceptions... menu I enabled, under Win32 Exceptions, catching of thrown Access Violations:

After this, I managed to finally get proper trapping in the debugger, and easily find out what code actually caused the access violation:

It really yanks my chain when I put my faith into what is presumably a solid foundation for my code, and end up running into a huge number of unexpected pitfalls.

Be warned: .NET Compact Framework is incomplete. Oh yes, it's a fleshed out version of CF 1.0 with generics and various important bits and pieces finally included (COM interop. I mean, seriously, .NET is useless without it even on the desktop), but it's still lacking a lot of vital components (ActiveX hosting) and has major shortcomings in others (no XSLT support). But that isn't the worst of it: the documentation is sparse at best, and flat out wrong in some cases.

You cannot use asynchronous delegates in .NET CF 2.0.

In case it isn't clear to you, let me repeat it: you can't use Delegate.BeginInvoke. If you're really unlucky, like me, you'll write a bunch of code and go on to compile, run, test and debug it and then run into a completely bogus NotSupportedException on a seemingly innocuous internal method call, and have to do some serious digging to figure out the culprit. This goes particularly well with (admittedly documented) fineprint in methods such as Windows Media Player's player.URL property setter, which for some reason mustn't be called from an event handler.

Update (January 16th, 2007): Not only does this apparently only work on the emulator, you would do well to stay away from Managed DirectX in general because common drivers, such as those on the Willcom W-Zero3 - do not support rendering in landscape mode (I wonder which devices DO support those features). After messing with this collectively for weeks we eventually went with the obsolete GAPI. The code here will probably not work for you.

For some reason, elementary DirectX operations are not very well documented in the .NET Compact Framework documentation; I kept running into InvalidCallExceptions for no aparent reason, and couldn't figure out the "simple" way to lock surfaces from the documentation (it's worth noting that I was only interested in basic 2D functionality).

Basically you just need the right set of flags and the right order of operations. Here's the code I used and that worked for me (even under the considerably buggy emulator). To create the device, use the following parameters:

PresentParameters p = new PresentParameters();
p.SwapEffect = SwapEffect.Discard;
p.Windowed = true;
p.EnableAutoDepthStencil = false;
p.PresentFlag |= PresentFlag.LockableBackBuffer;
p.MultiSample = MultiSampleType.None;
m_device = new Device( 0, DeviceType.Default, this, CreateFlags.None, p );

To draw on the backbuffer, you can use the following code:

m_device.BeginScene();

using ( Surface s = m_device.GetBackBuffer( 0, BackBufferType.Mono ) )
{
    int pitch;
    using ( GraphicsStream gs = s.LockRectangle( this.ClientRectangle, LockFlags.None, out pitch ) )
    {
        // Your code goes here...
    }   
    s.UnlockRectangle();
}

m_device.EndScene();
m_device.Present();

I hope this helps someone avoid a couple hours of frustration.

Quick link: To download the wrapper classes click here

While working on a large application targetting the .NET Compact Framework 2.0 I realized that I'll need to feed some native code (specifically, the XSLT processor in MSXML 3.0 SP1) with an IStream implementation.

Articles about interoperating with unmanaged code in the CF are not exactly abundant; to save you the time I spent on incorrect and/or conflicting research, here's the bottom line:

• There is no Managed C/C++ for .NET Compact Framework 2.0. To some this may be old news, but you should really pay attention to this point if you're going to do any serious development against CF. 2.0 adds support for managed COM/ActiveX interop, but otherwise you're completely stuck with P/Invoke.
• Although it's not immediately obvious, CF 2.0 does support exposing managed classes via COM.
• The CF is missing some usually-minor classes from the BCL; in this case I was missing System.Runtime.InteropServices.ComTypes.IStream. Annoying but easy to work around.
• Finally, as an aside, MSXML 3.0 SP1 for Windows Mobile does not support the IXslTemplate and IXslProcessor interfaces, meaning that MSXML 3.0's already lackluster performance in XSLT transformations is further hindered by not being able to cache the interpreted stylesheets. This means that, if you use XSLT, your application will not scale. I was not initially aware of this issue, so I hope our data sets are small enough to handle this, or I may yet come to regret the decision to use XSLT in this project.

I managed to save quite a bit of time by leveraging Oliver Sturm's work, which was originally intended for the desktop. Since the CF is missing a whole bunch of minor classes, the managed definition of IStream included, I originally mucked about with midl trying to generate these definitions from the .IDL files. After this proved to be a genuine chore, I just ripped the definitions straight out of the .NET Framework 2.0 assemblies with the ever-useful Reflector.

You can download the class file here. If you use this it would really be cool if you could drop me an e-mail, and I bet Oliver would be equally appreciative. Enjoy!