This is the second post of a two-parter. Please read part 1 for context and additional information.

3rd Party Libraries

As mentioned in part 1, Scala is not binary backwards-compatible between major releases. Although code compiled with 2.10 may work properly with libraries compiled against 2.9.x, this is hardly recommended and may result in a lot of subtle error conditions at runtime that are hard to track down. As a result, the recommended practice is to upgrade all Scala dependencies to versions compiled against 2.10. Fortunately the Scala ecosystem is very active, so most common libraries had 2.10 builds on announcement day, but you are still likely to encounter libraries that need to be upgraded because:

• They're lagging behind the Scala release schedule and simply haven't released 2.10-compatible binaries (fortunately, this seems to be extremely rare; in practice all our dependencies were otherwise resolved);
• The specific version you're using does not have a 2.10-compatible release and likely never will (Squeryl 0.9.5-2);
• A 2.10-compatible official release is out but hasn't yet been published to the central repository (Scalatra 2.0.5).

My experience shows that with a bit of dilligence, practically all our (myriad) dependencies were fairly easy to migrate to 2.10. In most cases this simply entails switching to the appropriate artifact ID suffix (_2.10 instead of _2.9.2, see part 1 for associated rant), in other cases I had to upgrade to a later official release and in rare cases I had to dig in and replace a dependency with an appropriate fork or snapshot version. Here is a list of migration notes for some of our dependencies:

Scala Library Changes

The Scala class library itself has a number of changes you ought to be aware of before migrating to 2.10. The really big change is that Scala actors are deprecated in favor of Akka. You can still use them by importing the scala-actors artifact from the Scala 2.10 distribution, but it is recommended to migrate fully to the new actor system as this is also likely to be obsoleted by 2.10.1. The gentle folk at Typesafe have provided a very comprehensive migration guide to assist your efforts.

The less prevasive API changes we ran into include:

• List.elements is deprecated in favor of List.iterator;
• TraversableOnce.toIndexedSeq no longer takes a type argument. This was actually quite pervasive in our codebase, causing plenty of compilation errors, and is easily worked around by removing the type parameter (which is extraneous to begin with);
• Scala actors' Actor.receive method is now public (previously protected). This had to be rectified in pretty much all of our existing actors by removing the protected modifer;
• Occasional subtle API changes requiring minor code fixes. For example, see this question on StackOverflow.

Summary

Opinions differ among members of our team - some predicted that the migration process will be much more complex whereas personally, given the relatively high level of maturity I've come to depend on in the 2.9 series, the migration process actually ended up being significantly harder than I anticipated. Particularly disconcerting were the occasional compiler failures which took a lot of time to track down. Practically speaking, though, the whole migration process took less than 3 days (including documentation), did not involve additional teammates, and all problems were either resolved or worked around rather quickly. The Typesafe compiler team has been very helpful in analyzing and resolving the single bona-fide compiler bug we've run into, and the community as a whole (on Stack Overflow, Google Groups and elsewhere) was also extremely supportive and helpful.

On the whole, am I happy with the process? So-so. There is nothing here an experienced Scala developer should have serious trouble with, but it will take a lot more stability and predictability for Scala to gain mainstream acceptance in the industry, and that includes a much easier and more robust migration path to upcoming releases (yes, that includes migrating from 2.10 to 2.11 when it comes along). That being said, Scala has been a terrific language to work with in the last couple of years, and the new features in 2.10 (particularly reflection, macros and string interpolation) should make this an extremely worthwhile upgrade. We still have a lot of regression testing to do on the new version, and if anything interesting pops up I'll be sure to post about it separately (or bitch about it on Twitter...)

I just spent a few days migrating our codebase from Scala 2.9.2 to Scala 2.10. A quick search found very little in the way of migration guides and studies, so it seemed appropriate to document my experiences and offer what tips I managed to collect on the way.

General Observations

This process should not be undertaken lightly, and its implementation inevitably depends on your resources and team composition. The group (or person) charged with migrating the codebase should, at the very least, consist of an experienced Scala developer not afraid to get his or her hands dirty.

The migration is not an entirely smooth process - there are plenty of (fortunately fairly minor) breaking changes in the Scala APIs, and since Scala does not yet feature binary backwards compatibility between major versions (2.9→2.10→2.11...), so expect some inevitable library upgrades and the potential complexities inherent in any such update. The specific issues I encountered are documented in their respective sections below.

Unfortunately there is not much I can offer in the way of preparation; these are the obvious steps:

• Familiarize yourself with Scala 2.10. Make sure you know what you’re getting in exchange for investing time and effort on the migration process and early-bird issues;
• Work in an isolated branch and pull changes from the master branch often;
• Commit early, commit often. Try to keep your commits small and well-documented, so you have good revert checkpoints for when things go wrong;
• Keep the affected code to a minimum. The fewer changes, the easier it will be to isolate problems down the road.

If you’ve read so far and still can’t wait for 2.10.1, I hope the next few sections will save you some time and pain.

Toolchain

To start things off I switched the build to use Scala 2.10. Our project runs on Maven in lieu of SBT (topic for another post), and we use David Bernard’s excellent scala-maven-plugin, which deduces the Scala version from the scala-library dependency.

The convention for the Scala version suffix has changed between 2.9 and 2.10: for example, argot2.9.2 is now argot2.10 - no revision for the 2.10 series. This rendered the suffix macro we employ useless, because annoyingly the Scala librariesthemselves (e.g. scalap or scala-compiler) actually use the full version number, so we ended up needing two macros (a “raw” Scala version and a suffix).

Fortunately, at least as far as Maven is concerned the rest was smooth sailing. I next turned my attention to IntelliJ IDEA - I wanted to make sure an existing workspace can be reused with the migrated codebase, otherwise my entire team will have to undergo the inconvenience of starting from scratch on a clean working copy. Fortunately the process turned out to be quite painless (on IDEA 12.0.3 with Scala plugin build 0.7.121), with the following caveats:

• Code analysis appears to miss some cases of stricter compilation compared to 2.9.x (see below);
• Code analysis occasionally identifies relative import statements as erroneous (e.g. import util.Random);
• Project FSC settings had to be manually changed to use the 2.10 compiler bundle (it remained on the default 2.9.2). This proved to be a moot point, because:
• The much-touted IDEA external build mode finally works consistently for the first time (and it uses the significantly better SBT compiler)!
• On the negative side, IDEA does not seem to handle compiler failures (as opposed to compilation errors) gracefully, missing a lot of detail in the output. As I ran into quite a few of these (details below), I ended up doing most of the compilation tests with Maven directly.

Beyond setting up a build job for the new branch, Jenkins posed no issues.
To summarize, from a toolchain standpoint, this was actually a fairly smooth process.

Language/Compiler

I was surprised and somewhat disheartened to find that, after some initial compilation attempts, it became evident that I missed every single mark in my code risk predictions. Where I expected most language-level headaches I encountered none, and seemingly simple and risk-free code ended up taking most of the time working on this process.

First off, the Scala 2.10 compiler is much stricter than the earlier 2.9.x series. It has a few new heuristics that ended up triggering very often on our codebase; some are simply the result of messy code or design, but most are new best practices adopted by the compiler team at Typesafe:

• “warning: This catches all Throwables. If this is really intended, use `case _ : Throwable` to clear this warning.”
  We wrap-and-throw or swallow a lot of exceptions, and for brevity’s sake often use the shorter catch { case _ => }. A catch block is analogous to a partial function from Throwable, and there are undoubtedly cases where the distinction matters. We’ll clean these up iteratively by qualifying that we’re only catching Exceptions.
  
  
• “non-variable type argument B[T] in type A[B[T]] is unchecked since it is eliminated by erasure”
  The 2.10 compiler is actually quite helpful in warning you when generic code may not behave as expected. This is just one example of such a warning.
  
  
• “error: overloaded method methodName needs result type”
  The 2.10 compiler requires you to specify result types on most (all?) types of method overloading. Not much work here.
• Case classes can no longer derive from other case classes, which makes a whole lot of sense if you consider the typesystem; 2.9.x had a whole variety of bugs and edge-cases relating to case class inheritance and companion objects, including some unwieldy constructor syntax. 2.10 simply does away with the complexity and potential ambiguity, which will break existing code. Since such class hierarchies are inherently problematic, I see this as an opportunity to refactor them properly; extractor objects were often useful for that task.
  
  
• In one case we ran into stricter cyclic dependency analysis on the part of 2.10 - an object extended a class and passed a member of another object as a constructor parameter. The second object referenced the first, which compiled fine under 2.9.2 but resulted in a cyclic dependency error with 2.10. As the original design was hard to understand I refactored the code to be a bit simpler, which resolved the problem satisfactorily.
  
  
• Manifests are deprecated in favor of TypeTags and the new reflection library built into 2.10 (certainly one if its most anticipated and celebrated features), but are still functional. We haven’t migrated to the new APIs yet, and that process will likely deserve a post all on its own; read this excellent answer on Stack Overflow for an introduction to TypeTags.
  

Beyond these and the occasional minor syntax change, the 2.10 compiler proved a somewhat difficult beast to tame:

• Ran into crazily-detailed compiler failures that clearly (though long-windedly) indicated that I need to manually add scala-reflect to the classpath;
  
  
• Missing or conflicting dependencies, typically due to mishandling POMs, resulted in what I ended up dubbing the Spontaneous Compiler Combustion phenomenon: a cryptic compilation failure, complete with what appears to be an AST dump and full-blown debug information. Tracking the occasional familiar type name in the compiler log can be helpful in tracking down the dependency at fault (this was the case in all but one of the occurrences), but the error itself is completely inscrutable.
  
  
• The one case was, unfortunately, a proper compiler bug logged as SI-7109, which has to do with consuming a package-protected (or -private) trait/class from another trait/class with the same accessibility. Jason Zaugg of Typesafe (@retronym) was extremely helpful in analysing the compiler output and producing a reproduction case, which I haven’t been able to do on my own. Until a fix is included and released, I’ve worked around this by temporarily commenting out the problematic qualifiers.
  
  
• Update (2013-02-12): Ran into another compiler bug that causes a runtime ClassFormatError. We've managed to identify, reproduce and work around the problem; see SI-7120 on the Typesafe JIRA.
  

Lastly, it’s worth noting that the 2.10 compiler is significantly heavier than the older 2.9.x series: it appears to require 50-100% more permgen space, and compile times are up x2 on average and up to x10 on a clean, fresh build. This seems consistent on both my laptop and the build server, which use different CPUs and OS. A quick check (top and jstat -gcutil) showed the compiler process to be single-threaded and CPU-bound, as it consistently utilizes 100% of a single core. GC activity was low to the point of negligible, so it appears the new compiler is actually a step back in terms of compilation throughput. I hope subsequent 2.10.x releases focus primarily on compilation stability and performance.

That’s it for today; next post will be up in a day or twois up and focuses on the Scala library, dependencies and miscellanea.

This post attempts to summarize and clarify a lecture given at the Botzia (Israeli Java user group) meeting on May 3rd, 2012. You can view the presentation in its entirety on SlideShare:

What is Scala?

Scala is a modern, statically-typed language designed to run on the Java platform. It's commonly perceived as an "evolutionary Java" and with good reason: Scala adds powerful features while significantly reducing Java's verbosity. It is a highly practical language in that it offers a very straightforward migration path for developers on existing platforms: Java developers will feel right at home with Scala, and a .NET developers will not have to give up on a lot of the advantages C# has over Java.

For organizations with a preexisting codebase Scala offers an excellent balance between power and feature set, as well as perfect interoperability with existing Java code: although you certainly don't have to, you can freely use familiar libraries without a hitch, call into existing code and, for the most part, use your existing tool-chain with little or no impact.

Why did we pick Scala?

As mentioned above, Scala offers a very compelling migration path from an existing Java codebase. When we were setting up the new R&D center at newBrandAnalytics, we were already serving paying customers via a complex codebase. A complete rewrite was out of the question as we had contractual obligations, and we had to keep improving the existing product in order to maintain the competitive edge; it became very clear that we needed to maintain the existing codebase and refactor it incrementally.

An additional data point was that our core team members were highly experienced with Java development and quite comfortable with the platform; we also had a distinct preference for statically typed languages, which ruled several alternatives (particularly Clojure) right out. Scala seemed like a great compromise in that it allowed us to maintain and extend the existing codebase while enjoying advanced language features and improved veracity at little or no risk to the product itself.

How should I "sell" Scala to my boss?

"Safe" choice with proven track record: Scala is not an entirely new language, and has reached a maturity tipping point at 2.8; over the last two years the language has been gaining significant momentum, and has a number of strong advocates and success stories, including Twitter, Foursquare, LinkedIn and a whole host of others.

Scala as a better Java: You don't have to go all-in on Scala's advanced feature set, but can instead think of Scala as an improved Java with type inference, closures, advanced collection framework, traits and pattern matching; these features alone will increase your developers' happiness by an order of magnitude, and productivity will see a corresponding improvement.

Less code ⇒ less bugs: Scala code is far more concise than corresponding Java code, an helps the developer focus on the "what" instead of the "how". The resulting code is usually much shorter, clearer and simpler, which in turn helps lower bug count.

Helps hire better engineers: Great engineers not only love working with the latest technologies, but also love practical, incremental improvements to well-respected technologies. Merely saying "we use Scala" will help attract the sort of seasoned, reliable and professional engineers that you most want to hire.

Where does Scala put you at risk?

Learning curve: Although Scala largely derives from Java, it's a whole new language with a lot of features that takes time to master. Don't expect your developers to be productive from day one; instead, provide opportunities for experimentation and encourage iterative development (and yes, give your developers time and mandate to throw away badly written early code and redo their work if necessary). There is actually a lot of material on Scala and an active and vibrant community; specifically, your developers can easily find help (and answers) on StackOverflow.

Rough edges: Scala is fairly new, and consequently a lot of things that you take for granted with Java are not as mature or robust with Scala. IDE support is a moving target: Eclipse and IntelliJ IDEA both have actively developed Scala plugins and both have occasional issues (spurious error highlighting, lackluster performance, wonky debuggers). The common build tools support Scala, but don't play as nicely; for example, only the native build tool (sbt) support incremental compilation. Even the compiler itself is not entirely bug-free, although these are getting very rare indeed. Bottom line: expect occasional problems, and be patient while working around them; even better, encourage your engineers to participate in the community, file bugs and even offer patches where possible.

Production characteristics: While it runs on the JVM, Scala is not Java; there are subtle differences that you should be aware of when maintaining large-scale or highly-available software. Scala generates a great deal of synthetic code, which puts additional pressure on the PermGen space; synthetic stack frames may also exhibit significantly increased stack usage. Scala also creates a lot of intermediate objects at runtime, which results in added eden generation churn. You may want to profile your applications and tune the GC accordingly.

What's the bottom line?

Scala is fantastic. Our team at newBrandAnalytics is remarkably happier with Scala despite the occasional hitches, and I doubt any of us will consider going back to Java given the option. As long as you keep in mind that on the bleeding edge you may get cut, I definitely recommend taking the plunge, with contractors and die-hard traditional enterprises the possible exceptions.

Download ant-intellij-tasks-1.0-b1.zip

A great but oft-ignored feature of Visual Studio 2005 and up is the inherent consolidation of an important developer tool: the build system. With a Visual Studio solution you can simply run MSBuild and you get accurate, automated builds. This is an invaluable capability: continuous integration is ridiculously easy to set up, as are nightly builds and automated deployment tools.

Since I started working for Delver (now Sears) I’ve been switching back and forth between C# (2.0 and later 3.0) and Java 1.6, and though the ecosystems share many similarities there are also several glaring differences. The first of these differences is that, in the Java world, it is perfectly acceptable – even traditional – to maintain a dual project structure, one using the IDE (usually Eclipse or IntelliJ IDEA) and one using one of the build tools (commonly Ant or Maven). The build scripts need to be continuously synchronized with the project structure, and output parity between the two separate build systems is almost unheard-of.

Because I had been a complete Java newbie when I started, I had never had the time to really sit down and set up a continuous integration server for our Java codebase, a mistake I did not intent to repeat when Sears took over. The first item on my agenda was to do away with the dual project structure; we originally used Eclipse, so I built a custom Ant script (my first, actually) around ant4eclipse and managed to come up with a semi-satisfactory solution. This also gave us invaluable insight when it was time to revisit our IDE choice; the lackluster project structure offered by Eclipse, along with firm positive comments on IntelliJ IDEA from several team members, tipped the balance and led us to switch to the alternative IDE, while also creating the necessity for a revamped build system can that work on top of the IntelliJ IDEA project structure.

Out of necessity, a project was born. ant-intellij-tasks is the result of several months of all-night itch-scratching on my part. While not directly affiliated with the company, we’ve been dogfooding the project at Sears for over a month now, and while there are certainly rough edges it finally seems stable enough for release! From the project website:

ant-intellij-tasks is a self-contained build system for IntelliJ IDEA projects based around Apache Ant. In essence, ant-intellij-tasks comprises three components:

1. An Ant task library that can extract and resolve the IntelliJ IDEA project and module files (.ipr and .iml respectively), and provides a set of tasks and conditions around the project structure;
2. A common build script which provides the four major build targets for modules: clean, build, test and package (see the quickstart guide);
3. A master build script which extends these targets to the entire project.

The build system is designed to be extensible (e.g. by adding targets), customizable (e.g. by overriding a target's behavior for a specific module) and self contained in that it's a drop-in solution that should not require any significant modifications to the code base.

This project is fully open source (distributed under an Apache license) and hosted at Google Code. Please report any bugs or issues on the project issue tracker.

ant-intellij-tasks makes use of, and redistributes, the ant-contrib task library.

Apparently Java has quite a few known but practically undocumented issues with its handling of UNC paths under Windows. I’ve specifically encountered this bug albeit in a slightly different scenario:

@Test
publicvoid test() throws URISyntaxException
{
final URI uri = new URI( "file://c:/temp/test/ham.and.eggs" );
new File( uri ); //
IllegalArgumentException thrown here

}

            

              /**
              

               * Resolves the specified URI, and returns the file 
              

               * represented by the URI.
              

               *
              

               * @param uri The URI for which to return an absolute
path.
              

               * @return The {@link File} instance represented by the
              

               * specified URI.
              

               * @throws IllegalArgumentException <ul><li>The
URI cannot 
              

               * be null.</li><li>Wrong URI scheme for
path resolution;
              

               * only file:// URIs are supported.</li></ul>
              

               */
              

              public
              static File
getFile( URI uri )
throws IllegalArgumentException {
if ( uri == null )
thrownew IllegalArgumentException(
"The URI cannot be null." );

            

              

              

              if ( !"file".equals(
uri.getScheme() ) )
thrownew IllegalArgumentException( "Wrong
URI " + 

"scheme for path resolution, expected \"file\" " +
"and got \"" + uri.getScheme() + "\"" );

            

              

              // Workaround for the following bug: 
              

              // http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5086147
              

              // Remove extra slashes after the scheme part.
              

              if ( uri.getAuthority() != null )
try {

uri = new URI( uri.toString().replace( 

"file://", "file:/" )
);

} catch ( URISyntaxException e ) {
thrownew IllegalArgumentException( "The
specified " +
"URI contains an authority, but could not be " +
"normalized.", e );

}

            

              

              

              return
              new File(
uri );

}

          

It seems nothing to do with maintaining this website is as easy or as simple as it should be. Whenever I switch hosts it’s an uphill struggle to get the site up and running again; whenever I upgrade dasBlog to a newer version I have to learn a lot about how it works, how ASP.NET works, how IIS is configured etc. It’s enough to make me seriously consider hosting my blog elsewhere and/or moving to another blogging platform, but the truth is I love dasBlog so much I simply forget how complex and volatile it can be and have to go through the same frustrating process whenever something changes.

The way I update my site is usually this:

1. Ensure I have an up-to-date local mirror of the website. dasBlog keeps all of its data in XML files, so backing up the website is simply a question of wget –-passive-ftp –m ftp://user:password@website.com; I have a daily scheduled task to take care of this.
2. Copy the latest mirrored version to a working directory; set the directory up as an IIS website/virtual directory.
3. Test the new working copy to make sure it works.
4. Perform whatever modifications are required.
5. Test again to make sure that the website works with multiple browsers (this time I tested with Chrome 2.0.170.0, Firefox 3.1 Beta 3 and IE 8)
6. Upload the website over FTP using the FileZilla client. I always verify that the relevant configuration files and binaries are overwritten and nothing else.

This process generally allows me to test upgrades before uploading them to the “production” website, as well as provides an easy rollback path if something goes wrong. Thing is, something always goes wrong. In this case, although nothing’s changed in the site configuration I started getting SecurityExceptions just after the upgrade:

Request for the permission of type 'System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.

This is one of the least informative error messages I have ever seen. All it tells me is that a request for some permission is denied; it doesn’t say what permission was requested, nor by whom (the stack trace seemed to indicate the permission was asserted from within System.Diagnostics.Trace, which doesn’t make much sense). A quick web search brought me to this page, which deals specifically with installing dasBlog on a GoDaddy-hosted website. Because GoDaddy runs ASP.NET applications under a modified medium trust that allows file-system access only to the virtual directory hierarchy, the site recommends adding a virtual directory for each of dasBlog’s writable directories (content, siteconfig, logs); I tried this out and the problem was not resolved.

At this point I was getting desperate, and was willing to try just about anything to get the site up and running again. Eventually I ran a diff between the site backup and the newly modified version, and found a new openidConsumerTrace.txt file in the site root. I’ve never seen that one before; where'd it come from? A quick search showed the following section in the web.config file:

            <
            system.diagnostics
            >
            <
            assert
            assertuienabled
            ="false"
            />
            <
            switches
            >
            <
            add
            name
            ="OpenID"
            value
            ="4"
            />
            </
            switches
            >
            <
            trace
            autoflush
            ="true"
            indentsize
            ="4"
            >
            <
            listeners
            >
            <
            add
            name
            ="fileLogger"
            type
            ="System.Diagnostics.TextWriterTraceListener"
            initializeData
            ="openidConsumerTrace.txt"
            traceOutputOptions
            ="None"
            />
            </
            listeners
            >
            </
            trace
            >
            </
            system.diagnostics
            >
          

A-ha! So the OpenID activity trace log is written to the virtual root, which is not writable (I set the ACLs to only allow writes to the above three directories). I tried changing the trace file path to ~/logs/openidConsumerTrace.txt (which is a virtual directory and has the appropriate write ACL), but this did not resolve the problem. At this point I was ready to roll back to the previous version and work on switching to another (perhaps hosted) blogging platform, and in my despair I simply commented out the whole system.diagnostics section; oddly enough, this resolved the problem…

Now I know dasBlog is free and there’s little or no point complaining, so I hope this post helps someone handle the problem. And if anyone from the dasBlog team is reading this… please be a little more careful with undocumented dependencies?

Update (8 May 2009): Eli Ofek pointed out in the comments a little-known but effective tool called the Managed Stack Explorer. Although it features a basic GUI, it can effectively be a .NET equivalent of jstack if you add to the path; then it’s just a question of typing mse /s /p <pid>. It’s a little slower than jstack but worlds better than the alternative suggested below.

I’ve been working mostly with Java over the last year, and the .NET code I write is usually limited to interface code between our .NET-based and Java-based components. A long while away from production-grade code on Windows means I need to brush up on my production debugging skills.

On today’s menu: thread dumps, or per-thread stack traces if you will. With Java code (at least starting with Java 5) this is as easy as jstack <pid>; with .NET it turns out to be quite a bit more complicated:

1. Download and install the appropriate Debugging Tools for Windows version for your architecture (x86/x64/Itanium)
2. If you need information about Windows function calls (e.g. you want to trace into kernel calls), download and install the appropriate symbols. This isn't strictly necessary if you just want a thread dump of your own code.
3. If you need line numbers or any other detailed information, make sure to place your assemblies' PDB files where the debugger can find them (normally you just put them next to your actual assemblies).
4. Start->Programs->Debugging Tools for Windows [x64]->windbg
5. Attach the debugger to your running process using the menu
6. Load the SOS extension with ".loadby sos mscorwks" for .NET 2.0 (".load sos" for .NET 1.0/1.1)
7. Take a thread dump using "!eestack"
8. Detach using ".detach"

Quite a bit of work for something as trivial as a thread dump. I hope .NET diagnostic and debugging tools improve with time (Process Explorer is definitely a step in the right direction).

Don’t ya think?

I was making modifications to one of our components, and running all of the unit tests revealed that all database-dependant integration tests were failing:

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 0 ms ago.
<snip> (cut for brevity’s sake)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2104)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:729)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)

Strange error message, but as it turns out the inner exception was far more revealing:

java.net.ConnectException: Connection refused: connect

As can be expected, the local MySQL server was up and running, and I was able to connect with the command line tool as well as with SQLYog, so it was obviously not a problem with MySQL or the local firewall. Next up I tried to telnet to the appropriate port (the easiest way I know to check port-level connectivity) without success:

I next tried to connect to the loopback IP (127.0.0.1), and experienced a major WTF moment when the connection succeeded. I use Windows Server 2008 and, as it turns out, it supports IPv6 out of the box. localhost has a slightly different meaning under IPv6 (it maps to ::1), and as I understand it traditional IPv4 traffic is tunneled over the looback IPv6 connection; I’m not yet familiar enough with IPv6 to draw any conclusions on why the above shouldn’t work, but the bottom line is there are several ways of resolving the problem:

1. Edit your hosts file (it’s hidden under Windows Server 2008, but you should be able to Start->Run->notepad %SYSTEMROOT%\System32\Drivers\etc\hosts) and change the mapping for localhost from “::1 localhost” to “127.0.0.1 localhost”. This does resolve the problem, although I can’t say what impact this will have on IPv6-enabled applications.
2. Set the TCP stack to prefer IPv4 to IPv6 when attempting to connect (it’s the reverse by default). According to this forum post, this entails setting the registry value HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\tcpip6\Parameters\DisabledComponents to the DWORD value 0x20.
3. Disable IPv6 altogether for your network connection: remove the IPv6 protocol from your network connection component list. At this point in time IPv6 is still very rare so I doubt this will cause any significant issues, but YMMV.

For me, changing the hosts file was the quickest solution because it works and is easy to revert. I’ll have to keep a very careful eye on the behavior of my machine though.

If you've ever tried to develop using PHP on IIS7 (Vista), you'll find that errors in your script result in the default IIS7 "friendly" HTTP 500 error page, which is useless for debugging. This happens in both FastCGI and ISAPI modes.

To save you hours of crawling through the 'net, the solution (found on this forum post) is very simple:

• Start a command prompt;
• Copy-paste the following: %windir%\system32\inetsrv\appcmd.exe set config -Section:system.webServer/httpErrors -errorMode:Detailed
• Run iireset
• Enjoy.

Update: I was wrong, and private setters do not appear to work (strange, I’m sure I verified this in my test code, but it didn’t work in practice). See below.

It’s been ages since I used XmlSerializer, or even written meaningful amounts of code in C# for that matter, which is why I was utterly stumped by this problem. Try running this code:

[XmlRoot( "test" )] public
class Test { private
readonly string _string1; private readonly string _string2; public Test( string string1, string string2
) { _string1 = string1; _string2 = string2; } [XmlAttribute( "attr" )] public
string String1 { get { return _string1;
} } [XmlElement( "element" )] public
string String2 { get { return _string2;
} } }
/// ... XmlSerializer xs
= new XmlSerializer( typeof( Test ), "" ); XmlSerializerNamespaces xsn
= new XmlSerializerNamespaces();
xsn.Add("", ""); //
Gets rid of redundant xmlns: attributes StringWriter pw
= new StringWriter();
xs.Serialize( pw, new Test( "1", "2" ),
xsn ); Console.WriteLine( pw.GetStringBuilder().ToString()
);

              <?
              xml
              version
              ="1.0"
              encoding
              ="utf-16"?><test/>

The solution? Add a setter to each of your serializable properties. Don’t need a setter because the class is immutable? Mark it as private and you’re good to go. Tough – you’re going to need one anyway, private/internal/protected setters don’t appear to work. If you must use XmlSerializer you should throw a NotImplementedException from these setters, but in my opinion the resulting contract clutter implies you should simply avoid XmlSerializer altogether.

Although this behavior makes sense in light of how XmlSerializer can be used for both serialization and de-serialization, what threw me off was that no exception is thrown – the contract doesn’t require a setter property, and the serializer output is corrupt. Beware!

Because the Java language lacks delegates, anonymous classes are prevalent as a syntactic replacement. Non-static nested classes are also often used in the language, a feature which is conspicuously absent from C#, albeit far less necessary with that language.

This brings me to the following language caveat. This may seem a contrived example, but it's a simplification of an actual issue I've encountered in the last few days. Suppose you have a generic base class, call it BaseClass<U>, which you extend with an anonymous class. Lets also assume that the extended class spawns a thread that needs to access the BaseClass<U> state:

            class BaseClass<U> { void getState()
{} } class Test { publicvoid test()
{ final BaseClass<String> instance = new BaseClass<String>()
{ publicvoid invokeStatefulThread()
{ // Create our runnablefinal Runnable
threadCode = new Runnable() { publicvoid run()
{ /* 1 */ getState(); /*
2 */this.getState(); /*
3 */super.getState(); /*
4 */ BaseClass.this.getState(); /*
5 */ BaseClass<String>.this.getState();
} }; new Thread( threadCode ).start(); } }; instance.invokeStatefulThread();
} }

I'll spare you the guessing game. Here's what happens with each of the five invocations of getState():

1. Compiles and behaves as expected.
2. Obviously won't compile; this points to the Runnable.
3. Obviously won't compile; the superclass of a Runnable is Object.
4. Although this is the correct raw class, it won't compile because "No enclosing instance of the type BaseClass<U> is accessible in scope", even though the raw type should still be accessible and U can be inferred.
5. Although this appears to be the correct fully-qualified form, this does not compile with a "Syntax error on token(s), misplaced construct(s)" error.

The Java language specification section on "qualified this" is very brief and does not mention generics at all (does "class name" include bounded type parameters?). Oddly enough, moving the class declaration outside of the test method actually lets 4 compile -- if there's a clue there, I haven't figured it out yet.

I still haven't found a syntactically-correct way to access BaseClass<string>.this, other than placing it in a temporary variable outside of the Runnable declaration. I searched yesterday for a couple of hours with no obvious solution in sight. Ideas are more than welcome!...

Among a lot of other tools and technologies we use Lucene here at Delver. Lucene is a very mature, high-performance and full-featured information retrieval library, which in simple terms means: it allows you to add search functionality to your applications with ease. I could spend an entire day talking about Lucene, but instead I'd like to show you how to write scripts for Lucene with Python on Windows (and thus differs from Sujit Pal's excellent instructions on the same subject matter).

First, make sure you have your Python environment all set up -- at the time of writing this I use the Python 2.5.1 Windows installer. Next you'll need Python bindings for the library, so go and grab a copy of PyLucene; to make a long story short I suggest you use the JCC version for Windows which is downloadable here. Grab it and install it, no fussing necessary. Finally you'll need the Java VM DLL in your path, and this depends on which type of JRE you're using:

• If you're using the Java JDK, add the mental equivalent of C:\Program Files\Java\jdk1.6.0_03\jre\bin\client to your path;
• If you're using a JRE, add C:\Program Files\Java\jre1.6.0_03\bin\client to your path.

(As an aside, you should probably be using %JAVA_HOME% and %JRE_HOME% and indirecting through those.)

Now you can quickly whip down scripts in Python, like this one which took about two minutes to write:

            #!/usr/bin/python # # extract.py -- Extracts
term from an index # Tomer Gabel, Delver, 2008
# # Usage: extract.py <field_name> <index_url> #
            import
            sys
            import
            string
            import lucene
from lucene import IndexReader, StandardAnalyzer,
FSDirectory def usage(): print "Usage:\n" printsys.argv[
0 ] + " <field_name> <index_url>" sys.exit(
-1 ) def main(): if (
len( sys.argv ) < 3 ): usage() lucene.initVM(
lucene.CLASSPATH ) term = sys.argv[ 1 ] index_location
= sys.argv[ 2 ] reader = IndexReader.open(
FSDirectory.getDirectory( index_location, False ) ) try: for i in range( reader.maxDoc()
): if ( not reader.isDeleted( i ) ): doc = reader.document(
i ) print doc.get( term ) finally: reader.close()
main()

Note the VM initialization line at the beginning -- for the JCC version of PyLucene this is a must, but even like so using the library is extremely painless. Kudos to the developers responsible!

The Java implementation for generics is radically different from the C# equivalent; I won't reiterate issues that have been thoroughly discussed before, but suffice to say that Java generics are implemented as a backwards-compatible compiler extension that works on unmodified VMs.

The implications of this are considerable, and I'd like to present one of them. Lets fast forward a bit and consider a relatively new language feature in Java (introduced, I believe, with J2SE 5.0): autoboxing. A thoroughly overdue language feature, autoboxing allows the seamless transition from regular value types (e.g. the ubiquitous int) to object references (Integer); before autoboxing you couldn't simply add a value to an untyped ArrayList, you had to box (wrap) it in a reference type:

ArrayList list = new ArrayList();
list.add( 3 ); // Compile-time error list.add( new Integer(
3 ) ); // OK

Eventually Java caught up with C# (which introduced autoboxing in 2002), and with a modern compiler the above code would be valid.

With the introductions out of the way, here's a pop-quiz: what does the following code print?

HashMap<Integer, Integer> map = new HashMap<Integer,
Integer>(); map.put( 4, 2 ); short n = 4; System.out.println(
Integer.toString( map.get( n ) ) );

As a long-time C# programmer I was completely befuddled when the code resulted in a NullPointerException. Huh? Exception? What? Why?

It took me a while to figure it out: because Java generics are compile-time constructs and are not directly supported by the VM, what actually happens is that the underlying container class accepts regular Object instances (reference types); the compile-time check merely asserts that n can be promoted from short to int, whereas the actual object passed to the container class (via autoboxing) is a Short! Since the container doesn't doesn't actually have a runtime generic type per se, the collection merely looks up the reference object in the map, fails to find it (I guess the Object.hashCode implementation for value types simply returns the reference value as the hash code as in C#) and returns null. Doh! *slaps forehead*

A few ReSharper tips for your viewing pleasure (according to rumors it can extend your penis too!):

Alt+Shift+L locates the current file in the Solution Explorer (at least with the IDEA-style keyboard bindings we all know and love). This is particularly useful for extremely file-heavy solutions where you often want to Get Latest (or Undo Check Out) the current file.

Note: you can get this behavior automatically through Tools->Options->Projects and Solutions->Track Active Item in Solution Explorer, but in my opinion it's annoying as hell.

Easier unit testing: you can bind shortcuts to run or debug unit tests by context. Context means that the behavior is exactly as expected: using the shortcut with the cursor on a test will run just that test, using it on a class will run it as a whole fixture, and using it on a directory in the Solution Explorer will run all tests that reside in that directory. This is easily accomplished by going to the Tools->Options->Keyboard menu and binding:

• ReSharper.ReSharper_UnitTest_ContextDebug to whichever shortcut you wish for debugging purposes (my favourite is Ctrl+T, Ctrl+D)
• ReSharper.ReSharper_UnitTest_ContextRun to whichever shortcut you'd like to use in order to just run tests (my favourite is Ctrl+T, Ctrl+T)

Note: You can get the same behavior with the ubiquitous TestDriven.net, but it makes no sense to me to pay twice for the same functionality.

Quick link: download

In my work at Semingo I often encounter situations where it's impossible to unit- or integration-test a component without accessing the web. This happens in one of two cases: either the component itself is web-centric and makes no sense in any other context, or I simply require an actual web server to test the components against.

Since I firmly believe that tests should be self-contained and rely on external resources as little as possible, a belief which also extends to integration tests, I wrote a quick-and-dirty pluggable web server based on the .NET HttpListener class. The unit-tests for the class itself serve best to demonstrate how it's used; for instance, the built-in HttpNotFoundHandler returns 404 on all requests:

    [Test]
    [ExpectedException( typeof(
WebException ) )] [Description( "Instantiates an HTTP
server that returns 404 on all " +

"requests, and validates that behavior."
)] publicvoid VerifyThatHttpNotFoundHandlerBehavesAsExpected()
{ using ( LightweightWebServer webserver = 

 new LightweightWebServer( LightweightWebServer.HttpNotFoundHandler
) ) { WebRequest.Create( webserver.Uri ).GetResponse().Close(); } }

    [Test]
    [ExpectedException( typeof(
AssertionException ) )] publicvoid VerifyThatExceptionsAreForwardedToTestMethod()
{ using ( LightweightWebServer webserver = new LightweightWebServer( delegate {
Assert.Fail( "Works!" ); } ) ) { WebRequest.Create(
webserver.Uri ).GetResponse().Close(); } }

    [Test]
    publicvoid VerifyThatContentHandlerReturnsValidContent()
{ string content = "The
quick brown fox jumps over the lazy dog"; using (
LightweightWebServer webserver = new LightweightWebServer( delegate(
HttpListenerContext context ) { using ( StreamWriter
sw = new StreamWriter( context.Response.OutputStream
) ) sw.Write( content ); } ) ) { string returned; using (
WebResponse resp = WebRequest.Create( webserver.Uri ).GetResponse() ) returned = new StreamReader(
resp.GetResponseStream() ).ReadToEnd(); Assert.AreEqual( content, returned ); } }

We use this class internally to mock anything from web services to proxy servers. You can grab the class sources here -- it's distributed under a Creative Commons Public Domain license, so you can basically do anything you want with it. If it's useful to anyone, I'd love to hear comments and suggestions!

So you want a web project, a build system and a reasonable IDE to take care of the annoying details for you, right? The good news are that it's actually quite possible and there're many ways to do this. The bad news are that it's nigh impossible to get them to play along if you don't already know how to do that. It took me days to find a solution that finally seems to work, and I'd like to share it with you. I'm probably missing a few important details or did something really really stupid along the way (I'd appreciate comments!), but this process does seem to work. I'm not going into essentials of Java-based web development here -- if you want a more basic explanation of the terminology, post a comment and I'll see what I can do...

1. Goal

I want to:

• Use a common, standard and powerful IDE to edit and debug my Java code, and preferably provide a usable GUI interface for dependency and project management;
• Use a common, standard servlet container to host my servlet and still be able to control and debug everything from the same IDE;
• Have a convenient way to handle internal-and-external dependencies without worrying too much about the details;
• Be able to quickly compile, test and package my servlet for deployment;
• Understand as little as possible about the dependency stack of the tools involved

I'm going to tell you how to achieve most of these goals, with two glaring omissions: I won't show you how to do testing (I haven't successfully managed servlet unit testing so far -- different post on that) and I can't help but delve into some of the more annoying details involved with these tools and their dependencies. Sorry about that. Additionally, some of the information here applies even if you use different tools, but you're bound to face issues not covered here; don't assume I know more than you do -- seek the answers, post them somewhere, and maybe the next person will actually find what they're looking for!

2. Tools

The tools used are:

• J2SE JDK is an obvious must-have. Version used: JDK 6 update 3;
• Eclipse (but please don't download it just yet) for code editing, debugging and project management;
• Web Tools Platform: this is the Eclipse plug-in that adds web development capabilities to the IDE, including J2EE dependency management, hosting and running servlets from within the Eclipse workspace etc. This would be a good time to run along to the WTP web-site and download the Web Tools Platform All-In-One package. I only used the release version (2.0.1 at the time of writing this), so if you use another version your mileage may vary;
• Apache Maven is the newfangled build system from Apache slated to replace ant. I've used it for the last few days and so far it appears to be quite robust and even fairly well-integrated into Eclipse (see next item). Version used: 2.0.8;
• M2eclipse is the Eclipse plug-in for Maven integration. I've only found one problem with it so far, which I'll detail later on;
• Apache Tomcat is a solid choice in servlet containers. It's robust, fast and open-source, and has terrific Eclipse integration. I haven't given any of the other containers a serious whirl yet though.

3. Preparations

Unlike Visual Studio, with the tools mentioned above there's no straightforward installation procedure. You'll have to designate at least a workspace directory (where your Eclipse projects, settings etc. go) and some location where the tools themselves go. For me, it's C:\Dev\Eclipse and C:\Tools respectively.

• Setting up Java:
  • Install the JDK and remember where it was installed (nominally in %PROGRAMFILES%\Java\jdk1.6.0_03)
  • Set up a system-wide JAVA_HOME environment variable pointing to the same directory
• Setting up Maven and Tomcat:
  • Extract both archives to your designated directory (e.g. for Maven it would be C:\Tools\apache-maven-2.0.8)
  • Add the Maven bin directory to your PATH environment variable (user- or system-wide, depending on your preference)
  • Add whichever J2EE libraries you desire from the Tomcat installation to your class-path. If you have no idea what I'm talking about, you'll probably just want to set the CLASSPATH environment variable to your equivalent of c:\tools\apache-tomcat-6.0.14\lib\servlet-api.jar;c:\tools\apache-tomcat-6.0.14\lib\jsp-api.jar
• Setting up Eclipse:
  • Extract the WTP all-in-one package (which contains Eclipse itself) to your designated directory (e.g. C:\Tools\Eclipse)
  • Load Eclipse and point it to your designated workspace location
  • Install M2Eclipse:
    • Go to Help->Software Updates->Find and Install..., select "Search for new features to install" and click Next
    • Click on New Remote Site..., use M2eclipse or whatever for the name and http://m2eclipse.codehaus.org/update/ for the URL
    • Click on Finish and let Eclipse install the M2Eclipse plug-in
  • Set up a web server runtime for Eclipse to host your servlets in:
    • Open Window->Preferences...
    • Under Server select Installed Runtimes and click on Add...
    • Choose (from Apache) the Apache Tomcat v6.0 runtime and click Next
    • Enter the Apache installation directory (e.g. C:\Tools\apache-tomcat-6.0.14) in the appropriate location and click Finish

4. Creating a new web project

First off, you must create the actual project, directory structure etc. To do this:

• Open a command prompt, go to your Eclipse workspace directory
• Decide on your Maven group and artifact IDs; it's worth noting that the artifact ID is also the directory name for the project
• Type in mvn archetype:create -DgroupId=your.group.id -DartifactId=your.artifact.id -DarchetypeArtifactId=maven-archetype-webapp
• You'll notice that a new directory was created under the workspace root
• Edit the Maven project descriptor POM.XML in the newly created directory:
  • Add (after the <url> tag, although I'm not sure the order matters) the following section:

    <profiles>
        <profile>
            <id>servlet</id>
            <activation>
                <activeByDefault>false</activeByDefault>
            </activation>
            <dependencies>
                <dependency>
                    <groupId>javax.servlet</groupId>
                    <artifactId>servlet-api</artifactId>
                    <version>2.5</version>
                    <scope>provided</scope>
                </dependency>
                <dependency>
                    <groupId>javax.servlet.jsp</groupId>
                    <artifactId>jsp-api</artifactId>
                    <version>2.1</version>
                    <scope>provided</scope>
                </dependency>
            </dependencies>
        </profile>
    </profiles>

  • Under <build>, add the following section:

    <plugins>
        <plugin>
            <groupId>org.mortbay.jetty</groupId>
            <artifactId>maven-jetty-plugin</artifactId>
        </plugin>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.5</source>
                <target>1.5</target>
            </configuration>
        </plugin>
    </plugins>

• You can now thank my colleague Aviran Mordo for finding out this bit of Voodoo. :-)
• In the command prompt, now enter the project directory
• Type in mvn eclipse:m2eclipse -Dwtpversion=1.5 to create an Eclipse project
• Run Eclipse if it's not already started, then from the package explorer right click anywhere and click on Import...
• Choose General->Existing Projects into Workspace. For root directory pick the workspace directory
• Choose the new project and click on Finish
• At this point you may encounter a "Java compiler level does not match the version of the installed Java project facet" error. If that's the case, just right-click on the error (in the Problems view) and select Quick Fix, which will allow you to change the Java project facet version to 6.0. If this isn't what you want, you probably know enough to resolve the issue on your own...
• You'll need a src/main/java directory as a root source folder (as per the Maven convention). Right-click on the project, select New->Source Folder and type in src/main/java.
• Finally, in order to execute or debug the project on an actual running server, right-click on the new project and select Properties. From there go to the Server tab and select the runtime you created in the previous chapter.

At this point you have a Maven web project with a corresponding Eclipse project ready for editing in your workspace. In practice you will have to do several things to have any meaningful results.

1. Add your own code into the mix, such as a servlet. When adding a new servlet (via right-clicking the project, New->Other and choosing Web->Servlet) your WEB.XML file is automatically updated with the new servlet.
2. Add your own dependencies. Maven handles dependencies quite well; for instance, in order to actually create a servlet you're going to need servlet-api.jar in your classpath; the easiest way to do this is to right-click the project, select Maven->Add Dependency and then simply type in servlet and choose javax.servlet servlet-api.
3. When you wish to run or debug your servlet, right-click on its Java file and select Run As->Run on Server (or Debug, as appropriate). Your applet should be happily up and running.

5. Converting an existing project to Maven

I'm not sure how to go about doing this for web projects, but converting regular projects to use Maven is actually pretty straightforward; move your sources to the appropriate directories according to the Maven conventions, right-click the project in Eclipse and choose Maven->Enable Dependency Management; this will implicitly create a Maven project descriptor for you (POM.XML) and that's pretty much it. From that point on your Eclipse and Maven projects should peacefully coexist, allowing you to leverage both tools for your purposes.

In case it wasn't obvious, I've been doing some Java development lately. One of the curious things about doing development in the Java world is that, whereas in the Microsoft world you get a fairly complete tool-chain direct from a commercial vendor, in the Java world you're pretty much dependant on the open-source ecosystem built around the essential Java technologies: Sun defines the APIs, the community provides the tools. In many ways this is really really cool: many Java tools like JUnit are so absolutely groundbreaking that they found their way into the common development idiom irrespective of language, and the availability of tools for just about any purpose is a tremendous advantage (being able to choose freely between Resin, Jetty, Tomcat or any other commercial container, for instance, is a huge boon).

This diversity and community-centric development ecosystem definitely comes with a price though. Java tools, even the high-profile ones such as Eclipse, are extremely difficult to work with for the uninitiated, with a learning curve somewhat like that of Linux: if you take the time to learn the tools you can do astounding things and remain in complete control of the system, but the sheer context required to do even the most trivial thing can be - and often is - daunting.

I've been battling these tools on and off for the last few weeks and often end up having to figure something out on my own. Unlike the .NET ecosystem, it's usually quite difficult to find a blog post detailing a solution to a particular problem. To that end I intend to document my successes - victories, if you will - over the tool-chain, and also the problems I encounter and haven't been able to solve, in the hope of helping others and maybe myself in the process. These posts will go under the Development->Java category, and I'd really appreciate any comments on the solutions (so that I can improve my own work) as well as the problems (so I can actually solve them). Here's to hoping :-)

I've been developing software with .NET professionally for the last five years or so, and aside from the occasional foray into other languages I've more or less specialized in that environment. While merrily hacking away at our back-end here at Semingo, we've recently made the decision to develop an aspect of said back-end in Java. As it's always a good practice to keep an open mind and experiment with other technologies I've happily accepted the challenge.

After working with Java and its associated tools for the past three or so weeks I have several observations to make:

1. The prominent free Java IDE, Eclipse, is actually a very full featured and impressive platform but takes a lot of getting used to. Some of the idioms and concepts are radically different than Visual Studio; for instance, whereas in Visual Studio you'd create an "ASP.NET Application Project", in Eclipse you create (or convert to) a dynamic web project and then add something called facet to your project; for instance, a Dynamic Web Module facet allows you to easily create and debug servlets, and the "Axis2 Web Services Core" facet allows you to create Axis2-based web services and work on them from within your IDE. To actually make use of these features, however, one needs to develop a pretty hefty knowledge base on the various technologies involved (J2EE and servlets, servlet containers like Tomcat, WTP etc.)
   
2. Eclipse is next-to-useless without some tinkering; in particular, what I originally attributed to very immature web development plug-ins - the WTP umbrella project I already mentioned - turned out to be the default memory settings of the Eclipse launcher. The launcher hosts the Java VM and its baseline configuration is simply inadequate. In my case adding the following switches: -vmargs -Xmx512M -XX:MaxPermSize=128M to the command line resolved all of the problems I had with the various WTP plug-ins, as well as the myriad crashes I've experienced with the IDE. In fact it's rock-stable now.
   
3. The Java language has some unexpected caveats; for instance, whereas in C# the designers eschewed fall through in the switch statement (you can group labels to implementations, but you cannot fall through from the implementation of one case statement to the next), the Java designers elected to maintain C-style behavior. I'm of the belief that switch statement fall through is the cause of a huge number of subtle, hard-to-find bugs, and was surprised to learn of this discrepancy between the two languages.
   
4. Enumerations in Java, a relatively new feature added in 1.5, are an impressively diverse feature which is a great deal more powerful than its C# counterpart. I only wish the designers would also allow for a more simplified "SOME_CONSTANT = 3" type syntax, as it's somewhat cumbersome to have to actually use constructors for the purpose. Additionally Java does not (to my knowledge) support implicit conversion operators, which makes necessary constructs such as SomeEnum.CONSTANT_VALUE.getConvertedValue(). It's not a huge issue but it's one of the many areas where syntactic sugar in C# is useful.
   
5. Speaking of syntactic sugar, there're several aspects where Java simply falls short of C#: disposables, iterators and delegates. Yes, I know delegates are an essentially religious issue for the Java designers (mostly for historical reasons, I suspect), and I won't deny that anything you can do with delegates you can do with nested classes, but at ridiculous verbosity. As for disposables, I find that the using keyword in C# is one of the most useful language constructs I've ever encountered, the use of which goes way beyond the original intention of elegantly scoping unmanaged resource use; finally, iterators are tremendously useful and cut a lot of unnecessary boilerplate code out of the equation.
   
6. The Java ecosystem is riddled with code- and buzz-words, to the point of being annoying. If you thought .NET has too many sub-technologies and acronyms, you should try Java. Just to get the taste buds going, here are some of the keywords I've been messing with for the past couple of weeks: J2SE, J2EE, Servlet, Eclipse, WTP, (Apache) Tomcat, Axis2, SAX, JAXP, JAX-RPC, JAR, WAR, AAR, EAR, JDBC, JavaDoc, JSP and JavaBeans. And that's just off the top of my head! To someone with any sort of Java experience this list wouldn't seem intimidating or even exhaustive, but to a new-comer that's simply too much. There is also an import cultural distinction: Visual Studio and its associated technologies (.NET, ASP.NET, ADO.NET etc.) are designed to ease you in as you learn the ropes; I found it much easier to simply start working with them and learn as I go, whereas with the Java counterparts I usually found myself trying to rework code samples found on the 'net while scratching my head.
   Now don't misunderstand me: the Java technologies are generally impressive, mature and usable, but the learning curve is not nearly as comfortable as the competing technologies from Microsoft, and the tools and documentation just aren't as polished.

The Windows SDK command shell, setenv.cmd, is immensely useful, so much so that I wanted it as my default command prompt (i.e. when CMD is run, no matter by whom). A quick Google search didn't turn out anything, so I eventually figured it out myself. The trick is to add it to the command processor's AutoRun value in the registry (run cmd /? from the command prompt if you don't know what I'm talking about):

reg add "HKCU\Software\Microsoft\Command Processor" /v AutoRun /f /t REG_EXPAND_SZ /d "\"%programfiles%\Microsoft SDKs\Windows\v6.0\Bin\SetEnv.Cmd\" /debug /x86 /vista"

You'll notice that I explicitly set the arguments for setenv.cmd; I can't explain it (nor bothered to delve into the script), but without these arguments the script gets stuck along with the command prompt. You should obviously change the values to your own environment.

Our test code makes extensive use of a lightweight web server implementation based on the .NET 2.0 HttpListener class (I'll write a separate post on this, source code included, in the near future). The web server implementation randomizes an incoming port and exposes its URI via a property; this is done for two reasons: to avoid conflicts with other listeners on the machine, and to facilitate multi-threading test runners (like TestDriven.NET). Even with a serial test-runner (like ReSharper's unit test driver, which is what most of us at Semingo use) this means that for an average test suite a large number of web-server instances are initiated and disposed of in very rapid succession. Our implementation creates each web server on its own dedicated port (using Socket.Bind to make sure that the port is free), and closes the listener via HttpListener.Abort when the test ends.

Although the entire test suite seems to run fine locally (both via ReSharper's test runner and using NUnit-Console), at some point we started getting strange errors from our continuous integration server:

System.Net.HttpListenerException : Failed to listen on prefix 'http://+:40275/' because it conflicts with an existing registration on the machine.

I verified this by running the tests with NUnit-Console on the CI server and it was consistent (but only on that server). Since the only instances of HttpListener used throughout the test suite are those used by the test web servers, this means that HttpListener.Abortdoes not properly unregister its own prefixes. Since the documentation for HttpListener is rather sparse and I couldn't find any mention of this issue on the web, I eventually went the Reflector route. Check out the Reflector decompiler output for both HttpListener.Dispose (called via Close) and HttpListener.Abort methods:

HttpListener.Dispose	HttpListener.Abort
if (this.m_State != State.Closed) { this.Stop(); this.m_RequestHandleBound = false; this.m_State = State.Closed; }	if (this.m_RequestQueueHandle != null) { this.m_RequestQueueHandle.Abort(); } this.m_RequestHandleBound = false; this.m_State = State.Closed;

The primary difference is the call to HttpListener.Stop. Here's a code snippet from that method:

            if (this.m_State
!= State.Stopped) { this.RemoveAll(false); this.m_RequestQueueHandle.Close(); this.m_RequestHandleBound
= false; this.m_State
= State.Stopped; this.ClearDigestCache(); }

I'll spare you the hunt and point the problem out. There are two tangible differences between the two calls:

1. HttpListener.Close closes the request queue handle (which is used in calls to the native HTTP API) whereas HttpListener.Abortaborts it. I didn't delve into this but the semantics seem to be the same as for the HttpListener itself.
2. HttpListener.Close calls RemoveAll before disposing of the queue handle, presumably in order to stop accepting incoming requests.

In order to solve the problem, you can either remove the prefixes manually, or call the internal method like so:

listener.GetType().InvokeMember(
"RemoveAll", BindingFlags.NonPublic
| BindingFlags.InvokeMethod | BindingFlags.Instance, null,
listener, newobject[]
{ false } );

It's actually pretty easy; get your web service up and running in axis, run svcutil.exe http://someserver/someurl/someservice?wsdl and you're good to go. Unless, of course, you're trying to return arrays from the Java side (primitive arrays at least, I didn't see much of a point testing with custom classes). Consider the following contract:

            public
            interface ServiceInterface
{  int[] doSomething( String someParameter );
}

The generated WSDL looks something like this (edited for clarity... I hope):

            <?xml version="1.0" encoding="UTF-8"?><wsdl:definitions
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/">
<wsdl:types><schemaxmlns="http://www.w3.org/2001/XMLSchema"><importnamespace="http://schemas.xmlsoap.org/soap/encoding/"/><complexTypename="ArrayOf_xsd_int"><complexContent><restrictionbase="soapenc:Array"><attributeref="soapenc:arrayType"wsdl:arrayType="xsd:int[]"/></restriction></complexContent></complexType></wsdl:types>
<wsdl:messagename="doSomethingResponse"><wsdl:partname="doSomethingReturn"type="impl:ArrayOf_xsd_int"/></wsdl:message><wsdl:messagename="doSomethingRequest"><wsdl:partname="someParameter"type="soapenc:string"/></wsdl:message>
<wsdl:portTypename="ServiceInterfaceService"><wsdl:operationname="doSomething"parameterOrder="someParameter"><wsdl:inputmessage="impl:doSomethingRequest"name="doSomethingRequest"/><wsdl:outputmessage="impl:doSomethingResponse"name="doSomethingResponse"/></wsdl:operation></wsdl:portType>

             <!--
More uninteresting stuff -->
            
              

              

            
            
              

              

</
            wsdl:definitions>

The WDSL is perfectly fine and is properly processed by Microsoft's svcutil.exe tool; the generated classes look and appear to function normally, but if you'll look at the deserialized array on the client (.NET/WCF) side you'll find that the array has been deserialized incorrectly, and all values in the array are 0. You'll have to manually look at the SOAP response returned by Axis to figure out what's wrong; here's a sample response (again, edited for clarity):

            <?xml version="1.0" encoding="UTF-8"?><soapenv:Envelopexmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/><soapenv:Body><doSomethingResponse><doSomethingReturn><doSomethingReturnhref="#id0"/><doSomethingReturnhref="#id1"/><doSomethingReturnhref="#id2"/><doSomethingReturnhref="#id3"/><doSomethingReturnhref="#id4"/></doSomethingReturn></doSomethingResponse><multiRefid="id4">5</multiRef><multiRefid="id3">4</multiRef><multiRefid="id2">3</multiRef><multiRefid="id1">2</multiRef><multiRefid="id0">1</multiRef></soapenv:Body></soapenv:Envelope>

You'll notice that Axis does not generate values directly in the returned element, but instead references external elements for values. This might make sense when there are many references to relatively few discrete values, but whatever the case this is not properly handled by the WCF basicHttpBinding provider (and reportedly by gSOAP and classic .NET web references as well).

It took me a while to find a solution (after which I stumbled onto this post, which wasn't trivial to find): edit your Axis deployment's server-config.wsdd file and find the following parameter:

            <
            parameter
            name="sendMultiRefs"value="true"/>

Change it to false, then redeploy via the command line, which looks (under Windows) something like this:

java -cp %AXISCLASSPATH% org.apache.axis.client.AdminClient server-config.wsdl

The web service's response should now be deserializable by your .NET client.

If you have a WCF service that returns a stream to the caller, you should be extra-careful with the source of these streams. In my case I was redirecting a file stream over a MessageContract that looks something like this:

    [MessageContract] publicclassItemInfo  {

...

        [MessageBodyMember] publicStream ItemStream;
}

It took one of the integration tests to fail... oddly to figure out that the stream wasn't getting disposed of after the operation was completed. The solution appears to be fairly simple, add the following lines to your service wrapper (if your concrete implementation is WCF-aware you can always just stick this straight in the implementation code):

          // Make sure stream gets
disposed at the end of the operation 
          OperationContext.Current.OperationCompleted
+= delegate { item.ItemStream.Dispose();
};

Installing Microsoft Team Foundation Server is a ridiculously arduous and difficult process. I'll spare you my own complaints and simply list the checklist for installing this beast. This assumes you're installing TFS in a domain-enabled environment and in single-server mode; this is the typical configuration for a small-to-medium-size organization:

1. Designate a machine to host your Team Foundation Server repository. This machine must not double as a domain controller as this configuration is not supported by TFS.
2. Set up at two regular user accounts (not administrators, and if you have any group policies you may - according to your configuration - want to keep these users out of the relevant OUs) in your Active Directory. I used the trivial TFSService and TFSReports accounts. Also you'll need a user with administrative privileges on the target server; I personally prefer to avoid the associated headaches, so I simply used a domain administrator user for installation purposes (but used the aforementioned two users to set the beast up).
3. If necessary, install Windows 2003 Server (whatever flavor) on the machine; don't forget the necessary service packs and updates. If your pipe is fat enough, just let Windows Update do its magic.
4. Add an Application Server role, make sure you enable ASP.NET 2.0 during the installation process
5. Install SQL Server 2005. Make sure you read the installation guide first though, as you'll need to set it up to "Use the built-in System account," enable all services except Notification and finally select Windows authentication as the preferred authentication mechanism. You'll also need to let the SQL Server installer install a bunch of prerequisites before actual installation begins.
6. Install SQL Server 2005 Service Pack 2.
7. From the TFS installation media, install hot-fix 913393 for .NET Framework.
8. Install Windows SharePoint Services 2.0 with Service Pack 2.0. Make sure you select server farm mode when installing, or you'll just have to redo the installation.
9. Install Team Foundation Server itself.
10. Back up the reporting services encryption key (you can find a description of the procedure here).
11. Install hot-fix 919156, a.k.a the Quiescence GDR (no, I have no idea what GDR stands for).
12. Install Team Foundation Server Service Pack 1.
13. Make sure TCP port 8090 is open in your firewall software if you want web access to your Team Foundation Server (to be honest, I haven't found any use for it yet.)
14. Install Team Explorer from the installation media (required for many add-ons, including eScrum).
15. Install Visual Studio 2005 Team Suite Service Pack 1. This can, and will, take forever.

If at this point you're not thoroughly exhausted, you might want to set yourself up a with a project. We're currently evaluating the Microsoft eScrum template for our purposes; my colleague Oren Ellenbogen, in his capacity as Scrum Master, will probably be posting his thoughts on eScrum as a platform. In the meantime here's a quick list of solutions to problems we've encountered while configuring the beast:

1. Make sure you install the various prerequisites; in this case, .NET Framework 2.0, IIS, TFS and Team Explorer, AJAX Extensions 1.0 and the Anti-Cross Site Scripting Library
2. At this point you're liable to get a strange SharePoint-related error if you try and create an eScrum-based project; if that's the case (or as a preemptive measure), just run iisreset on the TFS server.
3. If you can't seem to access the eScrum website (nominally at http://yourserver/eScrum) you may have to reconfigure the eScrumAppPool identity from the IIS manager (right click the application pool, chose Properties, go to the Identity tab and enter the right information under Configurable)
4. You may also get 404 errors from the eScrum website even though it's very obviously configured. We've found that the solution described here works as well:
   • From the command prompt, type cd "%ProgramFiles%\Common Files\Microsoft Shared\Web Server Extensions\60\BIN"
   • Run STSADM.EXE -o addpath -url http://localhost/eScrum -type exclusion
   • Run iisreset again
5. eScrum reports only update once every 1 hour. If this bothers you, follow the instructions here to reduce the lag.

Hope this saves someone out there a lot of time and headache (and if so, a comment or e-mail is always appreciated...)

Imagine that you work mostly with C#. Imagine that on occasion you sprinkle SQL, Java and JavaScript into your daily routine, and that you also use the command line a lot.

Can you see the problem?

Didn't think so. I'll clue you in: the last one in that list has one distinct difference with the first four. Look around you. Have you found what we're looking for?

That is correct. The answer is: the semicolon. A lightweight, generally meaningless way to end a statement in most common languages. Arguably useless to users, arguably beneficial to parsers, and the cause of almost half an hour of continuous head-scratching by both programmer and DBA. How so?

C:\Temp>mysql -C -usomeuser -phispassword thedatabase;
ERROR 1044 (42000): Access denied for user 'someuser'@'%' to database 'thedatabase;'

I'll let you draw your own conclusions.

I've been working with Rhino Mocks for a couple of days now and I'm duly impressed. Aside from being generally useful, the Rhino Mocks syntax is mostly natural and some very clever design hacks are used to make it work the way you'd expect it to. (In case you don't know what mocking frameworks are, the subject has been tackledad nauseam. Just take your pick.)

There is one caveat that is not immediately visible in the Record-Playback syntax, though:

            internal
            interface IFoo
{ int Bar(); }

[Test]
[ExpectedException( typeof(
InvalidOperationException ), "Type 'System.Object' doesn't
match the return type " +
"'System.Int32'
for method 'IFoo.Bar();'" )] publicvoid TestRecordSyntax_ReturnTypeMismatchBehavior()
{

MockRepository mocks = new MockRepository(); IFoo
mock = mocks.CreateMock<IFoo>(); using (
mocks.Record() ) Expect.Call( mock.Bar() ).Return( newobject()
); }

This simple test merely returns an incorrectly typed return value for the IFoo.Bar method, so with the ExpectedException in place you'd expect it to succeed, but you're in for a surprise:

[failure] RecordPlaybackTests.TestSetup.ConsistentReturnTypeMismatchBehavior
TestCase 'RecordPlaybackTests.TestSetup.ConsistentReturnTypeMismatchBehavior'
failed: Exception of type 'MbUnit.Core.Exceptions.ExceptionExpectedMessageMismatchException' was thrown. Expected exception message: "Type 'System.Object' doesn't match the return type 'System.Int32' for method 'IFoo.Bar();'", got "Previous method 'IFoo.Bar();' require a return value or an exception to throw.".

To understand this, consider the following flow:

1. A method call expectation is set (by calling mock.Bar())
2. Attempting to set a wrong return type results in the expected exception being thrown from Rhino Mocks. This happens before the Return constraint is set
3. Because using calls IDisposable.Dispose in the finally block, the mocks.Record object calls mocks.ReplayAll()
4. ReplayAll asserts because a required expectation (return value) is not yet set, masking the expected exception!

This is in fact how I first encountered this behavior, however there are several scenarios where this can happen. One example: setting up a return value that throws an exception on construction, possibly due to invalid arguments, causes the same behavior thereby masking the true cause of the error.

An alternative syntax suggested by Ayende (author of Rhino Mocks) is:

    With.Mocks( delegate { Expect.Call( mock.Bar()
).Return( newobject()
); } ); 

Aside from the esthetical aspect (i.e. butt-ugly), this syntax introduces a minor issue: you cannot Edit-and-Continue methods with anonymous delegates. This may not be a big deal for test code but it's worth noting.

All in all I don't consider this a big issue, but it can bite and should therefore be understood and documented. This post will be reflected in the Rhino Mocks wiki shortly and will hopefully save someone a little grief in the future...

Quick download link (v1.2)

I wanted to simple way to download an entire PicasaWeb album. Right-click and save simply will not do, installing the Picasa application is completely out of the question (I use FastStone, in case you were wondering) and I couldn't find any easy way of doing this, so I whipped up a quick and dirty application. It's pretty self-explanatory, really:

Download the application (source included) here.

I also used the chance to give #Develop a serious whirl, and I can honestly say that it's damn impressive; in several hours of use the only real qualms I've had with it are the non-configurable keyboard bindings (at least I couldn't find any configuration menu for this; it wouldn't be an issue if the default bindings weren't slightly different from what I'm used to) and the lack of immediate/watch debug windows. The dialog editor is full-featured and the environment seems to be quite responsive and robust. I really am impressed.

Update (20-Aug-2007) : Version 1.1 is now downloadable. I've added support for AuthKeys (as per skolima's request) so you should now be able to download private albums as well (assuming you have the appropriate key). Also added a compiled binary to the archive, doh!

Update (11-Sep-2008) : Version 1.2 is available for download, and includes the following improvements:

• The tool now works for PicasaWeb albums from any valid top-level domain (this means that albums hosted at www.picasaweb.ru will be properly handled.
• Existing files will not be overwritten; instead I've taken a cue from common browsers and am now adding a counter suffix to the filename. If an album has two pictures with the name example.jpg, you can now expect to find two files named example.jpg and example (2).jpg in your directory. This behavior is also applied if you already have such a file in your directory prior to running the tool - any feedback as to whether or not this is desirable will be appreciated!
• Authentication keys may now include dashes (thanks, Jakob).

My brother was going on and on about a coding challenge he found as part of the "Lisp as an Alternative to Java" study (the challenge instructions are outlined here, and you can find the sample dictionary and input data by following the study link); eventually I agreed to give it a try.

Spoiler alert: Do not read on if you want to take on this challenge!

Although I had to stop in the middle because I had company, the initial (hacked-together, very inefficient and completely uncommented C# 2.0 code) version took me 1 hour and 20 minutes. While technically correct this version performed abysmally, but it served as a good basis for further optimizations; a second, optimized version took another 30 minutes and with another 20 or so minutes to clean it up, the final version took a grand total of 2 hours and 20 minutes. Total length is 193 lines of code (including comments and white-space), which you can find here.

The crux of the solution is a very simple tree structure to hold the word dictionary. Each node has a 10-entry branch table, each entry representing a phone digit; a node can have an arbitrary number of leaves, each leaf representing a dictionary word. This allows for a very simple (and hopefully efficient) inner search loop:

• Use successive digits as indices for traversal into the tree;
• When a node with leaves is encountered, the leaves (dictionary words) are added to the potential result set. The rest of the input is then recursively encoded;
• If no leaves are encountered for the entire input set, a digit is added to the encoded string and the rest of the input is again encoded. A boolean flag is passed along with the recursion to make sure two consecutive digits are never encoded.

This is basically a prefix tree (or trie). With the provided dictionary and input sets, runtime on my 1.7GHz Pentium M-equipped laptop is approximately three seconds. I haven't properly profiled the application for CPU and memory utilization because, frankly, it's good enough. There's also a lot of leeway for optimizations: the current tree implementation is woefully inefficient, as tree traversal can result in additional memory pressure.

With my project finally nearing completion, it's nigh time for Microsoft to release yet another update to the .NET Compact Framework 2.0. Service Pack 2 ought to bring it to about "beta 2" level.

Check out these gems:

By far one of the most annoying aspects of the .NET Compact Framework is how heavily it relies on P/Invoke to fill in the gaps. The framework itself is missing huge pieces of functionality, and with the lack of C++/CLI for the Compact Framework a developer is often time left with no choice but to hack around the missing functionality with P/Invoke (or COM interop, if you have the patience to muck about with ATL).

The problem is that, on occasion, a P/Invoke call would result in a MissingMethodException (in fact, if you're really unlucky, the type loader will throw the same exception on loading the method actually making the P/Invoke call). Although a lot of the scenarios have been thoroughly ironed out by now and can be resolved through a Google search or some hacking on the developer's part, there is one scario that is esoteric enough that I couldn't find any references to it on the internet: you can get a MissingMethodException when you are out of memory.

We're working on an extremely large (in mobile proportions) and complex project, involving massive amounts of .NET logic combined with a very large, performance-concious and memory-hungry native codebase. We've had to hack around a lot of missing capabilities in the .NET Compact Framework (as well as some bugs and/or shortcomings in other parts of the OS), and one of our native calls would inconsistently throw a MisingMethodException; having reearched the problem for a day or two I was convinced that the problem was an incorrect function prototype for the exported function and added explicit calling convention declarations. This seemed to have resolved and I was content for a couple of days, until the problem resurfaced.

The exception content itself is next-to-useless, and since the same P/Invoke call would work intermittently I was hoping that enabling the loader log might supply some further information. Alas, all the log would provide was the following line:

Failed to find/load [SomeDll.dll] (even in [\Program Files\Somewhere\])

Not good. I then dug up another article on the subject and proceeded to enable the interop log, which provided some additional information:

JIT ERROR FOR PINVOKE METHOD (Managed -> Native):
[pinvokeimpl][preservesig]
int Workarounds::GetSinkWrapper(out IImageSink , IImageEncoder );
int (I4_VAL) GetSinkWrapper(IImageSink& ** (INTF_REF) , IImageEncoder *(INTF_VAL) );

I was originally interested in the bizarre native signature generation (specifically the IImageSink& ** parameter - where'd the reference come from?), but upon reading some valid log files for working methods I was convinced that it's a dead duck. I then set my attention to the JIT message: what has the JIT to do with a native function? I theorized that the JIT is responsible for the native signature generation for native functions and kept working under that assumption. That narrowed the question down to, "what went wrong with the JIT compiler?"

Eventually it occured to me that this might be yet another manifestation of the memory limitation issue (processes under Windows CE 5.0 are limited to a 32MB address space). The P/Invoke call is made fairly late into the application; I added a dummy function to the library which I called on application initialization. This forced the JIT to take care of the native library before anything was sucking up memory, and the issue was resolved.

The moral of the story? If you get MissingMethodExceptions on your P/Invoke calls, fisrt make sure your DLL is actually deployed; then check the DllImport signature (you can find a lot of useful resources on this FAQ). Finally, make sure you're not out of memory.

I was screwing around with XmlSerializer and manual recompilations (you can enable raw code output from the XmlSerializer) and somehow this happened:

.NET Framework reinstall time, I guess...

Footnote: This is fast becoming a series of posts on the woes of using the .NET Compact Framework. In fact, I've added a subcategory for this exact purpose; you can access the category and its RSS feed from the Categories list on the right-hand side of the screen.

Here's another one of those "bang your head repeatedly against the wall and look for a sharp object" issues: closing a stream derived from HttpWebResponse.GetResponseStream takes an abnormally long time. In fact, the (blocking) call to Stream.Close takes approximately the same amount of time as it would to read to the end of the stream. I've only encountered one other reference to this in a newsgroup post from 2004, which (according to Todd, the original author of the post) was never actually answered.

Seeing as I was trying to use HttpWebRequest to create a minimalistic download manager for one of our applications, I had to come up with a solution: one option was to close the stream asynchronously and just let the download run its course, wasting valuable memory, CPU time and bandwidth; another was to never close the stream and make do with a resource leak. Not content with these solutions I decided to dig into the framework code with the ever-useful Reflector. The first hurdle was locating the assemblies; they're not trivial to find, but if you look hard enough you can find actually useful versions in your Visual Studio 2005 installation folder, under SmartDevices\SDK\CompactFramework\2.0\v2.0\Debugger\BCL. These are regular .NET Assembly PEs so it's trivial to load them up in Reflector.

Some digging into the BCL sources proved that HttpWebRequest does, in fact, read until the end of the stream when closing; this is the relevant code excerpt, from HttpWebRequest.doClose:

            if (!this.m_doneReading)

{
byte[] buffer1 = newbyte[0x100];
while (this.ReadInternal(buffer1,
0, buffer1.Length, true) != 0)

{

}

}

It only started to make sense when I did some reading on HTTP (a protocol I'm not deeply familiar with). Apparently, HTTP 1.1 connections are persistent by default, which means the connection is maintained even after the request is completed, and further requests are served from the same connection. Technically, this means that the KeepAlive property of HttpWebRequest is true by default, and a Connection="Keep-alive" header is added to the HTTP request. I can only surmise that, with persistent connections, the response must be read in full in order to allow future requests (if the response was cut off, some concurrency issues may apply). Unfortunately, setting the KeepAlive property to false did not resolve the issue and the connection was maintained until closed.

Since I could find no way to resolve the problem, I decided to hack around it:

stream.GetType().BaseType

.GetField( "m_doneReading", BindingFlags.Instance
| BindingFlags.NonPublic )

.SetValue( stream, true );

Screwing around with BCL internals is obviously NOT a recommended practise and it's something I tried very hard to avoid, however this problem had to be resolved in one way or another. I traced the code a bit deeper to see if this would actually have the expected results; apparently HttpWebRequest keeps an internal "reference counter" (actually a bitfield, but same principle) on the connection. According to the implementation for HttpWebRequest.connectionRemoveRef, if the reference counter reaches 0 the private m_connection field of the request becomes null, and the connection is released. I started out by testing this:

Before and after - you can see that the m_connection field becomes null

Unfortunately I have no idea how to test whether ot not the connection is actually disposed (conceptually, if KeepAlive is set to false then the connection should be immediately closed); when I find the time I'll try and track the HTTP session using Wireshark, but currently this horrible hack will just have to do.

I've been hacking away quite a bit at an application that contains large managed and unmanaged portions, and a necessarily complex interop layer in between. It appears that interop marshalling behaves somewhat differently between the full and Compact framework. Here's a somewhat laconic list of the issues I've encountered and how to resolve them:

1. You may encounter NotSupportedException on calls to Marshall.SizeOf. Although the documentation does not specify this as a possible exception, experience shows that this is a result of wrong marshalling attributes: for example, although trying to marshal a bool to UnmanagedType.I4 makes sense from a C programmer's perspective, it results in the behavior described above. The article describing this is called "Using the MarshalAsAttribute Attribute," but the version in my locally installed copy of MSDN Library (August 2006) does not contain this information.
2. Annoyingly, the default marshalling, or even an explicit UnmanagedType.Bool, results in corrupt values (probably some minor framework bug). I worked around this by defining the member as int and manually giving it the value 0 or -1.
3. It's not obvious, but you can't use UnmanagedType.LPArray from within structures - it only works on P/Invoke method declaration parameters. The only way to do this is to manually call Marshal.StructureToPtr and do some pointer arithmetic with IntPtr (annoying, but at least it's safe code).
4. The marshaller always frees up memory; although this makes a lot of sense from the .NET perspective, it probably means that you'll have to code in some sort of deep copying mechanism in your native code if you want any of that information in your state. This also has performance repercussions you should consider when architecting your interop layer.

I'll post updates to this as I come across more issues.

I was writing some code with Visual Studio 2005, and pressed Alt+Keypad 8 (under ReSharper, this moves the current method up/down). Not only did this produce the ASCII code 8 - backspace (control character) or inverse bullet (printable character) - but it also completely screwed up the font on the same line:

For some reason the Visual Studio 2005 debugger (on my native Smart Device project) flat out refused to halt on Data Abort errors. Instead it would just show the error information on the debug trace and crash the application: 

Data Abort: Thread=81e27040 Proc=804c68c0 'Client.exe'
AKY=00000041 PC=00eb1cc0(emnative.dll+0x00001cc0) RA=00eb6d6c(emnative.dll+0x00006d6c) BVA=0e537510 FSR=00000007

For obvious reasons, I wasn't particularly happy with this. Digging into Google and the documentation didn't help; after some serious headscratching I figured that the Windows CE kernel must be catching these exceptions at some point, so from the Debug->Exceptions... menu I enabled, under Win32 Exceptions, catching of thrown Access Violations:

After this, I managed to finally get proper trapping in the debugger, and easily find out what code actually caused the access violation:

It really yanks my chain when I put my faith into what is presumably a solid foundation for my code, and end up running into a huge number of unexpected pitfalls.

Be warned: .NET Compact Framework is incomplete. Oh yes, it's a fleshed out version of CF 1.0 with generics and various important bits and pieces finally included (COM interop. I mean, seriously, .NET is useless without it even on the desktop), but it's still lacking a lot of vital components (ActiveX hosting) and has major shortcomings in others (no XSLT support). But that isn't the worst of it: the documentation is sparse at best, and flat out wrong in some cases.

You cannot use asynchronous delegates in .NET CF 2.0.

In case it isn't clear to you, let me repeat it: you can't use Delegate.BeginInvoke. If you're really unlucky, like me, you'll write a bunch of code and go on to compile, run, test and debug it and then run into a completely bogusNotSupportedException on a seemingly innocuous internal method call, and have to do some serious digging to figure out the culprit. This goes particularly well with (admittedly documented) fineprint in methods such as Windows Media Player's player.URL property setter, which for some reason mustn't be called from an event handler.

Update (January 16th, 2007): Not only does this apparently only work on the emulator, you would do well to stay away from Managed DirectX in general because common drivers, such as those on the Willcom W-Zero3 - do not support rendering in landscape mode (I wonder which devices DO support those features). After messing with this collectively for weeks we eventually went with the obsolete GAPI. The code here will probably not work for you.

For some reason, elementary DirectX operations are not very well documented in the .NET Compact Framework documentation; I kept running into InvalidCallExceptions for no aparent reason, and couldn't figure out the "simple" way to lock surfaces from the documentation (it's worth noting that I was only interested in basic 2D functionality).

Basically you just need the right set of flags and the right order of operations. Here's the code I used and that worked for me (even under the considerably buggy emulator). To create the device, use the following parameters:

PresentParameters p = new PresentParameters();
p.SwapEffect = SwapEffect.Discard; p.Windowed = true;
p.EnableAutoDepthStencil = false; p.PresentFlag
|= PresentFlag.LockableBackBuffer; p.MultiSample = MultiSampleType.None; m_device
= new Device( 0, DeviceType.Default, this,
CreateFlags.None, p ); 

To draw on the backbuffer, you can use the following code:

m_device.BeginScene();

using (
Surface s = m_device.GetBackBuffer( 0, BackBufferType.Mono ) ) { int pitch; using (
GraphicsStream gs = s.LockRectangle( this.ClientRectangle,
LockFlags.None, out pitch ) ) { //
Your code goes here... } s.UnlockRectangle(); } m_device.EndScene(); m_device.Present();

I hope this helps someone avoid a couple hours of frustration.

... from Alex Feinman. Don't ask me why.

In other news, the .NET Compact Framework doesn't decode alpha channels even in bitmaps that has them (well, it might decode the alpha channel, but it doesn't survive a Bitmap.LockBits call - maybe because there's no ImageFormat with alpha...) Still working around that.

You might want to take a look at this post from Omar Shahine. His explanation (excuse? call it whatever you want) for why Windows Mobile will probably never be as good as Blackberry in some respects is honest to the point of being uncanny.

I've often remarked how much I appreciate Raymond Chen's ramblings about the decisions behind some of the more peculiar aspects of Windows. However, where Raymond is uncompromising and unapollogetic (to the point where readers sometimes comment that he is a "douchebag"), Omar is brutally honest but not grounded to the point of being cynical.

I reckon that top management cannot usually afford this sort of uncompromising honesty, but given what is often seen in this industry such an attitude is refreshing even coming from lower rings in the leadership ladder.

To make a long story short, I'm building a trivial query engine over a dataset using XPath; for expressiveness purposes, I've allowed the users of said engine to query for attributes, and assumed that I can simply go up the hierarchy from there and access the elements directly.

I was somewhat dumbfounded to find that each node in the resulting node set had its parentNode property set to null; at first I was sure this has to do with an implementation detail of MSXML3 (perhaps it returns copies of the attributes for... I have no idea what possible gain could be derived from this.) A little more digging proved that, and I quote from MSDN:

In C/C++, IXMLDOMAttribute inherits IXMLDOMNode but are not actually child nodes of the element and are not considered part of the document tree. Attributes are considered members of their associated elements rather than independent and separate. Thus IXMLDOMAttributeparentNode, previousSibling, and nextSibling members have the value Null.

Working under the assumption that this is an MSXML3-only problem, I digged a bit into the W3C DOM specification only to find that it is, in fact, a specification issue:

Attr objects inherit the Node interface, but since they are not actually child nodes of the element they describe, the DOM does not consider them part of the document tree. Thus, the Node attributes parentNode, previousSibling, and nextSibling have a null value for Attr objects. The DOM takes the view that attributes are properties of elements rather than having a separate identity from the elements they are associated with; this should make it more efficient to implement such features as default attributes associated with all elements of a given type.

This seems to me a completely arbitrary design decision. It makes absolutely no sense to inherit from what at first glance is an interface for first-class citizens in the document (Node) and then castrate the interface with useless implementations. If, contractually speaking, attributes never have siblings or parents, then they shouldn't have those properties at all, otherwise this just frustrates the developers wishing to make use of those properties, and wastes their time by forcing them to dig through documentation to figure out what's wrong.

And on top of it all, the decision doesn't even make sense in light of the "efficiency" claim - how is it more efficient to partially implement an interface, instead of defining a new one that's more contractually sound?

Quick link: To download the wrapper classes click here

While working on a large application targetting the .NET Compact Framework 2.0 I realized that I'll need to feed some native code (specifically, the XSLT processor in MSXML 3.0 SP1) with an IStream implementation.

Articles about interoperating with unmanaged code in the CF are not exactly abundant; to save you the time I spent on incorrect and/or conflicting research, here's the bottom line:

• There is no Managed C/C++ for .NET Compact Framework 2.0. To some this may be old news, but you should really pay attention to this point if you're going to do any serious development against CF. 2.0 adds support for managed COM/ActiveX interop, but otherwise you're completely stuck with P/Invoke.
• Although it's not immediately obvious, CF 2.0 does support exposing managed classes via COM.
• The CF is missing some usually-minor classes from the BCL; in this case I was missing System.Runtime.InteropServices.ComTypes.IStream. Annoying but easy to work around.
• Finally, as an aside, MSXML 3.0 SP1 for Windows Mobile does not support the IXslTemplate and IXslProcessor interfaces, meaning that MSXML 3.0's already lackluster performance in XSLT transformations is further hindered by not being able to cache the interpreted stylesheets. This means that, if you use XSLT, your application will not scale. I was not initially aware of this issue, so I hope our data sets are small enough to handle this, or I may yet come to regret the decision to use XSLT in this project.

I managed to save quite a bit of time by leveraging Oliver Sturm's work, which was originally intended for the desktop. Since the CF is missing a whole bunch of minor classes, the managed definition of IStream included, I originally mucked about with midl trying to generate these definitions from the .IDL files. After this proved to be a genuine chore, I just ripped the definitions straight out of the .NET Framework 2.0 assemblies with the ever-useful Reflector.

You can download the class file here. If you use this it would really be cool if you could drop me an e-mail, and I bet Oliver would be equally appreciative. Enjoy!

As I mentioned before, I've had well over 300 trackback/pingback spam notifications from dasBlog. Since this was well beyond what I was willing to mess with by hand, I whipped up a quick Outlook macro to do the work for me:

          Sub DeleteTrackback() Dim oSel
As Outlook.Selection Dim oItem As Outlook.MailItem Dim oShell Set oShell
= CreateObject("Shell.Application") Set oSel
= Application.ActiveExplorer.Selection For x
= 1 To oSel.Count Set oItem = oSel.Item(x) If (Left(oItem.Subject,
19) = "Weblog trackback by") Or _
(Left(oItem.Subject, 18) = "Weblog
pingback by") Then Index = InStr(1,
oItem.Body, "Delete Trackback:") If (Index
<> 0) Then URL = Mid(oItem.Body,
Index + 18) URL = Left(URL, Len(URL)
- 1) oShell.ShellExecute URL, "", "",
"open", 1 EndIfEndIfNextEndSub

To use this macro, create a new macro and paste the source code; then select all the "Trackback/Pingback" notifications messages and run the macro. It could obviously be customized to work on entire folders or whatever, but that I leave to you. One final suggestion: if you (like me) keep a 15-tab Firefox window open at all times, you may want to open a new window (not tab, window!) so that you can then close all the URLs at once.

The previous post in this series was an overview of the various communication hurdles developers and managers face. Here I will describe one of the methods we use at Monfort to tackle these issues.

We've been working very closely with one of our clients on a large variety of projects, most of which are based off of a shared technological foundation (I suppose you could call it a "framework," although it technically isn't). These projects generally fall into one of three categories:

1. Improvements to our technological infrastructure. These are either extremely large projects (measured in man-years) or fairly small projects (usually several weeks);
2. Product derivatives, demonstrations and platform ports that make use of the pre-existing technological infrastructure;
3. New products that make little or no use of the pre-existing infrastructure.

Over the years the volume of work we do for this client has increased considerably, creating the need for additional personnel in order to meet the demand. The caveat is, obviously, increased managerial overhead. This can take one of two forms:

1. Project management, in which a manager is directly related to all aspects of a particular project. As soon as an order is placed for the project, the manager is responsible to bring the project to fruition. More projects equate more project management;
2. Customer relations, in which a manager is responsible for maintaining contact with the client and providing the initial technical contact point. The person in charge has to have a very deep technical grip of the relevant technologies and be able to communicate effectively with both business and engineering personnel (the client's representatives as well as their customers). Finally, he has to maintain constant vigilance so that new projects will actually move beyond the initial "what if" stages.

Although good project managers are hard to find, the real problems became evident only when we added personnel to the customer relations position, which up until that point was (for the most part) filled by just one person. The first hurdle was in bringing additional people up to speed; the relevant knowledge was kept in e-mail archives and the heads of several people, which adds up to very inefficient indexing. A newly commissioned customer relations manager has to know a lot of non-trivial details: the representatives of the various clients he'll be working with, the work methodology against a variety of clients and similar information. The issue of knowledge sharing becomes a much bigger issue when the manager moves beyond that point and goes out to the field: he has to be kept up-to-date on all concurrent projects for those customers, prospects for future projects and the various proposals and discussions that had taken place, whether he was involved or not. When two or three different people all share in those responsibilities, they have to be constantly synchronized

This is a lot of information, and it became very obvious very quickly that we needed some system to manage and store it. After spending a lot of time thinking about this, I arrived at a fairly coherent list of requirements from the knowledge sharing platform we will employ:

• Any one of the customer relations managers should be able to freely access and update information;
• There should be no artificial limitations to the way information is written, presented or interlinked;
• The platform should be easy, if not trivial, to learn and use;
• The platform should be web-based and accessible from everywhere (we often require access to information for client sites).

Eventually, I came to the conclusion that the only knowledge/content management system that could actually work in our corporate environment is a wiki.

If you've been living under a rock for the past few years it's possible that you've managed not to hear about Wikipedia; the collaborative encyclopedia is one of the most ambitious and impressive projects to have attained a degree of success on the Internet. What distinguishes Wikipedia from similar, less successful projects is that the knowledge stored in Wikipedia is completely freeform: anyone can edit it, there are no predetermined schemas and no mandatory information fields - anyone can enter data in the way they deem fit. Here-in lies both the beauty and danger of a wiki: it doesn't brute-force the information into patterns. This means that knowledge can be shared in any conceivable way and content editors are free to interlink this knowledge in any way they deem appropriate. The danger here is that most people aren't disciplined enough to invent their own patterns. In our case this did not pose a problem, since only a small number of people were expected to be involved in the effort.

In the next post: installing, configuring and learning to effectively use the wiki.

(This is the first in a series of posts, in which I will try to articulate the various problems of communication in the software field and possible ideas on how to solve those problems.)

One of the most interesting aspects of being a software developer is communication. Perhaps the single most important trait of a good software developer is the ability to communicate with varying audiences:

• First and foremost, a software developer must be able to communicate effectively with the other developers on his team. The usual tools apply: e-mails, whiteboard, hall meetings, source code comments. For a developer this is relatively easy, since it is unnecessary to translate the technical abstractions into a coherent conceptual model: the abstractions themselves are familiar to all sides of the conversation.
• A software developer must also be able to communicate effectively with his team leader, project manager, boss - whatever you want to call it. This isn't quite as easy, as such communication can have serious repercussions on the project, the work environment and - potentially - the developer's career. Managers are usually not privy to the ultra-technical subculture of software developers, and although they almost always come from the same background they have just as often given up on subtle technicalities.
• Finally, a software developer must be able to communicate effectively with clients. At some point in almost every developer's career he is required to meet and at least converse with a client, sometimes in order to figure out a bug, sometimes in order to gather requirements for a new project. A lot of developers would rather avoid these encounters because they are forced to re-think everything they say and translate abstractions. A client is not, generally, heavily interested in technical matters, so these must be explained carefully and coherently; most developers I know find this process frustrating and draining.

A developer that delves into management is assumed to already be proficient in the above scenarios, but management brings a whole new slew of communication problems to the table:

• A manager is required to communicate with clients on a whole different level. When a developer is asked to make an estimate, the estimate forms a basis for the cost and schedule estimates his manager provides to the clients. The difference is that the manager is required to explain these estimates and is personally accountable to same. It is the manager's job to continuously communicate with the client, reassure, explain and assume responsibility for whatever pitfalls and hurdles the project encounters, and generally make sure the project is on track. A manager is also required to act as a filter between the client and the developers on the team for everyone's sake: on the one hand the manager is required to "protect" the developers and the project from ridiculous requests and whims on the client's side, and on the other hand he is required to make sure the client's requirements are fulfilled to his satisfaction.
• A manager must communicate effectively with the developers on his team. He must establish trust and confidence, so that he can constantly be aware of what goes on with his team while avoiding micromanagement and wasteful status meetings. This isn't easy because the manager and the developer are fundamentally on opposite sides: efficiency vs "doing the right thing", choosing wisely instead of following trends, cutting corners instead of continuous refactoring and improvement. The only glue they have is the desire for a successful project. A good manager must communicate his decisions to the developers on his team to constantly maintain the trust and confidence he has built; a frustrated or uptight developer is an inefficient developer.
• Finally, a manager must be able to communicate effectively with other managers. This is the least trivial form of communication for a variety of reasons: sometimes the corporate atmosphere requires managers to be ultra-competitive. It is hard working for a shared goal ("the good of the company") when one is constantly competing with one's peers. Even lack of competition can be an issue when two managers from different schools of management must collaborate, or when one manager is frustrated by the other's inefficiency. And even when all managers are equally efficient and equally dedicated to the shared goal, they must still share information between them and make sure the collective company process is on track.

You'll notice that the two lists are in opposite order. This is no coincidence; I ordered the lists so that the more difficult of the tasks are further down the list, so for developers the most difficult form of communication is usually with customers, whereas for managers it's communication with other managers.

The next post will deal with knowledge management and how we've used a wiki internally in our organization to alleviate the overhead of communication between managers.

This wasn't an easy riddle by any means. One has to either have a lot of experience developing both C and C++, or take a very hard look at the standards to spot the differences. Bottom line? I couldn't find a solution. Fortunately a brother and a colleague both came through:

Solution 1: Ugly Hack

Tal (one of my colleagues at Monfort) came up with the following concept:

#include <stdio.h>

int main()
{ int a = 1//**/10 ; printf(
"%d\n", a ); return 0;
}

This deserves some explanation: originally C did not support single-line comments (//); these were added to a later version of the standard called C99. So what happens is, if you compile this code with an old C compiler the "//**/" string is parsed as a division operator and an empty comment, resulting in 1/10=0. A modern C compiler (or any C++ compiler) will treat the entire line as a comment and the result is 1.  One way to easily test this is to compile with gcc -std=c89 file.c.

Solution 2: Elegant Hack

I didn't have the time to properly look for a detailed comparison of C and C++ standards (I'll save you the time: this looks to be a good source). There are probably quite a few ways to achieve the goal, however my brother Mickey suggested one I find quite elegant:

#include <stdio.h>

int main()
{ printf( "%d\n", sizeof(
'c' ) == sizeof( char )
); return 0; }

Seems odd? Apparently one of the (relatively small) breaking changes between C and C++ is how character literals are handled. In C a character literal takes the type int, whereas in C++ a character literal has the expected size (source).

Write a program that compiles under both C and C++ compilers, and outputs "0" for C and "1" for C++ (or whatever, the actual numbers are not the point). You obviously may not use compiler-dependant macros, __CPLUSPLUS__-type macros or anything of the sort.

Once again, I'll post the solution in a couple of days.

Here's one possible solution for the riddle I posted a few days ago:

1. Replace i < N with -i < N
2. Replace i-- with N--
3. Replace i < N with i + N

This isn't a particularly hard riddle - it took me about 10 minutes to come up with the solution, although I imagine a developer with more recent C experience will be much quicker. Regardless it's a pretty good way to tone those C muscles.