This is the second post of a two-parter. Please read part 1 for context and additional information.

3rd Party Libraries

As mentioned in part 1, Scala is not binary backwards-compatible between major releases. Although code compiled with 2.10 may work properly with libraries compiled against 2.9.x, this is hardly recommended and may result in a lot of subtle error conditions at runtime that are hard to track down. As a result, the recommended practice is to upgrade all Scala dependencies to versions compiled against 2.10. Fortunately the Scala ecosystem is very active, so most common libraries had 2.10 builds on announcement day, but you are still likely to encounter libraries that need to be upgraded because:

• They're lagging behind the Scala release schedule and simply haven't released 2.10-compatible binaries (fortunately, this seems to be extremely rare; in practice all our dependencies were otherwise resolved);
• The specific version you're using does not have a 2.10-compatible release and likely never will (Squeryl 0.9.5-2);
• A 2.10-compatible official release is out but hasn't yet been published to the central repository (Scalatra 2.0.5).

My experience shows that with a bit of dilligence, practically all our (myriad) dependencies were fairly easy to migrate to 2.10. In most cases this simply entails switching to the appropriate artifact ID suffix (_2.10 instead of _2.9.2, see part 1 for associated rant), in other cases I had to upgrade to a later official release and in rare cases I had to dig in and replace a dependency with an appropriate fork or snapshot version. Here is a list of migration notes for some of our dependencies:

Scala Library Changes

The Scala class library itself has a number of changes you ought to be aware of before migrating to 2.10. The really big change is that Scala actors are deprecated in favor of Akka. You can still use them by importing the scala-actors artifact from the Scala 2.10 distribution, but it is recommended to migrate fully to the new actor system as this is also likely to be obsoleted by 2.10.1. The gentle folk at Typesafe have provided a very comprehensive migration guide to assist your efforts.

The less prevasive API changes we ran into include:

• List.elements is deprecated in favor of List.iterator;
• TraversableOnce.toIndexedSeq no longer takes a type argument. This was actually quite pervasive in our codebase, causing plenty of compilation errors, and is easily worked around by removing the type parameter (which is extraneous to begin with);
• Scala actors' Actor.receive method is now public (previously protected). This had to be rectified in pretty much all of our existing actors by removing the protected modifer;
• Occasional subtle API changes requiring minor code fixes. For example, see this question on StackOverflow.

Summary

Opinions differ among members of our team - some predicted that the migration process will be much more complex whereas personally, given the relatively high level of maturity I've come to depend on in the 2.9 series, the migration process actually ended up being significantly harder than I anticipated. Particularly disconcerting were the occasional compiler failures which took a lot of time to track down. Practically speaking, though, the whole migration process took less than 3 days (including documentation), did not involve additional teammates, and all problems were either resolved or worked around rather quickly. The Typesafe compiler team has been very helpful in analyzing and resolving the single bona-fide compiler bug we've run into, and the community as a whole (on Stack Overflow, Google Groups and elsewhere) was also extremely supportive and helpful.

On the whole, am I happy with the process? So-so. There is nothing here an experienced Scala developer should have serious trouble with, but it will take a lot more stability and predictability for Scala to gain mainstream acceptance in the industry, and that includes a much easier and more robust migration path to upcoming releases (yes, that includes migrating from 2.10 to 2.11 when it comes along). That being said, Scala has been a terrific language to work with in the last couple of years, and the new features in 2.10 (particularly reflection, macros and string interpolation) should make this an extremely worthwhile upgrade. We still have a lot of regression testing to do on the new version, and if anything interesting pops up I'll be sure to post about it separately (or bitch about it on Twitter...)

I just spent a few days migrating our codebase from Scala 2.9.2 to Scala 2.10. A quick search found very little in the way of migration guides and studies, so it seemed appropriate to document my experiences and offer what tips I managed to collect on the way.

General Observations

This process should not be undertaken lightly, and its implementation inevitably depends on your resources and team composition. The group (or person) charged with migrating the codebase should, at the very least, consist of an experienced Scala developer not afraid to get his or her hands dirty.

The migration is not an entirely smooth process - there are plenty of (fortunately fairly minor) breaking changes in the Scala APIs, and since Scala does not yet feature binary backwards compatibility between major versions (2.9→2.10→2.11...), so expect some inevitable library upgrades and the potential complexities inherent in any such update. The specific issues I encountered are documented in their respective sections below.

Unfortunately there is not much I can offer in the way of preparation; these are the obvious steps:

• Familiarize yourself with Scala 2.10. Make sure you know what you’re getting in exchange for investing time and effort on the migration process and early-bird issues;
• Work in an isolated branch and pull changes from the master branch often;
• Commit early, commit often. Try to keep your commits small and well-documented, so you have good revert checkpoints for when things go wrong;
• Keep the affected code to a minimum. The fewer changes, the easier it will be to isolate problems down the road.

If you’ve read so far and still can’t wait for 2.10.1, I hope the next few sections will save you some time and pain.

Toolchain

To start things off I switched the build to use Scala 2.10. Our project runs on Maven in lieu of SBT (topic for another post), and we use David Bernard’s excellent scala-maven-plugin, which deduces the Scala version from the scala-library dependency.

The convention for the Scala version suffix has changed between 2.9 and 2.10: for example, argot2.9.2 is now argot2.10 - no revision for the 2.10 series. This rendered the suffix macro we employ useless, because annoyingly the Scala librariesthemselves (e.g. scalap or scala-compiler) actually use the full version number, so we ended up needing two macros (a “raw” Scala version and a suffix).

Fortunately, at least as far as Maven is concerned the rest was smooth sailing. I next turned my attention to IntelliJ IDEA - I wanted to make sure an existing workspace can be reused with the migrated codebase, otherwise my entire team will have to undergo the inconvenience of starting from scratch on a clean working copy. Fortunately the process turned out to be quite painless (on IDEA 12.0.3 with Scala plugin build 0.7.121), with the following caveats:

• Code analysis appears to miss some cases of stricter compilation compared to 2.9.x (see below);
• Code analysis occasionally identifies relative import statements as erroneous (e.g. import util.Random);
• Project FSC settings had to be manually changed to use the 2.10 compiler bundle (it remained on the default 2.9.2). This proved to be a moot point, because:
• The much-touted IDEA external build mode finally works consistently for the first time (and it uses the significantly better SBT compiler)!
• On the negative side, IDEA does not seem to handle compiler failures (as opposed to compilation errors) gracefully, missing a lot of detail in the output. As I ran into quite a few of these (details below), I ended up doing most of the compilation tests with Maven directly.

Beyond setting up a build job for the new branch, Jenkins posed no issues.
To summarize, from a toolchain standpoint, this was actually a fairly smooth process.

Language/Compiler

I was surprised and somewhat disheartened to find that, after some initial compilation attempts, it became evident that I missed every single mark in my code risk predictions. Where I expected most language-level headaches I encountered none, and seemingly simple and risk-free code ended up taking most of the time working on this process.

First off, the Scala 2.10 compiler is much stricter than the earlier 2.9.x series. It has a few new heuristics that ended up triggering very often on our codebase; some are simply the result of messy code or design, but most are new best practices adopted by the compiler team at Typesafe:

• “warning: This catches all Throwables. If this is really intended, use `case _ : Throwable` to clear this warning.”
  We wrap-and-throw or swallow a lot of exceptions, and for brevity’s sake often use the shorter catch { case _ => }. A catch block is analogous to a partial function from Throwable, and there are undoubtedly cases where the distinction matters. We’ll clean these up iteratively by qualifying that we’re only catching Exceptions.
  
  
• “non-variable type argument B[T] in type A[B[T]] is unchecked since it is eliminated by erasure”
  The 2.10 compiler is actually quite helpful in warning you when generic code may not behave as expected. This is just one example of such a warning.
  
  
• “error: overloaded method methodName needs result type”
  The 2.10 compiler requires you to specify result types on most (all?) types of method overloading. Not much work here.
• Case classes can no longer derive from other case classes, which makes a whole lot of sense if you consider the typesystem; 2.9.x had a whole variety of bugs and edge-cases relating to case class inheritance and companion objects, including some unwieldy constructor syntax. 2.10 simply does away with the complexity and potential ambiguity, which will break existing code. Since such class hierarchies are inherently problematic, I see this as an opportunity to refactor them properly; extractor objects were often useful for that task.
  
  
• In one case we ran into stricter cyclic dependency analysis on the part of 2.10 - an object extended a class and passed a member of another object as a constructor parameter. The second object referenced the first, which compiled fine under 2.9.2 but resulted in a cyclic dependency error with 2.10. As the original design was hard to understand I refactored the code to be a bit simpler, which resolved the problem satisfactorily.
  
  
• Manifests are deprecated in favor of TypeTags and the new reflection library built into 2.10 (certainly one if its most anticipated and celebrated features), but are still functional. We haven’t migrated to the new APIs yet, and that process will likely deserve a post all on its own; read this excellent answer on Stack Overflow for an introduction to TypeTags.
  

Beyond these and the occasional minor syntax change, the 2.10 compiler proved a somewhat difficult beast to tame:

• Ran into crazily-detailed compiler failures that clearly (though long-windedly) indicated that I need to manually add scala-reflect to the classpath;
  
  
• Missing or conflicting dependencies, typically due to mishandling POMs, resulted in what I ended up dubbing the Spontaneous Compiler Combustion phenomenon: a cryptic compilation failure, complete with what appears to be an AST dump and full-blown debug information. Tracking the occasional familiar type name in the compiler log can be helpful in tracking down the dependency at fault (this was the case in all but one of the occurrences), but the error itself is completely inscrutable.
  
  
• The one case was, unfortunately, a proper compiler bug logged as SI-7109, which has to do with consuming a package-protected (or -private) trait/class from another trait/class with the same accessibility. Jason Zaugg of Typesafe (@retronym) was extremely helpful in analysing the compiler output and producing a reproduction case, which I haven’t been able to do on my own. Until a fix is included and released, I’ve worked around this by temporarily commenting out the problematic qualifiers.
  
  
• Update (2013-02-12): Ran into another compiler bug that causes a runtime ClassFormatError. We've managed to identify, reproduce and work around the problem; see SI-7120 on the Typesafe JIRA.
  

Lastly, it’s worth noting that the 2.10 compiler is significantly heavier than the older 2.9.x series: it appears to require 50-100% more permgen space, and compile times are up x2 on average and up to x10 on a clean, fresh build. This seems consistent on both my laptop and the build server, which use different CPUs and OS. A quick check (top and jstat -gcutil) showed the compiler process to be single-threaded and CPU-bound, as it consistently utilizes 100% of a single core. GC activity was low to the point of negligible, so it appears the new compiler is actually a step back in terms of compilation throughput. I hope subsequent 2.10.x releases focus primarily on compilation stability and performance.

That’s it for today; next post will be up in a day or twois up and focuses on the Scala library, dependencies and miscellanea.

This post attempts to summarize and clarify a lecture given at the Botzia (Israeli Java user group) meeting on May 3rd, 2012. You can view the presentation in its entirety on SlideShare:

What is Scala?

Scala is a modern, statically-typed language designed to run on the Java platform. It's commonly perceived as an "evolutionary Java" and with good reason: Scala adds powerful features while significantly reducing Java's verbosity. It is a highly practical language in that it offers a very straightforward migration path for developers on existing platforms: Java developers will feel right at home with Scala, and a .NET developers will not have to give up on a lot of the advantages C# has over Java.

For organizations with a preexisting codebase Scala offers an excellent balance between power and feature set, as well as perfect interoperability with existing Java code: although you certainly don't have to, you can freely use familiar libraries without a hitch, call into existing code and, for the most part, use your existing tool-chain with little or no impact.

Why did we pick Scala?

As mentioned above, Scala offers a very compelling migration path from an existing Java codebase. When we were setting up the new R&D center at newBrandAnalytics, we were already serving paying customers via a complex codebase. A complete rewrite was out of the question as we had contractual obligations, and we had to keep improving the existing product in order to maintain the competitive edge; it became very clear that we needed to maintain the existing codebase and refactor it incrementally.

An additional data point was that our core team members were highly experienced with Java development and quite comfortable with the platform; we also had a distinct preference for statically typed languages, which ruled several alternatives (particularly Clojure) right out. Scala seemed like a great compromise in that it allowed us to maintain and extend the existing codebase while enjoying advanced language features and improved veracity at little or no risk to the product itself.

How should I "sell" Scala to my boss?

"Safe" choice with proven track record: Scala is not an entirely new language, and has reached a maturity tipping point at 2.8; over the last two years the language has been gaining significant momentum, and has a number of strong advocates and success stories, including Twitter, Foursquare, LinkedIn and a whole host of others.

Scala as a better Java: You don't have to go all-in on Scala's advanced feature set, but can instead think of Scala as an improved Java with type inference, closures, advanced collection framework, traits and pattern matching; these features alone will increase your developers' happiness by an order of magnitude, and productivity will see a corresponding improvement.

Less code ⇒ less bugs: Scala code is far more concise than corresponding Java code, an helps the developer focus on the "what" instead of the "how". The resulting code is usually much shorter, clearer and simpler, which in turn helps lower bug count.

Helps hire better engineers: Great engineers not only love working with the latest technologies, but also love practical, incremental improvements to well-respected technologies. Merely saying "we use Scala" will help attract the sort of seasoned, reliable and professional engineers that you most want to hire.

Where does Scala put you at risk?

Learning curve: Although Scala largely derives from Java, it's a whole new language with a lot of features that takes time to master. Don't expect your developers to be productive from day one; instead, provide opportunities for experimentation and encourage iterative development (and yes, give your developers time and mandate to throw away badly written early code and redo their work if necessary). There is actually a lot of material on Scala and an active and vibrant community; specifically, your developers can easily find help (and answers) on StackOverflow.

Rough edges: Scala is fairly new, and consequently a lot of things that you take for granted with Java are not as mature or robust with Scala. IDE support is a moving target: Eclipse and IntelliJ IDEA both have actively developed Scala plugins and both have occasional issues (spurious error highlighting, lackluster performance, wonky debuggers). The common build tools support Scala, but don't play as nicely; for example, only the native build tool (sbt) support incremental compilation. Even the compiler itself is not entirely bug-free, although these are getting very rare indeed. Bottom line: expect occasional problems, and be patient while working around them; even better, encourage your engineers to participate in the community, file bugs and even offer patches where possible.

Production characteristics: While it runs on the JVM, Scala is not Java; there are subtle differences that you should be aware of when maintaining large-scale or highly-available software. Scala generates a great deal of synthetic code, which puts additional pressure on the PermGen space; synthetic stack frames may also exhibit significantly increased stack usage. Scala also creates a lot of intermediate objects at runtime, which results in added eden generation churn. You may want to profile your applications and tune the GC accordingly.

What's the bottom line?

Scala is fantastic. Our team at newBrandAnalytics is remarkably happier with Scala despite the occasional hitches, and I doubt any of us will consider going back to Java given the option. As long as you keep in mind that on the bleeding edge you may get cut, I definitely recommend taking the plunge, with contractors and die-hard traditional enterprises the possible exceptions.

New brand what now?

newBrandAnalytics is a newish startup (less than 2 years old) which started out with a simple premise: the internet is a goldmine of customer opinions, to the point where it's impossible to track and respond to it all with traditional techniques and purely human oversight. We mine the net for opinions, reviews, ratings and other data; we analyze this data and break it down into a structured form that can then be aggregated and measured against the customer's own operational metrics.

To illustrate, one example of what we do is take the impossibly huge number of "Just ate at McDonalds in Copenhagen airport, it was horrible! Stale fries, messy bathroom and facetious service!" tweets (alright, maybe I'm giving Twitter users too much credit :-)), turn them into what amounts to a scoreboard (cleanliness: -1, freshness: -1, service: -1), and finally crunch numbers until we can give the customer a dashboard that says, in plain English, "there's something rotten in the kingdom of Denmark." Not only that, you can easily drill down and contact the author of the complaint, pull up historical graphs to see if there's a growing negative trend in service quality for that area etc. And that's just scratching the surface.

The company itself is US-based, and is now just past the incubation stage; there's a steadily growing revenue stream, and now the business is catching up with what was essentially proof-of-concept technology, originally developed by an outsourcing firm (which actually did a surprisingly good job). The company has decided to bring engineering in-house, hired a veteran CTO and is now in the process of setting up its R&D center in Ceasarea, Israel.

What's in it for you?

• Join a small, tightly-knit team of extremely smart, experienced engineers who absolutely love their profession!
• Work with cutting-edge technologies, including Node.js, Scala and Cassandra/Hadoop:
  • We're all in when it comes to DevOps;
  • Take your pick on tools. We don't care if you use Windows, Mac or Linux, as long as you have what you need to be productive. In fact, you get to pick your own machine spec;
  • We're big believers in open-source. Team members often (some regularly) contribute to open-source projects, and they're pervasive in our technology selection;
• Work at a revenue-generating startup with huge promise;
• Competitive compensation package;
• Terrific working conditions: less traffic, noise and air pollution and much more space compared with most locations in the Gush-Dan area

What's in it for us?

We have several positions to fill. Officially, these are: QA automation engineer, frontend engineer, senior versatile backend engineer. In practice, we're just looking for really smart people that can help us get things done faster and better. Proficiency, or talent and inclination to become proficient, in any of the following areas is a definite plus:

• Java and associated technologies (Scala experience will impress us further still!);
• DevOps, specifically: monitoring, deployment automation and build automation;
• NLP, machine learning and text analysis;
• Large-scale data crunching (experience with modern NoSQL/big-data solutions, e.g. Hadoop/Pig, GigaSpaces or Cassandra, is a major plus);
• Testing methodology: we have several major subsystems with separate testing requirements (data acquisition, NLP and UI automation, to name but a few);
• Client-side and client-facing development (MVC frameworks, JavaScript, CSS, HTML);
• HTTP crawlers and associated domains (bot idioms and challenges, scale and cost concerns).

Interested? Send in your resume to tomer@newbrandanalytics.com or contact me!

With the question of what DevOps is all about out of the way, let’s consider the challenge of setting up an effective DevOps team. I’ll try to present this as a series of questions, which in order hopefully comprise a good overview of the process. I certainly can’t cover everything in s single post, so let’s start with the basics:

Do I even need a DevOps team?

Not necessarily. What you do most certainly need, and probably already have, is an application engineer role. The formal makeup of such a team derives from business necessity: one-man ISVs handle all aspects of the application on their own, including the operational aspects, and are therefore not interesting for the purpose of this discussion. But even with garage startups that comprise very few people (as low as three, in practice), someone always ends up taking the DevOps mantle; there’s always the one guy who ends up taking care of the hosting, deployment and all that other business because someone has to. As your organization grows, as your engineering throughput exceeds your operational capacity and the complexity of your product portfolio increases dramatically, that’s when you need to consider an actual team.

How do I split up the work?

There is no absolute truth here; the answer depends on the size, organizational structure and corporate culture of your specific organization. Your requirements may be satisfied by having just one übergeek take care of everything; in other cases (such as multiple long-term projects, very high system complexity or simply too much work for one person) you may need to divide the work up into more specific agendas. There are numerous strategies you can employ, but here’s a few ideas to get you started:

1. Division by domain

This is a fairly trivial model: figure out what work gets done, categorize it into a reasonably small number of domains, and assign them to specific team members according to their respective skillsets, interests and aspirations. To illustrate, consider the following domains: capacity planning, monitoring, high availability, infrastructure (networking, storage) and deployment. Your engineers probably already handle all of the above.

While this model has the benefit of generally placing the most qualified personnel in charge of any particular domain, be careful to have a least a pair of engineers on each domain so you can scale up when necessary, and to ensure knowledge is retained. In case of small teams (3-5 people) this means each person is likely to have more than a single area of responsibility; depending on your team members, the ensuing overhead (or even personality clash) may have a significant impact on the efficiency of your team.

2. Division by feature

Another fairly simple strategy is to have shared ownership of all the fundamental tools and processes your team employs, and scale by assigning specific features to team members. Not everyone on the team has to have the same level of expertise on every domain, but every team member does need to have a good grasp of all aspects of your work. The level of expertise of any specific team member becomes an evaluation criterion when assigning a new feature, project or issue, and has to be weighed alongside other criteria, such as the need to challenge less-experienced team members, pressing schedules and the other myriad details that managers have to handle. Ideally, any feature can be assigned to any team member and executed effectively.

The primary drawback to this model is efficiency: it is irresponsible to expect the same level of productivity for a specific task from each team member, which makes efficiency the direct responsibility of whomever divides up the assignments, usually the team lead or group manager†. Unfortunately, not all managers are apt at identifying bottlenecks and shifting responsibility around to optimize for efficiency; whether this model suits your organization or not depends on your ability to effectively track your team members’ abilities and provide effective guidance.

A possibly bigger challenge here is knowledge sharing. As the team grows, so does the communication overhead and, perhaps more importantly, the opportunity for miscommunication, necessitating meticulous documentation and proactive knowledge management. The ensuing overhead can kill productivity if not carefully managed.

† Small-enough teams may be able to split the work up by consensus, but success depends on the social composition of the team and is hard to predict.

3. Divide and conquer

The problem can also be attacked from a completely different angle. Your team comprises members with different skills, mindsets and even ambitions; sometimes these do not mesh together naturally, possibly due to too much overlap or, in other cases, too little. This can happen for any number of reasons, but let’s assume that this is not a case of simple ego clash or, as Rands put it, a toxic asset; your team members are hard workers, they like each other well enough and want to get things done, but they just can’t agree on the mechanics. It’s time to divide and conquer.

There are two ways to go about doing this: the simplest is to split up domain ownership between the groups. It’s not likely that anyone will be truly satisfied with your choices (if the division of expertise or interest is natural, there probably wouldn’t be any significant friction in the first place), but the grumbles will eventually subside, if only through relief that the constant bickering is over. When the division of responsibilities is clear, people feel there’s less at stake and are more likely to concede a draw in an argument (don’t expect capitulation here, these are engineers after all…). Once the emotional involvement is out of the equation, engineers are quite rational; if they can’t make a convincing argument and they’re not personally vested in the result, the argument ends with an amicable “yeah, alright, it’s up to you.” Unfortunately, turning this to your advantage in the long-term entails a clever assignment of domains, balanced well enough so that both sides rate their level of interest above the “meh” threshold. How to do that is entirely up to you.

The second way is to introduce a new element into the mix. In particular, if the areas of contention typically resolve around choice of tools, it might be wise to assign ownership of tool selection to a third party; for instance, friction may ensue over the choice of monitoring system. You may experience heated arguments about the relative merits of e.g. Nagios and Zabbix, whereas a level-headed analysis would show that a customized system based on Graphite is your best strategy. Or perhaps the debate is on httpd vs nginx; or any strategic technical decision, really. See what I’m getting at here? At a previous employer we ended up defining an Infrastructure Architect role, an operations counterpart to the System Architect role in R&D. These rolls fill very much the same need, except on opposite ends of the spectrum; whereas the SA might pick Hadoop and log4j for data storage and logging respectively, the IA will likely pick Graphite and LogStash to monitor the cluster and aggregate logs, respectively. Interestingly, the necessary skillset is also quite similar. Identifying and mentoring likely candidates is a subject that I’ll likely tackle in a future installment.

To be continued

There’s much more to say; this is an entirely new branch of the industry, with few acknowledged best practices and even fewer well-known truths. The main point here is that you have to experiment in order to build an effective team, and manage it efficiently. I’d love to hear actual feedback on what works and what doesn’t, and likewise I’d be interested in feedback regarding what I should focus on next: which is more interesting, recruiting techniques and pains, or building an effective interface with the other groups in the organization? Or perhaps an entirely different domain, such as knowledge management techniques? Fire away…

Sometimes you know it’s time to move on. I’ve been working at Sears (née Delver) for just under four years, making this the longest stretch in my career. Even so, my tenure at Delver has seen several major upheavals, including a name change (we originally started off as Semingo), product strategy shifts (Delver started out as a people search engine, similar to what pipl.com are doing today), launching our product, failing to survive the market crash and shutting down, being bought outright by Sears and a drastic personal shift from R&D to operations.

So yeah, a lot has happened in the last four years: I’ve worked alongside some amazingly smart people, helped build and take care of systems way more complex than any I had encountered before, and perhaps even learned to curb my temper a little bit. Delver/Sears is a great company to work for, but it’s time for me to move on.

As of two weeks ago I’m a software architect working for newBrandAnalytics, a startup company that provides social business intelligence for various industries; I’ll skip the business spiel as you can go ahead and read about our solutions and existing customers. Suffice to say that we handle massive amounts of data from various sources (including social media such as Twitter and Facebook); scaling concerns aside, this also entails bleeding-edge text analysis and NLP, rapid response to growing (and changing) customer demand and other wonderful aspects of complex, scalable software systems built to support an actual business. Exciting times are ahead!

Coincidentally, we’re hiring! Our careers page isn’t up yet, but we’re looking for top notch engineers, QA engineers and NLP/algorithm specialists for our Israeli R&D center. I’ll update this post with more concrete details later on, but will leave you with the following points to consider in the meantime:

• nBA is a fresh, privately-funded company with a fast-growing customer base;
• We’re a core group of extremely smart, experienced people who love (and know how!) to get things done;
• Tackle challenges such as massive data volume, increasing scale and deceptively simple business requirements which translate to bleeding-edge software;
• The Israeli R&D center is the only in-house engineering center for nBA. Employees have unprecedented influence on the company, as well as direct communication with customers. There’s nothing like actual business feedback to motivate and ensure success!

If any of this resonates with you, go ahead and send your CV to jobs@newbrandanalytics.com, or contact me directly at tomer@tomergabel.com.

Download ant-intellij-tasks-1.0-b1.zip

A great but oft-ignored feature of Visual Studio 2005 and up is the inherent consolidation of an important developer tool: the build system. With a Visual Studio solution you can simply run MSBuild and you get accurate, automated builds. This is an invaluable capability: continuous integration is ridiculously easy to set up, as are nightly builds and automated deployment tools.

Since I started working for Delver (now Sears) I’ve been switching back and forth between C# (2.0 and later 3.0) and Java 1.6, and though the ecosystems share many similarities there are also several glaring differences. The first of these differences is that, in the Java world, it is perfectly acceptable – even traditional – to maintain a dual project structure, one using the IDE (usually Eclipse or IntelliJ IDEA) and one using one of the build tools (commonly Ant or Maven). The build scripts need to be continuously synchronized with the project structure, and output parity between the two separate build systems is almost unheard-of.

Because I had been a complete Java newbie when I started, I had never had the time to really sit down and set up a continuous integration server for our Java codebase, a mistake I did not intent to repeat when Sears took over. The first item on my agenda was to do away with the dual project structure; we originally used Eclipse, so I built a custom Ant script (my first, actually) around ant4eclipse and managed to come up with a semi-satisfactory solution. This also gave us invaluable insight when it was time to revisit our IDE choice; the lackluster project structure offered by Eclipse, along with firm positive comments on IntelliJ IDEA from several team members, tipped the balance and led us to switch to the alternative IDE, while also creating the necessity for a revamped build system can that work on top of the IntelliJ IDEA project structure.

Out of necessity, a project was born. ant-intellij-tasks is the result of several months of all-night itch-scratching on my part. While not directly affiliated with the company, we’ve been dogfooding the project at Sears for over a month now, and while there are certainly rough edges it finally seems stable enough for release! From the project website:

ant-intellij-tasks is a self-contained build system for IntelliJ IDEA projects based around Apache Ant. In essence, ant-intellij-tasks comprises three components:

1. An Ant task library that can extract and resolve the IntelliJ IDEA project and module files (.ipr and .iml respectively), and provides a set of tasks and conditions around the project structure;
2. A common build script which provides the four major build targets for modules: clean, build, test and package (see the quickstart guide);
3. A master build script which extends these targets to the entire project.

The build system is designed to be extensible (e.g. by adding targets), customizable (e.g. by overriding a target's behavior for a specific module) and self contained in that it's a drop-in solution that should not require any significant modifications to the code base.

This project is fully open source (distributed under an Apache license) and hosted at Google Code. Please report any bugs or issues on the project issue tracker.

ant-intellij-tasks makes use of, and redistributes, the ant-contrib task library.

Apparently Java has quite a few known but practically undocumented issues with its handling of UNC paths under Windows. I’ve specifically encountered this bug albeit in a slightly different scenario:

@Test
publicvoid test() throws URISyntaxException
{
final URI uri = new URI( "file://c:/temp/test/ham.and.eggs" );
new File( uri ); //
IllegalArgumentException thrown here

}

            

              /**
              

               * Resolves the specified URI, and returns the file 
              

               * represented by the URI.
              

               *
              

               * @param uri The URI for which to return an absolute
path.
              

               * @return The {@link File} instance represented by the
              

               * specified URI.
              

               * @throws IllegalArgumentException <ul><li>The
URI cannot 
              

               * be null.</li><li>Wrong URI scheme for
path resolution;
              

               * only file:// URIs are supported.</li></ul>
              

               */
              

              public
              static File
getFile( URI uri )
throws IllegalArgumentException {
if ( uri == null )
thrownew IllegalArgumentException(
"The URI cannot be null." );

            

              

              

              if ( !"file".equals(
uri.getScheme() ) )
thrownew IllegalArgumentException( "Wrong
URI " + 

"scheme for path resolution, expected \"file\" " +
"and got \"" + uri.getScheme() + "\"" );

            

              

              // Workaround for the following bug: 
              

              // http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5086147
              

              // Remove extra slashes after the scheme part.
              

              if ( uri.getAuthority() != null )
try {

uri = new URI( uri.toString().replace( 

"file://", "file:/" )
);

} catch ( URISyntaxException e ) {
thrownew IllegalArgumentException( "The
specified " +
"URI contains an authority, but could not be " +
"normalized.", e );

}

            

              

              

              return
              new File(
uri );

}

          

Note: This is more or less a translation of two forum posts in an Israeli home theater website; if you can read Hebrew, you may be interested in the comments as well. First post, second post

Lately I have been bit by the audio bug again, and have decided to experiment with my headphone setup. This is the setup I use at work and listen to for hours at a time (often 5 or more hours a day), and which consists of a pair of 2003 model Beyerdynamic DT880 cans I bought on a previous trip to China, hooked up to a G&W T-2.6F amp I bought on the same trip after having spent an hour listening to various equipment combinations. Unlike products from other, internationally recognized brands with which I am familiar (Creek, Musical Fidelity etc.) this product was completely out of my comfort zone: a Chinese-made amplifier, which like many others I normally associate with cheap components and subpar build quality. Additionally it is  rather large and bulky, and to top it off, it is a hybrid design based on two Electro Harmonix 6922EH preamplifier tubes and a solid state power section. Up until that point I had heard solid state equipment exclusively, with the exception of two loudspeaker demonstrations in which the amplification included tube components and was significantly more expensive than I could even consider. Since I had not known what to expect I could not detect the subtleties of tube sound, and had chosen this headphone/amp combination strictly on being the best I had heard during that visit.

This setup was, in turn, hooked via a generic interconnect to my work computer with onboard HD audio, through which I play mostly lossless rips from my own CDs via Exact Audio Copy. It had faithfully served me for over three years, at which point I decided to do some research and was first exposed to the vast world of tube-based amplification and tube rolling. After several days of forum lurking and reading well into the night I had placed my first ever tube order at thetubestore.com. With the help of the shop representative, Jon, and general recommendations around the web, I had selected a matched pair of JAN-Sylvania 7308 tubes and yet another matched pair of Mullard E88CC tubes and placed my order. I did not have to wait very long, as the UPS delivery arrived amazingly fast (a single weekend, not too shabby for an international delivery!), and decided to spend a few weeks with each pair to be able to form an honest, educated opinion. I begun my experiments with what is, according to general consensus, the weaker tube: the JAN-Sylvania 7308.

What a shock! While I do not, for a moment, assert that similar or better sound cannot be found in solid state amplifiers, I certainly did not expect such a dramatic difference in sound quality. The soundstage, previously wide but shallow, simply exploded! It’s as though the sound instantly multiplied its volume tenfold or more; highs became wonderfully airy and distinct, and the resolution… let me put it this way: in every audio enthusiast’s life there are but few such moments of enlightenment, where you suddenly realize how much more is possible, and even attainable. The first time I’ve listened to the very same equipment with upgraded tubes provided me with one of those rare occasions, and from that point on I can never settle for less.

At the same time I had a second such revelation, albeit by accident: because of the physical layout of my desktop at work I was forced to place the amp further away from the computer, which necessitated a longer interconnect cable. I did not have one at hand and until I was done for the day I was left with no alternative but to use the iPod’s standard analog output. The iPod is generally considered to have very poor analog performance, which is why I was thoroughly surprised when, having brought a longer (and higher quality) interconnect from home and hooked the computer up, I found that the iPod actually sounded better. I recall when it was almost impossible to find a decent quality audio card for your computer, and assumed that contemporary solutions were at least adequate; indeed, the computer sound output was cleaner (better SNR) but also had significantly diminished dynamic range and volume. This led me to the conclusion that an audio card upgrade was in order.

A little research into the subject brought me to the the conclusion that what I’m interested in is not, in fact, a computer audio card; what I want is an external DAC, or more specifically a USB DAC. Getting a computer to output even half-decent analog audio is pretty much a futile quest, and while hooking it up via coaxial/optical S/PDIF would certainly work there are some significant disadvantages, namely: digital (lossy) volume control, and jitter. With S/PDIF, both clock and data signals are encoded together on a single data line, and the click has to be regenerated. This introduces subtle timing inaccuracies, generally known as jitter, which in have an undesirable impact on digital-to-analog conversion (a more scientific explanation can be found here). Just how significant an impact is a subject of much controversy, but at a USB DAC has the theoretical advantage of significantly reduced jitter on the protocol level, as well as removing the question of the onboard S/PDIF encoder’s clock accuracy from the equation.

With the advice of fellow forum members I resolved to try one of the following DAC trio: Cambridge Audio DacMagic, Oritek OMZ DAC or Musical Fidelity V-DAC. Following a lead from a fellow forum member I eventually bought the V-DAC for a very good price from a head-fi.org forum member. The V-DAC is a 192KHz/24-bit upsampling DAC with optical, coaxial and USB inputs that has received high praise in the head-fi circles and is even available in Israel for a surprisingly reasonable price.

I hooked the unit up with an unnamed but high quality silver interconnect, and after significant critical listening I can draw the following conclusions: compared with both iPod and onboard audio card (as well as an old Audigy 4 I had lying around) the V-DAC features significantly improved accuracy and resolution, and more and more often I’ve been rewinding tracks just to make sure that, yes, I wasn’t imagining, I really have never heard this or that detail before. This is exactly what I got into audio for in the first place! The bass is also much tighter, and in my opinion also extends further down than it ever did. In fact, the only disadvantage is in a certain change in the soundstage, as though the stereo separation grew just a littler wider than I’d like. Don’t get me wrong, the soundstage is huge and imaging is terrific, but it sometimes seems to be that sounds tend to cluster a little closer to the extremes of the soundstage than they should.

All in all I’m extremely happy with the upgrade, and luckily I still have some new equipment left to play with: the Mullard E88CC tubes, patiently awaiting my pleasure. Still, now that there’s such a significant difference in fidelity between my work and home setup (which consists of an Aqua Mini-Head amp and Beyerdynamic DT880 2005 edition cans) I feel compelled to experiment with new equipment. Whatever shall I try next, a new DAC? Another amp? Different cans, perhaps? The choices are endless, and that’s the beauty of it. See you on the next upgrade.

Update: Some more pictures can be found here.

A couple of years back I went to Breakpoint 2007, which was my first international (read: non-Israeli) demo party, as well as the first proper demo party I went to in 9 years (the Israeli demo scene had a couple of small get-together events in Kamon in 2000 and later in 2005, but I don’t consider those actual demo parties). Along with Bacter and my brother Mickey, we three were the only Israelis to be found at the party.

I missed Breakpoint 2008 due to product release pressures, the normal state of affairs while working for a small startup; I resolved not to miss it again this year, and made plans with Bacter and Mickey to meet up at the party place. Executive summary: amazing people and amazing scene spirit. I spent nearly all of the party outside with a beer in my hand chatting with people. It’s amazing just how much diversity one can find in such a small group; really the only common grounds is a general love for art, freedom of expression and the demoscene in particular. One moment I may be involved in a deep political discussion with a bunch of Germans (greets Streettuff/TRSI), and the next I’ll be drawing a comparison between English, Dutch, German and Hebrew with a bunch of Dutch guys (hi Cosmiq!) or quoting Borat with our resident Portuguese Jeenio. While I’m at it, greets to Luise, Julius, Jan and Manu from München, Okkie, the Misfit of the C64 scene and everyone else with whom I’ve spent with and whose name I can’t remember :-)

Although some of the compos this year had disappointing turn out (in particular, out of 25 or so demos maybe two or three are noteworthy) the party was still great fun. The lack of sponsors did very little to detract from the quality of the party, possibly the opposite in fact: the event was sponsored out of the entrance fee and donations made by sceners in the few months before it took place, the net result being that everyone present was happy to be there and the party really took off. Kudos Breakpoint organizers!

I recommend watching the following productions from Breakpoint 2009:

Everything is Under Control by the ever-prolific mfx is the invitation demo to Breakpoint 2009, which brings to the table mfx’s usual array of amazing 3D graphics, 2D effects, fast code and coherent, though disturbing, design. With its 1984-esque theme this demo set the theme for the entire party.

One of the noteworthy demos from Breakpoint 2009 is Freedom From State by Hullabaloo: this demo was entirely written at the party-place by blala, who had been sitting with has MacBook right next to us the whole party and coding furiously in Haskell. Yes, you heard me right: the demo is written in Haskell, which (along with the party theme) is why Freedom From State is such an excellent name, even though the demo itself is quite unremarkable.

And in the wicked cool department, lft (of Craft fame) is at it again with another microcontroller-based demo: Turbulence (or on pouët). This time the custom hardware platform is based on a Parallax Propeller chip, and the demo itself is both good (in an oldskool kind of way) and damned technically impressive at that. Kudos!

Excelence by the group with the awesome name BraadWorsten Brigade is probably the world’s first Excel demo, and proves just how fortunate we are that this is the case :-) Don’t take me wrong, it’s awesome and even funny, but if no-one else ever makes another VBA demo it won’t be soon enough…

Breakpoint 2009 has seen a lot of first-time productions by new demo groups; of these my favorite is PC-03 ON/OFF by Panda Cube. A stylized 3D flyby with subtle shades and nice presentation. I hope these guys go on to make demos, lots of potential there!

Although this was not strictly their first production, Conscious of Blue by System-K is another favorite of mine: a clean, well-designed and imaginative demo that’s very different from the typical European demo style. No surprises there; these guys come from Japan. I didn’t even know Japan had an active demoscene, although for the life of me I don’t see why not. Kudos guys!

While Crush by Anadune and Floppy was not the only enjoyable PC demo at Breakpoint 2009, it was certainly the most impressive: the right blend of technology, design and music. Borrowing a leaf from Debris by farbrausch, this demo features plenty of deformable objects and lots of glow, but is different enough in style, pacing and content to stand out on its own. Two or three scenes here (such as the one pictured) are simply astounding.

It seems white is the new black, with at least three white-themed productions at this party alone. With that in mind, 060659 by Rebels is an excellent (if not groundbreaking), stylized 64k intro that’s always great to watch. The commodore fan-service in the middle is gratuitous, although the effect itself is absolutely brilliant. Music is also subpar, but the design more than makes up for it IMO.

One of the most technically impressive C64 demo I’ve ever seen (possibly on par with Second Reality 64), Das Gotler by Extend and Dekadence hits you from the very first moment (with how the C64 basic window is cleared). The downside? Horrible, horrible music.

The last few years have seen some amazing new artistic outlets for the scene, particularly commercial-quality animations in the compos. Breakpoint 2009 had a couple of fantastic entries, notably the winning duo. 2nd place animation compo winner Julie by Nuance is both a fantastic artistic expression and a terrific tech demo: with a 300 Euro budget and stuff they had lying around at home, the team tried (rather successfully) to imitate the bullet-time effect popularized by The Matrix. They’ve also released a “making of” document that’s a fairly interesting read.

On the other end of the spectrum you’ll find JCO’s Spiechergurke, a fake commercial for a new kind of storage product (watch with subtitles). Other than being very well made it made me laugh my ass off. I think all in all Julie was the better production, but it was a very hard toss-up between the two; at any rate I’m glad both won the competition (Speichergurke took 1st place).

Jesus Christ Motocross by Nature and Traktor is, other than being a heavy hitting, funny and fun to watch, an amazingly impressive Amiga demo. Nontrivial effects (all in software, obviously), psychotic pacing and music and apparently artifact-free code are all fine and dandy, but the Tron tribute pictured on the right won my heart.

That said, Lightshaft by Elude is a very strong runner-up; 2nd place Amiga demo compo winner, this demo combines an incredibly impressive array of 3D scenes with epic design and pacing, terrific graphics and excellent music. It’s ironic that the two winning Amiga demos were so impressive whereas the PC demo compo suffered from general lack of enthusiasm and polish.

Easily the best PC 4k intro I have ever seen, Elevated by Rgba and TBC is also possibly the first to ever get me excited. Astounding visuals, top notch design and excellent music are only part of it; the picture on the right really does not do this production justice, and you should definitely watch it in its entirety at least once to appreciate just how amazingly good a demo can be at 4096 bytes!

Other notable productions from Breakpoint 2009:

• Defcon Zero by Scarab for the Nintendo DS
• Syntax Infinity by Tulou and Traktor for the MSX2 platform
• fr-065: euphotic by farbrausch, a technically impressive but boring and uninspired PC demo
• Enigma Sequence by Approximate, a 64k intro that’s really close to being awesome. I think a couple more weeks of polish would’ve really turned this one into a winner, but as is it’s quite raw.
• Luminagia by Loonies, Amiga 4k intro. Not quite as polished as the PC 4ks of the last few years, but damned impressive never-the-less.

I also got to watch the following productions on the big screen at Breakpoint, each of which is a recommended watch:

While Breakpoint was still in progress, Rupture by Andromeda Software Development won 1st place in the demo party at The Gathering 2009 and with all due reason: this demo is fantastic. Coherent design, astounding visuals, excellent pacing and music – it does everything well. For a demoscene fan, watching this on the big screen was a little like watching Terminator 2 in the theater for the first time: it gives you a profound sense of “this is what production values are all about.” The screenshot can’t do it justice, just go watch it already!

Conversely, NVScene 2008 winner Stargazer by Andromeda and Orb is not as fluently directed but at least as technically impressive. I simply love Andromeda’s flow, the way they always manage to bring closure to a scene before moving on to the next effect, even if the two aren’t related in any way. Stargazer is a slideshow of some of the most impressive effects ever seen in a demo, with astonishing visuals and excellent techno music; I’m not sure which of the two (Stargazer or Rupture) I like better, but I guess they each appeal to a different school. Both are definitely must-see.

A veritable demoscene poster-boy, the NVScene 2008 invitation intro Masagin is the brainchild of Paniq (the guy behind Die Ewigkeit Schmerzt). A high quality production with an obvious artistic bent, Masagin blends excellent music with unique effects and fluid design and is one of the most engaging demos I’ve seen in years.

Andromeda Software Development demos typically fall into one of two categories: artistically done 3D slideshows (Dreamchild, Rupture), and technically impressive video art (Evolution of Vision, Beyond the walls of Eryx). Midnight Run, 3rd place winner at NVScene 2008, is definitely of the latter sort, seamlessly blending 2D and 3D graphics with a bizarre screenplay and excellent music. Although not trivial by any means, if you’re looking for a technical demo to boggle your friends’ minds with, look elsewhere; Midnight Run is definitely for those not looking at demos with just an analytic eye.

Proving my previous point, Euskal 2008 demo compo winner Size Antimatters by Andromeda Software Development is precisely the opposite of Mindight Run: it’s a technological powerhouse with amazing effects and great techno music, a lot faster paced than Midnight Run and built for a different audience. Along with Rupture and Stargazer, these are my current “show off your rig” demos. Kudos!

½-bit Cheese are fast becoming my favorite demoscene animators. Their Assembly 2008 wild compo tour de force Field Trip features some of the most amazing animation, visual effects, music and direction I’ve ever seen, taking the already-excellent talents of Maxson and D-Fast (of Realtime Demo Wannabe fame) to the next level. Groundbreaking!

I ran a Google image search today, and was surprised to see this:

New feature, hurray! (and may actually prove useful…)

It seems nothing to do with maintaining this website is as easy or as simple as it should be. Whenever I switch hosts it’s an uphill struggle to get the site up and running again; whenever I upgrade dasBlog to a newer version I have to learn a lot about how it works, how ASP.NET works, how IIS is configured etc. It’s enough to make me seriously consider hosting my blog elsewhere and/or moving to another blogging platform, but the truth is I love dasBlog so much I simply forget how complex and volatile it can be and have to go through the same frustrating process whenever something changes.

The way I update my site is usually this:

1. Ensure I have an up-to-date local mirror of the website. dasBlog keeps all of its data in XML files, so backing up the website is simply a question of wget –-passive-ftp –m ftp://user:password@website.com; I have a daily scheduled task to take care of this.
2. Copy the latest mirrored version to a working directory; set the directory up as an IIS website/virtual directory.
3. Test the new working copy to make sure it works.
4. Perform whatever modifications are required.
5. Test again to make sure that the website works with multiple browsers (this time I tested with Chrome 2.0.170.0, Firefox 3.1 Beta 3 and IE 8)
6. Upload the website over FTP using the FileZilla client. I always verify that the relevant configuration files and binaries are overwritten and nothing else.

This process generally allows me to test upgrades before uploading them to the “production” website, as well as provides an easy rollback path if something goes wrong. Thing is, something always goes wrong. In this case, although nothing’s changed in the site configuration I started getting SecurityExceptions just after the upgrade:

Request for the permission of type 'System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' failed.

This is one of the least informative error messages I have ever seen. All it tells me is that a request for some permission is denied; it doesn’t say what permission was requested, nor by whom (the stack trace seemed to indicate the permission was asserted from within System.Diagnostics.Trace, which doesn’t make much sense). A quick web search brought me to this page, which deals specifically with installing dasBlog on a GoDaddy-hosted website. Because GoDaddy runs ASP.NET applications under a modified medium trust that allows file-system access only to the virtual directory hierarchy, the site recommends adding a virtual directory for each of dasBlog’s writable directories (content, siteconfig, logs); I tried this out and the problem was not resolved.

At this point I was getting desperate, and was willing to try just about anything to get the site up and running again. Eventually I ran a diff between the site backup and the newly modified version, and found a new openidConsumerTrace.txt file in the site root. I’ve never seen that one before; where'd it come from? A quick search showed the following section in the web.config file:

            <
            system.diagnostics
            >
            <
            assert
            assertuienabled
            ="false"
            />
            <
            switches
            >
            <
            add
            name
            ="OpenID"
            value
            ="4"
            />
            </
            switches
            >
            <
            trace
            autoflush
            ="true"
            indentsize
            ="4"
            >
            <
            listeners
            >
            <
            add
            name
            ="fileLogger"
            type
            ="System.Diagnostics.TextWriterTraceListener"
            initializeData
            ="openidConsumerTrace.txt"
            traceOutputOptions
            ="None"
            />
            </
            listeners
            >
            </
            trace
            >
            </
            system.diagnostics
            >
          

A-ha! So the OpenID activity trace log is written to the virtual root, which is not writable (I set the ACLs to only allow writes to the above three directories). I tried changing the trace file path to ~/logs/openidConsumerTrace.txt (which is a virtual directory and has the appropriate write ACL), but this did not resolve the problem. At this point I was ready to roll back to the previous version and work on switching to another (perhaps hosted) blogging platform, and in my despair I simply commented out the whole system.diagnostics section; oddly enough, this resolved the problem…

Now I know dasBlog is free and there’s little or no point complaining, so I hope this post helps someone handle the problem. And if anyone from the dasBlog team is reading this… please be a little more careful with undocumented dependencies?

I’ve been meaning to change the site’s theme for ages, and as often happens with these things this gave me an excuse to overhaul various aspects of the site. Here’s a bunch of stuff that’s changed:

• Upgraded to dasBlog 2.3 and tweaked a whole bunch of settings. Hopefully this will enable all sorts of interesting stuff, such as OpenID commenter identification and coComment support. I’ll post my upgrade experiences separately;
• Switched the theme to a slightly tweaked version of the dasBlog “business” theme by Christoph De Baene (thanks for the help, Ken!);
• Got rid of the “advocacy” section on the right. I still strongly advocate Firefox, Vorbis and OpenOffice.org (among others), but there doesn’t seem to be much point in placing banners just for that;
• Updated the blog-roll with my latest list.

Some code examples on the site may look a little weird on account of the CSS changes; over the coming days/weeks I’ll be fixing those, as well as recompressing images and other behind-the-scenes changes that will hopefully make the site look better and load faster.

Update (8 May 2009): Eli Ofek pointed out in the comments a little-known but effective tool called the Managed Stack Explorer. Although it features a basic GUI, it can effectively be a .NET equivalent of jstack if you add to the path; then it’s just a question of typing mse /s /p <pid>. It’s a little slower than jstack but worlds better than the alternative suggested below.

I’ve been working mostly with Java over the last year, and the .NET code I write is usually limited to interface code between our .NET-based and Java-based components. A long while away from production-grade code on Windows means I need to brush up on my production debugging skills.

On today’s menu: thread dumps, or per-thread stack traces if you will. With Java code (at least starting with Java 5) this is as easy as jstack <pid>; with .NET it turns out to be quite a bit more complicated:

1. Download and install the appropriate Debugging Tools for Windows version for your architecture (x86/x64/Itanium)
2. If you need information about Windows function calls (e.g. you want to trace into kernel calls), download and install the appropriate symbols. This isn't strictly necessary if you just want a thread dump of your own code.
3. If you need line numbers or any other detailed information, make sure to place your assemblies' PDB files where the debugger can find them (normally you just put them next to your actual assemblies).
4. Start->Programs->Debugging Tools for Windows [x64]->windbg
5. Attach the debugger to your running process using the menu
6. Load the SOS extension with ".loadby sos mscorwks" for .NET 2.0 (".load sos" for .NET 1.0/1.1)
7. Take a thread dump using "!eestack"
8. Detach using ".detach"

Quite a bit of work for something as trivial as a thread dump. I hope .NET diagnostic and debugging tools improve with time (Process Explorer is definitely a step in the right direction).

Don’t ya think?

So I had to find myself a new job. Delver was about close down, the employees (including yours truly) were handed notices and the next few weeks were spent searching for my next job. I guess breaks come not only when you least expect them but also from the least likely direction: Delver was bought by Sears and made into SHC Israel, not to mention the company’s first overseas headquarters and development center.

You’ve read correctly: Sears. Not Amazon, not Google, not Microsoft. Delver, a strictly web-based startup, wasn’t acquired by a web company; not even by a technology company at that. Instead we were acquired by one of the United States’ largest retailers. Why, you ask? Well, with any luck you’ll find out in a few months :-)