In addition to the wide press coverage on US-oriented technology sites we've seen coverage from two major Israeli news providers (Hebrew only, for now): Calcalist and TheMarker.

Now comes the fun part; Delver is still borderline-alpha. We've been working hard testing and tweaking it, and getting a system of this complexity working in good order on a ridiculously short schedule feels astounding. I sincerely believe the Delver premise is a solid one, and we're giving you a mere inkling of what's in store for the concept; now all we have to do is work harder, growing along with the product and slowly but surely realizing its full potential.

The brilliant part? Beyond the dreams of rich and fame, this product already is useful; with relentless improvements it may yet become as indispensable a tool to Internet denizens as Google, Wikipedia and Facebook are today.

An alpha version of our search engine is now open for all users!

We've been working towards this day for the past year, building a complete and functional search engine from scratch on a completely original premise. I'm both amazed and proud of the work done by the various teams, and I'm still can't believe we've managed to pull this off in so little time. Launching the search engine publicly seems like a great way to celebrate the year I've been working for Delver (as of July 1^st).

Mind you, the service is still new and we're hammering away at the kinks, but so far we've had overwhelmingly positive press coverage and the various comments are sincerely flattering. Here's to another amazing year!

As an aside, we're got openings on my team (search back-end) for extremely talented software developers who are interested in building performance-driven, robust back-end software in a variety of technologies. Interested? Contact me for details at tomer@delver.com!

After figuring out the problem with the old dasBlog permalinks I had to figure out a way to convert all existing links in my blog to the new format. Lately whenever I need a script I try and take the opportunity to learn a bit of Python, so it took an hour or two to write the conversion script.

Here it is; if you want to use this for your own copy of dasBlog, change the "domain" global variable to wherever your blog is located and run this from your ~/Content directory (you can also download the script here):

#!/usr/local/bin/python
#
# convert_permalinks.py
# Quick and dirty permalink converter for dasBlog content files
#
# Tomer Gabel, 22 June 2008
# http://www.tomergabel.com
#
# This code is placed in the public domain (see http://creativecommons.org/licenses/publicdomain/)

from __future__ import with_statement
import os
import glob
import re
import urllib

# Static constants
domain = 'tomergabel.com'
href_lookup = re.compile( 'href="(http:\/\/(www\.)?' + re.escape( domain ) + '/[^"]*\+[^"]*?)"' )

# Globals
conversion_map = {}

# Takes a URL and removes all offensive characters. Tests the new URL for validity (anything other than a 404 error is considered valid).
# Returns a tuple with the converted URL and a boolean flag indicating whether the converted URL is valid or not.
def convert( url ):
	new_url = url.replace( "+", "" )
	# Check URL validity
	valid = True
	try:
		resp = urllib.urlopen( new_url )
		resp.close()
	except:
		valid = False
	return [ new_url, valid ]

# Processes the source file, converts all URLs therein and writes it to the target file.
def process( source_file, target_file ):
	with open( source_file, "r" ) as input:
		source_text = input.read()

	conv_text = source_text
	match_found = False
	for matcher in href_lookup.finditer( source_text ):
		if ( matcher != None ):
			match_found = True
			original_url = matcher.group( 1 )
			print "\tConverting permalink " + original_url
			if not conversion_map.has_key( original_url ):
				conversion_map[ original_url ] = convert( original_url )

			conversion = conversion_map[ original_url ]
			if conversion[ 1 ]:
				print "\tConversion successful, new URL: " + conversion[ 0 ]
				conv_text = conv_text.replace( original_url, conversion[ 0 ] )
			else:
				print "\tConversion failed!"

	# Write out the target file
	if match_found:
		with open( target_file, "w" ) as output:
			output.write( conv_text )

# Entry point		
for file_name in glob.iglob( "*.xml" ):
	print "Processing " + file_name
	process( file_name, file_name + ".conv" )

What a stroke of luck! With more and more sites foregoing captchas, I was starting to think weird-ass captchas are a thing of the past. Fortunately trusty old RapidShare brought me to task:

Apparently not only am I meant to figure out the convoluted glyphs (it took me a while to figure out that there are no numbers -- their G looks exactly like a 6, O and 0 look the same etc.), but I'm supposed to match only the letters connected to a cat. Conceptually amusing, but what the hell? Can you tell me which of the above are cats? Is it reasonable to expect someone to spend more than about 5 seconds on a captcha?

Oddly enough I re-entered the link a little while later to discover the amazing RapidShare feature called "Happy Hour":

And I'm just left scratching my head.

Update (22 June 2008): I've posted a Python script for converting the invalid permalinks to their "proper" form.

After moving to GoDaddy I found to my chagrin that none of the site's article links (a.k.a permalinks) seem to work. There are any number of reasons why this is a bad thing, the two primary reasons being that Google cannot crawl the actual articles and that historical, incoming links no longer work. As they say, crap on a stick.

I originally looked into the URL rewriting rules and HTTP handler configuration in web.config, thinking that perhaps some of the handlers need to be manually registered with IIS for some reason; eventually I installed the blog locally, migrated to IIS pipeline mode (which might be cool but has no tangible benefit for me) but the problems persisted. Until I tried accessing a permalink URL locally, that is. Then I discovered that plus signs in URLs (dasBlog's way of avoiding encoding spaces to %20) are considered "double-encoded" characters, and are automatically rejected by IIS 7.0 by default because under some circumstances they pose a security threat.

There's a knowledge-base article detailing how to resolve this on a server or per-application level, but either solution requires server reconfiguration. Working under the assumption that this entails special requests from each and every potential future web host, and that having plus signs in my URLs is not the "right thing to do" anyway, I just opted to disable them and try to rework the existing links as best I can. I suppose blogs with considerably higher traffic cannot afford that luxury, but then blogs with considerably higher traffic usually work with much more specialized hosting providers and wouldn't have to worry about server reconfiguration in the first place...

Anyway, hope this helps someone.

Note: This is a translation of a forum post in an Israeli home theater site. If you're Hebrew-enabled, the ensuing thread might be of interest to you.

After a long period of running around, testing and making arrangements I've finally bought a new set of speakers for my home theater setup. I intend to describe the process, so if you're impatient feel free to scroll to the conclusion.

I've used the same set of CDs, same amplifier and same testing methodology for each system I was demonstrated:

1. A Yamaha RX-V3800 A/V receiver, hooked up with HDMI where available and regular optical cable where it wasn't
2. I started each demonstration with a consistent set of musical segments, with the receiver set to Pure Direct mode. Specifically:
   • Telegraph Road (out of Love Over Gold by Dire Straits)
   • Converting Vegetarians (out of the album by the same name)
   • Elation Station (same album)
   • Sharp switch to Creedence Clearwater Revival, with Lookin' Out My Backdoor
   • Followed by Heard It Through the Grapevine by the same band
   • Another sharp switch to Ear Parcel (out of Fear of Fours by Lamb. It's kind of "busy acid jazz" -- recommended for giving amplifiers a serious run for their money)
   • And ending with Telegraph Road, again out of the Dire Straits album Love Over Gold
   • In some cases I also played excerpts of Time and The Great Gig in the Sky by Pink Floyd
3. I then proceeded to the home theater tests:
   • Master and Commander: The Far Side of the World (the first naval warfare scene, as well as the scene where the captain and doctor play music together)
   • I Am Legend on BluRay (Dolby TrueHD, mostly the chase scenes)
   • Pan's Labyrinth on BluRay (DTS HD Master Audio, mostly dialogue-oriented scenes)
   • Harry Potter and the Order of the Phoenix (several segments, the O.W.L test scene in particular)
   • Gladiator (opening scene)
4. After hearing all available options I've narrowed the selection down to three main (front) speakers and borrowed them for testing at home.
5. The prices mentioned are from memory, in New Israeli Shekels and can be considered MSRP. In other words, you can usually get the same goods for a considerably lower price. Additionally, Israeli customs have a ~30% tax (VAT included) on loudspeakers, so prices here aren't really representative of the market price in the US.

The setups I've tested were:

Klipsch

• Front: Klipsch RF-82
• Surround Back: Klipsch RS-42
• Center: Klipsch RC-62
• Subwoofer: Klipsch RW-10d

This system is particularly characterized by a "large" sound-stage, and seems to be designed mostly for home theater use. Compared to most other systems I've listened to, the Klipsch speakers exhibit full and punchy sound in home theater, but which sounds slightly anemic when listening to music (the sound produced lacks volume - particularly noticeable with Dire Straits) and lacks resolution. The center, however, exhibited great potential in dialogues (Pan's Labyrinth in particular); the next center speaker high up in Klipsch's product offerings is the RC-64, which sounds even more impressive and is one of my candidates for purchase.
Bottom line: a good choice for those interested specifically in home theater, but Paradigm's more impressive offerings are available in the same price range in Israel.
Total cost: approximately 24,000 NIS.

Focal, Chorus series

• Front: Focal 816V
• Center: Focal 800CC
• No surround back
• No subwoofer

In a word: disappointing. I didn't even bother to continue with the home theater benchmarks. I had high hopes for this setup, both because of recommendations by several forum members in various conversations and because I have a Focal component set in my car and was pleasantly surprised by their performance in that market segment. With this setup, however, the result was a "dirty" sound that lacks proper resolution, is sharply biased towards low bass and over-bright treble. I consider this by far the weakest setup I have tested in my search.
Cost of the front speakers: about 8,000 NIS. Update: Forum members have pointed out that the actual price may be closer to 12,000 NIS.

Triangle, Esprit EX series

• Front: Triangle Antal EX
• Center: Triangle Voce EX
• No surround speakers
• No subwoofer

These speakers exhibit a very French sound, open and airy with terrific mid-bass and no shortage of punch in the low-end. The Antal were some of the most impressive speakers I've examined (in price as well as performance) and I ended up borrowing them for testing at home. The center speaker had terrific resolution and a very pleasant sound, despite the fact that the particular speaker tested was new and not yet broken in.
Cost of the front speakers: About 11,000 NIS.

ProAc

• Front: ProAc Studio 140
• Center: Focal Chorus 800CC
• Surround back: Focal Electra 1007 Be (I think)
• Subwoofer: B&W ASW608

I am not really partial to British loudspeakers in general, and ProAc in particular; I had previously listened extensively to both Tablette Signature and Response One models and had disliked their tonal character, which in my opinion is overly analytical and dry. This is why I entered this particular demonstration with rather low expectations, but the Studio 140 speakers simply blew me away from the get-go: they combine astonishing detail and resolution with huge sound-stage, and provide low, tight bass that's way out of their size league. At an MSRP of 15,900 NIS (actual price considerably lower) they are 30-40% more expensive than my other favorites.
The rear speakers also impressed me, particularly with the ambient effects in Master and Commander, but I think they are from a very highly priced series in Focal's line.

Paradigm, Monitor series

• Front: Paradigm Monitor 11
• Center: Paradigm CC-290
• No surround speakers
• Subwoofer: Paradigm DSP-3400

I'll admit that I came to this particular demonstration positively biased; I've known the old Monitor 9 v2 model for years and very much like the sound character of Paradigm's products. This system is one of the finest I've heard: the Monitors dealt incredibly well with just about any music I could throw at them and produce a deep, wide sound that I find very attractive. In the home theater benchmarks this setup absolutely shined: the Monitors themselves produce deep, penetrating bass and overall sound that is far tighter and more detailed than the Klipsch RF-82s. The center speaker is very convincing in dialogues, but lacks in the bass department, which is why I'm hoping to test the CC-390 and add it to my list of candidates for purchase. Regardless, I ended up borrowed the Monitor 11 pair for testing at home.
The subwoofer deserves a separate discussion which I get to further down the page.
Total cost: about 20,000 NIS, making this an amazing value for the money.

Paradigm, Studio series

• Front: Paradigm Studio 60
• Front: Paradigm Studio 100
• Center: Paradigm CC-290
• No surround speakers
• Subwoofer: Paradigm DSP-3400

Since the previous system left me some room in my budget I've allowed myself to test two of Paradigm's much-vaunted Studio series loudspeakers. The Studio 60 failed to impress me despite their impressive detail and resolution, this due to lacking mid-bass and low-end; I moved on quickly to comparing the Studio 100s directly with the Monitor 11. This model was far more to my liking: clarity and high resolution combined with tight, bunchy bass, but the sound still felt too thin and lacking in authority compared to the Monitor 11. To those of you looking for a highly detailed, analytical loudspeaker I definitely recommend giving the Studio 100 speakers a listen, but they just don't match my own preferences.
Cost of the Studio 100: roughly 16,000 NIS.

Finally I've heard several front and center speakers that I did not mention (certain Canton and Wharfedale models, for instance) because in my opinion they are not in the same league as those products. I will not bother expanding on those.

Eventually I've reached the dilemma of choosing between Triangle Antal EX (a beautiful loudspeaker with a deep, warm tonal character), Paradigm Monitor 11 (an impressive loudspeaker with serious presence, nor is it shy playing just stereo) and the considerably more expensive ProAc Studio 140. I was given all three pairs for testing at home during the Shavuot holiday, which gave me a relatively long time to spend with each speaker. I had four days to experience a variety of music, movies and TV series in a wide variety of genres, and have managed to reach several conclusions about these speakers:

1. First and foremost it's worth noting that each and every pair sounded simply magnificent. I felt privileged to be able to simply switch from one pair to another and experience their magnificent respective sounds.
2. I don't care what other people say, the Antal EX speaker is simply gorgeous.
3. For all you married people: my SO actually preferred the looks of the Monitor 11.
4. The Antal and Monitor speakers have similar character to their sound; where the Antal set was more detailed the Monitors compensated with glorious mid-bass, and where the Antals exhibited great low-end muscle the Monitors shined with excellent high-end resolution. If I had to pick between the two I'd be facing a bothersome dilemma, however...
5. The Studio 140 set simply offers the best of all worlds and was the clear winner in my mind. Accurate and detailed yet exhibiting an amazingly wide sound-stage with deep, low, tight bass to boot? Not only that, the rich mid-bass and pleasant overall tone completely eradicated my expectations of dryness. They are indeed quite a bit more expensive, but I think they completely justify the price delta (5,000 NIS on paper, probably less in practice).

Bottom line: a pair of ProAc Studio 140 speakers will soon find its new home in my apartment.

As an unexpected bonus I was also provided the Paradigm DSP-3400 subwoofer for testing at home. When the sub was initially demonstrated for me at Stereo-Star, Rami (the sales person who was conducting the demonstration) walked in the room with a gargantuan cardboard box, and with a smile on his face told me "not to get excited, the sub is smaller than the box." Although technically true, the statement hadn't prepared me for just how amazingly big this subwoofer is -- a 14" woofer with two 6" ports underneath:

This subwoofer was my companion for the holidays and indirectly thought me a couple of tricks on how to reinforce cupboard hinges. I'm fairly a bass aficionado, particularly when it comes to home theater, and this little monster went above and beyond my expectations. After returning the demo model I went to a short demonstration of another subwoofer, the B&W ASW610, a little monster in its own right; the problem was that compared to the gut-wrenching effect produced when the DSP-3400 comes to life this was simply an underwhelming experience. Conclusion 1: I should sample equipment in order of ascending cost, not the other way around. Conclusion 2: I'm buying the DSP-3400. Its MSRP: about 7,500 NIS.

The demonstrations were held courtesy of Yaki of Z-Max, Rami Kasher of Stereo-Star, Zohar of Armageddon Audio and Eyal Por of Audio Fidelity. I heartily recommend to anyone looking for similar equipment to contact them.

And for those of you with me so far, here's a bonus kitty:

Sorry for the downtime and the lack of e-mail response over the past couple of months; I've had a ridiculous lot of problems with my previous web host (UCVHOST), among them:

• They switched servers on me with no notification or announcement;
• This caused a reset of both my mail forwarding settings (causing numerous mailbox outtages for several days each), my ASP.NET settings (which caused dasBlog to stop functioning) and the directory security settings (which disallowed comments, trackbacks or even blog posts);
• They blamed me for the e-mail problems because supposedly I've changed the MX records for my domain to point to a different provider. This is ludicrous because the domain NS was under their control;
• And the final nail in the coffin, they decided on a new policy that disallows forward e-mail to a GMail account (which I use as my primary mail provider).

In other words, what the hell? My recommendation is strictly to stay away from UCVHost.

I was looking for a new host for quite a while, and since I was in a pinch with no time for real research I decided to put my faith in my web registrar GoDaddy. To make a short story shorter, the setup was quick and painless and I got everything working within hours. GoDaddy's web-based control panel is exhaustive and usable, and everything "just worked" from the first instant -- which is the way I like it, really. So now I'm with a new host and can peacefully blog again.

Update: Link to the Omea wiki page with more details.

My favourite RSS aggregator, newsgroup client and all-around cool application I always have in the background is JetBrains' Omea Pro. While I don't use those features, Omea also supports aggregating and searching Outlook mail, contacts and a host of other connectors. Now that it's open source there's even be room for improvement, so those of you who do not like online aggregators like Google Reader would do well to go to the Omea site and check it out!

Among a lot of other tools and technologies we use Lucene here at Delver. Lucene is a very mature, high-performance and full-featured information retrieval library, which in simple terms means: it allows you to add search functionality to your applications with ease. I could spend an entire day talking about Lucene, but instead I'd like to show you how to write scripts for Lucene with Python on Windows (and thus differs from Sujit Pal's excellent instructions on the same subject matter).

First, make sure you have your Python environment all set up -- at the time of writing this I use the Python 2.5.1 Windows installer. Next you'll need Python bindings for the library, so go and grab a copy of PyLucene; to make a long story short I suggest you use the JCC version for Windows which is downloadable here. Grab it and install it, no fussing necessary. Finally you'll need the Java VM DLL in your path, and this depends on which type of JRE you're using:

• If you're using the Java JDK, add the mental equivalent of C:\Program Files\Java\jdk1.6.0_03\jre\bin\client to your path;
• If you're using a JRE, add C:\Program Files\Java\jre1.6.0_03\bin\client to your path.

(As an aside, you should probably be using %JAVA_HOME% and %JRE_HOME% and indirecting through those.)

Now you can quickly whip down scripts in Python, like this one which took about two minutes to write:

#!/usr/bin/python
#
# extract.py -- Extracts term from an index
# Tomer Gabel, Delver, 2008
#
# Usage: extract.py <field_name> <index_url>
#

import sys
import string
import lucene
from lucene import IndexReader, StandardAnalyzer, FSDirectory

def usage():
	print "Usage:\n"
	print sys.argv[ 0 ] + " <field_name> <index_url>"
	sys.exit( -1 )

def main():
	if ( len( sys.argv ) < 3 ):
		usage()

	lucene.initVM( lucene.CLASSPATH )

	term = sys.argv[ 1 ]
	index_location = sys.argv[ 2 ]
	
	reader = IndexReader.open( FSDirectory.getDirectory( index_location, False ) )
	try:
		for i in range( reader.maxDoc() ):
			if ( not reader.isDeleted( i ) ):
				doc = reader.document( i )
				print doc.get( term )
	finally:
		reader.close()

main()

Note the VM initialization line at the beginning -- for the JCC version of PyLucene this is a must, but even like so using the library is extremely painless. Kudos to the developers responsible!

I just encountered a really weird issue with Windows Vista, where an external Western Digital hard drive (an older My Book 250GB) would show up as a mass storage device, but was not allocated a drive letter and was basically inaccessible. The weirdest thing is that the USB "Safely remove USB Mass Storage Device" icon did show, except with no drive letter.

Anyway the way to deal with it was to fire up the Device Manager (Start->type in Device Manager) and double-click the external hard drive:

Then go to the Volumes tab and click on Populate:

If the volumes show up, you're good to go.