Two Steps Forward, Three Steps Back, Warp Speed Mr. Sulu

One of the very first programming challenges I tackled as a young person in the computer industry was implementing a fast search algorithm. Since computers have existed, there has always been a need to search large, constantly-changing data stores. In my case, the challenge was to search a historical list of retail sales transactions. This was a file that grew daily and went back to the beginning of the implementation of the software. If I’m remembering correctly, this file was 4 or 5 megabytes and growing. Today this sounds like nothing, but at the time I was working with a 386 computer running at 16 Mhz with, at most, 2 megs of RAM that was acting as a server for at least five terminals. The idea was that one of these terminals should be able to initiate a search of this database. On a good day, these terminals had 300 or 400K of RAM to work with. With the standard tools available at the time–dBase, Clipper–this search could be accomplished in maybe 5 or 6 minutes. To solve the problem, I learned assembly language so that I could forgo all the overhead of database programs and compilers and search the file directly. I got the search time down to 5 or 6 seconds. Theoretically, I was forcing the system to search at the limit of the hardware. If you ran this program today, it would have one of two outcomes: either it would search 5 gigabytes of unindexed data faster than any program commercially available or it would crash your Windows machine catastrophically. If pressed, I can probably come up with the application and source code. The lesson I learned was a simple one: applying superior skills and technology, the same effort will get you 60 times the outcome. That’s pretty much been the story of my life ever since.

Fast forward to today where Microsoft Outlook takes several minutes to search my message archive for a keyword. That’s if I only want to go back a year. If I want to go into my actual archives, spanning tens of thousands of messages and several years, I’m back up around the five or six minute mark. Plus a lot of user-interface hassle. Enter Google Desktop Search Version 2.0. First of all, let me say, this is the first computer program I’ve ever seen that is so powerful it frightens me. This is science-fiction-level computing. Secondly, let me say that if you aren’t afraid of what’s on your computer and you want to be able to search the entire thing instantly (including Outlook) you absolutely must install this program. Thirdly, let me say, whatever you do, if you value your time and your attention span, do not download this program. You cannot possibility imagine how addictive it is to have the entire contents of your computer instantly accessible, the RSS feeds for all the blogs you read running in a constant ticker, a slide show of all the images on your hard drive constantly running, a section of news headlines tuned to your reading preferences, a scratch pad of your most recent thoughts, whatever happens to be going on in your gmail account, the current weather where you live (it just knows), and everything else in the fucking world right in a little bar on the right of your screen. Your life will grind to a halt and you will worship Google. The difference between running this program and not running it is the difference between turning your computer on and unplugging it and taking it into the shower with you. Computing, as you know it, is over.

But here’s the question: why the hell didn’t Microsoft do this back in 1990 when I did? They had people who already knew assembly language! They’ve had the technology all along to give us instantaneous comprehensive search and they’ve squandered it. This is what Google has accomplished. With only surface access, no hooks into the operating system, no access to the origins of the data at hand (that is, we didn’t supply meta information up front), not even massive outside computing resources, they’ve taken this machine, one I paid hard-earned money for, and made it do exactly what it should have been doing all along. They took a seemingly insurmountable challenge and made it into a trivial operation. That is not only the definition of success, but the definition of genius.

One thought on “Two Steps Forward, Three Steps Back, Warp Speed Mr. Sulu”

  1. Just a follow-up to say that I’ve now uninstalled Google Desktop search. I’m not sure if it was the search itself or one of the plugins, but it ate huge chunks of memory (200megs at times) and slowed/crashed my system consistently. This makes me wonder what, if any, sandboxing Google does for the plug-ins. They’re pretty vague about that and I didn’t dig into the SDK to find out. I’ll check it out again in the next revision. And maybe I’ll have a faster computer by then.

Leave a Reply

Your email address will not be published. Required fields are marked *