The rise of search as navigation

Arthur C. Clarke isn't credited with creating the first search engine; that honor goes to a student at McGill University during the infancy of the Internet. Clarke, however, certainly had a handle on what we're still looking for in his HAL 9000 creation—perhaps the most famous example of artificial intelligence of its era. If you'll pardon the quick trip back in movie history, HAL 9000 was the spaceship Discovery's paranoid central computer that attempted to kill the crew in the movie classic 2001: A Space Odyssey.

Perhaps overshadowed by its homicidal tendencies, HAL's genius lay in its role as the primary link to the ship's information stores and in its relevant answers to standard questions ("natural language search"). Yet almost forty years later, despite advances in search technology and with multi-billion dollar gorilla Google lending search high visibility, many companies neglect to integrate even the most basic search into their site architecture. Putting aside the as yet unattained dream of true natural language search, why are so few companies taking advantage of existing search functionality and thus turning away potential users, customers, and sales?

In the heyday of the dot com boom, Web site navigation was seen as top-down and hierarchical due to technology restrictions and, in a vicious cycle, user expectations. Search was a marginalized function not expected to deliver relevant results—in fact, users presented with even a single relevant search result rejoiced (some even saw a great light and heard doves singing).

One of the many problems with relevancy was that Web searches directed users to general areas rather than specific content items. Search operated on a conceptual model of a house: arriving at a Web site meant walking through the front door (homepage) at risk of starting off in some storage closet in the basement with no context or way out. To be fair, that was seen as okay, even expected, and was the result of search catering to the Web site architecture popular at the time. Today, however, with better search technology, better-architected Web sites with more context and accessible distinct content, user expectations of search are increasing and companies are left trying to catch up.

With search engines of all sorts becoming increasingly savvy, your navigation practices, indeed even your content planning needs to take powerful search into account. Instead of being forced through your homepage, users can and will be consistently sent deep within your site to find content truly relevant to their search. Sites no longer mimic the old brick and mortar structure with a front door; they've evolved into virtual balls, able to be spun and manipulated so that the entry point can be anywhere to any type of content. This is particularly relevant for internal Web site search, which was more affected by the "front door" Web site model than external search—at least the latter had several front doors from which to choose.

Research from Jupiter and Forrester has found that 90 percent of companies report search as the primary means of navigation on their site[1] and 82 percent of visitors use site search to find the information they need.[2] 85 percent of site search users are not satisfied with search results.[3] Those are shocking numbers—so why are businesses (some Web leaders excepted) still not looking at effective solutions to this obvious problem and opportunity for competitive advantage? The expectation of having to pay a hefty amount is usually the first stumbling block. Yes, three or more years ago internal site search would have set back large companies tidy sums of money and been out of reach for most small companies, but many internal search tools have been commoditized, with prices falling (sometimes to zero) as a result. Shoestring startups or that mom-and-pop online recipe book can now afford the benefits of a powerful search engine—in fact, they can't afford to pass up those benefits if they want to compete successfully for users' online time.

One free example of a powerful internal search engine is IBM and Yahoo's free version of the robust OmniWare tool. After testing this version, you'll have to admit to yourself (as I did) that it's one heck of a product, particularly given its lack of cost. Coveo, Thunderstone, Autonomy, DT Search, Google, and IBM all offer relatively low-priced search engines for smaller clients who are looking to provide a great user experience. For larger companies, again, there are no shortages of quality search engines that fulfill content discovery needs at varying budgets.

Return on Investment. Without it, businesses don't exist. So what is the ROI for an internal search engine and how can I measure it? What you can monitor is site traffic, sales leads resulting from search engine use, and the use of your search tool itself. What you'll find and what many consultancy firms and business owners have also found, is that with a tuned, correctly configured, internal search engine, users will find far more relevant results, satisfying users and making them more likely to use your company's services.

Your internal search engine will also show you exactly – in their words – what your customers are looking for on your site. That wealth of information will come from the log of keywords they typed in. Search statistics will also show you what users aren't finding and what happens after an unsuccessful search. Usually the users will leave, but sometimes they'll give you alternate phrases or other clues to areas they expect will contain what they are looking for on. A good business owner can capitalize on this information by making adjustments to his site, e.g., improving navigation labels or pathways or grouping products and services differently. The goals are standard: greater conversion, time on the site, pageviews, and user satisfaction.

A primary purpose of your Web site is to get your information to the users' eyeballs as quickly as possible. The old multi-step process of wading through layer after layer of navigation is inefficient for most applications, and Web sites that still use it are beginning to show their age. As discussed above, powerful new internal search technology can dump your potential customers into your Web site wherever it's relevant, regardless of section, layer, or other elements about which users are mostly indifferent.

This laser accuracy brings up the challenge of providing context and intuitive pathways back from the potentially buried content your internal search provided. One particularly good example of addressing this challenge is using relevant suggestions of other content or items in which the user might be interested. Leading e-commerce sites like Amazon.com have adopted this, as have content-based sites like the Washington Post Web site. If the user arrives directly at a desired page and is confused by or not interested in what surrounds the desired item, an exit often follows. Providing relevant related items, however, can lead to increased interaction and help users discover what they missed by drilling straight to their result.

We want our information like we want everything else: at our fingertips and fast. While a search engine as brilliant (and sinister) as the Hal 9000 isn't quite on the market yet, many intelligently engineered and well packaged search engines are. As many sayings go, you could have the best product or service in the world but never make a sale or gain a user if no one knows about it. And with user preferences shifting away from browsing a hierarchical tree and toward finding exactly what's desired, much more of the Web is in danger of remaining unknown and unexplored. Perhaps it's time to investigate internal site search as one way of letting users know exactly what you have to offer.