How To Improve Your Search Experience

As a high-class consultant, I spend a lot of time helping companies improve their search experiences. Every company has a unique set of concerns, but I’ve seen a number of common themes.

Does search contribute to your company’s core business? If so, please consider this top-10 list a free sample of the advice I offer to my clients.

1. Invest in query understanding and query rewriting. Search engine developers invest significant effort into improving ranking. Query understanding is about what happens before the search engine scores and ranks results —it’s the process of establishing the searcher’s intent. Query rewriting automatically transforms search queries in order for them to better represent that intent. Investments in query understanding and rewriting often provide larger and faster returns than efforts to improve ranking.
2. Focus on your head queries before chasing your tail. Most search engines have a power-law distribution: a relatively small set of unique queries accounts for a sizeable fraction of traffic, while a long tail accounts for the rest. Hopefully you’re already seeing great performance from your head queries. But if not, then make it so. Investments in improving performance for your head queries generally provide disproportionate returns.
3. Size matters?—?result set size, that is. Hopefully you already track queries that return no results. But be sure to track queries that return very small or very large result sets. The former indicate recall problems, while the latter indicate precision problems. Search for Goldilocks.
4. Capture head queries using autocomplete. Your head queries?—?whether the top thousand or top hundred thousand?—?should be handled entirely by your autocomplete (aka typeahead) system. Autocomplete not only reduces searchers’ effort, but also guides searchers towards good queries, helping to ensure a successful search experience.
5. Pay attention to the overall search experience. When you’re working on hard problems like ranking and query understanding, it’s easy to lose sight of details like the size of the search box and the layout of the search results page. Don’t assume you know what is and isn’t important. Try to treat each decision about your search experience as a testable hypothesis.
6. Measure everything, but keep your metrics simple. As Lord Kelvin said, you can only improve what you measure. Instrument your entire search stack so that you can see what is and isn’t working. But keep your metrics simple and interpretable, e.g., click-through rate on the first page. Search evaluation is a rich enough topic to fill a book, but you should resist the urge to use complex metrics. Complex metrics may feel sophisticated, but it’s harder to derive actionable insights from them.
7. Use a combination of analytics and human evaluation. Analytics are cheap and scalable, and they serve as the most robust indicators of how searchers are experiencing your product. But human evaluation is sometimes the most cost-effective way to measure quality issues that elude analytics. But don’t go overboard with your evaluation tasks?—?even humans have their limitations.
8. Analyze failures. Look at searches that return no results, and figure out why. Look at common search queries that underperform as measured by clicks or conversions, and generate hypotheses based on your observations. If you can classify queries into distinct classes, look at classes that underperform?—?or queries that underperform within their query class. Failures are often your best sources of insight, so pay attention to them.
9. Treat data issues like production issues. At many companies, publicly visible site issues have a rapid escalation path, but data issues can languish on the backburner. Search quality, however, is extremely dependent on robust and timely data, and in some cases that data directly drives the search experience. If you can’t measure search behavior and trust the data you collect, you won’t even know where you need to improve your search experience, let alone how to improve it.
10. Finally, dream big, but execute incrementally. It’s easy to dismiss incremental improvements as boring or unimaginative. But moonshot visions don’t materialize overnight. Almost all innovation happens one step at a time, through a series of controlled experiments. Dream big, execute incrementally.

If you’re already doing all of this, then congratulations: you’re way ahead of the pack. If not, then you might want to revisit your roadmap.

And if search experience is critical to your business, you may want to engage a high-class consultant.