The AI Revolution - time to throw away your search engine (again)!

I've been working in search for nearly 25 years, first with Dr. Martin Porter at Muscat in the late 1990s, then as a co-founder of Flax where we helped clients ranging from the British Government to the NHS to major newspaper publishers build better search, then at OpenSource Connections where we focus on the whole picture, developing our client's search overall maturity with a combination of training & consulting. During this time I've seen a good few 'revolutions' in search that promised to change everything: BM25 and probabilistic search engines, the rise of open source, commoditized search platforms that everyone could download and use; Big Data and even bigger promises; Insight Engines (I'm not sure anyone ever explained to me how you quantify 'insight') and now AI, not-actually-personified-yet by ChatGPT and its cousins, powered by Large Language Models (LLMs).

My personal strategy for dealing with these revolutions has been to surround myself with people far cleverer than I am, in the hope they can explain it to me sufficiently that I can then explain it to others (as my role has moved on from the purely technical to management, sales and marketing). My hope is that I'll be able to cut through the hype, show what actually works and what doesn't and help people decide what will benefit their business as opposed what might be an interesting but expensive research project that doesn't produce any useful results. This is why I love organising and attending search events like Haystack .

At the moment we're right in the middle of one of these hype cycles. If certain commentators to be believed (and I'm being kind to some of them here, having a .ai after your Twitter handle doesn't make you any more of an authority than the .eth or .web3 did a few months ago) we might as well throw away all our search engines today. Google is doomed ; Microsoft are back and they're winning the search race; Skynet is just around the corner. Others have a more sober take: these chatbots are just stochastic parrots , blindly repeating what they've been trained on - and we don't actually know what they've been trained on or the inherent biases as a lot of the technology is closed source (OpenAI being not as Open as it was meant to be) - so why should you trust them? There are calls for regulation, Open Letters signed by many people (and some of these people are even real) and politicians looking for emotive buttons to push .

In my world of search there's a great deal of patient sighing from technical folks as they explain no, although some of these techniques are useful, it's not all quite as clever as all that, and some considerable bandwagon jumping from marketing teams who get to publish yet another picture of a beautiful female android to illustrate their AI articles. It's remarkable how many 30-year-old NLP techniques have been hastily rebadged as AI and the demo factories are operating at high speed (although remember the phrase 'the demo is never the product'), while some AI companies gain huge valuations . No-one wants to miss the AI train that's steaming into the future, even if we're not really sure of its destination, the price or if there's a bridge out ahead.

So is this an actual search revolution this time? Frankly, I have no idea, although past experience has led me to assume that some of the new ideas will be useful, in some places and situations, while a lot else will be just fluff and nonsense. It's all very exciting though and it's definitely time to get those clever search folks together again so I can find out what it's all about - luckily in a few weeks we'll be doing just that at our search relevance conference Haystack US 2023 in Charlottesville, VA. As our keynote speaker Trey Grainger writes: "The search relevance landscape is rapidly shifting...public search engines are now rushing to integrate models like chatGPT directly into the search experience...As search relevance practitioners, it’s important for us to know how these technologies work and how to best integrate them into search experiences to drive accurate, relevant results."

The revolution will be televised (well, you can join online ) or you can come and meet in person some of the people leading the charge. If you need to know whether to throw away your old search engine and how to jump on the AI train, this is your chance.

I hope to see you there!