AI-powered search: From keywords to conversations

By now, everyone’s had the opportunity to experiment with AI tools like ChatGPT or Midjourney and ponder their inner workings. They’ve seen for themselves the remarkably fluid, sophisticated, and sometimes silly responses they produce from simple queries. The result is a torrent of ideas about what AI means and where organizations can take it.

The new eBook —?From Keywords to Conversations: Get Ready for AI-powered Search?— explains the significance of recent advances in AI-powered search in easy-to-understand language. As we share in the eBook, thanks to quantum leap advancements in Artificial Intelligence (AI) and Natural Language Understanding (NLU), search engine technology is also undergoing major changes. People and businesses will be able to discover things and information more naturally, faster, and with more accuracy than ever before, and with remarkable computational ease.

If you’re new to AI search, or just want to have an understanding of where AI technology is coming from, this blog is a good place to start.?

Language understanding has come a long way

Search has been?around for decades. Built on keyword matching technology, search worked fairly well for rudimentary queries. However, it would still struggle with more natural language queries — often returning few or sometimes no results much to the consternation of customers.

Language is naturally hard for computers to process and understand. Words can have multiple meanings and those meanings shift depending on context. Words like “it,” “that,” and “this” are constantly in flux and can refer to many different things over the span of a few short phrases.

For people, a phrase like “Get that?” at the end of a paragraph is easy to grasp. Computers have a much harder time solving for this kind of ambiguity — what might “Get that?” refer to? They have trouble filling in the blanks, reading between the lines, and making inferences.

The subfield of AI that works on this problem is called?Natural Language Processing (NLP). The subfield that focuses on how computers understand human generated text is called Natural Language Understanding (NLU). The joint fields of NLP/NLU have been working on these problems for decades – making impressive forward strides, then always hitting a wall.

With new AI models, computers now appear to genuinely understand natural human text and can generate responses that seem entirely fluid and comprehensible, even thoughtful. The joint fields of NLP/ NLU have just plowed through their biggest wall yet. The key to that success is machine learning and Large Language Models (LLMs).

Transformers, like the kind used in ChatGPT, took it to the next level. LLMs can process input text and respond in real time by making astoundingly accurate “guesses” about what the next word is likely to be. It doesn’t actually “understand” the meanings of input text, but rather, it performs complex statistical calculations based on petabytes of training examples to assess likely relationships between words in this specific case.

Vector search and intent intelligence

The development of transformers led Google and AI to another critical innovation: vector search technology, a method for connecting text inputs with objects in an index to vastly improve information retrieval in ways that mirror human understanding.

Vector search?is a huge advance over keyword matching. It lets computers map the semantic relationships between words using statistics and probabilities. By making connections between words, it understands what users mean and captures valuable “intent intelligence.”

Vectors are key to helping computers understand natural language. For example, how would a computer know that a computer user who types “espresso with milk thingy” means an appliance, and not coffee or grocery items? Vectors help by plotting and clustering word tokens in multiple dimensions (n-dimensional space) representing any number of variable attributes. When search queries are made, they’re converted into vectors and compared to other vector representations by computing the distance between a range of attributes. The closer the match, the more accurate the result.

Now, search algorithms know so much about the semantics of natural language that they can pinpoint exactly what users are seeking even from remarkably varied and fuzzy input. Users don’t need to work as hard to define what they’re after. Powered by vector search, machines understand user intentions and meet them halfway – much like human partners in conversation do. The result is a watershed moment in AI, as computers now capture?intent intelligence.

Vectors can greatly improve search relevance, and they can be assisted with another AI technology —?reinforcement learning?— for even better results that improve over time with enough data. As users click results, or trigger other events such as conversions or purchases, the AI model gets smarter and boosts better results to the top.?

The next step

Enabled by LLMs and intent intelligence, chatbots and virtual assistants will be able to grasp exactly what site visitors are looking for more quickly and with more precision than ever before. The potential for site search is truly out of this world.

To learn more, download this free eBook today,?From Keywords to Conversations: Get Ready for AI-powered Search.