Creating an Intelligent Website Search with Vectors and AI

Overview

Website search features are invaluable tools for users, offering a quick and efficient way to find relevant information. However, implementing a robust search functionality has traditionally been challenging, especially for static websites not connected to a database. Poorly implemented searches can result in a worse user experience than not having one at all, often due to issues like spelling correction, disambiguation, and relevance.

In this post, we'll explore how leveraging vectors and Language Models (LLMs) can streamline the process of creating a powerful and user-friendly search experience on any website. We'll cover both basic and advanced implementations of this idea.

Vector Similarity Search

What is a Vector and How They are Created

Vectors are mathematical representations of objects in a multi-dimensional space. In the context of search, these vectors capture the semantic meaning of words and phrases. For instance, similar words are represented by vectors that are close together in this space.

Let's take a very high-level view of how this works with similar words:

from scipy.spatial.distance import cosine 

vector_word_apple = [0.4, 0.9, 0.2] 
vector_word_fruit = [0.3, 0.8, 0.1] 

# Calculating cosine similarity
similarity = 1 - cosine(vector_word_apple, vector_word_fruit) 

print(f"Similarity between 'apple' and 'fruit': {similarity}")        

Vector Database

A vector database stores these representations efficiently, allowing for quick and accurate similarity searches. It is tailored for scenarios where capturing and leveraging semantic relationships between data points are crucial, making it a powerful choice for tasks like intelligent search and recommendation systems.

There are a few different types of vector databases, but for our use case we'll use Weaviate (https://weaviate.io/).

How to Implement a basic site search with Weaviate as the Vector Database

Scraping the website

To implement vector-based search, first scrape the website and save all the content from each page. You can optionally removing unimportant sections like headers and footers. For text-heavy pages, consider chunking the data out into several sections.

import requests
from bs4 import BeautifulSoup

url = 'https://example.com'  # Replace with the URL you want to scrape
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

# Extract the content you're interested in, for example, paragraphs
paragraphs = soup.find_all('p')

# Print or process the scraped content as needed
for paragraph in paragraphs:
    print(paragraph.text)
        

Next, vectorize the data using a suitable algorithm; in this example, we'll use the spaCy library, but Weaviate works well with the OpenAI API.

import spacy

# Load the spaCy model
nlp = spacy.load('en_core_web_sm')

# Process the text
text = " ".join([paragraph.text for paragraph in paragraphs])
doc = nlp(text)

# Access the document vectors
vectors = doc.vector

# Print or use the vectors as needed
print(vectors)
        

Storing Data in Weaviate

Store the vectorized data in Weaviate along with relevant metadata such as the URL and page title.

import weaviate

# Initialize Weaviate client
client = weaviate.Client("https://localhost:8080")

# Create a class for webpages
webpage_class = client.schema.create_class("Webpage")

# Add a property for vectorized text
client.schema.add_property(webpage_class["class"], "vector", "vector")

# Store data
data_to_store = {"vector": vectorized_text, "url": "example.com", "title": "Example Page"}
client.data.create("Webpage", data_to_store)        

Implementing Search

To implement search, take a user query, vectorize it, and return the a specific number of documents closely related to the query.

user_query = "Python web development"

vectorized_query = vectorize_text(user_query) 

# Perform a similarity search in Weaviate 
results = client.query.search({"vector": {"cosineSimilarity": {"vector": vectorized_query}}})        

One Step Further - Using Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is a natural language processing (NLP) framework that combines elements of both retrieval and generation models to enhance the capabilities of language models. RAG aims to address the limitations of traditional language models by incorporating a two-step process: retrieval and generation. In the retrieval phase, a search is performed over a pre-existing knowledge base, in this case our web pages, to find relevant documents or passages. These retrieved contexts are then used to augment the language model's understanding before it generates a response. This approach enables the model to leverage external information, making it more contextually aware and capable of producing more accurate and contextually relevant responses.

RAG is particularly beneficial for tasks that require a deep understanding of specific domains or topics, such as question answering and content generation. By incorporating information from the website through retrieval, the model can access the specific context and return rich outputs.

# Code example using OpenAI API for generative search 
# (API key and endpoint setup required) 
import openai openai.api_key = "your_api_key" 

generate prompt = "Summarize this content into two sentences: {content}"

response = (
  client.query
  .get("Webpage")
  .with_generate(single_prompt=generate_prompt)
  .with_near_text({
    "concepts": ["SEARCH QUERY HERE"]
  })
  .with_limit(3)
).do()

print(json.dumps(response, indent=2))        

The output to the user would be specific answers to their search as well as the page location to find more information.

Conclusion

By harnessing the power of vectors and AI, implementing an intelligent search feature on a website becomes not only feasible but also efficient. We've explored the basics of vector similarity search using Weaviate as a vector database and delved into advanced techniques like Retrieval-Augmented Generation. These approaches not only enhance user experience but also open up possibilities for further innovation in website search functionalities.