A Beginner’s Guide to Vector Databases - With Example

In the first part (ref link), we explored the foundational concepts of vector databases, understanding their key features, advantages, and how they differ from traditional relational databases. In this second part, we'll deepen our understanding through a practical implementation, demonstrating how vector databases can power a system such as a movie recommendation engine.

Problem Statement

We want to create a movie recommendation system that can understand and process natural language inputs from users, like "Cartoon Movies", and provide relevant suggestions. The system should be able to go through list of all movies and deliver quick, accurate recommendations.

Why Vector Databases?

While movie data used in this example is largely structured and can be efficiently handled by traditional databases, vector databases offer unique advantages for recommendation systems, in a conventional database, searching for a movie would involve:

• Keyword matching across multiple fields (title, description, genre)
• Complex joins to link related data (actors, directors, franchises)
• Potentially slow full-text searches
• Difficulty in capturing semantic similarity beyond exact matches

This process becomes increasingly complex and slow as the database grows.

Vector databases simplify this by:

• Representing movies and queries as vectors, capturing semantic meaning
• Enabling efficient similarity searches in high-dimensional spaces
• Maintaining performance even with millions of entries

This approach allows for more intuitive, faster, and more accurate recommendations, especially when dealing with natural language inputs and concept-based searches.

Example 1:

Example 2:

Let's Understand Vector Similarity Search

Vector similarity search is at the core of many modern applications, from recommendation systems to image recognition. The concept involves representing items (like movie plots, user preferences, or images) as vectors in a high-dimensional space. The goal is to find vectors that are close to a given query vector, indicating that the corresponding items are similar.

How Does Vector Similarity Search Work?

1. Vector Representation: Data is converted into vector representations using pre-trained models like BERT, Word2Vec, or Sentence Transformers.
2. Indexing: Vectors are organised using indexing structures (like trees or graphs) to enable efficient search without having to compare the query vector to every vector in the dataset.
3. Distance Metrics: Various metrics, such as Euclidean distance, cosine similarity, or dot product, are used to determine the similarity between vectors.
4. Querying: The query vector is compared against indexed vectors, and results are ranked based on their similarity scores.
5. Post-processing: Additional filtering or ranking may be applied to refine the search results according to specific user preferences or requirements.

The Use Case

For our movie recommendation system, we start by preparing the data. We'll use a movie dataset from IMDb, which includes features like genre, overview, director, and starring actors. (ref link)

Generating Embeddings

The next step is to generate vector embeddings for each movie. We use a pre-trained Sentence-BERT model to encode the combined features into high-dimensional vectors that capture semantic meaning. These embeddings represent each movie in a way that allows us to compare them based on their content, rather than just keywords.

Measuring Similarity

Once we have the embeddings, the next challenge is to measure how similar two movies are based on their embeddings. Cosine similarity is particularly well-suited for high-dimensional data like embeddings, where the magnitude of the vectors can vary widely. By focusing on the angle between vectors, it ensures that the similarity is purely based on the content, making it ideal for our recommendation system.

Here's how we implement cosine similarity:

The Role of FAISS

For small datasets, computing cosine similarity directly is feasible. However, when dealing with large-scale datasets, the process can become computationally expensive. This is where FAISS (Facebook AI Similarity Search) comes into play. FAISS is a library developed by Facebook AI Research that efficiently searches for similar vectors in large datasets. By using it, we can quickly retrieve the most similar movies based on input query, even when dealing with large datasets.

Here's how we use FAISS in our system:

Alternative Approaches to Similarity Search

While we focused on using cosine similarity in this article, it's important to note that other approaches could be equally effective, depending on the specific use case. For instance, Euclidean Distance and Dot Product might be more suitable for certain types of data or applications. Similarly, when it comes to vector similarity search, FAISS is just one of many tools available. Annoy, Milvus, and others offer unique features and optimisations that could be better suited for different scenarios.

Efficiency of Precomputed Embeddings

In this example, I precomputed the vector embeddings and stored them in a PKL file to enhance the efficiency of the recommendation system. This approach is particularly useful when dealing with static datasets where the content doesn't change frequently. Vector databases like Milvus, Chroma, and FAISS are designed to handle real-time data, making them a better choice for applications where data is continuously updated.

You can explore the movie recommendation system here:

• Try the App - Just Another Movie Recommendation System

Conclusion

In this article, we explored how to implement a movie recommendation system as a practical way to better understand vector databases and related concepts. By combining different similarity measures, we demonstrated how to capture both lexical and semantic similarities, providing users with relevant and personalized results.

Whether you're aiming to build a recommendation system, an image retrieval application, or any other search-driven solution, vector databases offer a scalable and efficient approach that goes beyond the limitations of traditional databases.