Building your enterprise grade GenAI app in minutes!

Whether you are in the data space or not, you sure have heard enough and more about "Generative AI". Speed & Precision are extremely key for any effective Generative AI chat application. RAG - Retrieval Augmented Generation is at the heart of both of these factors.

When you want to build a GenAI application that specialises in YOUR data, RAG plays a key role.

Think of a RAG Enhanced (RAGE) foundational model as a Subject Matter Expert and a simple foundational model as more of a generalist.

How do you make your foundational model RAGE? (How do you make your generalist an SME? )

Well you supply the open source foundational model with context-specific documents of your chosen domain.

So when someone asks a question specific to your context, the model is able to delve into YOUR documents and find a highly precise answer to the question.

Great! But what is the underlying technology that enables this?

"Vector Search" has entered the building.

How does Vector Search help in leveraging YOUR documents to return precise answers quickly?

First, you build a high dimensional vector space where all these documents are mapped/represented.

Then you represent the question as a vector in the same high dimensional space.

Finally, you pick the vectors that are closest to your question's vector to get the documents that are most similar to your question and therefore contain the answer to your question.

?? Wait What??

You want the SME to understand the meaning behind your question (not just search for the words) and find the relevant documents that have your answers.

Vector search does this using Machine Learning to convert your question into word embeddings or a vector.

This vector is a set of numbers that mathematically represent the meaning of your question in a high dimensional space.

So, all the SME needs to do is to look at other vectors "close" to your question's vector to find relevant documents that are "similar".

?? You wanna try that again? This time, try to make it a bit more meaningful?!

Think of the vector space as a 2 dimension space for a moment.

Let's go back a few years, remember you learnt plotting maps with an X-axis and a Y-axis. Every point on the map had an address (X,Y).

If a Pet company were to build a vector search index with its documents, then you would find words like Dog and Puppy located close to each other but far away from words like Cat and Kitten (which would themselves be closely-located).

So when you submit a question about puppy adoption, thanks to vector search you would be able to get a response from a document that details adoption procedure for dogs.

Why do you need a high dimensional space?

Well, higher the dimensions, higher the precision. Consider this Gartner Magic Quadrant as an example.

Question: Which company has the best Cloud Database Management System? (Exclude companies that are cloud vendors)

If you were to only use 1 dimension (Ability to Execute) to answer the question you would incorrectly respond with InterSystems, MongoDB, Snowflake and Databricks.

Where as when you add the 2nd dimension (Completeness of Vision), Databricks emerges as a clear winner and you will have a highly precise response!

Ok, but i still don't know how this helps me build my Generative AI application!

Well, by now you know that Vector Search and foundational models with RAG are crucial and why.

The Databricks Data Intelligence Platform allows you to build "vector search" indexes for enterprise data stored in your data lakehouse with the click of a button!

Once the vector index is created, you can then combine this with a foundational model and serve it using "ml model serving" and start asking questions to this "SME" model. Watch the detailed video here.

To run the demo, get a free Databricks workspace and execute the following two commands in a Python notebook:

%pip install dbdemos        

import dbdemos
dbdemos.install('llm-rag-chatbot', catalog='main', schema='rag_chatbot')