From Vector databases to hybrid RAG for Enterprise Gen AI

Vector databases have become synonymous with Retrieval Augmented Generation (RAG) for LLM applications in recent discourse. The embeddings approach that underpins these databases makes a lot of sense for a general-purpose Gen AI solution like ChatGPT; vector matching provides a broad, content-agnostic solution for identifying relevant information about any conceivable topic.

The use case for internal AI agents within an Enterprise is very different, in three important ways:?

1. Data: The data context is usually topical, focused on a specific industry vertical and company, along with its products, solutions, suppliers, partners, etc. There is also a great deal of structured data available (e.g. sales or purchase transactions) which can be looked up directly using keys and indexes.
2. User segmentation: Most internal users belong to one of a small number of user types, broadly aligned with team and function; each user type has a small number of use cases that are most frequently used. These use cases however can be quite complex and sophisticated.
3. Success parameters: Accuracy is the most important (except requests where creativity is the goal), followed by completeness and speed of response. It’s better for the AI to explicitly say that it does not know the answer than to make something up.

As foundation models increasingly become commoditized and prompt optimization gets systematized, e.g. using DSPy, the value differentiation of an AI system will shift from its inference engine to its “data power”, i.e. how well it understands the user request and locates relevant data across disparate data sources to provide in context. Equally important, it needs to seamlessly absorb new data as it is created so that the AI system continues to “learn” over time.

On the technology side, data within an Enterprise is typically distributed across many different data sources: transactional SQL systems, NoSQL/object databases, graph databases, file systems, APIs, etc. In addition, domain-specific taxonomies and rich metadata can add semantic depth to the search.

At ADS, we’re designing a hybrid RAG system that’s carefully designed to fetch data from this rich variety of data sources using multiple technologies (SQL queries, knowledge graphs, keyword search, vector search, etc.) and then to stitch the results together appropriately to feed into the LLM prompt.?

This sophisticated RAG system then becomes the backbone of the AI system, the growing knowledge base that enables AI agents to provide correct answers.