Building Enterprise-Grade RAG Systems With Epsilla
Epsilla (YC S23)
All-in-one platform for building production-ready AI agents powered by your private data and knowledge.
Welcome back to the Epsilla Blog! Today, we’re diving into one of the most exciting and challenging topics in AI: building enterprise-grade RAG systems.
While multiple advanced RAG optimization technologies are discussed in previous blog post series, scaling up to meet the demands of enterprise environments requires combining them into a holistic system. The stakes are higher, the architecture is more comprehensive, and the solutions need to be both scalable and secure.
This isn’t just a conceptual overview—we’re here to show you how Epsilla’s platform tackles real-world challenges in creating robust enterprise RAG systems. From data ingestion and synchronization to advanced retrieval techniques and observability, we’ll walk you through how our tools empower enterprises to unlock the full potential of their data.
Whether you’re a data scientist fine-tuning workflows or a business leader exploring AI solutions, this guide is packed with actionable insights and hands-on strategies to get you started. Let’s dive in and see how Epsilla can redefine what’s possible with enterprise RAG systems!
Case Studies: How Enterprises Use Epsilla
AI Financial Analyst
Financial firms use Epsilla’s AI Financial Analyst to solve problems with manual data analysis. This tool automates the retrieval and analysis of financial reports and news. It reduces costs by 90% and improves efficiency by 10 times, giving financial professionals more time to focus on strategy.
AI Lawyer Assistant
Epsilla’s AI Lawyer Assistant helps legal professionals by analyzing and summarizing legal documents. It works with both public and private databases, saving up to 80% of the time and cost spent on legal research. This tool allows law firms to deliver faster and better results.
AI Customer Support
Epsilla’s AI Customer Support improves the way companies handle customer questions. It provides quick answers, automates responses, and escalates complex issues to human agents when needed. By speeding up response times, this tool boosts customer satisfaction and makes teams more efficient.
Boost Content Engagement
Content creators use Epsilla to increase content visibility and engagement. This tool helps publishers deliver relevant content to their audience by using advanced search and summarization features. It simplifies content discovery and drives more traffic, helping publishers achieve better results.
AI Tutor and Study Buddy
Epsilla’s AI Tutor and Study Buddy make learning easier for students and educators. This tool helps users search and summarize educational materials, creating personalized learning paths. Students can focus on what they need to learn while enjoying an immersive and efficient learning experience.
Personal AI Assistant
Epsilla’s Personal AI Assistant helps users build their own AI tools. It allows people to research topics, answer questions, and automate daily tasks. This tool is perfect for anyone who wants to save time and simplify their workflows.
How to Build an Enterprise RAG System
Access Control
Epsilla’s platform provides detailed access control at both the project and application levels. For each project, administrators can decide who has access and specify the permissions for each user. Similarly, at the application (or agent) level, users can publish applications with varying access levels, offering precise control over who can interact with specific features or agents.
Data Security
Security is a cornerstone of Epsilla’s design. By managing access at granular levels, the system ensures that only authorized individuals can view or interact with sensitive project data. This robust approach minimizes risks such as data breaches and unauthorized access, safeguarding enterprise information.
Personalization and Customization
Epsilla supports personalized user experiences by identifying individual users and tailoring interactions accordingly. The system leverages user-specific chat histories as context, ensuring that agents can recognize who they are interacting with and respond appropriately.
Branding
Epsilla allows enterprises to customize the appearance of their agents to align with their branding. Businesses can adjust colors, logos, and other visual elements to ensure the AI agents reflect their corporate identity. This feature enhances user trust and creates a seamless branding experience.
Input Guardrail
When users interact with the RAG system, it’s crucial to manage the type of inputs being processed. Epsilla’s input guardrails ensure that only appropriate and safe queries are handled, helping enterprises maintain security, compliance, and user trust. Let’s explore how Epsilla applies these guardrails across various scenarios.
Anonymization
Epsilla’s platform can identify and redact sensitive information, such as Social Security Numbers (SSN), credit card details, or other personally identifiable information (PII). If such information is detected in a user’s input, the system blocks the query and ensures no sensitive data is processed or stored.
Restrict Topics
To maintain professionalism and compliance, Epsilla can filter and reject queries related to inappropriate or sensitive topics. For example, the system denies responses to questions involving political controversies, criminal activities, or content that violates enterprise guidelines.
Restrict Language
Epsilla ensures that inputs adhere to language preferences set by the enterprise. For example, the system can restrict interactions to English or another specified language, reducing errors caused by unsupported scripts or misinterpreted inputs.
Detect Toxicity
Epsilla incorporates filters to identify and block harmful or abusive language in user queries. This safeguard ensures that the system does not engage in or propagate harmful content, protecting both users and enterprises from reputational risks.
Limit Tokens
To prevent resource overuse and ensure efficient system performance, Epsilla enforces input length limits. For instance, queries exceeding a predefined token or character limit are flagged and rejected, avoiding potential issues like denial-of-service (DoS) attacks.
Best Practices for Input Guardrails
Epsilla allows enterprises to customize input guardrails based on their unique needs. Administrators can configure rules for anonymization, topic restrictions, language preferences, toxicity detection, and token limits directly through the platform. These guardrails empower organizations to manage user interactions effectively while safeguarding sensitive data and maintaining system integrity.
Here’s how we can simplify, rephrase, and explain the input guardrail workflow in a clear and easy-to-understand manner:
How Input Guardrail Works in Epsilla Workflows
Epsilla ensures that only safe and appropriate inputs are processed through its input guardrail system. Here’s how you can set up input guardrails using Epsilla's workflow nodes:
Step 1: Add the LLM Completion Node
Start by creating an LLM Completion Node to screen incoming user questions. This node uses a custom prompt to determine whether the input is safe or unsafe. The prompt should look like this:
If the user's question falls into any of the categories below, respond with 1 for unsafe to answer. Otherwise, respond with 2 for safe to answer:
- Contains PII (e.g., Social Security Number, credit card number)
- Discusses politics
- Relates to crime topics
Respond with a single number, either 1 or 2. DO NOT INCLUDE ANY ADDITIONAL TEXT BEFORE OR AFTER THE NUMBER.
This ensures the system correctly identifies the user query as safe or unsafe.
Step 2: Add a String Template Node for Unsafe Responses
Next, add a String Template Node to handle unsafe inputs. If the input is flagged as unsafe, the node will generate the following response:
"Sorry, but I cannot answer your question."
This step ensures users receive a polite and clear response when their input is inappropriate or violates the system's guidelines.
Step 3: Use a Router Node to Manage Outputs
Insert a Router Node to connect the LLM Completion Node’s outputs to the appropriate next step:
This ensures that the system handles both safe and unsafe inputs properly.
Step 4: Finalize with a String Reducer Node
Finally, use a String Reducer Node to combine the outputs into a single, unified response. This node merges the outcomes of both safe and unsafe workflows, ensuring the user receives a seamless reply based on their input.
A user asks the agent to check the bill for their credit card. The system detects that the question contains sensitive information (credit card data) and classifies it as unsafe using the LLM Completion Node. The query is then routed to the String Template Node, which provides a polite response: "Sorry, but I cannot answer your question. This ensures the system respects privacy and adheres to input guardrail rules.
Query Rewriting
Once a user’s input passes the guardrail checks, the next step is query rewriting. This process refines the user’s query to make it clearer, more precise, and more relevant for retrieval. Epsilla supports at least three techniques for query rewriting: Rewrite Based on History, Create Subqueries, and Create Similar Queries. Let’s explore how each technique works.
Rewrite Based on History
In many conversations, a user’s final query may miss important details mentioned earlier. Epsilla’s query rewriting leverages the user’s query history to retrieve this missing context and incorporate it into the rewritten query. This ensures that the system fully understands the user’s intent.
Example:
Query History:
Without rewriting, the final query lacks context. Using the history, Epsilla rewrites the query as: Rewritten Query: "Compare the rewards features of gold and platinum credit cards."
This rewriting process improves retrieval precision by clarifying the user’s intent and providing all necessary context for an accurate response.
Workflow Explanation: In this workflow, the String Template Node uses the conversation's context, including prior chat history and the follow-up question, to generate a standalone query. The prompt provided ensures that missing context from earlier messages is included, making the query clear and complete. This standalone query is then passed to the LLM Completion Node for processing, ensuring precise retrieval and response generation.
Prompt in the String Template Node:
Given the following conversation and a follow-up question, rephrase the follow-up question to be a standalone question:
<CHAT_HISTORY>
{{ chat_history }}
</CHAT_HISTORY>
<FOLLOW_UP_QUESTION>
{{ the_question }}
</FOLLOW_UP_QUESTION>
Standalone question:
This setup ensures the rewritten query accurately reflects the user’s intent by leveraging prior context.
Create Subqueries
Sometimes, a complex query involves multiple parts that need to be addressed separately. Epsilla’s query rewriting can break down such queries into more specific subqueries, a technique referred to as query decomposition. This makes it easier for the system to retrieve accurate and detailed information for each part.
Example:
Original Query: "Compare features of gold and platinum credit cards." Rewritten Subqueries:
This approach ensures that each entity in the query is addressed independently, boosting the accuracy and completeness of the final response. For more details, check out our blog post on query decomposition.
Create Multiple Versions Of The Query
When retrieving information, lexical or semantic matching may sometimes fall short in identifying the most relevant documents. To overcome this, Epsilla leverages RAG Fusion to generate multiple versions of the user query from different perspectives that align more closely with the user’s intent. This involves rephrasing the query using synonyms, domain-specific terminology, or related concepts.
Example:
User Query: "Tell me about the benefits of platinum credit cards." Generated Similar Queries:
This technique expands the range of potential retrieval matches, ensuring that the system captures all relevant information. Learn more about this in our blog post on RAG fusion and multi-perspective retrieval.
Document Ingestion
Efficiently managing large-scale data ingestion is crucial for the success of any RAG system. Epsilla's platform is designed to handle massive datasets from diverse sources, ensuring seamless integration and synchronization.
Epsilla's approach to Extract, Transform, Load (ETL) processes optimizes large-scale data synchronization and processing. This strategy significantly enhances system stability and observability, addressing scalability challenges and improving performance and data accuracy in demanding production environments.
领英推荐
Data Ingestion at Scale
Epsilla excels at ingesting extensive datasets from various repositories, including Notion, Google Drive, and other content management systems. Our platform optimizes both the speed and cost of data ingestion, effectively managing hundreds of thousands of data files to meet production-level demands.
Transforming unstructured data into manageable pieces is essential for efficient processing and retrieval. Epsilla employs advanced chunking techniques to optimize this transformation and make sure the chunk preserves contextual cohesive information.
Embedding transforms text into numerical representations (vectors) for retrieval, and selecting the right embedding model is a critical decision for building an effective RAG system. At Epsilla, we provide flexible embedding options to meet diverse needs and ensure optimal performance.
Which Embedding Model Fits My Needs Best?
There is no one-size-fits-all embedding model. The best choice depends on the specific requirements of your use case, including performance needs, budget, and language considerations. OpenAI offers both large and small embedding models. The large model excels in general-purpose scenarios where high-quality embeddings are essential, while the small model is cost-effective for projects with tighter budgets. For multilingual applications, JinaAI models are ideal, offering strong support for diverse languages. For specialized domains such as finance or law, VoyageAI models provide domain-specific insights and improved accuracy tailored to these industries.
To evaluate models, the Massive Text Embedding Benchmark (MTEB) is a helpful starting point. It compares embedding models across various metrics like vector dimensions and retrieval performance. However, it is important to test embedding models within the context of your specific data and use case, as MTEB results may not fully reflect domain-specific needs.
Epsilla’s Flexible Embedding Configuration
Epsilla simplifies embedding configuration by offering an intuitive dropdown menu where users can select models. Our default models, epsilla/text-embedding-3-small provide seamless integration and require no additional setup. For more specialized needs, users can integrate external models, such as JinaAI or VoyageAI, ensuring adaptability for various scenarios. Epsilla allows users to balance cost-efficiency and performance by tailoring the embedding model selection to their unique use case.
A robust indexing system is the cornerstone of rapid and accurate information retrieval in RAG systems. Epsilla's indexing solutions are designed to meet the demands of large-scale, unstructured data environments.
Data Synchronization
Maintaining up-to-date information is vital. Epsilla ensures timely synchronization across multiple data sources, preventing data loss or duplication. This reliability guarantees that your knowledge base remains current, even as new files are added or old ones are deleted.
User Interface and API Gateway
Our user-friendly interface supports a wide range of data source configurations, allowing for seamless file uploads from local systems, website URLs, object buckets, and network storage. With features like periodic data synchronization and real-time progress updates, managing your data is straightforward. Additionally, our robust API Gateway handles large data volumes and high user concurrency, integrating advanced features such as authentication, routing, dispatch, and dynamic scaling.
System Robustness and Monitoring
Our architecture includes robust monitoring and operational resilience features, such as intuitive progress reporting, comprehensive logging, efficient alert systems, and mechanisms for gracefully handling operational failures. These tools maintain continuous service availability and high performance as your system scales.
Customer Stories
Epsilla’s large-scale Smart ETL for document loading and knowledge indexing architecture has empowered many customers to develop their RAG solutions, with the following examples representing just some of our success stories across various industries:
1?? Legal: A Legal-Tech company processed over 120,000 legal case files from court databases to create a junior legal assistant AI. This tool helps streamline the search and analysis of past cases, enhancing efficiency in new case processing.
2?? Construction: A Construction-Tech company managed the ingestion and analysis of tens of thousands of zoning law documents, some of which exceed 10,000 pages and several hundred MBs in size. They developed an AI assistant that leverages this data to aid in construction planning, ensuring compliance and strategic decision-making.
3?? Education: A Faith-Tech company used Epsilla to develop a AI-powered virtual study buddy providing a personalized and engaging conversational learning experience , incorporating thousands of books and annotations. This comprehensive tool aids faith-seekers, biblical students and scholars in their pursuit of knowledge on faith.
By integrating these advanced document ingestion, chunking, and indexing capabilities, Epsilla empowers organizations to harness the full potential of their unstructured data, driving informed decision-making and operational efficiency.
Data Storage
In a RAG system, storing chat history and user feedback is essential for ensuring adaptive, efficient, and user-centric interactions. Epsilla provides robust storage solutions to capture and utilize these data types effectively.
Chat History
Epsilla’s platform stores complete chat histories within the application, accessible through the Analytics interface. This feature allows users to review all previous interactions, providing a detailed record of queries and responses.
By maintaining these histories, the system can leverage past interactions to enhance personalization, adapting responses to match user preferences and context. For instance, the chat history enables the system to recognize recurring patterns in user queries and deliver more relevant and accurate answers over time.
User Feedback
Epsilla goes beyond simply storing chat histories by integrating a comprehensive feedback mechanism. Users can provide feedback through features such as thumbs-up, thumbs-down, or bug reports directly within the chat interface.
The platform records these interactions and presents them in the Analytics dashboard, offering actionable insights into user experiences. For example, if a user submits feedback after asking specific questions, the system captures this data alongside the interaction history, enabling targeted system improvements.
By combining feedback with chat history, Epsilla empowers teams to identify issues, fine-tune system performance, and ensure a continuously improving user experience.
Vector Database
Epsilla’s all-in-one RAG platform leverages our own open-source vector database that empowers the knowledge bases. The Epsilla vector database sets a new standard in semantic search. With its innovative architecture and groundbreaking performance, Epsilla delivers unparalleled speed, efficiency, and scalability tailored to modern enterprise needs.
Innovative Architecture
Epsilla introduces a revolutionary approach to vector search by leveraging parallel graph traversal technology. Unlike conventional systems built on HNSW (Hierarchical Navigable Small World) indexes, Epsilla employs a single-layer nearest neighbor graph to achieve exceptional query speed and accuracy.
Benchmark Results
Epsilla outperforms leading vector databases in key performance metrics, as demonstrated in recent benchmarking tests using the gist-960-euclidean dataset.
These benchmarks highlight Epsilla’s ability to handle real-time vector search demands, making it a game-changer for businesses relying on Large Language Models (LLMs) and generative AI solutions.
Efficiency and Features
Epsilla’s vector database is designed with efficiency and enterprise-grade functionality in mind:
The Epsilla vector database bridges the gap between general-purpose LLMs and specific business requirements, providing a high-performance solution that adapts to the challenges of modern enterprises. Its 10x performance advantage, robust feature set, and efficient architecture redefine what’s possible for enterprise RAG systems.
Techniques for Improving Retrieval
Efficient retrieval is the foundation of any high-performing RAG system. However, LLMs can often be distracted by irrelevant context, and retrieving a large volume of documents can lead to missing critical information due to attention limitations. At Epsilla, we leverage advanced techniques to ensure retrieval is not only accurate but also diverse and contextually relevant.
Hypothetical Document Embeddings (HyDE) & Hypothetical Questions
A big challenge in traditional embedding-based search systems is something called misalignment—basically, a mismatch between how user questions and documents are represented in the embedding space. Ideally, an embedding model should “get” the main patterns in both questions and documents, putting them into a shared space so that it can easily match questions with the closest relevant documents.
The HyDE technique enhances retrieval performance by bridging the gap between user questions and document embeddings. It achieves this by generating hypothetical documents that align more closely with the actual documents in knowledge base, using generative models like GPT. These hypothetical documents are converted into embeddings, which serve as intermediaries to locate real, relevant documents in the database with greater precision. Epsilla has effectively integrated HyDE, enabling seamless alignment between question and document embedding spaces, resulting in significantly improved retrieval accuracy. For a deeper dive into how HyDE enhances retrieval in RAG systems, check out our article on HyDE.
Hypothetical Questions takes an inverse approach by leveraging a large language model (LLM) to generate potential questions for each chunk of a document. Instead of embedding document chunks directly, the generated questions are embedded into vector space. When a user submits a query, it is compared to these hypothetical question embeddings to identify the most relevant matches, which are then linked back to the corresponding document chunks. This approach resolves the common misalignment between user questions and document embeddings by transforming document content into a question format, ensuring better alignment and more accurate retrieval. Read our article on Hypothetical Questions.
Query Routing
Query routing is an effective method for directing queries to the most appropriate sub-workflows, optimizing the response by ensuring each query is processed in the most relevant context. In enterprise environments where data spans diverse domains like product documentation, technical papers, and code repositories, query routing ensures that users receive tailored, accurate results. For example, a query about a specific product feature would be routed to the sub-workflow designed to process product-related questions, resulting in more precise and contextually relevant answers. This approach enhances the efficiency of query processing by leveraging specialized workflows for different scenarios rather than relying on multiple indices.
Epsilla’s approach to query routing simplifies this process. Learn more in our article on Query Routing.
Reranker
When initial retrieval results lack precision, Epsilla leverages rerankers to refine document rankings and enhance the quality of search results. By employing a combination of heuristic rerankers including RRF, RSF, and DBSF, and model-based rerankers like Jina and Cohere, RAG systems can significantly reduce LLM hallucinations and improve generalization across out-of-domain queries. This process ensures more accurate, contextually relevant outcomes, boosting the reliability of the retrieval-augmented generation process.
While sophisticated rerankers may introduce computational overhead, Epsilla’s efficient implementation strikes a balance between performance gains and resource utilization. Rerankers are especially effective in prioritizing the most relevant information, ensuring users get precise and actionable insights. For more details, read our article on Reranking.
By incorporating advanced techniques like HyDE, Query Routing, and Reranking, Epsilla ensures that RAG systems deliver accurate, reliable, and contextually relevant results, setting a new standard for retrieval optimization.
Generator
In the context of RAG systems, the generator plays a critical role in producing meaningful and accurate completions. Epsilla takes a vendor-agnostic approach, enabling seamless integration of both open-source and closed-source Large Language Models (LLMs) into our workflows.
This flexibility allows users to choose the best LLMs for their specific needs, whether they require the capabilities of proprietary models like GPT, Claude, Gemini, or prefer the adaptability of open-source alternatives like Llama and Mistral. By integrating all these models into Epsilla's workflow engine, users can create custom AI agents that are not only powerful but also tailored to their unique requirements.
Epsilla’s approach ensures that businesses have access to cutting-edge generative AI capabilities, all within a single, cohesive platform designed for efficiency and scalability.
Output Guardrail
The output guardrail serves as a critical layer to ensure that generated responses align with both user expectations and enterprise’s core values. While similar to input guardrails, output guardrails specifically focus on identifying and mitigating issues in the generated content, such as hallucinations, competitor mentions, and information that could potentially harm the brand's reputation.
Purpose and Functionality
The goal of the output guardrail is to verify that the responses are:
How It Works
Epsilla supports integrating an LLM Node in the workflow as part of the output guardrail, configured with the following prompt to validate and adjust responses:
Sample Prompt on LLM Node:
Check if the AI response to a user’s question aligns with our company’s values:
- Regarding RAG vs. Fine-tuning: We believe these approaches can be combined rather than favoring one over the other.
- Regarding GraphRAG: We believe it can improve the accuracy of the answer.
Instructions:
1. If the provided user answer relates to the topics above and conflicts with our stated views, modify it to ensure compliance with our values.
2. If the provided user answer does not relate to these topics or already aligns with our values, return the answer exactly as is, without changing a single word.
3. Always include all UUIDs from the response.
Observability
Epsilla enables detailed observability throughout the system’s operations. Every step in the workflow—from data retrieval to response generation—can be traced and analyzed. This step-by-step debugging capability allows operators to fine-tune prompts, monitor intermediate outputs, and optimize performance.
Real-time monitoring tools provide visibility into key metrics such as response latency, resource usage, and system behavior, ensuring your RAG application operates smoothly and efficiently. By tracking and addressing potential issues proactively, Epsilla delivers reliability at scale.
Multi-Tenancy
Epsilla’s platform is designed with seamless multi-tenancy in mind, ensuring that each user’s interaction is fully isolated and secure. Each user’s chat history and data are stored separately, and the system uses metadata to maintain this separation during retrieval.
This architecture allows multiple users to interact with the platform simultaneously without compromising data privacy or performance. Whether for small teams or large enterprises, Epsilla’s multi-tenancy approach guarantees both scalability and security.
Conclusion
Epsilla’s all-in-one platform redefines how enterprises build and scale RAG systems, offering a seamless and efficient solution for modern AI applications. By integrating key features like efficient data management, a cutting-edge vector database, advanced retrieval techniques, robust output guardrails, and user-centric design, Epsilla simplifies the creation of enterprise-grade RAG solutions.
What truly sets Epsilla apart is its ability to streamline the entire process—from ingestion to deployment—on a single platform. This unified approach accelerates iteration cycles, ensures adaptability, and delivers scalable, accurate, and secure AI solutions tailored to evolving business needs.
With Epsilla, enterprises can unlock the full potential of generative AI, confidently meeting the demands of real-world applications while staying agile in a dynamic landscape.
Sign-up Epsilla for FREE today and start building your enterprise-grade RAG systems!
Professor
2 个月Interesting