Leveraging LLM Tools for Beyond Language Tasks

Enhancing ArangoDB Text Search with Haystack Tools

TL;DR: This article explores the innovative application of Large Language Model (LLM)-related tools, such as Haystack for document processing and FastEmbed for generating semantic embeddings, to improve document management and search functionalities within ArangoDB, a multi-model database that does not natively support vector search. By storing semantic embeddings alongside textual content in ArangoDB and leveraging external vector databases like Qdrant for advanced search capabilities, organizations can create a hybrid system that combines the structured data management strengths of ArangoDB with the semantic search prowess of vector databases.

Key Points:

• Document Processing with Haystack: Utilizes advanced document conversion, cleaning, and semantic splitting to prepare data for efficient storage and retrieval.
• Semantic Embeddings with FastEmbed: Generates numerical representations of text to enhance semantic understanding and search capabilities.
• Hybrid Search System: Combines ArangoDB's document storage with Qdrant's vector search to enable semantic searches, overcoming ArangoDB's lack of native vector search support.
• Human-in-the-Loop for Data Quality: Ensures high data quality for vector search applications by requiring human approval before storing data in Qdrant, enhancing the accuracy and relevance of search results.
• Practical Implementation: Involves generating and storing embeddings in ArangoDB, developing custom retrieval logic, duplicating data between ArangoDB and Qdrant, and implementing a human-in-the-loop system for maintaining data quality.

This approach not only addresses the challenges of managing and searching through large volumes of documents but also sets a new standard for document management systems by leveraging the latest advancements in AI and machine learning for non-traditional applications.

In the realm of digital information management, the ability to swiftly navigate and extract value from vast repositories of text data is paramount. Large Language Models (LLMs) have been at the forefront of transforming our capabilities in natural language processing, offering unprecedented insights and automation. However, the utility of technologies developed in the orbit of LLMs—such as embedding generators and document processing frameworks—extends well beyond their initial language-centric applications. Among these, the integration of Haystack tools with ArangoDB for optimizing text search represents a groundbreaking approach to overcoming traditional challenges in document management systems.

This article embarks on an exploration of how LLM-related tools, particularly those associated with the Haystack ecosystem, can be adeptly repurposed to enhance the functionality and efficiency of ArangoDB, a powerful multi-model database renowned for its flexibility and performance in managing and searching text data. By leveraging the advanced capabilities of Haystack for document processing and embedding generation, users can unlock new levels of search efficiency, precision, and scalability in ArangoDB, transforming the landscape of text search and retrieval.

Our journey will uncover the innovative application of these tools in streamlining document management processes, from the initial collection and processing of documents to their storage and retrieval in ArangoDB. We aim to illuminate the path for end users to harness the sophisticated features of LLM-related technologies in novel contexts, thereby enhancing operational efficiency and discovering new solutions to longstanding challenges. Through this exploration, we invite readers to expand their perception of LLM tools, recognizing their potential as versatile instruments capable of driving significant improvements in a wide array of non-language-specific tasks, with a special focus on optimizing ArangoDB text search.

Understanding LLM-Related Tools

The advent of Large Language Models (LLMs) has revolutionized the field of natural language processing and led to the development of many tools and technologies designed to harness their capabilities. These LLM-related tools, such as embedding generators and document processing frameworks, play a pivotal role in enhancing the performance of LLMs across various tasks. Understanding these tools is the first step toward realizing their potential in applications beyond traditional language tasks, including optimizing text search in databases like ArangoDB.

Embedding Generators: The Power of FastEmbed

Embedding generators, such as FastEmbed, by Qdrant , are instrumental in transforming textual data into numerical representations, known as embeddings. These embeddings capture the semantic essence of the text, allowing for a more nuanced and efficient approach to text analysis and search. FastEmbed, for instance, leverages quantized model weights and ONNX Runtime for inference, offering a lightweight yet powerful solution for generating high-quality embeddings. This capability is crucial for enhancing database search functionality, as it enables matching queries with documents based on semantic similarity rather than mere keyword matching.

Document Processing Frameworks: The Versatility of Haystack

Haystack represents a comprehensive ecosystem for building search applications that can handle complex document processing tasks. It provides a flexible pipeline for converting, cleaning, and preprocessing documents, making it an invaluable tool for preparing data for storage and retrieval. With components like the TextFileToDocument and PyPDFToDocument converters, Haystack facilitates the transformation of raw text and PDF files into structured formats suitable for further analysis. Moreover, its preprocessing capabilities, including document cleaning and splitting, ensure that the data is optimized for both human readability and machine processing.

The Synergy with Large Language Models

While embedding generators and document processing frameworks are powerful in their own right, their true potential is unlocked when combined with the capabilities of LLMs. These tools can preprocess data to maximize the effectiveness of LLMs for tasks such as document summarization, question answering, and text classification. By generating embeddings, they also enable LLMs to efficiently search through large volumes of text, identifying relevant information based on semantic context rather than explicit keywords.

Beyond Language: A New Frontier for LLM Tools

The application of LLM-related tools extends far beyond the boundaries of language processing. By leveraging the advanced capabilities of tools like FastEmbed and Haystack, users can enhance the functionality of multi-model databases such as ArangoDB. This improves the efficiency and accuracy of text search and opens up new possibilities for managing and analyzing textual data at scale.

The Challenge of Document Management and Search

In the digital age, organizations across all sectors are inundated with vast amounts of textual data, ranging from internal documents and reports to customer feedback and external publications. Efficiently managing, searching, and extracting value from this data is a significant challenge that directly impacts operational efficiency and decision-making processes. The core issues in document management and search can be broadly categorized into three main areas: volume, search efficiency, and relevance.

Volume: Navigating the Data Deluge

The sheer volume of textual data that organizations need to handle is staggering. This data deluge presents a logistical challenge, requiring robust data storage, organization, and retrieval systems. Traditional document management systems often struggle to scale effectively, leading to inefficiencies in data handling and increased costs.

Search Efficiency: Beyond Keyword Matching

Traditional text search mechanisms primarily rely on keyword matching, which can be inefficient and imprecise. This approach often fails to capture the semantic nuances of human language, resulting in searches that are both time-consuming and fraught with irrelevant results. As data volumes grow, the limitations of keyword-based search become increasingly apparent, necessitating more sophisticated search methodologies.

Relevance: Finding the Needle in the Haystack

Ensuring the relevance of search results is perhaps the most critical challenge in document management and search. Users need to quickly find the most pertinent information without sifting through vast amounts of irrelevant data. Achieving high relevance in search results requires understanding the context and meaning behind user queries and the content within the documents, something that traditional search engines often struggle to accomplish.

The Need for Advanced Solutions

The limitations of traditional document management and search systems highlight the need for more advanced solutions capable of addressing these challenges. Such solutions must be able to handle large volumes of data efficiently, improve search accuracy by understanding the semantic context of both documents and queries, and ensure the relevance of search results to meet the user's needs.

In this context, applying LLM-related tools—originally developed for natural language processing tasks—presents a novel approach to overcoming the challenges of document management and search. By leveraging the capabilities of these tools, organizations can enhance their document management systems, making them more scalable, efficient, and capable of delivering highly relevant search results.

In the next section, we will explore how the innovative application of LLM-related tools, specifically within the framework of Haystack and the embedding capabilities of FastEmbed, can transform the landscape of document management and search, particularly in optimizing text search within ArangoDB.

Innovative Application of LLM-Related Tools in Document Management

The transformative potential of LLM-related tools in addressing the challenges of document management and search lies in their ability to understand, process, and organize textual data in ways that traditional systems cannot. By leveraging the advanced capabilities of tools like Haystack for document processing and FastEmbed for generating semantic embeddings, organizations can significantly enhance the efficiency and relevance of their text search functionalities, especially when integrated with databases like ArangoDB.

Leveraging Haystack for Document Processing

Haystack provides a flexible, modular framework designed for building search applications that can handle complex document processing tasks. Its ability to convert, clean, preprocess, and split documents into manageable chunks makes it an invaluable asset in preparing data for efficient storage and retrieval.

• Document Conversion and Cleaning: Haystack's components for converting text files and PDFs into structured documents, coupled with its cleaning capabilities, ensure that the data is readable and optimized for machine processing. This step is crucial for removing noise and standardizing the format of the stored documents.
• Semantic Document Splitting: By splitting documents into smaller, semantically coherent chunks, Haystack facilitates more granular indexing and search. This approach allows users to retrieve only the most relevant sections of documents, improving search efficiency and user experience.

Here is the basic example of processing text and pdf documents using document cleanup and sentence splitting:

def prepare_docs(source_docs, metadata):
    pipeline = Pipeline()
    pipeline.add_component("text_converter", instance=TextFileToDocument())
    pipeline.add_component("pdf_converter", instance=PyPDFToDocument())
    pipeline.add_component("text_cleaner", instance=DocumentCleaner(
        remove_extra_whitespaces=False
    ))
    pipeline.add_component("pdf_cleaner", instance=DocumentCleaner(
        remove_extra_whitespaces=False
    ))

    pipeline.connect("text_converter", "text_cleaner")
    pipeline.connect("pdf_converter", "pdf_cleaner")

    print("√: Running a pipeline for splitting all files in the folder")

    prepared_docs = pipeline.run({
        "text_converter": {
            "sources": [item["path"] for item in source_docs if item["type"] == 'txt'] or [],
            "meta": metadata or {}
        },

        "pdf_converter": {
            "sources": [item["path"] for item in source_docs if item["type"] == 'pdf'] or [],
            "meta": metadata or {}
        }
    })

    # merge two arrays into a new one
    parsed_docs = (prepared_docs["text_cleaner"]["documents"] or []) + prepared_docs["pdf_cleaner"]["documents"]

    # split docs
    splitter = DocumentSplitter(split_by="sentence", split_length=5)
    split_docs = splitter.run(documents=parsed_docs)["documents"]

    for item in split_docs:
        item.meta = {
            "createdOn": int(datetime.timestamp(datetime.now())),
            "source": os.path.basename(item.meta["file_path"])
        }
        item.meta.update(metadata)

    print("√: Adding embeddings to documents")

    for doc in split_docs:
        embeddings = list(embedding_model.embed(doc.content))
        doc.embedding = list(embeddings[0])

    return split_docs        

And, to walk through some paths and collect the files, you would need something like this:

def collect_docs(path):
    found_docs = []
    for root, dirs, files in os.walk(path):
        for file in files:
            if file.endswith(".txt"):
                found_docs.append({"name": file, "path": os.path.join(root, file), "type": "txt"})

            if file.endswith(".pdf"):
                found_docs.append({"name": file, "path": os.path.join(root, file), "type": "pdf"})
    return found_docs        

Enhancing Search with FastEmbed Embeddings

FastEmbed plays a critical role in transforming textual data into numerical representations that capture the semantic essence of the text. These embeddings can be used to enhance the search capabilities of ArangoDB by enabling semantic search functionalities.

• Semantic Similarity: Embeddings allow for the matching of queries with documents based on semantic similarity rather than exact keyword matches. This leads to more relevant search results, as the system can understand the intent behind a query and find documents that match this intent, even if they don't contain the exact keywords.
• Efficient Indexing and Retrieval: When stored in ArangoDB, these embeddings can be indexed efficiently, enabling rapid retrieval of semantically relevant documents. This significantly reduces the time and computational resources required to find relevant information within a large corpus.

Integration with ArangoDB

ArangoDB's multi-model database structure is ideally suited to utilizing Haystack and FastEmbed's document processing and embedding generation capabilities. By storing document chunks along with their semantic embeddings, ArangoDB can offer advanced text search functionalities that are both fast and accurate.

• Scalable Document Storage: ArangoDB's flexible data model accommodates the storage of document chunks and their associated metadata seamlessly, allowing for efficient organization and retrieval of large volumes of data.
• Advanced Search Capabilities: Integrating semantic embeddings into ArangoDB's search mechanism enables the database to support complex search queries, including semantic search, similarity search, and more. This allows users to quickly find the most relevant information based on the content's meaning, not just its keywords.

Integrating ArangoDB with vector search capabilities, such as those provided by Qdrant or Weaviate , offers a nuanced approach to enhancing search functionalities within document management systems. While ArangoDB excels in managing and querying structured data, its native capabilities do not extend to vector search, which is essential for performing semantic searches based on embeddings. However, by storing embeddings alongside textual content within ArangoDB and leveraging external vector databases like Qdrant for search, organizations can create a hybrid system that combines the strengths of both worlds.

Even though ArangoDB does not directly support vector search, it can still store embeddings generated by LLM tools alongside the original text content. This setup allows for the reuse of embeddings in various LLM pipelines, facilitating tasks that require a semantic understanding of the content.

With the embeddings stored in ArangoDB, organizations can develop custom retrieval mechanisms for specific applications, such as the Retrieval-Augmented Generation (RAG) project. This involves creating algorithms to query embeddings for similarity and return the most relevant text content based on semantic matching.

To enable advanced vector search capabilities, data can be duplicated between ArangoDB and a dedicated vector database like Qdrant. This approach allows organizations to utilize ArangoDB for managing structured data and textual content while leveraging Qdrant for its powerful vector search functionalities.

Implementing a human-in-the-loop system ensures high data quality before storing information in Qdrant. By requiring human approval for data entries, organizations can maintain a high standard of accuracy and relevance in their vector search database, which is crucial for applications like RAG that rely on precise semantic matching.

By combining the capabilities of ArangoDB with vector databases like Qdrant, organizations can overcome the limitations of traditional search mechanisms. This hybrid approach enhances document management and search systems with advanced semantic search capabilities and ensures the flexibility and scalability for handling large data volumes. Through strategic integration and implementing quality control mechanisms like human-in-the-loop, organizations can achieve high precision and relevance in their search functionalities, paving the way for more intelligent and efficient information retrieval systems.

Practical Implementation Steps

1. Generate and Store Embeddings: Use LLM-related tools to generate embeddings for your textual content and store these embeddings in ArangoDB alongside the original text.
2. Custom Retrieval Logic: Develop custom retrieval logic in ArangoDB to utilize the stored embeddings for semantic search or as part of a larger LLM pipeline.
3. Data Duplication and Synchronization: Implement a system to duplicate and synchronize data between ArangoDB and Qdrant, ensuring that both databases are up-to-date and consistent.
4. Human-in-the-Loop for Data Quality: Before transferring data to Qdrant, incorporate a human-in-the-loop process to review and approve content, maintaining high data quality for vector search applications.
5. Leverage Qdrant for Vector Search: Utilize Qdrant's vector search capabilities to perform advanced semantic searches, enhancing the overall search experience and accuracy.

Example on how to use the same prepare_docs method mentioned above to load data into Qdrant:

import os

from haystack_integrations.document_stores.qdrant import QdrantDocumentStore

import utils

document_store = QdrantDocumentStore(
    url=os.getenv('QDRANT_URL'),
    api_key=os.getenv('QDRANT_API_KEY'),
    embedding_dim=384,
    return_embedding=True,
    wait_result_from_api=True,
)

folder_with_files = "data/"

docs = utils.collect_docs(folder_with_files)

print(f"√: Found {len(docs)} files")

prepared_docs = utils.prepare_dcos(docs, {"importer": "my-demo-app"})

print(f"Prepared {len(prepared_docs)} documents")

document_store.write_documents(documents=prepared_docs)The Outcome: A Synergistic Solution        

And the one for ArangoDB would be something like that:

import os
import sys

from arango import ArangoClient

import utils

client = ArangoClient(hosts=[os.getenv('DB_HOST')])
db = client.db(os.getenv('DB_DATABASE'), username=os.getenv('DB_USERNAME'), password=os.getenv('DB_PASSWORD'))

try:
    print(f"√: {os.getenv('DB_HOST')}: Connecting to {os.getenv('DB_DATABASE')} database")
    print(f"√: {db.version()}")
except Exception as e:
    print(f"Error connecting to {os.getenv('DB_HOST')}: {e}")
    sys.exit(1)

print('√: Connected')

COLLECTION_NAME = 'haystack_docs'

if db.has_collection(COLLECTION_NAME):
    database_docs = db.collection(COLLECTION_NAME)
else:
    database_docs = db.create_collection(COLLECTION_NAME)

folder_with_files = "data/"

docs = utils.collect_docs(folder_with_files)

print(f"√: Found {len(docs)} files")

prepared_docs = utils.prepare_dcos(docs, {"importer": "arangodb-importer"})

print(f"Prepared {len(prepared_docs)} documents")

print(f"√: Adding docs into {COLLECTION_NAME} collection")
for doc in prepared_docs:
    doc_source = doc.meta["source"]
    print(f"√: Processing {doc_source}/{doc.id}")
    database_docs.insert({
        "metadata": doc.meta,
        "page_content": doc.content
    })
        

The innovative application of LLM-related tools in document management and search represents a synergistic solution that addresses the core challenges of volume, search efficiency, and relevance. By combining the document processing strengths of Haystack with the semantic understanding provided by FastEmbed embeddings, and integrating these capabilities with ArangoDB, organizations can create a powerful document management and search system that is both scalable and capable of delivering highly relevant search results.

In the next section, we will delve into a case study that illustrates the practical benefits of this approach, showcasing how integrating LLM-related tools with ArangoDB can transform the efficiency and effectiveness of text search and retrieval processes.

A few notes on Splitters

The approach to storing documents in sliced chunks rather than whole entities in ArangoDB, especially when leveraging ArangoSearch, is strategic for several reasons. This method enhances search performance, precision, and efficiency in data retrieval. Here's a deeper look into the benefits and considerations of this approach:

Enhanced Search Performance

• Indexing Efficiency: ArangoSearch can index these chunks more efficiently by storing documents in smaller chunks. Smaller documents mean faster indexing times, as each chunk can be processed faster than a large document. This results in a more responsive search experience.
• Query Performance: Searches can be executed faster because the search engine has to scan through smaller indexed units. This reduces the computational load and speeds up query response times, as the engine can more quickly identify relevant chunks without processing the entire content of large documents.

Precision in Data Retrieval

• Relevance of Results: When documents are chunked, the search engine can return just the relevant chunks containing the queried terms or phrases. This increases the relevance of the search results, as users receive precisely the segments of text that match their search criteria, rather than having to sift through entire documents to find the information they need.
• Contextual Accuracy: This method also allows for a better contextual understanding of search queries. Since each chunk is more focused, the search engine can more accurately determine the context of the query within each chunk, leading to results that are not only relevant but contextually accurate.

Efficiency in Data Management

• Storage Optimization: Storing documents in chunks can lead to more efficient storage management. It allows for better compression and deduplication strategies, as similar chunks can be identified and managed more effectively than entire documents.
• Scalability: This approach scales well with the growth of data. As the document corpus expands, adding and indexing new chunks remains efficient, ensuring that the search system remains responsive and scalable.

Related links:

• ArangoDB - https://arangodb.com/ the most complete and scalable graph database
• Haystack - https://haystack.deepset.ai/ Open-source LLM framework to build production-ready applications
• PyPDFToDocument - https://docs.haystack.deepset.ai/v2.0/docs/pypdftodocument a component that converts PDF files to Haystack Documents
• TextFileToDocument - https://docs.haystack.deepset.ai/v2.0/docs/textfiletodocument a component that converts text files to a Document.
• FastEmbed - https://qdrant.github.io/fastembed/ a lightweight, fast, Python library built for embedding generation
• Qdrant - https://qdrant.tech/ a vector database & vector similarity search engine