Building a Ranking System to Enhance Prompt Results: The New PageRank for RAG/LLM

In this document, you will learn how to build a system that decides, among dozens of candidate paragraphs selected from the corpus to answer a prompt, which ones to show in the results, and in what order. The goal is to maximize relevancy while not overwhelming the user with a long, cluttered answer. Think of it as the new PageRank for RAG/LLM, although the algorithm is radically different, and much simpler.

The approach is generic and works for all RAG/LLM systems whether based on neural networks or not. It is implemented in xLLM. The main steps are:

Backend processing (linked to the corpus)

1. Split your corpus into text entities such as webpages, paragraphs, sections and so on. This step is similar to chunking. Attach an ID (called index) to each text entity.
2. Text entity have two types of fields: regular text like in all LLMs, and knowledge graph elements such as categories, related items, URL, tags, parent categories, title, and so on. These knowledge graphs elements are found while crawling and part of the original corpus. Or they can be added after the full crawl. For instance, in xLLM, agents are assigned to text entities post-crawling, using a clustering algorithm.
3. You need two types of tokens: regular ones, and those found in the knowledge graph elements. The latter are called graph tokens. You then create a key-value table Hash_ID, where the key is a token, and the value is the list of text entity IDs attached to the token in question, with a token count for each one. Graph tokens start with “__”, to differentiate them from regular tokens.

Frontend processing (linked to the prompt)

1. You create a local, small key-value table ID_Hash, a transposed version of Hash_ID, where the key is a text entity ID, and the value is a list of tokens t found in the prompt, with ID in Hash_ID[t].
2. For each ID, you compute (say) 4 relevancy scores: [..]

?? Follow this link to read the full article with all Frontend steps and smart ranking, download the technical document with Python code (with links to GitHub) and case study featuring the anonymized augmented corpus of a fortune 100 company, as well as future LLM developments (auto-indexing and LLM for cataloging and glossary generation).