Methods for Ranking LLM-Generated Results

LLM ranking systems hold immense potential for businesses and marketers, offering new ways to prioritize information, products, leads, and competitors. By developing ranking systems that account for lists, frequency, confidence intervals, and the LLM’s internal confidence, businesses can gain actionable insights in an otherwise unpredictable AI world.

Future Possibilities for LLM Ranking Systems in Business

1. Personalized Ranking Systems

As businesses gather more data on their customers, personalized ranking systems can be developed that tailor LLM-generated rankings to specific user preferences. For example, an e-commerce platform could use customer browsing history and purchase data to create a personalized list of recommended products.

2. Reinforcement Learning to Improve Rankings

Reinforcement learning techniques could be applied to improve LLM-generated rankings over time. As users interact with the rankings (e.g., by purchasing a recommended product or clicking on a suggested link), the system could learn from these interactions to improve future rankings.

3. Domain-Specific Rankings

Different industries require different ranking approaches. For example, in healthcare, ranking based on expert consensus might be more relevant than frequency alone. In marketing, SEO performance or customer engagement might be more important. Future ranking systems can incorporate domain-specific knowledge to improve the accuracy of LLM-generated rankings.

LLM-Generated Ranking Methods

1. List-based Ranking

The simplest way to rank results from LLMs is through ordered listed-based ranking. This method ranks entities (such as products, services, or websites) based on the order in which they appear in LLM-generated results across multiple runs.

How It Works:

• Run an LLM multiple times using the same query (e.g., "Best social media marketing tools").
• Record the list position each tool appears in the results.
• Average the list positions of each run. Track the highest and lowest list positions which ensures that the entity appearing most frequently at the top of the list has the highest position or best average.

Example: If a company runs an LLM 4 times with the query “Best digital marketing software” and Software A appears as first in the list 4 times while Software B appears as second in the list 4 times, Software A would be ranked higher than Software B.

Advantages:

• Closest equivalent to SEO rankings
• Reflects position of appearance, which implies the quality of the recommendation.

Disadvantages:

• List position doesn’t necessarily reflect the frequency of the results. An entity could appear at a higher list position due to bias in the LLM’s training data, not because it’s truly the best option.

2. Frequency-Based Ranking

Another way to rank results from LLMs is through frequency-based ranking. This method ranks entities (such as products, services, or websites) based on how often they appear in LLM-generated results across multiple runs.

How It Works:

• Run an LLM multiple times using the same query (e.g., "Best social media marketing tools").
• Record how often each tool appears in the results.
• Normalize the frequency using min-max normalization, which ensures that the entity appearing most frequently is ranked at the top.

Example: If a company runs an LLM 20 times with the query “Best digital marketing software” and Software A appears 15 times while Software B appears 10 times, Software A would be ranked higher than Software B.

Advantages:

• Simple and transparent.
• Reflects the frequency of appearance, which is crucial when LLMs generate variable results.

Disadvantages:

• Frequency doesn’t necessarily reflect the quality or relevance of the results. An entity could appear frequently due to bias in the LLM’s training data, not because it’s truly the best option.

3. Incorporating Confidence Intervals

While frequency-based ranking provides a clear picture of how often an entity appears, it doesn’t account for uncertainty in the model’s predictions. Incorporating confidence intervals helps address this issue by measuring the variability in the LLM’s results. Confidence intervals provide a range within which the true frequency of an entity is likely to fall, allowing businesses to gauge the reliability of the rankings.

How It Works:

• Calculate the estimated probability of each entity appearing in the results.
• Compute the standard deviation to measure the uncertainty in the estimate.
• Generate a confidence interval (e.g., 95% confidence interval) to indicate how stable the ranking is.

Example: If Supplier A appears in 70% of the LLM's results but with a wide confidence interval (e.g., 50% to 90%), the company might consider that Supplier A is less reliable than Supplier B, which appears in 60% of results but with a much narrower confidence interval (e.g., 58% to 62%).

Advantages:

• Quantifies the uncertainty in the rankings, helping businesses make more informed decisions.
• Reduces the influence of outliers or random variability.

Disadvantages:

• Confidence intervals require more complex statistical calculations, which can be harder to explain and understand.
• For small sample sizes, confidence intervals may be too wide to provide useful insights.

4. Averaging GPT-4's Internal Confidence

LLMs like GPT-4 assign internal probabilities to the tokens or entities they generate in a response. By averaging these internal confidence scores over multiple iterations, businesses can rank entities based on how confident the model is about each result.

How It Works:

• Track the internal confidence score GPT-4 assigns to each entity in each run.
• Average these confidence scores across multiple iterations.
• Rank entities based on their average confidence scores.

Example: If Product A consistently receives high internal confidence scores (e.g., 0.85) across multiple runs, while Product B receives lower scores (e.g., 0.65), Product A would be ranked higher in the final list.

Advantages:

• Directly reflects the LLM’s internal belief in its predictions.
• Helps businesses understand which results the model is most confident in.

Disadvantages:

• GPT-4’s confidence scores are based on its training data and may not always align with real-world relevance or quality.
• Internal confidence doesn’t necessarily reflect actual performance or correctness.

Challenges in Implementing LLM Ranking Systems for Business

1. Prompt Variability

The way a prompt is phrased can have a significant impact on the results generated by an LLM. Even slight changes in wording can produce vastly different responses, affecting the consistency of the frequency-based ranking system.

Solution: To mitigate prompt variability, businesses should use standardized prompts and run multiple variations of the same query to smooth out any differences. By averaging the results across different prompt variations, companies can obtain a more reliable ranking.

2. Web Browsing Variability

When LLMs are integrated with web browsing capabilities, they can return real-time data from the web. However, web content is dynamic and subject to change, which can lead to inconsistent results across different runs.

Solution: Implement time-based aggregation to control when web-based data is retrieved. By collecting results within a specific time window, businesses can ensure consistency in the data used for ranking. Web result caching can also be used to standardize the data across multiple runs.

3. Balancing List Position, Frequency, Confidence Intervals and LLM Confidence

A core challenge in creating an LLM ranking system is finding the right balance between these 4: list position, frequency, confidence interval and LLM confidence. While frequency-based rankings and list positions are useful, they may overemphasize entities that appear often but aren’t necessarily the best choice. Confidence-based rankings, on the other hand, may prioritize entities that the model believes in but aren’t frequently mentioned.

Solution: Create a hybrid ranking system that combines list position, frequency, confidence internal and LLM confidence. By assigning weights to both metrics, businesses can strike a balance that ensures both reliable and consistent results.

Today, that solution is called RankLens.