How do you evaluate ranking and recommendation metrics?

1 Relevance metrics

Relevance metrics assess how well the ranked or recommended items match the user's preferences or needs. Some examples of relevance metrics are precision, recall, F1-score, and mean average precision (MAP). These metrics are usually computed at different levels of granularity, such as per query, per user, or per system. Relevance metrics are easy to calculate and interpret, but they do not capture the order or diversity of the items, nor the user satisfaction or behavior.

2 Ranking metrics

Ranking metrics measure how well the order of the items reflects their importance or utility for the user. Some examples of ranking metrics are normalized discounted cumulative gain (NDCG), mean reciprocal rank (MRR), and rank-biased precision (RBP). These metrics are usually computed based on the position of the relevant items in the ranked list, and they often use a discount factor to penalize lower ranks. Ranking metrics are more sensitive to the order of the items than relevance metrics, but they still do not account for the diversity or novelty of the items, nor the user feedback or engagement.

3 Recommendation metrics

Recommendation metrics evaluate how well the system suggests novel and diverse items that the user might like or benefit from. Some examples of recommendation metrics are serendipity, coverage, diversity, and novelty. These metrics are usually computed based on the similarity or dissimilarity of the recommended items to the user's profile, history, or expectations. Recommendation metrics are more focused on the quality and variety of the items than relevance or ranking metrics, but they are harder to define and measure, and they may require additional data or user feedback.

4 Offline evaluation

Offline evaluation is the most common and convenient method to evaluate ranking and recommendation systems. It involves using historical or simulated data to compare the performance of different systems or algorithms based on predefined metrics. Offline evaluation is fast, cheap, and reproducible, but it has some limitations. It may suffer from data bias, sparsity, or staleness, it may not reflect the real user behavior or feedback, and it may not capture the dynamic and interactive nature of ranking and recommendation systems.

5 Online evaluation

Online evaluation is the most realistic and reliable method to evaluate ranking and recommendation systems. It involves conducting experiments with real users in a live or controlled environment to measure the impact of different systems or algorithms on user behavior, satisfaction, or outcomes. Online evaluation is more accurate, relevant, and generalizable than offline evaluation, but it also has some challenges. It may require large and representative samples, ethical approvals, and careful designs, it may be costly, risky, or disruptive, and it may be affected by external factors or noise.

6 Hybrid evaluation

Hybrid evaluation is a method that combines offline and online evaluation to leverage their strengths and overcome their weaknesses. It involves using offline evaluation to filter out or optimize the candidate systems or algorithms, and then using online evaluation to validate or compare them with real users. Hybrid evaluation is more efficient, flexible, and robust than offline or online evaluation alone, but it also requires more coordination, integration, and alignment between the two methods. Hybrid evaluation is a promising approach to evaluate ranking and recommendation systems in a comprehensive and rigorous way.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?