Comparison of Search API MRR

## Background

In modern information retrieval, the quality of search engine APIs directly impacts user experience and the accuracy of search results. To evaluate the performance of different search engine APIs, we compared the APIs of Bing, Google, and SearxNG, focusing on their accuracy in different language environments. Our evaluation metric is the Mean Reciprocal Rank (MRR), a commonly used indicator for measuring the quality of search results.

## Methodology

### Data Samples

We selected queries in four languages (Traditional Chinese, Simplified Chinese, Japanese, and English), with 50 queries for each language, totaling 200 queries. These queries cover a wide range of topics and complexities to comprehensively evaluate the performance of the search engine APIs.

### Testing Steps

1. Query Preparation: Prepare 50 queries for each language.

2. API Calls: Use the APIs of Bing, Google, and SearxNG to perform the queries.

3. Result Evaluation: Calculate the MRR value for each query.

4. Data Aggregation: Aggregate the MRR values for each API and perform a comparative analysis.

## Comparative Metrics

### Mean Reciprocal Rank (MRR)

MRR is a commonly used metric for evaluating the quality of search results. The formula for MRR is:

\[ \text{MRR} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\text{rank}_i} \]

where \( |Q| \) is the total number of queries, and \( \text{rank}_i \) is the rank of the first relevant result for the \( i \)-th query. A higher MRR value indicates that the search engine returns relevant results more prominently, reflecting better performance.

## Test Results

The table below shows the MRR values of different search engine APIs in various language environments:

| | Traditional Chinese | Simplified Chinese | Japanese | English |

|--------------------|---------------------|---------------------|----------|---------|

| Bing | 54.90% | 62.90% | 56.73% | 56.50% |

| Google | 43.37% | 54.40% | 41.90% | 39.50% |

| SearxNG (combined) | 51.57% | 66.20% | 60.13% | 59.07% |

| SearxNG (DuckDuckGo) 2S timeout | 40.67% | 46.97% | 55.83% | 47.30% |

| SearxNG (DuckDuckGo) | 39.83% | 53.07% | 56.67% | 55.87% |

### Results Analysis

1. Bing: Bing's MRR values are relatively stable across the four language environments, with particularly strong performance in Simplified Chinese (62.90%) and Traditional Chinese (54.90%).

2. Google: Google has relatively low MRR values in all language environments, especially in Japanese (41.90%) and English (39.50%).

3. SearxNG (combined): SearxNG, which uses multiple search engines (Google, Bing, DuckDuckGo, Baidu), performs best in Simplified Chinese (66.20%) and Japanese (60.13%).

4. SearxNG (DuckDuckGo): SearxNG using only DuckDuckGo, both with a 2-second timeout and without, shows lower MRR values, particularly in Traditional Chinese (40.67% and 39.83%).

## Conclusion

By comparing the MRR values of different search engine APIs, we can see that Bing and SearxNG perform relatively well in different language environments, with SearxNG performing best when combining multiple search engines. However, Google consistently underperforms across all language environments. Future work can further optimize the query set and evaluation metrics to obtain a more comprehensive performance assessment.