LLM LLM On the Wall, Who's the Best of Them All? Answer: It's Complicated!

The world finds itself characterized by a divide between two groups: those who possess a comprehensive understanding of artificial intelligence and those who utilize its capabilities without necessarily comprehending its underlying principles.

This trend has further fueled the proliferation of tools that make ambitious claims about their capabilities, such as detecting hallucinations in large language models, automating prompt engineering, identifying the most effective large language model, providing the safest large language model tool ever, detecting bias in large language models, enabling artificial intelligence to operate at an unprecedented level, creating AGI in your backyard and other such.

It is often less understood that the validity of the research underpinning those tools is limited to the specific scenarios and assumptions employed during their testing and the outcomes are usually non-transferable unless there exists an identical scenario with an identical set of assumptions in a business case. Consequently, there exists no universally applicable solution for determining the best large language model across all contexts and all business scenarios.

If models such as GPT 4, PaLM 2, Llama 2, Cohere, Claude, Mistral, Falcon and so on were not sufficient, there are over 38,000 language models in Huggingface.

There are a number of factors that need to be considered when evaluating large language models. These include the following.

• Accuracy: How accurate is the LLM at performing a given task?
• Fluency: How fluent is the LLM in its output for the given task?
• Creativity: How creative is the LLM in its output for the given task?
• Efficiency: How efficient is the LLM in terms of utilization of its computational resources for the given task?
• Interpretability: How interpretable is the LLM's output for the given task?

• Fairness and bias: LLMs can be biased, potentially reflecting any biases present in the data they were trained on. It is important to take steps to mitigate such biases for the given task.
• Transparency and explainability: It is difficult to understand how LLMs make their language predictions, so it's important to consider explainability mechanisms into the inference for a given task.
• Safety and security: LLMs can be used to generate harmful content. It is important to develop safeguards to prevent LLMs from being used for malicious purposes and to mitigate the risks of hallucination.

The overarching theme underlying the above points is the centrality of the "specific task" at hand. In other words, a model tuned for a particular task may effectively address the majority of the above requirements for that specific task. However, the same model or tuning technique may prove entirely inadequate for a different task and may fail to meet the majority of the aforementioned criteria for that new task.

A model tuned for excelling in generating creative text formats like poems or scripts might not necessarily outperform one designed for factual question answering. Similarly, the number of parameters may not necessarily play a crucial role in shaping an LLM's capabilities, while high quality, domain-focused and diverse (albeit relatively small) datasets used for tuning could lead to an improved performance, especially when coupled with tuning mechanisms like Distilling step-by-step.

Furthermore, the evaluation of LLM performance is often hindered by the lack of standardized benchmarks and metrics. While certain metrics, such as accuracy and fluency, can provide valuable insights, they fail to capture all the nuances of language and the ability to adapt to different contexts. Moreover, subjective factors like human judgment, uniqueness of business needs, regulatory needs of specific industries and a general aesthetic preference of specific scenarios can all influence the perceived quality of LLM output.

In the light of these complexities, attempting to identify a single LLM as the absolute best or the least hallucinating is an oversimplification. On the other hand, the choice of LLM should be tailored to the specific task and requirements, considering factors such as task performance needs, data availability, regulatory obligations, safety, privacy and computational resources.

While the pursuit of a universally superior LLM will remain an ongoing open-ended quest, the true potential of LLMs is not in searching for a single, dominant all-encompassing model, but in harnessing their adaptability to excel in diverse, task-specific applications.