AI Showdown: Which Language Model Gets It Right?
Putting AI Models to the Test with Simple Questions
As we approach the two-year anniversary of ChatGPT's debut, it's astonishing to reflect on how rapidly conversational AI capabilities have evolved.
I've been probing these advancements to see just how far AI systems have come in creativity, problem-solving, reasoning, and common knowledge.
As the developer of the Kognetiks Chatbot for WordPress plugin, I’m very curious to see how different models handle tasks that seem simple to humans but can be challenging for AI.
I’ve been on the lookout for interesting prompts and found this one on Quora that challenges AI models' reasoning and counting abilities.? It’s a simple yet effective test of AI capabilities.
Come up with two phrases, phrase 1 and phrase 2.? Phrase 1 should have a word count that is two times the word count of Phrase 2.
I put this prompt to the test with three leading Large Language Models (LLMs).? I included three LLMs that I have access to:
With each of these LLMs, I tested the same prompt.? The outcomes are detailed below.
OpenAI – o1-preview
For the first test, I chose OpenAI’s o1-preview model as it is purported to be the best model for tasks that require reasoning, as well as handling brainstorming, coding, mathematics and more.
When prompted, the model responded with the following:
You can see from the screen capture (above) the process that the model followed.? OpenAI’s o1-preview model now explains itself.? The explanation is a relatively new feature and provides insight into just how it arrived at the answer.
The model responded to the prompt by correctly generating two phrases, one with six words and one with three words.? Here is the model output:
Assessment: OpenAI’s model response was correct with six words in phrase one and three in phrase two.
Anthropic – Claude 3 Haiku
For the second test, I chose Anthropic’s Claude 3 Haiku model as it is the best one that I can access.? The Haiku model is one of three Claude models which include successively more powerful models called Claude 3 Sonnet and Claude 3 Opus.
When prompted, the model responded with the following:
You can see from the screen capture (above) the process that this model followed.? The explanation for how it solved the problem was just as insightful as that provided by OpenAI’s model response.?
Interestingly the model acknowledged that it made a mistake in counting and attempted to correct the mistake.? Even after Claude 3 Haiku’s response “Now I’ll verify again”, it returned an incorrect answer.? Here is the model output:
Assessment: Anthropic’s model response was incorrect, returning seven words in phrase one and three words in phrase two.
领英推荐
NVIDIA – Llama-3.1-Nemotron-70B-Instruct
For the third and final test, I chose the NVIDIA Llama-3.1-Nemotron-70B-Instruct model because it NVIDIA states that it has been customized in order to improve its usefulness.
When prompted, the model responded with the following:
As you can see from the screen capture (above) the process of arriving at an answer is not extensively described in the model’s response.? However, it should be noted that other interactions I’ve had with NVIDIA’s model have been more verbose.
As you can see the model responded with seven words.? Here is the model output:
Assessment: NVIDIA’s model response was incorrect, returning seven words in phrase one and three words in phrase two.
Key findings and Implications
The prompt is a simple test but a revealing one.
Two out of the three models failed to return the right answer.
The miscounts encountered by Anthropic’s model and NVIDIA’s model suggest limitations in their model’s short-term memory or counting abilities, highlighting areas for improvement in these types of tasks.
For end-users relying on AI for accurate information, these discrepancies can lead to misunderstandings or misinformation.
LLMs can generate responses and outputs that may be inaccurate.? And there are disclaimers that these models can make mistakes.? And important results should be checked for completeness and accuracy.
These results highlight that even the most advanced AI models can struggle with basic reasoning tasks, which may have significant implications for their use in critical applications.
What does this mean for the future of Conversational AI?
The results suggest that some models lag other models when it comes to reasoning and accurate problem solving, especially for conversational AI.
This didn’t come as a surprise to me, nor should it to you.
The model makers continue to tout the capabilities of their newly released products as they pursue delivery of the smartest, most capable models at the lowest cost.
Scaling laws in AI suggest that increasing a model's parameters and data can lead to predictable improvements in performance. ?Huang's Law observes that GPU performance doubles approximately every two years, accelerating AI training capabilities.
Researchers are working on metrics to assess performance of LLMs as they improve with increased computational resources, model size, and data.
While there’s no direct equivalent yet to Moore’s Law for LLMs, larger models and more data generally lead to better results.? It should be noted that the focus has shifted to more efficient algorithms and architectures.
So, what does this tell us about the abilities of artificial intelligence so far?
While AI has made significant strides, simple tests, such as the one demonstrated here, reveal that even the most advanced models have room for improvement. ?Despite rapid advancements and groundbreaking achievements in artificial intelligence, I still find myself asking: Are we there yet?
Have you had similar experiences with AI models? ?Share your thoughts in the comments.
#AI #AGI #LLAMA #ChatGPT #Claude #LLM #BenchMarks #Kognetiks
Was this helpful?? ???Repost if it resonates.? Follow Stephen Howell for more.
Empowering digital experiences with conversational AI.
1 个月I'm excited to share my latest experiment comparing leading AI models on a simple yet revealing task. The results might surprise you! ???? Dive into the article and let's discuss which AI you think is leading the pack.