State of AI 2024: open (weight) source LLMs now run similar to GPT-4o (and we got it)
According to recent July 2024 news Mistral AI just released the open (weight) source Large 2 (ML2) model which is just slightly worse than the more famous OpenAI GPT-4o model.?
ML2, a language model boasting 123 billion parameters, has demonstrated impressive capabilities in the Massive Multitask Language Understanding (MMLU) benchmark, achieving a score of 84%. While falling slightly short of competitors like GPT-4 and Llama 3.1, it's important to note that even human domain experts are estimated to score around 89.8% on this challenging test.
ML2's smaller size offers distinct advantages in terms of deployment and commercialization. Requiring approximately 246GB of memory at full 16-bit precision, it is significantly more efficient compared to larger models like Llama 3.1, which demands over a terabyte (you need above TB of memory for Llama 3.1 or reduced bit precision). This efficiency makes ML2 a compelling option for various applications with constrained computational resources.
And..? we said that in January 2024: in our State of AI report (here), we predicted that open source (weight) models will soon compete with proprietary companies (prediction 5). And it is happening in front of our eyes.
The interesting fact is, Mistral Large 2 is way cheaper than GPT-4 (3x to 10x depending on benchmarks and tasks).?
The question is … are we going to see people switching from OpenAI or 微软 Copilot to Mistral AI or other open (weight) source because of that?
No, not really.
Many reasons for this and the actual price may not be that relevant.
I will explain why in another post soon.
Have great holidays.
#ai #artificialintelligence #business #technology #innovation
CPM | IBP | Professional Services
2 个月Another one can’t differentiate between open source and open weights. Makes whatever follows very trustworthy.
Leadership and Keynote Speaker and member of the Data Science Research Centre at University of Derby
2 个月What is obvious is that all LLMs are now reaching parity. However, they are no longer advancing in capability. They have plateaued and will not improve however much material or training compute.
Sr. Business Development Executive at VKAPS IT Solutions Pvt. Ltd.
2 个月It's fascinating to see how open source LLMs are rapidly advancing and closing the gap with proprietary models like GPT-4o. Your insights on their performance would be greatly valued, Dr Andrea Isoni.