The Fault in Gemini’s Stars
The excitement was on as soon as Google released Gemini after several delays. The company published an impressive demo video of the AI model answering questions by analysing drawings of ducks, playing rock-paper-scissors, and sorting out objects – all in real time!
It was even replying in human-like voices. Sundar Pichai posted on X saying, “Best way to understand Gemini’s underlying amazing capabilities is to see them in action, take a look ??”
But turns out, the demo was fake, and Google admitted to it.
The video was just a recording that had cherry-picked frames later edited to impress the crowd. Some Google employees said the video was created “using still image frames from the footage, and prompting via text”.
Google’s blog on Gemini explained how the researchers did multimodal prompting by showing images to the model along with a prompt, for it to give the correct answers.?
Moreover, the vision and speech capabilities showcased are not even present in the current version of the AI model. These models will be released next year, not right now. It was all just going overboard with marketing.
The benchmark problem
Google seems to have rushed the release of Gemini given all the pressure from the world, also just to compete with OpenAI – which it did, but only on paper.?
Gemini’s metrics were the talk of the tech world when Google published its technical paper. Gemini Ultra created a buzz as it outperformed OpenAI’s GPT-4 on various benchmarks, including MMLU.?
But turns out, even that was fake.
If one delves into the technical report, Google has cleverly employed COT@32 instead of 5-shot learning to enhance the perceived performance of Gemini. The model only beat GPT-4 on MMLU when uncertainty routed CoT@32 is used, which Google did through GPT-4 API, as the paper mentions.?
In short, Google invented a different methodology around CoT@32 to claim that it’s better than GPT-4, which clearly shows that something is fishy.
However, Gemini Ultra isn’t out yet. Who knows, it might actually be better than GPT-4. Google can only hope that OpenAI doesn’t release GPT-5 by then, which is already in the works.
Google is going to release different versions of Gemini slowly, starting with Gemini Pro, ideal for scaling across various tasks; Gemini Ultra, tailored for highly complex tasks; and Gemini Nano, a highly efficient model for on-device applications.
While Google has its “overhype” strategy of launching products, Mistral AI, the open source generative AI startup from France, released its model, Mixtral, just the next day. The best part is that the company posted it as a torrent link on X, a move, which clearly shows that the company knows its customers – a mic drop on Google’s “coming soon” models.
It is not like nothing changes. Now powered by Gemini Pro, Bard has gotten way better, ideally competing with ChatGPT-powered by GPT-3.5. What currently gives Bard an edge over ChatGPT is that the former is connected to the internet, giving real time information, which is similar to xAI’s Grok.
Can’t wait for the real Gemini to be out soon. Click here to read more.
AI Forum for India. Our Discord Community for AI Ecosystem, In collaboration with NVIDIA.?Join Today >>
领英推荐
AIM Videos >>
[Reaction Video] In this latest video, AIM follows up on the announcement of Google’s Gemini and ponders the potential impact of Gemini on the tech industry. The video also speculates the responses from other tech giants like OpenAI, Microsoft, Amazon, and Elon Musk.
Top Stories of the Week >>?
What is Meta doing with 150K GPUs?
According to a recent report, Microsoft and Meta have received 150,000 H100 GPUs each from NVIDIA, surpassing allocations to Google, Amazon, and Oracle. From Microsoft’s standpoint, it’s evident that they are supplying GPUs to support OpenAI in developing their upcoming models.
But the quantity is notably substantial for Meta, the parent company of Facebook and Instagram, especially considering that it’s not a cloud service provider like Microsoft, but rather the creator of the open-source Llama models. Click here to read more.
Lisa Su Takes AMD’s AI Beyond Just GPUs
AMD held its ‘Advancing AI’ event last week, which saw a bunch of announcements besides the launch of its Instinct MI300X AI accelerator. Lisa Su, the chair and CEO of AMD, announced its partnering with Microsoft, Meta, and Oracle, along with showcasing its capabilities with AI startups such as Databricks, Essential AI, and Lamini.
Leading OEMs, such as Acer, Asus, Dell, HP, Lenovo, and Razer, are all set to feature the new AMD Ryzen 8040 Series processors as well. Read the full story here.
How Redis Finds Moat in the Indian Market
India boasts a staggering Redis user base of approximately 12 million downloads per day, securing its position as the third-largest adopter on the planet, closely trailing behind the technology giants, the United States and China.
“Every month, half a billion people use Redis Enterprise in India alone,” co-founder and CTO Yiftach Shoolman told AIM during his recent visit to India. Click here to read the full interview.
AIM SHOTS >>