ChatGPT vs Gemini: What Does a Game Changer in AI Look Like?

TLDR: Look at the time dependence of the distribution of metrics from an ensemble of strategies

When I evaluate potential game changers, I look for transformative leaps in performance. I focus on advancements that go beyond marginal gains and instead concentrate on delivering substantial improvements, having a tangible impact on users or industries. It's these step changes that capture my attention and disrupt the status quo, pushing the boundaries of what is possible.

Let’s look at a recent Kaggle competition as an example how game changes look in the open source world. Kaggle competitions attract data scientists and machine learning experts like me who strive to achieve significant improvements in performance metrics. These competitions often showcase game-changing approaches and techniques that push the boundaries of what is currently possible in the field of AI.

The graph of the leaderboard over time show a solution philosophy and general approach was adopted around July 1. A clear game changer on July 17 diffused through the community before settling in for a few weeks with some people holding their cards close until about August 16. The winners held back a few critical details until the competition closed.

While incremental improvements in the third decimal place can be valuable for fine-tuning existing technologies, they typically don't qualify as game changers. True game changers redefine the landscape, introducing novel approaches, capabilities, or functionalities that have a profound impact on how things are done. By recognizing and celebrating these step changes in performance metrics, the technology community encourages further innovation and sets the stage for future breakthroughs.