AI: There is a Measurement Transparency Gap

Which AI system writes the best computer code or generates the most realistic image?? Right now, there isn't a straightforward way to answer this question.

As someone who's been diving deep into AI over the last year, I must admit it’s been a wild ride. It seems like every day; a friend or colleague is asking me which AI tool they should use. For example, "Hey, does ChatGPT or Gemini write better code?" or "Is DALL-E 3 or Midjourney better at making realistic images of people?"

And honestly, I'm just like, "It’s hard to say for certain." Even though I'm researching AI tools left and right, it's challenging to stay on top of the relative strengths and weaknesses of various AI products. The tech companies behind these tools aren't exactly handing out user manuals or detailed release notes on what's new. Plus, they're updating the AI models so fast that a chatbot might struggle with a prompt one day and then totally nail it the next.

The Stanford University 2024 AI Index report states that poor measurement is one of the biggest challenges facing AI researchers. The lack of proper measurement is not merely an inconvenience; it also poses a significant safety risk. Without robust testing methods for AI models, it becomes difficult to identify which capabilities are advancing or which products might present genuine threats of harm.

But here's the thing – not having good ways to measure these AI models isn't just annoying, it's somewhat dangerous. Without solid test methods, we can't tell which AI capabilities are growing faster than we expected, or which products might end up causing threats or real harm.

So, my advice? don't be afraid to admit when you're not sure which AI to use. We're all learning as we go in this brave new world of artificial intelligence. It's crucial to remain open to learning as we progress.

https://aiindex.stanford.edu/report/ ?? #cambridgetransformationpartners