Scale AI

Today, we’re announcing Scale has closed $1B of financing at a $13.8B valuation, led by existing investor Accel. For 8 years, Scale has been the leading AI data foundry helping fuel the most exciting advancements in AI, including autonomous vehicles, defense applications, and generative AI. With today’s funding, we’re moving into the next phase of our journey: accelerating the abundance of frontier data to pave the road to Artificial General Intelligence (AGI). “Our vision is one of data abundance, where we have the means of production to continue scaling frontier LLMs many more orders of magnitude. We should not be data-constrained in getting to GPT-10.” - Alexandr Wang, CEO and founder of Scale AI. This new funding also enables Scale to build upon our prior model evaluation work with enterprise customers, the U.S. Department of Defense, and collaboration with the White House to deepen our capabilities and offerings for both public and private evaluations. There’s a lot left to do. If this challenge excites you, join us: https://scale.com/careers Read the full announcement: https://lnkd.in/gVBhaPZ5

Today, we’re excited to announce our strategic partnership with Inception, to accelerate global AI adoption across public and private sectors. By combining Scale and Inception’s expertise, we’re driving the next wave of AI innovation—starting in the UAE. Learn more: https://lnkd.in/ejriMvKs

We’ve added GPT-4.5 to the SEAL LLM Leaderboards. GPT-4.5 ranks 2nd on the Agentic Tool Use - Chat Leaderboard! View the rankings: HLE: https://lnkd.in/gKSZptdg EnigmaEval: https://lnkd.in/g6qEfqCS? Visual-Language Understanding (VISTA): https://lnkd.in/g43qgC_R? Multichallenge: https://lnkd.in/gZVmy6SW Agentic Tool Use (Enterprise): https://lnkd.in/g4FhA55w Agentic Tool Use (Chat): https://lnkd.in/gjCiU4j6

In collaboration with Center for Strategic and International Studies (CSIS), we’ve developed the Critical Foreign Policy Decision (CFPD) Benchmark—a new way to assess LLMs based on their national security and foreign policy decision-making tendencies. This benchmark systematically assesses how LLMs respond to complex geopolitical prompts and offers insight into models’ geopolitical preferences that are critical to understand. Learn more: https://lnkd.in/gWuTp4mM

Claude 3.7 Sonnet is rolling out now on the SEAL LLM Leaderboards, starting with Humanity’s Last Exam. Throughout this week, we will continue to add Claude 3.7 Sonnet to the remaining leaderboards. Stay tuned. View the rankings: ?? HLE: https://lnkd.in/gKSZptdg

Yesterday we announced a strategic, long-term partnership with the Ministry of Communications and Information Technology to drive Qatar’s digital transformation. Together, Scale and MCIT will work together to support Qatar’s National Development Strategy and Digital Agenda 2030. This is a significant milestone in Qatar’s efforts to modernize government operations, enhance public services, and build a future-ready workforce. A key part of the partnership will be the development of an AI talent ecosystem in Qatar. This initiative aims to empower government employees, students, and young professionals with the skills and expertise needed to succeed in an AI-driven digital landscape. Learn more: https://lnkd.in/d9MdZWp9

Our latest collaboration with the Defense Innovation Unit (DIU) and the United States Air Force showcases how AI-driven computer vision can autonomously detect potential maritime threats, bolstering national security without increasing manpower. We developed a bespoke dataset of 88,000 maritime vessel labels, allowing our AI models to distinguish between authorized and unauthorized vessels with greater accuracy, providing more tools to those who keep us safe. Learn more: https://lnkd.in/gzU57zjs

New SEAL Leaderboard today! Introducing EnigmaEval, a benchmark assessing models’ puzzle-solving capabilities. o1 leads. Puzzles demand sophisticated reasoning, creative problem-solving, and the ability to synthesize information across diverse fields—from mathematical and logical reasoning to cultural knowledge and linguistic manipulation. EnigmaEval is part of a new class of extremely challenging benchmarks that expose current models' limitations. The benchmark comprises 1184 puzzles collected from eight diverse sources. View the leaderboard: https://lnkd.in/g9bcDy5X

Our researchers have developed a novel approach to scale vulnerability testing that involves teaching LLMs to systematically break through their own safeguards and those of other models. We are calling this new method of safety testing J2 – Jailbreaking-to-Jailbreak. Our J2 approach combines the strategic reasoning of human red-teamers with the scalability of automated methods, achieving up to 94% attack success rate. With this method, builders no longer have to decide between resource-intensive testing or less effective automated testing. Read the paper here:? https://scale.com/blog/j2

At Scale, our research challenges fundamental assumptions about how AI systems develop capabilities while establishing new frameworks for their evaluation and improvement. Take a metaphorical tour of our lab in today's blog. We cover our research team’s latest findings from critical focus areas: ? Post-training optimization and model behavior ? Advanced reasoning and problem-solving capabilities ? Multimodal systems and hallucination challenges ? Data quality and interpretability ? Safety evaluation frameworks ? Agent capabilities and robustness ? Scalable oversight methods Read here: https://lnkd.in/gpa5WRG7