Making a Better AI to Solve the World’s Greatest Problems - How Much 2022 Training Compute (in FLOP) and Data Is Needed? – What is Quetta (Q)
Doug Hohulin
To Save 1 Billion Lives with AI, Exponential Blueprint Consulting LLC, President/Founder, When the AI System Has to Be Right: Healthcare, AV, Policy, Energy. Co-Author of 2030: A Blueprint for Humanity's Exponential Leap
In the next 10 years, we'll increase the computational capability for deep learning by another million times. - CEO Jensen Huang
Short version of the talk?
"Sam Altman, the CEO of OpenAI recently came on Stanford eCorner’s talk and said, “Whether we burn $500 million, $5 billion, or $50 billion a year, I don’t care. I genuinely don’t as long as we can stay on a trajectory where eventually we create way more value for society than that and as long as we can figure out a way to pay the bills,” he said.?
“We are making AGI, and it is going to be expensive and totally worth it,” he added." - "I don't care if we burn $50 billion a year, we're building AGI," says Sam Altman (analyticsindiamag.com)
This LinkedIn article is a review of the paper “Future-Proofing AI Regulation,” a new report from CNAS, finds the cost of training an AI system of a given level of capabilities in the current paradigm fall by a factor of ~1000 over 5 years.
Future-Proofing Frontier AI Regulation | Center for a New American Security (en-US) (cnas.org)? by By:?Paul Scharre Paul Scharre
“Research Questions This paper aims to answer several questions about how trends in cost and compute could affect the future of AI:
This is a must-read paper to better understand how AI can help solve some of the world’s greatest problems.??
I asked the question: What Problem Are You Trying to Solve? ?Can AI Help You Solve It Better with CO-INTELLIGENCE (see the book by ETHAN MOLLICK) Ethan Mollick
Will better AI Solve?Ignorance, Disease, Poverty, Hunger, and War in the next 10 years? Or Will AI make these problems worse? If AI can solve these problems, I would call that AGI and maybe even ASI.
?“The cost of making almost everything is mostly energy and intellect, not raw materials. The energy—sometimes human exertion, sometimes mechanical exertion—has always come at a cost. But what if that energy cost fell to zero? As my economics professors insisted, cost is determined by scarcity and demand. But is energy really scarce—or is it like air? Is it finite, or is it for all practical purposes infinite??-? Byron Byron’s book: Infinite Progress: How the Internet and Technology Will End Ignorance, Disease, Poverty, Hunger, and War Byron Reese
Per the book: A Question of Power: Electricity and the Wealth of Nations by ?Robert Bryce Robert Bryce ??
He discusses "THE TOP TEN PROBLEMS FACING THE WORLD “ Figure 4
1.???? Energy
2.???? Water
3.???? Food
4.???? Environment
5.???? Poverty
6.???? Terrorism and war
7.???? Disease
8.???? ?Education
9.???? Democracy
10.? ?Population
How Much 2022 Training Compute in FLOP Will Be Needed to Solve these Problems.? If we can solve these problems, will we call the AI: AGI/ASI?? If we have a Quetta Q 10^30 Flops of Training Compute by 2030, how likely will it be to solve these problems?
?Author’s note: I find it interesting the New SI unit name: Quetta Q 10^30 Latin: “ten” — (10^3)10
PREFIX SYMBOL FACTOR??? NAME
Quetta? ????? ?Q?? ?????? 10^30???????? nonillion
Humanity is or will be living in the world of large numbers: giga, tera, peta, exa, zetta, yotta, ronna, quetta.
Fun facts, there are ~5 Q (nonillion or Quetta) bacteria (10^30) on earth?
Total energy output of the Sun each day ?33 QJ (3.3×10^31J ) Orders of magnitude (energy) - Wikipedia
If this is the first time you heard of Q, let me know.? Will we be living in the Q Continuum soon?
?In 2017, I wrote the article: Are You Smarter than your Grandparents? Intelligence X - Intelligence Augmentation IA- Collaborated | LinkedIn
How to Make AI Better (Question asked and discussed by Byron Reese)
??????? More Processing Power – Training Compute
??????? Better Algorithms
??????? More Data / Better – Communications
Human Intelligence
??????? Natural Ability – Effort to Learn
??????? Better Education – Standing on the Shoulder of Giants
??????? More Education - Time
??????? Intelligence Augmentation (IA) - Standing on the Shoulder of AI Giants
The following are quotes and charts I found most interesting in the paper: Future-Proofing Frontier AI Regulation:
“The effective compute used to train frontier models is projected forward over time, starting from an initial estimate of 2.1 × 10^25 FLOP to train GPT-4 in 2022 and using a 7.0-month doubling rate for compute and an 8.4-month doubling rate for algorithmic efficiency (95 percent CI: 5.3 to 13 months).”
?“Accounting for algorithmic progress, effective compute could increase by 2030 to approximately 1 millionfold above GPT-4, to around the equivalent of 10^31 FLOP in 2022. If algorithms continue to improve, effective compute could increase by the mid-2030s to approximately 1 billionfold above GPT-4, to around the equivalent of 10^34 FLOP in 2022."
?“If current trends continue, using GPT-4 as a starting point, the 10^26 FLOP threshold is projected to be reached in early 2024 with a cost of roughly $155 million (95 percent CI: $135 million to $170 million) for the final training run. (In practice, the actual timing for when the first model will cross the notification threshold will depend on the compute availability, experiment pipeline, and strategies of the handful of labs that are able to train a 10^26 FLOP model today.)
Initially, only a small number of leading AI labs would be able to train such a model. Training costs would fall over time, however, due to hardware improvements. Machine learning GPU price-performance is doubling every 2.1 years. Figure 7.1 projects the cost to train a 10^26 FLOP model declining over time due to hardware improvements, starting with a $155 million cost in 2024.
Under this projection, the cost to train a 10^26 FLOP model would decrease relatively slowly from $155 million in 2024 to $30 million five years later in 2029.
This would limit the number of actors that could train a model at the regulatory threshold.”
领英推荐
?"The estimated cost limit for private companies, $20 billion, is reached in the late 2020s or early 2030s under all growth rates. Similarly, 10^28 FLOP, or approximately 1,000 times the compute used to train GPT-4, is reached in the late 2020s under all compute growth rate projections.
?The Paper "Future-Proofing AI Regulation" by Paul Scharre from the Center for a New American Security, ?provides analysis into the trends in the costs and computational needs for training advanced AI models, and using this paper I offer a glimpse into how AI could potentially tackle global challenges such as energy scarcity, water shortage, poverty, and more.
The paper poses pivotal research questions aimed at understanding the trajectory of AI development costs and the implications for future regulatory measures. It explores how algorithmic improvements and hardware advancements might alter the landscape of AI capabilities and accessibility. Particularly compelling is the discussion on the limitations of hardware improvements and the impact of these constraints on various actors, including those restricted by hardware availability due to policies like export controls.
As we stand on the brink of an AI revolution, the possibility of AI systems achieving or surpassing human-level capabilities in various domains raises both opportunities and challenges. The notion of AI contributing to solving existential problems such as disease, poverty, and war—or exacerbating them—is a dual-edged sword that requires thoughtful consideration and proactive governance.
Drawing from the discussions in the paper and further enriched by insights from Byron Reese's "Infinite Progress" and Robert Bryce's "A Question of Power," it becomes evident that the integration of AI could dramatically shift our approach to these global challenges.
Call to Action:
·?????? Professionals and leaders in AI, technology, and policy-making are invited to engage with this critical dialogue.
·?????? As we navigate this rapidly evolving field, your insights and contributions are vital in shaping a future where AI not only enhances our capabilities but does so in a way that is beneficial and equitable for all. Join the conversation on how we can collaboratively ensure that AI serves as a force for good, helping to alleviate the pressing challenges of our times.
·?????? Let's not just imagine a future dominated by advanced AI—let's actively participate in creating it.
·?????? Engage with the full article on CNAS and join in shaping the path forward for AI in solving the world’s greatest problems.
Additional information
The Secretary of Commerce, in consultation with the Secretary of State, the Secretary of Defense, the Secretary of Energy, and the Director of National Intelligence, shall define, and thereafter update as needed on a regular basis, the set of technical conditions for models and computing clusters that would be subject to the reporting requirements of subsection 4.2(a) of this section.? Until such technical conditions are defined, the Secretary shall require compliance with these reporting requirements for:
? ? ?? ? ?(i)? ?any model that was trained using a quantity of computing power greater than 10^26?integer or floating-point operations, or using primarily biological sequence data and using a quantity of computing power greater than 10^23?integer or floating-point operations; and
? ? ?? ? ?(ii)? any computing cluster that has a set of machines physically co-located in a single datacenter, transitively connected by data center networking of over 100 Gbit/s, and having a theoretical maximum computing capacity of 1020?integer or floating-point operations per second for training AI.
Threshold of 10^25 FLOPs for a default categorization of systemic risk models is too high. (see the very useful study by The Future Society: EU AI Act Compliance Analysis: General-Purpose AI Models in Focus (thefuturesociety.org)?
I am on Team Human - Being Pro-Human - As We Solve Some of the World's Largest Problems
“Discussions that I had with Larry Page, I talked to him about AI safety, and Larry did not care about AI safety, or at least at the time he didn’t. And at one point he called me a speciesist for being pro-human, and I’m like, well, what team are you on, Larry? He’s still on Team Robot to be clear. And I’m like, okay. So at the time Google had acquired DeepMind, they had probably two thirds of all AI researchers in the world. They had basically infinite money and compute, and the guy in charge, Larry Page, did not care about safety and even yelled at me and caught me a speciesist for being pro-human.” - Elon Musk: War, AI, Aliens, Politics, Physics, Video Games, and Humanity | Lex Fridman Podcast #400 - Lex Fridman
California’s new AI bill?“Is this GPT-5?!”, Inside Microsoft’s $10B Deal with OpenAI, California Bill 1047 (youtube.com)? ?09:52 California Bill 1047
(b)?“Artificial intelligence model” means a machine-based system that can make predictions, recommendations, or decisions influencing real or virtual environments and can use model inference to formulate options for information or action.
(d)?“Computing cluster” means a set of machines transitively connected by data center networking of over 100 gigabits that has a theoretical maximum computing capacity of 10^20 integer or floating-point operations per second for training artificial intelligence.
(f)?“Covered model” means an artificial intelligence model that meets either of the following criteria:
(1)?The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024, or a model that could reasonably be expected to have similar performance on benchmarks commonly used to quantify the performance of state-of-the-art foundation models, as determined by industry best practices and relevant standard setting organizations.
(2)?The artificial intelligence model has capability below the relevant threshold on a specific benchmark but is of otherwise similar general capability.
(n)?(1)?“Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)?The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)?At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)?At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)?Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
Author's Note: My question, Is this not Illegal Under Federal or California Law Already?
AUTOCOMPLETE EVERYTHING: THE RISE OF LARGE LANGUAGE MODELS
Exa FLOP (10^18) x? 2.6 M Sec/Month (10^6) = 10^24
“Today at inflection my new company, we have one of the largest superclusters in the world, the second largest and
We now train our current model of Pi is 10 billion (10^9) Petaflops (10^15) so
10 billion million billion (10 x 10^9 x 10^15? = 10^25)
It's a very basically impossible number to wrap your head around a lot but just to see how it's changed so 10x more compute every single year for the last 10 years so 10x 10x 10x 10x 10x for 10 years exponential growth that's exponential growth.
full executive order reporting requirement:
- any model trained with ~28M H100 hours, which is around $50M USD or
- any cluster with 10^20 FLOPs, which is around 50,000 H100s, which only two companies currently have
The Memo - 30/Aug/2023 - by Dr Alan D. Thompson (substack.com) Alan D. Thompson
Dr Alan D. Thompson's presentation? Devoxx presentation - What’s in my AI? (VIMA, Gato, GPT-2, GPT-3, PaLM, Chinchilla) 36 minutes
https://www.youtube.com/watch?v=ODLjaoWxT98? https://lifearchitect.ai/chatgpt/? https://lifearchitect.ai/
Forget About Your Real Data — Synthetic Data Is the Future of AI Synthetic data will become the main form of data used in AI source gartner?? Leinar Ramos,? Jitendra Subramanyam?? 24 june 2021 https://www.gartner.com/en/documents/4002912