FOD#55: AI Factories, New Industrial Revolution, and Look Back

+ cool news from the usual suspects and the best weekly curated list of research papers

Next Week in Turing Post:

• Wednesday, AI 101:?In the third episode, we discuss PEFT – Parameter-Efficient Fine-Tuning technique;
• Friday: A new investigation into the world if AI Unicorns.

A new industrial revolution is unfolding, driven by the rise of AI factories. These facilities are changing computing across all scales, from massive data centers to everyday laptops, that very likely soon all turn into AI laptops. At Computex 2024, Jensen Huang highlighted this shift, emphasizing the need for industry-wide cooperation: meaning that hardware vendors, software developers, and enterprises have to make collective effort to transition from data centers to AI factories. Jensen Huang manifests that transformation is not just about technology but about reshaping the entire computing landscape. He usually knows what he says.

At the Nvidia pre-brief, the executives underlined the significant focus on the AI PC, a technology Nvidia introduced six years ago, in 2018. This innovation has revolutionized areas such as gaming, content creation, and software development.

AI PCs were not something widely discussed over the last six years, but now –thanks to Microsoft and Nvidia – they are becoming ubiquitous. Along with conversations about the new industrial revolution. And while still at the threshold, it is indeed important to remember history. In 2018 and early 2019, another significant event sent shockwaves through the ML community. This event made possible the groundbreaking milestone: ChatGPT. Let’s walk through this timeline :

• 2018 the creation of GPT (generative pre-trained transformer) led to –>
• Feb, 2019 GPT-2 – an LLM with 1.5 billion parameters. It was not fully released to the public, only a much smaller model was available for researchers to experiment with, as well as a technical paper , that led to –>
• 2020 GPT-3 and the paper Language Models are Few-Shot Learners , which evolved to –>
• 2022 GPT-3.5 and the fine-tuned version of GPT-3.5, called InstructGPT with the research paper Training language models to follow instructions with human feedback .
• Nov, 2022 – ChatGPT. It was trained in a very similar way to InstructGPT, the magic behind it is based on the research paper Deep reinforcement learning from human preferences , the technique is called reinforcement learning from human feedback (RLHF).

Jack Clark, now a co-founder of Anthropic and formerly the Policy Director of OpenAI, reflected today on the launch of GPT-2, which he described as "a case of time travel." In 2019, OpenAI's decision to withhold the full release of GPT-2 due to concerns about misuse sparked a loud debate within the AI community. This debate centered on balancing innovation with ethical responsibility. Critics argued that withholding the model could slow scientific progress, while supporters applauded OpenAI's cautious approach.

Jack argues that departing from norms can trigger a counterreaction. By gradually releasing GPT-2, OpenAI unintentionally fueled interest in developing open-source GPT-2-grade systems, as others aimed to address the perceived gap. If GPT-2 had been fully released from the beginning, there might have been fewer replications, as fewer people would have felt compelled to prove OpenAI wrong.

There are many interesting questions in the Clark’s reminiscence of that turbulent times. It’s worth reading in full but here are a few quotes for your attention:

• “Just because you can imagine something as being technically possible, you aren't likely to be able to correctly forecast the time by which it arrives nor its severity.”
• "I've come to believe that in policy 'a little goes a long way' – it's far better to have a couple of ideas you think are robustly good in all futures and advocate for those than make a confident bet on ideas custom-designed for one specific future."
• "We should be afraid of the power structures encoded by these regulatory ideas and we should likely treat them as dangerous things in themselves. I worry that the AI policy community that aligns with long-term visions of AI safety and AGI believes that because it assigns an extremely high probability to a future AGI destroying humanity, that this justifies any action in the present."
• "Five years on, because of things like GPT-2, we're in the midst of a large-scale industrialization of the AI sector in response to the scaling up of these ideas. And there's a huge sense of deja vu – now, people (including me) are looking at models like Claude 3 or GPT-4 and making confident noises about the technological implications of these systems today and the implications of further scaling them up, and some are using these implications to justify the need for imposing increasingly strict policy regimes in the present. Are we making the same mistakes that were made five years ago?"

We don’t have an answer but can certainly make a few confident noises about this new industrial revolution, made possible due to scaling laws and now powered by AI factories. People like Jensen Huang argue that we are at the moment of redefining what is possible in technology. What do you think? To see the bigger picture of the future, we – as always – encourage you to know the past.

Additional reading: And even play with the past, like Andrej Karpathy did: he just released a method to train GPT-2 models quickly and cost-effectively. Training a tiny GPT-2 (124M parameters) takes 90 minutes and $20 using an 8xA100 GPU. The 350M version takes 14 hours and $200, while the full 1.6B model requires a week and $2.5k. The method uses Karpathy’s llm.c repository, which leverages pure C/CUDA for efficient LLM training without large frameworks.

Twitter Library

News from The Usual Suspects ?

The State of AI in Early 2024

• According to McKinsey, the adoption of generative AI is spiking, beginning to generate substantial value.

OpenAI's: Threat Actors, Security Committee and Back to Robotics

• OpenAI has released a detailed report highlighting the use of its AI models in covert influence operations by threat actors from Russia, China, Iran, and Israel. These operations, aimed at manipulating public opinion and political outcomes, were largely ineffective in engaging authentic audiences despite increased content generation. In response, OpenAI has implemented measures such as banning accounts, sharing threat indicators, and enhancing safety protocols. Notable campaigns include "Bad Grammar" (Russian), "Doppelganger" (Russian), "Spamouflage" (Chinese), "IUVM" (Iranian), and "Zero Zeno" (Israeli). This underscores the dual role of AI in both conducting and defending against covert IO, highlighting the necessity of a comprehensive defensive strategy.
• They also established a Safety and Security Committee to address critical safety concerns. Led by Bret Taylor, Adam D’Angelo, Nicole Seligman, and Sam Altman, this committee is tasked with making safety recommendations within 90 days, supported by consultations with cybersecurity experts. These recommendations will be publicly shared to ensure robust safety and security measures across OpenAI’s projects.
• In other news, OpenAI has relaunched its robotics team and is currently hiring . From the past (also 2018): L earning Dexterous In-Hand Manipulation . Back then, they developed a system called Dactyl, which is trained entirely in simulation but has proven to solve real-world tasks without physically-accurate modeling of the world.

Claude 3 Enhances Tool Integration

• The Claude 3 model family now supports tool use, enabling interaction with external tools and APIs on Anthropic Messages API, Amazon Bedrock, and Google Cloud's Vertex AI.

NVIDIA's New AI Chip: Vera Rubin

• At Computex, NVIDIA CEO Jensen Huang announced in his keynote the 'Vera Rubin' (named after American astronomer who discovered dark matter) AI chip, set to launch in 2026. This new chip will feature cutting-edge GPUs and CPUs designed for AI applications. NVIDIA plans to upgrade its AI accelerators annually, starting with the Blackwell Ultra in 2025, focusing on cost and energy efficiency.
• They also introduced Earth Climate Digital Twin to be able to not only predict but know what’s happening with our planet and its climate.

Mistral AI Introduces Codestral

• Researchers at Mistral AI have unveiled Codestral, a 22B open-weight generative AI model tailored for code generation. Supporting over 80 programming languages, Codestral excels in tasks like code completion and test writing, outperforming other models in long-range repository-level code completion. Available for research and testing via HuggingFace, Codestral also integrates with popular tools like VSCode and JetBrains, enhancing developer productivity.

Eleven Labs Introduces Text to Sound Effects (via video). Very impressive!

The freshest research papers were published. We categorized for your convenience ????