Research to Reality: The Evolution of the New AI Ecosystem

INTRODUCTION

If you were not paying attention, the AI boom seems like it came out of nowhere, arriving at a stroke with the release of ChatGPT in November 2022. In reality, it’s been building for years through the accumulation of discoveries in basic research. Some of these led directly to the creation of pacesetting companies, while others established the foundation for entirely new ecosystems of innovation. Many did both.

At Lightspeed, we’ve been close observers of AI research over the last decade. We’ve also often been active participants in helping leading researchers turn their ideas into groundbreaking startups. We were early backers of Mistral, SKILD and Snorkel, all of which germinated from fundamental discoveries about AI technology. The breadth of our portfolio in this area reflects the deep expertise Lightspeed investors and advisors have at recognizing and cultivating the new discoveries in basic AI research that will lead to the next generation of breakout technology companies.

In this post we identify the most influential AI research papers of the past 15 years. We also trace how they reverberate in the startup world today, in the form of scientists who’ve become founders and institutions that have turned into hubs of the new AI ecosystem. Through this lens, we observe that there have been four main research waves that have built on each other and propelled AI to its current prominence:

1. MODEL ARCHITECTURE IMPROVEMENTS

Since the 2010s, advancements in AI model architecture have driven significant breakthroughs and startup innovation. These include AlexNet’s 2012 work on deep convolutional neural networks and the acclaimed paper Attention Is All You Need , which revolutionized natural language processing.

2. DEVELOPER PRODUCTIVITY GAINS

The past decade has seen significant advancements in tools and frameworks that have markedly improved developer productivity, which is essential for startup growth. Milestones include the? introduction of?TensorFlow ?(and others, like PyTorch, in 2015), the?HuggingFace Transformers library , introduced in 2018, and Meta’s?open-sourcing of the Llama models ?in 2023.

3. TASK PERFORMANCE

Increasing public interest in AI models has been driven by their consumer, developer, and enterprise applications for task execution, sparking additional research focus. Several different papers released in the past 10 years have revolutionized the efficiency and variety of task executions performed by AI: on training deep neural networks ?for complex task execution,?joint learning for alignment and translation ?which led to reduced training complexity, a breakthrough in?unsupervised learning ?which led to improved task performance without any fine-tuning, and the use of?retrieval augmented generation (RAG) and external data stores ?for knowledge-intensive tasks

4. COMPUTE OPTIMIZATION

In the 2010s, new optimization techniques like dropout and batch normalization improved model performance and stability. In 2020, OpenAI’s landmark paper highlighted how model performance scales predictably with increased computational resources. This was followed in 2022 by DeepMind, which?demonstrated ?the importance of balancing model size and training duration for optimal performance.

The rest of the post explores the broad currents and specific discoveries that underlie AI technology today—providing a better understanding of how we got where we are, and making it possible to anticipate where the most dynamic technology in generations is headed next.

AI RESEARCH FAMILY TREE

GOOGLE

OPEN AI

FACEBOOK AI

STANFORD UNIVERSITY

PAPERS

EARLY BREAKTHROUGHS:

Early papers laid the groundwork for today’s AI ecosystem by introducing the frameworks, models, and methodologies that have become foundational to startup development and subsequent research. Frameworks like Transformers, GPT, Tensorflow, Bert, and others in these papers have introduced new architecture for natural language processing, training language models, and fine-tuning model development.

• ImageNet Classification with Deep Convolutional Neural Networks ?(2012), Geoffrey Hinton, Ilya Sutskever, Alex Krizhevsky
• Deep Residual Learning for Image Recognition (2015), Kaiming ?He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
• Neural Machine Translation by Jointly Learning to Align and Translate ?(2016), Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio
• TensorFlow: A system for large-scale machine learning ?(2016), Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, Xiaoqiang Zheng
• Attention Is All You Need ?(2017), Ashish Vaswani, Noam Shazeer, Niki Parmar, Jacob Uszkoreit, Lilon Jones, Aidan Gomez, Lukasz Kaiser
• Language Models are Unsupervised Multitask Learners ?(2019), Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever
• Roberta: A robustly optimized bert pretraining approach ?(2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov
• Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences ?(2019) Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, Rob Fergus

RECENT ADVANCEMENTS:

After 2020, the speed of AI development and adoption accelerated. Recent AI research has made significant advancements in learning and processing, making technology more efficient and scalable to a wider set of applications. We’ve also seen the development of AI solutions to real world applications, startups built off of early models flourish, and startups built off of newer models emerge.

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ?(2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rockt?schel, Sebastian Riedel, Douwe Kiela
• Transformers: State-of-the-art natural language processing (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander M Rush
• Language Models Are Few-Shot Learners ?(2020) Amanda Askell, Tom Henighan, Jack Clark, Benjamin Mann, Dario Amodei, Sam McCandlish, Tom Brown, Pranav Shyam, Rewon Child, Aditya Ramesh, Arvind Neelakantan, Christopher Burner, Christopher Hesse, Clemens Winter, Girish Sastry, Gretchen Krueger, Jeffrey Wu, Mark Chen, Matusz Litwin, Nick Ryder, Prafulla Dhariwal, Sanhini Agarwal, Scott Gray, Ilya Sutskever
• Scaling Laws for Neural Language Models ?(2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei
• Efficiently Modeling Long Sequences with Structured State Spaces ?(2021) Albert Gu, Karan Goel, Christopher Ré
• Flamingo: a Visual Language Model for Few-Shot Learning ?(2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan
• Training Compute-Optimal Large Language Models ?(2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W Rae, Oriol Vinyals, Laurent Sifre
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models ?(2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou
• Llama: Open and efficient foundation language models ?(2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
• Legged locomotion in challenging terrains using egocentric vision ?(2023) Ananye Agarwal, Ashish Kumar, Jitendra Malik, Deepak Pathak
• Multimodal Foundation Models: From Specialists to General-Purpose Assistants ?(2023) Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, Xiaohua (Michael) Xuan, Zhengxin Li, Lin Ma, Shenghua Gao

LOOKING AHEAD -- LIGHTSPEED’S VIEW ON FUTURE OF AI RESEARCH AND INNOVATION

ROBOTICS

Today, engineers are still working through basic issues around embodied AI. Current challenges include understanding data from multiple sensors, ensuring robots are adaptable to different contexts and settings, and working towards reasoning capabilities that could imbue robots with a degree of “common sense.” As AI research in robotics advances, breakthroughs in autonomous systems will enable more sophisticated and adaptable robots capable of navigating complex environments and performing a broader range of tasks. This is particularly important for industries like agriculture, where autonomous robots can revolutionize crop monitoring, harvesting, and precision farming, leading to increased efficiency and reduced labor costs. The same holds true in manufacturing, where advanced robotics has the potential to enhance existing automation, leading to safer and more efficient production processes.

AI AGENTS

Generative AI made waves for its ability to generate outputs based on user prompts, but many of the most pronounced and groundbreaking paradigm shifts will occur when AI begins doing work for us. In addition to simply querying AI for vacation ideas, we’ll be able to ask it to book the trip.

We already have early examples of agentive style AI in Github’s Copilot and other ‘AI assistants’, but to date, agentive technology has been limited to niche use cases with limited enterprise impact. As chain-of-thought reasoning capabilities improve and AI develops the ability to logic its way through a more generalized and complex set of tasks, the potential impact for businesses in every industry is vast.

AI INTERPRETABILITY

One of the biggest challenges with AI today is the ‘black box’ phenomenon – a lack of clarity on provenance in decision-making – which creates inherent distrust in outcomes. Improved interpretability techniques – understanding what models are ‘thinking’ about – or will allow users to gain deeper insights into AI decision-making processes, fostering wider adoption and ethical use of AI technologies.

This is particularly important in sensitive areas like finance and healthcare, where transparency is essential for regulatory compliance and maintaining user trust. Enhanced interpretability will ensure AI systems are more accountable and aligned with human values, promoting their safe and effective deployment across various industries.

VERTICALIZED APPLICATIONS OF AI

Verticalized applications of AI will increasingly streamline industry-specific workflows and transform the economics behind traditional engagement models for products— automating time-consuming tasks and reducing costs to free up humans to think bigger. We’re already seeing AI-driven contract analysis and research streamline and increase accuracy in the legal profession, while in healthcare, advancements in image recognition will revolutionize diagnostics, enabling radiologists and oncologists to detect diseases earlier and with greater accuracy, ultimately improving patient outcomes.

GET IN TOUCH

If you are a researcher exploring where and what to build in AI, a founder building on prior research, or we omitted an excellent piece of AI research from this roundup – we’d love to hear from you.

Get in touch with our team at?[email protected] .

Dive even deeper on the blog: https://lsvp.com/research-to-reality/