Cognition AI - Devin: The first 'AI software engineer'

>> More Posts and Articles from David Cronshaw

In what will be the first of many use-cases for Autonomous Agents - #Devin, touted as "the first "AI software engineer", was announced by a recently stealth-mode startup called Cognition AI, which is funded by Peter Thiel's Founders Fund and tech industry leaders, including former Twitter executive Elad Gil and Doordash co-founder Tony Xu.

Cognition AI's founders are Scott Wu, its chief executive officer; Steven Hao, the chief technology officer; and Walden Yan, the chief product officer.

Here is the Devin announcement from Cognition's CEO Steven Hao:

"Meet Devin, the world’s first fully autonomous AI software engineer. Devin is a tireless, skilled teammate, equally ready to build alongside you or independently complete tasks for you to review. With Devin, engineers can focus on more interesting problems and engineering teams can strive for more ambitious goals." - Cognitive founder and CEO Scott Wu

Cognitive's Devin is an AI agent that can help you do software development-related tasks on its own. With minimal to no human involvement.

While there are numerous coding assistants presently available, including Microsoft's Github Copilot, Devin is noteworthy because of its ability to handle entire development projects end-to-end - from writing code, fixing the bugs - to final execution.

Devin's Capabilities

>> Cognition YouTube Channel

Devin is able to handle a range of tasks. This includes engineering projects such as deploying and improving apps/websites end-to-end and also finding and fixing bugs in codebases to more complex tasks such as setting up fine-tuning for a large language model. It also handled an Upwork project to run a computer vision model by writing and debugging the code for it.

"With our advances in long-term reasoning and planning, Devin can plan and execute complex engineering tasks requiring thousands of decisions. Devin can recall relevant context at every step, learn over time, and fix mistakes." - Cognition

Ashley Vance from Bloomberg Businessweek was able to review Devin and wrote an article about it:

Gold-Medalist Coders Build an AI That Can Do Their Job for Them

Devin's Benchmarks

The benchmarks for coding looks impressive, with 13.86% of tasks assigned to it being completed on their own. In the benchmarks below, Devin completes the tasks on its own. The other results are with users using the LLM which are not automatic like Devin.

In comparison, Claude 2 could resolve just 4.80% while SWE-Llama-13b and GPT-4 could handle 3.97% and 1.74% of the issues

Even if it only completes 13.86% of assignments on its own, Devin is still a big step forward.

Conclusion

Devin is less like an assistant helping with coding and more like a real worker. Devin is different because it is an autonomous system that actually does things for you, from start to finish.

This will bring up a lot of philosophical questions about the future of specific jobs and work, and what this means when non-developers, can now create their own websites, services and apps.

There are two important points to to realize here:

1) There will be job losses, and it's unrealistic to think you'll remain unaffected without adapting to new technologies.

2) The field of software engineering is evolving, not disappearing. We're experiencing a transition where artificial intelligence assumes a significant portion of the development tasks, transforming the dynamics within software teams.

The future of software autonomous and multi-agents

There will be a progression of software autonomous and multi-agents from coding to broader applications.

From revolutionizing healthcare to pioneering space colonization, the potential applications are transformative:

Healthcare and Biomedical Research - will witness a paradigm shift with personalized medicine and accelerated drug discovery, leveraging agents to analyze genetic data and navigate vast drug compound datasets.

Environmental Monitoring and Sustainability - efforts will be bolstered by these technologies, offering precise climate change models and effective pollution control strategies, marking a significant stride toward preserving our planet.

In the development of Smart Cities and Urban Planning, autonomous systems will optimize traffic flow and predict infrastructure maintenance needs, ushering in an era of unprecedented efficiency and sustainability.

Space Exploration - will enter a new chapter as autonomous rovers and habitat management systems take the lead in scientific experiments and ensuring the viability of life beyond Earth.

Education - will be revolutionized through personalized learning platforms, adapting to each student’s unique needs and fostering a more inclusive and effective learning environment.

The Creative Industries - stand to gain immensely, with autonomous agents pushing the boundaries of content creation and design, blending creativity with computational power to produce novel artworks and architectural marvels.

Financial Services - will see enhanced market analysis and trading strategies, thanks to predictive models capable of navigating the complex landscape of global markets.

Lastly, Social Services and Governance will benefit from public policy simulations and efficient disaster management, showcasing the role of autonomous systems in building a more resilient and equitable society.

More Questions Than Answers

After reading this article, Sam Godwin came up with some very good philosophical questions:

"With autonomy comes the challenge of ensuring robustness and reliability in complex engineering tasks. How does Cognitive AI address the intricate balance between autonomy and accountability in Devin's operations, particularly in scenarios requiring nuanced decision-making and adaptability to evolving requirements? Moreover, considering the intricate nature of software engineering, how does Devin navigate ambiguities and unforeseen challenges while maintaining efficiency and precision?" - Sam Godwin

The Pragmatic Engineer - The Pulse from Gergely Orosz:

Is the “AI developer”a threat to jobs – or a marketing stunt?

"I suspect the media campaign is overreaching, though. Press reports say Devin is ready to do the job of developers, but even Cognition AI admits that the tool only solved about 1 in 7 GitHub issues unassisted in tests. That’s impressive, but there’s a very long way to go! You’d expect junior engineers to tackle at least as many (if not more,) and for senior engineers to get almost 7, unassisted." - - Gergely Orosz: The Pragmatic Engineer

What's Next in Generative AI?

The next generation of AI is already emerging, creating highly personalized assistants that learn from your interactions and tailor their services to your specific needs and preferences. This leap in technology promises a future where AI companions can anticipate your needs, offering support that feels uniquely designed for you, effectively creating a unique AI assistant for every single user.

>> More Posts and Articles from David Cronshaw

#ai #devin #cognitionai #autonomousagents