Devin: The future of software engineering?

Cognition AI has launched Devin, an autonomous agent revolutionizing software engineering. Equipped with cognitive abilities and essential tools, Devin autonomously handles complex coding tasks and adapts to diverse scenarios, demonstrated through rigorous tests with Meta's Llama 2.

According to Scott Wu, Co-founder and CEO of Cognition, Devin represents a transformative force in software engineering. Devin is designed to work like a human software engineer, capable of doing many tasks that a human engineer would do.

Cognition is not marketing Devin as a replacement for human software engineers, describing it as a “teammate.”

Devin's capabilities:

Devin can handle complex engineering tasks that involve making lots of decisions. It remembers what needs to be done at each step, learns from experience, and corrects mistakes along the way.

Devin has tools that developers commonly use, like a shell, a code editor, and a browser. Devin can work together with a human user, providing updates on its progress in real-time. It listens to feedback, makes changes as needed, and helps with design choices along the way.

Here's what else Devin can do:

Learn New Technologies: Devin can learn how to use new and unfamiliar technologies, like reading a blog post and then using what it learned to create images with hidden messages.

Build and Deploy Apps: Devin can create applications from start to end, like making an interactive website that simulates a game and adding new features to it as requested by the user. Then it can deploy the app online for others to use.

Find and Fix Bugs: Devin can search through codebases to find bugs and then fix them automatically, like helping someone debug their programming book.

Train and Fine-tune AI Models: Devin can set up and improve its AI models, even if it's given just a link to some research online.

Address Issues and Requests in Open Source: Devin can handle problems and requests for changes in open-source software projects. It gathers all the information it needs and makes the necessary changes.

Contribute to Production Repositories: This example is part of the SWE-bench benchmark. Devin solves a bug with logarithm calculations in the sympy Python algebra system. Devin sets up the code environment, reproduces the bug, and codes and tests the fix on its own.

Real-world Jobs: Devin has been tested on real-job tasks and was successful, like writing and fixing code for a computer vision model and creating a report based on the results.

Overall, Devin is a great assistant for coding tasks, capable of learning, problem-solving, and collaborating with humans to get things done efficiently.

Performance

Devin has been tested on a benchmark called SWE-bench, which measures how well it can solve real-world coding problems found in open-source projects.

Devin was able to solve 13.86% of these issues completely on its own, which is much better than GPT 4 (1.74%). Even when given exact instructions on what to do, other models could only solve 4.80% (Claude 2) of the issues.

How to Access Devin?

Devin is not currently publicly available. Wu said in his blog post that it is currently in early access as Cognition “ramp up capacity.”

For those interested in using Devin for engineering work, you will have to reach out to Cognition directly via email or through the contact form.

Final Thoughts

Devin AI could be the future of software engineering. Its ability to work autonomously alongside human engineers, streamline coding tasks, and boost productivity hints at its potential to revolutionize the industry. While it won't replace human input, Devin's role in shaping the future of software development looks promising.

Let me know your thoughts!