Anthropic’s New AI Model to Control PC with AI Agents

Welcome to Tech Tips Tuesday ?? where we explore the latest news, announcements and trends around the tech world.

Last spring, Anthropic shared its exciting plans with investors to create advanced AI technologies designed to build virtual assistants. These assistants would be capable of handling tasks like researching information, managing emails, and automating various office functions all on their own. This project is being called a "next-generation system for AI learning," and it has the potential to influence many parts of the economy in the future significantly.

Recently, Anthropic has made progress toward this vision. On Tuesday 22nd October 2024, the company launched an upgraded version of its Claude 3.5 Sonnet model, which is now capable of interacting comprehensively with any desktop application. The introduction of the “Computer Use” API, currently in open beta, enables this by allowing the model to emulate user actions, including keystrokes, mouse clicks, and other gestures, effectively simulating human operation at a computer.

Anthropic reports that Claude has been trained to interpret on-screen visuals and utilize available software tools to perform assigned tasks. When prompted to operate specific software, Claude first analyzes screenshots of the user interface, then calculates the necessary pixel movements—both vertically and horizontally—to accurately position the cursor for interaction.

Developers can access the Computer Use features through Anthropic’s API and platforms such as Amazon Bedrock and Google Cloud’s Vertex AI. Furthermore, the new 3.5 Sonnet model, which is available without the Computer Use functionality, is being rolled out to Claude applications and includes various performance enhancements compared to its predecessor.

AI Agents

Anthropic’s Claude models represent a new type of artificial intelligence known as "AI agents." Unlike traditional chatbots that simply reply to questions with text, AI agents can perform a variety of tasks on their own, making them much more flexible and useful. Although "AI agent" doesn't have a set definition, it generally refers to systems that can automate many different software-related tasks. For instance, these agents can send emails, browse websites, or handle complex processes that involve multiple steps.

Businesses are starting to see AI agents as key to improving productivity. Many tech companies are investing in this technology, recognizing its potential to create significant benefits. A survey from Capgemini found that 10% of organizations are already using AI agents, and 82% plan to start using them in the next few years. Major companies like Microsoft, Salesforce, and Google are all working on developing AI solutions that can take care of a wide range of tasks independently.

Anthropic’s latest product, the Claude 3.5 Sonnet, goes a step further by allowing AI to perform tasks typically done on a computer. This model not only works within software programs but can also browse the web in real time, expanding its ability to interact with both online and offline applications. This makes Claude 3.5 Sonnet a strong competitor to other AI tools like OpenAI’s ChatGPT and Google’s Gemini.

Practical Applications of Anthropic’s AI Agents

The introduction of the 3.5 Sonnet model is an exciting development for Anthropic, opening up new possibilities for businesses to use AI in practical ways. Some early users, including popular companies like Asana, Canva, and Notion, have been testing a feature called the Computer Use API to help automate tasks that are repetitive and time-consuming.

One of the standout features of this AI is its ability to manage complex tasks. This means it can handle activities that involve many steps, such as working with complicated software or analyzing large amounts of data.?

While many AI tools can help with single tasks—like automating simple processes—Anthropic's approach aims to do much more. For example, future versions of their AI, named Claude, could potentially handle entire operations within a company without human intervention. Currently, AI like this is already being used for tasks such as scheduling meetings, researching markets, and even managing expense reports.

On top of these business applications, Anthropic is also looking into ways to make Claude useful for everyday tasks, like booking flights or filling out online forms. This flexibility could make AI agents valuable not just for companies, but also for everyday consumers.

Anthropic vs. Other AI Agents

Anthropic is part of a competitive race to create advanced AI systems. Many companies, including OpenAI, Microsoft, and Google, are also working hard to enhance AI technology. These companies are developing tools that help people be more productive by automating complicated tasks.

What sets Anthropic apart is its focus on safety and giving users control over the AI. For instance, their Claude 3.5 Sonnet model only takes action when a user specifically asks it to. Users can give clear instructions, like "use information from my computer and online to fill out this form," and the AI will carry out those tasks—like clicking or typing—without acting on its own.

Despite these safety measures, Anthropic recognizes that there are still many challenges to tackle in the development of AI.

Limitations of AI Agents

Anthropic’s Claude 3.5 Sonnet is an advanced AI tool that has some impressive features, but it also has its problems. Early tests show that the AI struggles with simple tasks like scrolling and zooming on a screen. Sometimes, it doesn’t notice quick actions and ends up making mistakes. For example, when tested for booking flights, it managed to complete less than half of the tasks correctly, and it failed about a third of the time when trying to help with product returns.

These issues remind us that AI technology still has a long way to go before it can fully take over many jobs and tasks. Because of this, Anthropic suggests that developers start with simple and low-risk tasks to avoid any major problems caused by the AI's occasional errors.

Moreover, allowing AI to perform commands on a computer raises some safety concerns. If misused, this technology could be used for harmful activities, like accessing private data or disrupting computer systems. To address these risks, Anthropic has put in place safety measures, such as a system that discourages AI from taking actions that could be considered risky, like posting on social media or interacting with government sites.

Conclusion

Anthropic has introduced a new AI model called Claude 3.5 Sonnet, which is making strides in the way computers can be controlled automatically. While this technology is still developing and needs improvement, it shows great potential for businesses looking to automate everyday tasks. Companies like Asana, Canva, and Notion are already looking into using AI to enhance their operations, and we can expect consumer-friendly applications to emerge soon.

However, with such powerful AI systems, there are risks involved. Anthropic is taking a careful approach by limiting what the model can do and prioritizing safety, which is a good step towards preventing misuse. Like any new technology, these AI tools will need to demonstrate that they are reliable and secure before we can fully trust them with more important jobs. As AI technology continues to progress, we can look forward to a future where these intelligent virtual assistants play a role in both work and everyday life.