AI Now Controls Your Desktop - Directly

Imagine delegating tasks like file organisation, email management, and form filling—all without needing to touch your computer. Anthropic’s new Claude “Computer Use” feature makes this possible, introducing a whole new level of AI-powered productivity that could fundamentally change how we operate day-to-day. Don't believe me? Watch this video shared on Youtube:

#ShoryuWill Newsletter #27 By William Zhang

Click below to Listen to this Newsletter Edition??

What This Edition Will Cover:

• What is the Computer Use feature, and why it matter?
• How to use it for your business.
• The mindset shift needed to maximise AI in your workflow.
• 3 book recommendations to help you adapt and thrive.
• 1-2-3 Punch to elevate your success.

The Common Problem and Why It is Important

As business owners, especially in SMBs, we constantly juggle multiple roles—handling admin, engaging with clients, and keeping operations on track. On average, small business owners spend over 40% of their time on repetitive tasks like data entry, email responses, and scheduling. These mundane tasks drain valuable time that could be spent on strategy and growth. AI promises to alleviate this, but until now, most tools only offered passive support without directly performing the work.

This gap is precisely where Claude’s Computer Use feature can change the game, enabling an AI to actively manage these tasks autonomously, something Microsoft and Salesforce are also exploring through their new agent-based solutions. But with Claude, you don’t need deep integration or complex programming—it’s AI that interacts with your desktop as if it were you.

The "Computer Use" feature, introduced by Anthropic’s Claude, enables this AI assistant to directly interact with your computer, effectively allowing it to execute tasks on your desktop as though it were a virtual set of hands. This marks a significant leap from AI's traditional role of providing suggestions or information; with Computer Use, Claude can navigate, open applications, manage files, fill forms, and even perform tasks like sending emails—all autonomously, based on your commands. This feature matters because it brings practical, hands-on productivity to businesses and individuals who manage multiple daily tasks. By handling repetitive operations, Claude's Computer Use allows users to save time and focus on strategic work, thereby increasing efficiency. For example, small business owners or project managers who typically juggle administrative duties can offload these to Claude, freeing them up for higher-priority decisions.

Anthropic's Computer Use also incorporates safeguards, limiting access to sensitive sites and preventing unsupervised interactions on social media or government platforms, making it a safer option for business use. This combination of increased utility and built-in safety makes Claude’s Computer Use feature a valuable tool for businesses looking to optimize their workflows without compromising control.

Proof of Concept

Many companies are already testing Claude's Computer Use feature for real-world productivity gains. GitLab integrated it to support DevSecOps, reporting up to a 10% boost in task efficiency without latency issues, helping their teams keep up with security tasks while saving costs. Another lab, Cognition, noted improvements in Claude’s coding and planning, making it an ideal assistant for tech and data-driven industries. The demand for these kinds of AI "agents" is rising across sectors from finance to customer service, illustrating that businesses want an AI that can do more than just advise; they want it to act.

Where We Went Wrong

In the past, many businesses (my own included) invested in AI solutions without clear outcomes. I’ve tried multiple tools, only to find they added complexity without solving core pain points. These early tools offered suggestions but required constant supervision. Implementing AI without defined goals or proper integration was a mistake, as it led to more overhead than actual help.

Why the Problem Was Challenging to Resolve?

The real barriers to full automation were time, cost, and technical limitations. Setting up AI to manage tasks independently often required large budgets and in-house tech teams. On top of that, the complexity of human interaction (handling emails, navigating software, etc.) required intelligence that went beyond standard automation scripts. Lack of customization and integration options also limited the effectiveness of many solutions.

Why It’s Easier to Resolve Now

With Claude’s new Computer Use feature, AI setup and management are streamlined. Rather than relying on complex APIs or needing software integration for each task, Claude’s technology allows it to use standard desktop apps and websites directly, minimizing setup time. Anthropic has also placed significant focus on safety, ensuring Claude avoids sensitive sites like social media or government pages, which reduces risk. Now, we have an AI tool that’s cost-effective and directly useful for SMB owners.

The New Thing That Makes It Solvable at Low Cost but High Returns

Claude’s “Computer Use” is about more than automation—it’s a “virtual hand” for managing everyday tasks affordably. Since the feature is in beta, it’s accessible to smaller companies through Anthropic’s API and integrations with major cloud providers like Amazon Bedrock and Google Cloud’s Vertex AI. The real return here lies in the time it saves, allowing small businesses to focus on high-impact activities without burning through budgets on complex AI integrations.

What Knowledge, Experience, and Network You’re Missing (and How to Fulfill It)

To fully leverage Claude’s feature, you’ll need some essential skills and connections:

1. AI Fundamentals: Familiarize yourself with general AI principles and how agents work. This will help you maximize Claude’s potential. Coursera and edX offer introductory AI courses that can be completed in a few weeks.
2. Anthropic API Training: Spend time understanding how Claude connects with your desktop and workflows. Anthropic’s documentation offers valuable insights and examples, and many online communities provide support for API integration questions.
3. Network with Early Adopters: Join AI-focused forums and LinkedIn groups where early adopters share tips and troubleshooting insights. Hearing from others can help you avoid mistakes and discover creative uses for Claude in your industry.

Step-by-Step to Fulfill It

1. Review AI Basics: Start with a course on AI fundamentals. This helps you understand the tech behind Claude and its strengths.
2. Explore the API: Access the Computer Use beta via Anthropic’s API documentation. Test basic tasks like email management or file sorting to gauge Claude’s compatibility with your workflow.
3. Join Industry Groups: Engage with LinkedIn communities for SMBs and AI adopters. Sites like Stack Overflow and Reddit's r/ArtificialIntelligence can also provide practical advice.

Final Words

Claude’s Computer Use feature opens a new era in AI-led productivity for small businesses. While it’s still evolving, the early access we have to this technology is a huge opportunity for those willing to experiment. The possibilities extend from cost-cutting to true workflow transformation, and I’m excited to see how it will help us all work smarter.

3 Book Recommendations:

1. "The Age of AI" by Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher – A deep dive into how AI is reshaping industries and society.
2. "Human + Machine" by Paul R. Daugherty and H. James Wilson – A look at how AI and humans can work together, with practical case studies.
3. "Prediction Machines" by Ajay Agrawal, Joshua Gans, and Avi Goldfarb – Focused on how AI is transforming business decisions and lowering costs.

1-2-3 Punch

1 Quote: “Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.” - Stephen Hawking

2 Questions:

• What tasks in my daily workflow can I hand over to AI?
• How can I use AI to focus more on strategy and less on operations?

3 Actions:

• Identify three repetitive tasks you perform daily and try automating one of them with Claude.
• Set aside one hour to explore the Claude API or other AI tools that could save time in your workflow.
• Reach out to a network of AI users or SMB owners to exchange tips on AI adoption.

Reminder to Subscribe:

Enjoyed this edition of #ShoryuWill? Subscribe for more insights to help simplify complex business strategies into actionable steps or simply for daily insights that will help you 10x your business growth, and get exclusive access to AI tools, leadership strategies, and market trends that you won’t find anywhere else. Don’t miss out—hit that subscribe button now!

Your friend, William Zhang

Here are the full URLs for the references used:

1. PC-Tablet on Claude’s Computer Use Feature https://www.pc-tablet.com/anthropics-computer-use-feature-is-this-the-dawn-of-ai-superpowers/
2. TechRepublic on Claude’s Applications https://www.techrepublic.com/article/claude-3-5-sonnet-can-control-your-computer/
3. TechRadar’s Insight on Claude’s Control https://www.techradar.com/claude-ai-computer-use-control