ChatGPT can now read some of your Mac’s desktop apps

OpenAI Introduces ChatGPT Integration with Desktop Apps on macOS

OpenAI has unveiled a new feature for its ChatGPT macOS desktop app that allows it to interact with specific coding applications. Announced on Thursday, the Work with Apps feature enables ChatGPT to read code from apps like VS Code, Xcode, TextEdit, Terminal, and iTerm2.

This functionality eliminates the need for developers to copy and paste their code into ChatGPT. When enabled, the chatbot can automatically analyze the code section in use and incorporate it into its responses.

However, ChatGPT still cannot directly write or edit code within these apps. Unlike tools like Cursor or GitHub Copilot, developers will need to transfer ChatGPT’s suggestions back into their coding environment manually.

A Step Toward AI Agents

While this feature is not yet a fully autonomous AI agent, OpenAI sees it as a critical milestone in building systems that can collaborate with other applications. According to OpenAI, understanding content from different apps is a “key building block” toward creating agentic AI systems.

Initially focused on coding tools, OpenAI plans to expand ChatGPT’s compatibility to other types of text-based applications, such as those used for writing and editing tasks. The feature is currently available to Plus and Teams users, with rollout to Enterprise and Edu users expected in the coming weeks.

How It Works

To enable ChatGPT to read app content, OpenAI leverages the macOS accessibility API, which translates text for the chatbot. For apps like Microsoft’s VS Code, an additional extension is required. The feature works by sending either the last 200 lines of code or the content from the foremost window to ChatGPT as input. Users can also highlight specific sections to refine the AI’s focus.

However, the system is limited to text and cannot interpret visual elements like images, object orientation, or videos. This reliance on macOS’s accessibility tools may pose challenges for expanding the feature to apps not compatible with Apple’s screen reader.

Future Developments

Competitors like Anthropic are experimenting with more generalized AI systems that analyze screenshots of user desktops. While this approach is broader in scope, it is currently slower and less reliable. OpenAI, meanwhile, is building upon its foundational tools to support future advancements.

Reports indicate OpenAI is working on a general-purpose AI agent, codenamed Operator, expected to launch in early 2025. This tool would compete with similar systems like Anthropic’s and Google’s rumored “Jarvis.”

For now, the macOS feature is a step forward. OpenAI is focusing on coding applications ahead of Apple’s anticipated ChatGPT integration in December. The release date for Windows compatibility remains uncertain, despite Microsoft’s significant investment in OpenAI.

If you liked this list, give me a follow for more such content!??