Apple Vision Pro: It’s the OS, Stupid

The Vision Pro sets a new high bar for Spatial Computing hardware, but its operating system, visionOS, is the real game changer.?

In spatial computing the operating system plays a more prominent role than in mobile computing, and visionOS is the harbinger of this new wave of Spatial OS. Its introduction of natural interaction that “just works” and true multitasking is unmatched, but it falls short of delivering on the promise of spatial computing. VisionOS currently limits developers’ access to the device’s cameras and sensors, preventing apps from seamlessly blending experiences with users’ surroundings for complete suspension of disbelief.?

What’s so special about visionOS and its multitasking? And what would it take for Apple and its rivals to open up and allow developers to realize the full potential of Spatial Computing?

The Role of the OS

An operating system (OS) of any device is the software foundation that manages computer hardware and takes care of running software programs. From a developer perspective, its primary role is to act as an intermediary between software and hardware. From a user perspective, its most visible function is the interaction between the user and the software and hardware. Spatial operating systems must take on a new role: user interaction with the world.

The Comeback of the OS

In the PC era, the OS was king. Windows, the cornerstone of Microsoft's OS user interface, was so dominant that the operating system itself was named after it. In the smartphone era, the OS took a back seat. Mobile computing gave rise to a user experience ruled by apps: If you wanted to do anything, there’s an app for that. The OS’ job was to quickly get you in and out of an app and onto the next one. Limited screen real estate and touch gestures made multitasking impractical on smartphones.

The New Elements of a Spatial OS User Experience

In the brave new spatial world, where computing is all around you, the OS has opportunities to literally add a new dimension to interactions—the user experience goes spatial. Here are key new elements introduced by a spatial OS:

• Spatial Interaction: Menus are presented spatially (in the physical space all around), and the user embodies the interface, navigating menus using eye movements, hand gestures, and voice control.
• Multitasking: Multiple programs run in parallel and are persistent in different locations in the real world. The user switches between programs simply by naturally moving or glancing around.
• Multiusers: Multiple users (local and remote) may interact with the same content in the same space.

The OS makes a comeback and becomes front and center of the experience.?

Plenty has been said about how Vision OS perfected spatial interaction by using a combination of gaze and air pinch to select and interact with menu elements. But it truly sets itself apart when it comes to multitasking.

Among the leading spatial operating systems, visionOS is the first to offer true multitasking in the real world, or more accurately—in mixed reality—where, thanks to the device’s video passthrough, the user sees a mix of both virtual content and the camera view of the real world.

Multitasking Goes Spatial

Cognitive neuroscientists argue multitasking is a myth. The human brain can process only one task at a time. “In reality, it’s rapidly switching from one task to another, and then back again. And every time you make that switch, you pay a ‘tax’ on both your time and your energy.” —Dr. Sahar Yousef.

That’s a keen observation about multitasking on PCs: you can only focus on one window at a time. And it’s certainly true about mobile devices.?

Can spatial devices enable true multitasking?

The Meta Quest OS offers a limited 2-dimensional (2D) type of multitasking. A user playing a 3D game can flip their wrist while being fully immersed in a game and trigger the Quest browser to launch 2D apps. This is helpful, for example, when seeking a game walkthrough guide from the web. Moreover, Quest apps such as Immersed allow users to run different apps on multiple 2D screens that can be arranged around the user as in a cockpit. But true multitasking is not offered on the Quest OS level.

In visionOS, apps automatically launch into a Shared Space, where each app is allocated a 3D volume and shown side by side with other apps. They can be resized to be ultra-large or small. App volumes can be arranged around a room and manipulated at will. These programs are persistent in a specific location, say on your living room rug or your kitchen counter, so when you walk away or remove the headset and then come back, the apps will remain positioned in the exact same spot. Apple describes visionOS as offering an “infinite canvas”, but it is unclear if there is an upper limit beyond what fits in the visual space. Of course, running many apps in parallel, or multitasking, requires upgraded hardware resources, which the Vision Pro seems to handle well with its superpowered M2 and R1 chips.

And here’s the real impact of this unique capability of visionOS: Since multiple spatial apps or virtual objects can be positioned in space around the user, she can focus on several things at a time, hence experiencing true multitasking—just like in the real world. In the real world, things are of course naturally persistent, and users can switch their attention without switching “apps” or “window” without paying any “tax” on time or energy.

A Perfect Spatial Experience for Perfect cooking

When the first Vision Pro units started shipping to users, we heard enthusiastic cooks clamor about the newfound power to place virtual timers near real boiling pots and pans on the kitchen stove, side by side with virtual recipes from a different app. It can go much farther.?

To compare my cooking progress against the desired end result dish, visionOS allows me to access the Files folder (shared from my iPhone) and pull up an old 3D model of the aforementioned dish which I scanned with an iPhone while on vacation, and place it on the counter. And to get into the right ambiance, I can strategically position in the background a panorama of the scene in which I enjoyed that same dish in Costa Rica. Will it improve my odds for a successful dish? Given my cooking skills it’s unlikely. But I am sure to have a wonderful holistic experience thanks to multitasking several different apps, all present in my field of view.

Spatial computing gives multitasking a new meaning. Since it effectively mixes the virtual and the real world, it extends the idea of multitasking to encompass both tasks in the real world and tasks in the virtual world. A Vision Pro user and dad reported playing a virtual puzzle on the Vision Pro while watching in the background a baseball game with his family and sharing their reactions to the game.

It’s not surprising that visionOS is the first to introduce spatial multitasking. Since Tim Cook first hinted about Apple’s ambitions for a new spatial device, the term Virtual Reality (VR), where the user is typically fully consumed in a single focused experience, has never been uttered. Apple consistently led with a strategic preference for Mixed Reality experiences where the users remain connected to their surroundings.

VisionOS is Leading But Has a Ways to Go

Apple’s visionOS is therefore a trailblazer in offering true multitasking in spatial computing, but it does have a long way to go for full suspension of disbelief. Developers complain the Volumes in which the apps live are detached from the real world. As Gabriele Romagnoli posts, “volumes trap the digital content in invisible boxes but besides some visual effects like shadows on furniture or glossiness on the UI, the content [doesn’t have] much to do with your physical environment.”

Apple Vision Pro’s cameras and its depth sensors do have the ability to understand the world in finer granular details but its operating system doesn’t allow app developers to access that critical information. The simple excuse which is also communicated by Meta is privacy concerns. The safe bet to protect privacy is to prevent apps from accessing the cameras. As VR developer and blogger Antony Vitillo writes: “if a malicious developer made a cute game and behind the curtains activated the cameras and streamed whatever they saw to its servers, that would be an enormous privacy violation. Evil developers could easily spy on our homes.”

But confronting this challenge is worthwhile for Apple and its competitors in order to achieve the true potential of spatial computing.

To find out more about why this is critical in a spatial OS and the potential rewards for users—read my upcoming part II in the series “Apple Vision Pro: It’s The OS, Stupid.”