Dispatches from WWDC24 - privacy, security and more...

Apple Intelligence

As expected the major push for Apple this year was what Apple is calling Apple Intelligence (or AI). Despite the cheesy name, Apple has taken a thoughtful approach to integrating AI into its products, including careful consideration of privacy that focuses on 3 different use-cases for AI.

• Context specific on-device AI, for dedicated and targeted AI features
• Privacy preserving Apple Private Cloud Compute, for broader privacy preserving AI queries in the cloud
• Partnering with AI providers for general AI.

Each use-case has particular concerns and an approach to helping protect the users privacy whilst being useful.

Context-specific on-device AI

This attempts to provide context specific AI queries done entirely on-device, so that no sensitive data leaves the device to answer a query. These models are generally very specific to a given task, but have access to personal information stored only on the device - for instance the summary feature for Email and Notifications in iOS 18. This ensures the most personal and sensitive data Apple holds on the device can be queried with AI, but it never leaves your device, which keeps it safe.

Private Cloud Compute (PCC)

For AI queries that need the compute power of the cloud, Apple has expended a huge amount of engineering and security architecture capability, to build a secure cloud platform. This will enable the answering of queries in an Apple private cloud platform, that still preserves the users privacy and security. I personally think this model for Private Cloud compute should become the de facto standard for how cloud AI is done, as it preserves a users privacy whilst still answering AI queries in the cloud at scale.

Apple has achieved this secure cloud model through the following architecture and security approach using Apple Silicon with a custom OS, leveraging existing security controls like the Secure Enclave and Secure Boot.

The requests to cloud ensure the personal data passed with the query is not persisted, is stateless and is not available to anyone, including Apple’s operation staff managing the cloud platform. User Data is removed from the cloud as soon as the request is complete. Additionally Apple is using cryptography to protect the request in transit.

The cloud node processing the request is also protected from tampering, using secure boot and code signing (similar to that employed on iOS), ensuring only cryptographically measured code is executed on the node. This code cannot be modified at runtime and is loaded by the Secure Enclave. Apple has used Swift as a memory safe language in the cloud, in combination with the principle of least privilege to remove entire classes of attacks. The cloud nodes are locked right down, with no shell access or debug tooling.

The entire software stack is digitally signed with a certificate from the Secure Enclave on the physical Cloud node.

Apple also allows independent security researchers to verify end-to-end security of the cloud platform to ensure public trust.

These controls along with others I have not listed, provide very strong and verifiable controls around users’ personal data, when using in the Apple Cloud environment.

Partnership with 3rd party AI providers

Initially Apple has only partnered with OpenAI, who are arguably the leaders in the general AI space, to support ChatGPT4o. In this model, Apple lets you opt-in to this style of query, providing transparency that you will be using chatGPT for your query. Contractually Apple is also putting in place controls around what these providers can do with the data given to the AI. This model lets customers take advantage of general AI with leading AI models, as Apple's approach is more targeted and context specific.

These 3 AI models provide a sensible and pragmatic approach to extending its AI capabilities without scaring its customers, as much as other vendors have with their ill-conceived AI additions (yes Microsoft, I'm talking about you).

AI for Developers

Apple is also helping developers with AI additions to Xcode (the Apple IDE), by providing code completion with on-device processing and SwiftAssist a Private Cloud Compute model for helping to build Swift code using AI, without having to worry that you have given away sensitive corporate intellectual property, or leaked sensitive material into the cloud provider. SwiftAssist can understand the context of your code and follow patterns and best practice to provide code snippets to help speed development along.

visionOS

Whilst Apple has limited success with their game changing visionOS platform, they have continued to extend its capabilities this year, building out even more spatial interactions. They will also start shipping the hardware to more countries including Australia (not NZ just yet though). Apple is also targeting the Enterprise market more with visionOS, as they probably see those with a large workforce as likely purchasers of this platform, to help with guided workforce tasks, given the cost of the hardware (at this time).

iOS 18

Apple Intelligence comes to a limited range of Apple Devices with iOS 18 over the next year. Apple has limited the ability to use its on-device AI to just iPhone 15 Pro and above, Mac on Apple Silicon and iPads running the Apple Silicon chipset, due to the heavy CPU, NPU and RAM requirements needed to process AI on device. This will frustrate some people, but is understandable given the processing needs of AI.

Apple has also introduced “Writing Tools”, which are a specific AI capability running on-device, to help you re-write your text in different styles, along with allowing you to summarise large blocks of text so it’s easier to understand. These are narrow focus things that AI can do well, without suffering too many hallucinations or errors.

There is now also the ability to do limited Image generation on the platform, with what they call Image Playground, where you can create non-realistic images based on prompts. Apple has taken this path to avoid the creepy photo-realistic image generation approach of others, so that these images can’t be used for mis-information or other ill intent.

Apple has also provided Genmoji a more freeform Emoji capability that can create emoji via generative AI.

In order to allow on-device AI to interact with custom Apps (those not built by Apple), Apple has extended its App Intents capability, so that 3rd party Apps can share specific content items with the system and effectively become part of the Apple Intelligence ecosystem. This also provides the ability to then deep link into the Apps to find appropriate content, helping the user to find information and take action upon it. As iOS 18 matures and developers take advantage of these integration points with Apple's ecosystem, we will get smarter Siri and App integrations over time.

And Apple will finally introduce the ability to customise your Home Screen in more flexible layout ways, with light/dark and a Tint style mode.

Then there’s AccessorySetupKit, which simplifies the setup of IoT accessories with Apple - think pairing your AirPods to iPhone style simplicity.

Passwords app

Apple will finally release a Password App, that will work within the bounds of the Apple walled garden. This will help take Password managers to the broader consumer base that still find these things too hard to use. The password app will help with passwords, passkeys (the new password-less logins) and also One Time codes, all from a single App, synced across your devices using iCloud. For those that don’t already use a password manager and operate primarily within the Apple ecosystem, this will be a great advance in security for the average consumer.

Apple has released the ability for users to upgrade a normal password based identity to passkey, as part of its iOS 18 SDK updates. This means certain apps will allow the customer to move away from passwords, towards password-less logins using passkey as part of a customer registration and setup. This will allow for login credentials that are no longer ‘phishable’, another step towards reducing the number of data breaches occurring.