AR…ARKit…ARCore

In my previous post I briefly mentioned about AI and how it influenced AR at Facebook; here I want to take the opportunity to discuss it further more in detail.

Reflecting back AR (Augmented Reality) has been talked about for a while. Anyone remember this TED talk (2012) by Matt Mills (Matt Mills: Image recognition that triggers augmented reality) ?

Before we begin, lets clear out the AR and VR differences. Both play a part in immersive computing experience. In simple terms, if we start adding digital content to the real world, we end up with Augmented Reality (AR) whilst if we fully replace it with virtual (i.e. replace the real world with virtual world..kind of like transporting you to a different place), we end up with Virtual Reality (VR).

Come 2017 it was a big year for mobile AR. We saw the likes of Facebook(SLAM), Apple(ARKit) and Google(ARCore) bring it to main stream. With Apple and Google embracing it, it meant developers can now build more advanced AR experience at scale for millions of smart phone users.

Apple will win in the short term as they have the best platform (unified hardware and software set) which supports wider compatibility for AR. Having said that Google (Android) which is fragmented will take longer but has a better scale and the reach. More of the reasoning below.

Some of the benefits with AR are:

• See digital objects at a real size and scale in the real world ( e.g. placing digital furniture in the real world to see how it fits in your space)
• See things in context (e.g. whilst viewing a building site in progress, one can see how the fully completed model would look at the same location in 3D)

Yes, the business benefits which goes without saying are Brand Awareness (gets increased social shares!), help customers buying decisions (make sales to satisfied customers), understand user behaviour (based on desire, emotion and knowledge when shown the virtual object), thereby help customers truly connect to the published content!

Stepping back, lets take a look back what brought about this advancements in particular Apple. For instance, Apple's ARKit is down to the advancements in the AI space. Apple published its first AI paper (helping their computer vision tech in 2016), hiring a Carnegie Mellon Uni. professor as their director of AI, acquiring companies such as Lattice and Turi in the AI space. With the release of iOS 11 in fall 2017, alongside the new Depth API, Vision framework and CoreML, developers can now tap into the ARKit framework which uses a combination of motion sensor data and visual input from the camera to allow a user to freely explore around 3D graphics rendered in space.

Motion sensor in an Apple device was introduced in 2007 with the 3 axis accelerometer and come 2010, Steve Jobs introduced 3-axis gyroscope which gave 6-axis motion sensing and rotation enabling spatial awareness. Fast-forward to 2015, Apple acquired Metaio, a German firm who worked for internal Volkswagen project to develop tools for Augmented Reality visualizations. Come 2017, devices with at least A9 chip and iOS 11 can track motion sensor data for use in positional tracking in tandem with information from the camera using a technology called VIO (Visual Inertial Odometry).

VIO analyzes camera data ("visual") to identify landmarks it can use to measure ("odometry") how the device is moving in space relative to the landmarks it sees. Motion sensor ("inertial") data is used to fill in the blanks in providing complementary information that the device can compare with what it's seeing to better understand how it's moving in space.

Effectively VIO lets ARKit identify surfaces such as walls, table surfaces etc to place virtual objects thereby you have a mix of physical and virtual whilst you move the camera with 6-degrees of freedom (up/down, back/forth, in/out and its pitch, yaw and roll). Paired with light estimation the result is a photo-realistic model.

At the heart of this concept, is the chipset required to analyse and compute the vast amount of data at real-time to redraw the screen. Hence A9, A10 and the latest A11 Bionic chips are well suited for this.

Now, makes it easier to explain Android's world. It needs a fast chipset and standardised motion sensors across its fragmented space. When ARCore was announced (built on the work already done with Tango), it could only support on two devices: Pixel and Samsung's S8. Even if Google expands the hardware needed to run ARCore, the adoption percentage of the company's latest version of Android will be a major hindrance to adoption.

In my personal opinion, regardless of which platform makes it big, it benefits the user base of both platforms. Much like the "payment" instrument space, AR could be a space that both companies genuinely compete! 

Below are terms and concepts in ARKit and ARCore which highlights similarities:

ARCore fundamental concepts: Motion Tracking, Light Estimation and Environmental Understanding.

ARKit fundamental concepts: Tracking, Scene Understanding, Rendering and Light Estimates.

ARKit development support: Unity, Unreal, Sprite / Scene Kit

ARCore development support: Unity, Unreal, OpenGL

Basic ARKit API: Built on top of SceneKit/SpriteKit/Metal, Session, Frame, Anchors, Configurations, HitTest etc

Basic ARCore API: Built on top of OpenGL/SurfaceView/Renderers, Session, Frame, Anchors, Configurations, HitTest etc