ENTERPRISE XR AR VR MR with Michael Hoffman

Futurist Brian Federal

Published on February 7, 2020

Michael Hoffman is the head of product strategy and engineering for Portland Oregon based Object Theory , a leader in enterprise augmented reality services that Hoffman founded with partner Raven Zachary in 2015.  An early entrant into the AR field, Object Theory has been a member of the Microsoft Mixed Reality Partner Program (MRPP) since 2016. Prior to his current position, Michael contributed to the creation of XR experiences for Microsoft HoloLens as a Principal Engineering Lead at Microsoft. In his career michael has worked with teams including HoloLens work for Autodesk Fusion 360, Trimble, and the JPL/NASA Curiosity Mars rover. Michael’s experience also includes working with teams on high-profile projects for CDM Smith, Google, Nike, Stryker, as well as serving as senior executive at numerous startup ventures.

Most engineers remember destroying at least one home appliance as a kid in the pursuit of science. Do you remember being interested in how things worked from a young age? 

Indeed. I learned by building and experimenting. I labored for weeks creating complex contraptions with Fischertechnik , Lego, erector sets, and more. Fischertechnik sets were more technically-focused than Lego at the time. I recall building a functioning miniature motorized four-story elevator when I was eight years old!

That’s quite an accomplishment at that age. How did your interests and projects progress?

The next year, I experimented with chemical reactions, making gunpowder from scratch.At 11, I taught myself how to program on a Data General Eclipse mainframe at the college where my dad worked. That was really the beginning of my career. As a teen-ager, I discovered Byte Magazine at a Byte Shop, which was one of the earliest computer hobbyist stores. I meticulously studied Byte every month cover-to-cover, which led to such an intense interest in programming, I  wrote 6502 programs in notebooks and hand assembled them into the hexadecimal machine language instructions. At 15, I saved up enough money to buy a KIM, then was gifted an AIM-65 two years later. I finally got an Apple //e in college.

What were your primary interests when studying electrical engineering at North Carolina State University?

Throughout my teenage years, I had actually planned to pursue neurosurgery, yet as a high school senior, I had already started working at our local hospital in their mainframe department and had been programming for seven years. For three of those years, I also explored electronics and robotics, making printed circuit boards from Radio Shack kits, as well as getting involved with wire-wrapping and bread boarding circuits. If NCSU had offered a computer science degree, I would have majored in that area. Electrical Engineering felt like the closest field to my twin passions of software and electronics tinkering. What I enjoyed the most actually were my physics and chemistry courses. I found pure electronics too constrained a palette for all my interests.

How did you initially become interested in AR and spatial computing? Do you remember having a distinct “AHA!” moment?

That’s a clear memory. I read a Byte Magazine article entitled “The Mathematics of Computer Graphics” in 1978, which described the matrix math for 3D graphics. That captured my imagination! Within weeks, I wrote the 6502 machine language code needed to instruct my new KIM-I computer to draw rotations of a simple spaceship on a pen plotter. That triggered a domino effect. In 1983, I wrote a 3D graphics application for my Apple //e for designing and viewing 3D model rotations. In the early 1990s, I wrote a new 3D rendering engine for Microprose’s F117 Flight Simulator in 68000 assembly language for the Mac release.

At Microsoft, although I was tracking VR, I was not focused on spatial computing until I chanced upon the team at Microsoft working on what later came to be known as HoloLens. The entire program was so secret that, even as a current Microsoft employee, I had to accept a position on that team sight-unseen. On day one, my desk was next to a Kawasaki motorcycle stripped of all the plastic parts, and I saw holograms painted on that frame. The next day, I was standing on Mars next to the Curiosity robot.

"At that point, I was absolutely certain mixed reality would permanently change computing and how people worldwide interact with technology. I had the proverbial “Aha!” moment – I knew I was involved in a transformative shift in computing."

As a Principal Engineering Lead on the Microsoft HoloLens team, what were the most daunting challenges your team faced in designing for this technology?

I was on the pre-launch Studios team that was building applications to run on the HoloLens prototypes. Our mandate was to prove its value in solving real business problems, like planning out the Curiosity robot’s daily mission on Mars by enabling geologists to virtually stand on Mars and decide what the rover should do over the next 24 hours. When you’re inventing the future, it’s easy to get caught up in the “cool factor.” I was impressed with our team’s commitment to identifying the real value and not just focusing on how exciting this new medium is to present information in clever, visually appealing ways.  That commitment made every project much harder - so many ideas were cut because they didn’t meet that high bar of true utility.  That said, it was also an amazing education in what matters most.

Another of the many challenges related to developing such a revolutionary battery-powered device was managing battery consumption. We had contests to see which team could shave off the most milliamps of current draw. It reminded me of the Apollo 13 movie in which scientists had to figure out how to keep the current draw below a certain level to power up the capsule.

What was your motivation for leaving Microsoft to start Object Theory in 2015?

I must say, it was an excruciating decision. I loved my team and  Microsoft work. This was one of those rare moments in my career. I had identified a revolutionary computing technology advance and was positioned to become an early player by launching  a company.  My Portland entrepreneurial friend, Raven Zachary, had also identified "mixed reality" as a revolution around which he wanted to build a business.  I was tired of the long-distance weekly Portland to Seattle commute. The final factor in my decision involved the strong relationships I had built on the Microsoft team, which led to our selection as one of the first seven outside agencies to be nurtured by Microsoft in the HoloLens Agency Readiness Program, through which we were given pre-prelease access to devices. Those factors, in combination, told me the time was right for a big move.

What services does Object Theory offer customers? Basically, why would a company want to hire you?

We offer three primary services. We provide strategic guidance that helps companies find the best intersection between today’s XR device capabilities and their business processes that can benefit from spatial computing. Through a  discovery and analysis process, we give clients comprehensive recommendations for a spatial computing roadmap.We also offer the Prism spatial computing SaaS platform and backend management services on a subscription basis. This is an XR content delivery platform that meets the needs of large enterprise organizations.

Our third service, bespoke content creation, addresses the need for design and optimization of XR content and experiences. This content is deployed using the standard Prism application, or we can package the solution as a custom application.

Companies choose Object Theory based on our reputation. We’re extremely customer-focused and 100% committed to delivering exceptional quality. Our long history, impressive Fortune 500 company client base and a HoloLens team heritage give clients confidence that we can deliver answers that meet their needs.

Clients also appreciate our Prism platform’s flexibility, which supports a wide variety of capabilities in a coherent and non-siloed way. If you need training plus remote video assist, or space planning plus remote collaboration, or premium selling plus annotations, Prism makes that possible without going between many different applications.

Finally, Object Theory is known for fidelity. The quality of our experiences and content are considered to be the best in the industry, particularly when you’re aiming to give a great experience within the constraints of battery-powered devices such as HoloLens 2.

What are the primary challenges enterprise clients face when implementing the technology?

Historically, most use cases were hampered by at least one barrier to implementing a viable solution using today’s technology. Thankfully, many of these are being addressed in HoloLens 2, making them less of an issue going into 2020.  Some examples include reviewing 3D models of large infrastructure projects. These models typically have many millions of polygons, so clients use our new Azure Remote Rendering to HoloLens 2, or we optimize the model to 300K polygons, making it able to run on the HoloLens natively. Although this works in offices with great Internet access, clients often want to review large models at construction sites, where Internet connections usually aren’t ideal. Another solution – depending on model optimization costs and turn-around time - is to optimize the model down to around 300K polygons, so that it can run on the HoloLens natively. In addition, Object Theory can also help plan large new retail spaces in the design phase, providing guidance for complex assembly, diagnostic or repair procedures.

Can you provide brief examples of “use cases” in which Object Theory helped clients?

In one instance, we planned out a new large new retail space under design. This is one of the few use cases where the field of view (FOV) can present challenges. One of the most magical aspects of HoloLens is that you can experience a space at its true scale and walk around within an environment that does not yet exist. However, the small FOV creates challenges for gaining a genuine sense of spatial relationships, physical proximity, and potential choke points for navigating the space once it is built out. In other projects, we’ve provided task guidance for a complex assembly, diagnostic, or repair procedures.

Task guidance clearly offers high value. The creation of task guidance content is one where the cost of creation is low enough compared to the significantly high value, particularly for a task that will be performed countless times across a large workforce. Some of the current challenges include comfort, battery life, OSHA, and similar safety requirements, as well as maintaining good spatial tracking when close to the equipment.

"As strategic advisors, we ensure that any project we recommend to our clients take all constraints into consideration when identifying those specific processes, where spatial computing can deliver real value in light of those constraints."

Could you share a few examples of work done for clients by Object Theory that you feel have been successful and produced significant ROI?

The HoloLens companion we developed for Stryker’s ByDesign CAD tool helps their sales team show surgeons and other hospital personnel their future operating room and other patient care facility designs, before they are built. This has been saving Stryker significant time and money by reducing the need to ship and construct a temporary operating room design in a warehouse near a large prospective hospital client during the sales process. 

 Another project that has moved from PoC to pilot is a training solution for Jamco America.  They manufacture aircraft interiors for airlines such as premium class seats, lavatories and food galleys. We developed a task training PoC for dismantling their Venture business class seat to remove an electronics box. They demoed this at several industry events this year. It was quite amazing to see dozens of airline professionals dismantle this expensive seat in a matter of minutes, having no background in field repair. Right now, we're producing content for several complex repair procedures that they can put into production.

What key factors do you look at to prove ROI for your clients?

What we’ve found is that most of our clients are not accurately measuring many of the processes that are being moved to spatial computing. Instead, they rely on gut feelings and cursory observation. This is certainly true for most of the premium selling solutions we’ve developed for clients. The lack of metrics can create a challenge in comparing existing methods to spatial computing methods. For solutions where the opportunity exists to easily measure a process, such as in training situations, we do encourage them to perform comparison tests so that the increase in efficacy can be accurately measured. The factors we are looking for are:

1.    Reduced time to learn a task or a concept.

2.    Increased retention time of learned capabilities.

3.    Reduced errors in performing a spatially taught or guided task.

4.    Reduced rework.

5.    Lower labor cost of being able to leverage less trained workers.

6.    Increased training throughput by reducing dependency on expensive, limited access equipment.

Let’s say a customer is interested in setting up the HoloLens to assist technicians with training, maintenance, and repair of a piece of industrial equipment. Walk us though the process of setting this up for a client.

Here’s what we’re doing for a handful of current clients when it comes to producing spatial training and task guidance:

1.    Object Theory meets with client SMEs and ideally, our team experiences the exact training being created in HoloLens or reviews all existing training materials.

2.    A client provides CAD models for equipment, if possible, or we recreate what is needed.

3.    Each step is turned into a narrated and spatially-illustrated instructional experience that can be perfectly superimposed on the actual equipment. 

4.    The final training content is then placed in a Prism Space, which can then be managed and deployed as desired to trainers and trainees by our client.

It seems managing back end data will be key to successful implementation of the technology and getting the most out of your investment. Do you have any recommendations on how a company can integrate HoloLens into their current digital networks? How do you manage security concerns, such as protecting IP and sensitive data using HoloLens out in the field?

Yes, our enterprise clients need the ability to manage spatial content on a global scale. This includes providing and revoking access, updating revised content, and integration with existing processes and systems. For IP protection, all content is fully owned and managed by our client and access is controlled via Azure Active Directory authentication. Access to specific experiences contained in Prism Spaces, such as a particular training course, can be achieved either via authentication to one of these managed Prism Spaces, or via a secure unguessable URL that has been shared. That URL can then later be revoked at the end of a project or course.

In particularly sensitive cases, we will deploy a Prism back-end instance on our client’s own servers that they control behind firewalls.

What are your thoughts on integrating IoT and AI into the HoloLens workflow?

IoT, AI, and spatial computing are seen as the three pillars of the digital transformation and are becoming increasingly intertwined in solving complex business challenges.  

IoT provides sensory data, then AI analyzes that data and provides useful insights and more natural ways of interacting. Spatial computing enables experiencing the sensory data, combined with the insights in spatially useful ways. Finally, coming full circle via virtual controls, IoT can then be invoked to act on the environment using motors, pumps, and other actuators.

We’re embracing all three in our HoloLens workflows. For IoT, we’re starting to build the interfaces to standard IoT solutions, such as Azure IoT Hub, as well as custom integrations. On the AI front, we’re integrating natural language processing for more intuitive interaction and for multi-lingual collaboration. We’re also starting to look into computer vision to recognize objects both for specific parts identification, as well as for alignment.

Do you feel we are at the tipping point for enterprise MR mass adoption in early 2020?

We’ve already seen an increase in commitment from our existing clients at the end of 2019. With the launch of HoloLens 2, which is a significant step forward in meeting the needs of our clients, we anticipate that there will be a significant increase in MR activity and adoption, particularly in 2020 as new budgets start unlocking.

Do you have any closing thoughts or thoughts on HoloLens2?

As with all revolutionary advances in technology, when you’re deeply entrenched in the transformation in the early days it feels gruelingly slow, as all the requisite components of a complex emerging ecosystem are gelling. That ecosystem includes hardware, software, development tools, awareness, education, and financial commitments. This has certainly been our experience over the past five years, as we contribute to the viability of that transformation by producing content, applications, and delivery platforms.  

Now, with the advent of HoloLens 2, robust development tools and shrink wrap solutions, like Prism, combined with increased market awareness and success stories showing tangible ROI, I see 2020 as a breakout year where mixed reality is taken seriously for its ability to significantly impact operations in positive ways, particularly in training, task guidance, space planning, product design, and premium selling.