The Autonomous Vehicle Ecosystem- Extended Edition

The world of autonomous vehicles (AVs) extends far beyond the likes of Waymo, Cruise Automation, Uber and other well-known operators. Many specialized firms are vying to sell pickaxes during the gold rush that is the largest transportation revolution since the automobile. This is a deep dive into their expanding ecosystem in three parts: an overview of the major components, why to build vs buy and what’s next. These can be read independently. 

If you’re new to autonomous vehicle technology, it’s worth learning about the basics first. This blog post from Udacity is a good primer, as is their free online course if you have more time and interest.

Mapping

What is it: hosting, streaming and/or collecting data for machine-readable maps

Why does it matter: high-definition (HD) maps enable a self-driving vehicle to do two important things. To localize itself (i.e. know where it is in the world), and to more rapidly perceive its surroundings so as to spot anything transitory, such as a pedestrian. They are considered high-definition because they have much richer and more precise data than what’s on Google Maps or Apple Maps. In terms of detail, HD maps contain information like the height of a speed limit sign, the timing of a traffic light cycle or the location of on-street parking spots. As to their precision, millimeter-level detail is common.

Companies to watch:

AV-specific: Carmera, DeepMap, lvl5, Civil Maps

Mapping generalists: Google, Baidu, HERE, Mapbox

Simulation

What is it: software which lets a self-driving software system practice how it would respond to different road scenarios without actually physically driving the scenario. Like SimCity for a car.

Why does it matter: Simulation rapidly accelerates the improvement of self-driving vehicles since it means more practice for their software. As one data point, Waymo has driven over 7 billion miles in simulation while only 10 million miles on the roads. While these mileage figures are not directly comparable, and some companies don’t even track simulation miles in this fashion, it gives a sense of the different magnitudes.

Companies to watch: Applied Intuition, Cognata

Teleoperation

What is it: hardware and software solution which enables a human operator to remotely control a vehicle using wireless technology and a set of controls (steering wheel, pedals)

Why does it matter: This technology can let a company launch self-driving vehicles on the roads sooner, as it lets them still rely on humans to handle difficult and uncommon scenarios remotely. There is a spectrum of how much control the remote human can exercise on the vehicle. Some have full-blown remote driving solutions, while others merely are able to confirm decisions the vehicle proposes first. Cybersecurity and wireless connectivity are both factors in the decision. 

Companies to watch: Phantom Auto, Scotty Labs, Designated Driver, Ottopia

Data annotation

What is it: systems to efficiently handle the large volume of data produced by self-driving vehicles during operations

Why does it matter: AV companies produce terabytes of data daily from each of their vehicles. Quickly and comprehensively managing, labeling and storing this data is critical to system improvement and to safety. For example, when a system disengagement occurs, the situation should be analyzed promptly. This ensures the learnings are incorporated into future driving by the entire fleet’s vehicles. The AV company sends the scene data via API to a data annotation company. Then, the annotation company assigns the work to remote workers, not unlike Amazon’s Mechanical Turk. These workers label the road scene observed in the real world so the software system can understand it.

Companies to watch: Scale.ai, Deepen, Mighty AI, Datatang

Object detection & prediction

What is it: software system which analyzes objects in the environment and predicts their behavior

Why does it matter: Human beings are challenging for computers to predict, as we sometimes use subtle non-verbal forms of communication. A nod, a wave or brief eye contact all indicate what a pedestrian might do when standing next to a crosswalk. Interpreting these signals properly not only means a smoother ride but could even help avoid collisions.

Companies to watch: Perceptive Automata

Dispatching/network operations

What is it: software to ensure coordination and intelligent movement of all vehicles in the network, as well as with other road users. In essence, air traffic control for self-driving cars.

Why does it matter: Constant communications with vehicles will result in more efficient operations, and thus more affordable services. This will benefit operators, but potentially also other road users and cities. This is a very emerging field and tied closely to the mapping providers, as routing depends on real-time updated maps to handle situations like an accident or a construction zone. These firms also are providing building blocks such as routing APIs and ridehailing mobile apps for new mobility operators to launch services more quickly or their improve efficiency.

Companies to watch: rideOS, Wise Systems, INRIX

Driver monitoring

What is it: camera and software systems which watch a driver to ensure they are focused on the driving task and intervene if they are not

Why does it matter: Although unnecessary for self-driving vehicles, driver monitoring systems are critical for testing when vehicle operators are present. Waymo installed a DMS last year as it worked to address vehicle operator fatigue. The fatal 2018 Uber collision in Tempe, AZ was a tragic reminder of what can happen- the driver was not paying adequate attention to the vehicle just before the incident. There are different driver monitoring strategies- emotional state and attentiveness are two main ones. To go deeper, listen to this Autonocast episode.

Companies to watch: Affectiva, Seeing Machines, Nauto

Build versus buy

So far, many AV operators use few external service providers at all- they built bespoke systems. In fact, Voyage is one of the very few AV companies which has been open about its decision to use external specialists. How do other AV operators think about this?

Reasons to buy:

• focus (don’t use precious engineering resources to build tools that already exist)
• cost
• speed and scale
• demand fluctuation
• managing a remote workforce
• large up-front hardware costs (e.g. buying your own mapping vehicles)

Reasons to build:

• protect intellectual property (IP)
• competitive moat
• better interoperability by controlling the whole system
• lots of $$$ from investors
• pride (“our tools are better”)
• high SLA or quality standards
• reliance on a provider who could go out of business
• unique sensor stack (requiring too much integration work)

This last one is a meaty topic, with different use cases like sidewalk robots or semi-trucks requiring vastly different sensor setups. If curious, read this article on sensor placements.

What’s next?

There exists a tension between all the investment which has flowed into AV ecosystem companies and the many operators choosing to mostly do it themselves. Expect this to change. Much like the auto industry itself, vertical integration makes sense in the early, experimental days of an industry’s existence but will transition to outsourced specialty firms. During the 1930’s, Ford’s Rouge Complex employed over 100,000 workers and produced raw materials making glass and steel all the way through the final assembled vehicle. Now, more than 85% of a car’s internal systems are often produced by suppliers, leaving little more than the engine to the carmaker (source). This shift stems from increasingly complicated subsystems and a growing desire to control costs through outsourcing. A flywheel will accelerate this: increased sales of these services drive down their prices, attracting yet more sales.

Two criteria may predict which services are outsourced first: modularity and adjacent business. Modularity means the service is easily able to integrate with the rest of the operator’s AV system. An adjacent business means the service provider can build a solution that allows them to make money from other industries while adoption by AVs is initially gradual. Data annotation exhibits both traits and is widely outsourced today. The information involved is relatively standardized across companies. Also, such data review services can be sold to AR/VR, retail and drone companies. 

How do you see this ecosystem evolving? Will these new companies become major players much like today’s auto suppliers are in their industry? Or could AV operators buck the trend and keep building everything in-house?