The IT&Dev Challenges of Autonomous Driving

The automotive industry is undergoing a level of changes never seen before: Electrical vehicles, sharing services and, last but not least, autonomous cars.

Owning a vehicle in big cities is already an option (small towns still have to find a way to build a network effect), at least for the second car, in the same way as “portables” and laptops were a “2nd system” option many years ago. The car industry is fundamentally changing from its origins.

Working with many automotive companies I can say that autonomous driving is quickly becoming the most important area of investment in IT (if not the biggest). A benign form of cold war escalation is taking place in the form of a race for autonomous driving across the industry.

Either alone, in partnership, with OEMs or in complete outsourcing there is nobody out there not having or building a plan for its autonomous driving capabilities.

The task of developing autonomous driving software is daunting. Safety requirements in the cars are different by states and countries but them all are very complex. SW companies no longer get away with the standard software licenses (i.e. something like “this software MAY do something similar to what we say it does, may also contains errors that cause you to lose a lot of data and make your system stop working… anyway we are not liable for it and, since you accepted our terms and conditions by ticking this mandatory checkbox”).

The implications of safety requirements for autonomous driving development and the software business in general is something I am looking forward to learn. It may well reshape the software industry.

The short term implication for autonomous driving companies however is very clear: you need to test your software A LOT. And test it with a huge amount of data collected by car sensors… in all kinds of environments - different types of roads, all possible weather conditions, different light conditions (i.e. time of day), different traffic conditions etc… you get the point. All of the above need to be tested in countless combinations.

This means, in short, a large number of vehicles with many different sensors (cameras, radars, lidars…) collecting data at 10Mbps+ speed for hundreds of miles and hours…

With every car producing anything between 10 and one hundred Terabytes per day (and you need a fleet of test vehicles) there are software development and IT challenges that are very specific to the autonomous driving sector:

• Logistics – how do you collect data from all these cars going around the country and have it sent to your datacenter (either on premise or in cloud)? This is an interesting challenge, requiring logistics capabilities, organization and careful planning. Unfortunately to transfer 50TB of data at the end of the day is not enough to go back home and connect your Ethernet cable. You need dedicated (very) high speed lines and/or a way to send your data cheaply and reliably to your datacenter. Depending where your test car go (and you cannot have only cars going around your factories) you may have issue with data protection, export regulations etc. Most likely after removing your disk/tape/cartridge from the car collecting mechanism of your choice (the actual device who receive data from sensors) you need to provide a replacement for next day's test drive, i.e. you have to manage spare parts. The car is becoming a datacenter on wheels.
• Data Quality – how do you ensure that your data are correct and enriched with the right metadata and/or attributes (like weather forecast, for example)? It may seems easy right? Data are just “streams” coming from different sensors (think of the classic “time-stamped records”). However how do you “compress” it to make more manageable, who do you store in a way that make them easy to search, group, combine?  
• Store and Protect - How do you store and protect an always increasing amount of data that is continuously created every day (think about Petabytes a day, not Terabytes)? To store this quantity of data you need a scale-out-by-design system that allows you to grow as you need and where you need, and to protect it you need to have a strategy that cannot be just “weekly backup”. Data retention requirements for regulation purposes add to this complexity and are still being created. How much capacity do you need if you have to back-up everything and store for, says, 5 years? And what do you store? Full data? Changes only? And Where do you keep it? At this scale, even the high number of drives has an impact if you think how MTBF combine on thousands of drives…
• Data Analytics – How do you run data analytics on Petabytes of data per day and stream it to different processing units, simulators etc? You need not only computers with GPUs that can burn a lot of data, but also storage systems and networks capable of moving data fast enough. Streams become of paramount interest in the autonomous driving world, where you want to see your data as something you can easily filter to provide multiple views and feed to different systems at the same time.
• Security – securing the infrastructure, as always, is critical. However there is a high probability that the development is not a standalone effort from a single company. You probably have to share your data with partners that very often are also competitors or competitor’s partners. Identity management, multi factor authentication and multi-tenancy are no optional if you want to ensure your only real asset (your data) is protected.
• Software Testing - How do you test your software against a huge amounts of data sets? Every developer must have a way to run his code on a system that has access to tons of data, then run it against different huge data sets. The answer, here, is easy: automation of testing, automation of deployment and automation of everything. There is no way you can run test cases that may require hours to run or the setup of a large number of different systems. Automation and orchestration is a MUST.

I would be curious to have your view on what other challenges you see as specific to the autonomous driving industry and, of course, what do you think of the challenges above. Feel free to leave a comment.