Can we certify autonomous vehicles using synthetic simulations?

According to a report written by the RAND Corporation, autonomous vehicles will need to drive for billions of miles in order to get a definitive on whether it outperforms human driving in terms of safety or not. If this were to be a one-time process for regulatory purposes it might not be so bad, but unfortunately it won’t be. 

Once the autonomous market matures and the first Level 4 or 5 production vehicles hit the street, the number of off-chance accidents will increase resulting in daily media attention putting the AI under scrutiny from both the public as well as legislators. Was the manufacturer at fault? Could this have been prevented? Did they neglect to fix a recurring scenario?

This will mark the end of the wild-west where manufacturers update firmware whenever they see fit. Governments will start enforcing strict regulatory oversight by enforcing certification on all autonomous components requiring them to be put through stringent testing scenarios. 

Due to the way most deep learning based neural networks currently behave this process might have to be repeated every time a single sensor like a camera, LiDAR or radar is added, removed or even moved. In addition to the certification process, this will also be a huge marketing requirement. Everyone will want to buy a model that’s 5.0% safer than last year’s model. Given at this point we’re benchmarking against vehicles superior to human drivers the number of miles will balloon even more before a statistical significance is reached.

Arguably most autonomous vehicles developers are still in very early stages of development and L4/5 mass production is far from a reality. However, the problems that loom ahead present a huge opportunity for the quickly materializing simulator market that seeks to speed-up this process by using virtual worlds. 

Synthetic to the rescue?

While driving around in a virtual world seems like a great idea to cut costs and increase testing scalability it does come with a fair share of its own problems. 

The core problem and an active topic of research is the neural network’s ability to transfer its knowledge from one problem domain to another i.e. from synthetic environments to the real-world. As it turns out after training, our mighty AI overlord has problems cruising around its environment if does not resemble a comic book version of the world. This gap in realism will partially be bridged by more resilient deep learning algorithms with a greater ability to generalize, but in the meantime, many novel ideas have been presented to work around this: from style transfer and segmentation maps to realistic rendering; and safe to say we are working on all of the above. 

However, to build a fully realistic simulator suitable for certifying an autonomous vehicle we’ll have to go a lot further than just making it ‘look’ realistic. Most of the attention and research is focused on the use of synthetic camera data, but equal problems are present in the usage of synthetic LiDAR, radar and infrared data. While these types of data are arguably of a lower visual acuity than cameras, they do present their own unique problems. For example, current generation LiDAR devices are heavily affected by adverse weather conditions like snow and rain, while seeing black objects at a distance is near impossible. Similarly, near-infrared cameras have problems seeing objects that absorb infrared light or far-infrared/thermal cameras working in heat-soaked environments. 

While we have made great strides in modeling these issues, it remains an area of active research that sees us working closely with sensor manufacturers. Without attention to this level of detail, there is no guaranteeing the results produced by simulations.

Real-time Simulations

Assuming the problems sketched above will be resolved in the coming months to years there is still the matter of the actual simulations. While there are many types of simulations currently in use for testing electronics, there are two which are of particular interest when it comes to synthetic data:

The first solution is the SIL (Software-In-Loop) or full software simulation, in which both the virtual world and the operating neural network run on commodity hardware with GPU acceleration. This type of simulation allows for faster than real-time testing and scales easily in terms of the number of active seats. Simply running another copy of both and more miles are being simulated every minute. While this seems like an ideal situation the fact none of the actual hardware is involved in testing it takes out 90% of the validity of the simulator. There will be no way, for example, to test the validity of an NVIDIA PX2 or PolySync ECU.

The second, and in our opinion preferred solution, is a combination of SIL and HIL (Hardware-In-Loop). By simulating all connection types and protocols it's possible to fool the autonomous control unit of a car into believing it’s driving on an open road. Something akin to the VR experience for us humans. In this solution the unit to be tested can be made of its own complex set of hardware requirements and the end-result simulation will be as true as possible to the final product. Although this provides the most extensive capabilities for testing it comes with a huge caveat: the simulation has to run in real-time. Due to the complex interaction between different devices and their protocol, it is impossible to speed up world-time without breaking their connectivity and causing artifacts in the data. This brings us back to the original RAND article; but, instead of cars on the road we’ll now have hundreds of car ‘brains’ hooked up to simulators day dreaming away in a data center.

Our current research is heavily focused on reducing this requirement. The original estimates of billions of miles for safety calculations is based on world wide average human driving which inevitably consists of mostly boring event less highway driving. By focusing our simulations on problem areas and increasing event entropy e.g. near accidents and adverse weather conditions, it’s very possible to reduce this number to several millions of miles only. 

In Conclusion

As we are witnessing the birth of yet another industry surrounding the development of autonomous vehicles, I’m cautiously optimistic about the actual market potential. I strongly feel there will be very few SaaS opportunities when dealing with OEM’s. Given the extreme complexity of modeling accurate simulator-hardware interactions, the in-depth knowledge possessed by both parties, and combined with the significant investments required in digital assets and continuous customizations, their most sensible option would be to build in-house or make acquisitions. 

There will, however, be a huge long-tail of companies that are involved in problems of a reduced scope: smart sensors, drone manufacturers, rail transport, etc.

If you would like to learn more about Cvedia and SynCity and how we tackle our deep learning issues feel free to get in touch via LinkedIn or email at [email protected]