From Idea to Impact: How Toyota Connected India is Innovating Vehicle Safety with Advanced Hearing Technology

Dive into the dynamic world of Toyota Connected India, where innovation meets urgency to tackle the automotive industry's most pivotal challenges head-on. Our team is on a mission to craft transformative solutions that resonate globally, shaping the daily lives of millions. In our blog series, "From Idea To Impact," we unpack the steps that bring some of these nascent concepts to fruition. Discover what it truly means to ignite change, revolutionize our roads, enhance vehicle experiences, and ultimately champion sustainable mobility in a world in perpetual motion.

In this month’s deep dive, Sudharshan Atreya shares how Team EchoVision (that includes Apoorva R. Dasaradi Budhi Sravan P. Nimeesh Pitta Abuthair Hussain Jaai N. Sebastin A. Siva Chokkalingam ) took the idea of locating an ambulance siren's direction while driving to developing a full-blown Sentry Mode Solution that adds a new layer of safety and security to cars.

As part of the Data Services Team at Toyota Connected India, the team is at the forefront of pioneering data-driven products and using cutting-edge Computer Vision and Machine Learning (ML) technologies to enable them. The team is finding solutions to enable vehicles to perceive things around them just as humans would.

Can you tell us a little about what your team does and a project you've been working on?

Assume you are in the middle of traffic in your car, and you hear an emergency vehicle’s siren. Our first response would be to find out where the emergency vehicle is. The way humans do this is, we look around and localize the sound to find the source’s direction. We then look in a specific direction, followed by a look around for any emergency vehicles. At this point, we wouldn’t focus on any other audio (like conversation or music in a vehicle). We wouldn’t look for a different type of vehicle or the surrounding area as well. We would specifically look to identify the emergency vehicle. This is because the human brain has already perceived the audio as an emergency vehicle and has instructed our vision to look out only for distinct things in a specific direction. That is exactly the perception we want to give to our cars.

Our team focuses on enhancing vehicle intelligence through advanced sensory capabilities. While cars are already equipped with cameras to "see" their surroundings, our goal was to give them the ability to "hear" like humans. This idea led us to develop EchoVision, a software that allows cars to identify important audio cues and respond appropriately, making driving safer and more inclusive.

What problem are you trying to solve with EchoVision??

By incorporating this auditory perception into cars, we aim to enhance their environmental awareness and overall intelligence. Audio gives important cues that are very hard to identify with human eyes. For example, a bearing noise from the front-left wheel or a collision from behind. The audio cues first tell us what to look for and in what direction.?

So, in its most basic technical form, EchoVision uses an array of microphones strategically placed around the vehicle to capture sounds from all directions. Advanced audio signal processing algorithms then analyze these sounds in real-time, filtering out background noise and classifying critical audio cues using their frequency. The system employs machine learning models trained on vast datasets to recognize specific sounds, such as sirens or honking, and determine their location relative to the car. This information is then fed into the car's central processing unit, which can trigger appropriate responses, like alerting the driver or automatically slowing down.?

How do you take your ideas from concept to fruition? What does that process look like??

Our process typically begins with ideation sessions, where team members present their concepts. These ideas are then evaluated for feasibility and potential impact.?

For EchoVision however, the process started out a little differently. It was born out of TCIN’s flagship hackathon, Hackcelerate. With guidance and support from the leadership team and various members of the R&D teams at Toyota Connected India, the idea underwent various rounds of conceptual and technical iterations as we went into development.?

What are some challenges that you faced during this project? And how do you overcome it??

Audio processing is still an active research area. So, there are various complexities in the techniques and processes involved to help the camera focus on the exact cause and direction identified by the audio processing pipeline.?

Another big challenge with an idea such as this is the operational limitation that comes with it. Enabling a car with Sentry Mode is cool and a most-desired feature among customers. This means that the car needs to be actively listening for threats even when it is not operational. That brings us to the next big problem: how do we manage battery drain??

To help contextualize, cars need to listen to a vast number of signals from their surroundings and process them for anomalies for hours at a time without draining the battery by more than 20% in an 8-hour time span. Keeping the entire audio, CV, and ML processing is going to drain the typical 70Ah battery in just a few hours, leaving the car to starve power for the initial crank. Our teams explored various possibilities of just having the aDSP processors awake while putting the other cores, GPUs, and display components to sleep to save power.

Our ingenious team of core developers created a solution for delegating the audio signal processing to the DSP processors, which consumes very little power, and augmented the algorithms with computer vision-based only-on-need basis to ascertain the inferences. It was a tough problem to solve but offered incredible learning for the team.?

What is the real-world impact of the project you’re working on??

EchoVision has the potential to revolutionize vehicle safety and security. Imagine leaving your car in an unattended or open area. With its sentry mode, the car is equipped to monitor its surroundings actively and intelligently.?

In sentry mode, the vehicle's microphone array and cameras remain vigilant, continuously capturing and analyzing audio and visual data. If the system detects unusual sounds or movements, such as the noise of breaking glass, attempts to tamper with the tires, or any form of vandalism, it can trigger immediate responses. These include sounding an alarm, recording the incident, and sending real-time notifications to the owner's smartphone.?

Our hope is that through this proactive monitoring, we are able to ensure that your car can safeguard itself against external threats, providing you with peace of mind. Whether it's protecting against potential intrusions or alerting you to suspicious activities, EchoVision's integrated audio-visual surveillance significantly enhances vehicle security, making it a reliable guardian even in your absence.?