The Birth of Neuromorphic Vision: Part 2 - Applications
This is the second of a four-part series of short articles about the origins, applications, and future of neuromorphic event-based vision. In the first article, I provided a short overview of the long process behind the development of the Dynamic Vision Sensor (DVS). Here, I describe some of the key milestones on the way to bringing DVS technology to market, and outline some answers to the recurring questions about the technology.
What is DVS?
Unlike a normal camera, DVS does not capture frames (although there are also versions that can do this). Instead, its pixels operate independently and asynchronously to detect lighting changes, which are output as binary up/down signals. This occurs very fast, because only what has changed is being sent out of the camera. The result is much less output data, much faster (delay of microseconds instead of milliseconds). Because the changes detected by the sensor are always relative to the current local lighting, it also has very high dynamic range (typically around 120 dB), allowing it to respond well in scenes containing both very bright light and dark shadows. You can also watch a talk about DVS given by Tobi Delbruck at IBM below.
Cool Demos
As described in the previous article, the creation of the first DVS camera unlocked the imaginations of hundreds of people. Many technology demonstrators were created by labs and companies around the world. A small sample is available on the iniVation website here; you can also find an extensive list of resources on this page of the lab of Davide Scaramuzza. One of the first DVS demos, and still one of the coolest, was the RoboGoalie. It was designed to run on rather weak hardware, even for the time:
Despite these limitations, RoboGoalie demonstrated extremely fast reactions (<3 ms), beyond anything that had been previously possible on such a low computing budget, on the viscerally universal topic of football.
领英推荐
From Cool Demos to Industry Milestones
I helped to organize the first public industry trade show featuring the DVS in 2013, which built on several years of experience within the institute in presenting the technology at research conferences and workshops. For several years, the most common question was "What is that?", followed by "Does it work for X?". The DVS was so fundamentally new that a collective industry-wide effort was required to explore and understand its implications for computer vision. We have provided cameras to hundreds of organizations across all industries, including robotics, autonomous vehicles, consumer electronics, and aerospace. As of this writing, over 2600 papers on Google Scholar contain the phrase "Dynamic Vision Sensor".
Over the years, interest in neuromorphic vision has ballooned, with many customers and a number of competitors entering the scene. We have progressed steadily to reach a number of industry-first milestones:
Is DVS Useful for My Application?
The big question being asked by every industry is when (and where) DVS technology will go fully mainstream in their business. The answer is complex, and highly dependent on both the technology needs and the business environment of every individual niche. However, the key questions in each case can be summarized as follows:
The main technical trade-off can be summarized in the figure below. To date, neuromorphic vision sensors offer lower resolution (bigger pixels), in exchange for much higher speed and often lower power consumption. However, the technical sensor characteristics need to be considered together with the whole processing infrastructure: the type of processor used, the data bottlenecks, etc. For example, the ultra-low power consumption of the Speck vision SoC (system-on-chip) mentioned earlier is achieved by connecting a DVS directly to a spiking neural network processor, for which the DVS output is naturally suited.
While the technical considerations are an important part of determining the usability of DVS for applications, the other factors mentioned above are just as important. In the next article, I will dig into these questions for a number of applications, describe some of the technology and application gaps, and preview some of our future plans.
Your daily news source with 60K followers & 11 million impressions annually. Stay in the loop with exclusive semiconductor industry insights. INTERESTED IN COLLABORATION? E-MAIL ME at [email protected]
2 年This technology has been used in a variety of applications, including robotics, autonomous vehicles, consumer electronics, and aerospace. In recent years, there has been growing interest in neuromorphic vision and several companies have entered the market. I look forward to seeing the future of this market.
Verification Engineer
2 年BrainChip