The Birth of Neuromorphic Vision: Part 1 - Origins

This is the first of a four-part series of short articles on the origins, applications, and future of neuromorphic vision, which I have been fortunate to witness first-hand for a significant part of the journey. Here is a list of links to the other articles:

• Part 2: Applications
• Part 3: The Future (Section 1)
• Part 4: The Future (Section 2)
• Our next-generation Aeveon sensing technology

Origins

Neuromorphic engineering originated in the 1980s, in the lab of Carver Mead at Caltech. In a 2020 article, he describes how he was fascinated by how animals computed. If we could understand and replicate key elements of natural nervous systems, then perhaps we could unlock new levels of speed and efficiency in our own technology.

A key early figure in the field was one of Carver's PhD students, Misha Mahowald. In 1988 they published a seminal paper on a model of early visual processing - the "silicon retina". In 1991, she co-authored another breakthrough paper with Rodney Douglas, the "silicon neuron". A small, yet growing community of researchers held the first neuromorphic engineering workshop in 1993. This became known as the Telluride Neuromorphic Engineering Workshop, which still runs annually to this day.

In parallel, the first systematic institutional efforts crystallized to push research into neuromorphic engineering. The University of Zurich and ETH Zürich jointly created the Institute of Neuroinformatics in 1995. Rodney Douglas and his long-time collaborator Kevan Martin were hired as the founding directors, along with Misha Mahowald and several other former students of Carver Mead.

I started at the institute in 2000, entering research after working at ABB and Alstom. The institute was (and is) heavily influenced by its warm, informal Californian origins, which was a big contrast to the staid Herr-Doktor-Professor engineering labs that were prevalent in the German-speaking world at the time. The strategy of recruiting a mix of faculty in biology, engineering, and theory produced an eclectic mix of personalities and projects, as well as a pervasive atmosphere in which anything seemed to be possible.

A Usability Breakthrough

A significant step in the development of neuromorphic vision was the EU-funded Caviar project, combining components and efforts from several Zurich and other European labs. The silicon retina part of the project was led by J?rg Kramer and then by Tobi Delbruck, after the unfortunate passing of J?rg in 2002. The final demo implemented concepts such as the Address-Event Representation (AER) that are still in use today. It was also around this time that the term "event-based" computation emerged. As can be seen in the image below, the integration of the system was highly complex. With its heavily analog-based components, factors such as temperature variations and slightly-too-long-cables could directly impact its stability and performance. How could something like this be simplified to something that worked reliably?

Around this time, Tobi and his collaborators made what I believe were key practical contributions to the commercial viability of the silicon retina, and neuromorphic systems in general:

• A new, simple, stable switch-cap pixel design (in particular by Patrick Lichsteiner)
• A software-controlled bias generator, which allowed chips to be configured repeatably by uploading settings instead of manually adjusting potentiometers using plastic screwdrivers.
• Adding a USB cable, and then interfacing it to a library written in Java.
• Dropping features from Misha's original retina. In particular, the hexagonal pixel array was dropped in favor of a square array, and the connections between adjacent pixels were removed.

The difference in reaction from visitors was remarkable. Suddenly, what had been a baroque, complex maze of chips, adjustments, and wires, now looked just like a webcam. This abstraction of complexity greatly simplified the ongoing task of the researchers in explaining the technology and its potential benefits. People no longer had to comprehend what they were looking at, and could focus their thoughts on what they could do with it.

First Commercial Sale

In 2008, I was sitting in the institute's meeting room (the "red room") together with Tobi, Rodney, and institute general manager David Lawrence. The years invested by Tobi's group members explaining the technology were starting to yield fruit; the first silicon retina USB camera was being sold to an actual paying customer outside the lab. The only problem was that we didn't have a product name. After an hour or so we came up with a name - the Dynamic Vision Sensor, or DVS for short. The following year we founded an incubator named iniLabs to support this and other technologies coming out of the lab.

Interestingly, this was actually the world's second commercial sale of neuromorphic vision technology. Some years earlier, another member of the neuromorphic engineering community, André van Schaik, had designed a custom optical sensor for a trackball made by Logitech.

In 2012, we became involved in the SyNAPSE DARPA project coordinated by IBM, interfacing the DVS to their TrueNorth neuromorphic processor. Around the same time, a collaboration with Samsung Electronics also began. Commercial interest in DVS technology began to accelerate, and in 2015 we co-founded iniVation, together with Sven-Erik Jacobsen, Yvette K?rber, and Matti Weinberg. The next short article will cover our journey towards market.

Postscript: The Neuromorphic Engineering Prize

In 2016, Tobi, Rodney, and I co-founded the Misha Mahowald Prize for Neuromorphic Engineering, to recognize the best new developments and motivate progress in the field. We named the prize to recognize her unique contributions, which were all the more remarkable in that they were achieved in a heavily male-dominated arena.