The Rebirth of Robotics: How Covariant Unlocks the Power of Deep Learning for Robots

Most robots these days make use of some form of Deep Learning. These are basically large neural networks that allow the robot to learn both the perception of an object(s) it engages with as well as the motion plan that determines how the robot will act relative to the object at hand. Deep learning and neural networks have been around since the 90s. But, something fundamental changed in the last decade. The Internet allowed for the collection of massive data sets that machines could train on, and the advent of cloud computing brought large amounts of computational power to the field. This allowed the pioneers in the field - people like Geoffrey Hinton, Andrew Ng, Fei Fei Li, Peter Norvig, and Jeff Dean - to direct this somewhat forgotten field of computer science to practical applications. Google was certainly at the forefront of that movement, but many others followed and the use cases have grown massively.  

Robots themselves have been around forever, but, with a few exceptions, have been disappointments. There is some usage of robots in several manufacturing applications, but these are highly programmed applications with very little variability in the course of action that the robots perform. Humans, unlike robots, are extremely adaptive, capable of dealing with novel situations and edge cases with minimal explanation. The use of robots in more “human” applications has proved elusive. But as deep learning continues to “graduate” from academic to commercial use, robots have found a second (or perhaps third) spring.  

Silicon Valley has embraced this return of the robot, and a myriad of robotics companies have been funded in recent years. As my colleague Damir Becirovic and I went on our walkabout in this space, we found companies that had focused on specific applications (everything from toy manufacturing to parsing recycled goods), to ones that designed arms with creative and innovative mechanical designs. We collectively met with more than 30 companies, but never fell in love.

Damir is one of the most diligent venture capitalists I’ve known. When he decides he is going to pursue an investment thesis, he will tirelessly pursue every lead, meet every company, and interview every knowledgeable person in the field. Damir was convinced that the way eCommerce distribution centers worked was broken. With the continued rise in commerce conducted online, every kind of business that sold physical goods was putting their inventory in these giant cavernous warehouses filled with endless aisles packed with products from floor to ceiling. Products ranging from consumer packaged goods, to apparel, to creams or pills, could be found somewhere in those plastic crates. These distribution centers (DCs) were a bustling beehive of activity. Customer orders came in, crates and baskets would move around, orders would be assembled - and without fail, there would be a person grabbing toothpaste and Purell and shoving them into a cardboard box to be shipped to your home.

Damir thought this was ripe for re-invention. The work of packing and stacking boxes, moving crates, and picking items from bins isn't a fun job. It’s monotonous, tiring, repetitive and prone to errors and mistakes. People did it for now, but that wouldn’t be the case in 5 years. If eCommerce were to account for >40-50% of all retail (up from 12% today), the industry would need robotics to automate operations in the warehouse and in delivery. 

A year into our journey of discovery, our good friend Chris Urmson gave us a tip. Chris is a discerning guy when it comes to robotics; he was CTO at Waymo for 9 years and famously led his Carnegie Mellon team to victory in the 2007 DARPA Grand Challenge. He encouraged us to meet with Pieter Abbeel. Pieter is the Director of the Berkeley Robot Learning Lab at UC Berkeley and studied under Andrew Ng as a Ph.D. student. He is one of the premier roboticists in the world and, as we learned from Chris, he was working on something special.  

Damir and I dutifully trekked up to Berkeley and enjoyed a relatively mediocre Mexican lunch at Celia’s with Pieter. While secretive about the nature of his work, Pieter shared with us that he was working on the AI software for robotic arms used in pick-and-place applications. We were incredibly impressed with him but didn’t learn much beyond. In fact, Pieter was evaluating us more so than vice versa. About 6 months after our lunch, we got an email from Pieter. We had somehow not failed his test and he wanted us to talk about a financing round for his business - Covariant. This time he was willing to show us the secret sauce.  

We were impressed for many reasons. First, the technology that Covariant had architected and built is generically applicable. While its initial use case is for picking and packing goods in a distribution center, the core of what they had built was a range of skills modules that could be applied to a wide range of applications that involved recognizing objects and using an articulated arm and hand to manipulate them. Covariant technology could be used in industries spanning eCommerce, healthcare, parcel delivery, manufacturing, and so on.  

Second, the technology was shockingly advanced. Through the use of some of the most modern deep learning techniques like reinforcement learning and domain randomization, they had produced a robot brain that could recognize even the most tricky objects and manipulate them with speed and extremely low error rates (important, because having a human “baby-sit” a robot defeats the purpose).

Third, they had identified the exact customer problem that we felt needed to be solved. The initial application of Covariant is for picking and placing objects in DCs. Distribution centers across the world struggle to find and retain labor for this task. It’s ungratifying, tiring, and it does not pay particularly well for humans. Not surprisingly, there is an acute need for a robotic solution in the market and the maturity of the deep learning technology felt like a perfect fit for this need.

Finally, we were blown away by the team at Covariant. Pieter had co-founded the company with two of his best Ph.D. students and alumni from OpenAI: Peter Chen, Rocky Duan, and Tianhao Zhang. Peter Chen is the CEO of the company, and they had assembled some of the most extraordinary talents in the field of deep learning to tackle the challenge. We were particularly impressed to learn after just a few conversations with the Covariant team that they were not only smart AI technologists but also good business people. Peter had a firm grasp of every aspect of the business - from the go-to-market strategy, to organizational culture, hiring, negotiations, and so forth. This was just as much Peter’s company. It was quickly obvious that Pieter and Peter are a powerful combination. 

Covariant co-founders: Tianhao Zhang, Rocky Duan, Peter Chen, and Pieter Abbeel.

Following our investment in Aurora, Covariant is the second major robotics company that Index has invested in. When thinking about this industry, the topic of “replacing human jobs” inevitably comes up. And it becomes particularly sensitive in recessionary times when unemployment is high. We have a lot of empathy for this concern. At the same time, we firmly believe that humans will find more meaning in their work not when they are performing taxing, mindless, and repetitive tasks that strain their bodies in unnatural ways. Those types of tasks are better suited for machines. And, history has shown that, following periods of industrial transformation, we find much better ways of using human talents. Without a doubt, robots are partial substitutes for human tasks, but they are far away from being human substitutes. And, we firmly believe that robots, like the ones created by Covariant, will make our lives better, and more productive.  

It is in that spirit, that we are incredibly excited to announce our investment in Covariant. The field of robotics is still in its early innings. But, we are confident that our journey with Pieter, Peter, Rocky, and Tianhao will be the beginnings of the creation of the next great AI Robotics company. We can’t wait to get going.