Machines That See - Part 2

Welcome back to The Robotic Touch. Today’s discussion is the final installment of a two-part series exploring how computer vision is a vital part of a robotic piece-picking solution. Part one provided an overview of computer vision sensor technologies that can be brought to bear, and exposed some of the countless real-world scenarios that complicate the task of object detection. As promised, today I’ll introduce you to two engineers who are tackling these challenges head on. Those individuals will reveal details of the innovations RightHand Robotics, Inc is developing to meet the growing demand for autonomous, lights-out warehouse automation.

Engineering Insights

Staff Hardware Engineer Clark Teeple has a Ph.D. in Robotics from Harvard University, and has been with RightHand Robotics since 2022. Prior to joining RightHand, Clark’s work at Harvard focused on the design and control of dexterous, compliant robotic hands.

Clark was the technical lead on RightHand Robotics’ proprietary industrial camera development project. I asked Clark to share real-world problems that were considered when designing the product.

“When one of our RightPick systems receives a new tote, vision is the primary way the robot learns about its surroundings before deciding which item to pick. Our goal during development of this new 3D camera system was to increase the quality of visual data we collect. By providing our computer vision and machine learning algorithms with better raw data, we elevate their performance, and provide our robots with the best possible understanding of the world around them.?

One key example where data quality plays a big role is detecting objects: in other words, identifying what items to pick, and determining exactly where they are in 3D. This process is affected dramatically by a camera’s resolution, clarity, and dynamic range. Cameras with higher resolution (i.e. number of pixels) help our robots see smaller items, and allow them to determine item location to a higher degree of accuracy. Image clarity and dynamic range also come into play when looking for the edges of items, and are often directly tied to the quality of camera hardware itself. If images are too blurry, or have low contrast, it can be difficult to determine one item from another. Our new 3D vision system makes use of high-quality, high-resolution sensors to ensure the robot can detect every item as accurately as possible.

Another critical factor that often goes overlooked is workspace lighting. In the field, lighting in warehouses can vary dramatically, even within the same building, or throughout the day. Is the sun shining through skylights? Is there machinery overhead? Light brightness, shadows, and color casting can all affect our ability to clearly see items. Not only this, lighting can affect the reliability of 3D data, making it difficult to tell exactly where items are.

To make sure our robots are equipped to handle a wide variety of lighting scenarios, we use a two-pronged approach. Whenever possible, we control the lighting in our workcells by adding lights inside, and designing measures to block or filter light from outside. However, there are often other design constraints that limit our control over lighting, so we have designed our vision hardware and software to work reliably even in challenging lighting. In particular, our new 3D camera system has been designed from the ground up to be less sensitive to lighting conditions, and our vision algorithms leverage data from customers to ensure we can handle all sorts of different lighting.”

An immense amount of object detection “secret sauce” lies within RightHand Robotics’ cutting-edge software stack. Product Owner and CV/AI expert Madeline Goh is at the forefront of computer vision R&D for RightHand Robotics.

Madeline a Ph.D. in Mathematics from the University of Minnesota, 8+ years of experience in artificial intelligence, and holds six patents. She applies her deep knowledge of computer vision and machine learning to develop solutions for segmentation, object detection, anomaly detection, lane detection, simulation software, and barcode decoding.

I asked Madeline to share stories about how software innovation is solving everyday problems in warehouses around the world.

“We are constantly having to pick things we haven’t seen before, especially when we bring a new customer online with RightPick or an existing customer adds inventory we haven’t picked before. RightPick’s approach was designed with this in mind. One of the first tasks from a vision perspective is identifying items to pick in the tote. Our segmentation model is highly generalizable, and thanks to our many customers around the world, we have a very good representation of items to be picked. Our segmentation models, trained on a variety of customer data, give us the ability to routinely pick never-before-seen items. We are constantly improving our models by collecting training data from our customers and releasing updates to the field.

In addition to choosing what to pick, we need to choose how to pick it. We consider all sorts of factors in our decision: size of the item, height in the tote, quality of the data, approach angles, etc.

Piece-picking robots are a huge cost savings for our customers. However picking multiple items, i.e. overdelivery, is costly to the customer, particularly when the inventory is valuable or highly controlled. To reduce the rate of multipicks, RightHand utilizes several hardware and software approaches using various sensors for weight and visual recognition of multipicks. We’re constantly learning new information from the field that informs and enhances our detection methods.

In many of the warehouses utilizing RightPick, totes come and go on conveyors or are positioned by autonomous mobile robots (AMRs). This can introduce variability into the targeted pick location. An extreme example of this is when we are picking or placing to an unconstrained carton, meaning the carton isn’t locked into a predictable position when it arrives at the workstation. We take advantage of our rich depth data from our stereo camera to reconcile the tote’s location in 3D space with where we expect it to be. In the case of an unconstrained carton, we leverage the carton dimensions to search the depth data for a matching shape.”

A Bold Prediction

Within the next five years, I anticipate a transformative shift in the field of computer vision: off-the-shelf hardware/software vision solutions will become ubiquitous, affordable, and straightforward to integrate. This shift will commoditize object detection technology due to several converging trends:

• Widening Accessibility: The market will see an influx of vision devices and sensors, broadening options and reducing costs.
• Open Source Adoption: Platforms like Meta’s Segment Anything will democratize access to advanced vision capabilities.
• Data Availability: The proliferation of public datasets, such as Amazon's pick and place dataset, will fuel innovation in vision technology.
• Technological Advances: The advancement of Machine Learning models, particularly through the use of transformers in vision and Generative AI for object recognition, will not only boost precision and efficiency but also make object detection technology more accessible to non-experts.

These developments will level the playing field in object detection, compelling companies with vision-centric products to innovate beyond mere detection capabilities to stay competitive. Margins may narrow, and the competitive landscape will become more crowded.

What do you think about this prediction? If commoditization of vision occurs, what will be the implications?

Strategically Positioned

RightHand Robotics is uniquely prepared to navigate and thrive in this evolving ecosystem. Our turnkey piece-picking robot exemplifies our commitment to leading-edge technology, incorporating not only our cutting-edge vision system but also our proprietary hardware stack, including the RightHand Robotics Gripper, and a robust software suite.

Our vision system represents just one facet of our comprehensive solution. It's designed with the flexibility to seamlessly integrate new hardware, software, and AI advancements, ensuring our technology remains at the forefront of innovation.

As the landscape evolves, RightHand Robotics is not just poised to adapt; we're set to lead, offering our clients unparalleled efficiency, accuracy, and adaptability in automated object handling.

Onward Robots!