Why do robots in fulfillment automation fail at gripping and manipulation?

What’s the problem?

Robots must first locate and identify a target item among many others. This often requires avoiding obstacles in the workspace to reach the item and perform given tasks. The robot must have an excellent perception system, efficient navigation skills, and accurate manipulation capabilities.

A recent article September 2022 from Amazon | Science | Robotics called?“Pinch-grasping robot handles items with precision”?piqued my interest to do a deeper dive into this subject.

Pinch or vacuum debate:

“In robotics, we don’t have the mechanical ability of a five-finger dexterous hand,” said Aaron Parness, a senior manager for applied science at Amazon Fulfillment Technologies & Robotics Robotics AI. “But we are starting to get some of the ability to reason and think about how to grasp. We’re starting to catch up. Where pinch-grasping is really interesting is taking something mechanically simple and making it highly functional.”

This catching up is powered by breakthrough machine learning capabilities aimed at understanding the three-dimensional geometry of cluttered environments and how to navigate in them, according to Siddhartha Srinivasa, director of Amazon Robotics AI.

“Not only are we able to build robust three-dimensional models of the scene, but we’re also able to identify a specific item in the scene and use machine learning to know how best to pick it up and to move it quickly and without damage,” he said.

Ken Goldberg professor at UC Berkeley said “Humans are capable of between 400 and 600 mean picks per hour. In a contest organized by Amazon recently, the best robots were capable of between 70 and 95. The new machine reaches 200 to 300 mean picks per hour” in an?article from Ambi Robotics. . He goes on to say that “The key to its dexterity is not in its mechanical grippers but in its brain. The robot uses software called Dex-Net to determine how to pick up even odd-looking objects with incredible efficiency”.

Nimble Robotics?use a variety of different grippers and supervised autonomy to reliably handle nearly any object or product that fits into a bin,” said Simon Kalouche . “Our AI learns what grippers work best on different objects and automatically switches its gripper to properly pick, pack, and handle each object.”

Random robotic picking “is a niche today in the overall warehouse automation market, but it’s growing rapidly,” said Ash Sharma, managing director of market-research firm Interact Analysis. That sliver of the market will reach $1.34 billion in 2025 from an estimated $137 million last year, the firm projects. The broader warehouse automation market was nearly $36 billion last year, Interact Analysis estimates.

Customer Fulfilment Centre (CFC) have to handle random item picking day in and day out, this type of manipulation requires a convergence of technologies, particularly three main components that make a successful system: computer vision (eyes), software (brains), and end-of-arm tooling (hands). On the upside AI, machine learning techniques allow picking to improve over time making up for dropped items over time.

What is being done?

ASTM International’s committee on robotics, automation, and autonomous systems?(F45) has formed a new subcommittee on grasping and manipulation. This new subcommittee (F45.05) will develop standards that evaluate performance in several major areas of robotic manipulation. The subcommittee will be initially headed by two co-chairs, Joe Falco and Omar Aboul-Enein, both from the National Institute of Standards and Technology (NIST) . Aaron Prather , ASTM International ’s new director of robotics and autonomous systems programs, says “these standards will help speed up deployments and cut wasteful spending on selecting the wrong tool.”

The first three task groups of the committee will develop standards for the performance of grasping type end-effectors, mobile manipulators, and robotic assembly systems, covering their use in both fixed and mobile base systems.

“As robotics and automation continue to expand into new and diverse industries, performance standards that help end users better select their end-effectors and/or manipulators to the task they are working on will be key,” said Prather. “Seeing the number of experts from across the world joining this work shows just how much this group is needed.”

What it takes for a successful pick?

It all starts with the application and items being manipulated, boxes, bags, random items, and how the item is presented to the robot. The key is to thoroughly understand all aspects of the requirements, each have best solution for automated picking in an unstructured environment. The vision system, stereo vision or 3D and AI software process item recognition, position orientation, pick planning, motion planning, task planning, error detection to the robot and finally EOAT, eyes, brains, arm and hands all working together. While great strides have been made in robots, vision and software it seems that EOAT technology is lagging.

Eyes— Advances in vision stereo and 3D camera technology improve handling, picking challenges like complex shapes, opacity, and reflectivity. Photoneo, Zivid, Pickit, Solomon Cognex, Keyence, IDS, LMI Technologies, Roboception GmbH, Fraunhofer IPA, Euclid Labs, VISIO NERF, ISRA VISION, MVTec Software GmbH, OCTUM GmbH, SICK Sensor Intelligence, elunic AG, phil-vision GmbH, Teledyne Imaging, INSPEKTO, SensoPart, PSI Technics GmbH, SmartMore Corporation, Visometry, Fraunhofer IWU and more. Note that stereo vision systems, RGB-D cameras are less expensive but need well-lit environments to get grasp point from RGB data. Due to the lack of constraints on the range of motion on the target object like in 3D profiling technology, stereo vision can cope well with long distances and moving objects this is something that other 3D imaging falls short on.

Brains?– Autonomously manage the tasks. AI software, expansive data set and deep learning algorithms enable the robotic automation of unknown objects being picked from bulk. Software AI hybrid: Covariant, Ambi Robotics, Dexterity, Fizyr, OSARO, Nimble Robotics, Universal Logic Neocortex, Robominds GmbH, and Energid these companies provide a hybrid of software, hardware and AI combinations to make a very effective picking solutions with each having advantages.

Hands?– Grippers, also known as EOAT, end of arm tooling, end-effectors, grabber, grasping device, robot grasper, pinch-grasper, robotic hands, pinchers or manipulators are the last part of the robot to make contact with the item, one of the most important parts in the system.

The problem in random manipulation is that one gripper often doesn’t solve every random resting state of a part that’s in a bin, says?David Dechow a machine vision expert.?“Certainly, the imaging components are critical in determining what part can be imaged, but the end effector becomes one of the last components on the critical path in determining whether or not the part can be picked.”

Vacuum cups?are the most common forms of manipulation of unstructured items. Vacuum cups are the default technology for robots tasked to pick up and move items of different shapes and sizes. The major issue with vacuum cups is that the robots must reduce speed in order not to break the vacuum and fling the picked object during change in pose or deceleration. Some vacuum cup manufacturers are, Schmalz, piab, VMECA, SMC, DeStaCo, AGS, Festo, Coval, Anver, Joulin, and Aventics.

Oxipital AI Gripper

Mechanical option?is an open loop gripping via mechanical pincher. In this case the robot chooses a single gripping pose from a single image, and then blindly executes this grasp. This method has a 30% average failure rate on the first 30 picking attempts for this set of ecommerce objects out of a bin. The pick accuracy can be increased with multiple RGB-D cameras and AI software. The major disadvantage is the time it takes to grasp the items vs vacuum cups. Manufactures that offer off the shelf grippers are Schunk, Robotiq Onrobot, SoftRobotics, Zimmer GmbH, Festo, and PHD.

What’s new for commercially available EOAT that address the fulfillment automation?

Kindred AI fulfillment

New advancements in?hybrid multi-function EOAT?for manipulation of items for fulfillment application are starting to show up. The hybrid combines vacuum cup and a mechanical gripper, the following companies are making hybrid, mechanical pinch gripping combined with a vacuum system.?Swisslog, RightHand Robotics, and Kindred take advantage of the best of both worlds, the speed of vacuum and the firm grasp of a mechanical gripper.

Why aren’t more companies using this?

MaxxGrip from The Gripper Company

Companies that make these multi-function EOAT only sell systems and not a standard gripping solution, but a startup from Denmark already launched a solution.?The Gripper Company?is making a stand-alone hybrid gripper for automated picking solutions for more efficient manipulation. The?MAXXgrip?has an extendable/retractable linear actuator vacuum cup and four compliant grasping fingers. This hybrid technology stack involves both soft and hard kinematics and is another advantage of a hybrid TGC compliant gripper prevents product damage that a standard mechanical gripper may cause when picking. Its actuation is design so it can be executed in parallel with the approach/depart motion segments used in pick and place cycles, and due to its fast response 0.15 -0.4 sec., but as this motion typically takes around the same time to move, you barely have any dwell time loose in your major successive pick and place motion cycle. “Test results picking from a 340mm deep tote with a broad varying collection of consumer goods and placing them onto a moving belt conveyor – similar to placement into a cardboard box or equivalent, documents rates around 20-30 cycles per min. meaning fully utilized 1200-1800 pick per hour”, states CTO Preben Hj?rnet .?

Fizyr.AI gripper

New hybrid systems from?Fizyr?combines vision software and end of arm tooling. They offer a logistics vision software and adds a new gripper for picking efficiently, even from difficult angles and corners in bin. The benefits are, two extra degrees of freedom, gripper can cope with 6DOF (object) orientation, no external wiring, continuous rotation, easily adaptable to various lengths, optimized for high air flow; tube diameter ?10 mm, suction cup can rotate ± 80°.

KNAPP Pick-It-Easy robot, powered by the? Covariant ?Brain join forces to solve logistics with groceries, food retailers since they increasingly rely on central fulfillment and micro fulfillment centers to process orders.

New vision products from? Photoneo, now part of Zebra Technologies , MotionCam-3D outperforms standard 3D vision methods to allow shorter cycle times and higher productivity, efficiency, and throughput in countless applications. These include parcel sorting in order fulfillment, palletization and depalletization, dimensioning, counting, and other.

The fact is that random bin or conveyor picking requires a convergence of technologies to be more efficient, hybrid systems will continue to develop including new EOAT that combine gripper and vacuum to achieve speed and a firm grip for automated picking in an unstructured environment. Companies are using a trifecta of technologies, software, vision and EOAT to power robotic arms to pick and sort e-commerce parcels, pack food, supplies, and consumer products from bins and conveyors to prepare orders for 3PL, DC that deal with the fulfillment industry and as the technology matures the pick rate ROI will increase.

Peter Farkas : Disclosure statement: No potential conflict of interest was reported by the author.

#artificialintelligence ??#machinevision ??#industrialautomation ??#collaborativerobots #robots ??#grippers ??#motioncontrol #SoftwareSimulation??#softrobotics ??#supplychainsolutions #supplychain ??#logisticssolutions #logistics ??#warehouse ??#ecommercesolutions ??#materialshandling ?#picking ??