Is Tele-robotics over 5G combined with AI-based automated decision making the future of robotics?

Dr. Sanjoy Paul

Managing Director, Accenture Technology Labs

Haitao Zhu Sri Sadhan Jujjavarapu Lauren McKeown Trisha Ashish #5g #5gtechnology #robotics #teleoperation #innnovation

Introduction

The global Tele-operation and Tele-robotics market are poised to reach $81.9 billion by 2027. Narrowing the focus to 5G, the Global 5G in cloud robotics market will reach $10.6 billion by 2028, growing at 79.2% CAGR [RM22] which is one of the fastest growing market segments. AI software in support of the teleoperations market will reach $5.3 billion globally. Tele-maintenance solutions will grow at highest CAGR during forecasted period globally. Cloud robotics as a service use case will increase over 70% due to teleoperations solutions. All these market signals indicate the importance of Tele-robotics over 5G.

Given the huge potential growth in Tele-robotics, Tele-operations and Tele-maintenance, it’s important to understand the challenges that exist and the potential solutions.

Tele-robotics is the art and science of controlling the operations of a robot from a remote location. One of the examples of Tele-robotics that comes to mind is Tele-surgery as shown below where a surgeon remotely controls a robot that performs the surgery on the patient.

The physical operation is performed by the robot, but it is controlled remotely by the surgeon as shown above. There are a few points to be noted here:

1.??????The robot (performing the operation) is fixed or tethered to a location.

2.??????There are many cameras on top of the operation table to provide a visual representation of the patient as he/she is being operated on.

3.??????The surgeon looks at the camera feeds, figures out the situation of the patient, and decides on next steps.

?

In a factory with autonomous robots as shown below, there is no Tele-operation as the robots keep doing the same thing repeatedly unlike in the case of Tele-surgery where the cases vary depending on the situation. However, the robots need to be connected to the network for Tele-maintenance and occasional Tele-operation. Since the robots are fixed in position, they can be connected using wired Ethernet.?

However, there are situations where the robot cannot be tethered to a location. It needs to be mobile and hence untethered. Secondly, there may not be cameras in the environment as in the case of Tele-surgery because the robots may be operating in the wild where deploying fixed cameras is not possible. In the case of mobile robots, cameras, lidars and other sensors are usually mounted on the robot itself. These constraints introduce many technical challenges that need to be solved before Tele-operation for untethered robots becomes a reality.

Applications of Untethered/Mobile Robots

?

Every industry has applications of robots that need to be mobile. Here are a few sample applications:

Warehouses

Autonomous mobile robots (AMRs) are deployed in warehouses for carrying products and parts from one place to another. These AMRs are untethered as shown below.

Pipeline Inspection

Robot dogs are being deployed to inspect gas pipelines and detect potential leakage as shown below. These robots are also untethered as the nature of the job requires them to be mobile.

Disaster Areas

Disaster areas are dangerous and potentially harmful for humans. Robots are best deployed there for inspection and recovery. Most importantly, these robots must be mobile and completely untethered.

Utilities

Whether it is for inspecting windmills or for inspecting the power lines, drones are being deployed. These are also examples of untethered robots.

Technical Challenges

Teleoperation of a robot can be captured by the following diagram:

There are several steps involved in the process:

1.??????Data feed from the robot: the robot may have several sensors and devices mounted on it. For example, there may be several cameras, lidars, accelerometers etc. where the front camera would capture what the robot sees, other cameras might provide a view of the surroundings, the lidar would provide a 3D view of the objects, accelerometer would capture vibration and acceleration of the robot. All these information need to be fed back to the server/operator for analysis, just like in the case of Tele-surgery, except that the devices do not have wired connection. Note that some of the information like the video feed would require high bandwidth while some others like the accelerometer data would require higher reliability. These requirements must be met by the wireless communication network.

2.??????AI algorithms to analyze the information feed: the camera feed, lidar feed and the sensor feed data from the robot need to be analyzed. One way of analyzing is to present the feeds in a structured manner to the human Tele-operator (as shown in the figure above) and let him/her do the analysis. However, there are situations where the information contained in the feeds is too much for humans to analyze in real-time. This becomes even more complex as the speed at which the data is fed back increases. This is where AI algorithms kick in as they can analyze these data feeds in near real-time and generate the relevant notification/alerts for the human operator to take action. In the case of autonomous robots, the actions are automatically generated by the AI algorithms without the intervention of the human operator.

3.??????Control signals and interactions: The action taken by the Tele-operator is translated into control signals that are then sent to the remote robot over the communication network. In case of autonomous robots, control signals may be automatically sent back to the robot over the communication network as shown in the figure above. The control signals can be used for navigating the robot (to avoid obstacles – static or dynamic), or for controlling the instruments (like the surgical tools used in telesurgery or grippers used in tele-maintenance or cameras used in tele-operations, or for controlling the brakes used for emergency braking in the automobiles). These control signals need to be delivered with ultra reliability and low latency over the communication network.

Why is the 5G network indispensable for Tele-operation of untethered/mobile robots?

First, since the robots are untethered (whether in the warehouse or on the road or in the disaster areas or in the air), but must be connected to the cloud/infrastructure to send the data feed for analysis and for receiving the control signals, wireless becomes the de facto connection technology. 5G or beyond is the optimal way to connect, especially when there is no infrastructure for any other wireless technology, such as, WiFi.

Second, to send the high-fidelity (4K/8K) camera feeds, it is necessary to have high bandwidth in the communication network. 5G networks support eMBB (or Enhanced Mobile Broadband) with 10 Gbps peak uplink bandwidth.

Third, the control signals that need to be sent to the robot require high reliability and low latency. Specifically, if the signal to navigate a robot remotely is not delivered quickly (low latency) and reliably, it may run into an accident or injure a human worker in a warehouse or factory. 5G networks support uRLLC (ultra–Reliable Low Latency Communication) with more than 99.999% reliability and extremely low value of 1ms air interface latency and about 5ms of end-to-end latency between the device and the base station. uRLLC in 5G enables delivery of these low latency control signals in a reliable manner.

What are we doing in Accenture Labs in this area?

We, in Accenture Labs, working in close collaboration with WINLAB, Rutgers University, have developed a private 5G network called 5G-In-A-Box that enables the robot dog to connect to the network. The Edge server uses an AI-based vision algorithm to segment, classify, and understand the objects in the camera feed, and take decisions based on the result of the analysis as shown below. The decision is translated into control signals for the robot dog and transmitted over the uRLLC channel of the 5G network.

For more detailed results, please reach out to the author at [email protected]

Conclusion

Tele-robotics over 5G and beyond is the future of robotics as untethering the robot enables several real-world impactful use cases that were not possible before. The support of 5G for high bandwidth (eMBB) communication makes it possible to receive a high-fidelity video feed from the camera mounted on the robot. An AI-based computer vision algorithm processes the video feed, and it either provides the annotated video feed to the human operator to take an action, or it makes a decision on its own and takes an action based on the context of the use case. The action is translated into control messages that are sent out to the robot using the ultra-reliable low-latency (uRLLC) communication channel of 5G. There are several use cases in the manufacturing industry, energy and utilities industry, automobile industry, defense and public sector that are slowly being tried out in pilots before being rolled out widely.

Acknowledgement

I’d like to acknowledge the contributions of Haitao Zhu who has developed the 5G-In-A-Box with help from WINLAB, Sri Sadhan Jujjavarapu for help with the Robot Dog, Lauren McKeown and Trisha Ashish for connecting the Robot Dog over 5G network, streaming the video from the camera and sending the control signal to the robot dog from the Edge Server over 5G. Without the above team’s work, the above research work would not have been possible.

References

[RM22] Research and Markets Teleoperation and Telerobotics Market by Technologies, Solutions, and Applications for Enterprise and Industrial Automation 2022 – 2027

https://www.researchandmarkets.com/reports/5310700/teleoperation-and-telerobotics-market-by?utm_source=CI&utm_medium=PressRelease&utm_code=xl84gq&utm_campaign=1728643+-+Teleoperation+and+Telerobotics+Market+Report+2022%3a+Development+and+Penetration+of+5G+Technology+Offers+High+Potential+Opportunities&utm_exec=como322prd

[D15] DARPA Robotics Challenge Features Disaster-response Tasks

https://www.defense.gov/News/News-Stories/Article/Article/604236/darpa-robotics-challenge-features-disaster-response-tasks/