OpenCV with ROS2

Integrating OpenCV with ROS2: A Comprehensive Guide to Computer Vision in?Robotics

Introduction

In the world of robotics, vision is a crucial sense that enables machines to perceive and interact with their environment. This post will guide you through the process of integrating OpenCV, a powerful computer vision library, with ROS2. We’ll explore the architecture, set up the environment, and dive into some exciting projects.

ROS2 Architecture and Computer Vision Pipeline

ROS2, the next-generation robotics middleware, provides a flexible framework for developing robotic applications. Its publish-subscribe architecture allows for modular design and easy integration of various components, including computer vision systems.

In a typical ROS2 computer vision pipeline:

1. A camera node publishes raw image data to a topic.
2. A vision processing node subscribes to this topic, processes the images using OpenCV, and publishes the results.
3. Other nodes (e.g., control systems) can subscribe to the processed data for decision-making.

Required Libraries

To get started with OpenCV in ROS2, you’ll need the following libraries:

sudo apt-get update
sudo apt install ros-<distro>-cv-bridge
sudo apt-get install python3-opencv
pip install opencv-python        

Replace <distro> with your ROS2 distribution (e.g., foxy, galactic, humble).

Creating a ROS2 Package

To create a new ROS2 package for your computer vision project:

ros2 pkg create --build-type ament_python my_cv_package
cd my_cv_package        

Basic Computer Vision Projects

Let’s explore some fundamental computer vision projects you can implement in ROS2:

a) Image Subscriber

This basic node subscribes to a camera topic and displays the received images:

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import cv2

class ImageSubscriber(Node):
    def __init__(self):
        super().__init__('image_subscriber')
        self.subscription = self.create_subscription(
            Image,
            '/camera/image_raw',
            self.listener_callback,
            10)
        self.br = CvBridge()
    
    def listener_callback(self, data):
        self.get_logger().info('Receiving video frame')
        current_frame = self.br.imgmsg_to_cv2(data)
        cv2.imshow("camera", current_frame)
        cv2.waitKey(1)

def main(args=None):
    rclpy.init(args=args)
    image_subscriber = ImageSubscriber()
    rclpy.spin(image_subscriber)
    image_subscriber.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()        

b) Color Detection

This project detects a specific color in the image and make a centroid:

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from geometry_msgs.msg import Twist
from cv_bridge import CvBridge
import cv2
import numpy as np

class LineFollower(Node):
    def __init__(self):
        super().__init__('line_follower')
        self.subscription = self.create_subscription(Image, '/camera/image_raw', self.listener_callback, 10)
        self.subscription
        self.br = CvBridge()
        self.publisher_ = self.create_publisher(Twist, '/cmd_vel', 10)

        # Set the desired linear and angular velocity gains
        self.linear_velocity_gain = 0.2
        self.angular_velocity_gain = 0.3

    def listener_callback(self, data):
        self.get_logger().info('Receiving video frame')
        img = self.br.imgmsg_to_cv2(data, 'bgr8')
        img = cv2.resize(img, None, 1, 0.25, 0.25, cv2.INTER_CUBIC)

        imgHSV = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

        lower_yellow = np.array([0, 35, 0])
        upper_yellow = np.array([57, 255, 255])

        mask = cv2.inRange(imgHSV, lower_yellow, upper_yellow)

        roi_height = mask.shape[0] // 4
        roi = mask[mask.shape[0] - roi_height:, :]

        M = cv2.moments(roi)
        if M['m00'] > 0:
            cx = int(M['m10'] / M['m00'])
            cy = int(M['m01'] / M['m00']) + mask.shape[0] - roi_height
            centroid = (cx, cy)

            radius = 8
            img_center = (mask.shape[1] // 2, mask.shape[0] // 2)
            distance_from_center = np.sqrt((cx - img_center[0])**2 + (cy - img_center[1])**2)
            max_distance = np.sqrt(img_center[0]**2 + img_center[1]**2)
            color_intensity = int(255 * (1 - distance_from_center / max_distance))
            color = (color_intensity, color_intensity, 0)
            cv2.circle(img, centroid, radius, color, -1)

            # Calculate linear and angular velocities based on centroid position
            error = img_center[0] - cx
            linear_velocity = self.linear_velocity_gain
            angular_velocity = self.angular_velocity_gain * error

            # Keep the centroid in the center of the ROI
            if abs(error) > 20:
                angular_velocity = np.clip(angular_velocity, -0.3, 0.3)

            # Publish the velocity commands
            twist = Twist()
            twist.linear.x = linear_velocity
            twist.angular.z = angular_velocity
            self.publisher_.publish(twist)

        cv2.imshow("Line Threshold", mask)
        cv2.imshow("Region of Interest", roi)
        cv2.imshow("Centroid Indicator", img)
        cv2.waitKey(1)

def main(args=None):
    rclpy.init(args=args)
    line_follower = LineFollower()
    rclpy.spin(line_follower)
    line_follower.destroy_node()
    rclpy.shutdown()

if __name__ == '__main__':
    main()        

Advanced Projects

As you become more comfortable with OpenCV and ROS2, you can tackle more advanced projects:

• Object Detection: Use pre-trained models like YOLO or SSD for real-time object detection.
• Face Recognition: Implement face detection and recognition algorithms.
• Visual SLAM: Implement Simultaneous Localization and Mapping using visual data.

Conclusion

Integrating OpenCV with ROS2 opens up a world of possibilities in robotics and computer vision. From basic image processing to advanced perception tasks, the combination of these powerful tools enables the development of sophisticated robotic systems capable of understanding and interacting with their environment.

Remember, the key to mastering these technologies is practice and experimentation. Start with the basic projects, then gradually move to more complex applications.

References