"I Spy With My Industrial Eye" - 5 Things To Consider When Adding Vision To Your Industrial Edge Solution

So, you are thinking about adding computer vision to your industrial edge solution…

• Have your been hearing that your competitors added artificial intelligence (AI) and cameras on their production line and this helped save them money by catching defects earlier?
• Or, maybe you want to improve productivity and reduce costs by making your factory robots more autonomous?

Yes, vision at the industrial edge can do all of that and more!

As available computing power at the industrial edge continues to increase and with the introduction of artificial intelligence (AI), we are seeing increasingly sophisticated uses of machine vision. See my earlier blog, "Demystifying AI Acceleration at the Edge", to learn more about acceleration technologies for AI at the edge.?

It can be overwhelming trying to figure out where to get started and what to pay attention to.

Here are 5 things that you should consider as you start on your journey:

1. If One Camera Is Good, Are More Even Better?
2. It's What You Can See And How You See It...
3. Making Sense Of Camera Interfaces
4. Control Your Lighting To Control Your Destiny!
5. GenICam? - A Universal Language For Industrial Cameras

If One Camera Is Good, Are More Even Better?

Is there such a thing as too much of a good thing? Yes!

When it comes to the number of cameras in your industrial edge vision solution, you want to follow the "Goldilocks Principle" and find just the right number of cameras to accomplish the task.

Too many cameras for the task can mean unnecessary costs for the cameras, more computing power to handle the extra processing for each additional camera input and increased system complexity.

The "just right" number of cameras for you will require some careful analysis, including the types of camera you are using, fields of view and what you need to capture.

However, some questions you should ask yourself…

Q: Is my task stationary or moving? Stationary tasks normally involve observing objects as they move past the camera. Moving tasks normally involve guiding an autonomous robot through a controlled or uncontrolled space.

Moving machines require more cameras to maintain situational awareness and safety. You need enough cameras to cover your directions of movement.

Q: Can I capture the images I need from a single point of view? Or, do I need to see an object from more than one side as it moves by?

If your object is moving through the factory and you need to see it from different points of view, you may not be able to stop the production line to manipulate the object and get the views you need. Plan for enough cameras to get a complete view as it passes by without having to interfere with your existing operations.

Q: Do I need to synchronize my cameras?

Some tasks that require multiple cameras also require that you be able to very accurately determine which images from the different cameras were taken at the same time, especially if you have a fast moving assembly line and need to associate multiple images with a single object. If you need this kind of synchronization, you may be restricted to certain camera interfaces (see "Making Sense Of Camera Interfaces" below) that have time synchronization capabilities.

It's What You See And How You See It...

When most people think about vision, they are probably thinking about color cameras, similar to the webcam on your laptop or the camera in your smartphone. However, if your goal is not to take a cool selfie, but rather to detect, decipher or decode something in your factory, then color may not necessarily be the best choice for you.

Spectrum (Color, Grayscale, Infrared, Ultraviolet, X-Ray, ...)

When we talk about color cameras, we are mostly talking about a 2D RGB camera which returns a rectangular array of color values representing the real-world image. It "sees" color by measuring and reporting the individual levels of red (R), green (G) and blue (B) at each point.

Grayscale cameras, unlike color cameras, do not measure the levels of R, G and B at a particular point. Instead, for each point, they report the total intensity of light (luminance).

If color cameras provide so much more information, why would anyone want to use a grayscale camera?

Just like when we are deciding on the number of cameras to use, when we are choosing the richness of image data to capture, we need to find the "just right" point.

Color images contain much more data compared to grayscale images, requiring more bandwidth to transmit and more compute complexity to process. If that extra data is not needed to complete your task, the resources to capture, transmit and process that extra data are wasted, costing you more money to deploy and operate your solution.

Some vision algorithms may actually perform better or more reliably if you convert a color image into grayscale first. For example, shape detection and character recognition may be simpler in grayscale, if the shape or character is still recognizable without color differentiation.

In additional to visible light vision, you can also choose cameras which can sense outside the range of human eyes, such as: infrared, ultraviolet, X-Ray, ... These additional ranges can overcome operational challenges, such as low-light environments and also allow you to gain insights that you cannot with visible light. I put together a table of some common vision spectrums used in industrial edge solutions along with potential uses for them.

3D Depth

Another consideration for industrial vision is whether to capture depth information. As humans, we take for granted that we do not just see what is in our field of view, but we also understand how far are objects in front of us. We accomplish this by having 2 eyes, spaced slightly apart from each other, and give us slightly different views of the same scene (disparity). Our brains then use this disparity to figure out how far away something is.

Standard 2D cameras, regardless of the spectrum they operate in, lack the ability to capture this information.

As the amount of available computing power has increased, various techniques for determining depth information have become more accessible, including: stereo vision, structured light, time-of-flight and LiDAR.

Depth information is particularly useful for moving applications, such as: autonomous mobile robots (AMRs), robotic pick-and-place arms, ...

I found a nice overview of depth sensing technologies here. You can also read about how cameras based on Intel? RealSense? technology work. Intel RealSense technology is used in many applications which require depth information.

Resolution, Frame Rate, Shutter Type

Once you determine "what you want to see" (spectrum and depth), you need to decide "how you want to see it" (resolution, frame rate and shutter type).

The resolution of a camera refers to the size of the digital image that the camera produces. You will usually see this expressed in terms of "megapixels" which means how many million pixels can be captured and recorded in a single image. When deciding on how many megapixels you want your cameras to be able to capture, it is again tempting to think that "more is better". However, more resolution usually means a more expensive camera and more data to transmit and process.

To determine the resolution you need your cameras to be able to capture, you need to know the following:

• How much can your camera sensor see - Field of View (FOV)? If you increase your FOV, then every detail gets smaller by the time it reaches the sensor. This means you need higher resolution to avoid losing detail.

• What is the size of the smallest feature of interest - Minimum Feature Size (MFS)? The smaller the MFS, the higher resolution you need to get enough detail to distinguish that feature of interest.

• How many pixels are needed to resolve the MFS - Pixel Resolution (PS)? If the nature of the feature of interest is that you need more pixels before you can determine that it is the feature of interest, then you will need higher resolution to get more of those pixels in the same amount of sensor space.

Finally, frame rate and shutter type are greatly influenced by the speed of what you are trying to capture.

If the assembly line you are monitoring is moving at a rate of 600 objects per minute, this translates to 10 objects going by every second. At the lower end, this means you need to capture 10 images per second or 10 frames per second (FPS). However, it is generally recommended that you specify your frame rate at twice what you need, so this would mean 20 FPS.

There are 2 main shutter types in industrial machine vision: rolling shutter and global shutter.

With a rolling shutter, the controller will have the sensor capture and read out the values one row at a time. Rolling shutter sensors tend to allow higher frame rates, higher light sensitivity and lower noise. However, the biggest disadvantage of rolling shutters is that, since the frame is being captured one row at a time, if the object or the system moves significantly, the overall image may be distorted. However, if your scene is relatively slow moving or static, then rolling shutter sensors are a good option.

With a global shutter, when the signal is raised to capture an image, all of the sensor pixels sample the incoming light at the same time and hold that value, waiting to be read out. Then the controller reads out the entire frame before allowing another image to be captured. The advantage is that the motion distortion that affects rolling shutter sensors does not happen with global shutter sensors. However, global shutter sensors are more expensive to manufacture, have lower frame rates, have lower light sensitivity and have more noise. If your scene is faster moving and the motion distortions that exist with rolling shutter sensors adversely impact your results, choose a global shutter sensor.

Making Sense Of Camera Interfaces

Capturing the image on the camera is only part of the job. If you want to actually do anything useful, you need to connect the camera to the computer so that the computer can control the camera and process the data.

Your choice of which camera interface to use will be largely influenced by:

• How much bandwidth does your application need?
• How far away will the camera need to be?
• Are you able or willing to add an additional frame grabber card?
• Do you have any special requirements, such as time synchronization?
• What are your cost constraints?

Below is a table of describing the bandwidth, cable length constraints, advantages and disadvantages of popular industrial camera interfaces.

Control Your Lighting To Control Your Destiny!

Imagine if you had to consistently visually examine objects day in and day out but you could never be sure that the lighting stayed the same. Perhaps you have natural sunlight coming into the room that changes from morning to evening. Or there are multiple light sources inside the factory that reflect off of the object you are looking at. These changes could cause you to misjudge color, miss seeing a particular defect because it is covered up by a shadow, ...

These same issues impact industrial machine vision the same way they impact human operators.

If you want to "control your destiny" and have reliable and consistent industrial machine vision results, you need to provide consistent lighting for the objects you are examining that are not impacted by ambient lighting conditions and changes. Adding controlled lighting and providing enclosures around what you are examining are some of the ways this could be accomplished.

AMRs have some additional challenges since they need to navigate through an environment which they cannot control. Reflections and glare as they travel through their environment may impact their ability to detect obstacles. To combat this, you can consider adding polarizers and other filters depending on the most common lighting issues your AMR will encounter.

GenICam? - A Universal Language For Industrial Cameras

As any software engineer knows, having to rewrite their code to adapt to a new application programming interface (API) takes development time and can introduce bugs that will take time to debug. Trying to support multiple different APIs at the same time can be daunting and discourage a company from adopting better products and technology, effectively locking them to a particular vendor.

Thankfully, when it comes to industrial machine vision, there exists a standard to solve just that. The goal of Generic Interface for Cameras (GenICam) is to provide a generic programming interface for all types of camera devices, no matter what interface technology they use (GigE Vision, USB3 Vision, CoaXPress, Camera Link, ...)

If they want to be able to advertise that their products support GenICam, the camera vendor needs to provide a software implementation that complies to this standard. This lets the industrial machine vision software engineer concentrate on adding their unique value, instead of figuring out how to talk to whatever camera the team has chosen.

With major industrial camera vendors contributing and supporting this standard, there is really no reason not to choose a camera that has GenICam support.

Get Started Today Adding Vision To Your Industrial Edge Solution!

With the amazing advancements in industrial machine vision technology, it can seem daunting to know how to get started or what to pay attention to.

In this article I tried to highlight some of the important (and sometimes forgotten) considerations as you begin that journey.

What challenges or learning have you encountered in your pursuit of adding vision to your solution?

Share your thoughts and experiences.

Credits/Acknowledgements:

• Artificial intelligence elements collection image by Freepik - https://www.freepik.com/free-vector/artificial-intelligence-elements-collection-flat-style_2687173.htm
• Female detective image by brgfx on Freepik - https://www.freepik.com/free-vector/female-detective-wearing-brown-overcoat-hat_25539542.htm
• Robot arm picking up box image by user6702303 on Freepik - https://www.freepik.com/free-photo/robot-arm-picks-up-box-autonomous_18321474.htm
• Smart industry intelligent manufacturing isometric composition image by macrovector on Freepik - https://www.freepik.com/free-vector/smart-industry-intelligent-manufacturing-isometric-composition-with-text-automatic-assembly-line-operated-remotely-from-tablet_7378867.htm
• Cartesian icons created by Smashicons - Flaticon - https://www.flaticon.com/free-icon/coordinates_137072
• 4:3 screen resolutions image by Peter Forret on Flickr - https://www.flickr.com/photos/pforret/254101411
• Camera field of view image by macrodiego - https://openclipart.org/detail/323966/camera-field-of-view
• Line spacing icons created by Pixel perfect - Flaticon - https://www.flaticon.com/free-icon/line-spacing_889608