Background Removal Using Shutter Stream and a Background Image

In three earlier articles, Practical Algorithms for Background Removal,  Background Removal Using Shutter Stream and Mobile Devices, and Image Background Removal Using Shutter Stream and Magic Wand, I discussed several approaches our teams at Iconasys and DMMD have taken to remove backgrounds in images, and showed how the Iconasys Shutter Stream software can be used to remove backgrounds. Shutter Stream offers three types of background removal algorithms: (1) magic wand, (2) removal with a background image, and (3) chroma keying (sometimes known as green screen removal). For pure white (not transparent) and when objects are shot against close to white backgrounds, Shutter Stream also offers a colors levels-adjustment option.  The article Image Background Removal Using Shutter Stream and Magic Wand focused on the first option. This article takes a closer look at the second option, removing backgrounds using a background image, and discusses how this tool can be used to maximize image quality by minimizing camera vibrations. 

Shutter Stream

 A compatible camera or mobile device is connected to Shutter Stream, which allows full camera customization options. The different camera settings and their impact on image quality is discussed in details in the article Image Background Removal Using Shutter Stream and Magic Wand. The camera settings and configurations are the same for this type of background removal and therefore I’ll bypass the discussion on the camera settings and focus on the issues that make background removal with a background image different than using the magic wand tool.

Background Removal with a Background Image

The algorithm for removing the background of an image is straight forward conceptually but a little tricky in how the images need to be captured. The algorithm takes as input two images: (1) the input image containing the object and (2) the background image with the object removed. Ideally, the only difference between the two images is the object. Comparing the two images pixel by pixel allows us to determine the pixels that are part of the object and those that are part of the background. The rough block diagram of the algorithm is shown in Figure 1.

From a practical point of view, the algorithm works well when there are no shadows generated by the object in the input image. To keep the algorithm fast, the algorithm is based on pixel color information alone and shadows tend to cause problems. (To help with shadows, texture analysis can be used, in addition to color information, but this will slow the algorithm.)

To minimize shadows during image capture, we recommend the following setup: (1) if shooting with a backdrop, move the backdrop as far away from the object as possible; (2) if available, use acrylic risers with light flooding the riser from underneath, as discussed in the article Image Background Removal Using Shutter Stream and Magic Wand; (3) if possible, suspend the object in the air as shown in the setups of Figure 2.

The final aspect to consider is that the background image should have colors that are distinct, in terms of RGB values, from the colors in the object. If the colors are close in values, then holes in the object may appear for the pixels that have similar colors. For image holes, there are different adjustments that can be used, such as the ability to automatically fill them in or to add in the missing pixels using the lasso tool.

Measuring Camera Vibrations and Inconsistencies

Next, we tested the consistency of images across multiple shots. We set up the camera and a wire frame rack as shown in Figure 3. Camera was shooting the wire frame at an angle, which means that the captured image contained increasing frequencies from bottom to the top. (We expected errors in the high frequency areas.)

As part of the test, we shot the same image twice and then applied the background removal algorithm by varying only the threshold value. If the camera is consistent from shot to shot the background removal algorithm should have output a completely transparent image (in our case we set transparency to white, for better visualization) at small threshold values.

Figure 4 shows the output at threshold values of 2, 5 and 13. As it can be seen from the figure, the two images were fairly inconsistent. Shooting the same image twice generated images that were quite different, requiring thresholds of up to 13 or higher.

We experimented further with different camera settings (ISO, denoise settings, etc.), and in the end determined that mirror slap and switching the live view on/off was causing additional vibration in the camera sensor, amplifying the noise in the system.

After setting the live view to always on and running the same experiment, the consistency across images improved, and the threshold value for which the two images were almost identical dropped from 13 to 7, as shown in Figure 6. This particular observation about enabling live view is critical when shooting microphotography, such as diamonds and jewelry. In microphotography the smallest vibration errors can cause blur.

Applying the Background Removal

The process begins by capturing the image and its background, as shown in Figure 7. 

The background image is set as the background image for the algorithm, and the algorithm is applied to the main image. Besides the threshold values, there are a few other parameters that control the level of denoising, hole filling, speck removal, mask growth, and edge blending, as shown in Figure 8.

A final cleanup to cut the supporting wires is done with a manual lasso tool, as shown in Figure 9.

The final result is shown in Figure 10.

In conclusion, this article described how to use Shutter Stream’s background removal tool when a background image is available.  It also discussed how this tool can be used to measure camera consistency from shot to shot, and how we experimentally determined that for our particular camera, Canon 70D, enabling live view minimizes noise and vibration errors by 50 percent.

Keywords: #ProductPhotography, #ProductPhotographySoftware, #eCommerceImaging, #360Imaging, #360ProductPhotography, #BackgroundRemoval, #BackgroundClipping, #TransparentBackgrounds