How do you use the roto brush tool and the refine edge tool to complement your vector paths in rotoscoping?

1 Roto brush tool

The roto brush tool is a smart tool that automatically detects the edges of your subject and creates a mask around it. You can access it by selecting the layer you want to rotoscope and clicking on the roto brush icon in the toolbar. To use it, you simply paint over the area you want to keep, and the tool will expand the selection to the nearest edge. You can adjust the size, hardness, and feather of the brush, and use the Alt key to subtract from the selection. The roto brush tool works best on footage with clear contrast and minimal motion blur.

2 Refine edge tool

The refine edge tool is a sub-tool of the roto brush tool that helps you refine the mask around your subject, especially in areas with hair, fur, or other fine details. You can access it by holding the Ctrl key and clicking on the roto brush icon. To use it, you paint over the edges of your mask, and the tool will analyze the alpha channel and the color information to create a more accurate and smooth mask. You can adjust the feather, contrast, shift edge, and reduce chatter of the refine edge tool to improve the quality of your mask.

3 Vector paths

Vector paths are another way of creating masks or shapes in After Effects. Unlike the roto brush tool, vector paths are not automatic and require you to manually draw and adjust them using the pen tool or the shape tools. Vector paths are composed of anchor points and handles that define the direction and curvature of the path. You can access them by selecting the layer you want to rotoscope and clicking on the pen tool or the shape tool in the toolbar. To use them, you click and drag on the canvas to create and modify the anchor points and handles, and close the path to create a mask or shape.

4 Adjusting anchor points and handles

To adjust the anchor points and handles of your vector paths, you need to use the selection tool or the direct selection tool in the toolbar. The selection tool allows you to move, rotate, or scale the entire path or mask, while the direct selection tool allows you to select and modify individual anchor points and handles. You can also use the convert vertex tool to change the type of anchor point from corner to smooth or vice versa. To adjust the anchor points and handles, you click and drag on them to change their position and angle, or use the keyboard shortcuts to nudge them.

5 Complementing vector paths with roto brush and refine edge

While vector paths offer more precision and control over your masks or shapes, they can also be tedious and time-consuming to create and animate, especially for complex or organic subjects. That's why you can use the roto brush tool and the refine edge tool to complement your vector paths in rotoscoping. You can use the roto brush tool and the refine edge tool to quickly create a rough mask around your subject, and then use the vector paths to refine and tweak the mask as needed. You can also use the vector paths to create additional masks or shapes that are not covered by the roto brush tool and the refine edge tool.

6 Animating masks and shapes

Once you have created your masks or shapes using the roto brush tool, the refine edge tool, and the vector paths, you can animate them to follow the movement of your subject or create dynamic effects. You can animate them by setting keyframes for the mask path or shape path property of your layer, and adjusting the anchor points and handles accordingly. You can also use the motion tracker or the motion sketch to track or record the movement of your subject and apply it to your masks or shapes. You can also use the mask options or shape options to change the opacity, feather, expansion, stroke, fill, or blend mode of your masks or shapes.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?