Motion tracking and rotoscoping are two essential skills for visual effects artists who want to create realistic and seamless composites. But how do you know if you are doing them well and how can you improve your performance?
Explore some ways to measure and enhance your motion tracking and rotoscoping skills in VFX.
Motion tracking is the process of tracking the movement of an object or a camera in a video clip, and using that data to match the position, rotation, scale, and perspective of another element, such as a 3D model, a text, or a matte painting. Rotoscoping is the process of manually drawing masks or shapes around an object or a person in a video clip, and using them to isolate, remove, or replace that element with another one. Both techniques are often used together to create realistic and convincing composites.
The primary point in the section above is how do we measure and improve skills in motion tracking and rotoscoping. For motion tracking, I measure artist skill level by the quality of the artist's tracks. I quality check a track by first stabilizing it on a easily visible frame. A high quality track would show the plate fixed to that point stabilized for every frame. I kick back any tracks that pop or drift more than 50 pixels or so from the stabilization point. For roto, I measure artist skill level based on roto task difficulty. Experienced roto leaders have a measure of easy, medium and hard tasks. So if an artist can deliver only easy tasks, then they are level 1. As they deliver more difficult tasks, their level rises.
When I was a student at NYU I pitched my musical comedy short 'The Porns' to my class. I got to the part where it said "INT. 747 - CABIN - DAY" everyone burst into laughter. They thought it was impossible! We got ahold of airplane seats from a stripped Boeing 747 and filmed in my friends garage, over the course of several days, with nothing but a camera and a green screen. We used white "+" symbol tracking points to track the movement of the camera to recreate the scene in 3D on the computer. Anything the green screen couldn't cover, we used rotoscoping to 'delete' that background. So when we 'moved' with the camera, the 3D airplane interior moved relative to the camera, making it look like we were really there!
Motion tracking and rotoscoping are important because they allow you to integrate different elements from different sources into a single video clip, and make them look like they belong to the same scene. For example, you can use motion tracking to add a 3D spaceship to a live-action footage of a city, and use rotoscoping to remove the wires that hold the actors in a green screen studio. Motion tracking and rotoscoping can also help you correct or enhance the original footage, such as removing unwanted objects, adding effects, or changing the background.
This is pretty obvious, but after all, this is an AI generated article. So what can you expect. Likewise, AI will soon replace the need for traditional motion tracking and rotoscoping, just look at Nvidia's NeRF. Turn 2D photos and videos into high-res scenes. That same technology can and is being used for the next step up from traditional motion tracking, without the need for traditional tracking points, and simultaneously performing texture capture and an extra wealth of data not traditionally available to a standard (and often prohibitively expensive) motion capture setup. AI & ML models will only continue to become more advanced and refined. Regardless, both motion tracking and rotoscoping are still incredibly important to "film magic."
One way to measure your motion tracking and rotoscoping skills is to compare your results with the original footage and the reference material. You can use tools such as onion skinning, split screen, or difference mode to see how well your elements match the motion, shape, and color of the original footage. You can also use tools such as trackers, grids, or rulers to check the accuracy and consistency of your motion tracking and rotoscoping data. Another way to measure your skills is to get feedback from your peers, clients, or supervisors, and learn from their suggestions and critiques.
For motion tracking, I measure artist skill level by the quality of the artist's tracks. I quality check a track by first stabilizing it on a easily visible frame. A high quality track would show the plate fixed to that point stabilized for every frame. For roto, I measure artist skill level based on roto task difficulty. Experienced roto leaders have a measure of easy, medium and hard tasks. So if an artist can deliver only easy tasks, then they are level 1. As they deliver more difficult tasks, their level rises. Further, I measure an artist skill level based on the quality of their work. Roto quality is measured by two primary criteria: 1) Did they deliver what was in the task brief? 2) Did they follow our technical specifications?
One way to improve your motion tracking and rotoscoping skills is to practice on different types of footage and challenges. Find online tutorials or courses that teach you the basics and advanced techniques of motion tracking and rotoscoping, and follow along with the examples. Use online communities that offer feedback or resources for VFX artists. Another way to improve your skills is to use the right tools and settings for your motion tracking and rotoscoping tasks. Experiment with different software, plugins, or features that suit your workflow and preferences, and optimize your performance and quality.
Learn what the lead or supervisor's expectations are per task and per discipline. Then mentally focus on building that into each version of roto you deliver. Ask and memorize what their definitions of roto difficulty are. Then see which one can you deliver consistently. If you can only deliver easy tasks, then realize your limitation and practice on those harder tasks. If any artist wants to improve, there should be agreed upon definitions of expectation/level. Then artists should practice the specific skills they lack to level up. 5 key skills to improve in no particular order: 1- Task Expectations Clarity (understand all the task details) 2- Plate Analysis 3- Target Analysis 4- Target Motion Tracking and Roto Keyframe reduction 5- QC
When doing motion tracking and rotoscoping, it's important to avoid some common mistakes and pitfalls. Using too many or too few trackers or points for motion tracking can result in jittery, drifting, or inaccurate data. It's also important to check and correct the motion tracking data for errors, such as occlusions, outliers, or noise. Additionally, you should adjust the motion tracking data for perspective, lens distortion, or motion blur. When rotoscoping, you need to be careful not to use too many or too few keyframes or points, which can result in jagged, wobbly, or unnatural shapes. It's also essential to refine and smooth the edges or curves of the rotoscope shapes in order to achieve hard, soft, or consistent mattes. Lastly, you should animate and feather the rotoscope shapes according to the motion, depth, or lighting of the original footage.
If this is your first time doing extensive 3D VFX work from scratch like I did as an NYU student on 'The Porns', there is one specific technical note that you should know before shooting, especially if you do not have a background in VFX. - Make sure 100% you write down the focal length (/ how 'zoomed in' you are on the lens) and exactly the make, model of lens and camera you are using, and likewise the height of the tripod from the floor if you're using one. This will save an incredible amount of time in post as all major VFX / 3D software has 1) a lens correction tool / ability to match the way the lens distorts the world, and 2) ensure your tracking is properly applied.
To track and roto like a pro, you need to have a keen eye for detail, an accurate sense of timing and motion, and a solid understanding of VFX compositing principles and techniques. Additionally, you must have patience, persistence, and creativity to overcome any footage or tools challenges. Here are some tips to help you achieve success:
I believe it's really important to also look ahead at how current advancements, such as the Grounded-Segment-Anything (https://github.com/IDEA-Research/Grounded-Segment-Anything) will continue to disrupt how rotoscoping will be done inside of professional pipelines in the very near future. A lot is changing, so it's important to keep up to date on how these changes are affecting the best practices for tracking and rotoscoping!
Saying what's on everyone's mind... AI is drastically changing this traditionally labor-intensive process. Before "AI" / ML, rotoscoping was a meticulous task, done manually and optically. Now, better and more efficient algorithms have revolutionized it, speeding up workflows and enhancing precision. For instance, Autodesk's Flame leverages AI to transform hours-long rotoscoping into a 20-minute task. This efficiency allows artists to focus more on creativity. While AI significantly reduces the tedium of rotoscoping, it doesn't replace the artistic judgment essential in VFX —?as it lacks that second metacognitive layer of we attribute to consciousness, but rather complements it, marking a pivotal evolution in the field.