Rotoscoping introduction and resources

Rotoscoping (or just roto in casual usage) is the drawing or painting on frames of a movie, using visual elements in the movie as a reference. A common kind of rotoscoping is tracing a path around an object in a movie and using that path as a mask to separate the object from its background. This allows you to work with the object and the background separately, so you can do things like apply different effects to the object than to its background or replace the background.

Note: After Effects includes the Roto Brush tool, which can be used to accomplish many of the same tasks as conventional rotoscoping, but in far less time. For information about using the Roto Brush tool, see Roto Brush and Refine Matte.
If a background or foreground object is a consistent, distinct color, you can use color keying instead of rotoscoping to remove the background or object. If the footage was shot with color keying in mind, color keying is much easier than rotoscoping. (See Keying introduction and resources.)

Rotoscoping in After Effects is mostly a matter of drawing masks, animating the mask path, and then using these masks to define a matte. Many additional tasks and techniques make this job easier, such as using motion tracking on the object before you begin drawing masks, and then using the motion tracking data to make a mask or matte automatically follow the object.

Rotoscoping tips

  • Immediately after beginning to draw a mask, press Alt+Shift+M (Windows) or Option+Shift+M (Mac OS) to turn on keyframing for that mask and set a keyframe. This way, you won’t edit a mask frame-by-frame for several minutes (or longer) and then realize that you lost all of your work on previous frames because you forgot to click the stopwatch to make the mask path animated.

  • Draw your masks on a white solid layer with its Video (eyeball) switch off, above the (locked) footage layer. This way, you run no risk of accidentally moving the footage layer when you manipulate the mask, and you can also much more easily apply tracking data to the mask. (You apply the tracking data to the invisible solid layer that holds the mask.) This also means that you don't lose your cached RAM preview frames each time you manipulate the mask. (See Toggle visibility or influence of a layer or property group and Lock or unlock a layer.)

  • Turn on the Preserve Constant Vertex Count preference. (See Designate the first vertex for a Bezier path.)

  • When possible, transform (rotate, scale, move) the whole mask or a subset of the mask vertices instead of moving the vertices individually. This is both for efficiency and to avoid the chatter that comes from inconsistent movement across frames. (See Move vertices in free-transform mode.)

  • Manual motion tracking is less time-consuming than manual rotoscoping. The more effort you spend getting good tracking data for various parts of your scene and object, the less time you'll spend drawing and fine-tuning masks. (See Tracking and stabilizing motion (CS5).)

Online resources about rotoscoping

This video from the After Effects CS5: Learn By Video series shows how to combine motion tracking and rotoscoping to isolate and selectively color-correct an actor's face.

Sean Kennedy provides several good tutorials on the SimplyCG website, including some for rotoscoping in After Effects. Sean maintains an index of these tutorials on his website.

Scott Squires provides a pair of movies on his Effects Corner website that show how to rotoscope, both painting and masking:

Chris and Trish Meyer provide some tips on animating masks, including using Smart Mask Interpolation, on the ProVideo Coalition website.

Alejandro Pérez provides a script on the AE Enhancers forum with which you can use tracking data to position individual mask vertices.

Mathias Möhl provides the KeyTweak script on his website, with which you can modify many keyframes on a property simultaneously. With KeyTweak, you can modify a few keyframes manually, and the script modifies the remaining keyframes in between accordingly. KeyTweak is especially useful for Mask Path keyframes in a rotoscoping workflow.

Rich Young provides several resources for rotoscoping on his After Effects Portal website.