Keying is defining transparency by a particular color value or luminance value in an image. When you key out a value, all pixels that have colors or luminance values similar to that value become transparent.

Keying makes it easy to replace a background, which is especially useful when you work with objects too complex to mask easily. When you place a keyed layer over another layer, the result forms a composite, in which the background is visible wherever the keyed layer is transparent.

You often see composites made with keying techniques in movies, for example, when an actor appears to dangle from a helicopter or float in outer space. To create this effect, the actor is filmed in an appropriate position against a solid-color background screen. The background color is then keyed out and the scene with the actor is composited over a new background.

The technique of keying out a background of a consistent color is often called bluescreening or greenscreening, although you don’t have to use a blue or green screen; you can use any solid color for a background. Red screens are often used for shooting non-human objects, such as miniature models of cars and space ships. Magenta screens have been used for keying work in some feature films renowned for their visual effects. Other common terms for this kind of keying are color keying and chroma keying.

Difference keying works differently from color keying. Difference keying defines transparency with respect to a particular baseline background image. Instead of keying out a single-color screen, you can key out an arbitrary background. To use difference keying, you must have at least one frame that contains only the background; other frames are compared to this frame, and the background pixels are made transparent, leaving the foreground objects. Noise, grain, and other subtle variations can make difference keying very difficult to use in practice.

After Effects includes several built-in keying effects, as well as the Academy Award-winning Keylight effect, which excels at professional-quality color keying. (See Keying effects and Matte effects.)

For information on the Keylight effect, see its documentation in the folder in which the Keylight plug-in is installed, or on the Foundry website.

Keylight is not included with the trial version of After Effects CS5. (See Third-party plug-ins included with After Effects.) Keylight is included in the trial version of After Effects CS5.5 and later.

For a video tutorial on keying with Keylight, go to the Adobe website.

Mark Christiansen provides tips and techniques for using Keylight in an excerpt from his book After Effects Studio Techniques: Visual Effects and Compositing on the Peachpit Press website. In an excerpt from the “Color Keying in After Effects” chapter of After Effects Studio Techniques, Mark Christiansen provides detailed tips and techniques for color keying, including advice on which keying effects to avoid and how to overcome common keying challenges.

For a step-by-step tutorial demonstrating the use of the Color Difference Key effect, the Matte Choker effect, the Spill Suppressor effect, and garbage masks, see the “Keying in After Effects” chapter of the After Effects Classroom in a Book on the Peachpit Press website.

Jeff Foster provides free sample chapters from his book The Green Screen Handbook: Real World Production Techniques. The sample chapters cover basic compositing, color keying, garbage mattes, hold-out mattes, and how to avoid common problems with greenscreen shots. For more information, see the Adobe website.

Rich Young collects more tips and resources for keying on his After Effects Portal website.

Tips on color keying and compositing from experienced compositor, Chris Zwar.

Chris & Trish Meyer shares tips & resources for color keying with Keylight and other effects.

Keying footage that was not acquired correctly is much more difficult than is keying footage that was acquired with keying in mind. For best results, use Adobe OnLocation™ to monitor color and lighting as you acquire footage for color keying.

For tips on shooting footage so that color keying is easier and more successful, see Jonas Hummelstrand’s General Specialist website.

Alex Lindsay provides an article on the ProVideo Coalition website about setting up and lighting a green background in preparation for color keying work. In part two of his greenscreen primer on the ProVideo Coalition website, Alex Lindsay provides detailed information about cameras, codecs, and capture systems to use when acquiring footage for color keying.

• Light your color screen uniformly, and keep it free of wrinkles.

• Start with the highest-quality materials you can gather, such as film that you scan and digitize.

• Use uncompressed footage (or, at least, files with the least possible amount of compression). Many compression algorithms, especially the algorithms used in DV, HDV, and Motion JPEG, discard subtle variations in blue—which may be necessary to create a good key from a bluescreen. Use footage with the least color subsampling possible—for example, 4:2:2 rather than 4:2:0 or 4:1:1. (For information about color subsampling, see the Wikipedia website and the Adobe website.)

Robbie Carman and Richard Harrington provide an excerpt on the Peachpit website from their book Video Made On A Mac that demonstrates how to plan, shoot, key, and composite a greenscreen shot.

• Noise and compression artifacts can cause problems for keying, especially difference keying. Often, applying a slight blur before keying can reduce noise and compression artifacts enough to improve keying results. For example, blurring the blue channel for DV footage can smooth out noise in a bluescreen.

  The KeyerforDV animation preset available through the AE Enhancers forum automates the process of blurring the blue and green channels before keying DV footage.

• Use a garbage matte to roughly outline your subject so that you don’t have to waste time keying out parts of the background far from the foreground subject. (See Use a garbage matte.)

• Use a hold-out matte to roughly protect areas that are of a similar color to the background from being keyed out. (See Use a hold-out matte.)

• To help you view transparency, temporarily change the background color of the composition, or include a background layer behind the layer you are keying out. As you apply the keying effect to the layer in the foreground, the composition background (or a background layer) shows through, making it easy to view transparent areas. (See Composition settings.)

• For evenly lit footage, adjust keying controls on only one frame. Choose the most intricate frame of the scene, one involving fine detail such as hair and transparent or semitransparent objects, such as smoke or glass. If the lighting is constant, the same settings you apply to the first frame are applied to all subsequent frames. If lighting changes, you may need to adjust keying controls for other frames. Place keyframes for the first set of keying properties at the start of the scene. If you are setting keyframes for one property only, use Linear interpolation. For footage that requires keyframes for multiple interacting properties, use Hold interpolation. If you set keyframes for keying properties, you may want to check the results frame by frame. Intermediate keying values may appear, producing unexpected results.

• To key well-lit footage shot against a color screen, start with the Color Difference Key. Add the Spill Suppressor to remove traces of the key color, and then use one or more of the other Matte effects, if necessary. If you are not satisfied with the results, try starting again with the Linear Color Key.

• To key well-lit footage shot against multiple colors or unevenly lit footage shot against a bluescreen or greenscreen, start with the Color Range key. Add the Spill Suppressor and other effects to refine the matte. If you are not completely satisfied with the results, try starting with or adding the Linear Color Key.

• To key dark areas or shadows, use the Extract Key on the Luminance channel.

• To make a static background scene transparent, use the Difference Matte Key. Add the Simple Choker and other effects as needed to refine the matte.

• After you have used a key to create transparency, use Matte effects to remove traces of key color and create clean edges.

• Blurring the alpha channel after keying can soften the edges of the matte, which can improve compositing results.