A mask in After Effects is a path that
is used as a parameter to modify layer attributes, effects, and
properties. The most common use of a mask is the modification of
an alpha channel of a layer, which determines the transparency of
the layer at each pixel. Another common use of a mask is as a path
along which to animate text. (See Creating and animating text on a path.)
For more information on paths in general, see About paths.

Default behavior for a drawn mask (left); same mask inverted
(right)
Closed-path masks can create transparent areas for a layer. Open
paths cannot create transparent areas for a layer but are useful
as parameters for an effect. Effects that can use an open or closed
mask path as input include Stroke, Path Text, Audio Waveform, Audio
Spectrum, and Vegas. Effects that can use closed masks (but not
open masks) as input include Fill, Smear, Reshape, Particle Playground,
and Inner/Outer Key.
A mask belongs to a specific layer. Each layer can contain multiple
masks.
You can draw masks in common geometric shapes—including polygons, ellipses,
and stars—with the shape tools, or you can use the Pen tool to draw
an arbitrary path.
In most ways, drawing mask paths is the same as drawing shape
paths on shape layers, though the editing and interpolation of mask
paths have a few additional features. You can link a mask path to
a shape path using expressions, which allows you to bring the benefits
of masks into shape layers, and vice versa. See Creating shapes and masks and Managing and animating shape paths and masks.
The position of a mask in the stacking order in the Timeline
panel affects how it interacts with other masks. You can drag a
mask to different positions within the Masks property group in the
Timeline panel.
The Mask Opacity property for a mask determines the influence
that a closed mask has on the alpha channel of the layer inside
the mask area. A Mask Opacity value of 100% corresponds to an interior
area that is completely opaque. The area outside the mask is always
completely transparent. To invert what is considered inside and
what is considered outside for a specific mask, select Invert next
to the mask name in the Timeline panel.