A family of quasilocal mass definitions that includes as special cases the
Hawking mass and the Brown-York ``rest mass'' energy is derived for spacelike
2-surfaces in spacetime. The definitions involve an integral of powers of the
norm of the spacetime mean curvature vector of the 2-surface, whose properties
are connected with apparent horizons. In particular, for any spacelike
2-surface, the direction of mean curvature is orthogonal (dual in the normal
space) to a unique normal direction in which the 2-surface has vanishing
expansion in spacetime. The quasilocal mass definitions are obtained by an
analysis of boundary terms arising in the gravitational ADM Hamiltonian on
hypersurfaces with a spacelike 2-surface boundary, using a geometric time-flow
chosen proportional to the dualized mean curvature vector field at the boundary
surface. A similar analysis is made choosing a geometric rotational flow given
in terms of the twist covector of the dual pair of mean curvature vector
fields, which leads to a family of quasilocal angular momentum definitions
involving the squared norm of the twist. The large sphere limit of these
definitions is shown to yield the ADM mass and angular momentum in
asymptotically flat spacetimes, while at apparent horizons a quasilocal version
of the Gibbons-Penrose inequality is derived. Finally, some results concerning
positivity are proved for the quasilocal masses, motivated by consideration of
spacelike mean curvature flow of 2-surfaces in spacetime.Comment: Revised version, includes an analysis of null flows with applications
to mass and angular momentum for apparent horizon