We develop in full detail the formalism of tangent states to the manifold of
matrix product states, and show how they naturally appear in studying
time-evolution, excitations and spectral functions. We focus on the case of
systems with translation invariance in the thermodynamic limit, where momentum
is a well defined quantum number. We present some new illustrative results and
discuss analogous constructions for other variational classes. We also discuss
generalizations and extensions beyond the tangent space, and give a general
outlook towards post matrix product methods.Comment: 40 pages, 8 figure