We study Gaussian approximations to the distribution of a diffusion. The
approximations are easy to compute: they are defined by two simple ordinary
differential equations for the mean and the covariance. Time correlations can
also be computed via solution of a linear stochastic differential equation. We
show, using the Kullback-Leibler divergence, that the approximations are
accurate in the small noise regime. An analogous discrete time setting is also
studied. The results provide both theoretical support for the use of Gaussian
processes in the approximation of diffusions, and methodological guidance in
the construction of Gaussian approximations in applications
