We investigate graphene and graphene layers on different substrates by monochromatic and white-light confocal Rayleigh scattering microscopy. The image contrast depends sensitively on the dielectric properties of the sample as well as the substrate geometry and can be described quantitatively using the complex refractive index of bulk graphite. For a few layers (<6), the monochromatic contrast increases linearly with thickness. The data can be adequately understood by considering the samples behaving as a superposition of single sheets that act as independent two-dimensional electron gases. Thus, Rayleigh imaging is a general, simple, and quick tool to identify graphene layers, which is readily combined with Raman scattering, that provides structural identification.