Observation of cooling neutron stars can potentially provide information about the states of matter at supernuclear densities. We review physical properties important for cooling such as neutrino emission processes and superfluidity in the stellar interior, surface envelopes of light elements due to accretion of matter and strong surface magnetic fields. The neutrino processes include the modified Urca process, and the direct Urca process for nucleons and exotic states of matter such as a pion condensate, kaon condensate, or quark matter. The dependence of theoretical cooling curves on physical input and observations of thermal radiation from isolated neutron stars are described. The comparison of observation and theory leads to a unified interpretation in terms of three characteristic types of neutron stars: high-mass stars which cool primarily by some version of the direct Urca process; low-mass stars, which cool via slower processes; and medium-mass stars, which have an intermediate behavior. The related problem of thermal states of transiently accreting neutron stars with deep crustal burning of accreted matter is discussed in connection with observations of soft X-ray transients.