The Lyot project used an optimized Lyot coronagraph with Extreme Adaptive
Optics at the 3.63m Advanced Electro-Optical System telescope (AEOS) to observe
86 stars from 2004 to 2007. In this paper we give an overview of the survey
results and a statistical analysis of the observed nondetections around 58 of
our targets to place constraints on the population of substellar companions to
nearby stars. The observations did not detect any companion in the substellar
regime. Since null results can be as important as detections, we analyzed each
observation to determine the characteristics of the companions that can be
ruled out. For this purpose we use a Monte Carlo approach to produce artificial
companions, and determine their detectability by comparison with the
sensitivity curve for each star. All the non-detection results are combined
using a Bayesian approach and we provide upper limits on the population of
giant exoplanets and brown dwarfs for this sample of stars. Our nondetections
confirm the rarity of brown dwarfs around solar-like stars and we constrain the
frequency of massive substellar companions (M>40Mjup) at orbital separation
between and 10 and 50 AU to be <20%.Comment: 32 pages, 11 figures, 2 tables. Published in the Astrophysical
Journa