Model-free image segmentation approaches for automatic building detection, usually fail to detect accurately building boundaries due to shadows, occlusions and other low level misleading information. In this paper, a novel recognition-driven variational framework is introduced for automatic and accurate multiple building extraction from aerial and satellite images. We aim to solve the problem of inaccurate data-driven segmentation. To this end, multiple shape priors are considered. Segmentation is then addressed through the use of a data-driven approach constrained from the prior models. The proposed framework extend previous approaches towards the integration of shape priors into the level set segmentation. In particular, it allows multiple competing priors and estimates buildings pose and number from the observed single image. Therefore, it can address multiple building extraction from single panchromatic images a highly demanding task of fundamental importance in various geoscience and remote sensing applications. Very promising results demonstrate the potentials of our approach.