We identify rare and visually distinctive galaxy populations by searching for
structure within the learned representations of pretrained models. We show that
these representations arrange galaxies by appearance in patterns beyond those
needed to predict the pretraining labels. We design a clustering approach to
isolate specific local patterns, revealing groups of galaxies with rare and
scientifically-interesting morphologies.Comment: Accepted at Machine Learning and the Physical Sciences Workshop,
NeurIPS 202