A growing body of work studies Blindspot Discovery Methods ("BDM"s): methods
that use an image embedding to find semantically meaningful (i.e., united by a
human-understandable concept) subsets of the data where an image classifier
performs significantly worse. Motivated by observed gaps in prior work, we
introduce a new framework for evaluating BDMs, SpotCheck, that uses synthetic
image datasets to train models with known blindspots and a new BDM, PlaneSpot,
that uses a 2D image representation. We use SpotCheck to run controlled
experiments that identify factors that influence BDM performance (e.g., the
number of blindspots in a model, or features used to define the blindspot) and
show that PlaneSpot is competitive with and in many cases outperforms existing
BDMs. Importantly, we validate these findings by designing additional
experiments that use real image data from MS-COCO, a large image benchmark
dataset. Our findings suggest several promising directions for future work on
BDM design and evaluation. Overall, we hope that the methodology and analyses
presented in this work will help facilitate a more rigorous science of
blindspot discovery