Domain generalization (DG) seeks predictors which perform well on unseen test
distributions by leveraging data drawn from multiple related training
distributions or domains. To achieve this, DG is commonly formulated as an
average- or worst-case problem over the set of possible domains. However,
predictors that perform well on average lack robustness while predictors that
perform well in the worst case tend to be overly-conservative. To address this,
we propose a new probabilistic framework for DG where the goal is to learn
predictors that perform well with high probability. Our key idea is that
distribution shifts seen during training should inform us of probable shifts at
test time, which we realize by explicitly relating training and test domains as
draws from the same underlying meta-distribution. To achieve probable DG, we
propose a new optimization problem called Quantile Risk Minimization (QRM). By
minimizing the α-quantile of predictor's risk distribution over domains,
QRM seeks predictors that perform well with probability α. To solve QRM
in practice, we propose the Empirical QRM (EQRM) algorithm and provide: (i) a
generalization bound for EQRM; and (ii) the conditions under which EQRM
recovers the causal predictor as α→1. In our experiments, we
introduce a more holistic quantile-focused evaluation protocol for DG and
demonstrate that EQRM outperforms state-of-the-art baselines on datasets from
WILDS and DomainBed.Comment: NeurIPS 2022 camera-ready (+ minor corrections