Advancements in mathematical programming have made it possible to efficiently
tackle large-scale real-world problems that were deemed intractable just a few
decades ago. However, provably optimal solutions may not be accepted due to the
perception of optimization software as a black box. Although well understood by
scientists, this lacks easy accessibility for practitioners. Hence, we advocate
for introducing the explainability of a solution as another evaluation
criterion, next to its objective value, which enables us to find trade-off
solutions between these two criteria. Explainability is attained by comparing
against (not necessarily optimal) solutions that were implemented in similar
situations in the past. Thus, solutions are preferred that exhibit similar
features. Although we prove that already in simple cases the explainable model
is NP-hard, we characterize relevant polynomially solvable cases such as the
explainable shortest-path problem. Our numerical experiments on both artificial
as well as real-world road networks show the resulting Pareto front. It turns
out that the cost of enforcing explainability can be very small