Organic solar cells demonstrate external quantum efficiencies and fill factors approaching those of conventional photovoltaic technologies. However, as compared with the optical gap of the absorber materials, their open-circuit voltage is much lower, largely due to the presence of significant non-radiative recombination. Here, we study a large data set of published and new material combinations and find that non-radiative voltage losses decrease with increasing charge-transfer-state energies. This observation is explained by considering non-radiative charge-transfer-state decay as electron transfer in the Marcus inverted regime, being facilitated by a common skeletal molecular vibrational mode. Our results suggest an intrinsic link between non-radiative voltage losses and electron-vibration coupling, indicating that these losses are unavoidable. Accordingly, the theoretical upper limit for the power conversion efficiency of single-junction organic solar cells would be reduced to about 25.5% and the optimal optical gap increases to 1.45–1.65 eV, that is, 0.2–0.3 eV higher than for technologies with minimized non-radiative voltage losses