Up-conversion (UC) of near-infrared radiation to visible light has received much attention because of its use in the conversion of solar radiation, luminescence thermometry, biosensing, and anti-counterfeiting applications. However, the main issue hindering the successful utilization of UC is the relatively low quantum efficiency of the process. In order to design new UC systems with high quantum yield (ϕUC) values, we synthesized two series of co-doped BaF2 single crystals with nominal concentrations of Yb3+ (2–15 mol%)/Er3+ (2 mol%) as well as Yb3+ (3 mol%)/Er3+ (2–15 mol%). The highest ϕUC value of 10.0% was demonstrated for the BaF2:Er3+ (2 mol%) and Yb3+ (3 mol%) sample under 490 W cm−2 of 976 nm excitation. To study the natural limit of UC efficiency, quantum yield values upon direct excitation (ϕDS) of the 4S3/2 (ϕDS ≤ 26%) levels were measured. Comparison of experimental values of quantum yields to the ones obtained using Judd–Ofelt theory reveals strong quenching of the 4S3/2 state for all investigated compositions. In addition, we observed an unusually strong contribution of the Er3+:4I9/2 excited state to both UC and down-shifting luminescent processes. This contribution becomes possible due to the very low maximum phonon energy of BaF2 crystals (240 cm−1)