We report on the realization of a highly efficient optical spot-size converter for the end-face coupling of single photons from GaAs-based nanophotonic waveguides with embedded quantum dots. The converter is realized using an inverted taper and an epoxy polymer overlay providing a 1.3~μm output mode field diameter. We demonstrate the collection of single photons from a quantum dot into a lensed fiber with a rate of 5.84±0.01~MHz and estimate a chip-to-fiber coupling efficiency of ∼48~\%. The stability and compatibility with cryogenic temperatures make the epoxy waveguides a promising material to realize efficient and scalable interconnects between heterogeneous quantum photonic integrated circuits