We study the exchange interaction and the subsequent collective behavior of
magnetic impurities embedded in a disordered two-dimensional (2D) helical
metal. The exchange coupling follows a statistical distribution whose moments
are calculated to the lowest order in (pFββ)β1, where pFβ
is the Fermi momentum of itinerant electrons and β is the mean free path.
We find that i) the first moment of the distribution decays exponentially, and
ii) the variance of the interaction is long-range, however, it becomes
independent of the orientation of the localized magnetic moments due to the
locking between spin and momentum of the electrons that mediate the
interaction. As consequence, long-range magnetic order tends to be suppressed,
and a spin glass phase emerges. The formalism is applied to the surface states
of a three-dimensional (3D) topological insulator. The lack of a net magnetic
moment in the glassy phase and the full randomization of spin polarization at
distances larger than β excludes a spectral gap for surface states. Hence,
non-magnetic disorder may explain the dispersion in results for photoemission
experiments in magnetically-doped topological insulators.Comment: 5 pages, 3 figures; final version to appear in Physical Review B as a
rapid communicatio