We investigate the impact of spin anisotropic interactions, promoted by
spin-orbit coupling, on the magnetic phase diagram of the iron-based
superconductors. Three distinct magnetic phases with Bragg peaks at (π,0)
and (0,π) are possible in these systems: one C2 (i.e. orthorhombic)
symmetric stripe magnetic phase and two C4 (i.e. tetragonal) symmetric
magnetic phases. While the spin anisotropic interactions allow the magnetic
moments to point in any direction in the C2 phase, they restrict the
possible moment orientations in the C4 phases. As a result, an interesting
scenario arises in which the spin anisotropic interactions favor a C2 phase,
but the other spin isotropic interactions favor a C4 phase. We study this
frustration via both mean-field and renormalization-group approaches. We find
that, to lift this frustration, a rich magnetic landscape emerges well below
the magnetic transition temperature, with novel C2, C4, and mixed
C2-C4 phases. Near the putative magnetic quantum critical point, spin
anisotropies promote a stable Gaussian fixed point in the renormalization-group
flow, which is absent in the spin isotropic case, and is associated with a
near-degeneracy between C2 and C4 phases. We argue that this frustration
is the reason why most C4 phases in the iron pnictides only appear inside
the C2 phase, and discuss additional manifestations of this frustration in
the phase diagrams of these materials.Comment: 21 pages, 19 figures, published versio