Using spin-polarized scanning tunneling microscopy and density functional
theory we demonstrate the occurrence of a novel type of noncollinear spin
structure in Rh/Fe atomic bilayers on Ir(111). We find that higher-order
exchange interactions depend sensitively on the stacking sequence. For
fcc-Rh/Fe/Ir(111) frustrated exchange interactions are dominant and lead to the
formation of a spin spiral ground state with a period of about 1.5 nm. For
hcp-Rh/Fe/Ir(111) higher-order exchange interactions favor a double-row wise
antiferromagnetic or "uudd" state. However, the Dzyaloshinskii- Moriya
interaction at the Fe/Ir interface leads to a small angle of about 4{\deg}
between adjacent magnetic moments resulting in a canted "uudd" ground state
