Magnetism and topological phases in an interacting decorated honeycomb lattice with spin-orbit coupling

We study the interplay between spin-orbit coupling (SOC) and Coulomb repulsion in a Hubbard model on a decorated honeycomb lattice which leads to a plethora of phases. While a quantum spin hall insulator is stable at weak Coulomb repulsion and moderate SOC, a semimetallic phase emerges at large SOC in a broad range of Coulomb repulsion. This semimetallic phase has topological properties not observed in conventional metals such as a finite, non-quantized spin Hall conductivity. At large Coulomb repulsion and negligible spin-orbit coupling, electronic correlations stabilize a resonance valence bond (RVB) spin liquid state in contrast to the classical antiferromagnetic state predicted by mean-field theory. Under sufficiently strong SOC, such RVB state is transformed into a magnetic insulator consisting on S~3/2 localized moments on a honeycomb lattice with antiferromagnetic order and topological features.Comment: 13 pages, 10 figure