Competing topological and Kondo insulator phases on a honeycomb lattice

We investigate the competition between the spin-orbit interaction of itinerant electrons and their Kondo coupling with local moments densely distributed on the honeycomb lattice. We find that the model at half-filling displays a quantum phase transition between topological and Kondo insulators at a nonzero Kondo coupling. In the Kondo-screened case, tuning the electron concentration can lead to a new topological insulator phase. The results suggest that the heavy-fermion phase diagram contains a new regime with a competition among topological, Kondo-coherent and magnetic states, and that the regime may be especially relevant to Kondo lattice systems with $5d$-conduction electrons. Finally, we discuss the implications of our results in the context of the recent experiments on SmB$_6$ implicating the surface states of a topological insulator, as well as the existing experiments on the phase transitions in SmB$_6$ under pressure and in CeNiSn under chemical pressure.Comment: (v3) Published version including the main text (5 pages + 4 figures) and a supplementary material discussing the effects of quantum fluctuations of the slave bosons and antiferromagnetic ordering of the local moments on the transitions among the Kondo, magnetic and topological state

Calculation of a Class of Three-Loop Vacuum Diagrams with Two Different Mass Values

We calculate analytically a class of three-loop vacuum diagrams with two different mass values, one of which is one-third as large as the other, using the method of Chetyrkin, Misiak, and M\"{u}nz in the dimensional regularization scheme. All pole terms in \epsilon=4-D (D being the space-time dimensions in a dimensional regularization scheme) plus finite terms containing the logarithm of mass are kept in our calculation of each diagram. It is shown that three-loop effective potential calculated using three-loop integrals obtained in this paper agrees, in the large-N limit, with the overlap part of leading-order (in the large-N limit) calculation of Coleman, Jackiw, and Politzer [Phys. Rev. D {\bf 10}, 2491 (1974)].Comment: RevTex, 15 pages, 4 postscript figures, minor corrections in K(c), Appendix B removed, typos corrected, acknowledgements change