Using a Monte Carlo method, we study the finite-temperature phase transition
in the two-dimensional classical Heisenberg model on a triangular lattice with
or without easy-plane anisotropy. The model takes account of competing
interactions: a ferromagnetic nearest-neighbor interaction J1 and an
antiferromagnetic third nearest-neighbor interaction J3. As a result, the
ground state is a spiral spin configuration for −4<J1/J3<0. In this
structure, global spin rotation cannot compensate for the effect of 120-degree
lattice rotation, in contrast to the conventional 120-degree structure of the
nearest-neighbor interaction model. We find that this model exhibits a
first-order phase transition with breaking of the lattice rotation symmetry at
a finite temperature. The transition is characterized as a Z2 vortex
dissociation in the isotropic case, whereas it can be viewed as a Z vortex
dissociation in the anisotropic case. Remarkably, the latter is continuously
connected to the former as the magnitude of anisotropy decreases, in contrast
to the recent work by Misawa and Motome [J. Phys. Soc. Jpn. \textbf{79} (2010)
073001.] in which both the transitions were found to be continuous.Comment: 11pages, 16figures, accepted to JPS