Magnetic susceptibility and muon spin relaxation (muSR) experiments have been
carried out on the quasi-2D triangular-lattice spin S = 2 antiferromagnet
FeGa2S4. The muSR data indicate a sharp onset of a frozen or nearly-frozen spin
state at T* = 31(2) K, twice the spin-glass-like freezing temperature T_f =
16(1) K. The susceptibility becomes field dependent below T*, but no sharp
anomaly is observed in any bulk property. A similar transition is observed in
muSR data from the spin-1 isomorph NiGa2S4. In both compounds the dynamic muon
spin relaxation rate lambda_d(T) above T* agrees well with a calculation of
spin-lattice relaxation by Chubukov, Sachdev, and Senthil in the renormalized
classical regime of a 2D frustrated quantum antiferromagnet. There is no firm
evidence for other mechanisms. At low temperatures lambda_d(T) becomes
temperature independent in both compounds, indicating persistence of spin
dynamics. Scaling of lambda_d(T) between the two compounds is observed from
~T_f to ~1.5T*. Although the muSR data by themselves cannot exclude a truly
static spin component below T*, together with the susceptibility data they are
consistent with a slowly-fluctuating "spin gel" regime between T_f and T*. Such
a regime and the absence of a divergence in lambda_d(T) at T* are features of
two unconventional mechanisms: (1) binding/unbinding of Z_2 vortex excitations,
and (2) impurity spins in a nonmagnetic spin-nematic ground state. The absence
of a sharp anomaly or history dependence at T* in the susceptibility of
FeGa2S4, and the weakness of such phenomena in NiGa2S4, strongly suggest
transitions to low-temperature phases with unconventional dynamics.Comment: 13 pages, 6 figures, accepted for publication in Physical Review