Neutron scattering investigation of the interplay between lattice and spin degrees of freedom in geometrically frustrated magnets

Geometrical frustration in magnetic systems brought on by the incompatibility of structural and magnetic interaction symmetries leads to the suppression of a long-range order via introduction of macroscopic degeneracy of the system ground-state. As a result magnetic moments in frustrated systems remain disordered but highly correlated and may fluctuate down to very low temperatures. It results in variety of exotic physical phenomena ranging from structural distortions relieving the frustration to appearance of fractional quasiparticle excitations. This thesis presents results of studies on three examples of classical and quantum frustrated magnetic systems. The first is the family of chromate spinel breathing pyrochlore antiferromagnets \BP{}. These accommodate an alternating distortion to the pyrochlore lattice of Cr$^{3+}$ ions. This distortion with change of its magnitude drives the system between the singlet state of separated tetrahedron and the uniform pyrochlore lattice ground-state. Neutron and x-ray diffraction studies on $x=0$ composition identified two magnetostructural transitions. The first at $T=13.8$~K and the second at $12.5$~K. They result in the mixture of two phases, one tetragonal described with $C_{A}222_{1}$ magnetic space group and the second following complex multi-$\mathbf{k}$ order whose exact nature could not be resolved with the available data. A small departure from the $x=0$ stoichiometry to $x=0.05$ have not suppressed the single anomaly present in specific heat. However, no long-range magnetic order or lattice distortion were detected in diffraction data. Reverse Monte Carlo treatment of the diffuse feature observed in the neutron scattering allowed to identify this transition as the onset of classical spin nematic phase concomitant with spin freezing. Nonetheless, spectroscopic studies have shown the presence of persistent fluctuations of magnetic moments down to the lowest temperatures. The second example is \TGG{} (TGG). It hosts a hyperkagome magnetic sublattice of Tb$^{3+}$ ions. TGG orders at a very low $T_{\text{N}}=0.24$~K in an induced-moment type order. Neutron powder diffraction allowed to determine the value of the ordered magnetic moment $\mu=3.6$~$\mu_{\text{B}}$. The analysis of diffuse scattering have identified presence of correlated paramagnet phase above $T_{\text{N}}$. However, no dipolar order parameter for this phase was found in the refined spin structures. The inelastic neutron scattering on powder sample enabled the refinement of the set of CEF Hamiltonian parameters, using which the structure of CEF eigenstates was obtained. Following single-crystal experiment showed the presence of six dispersive magnetic excitons in place of the first excited CEF state. These are present in both the paramagnetic and ordered regimes, and reflect the collective character of usually single-ion crystal field effects acquired by magnetic interactions. A softening of one of these modes was observed at the magnetic propagation vector on cooling towards $T_{\text{N}}$. Nonetheless, no closing of the gap was detected. Measurements of phonon dispersion curves allowed to confirm the available results of density functional theorem (DFT) calculations. The third investigated system is RbNiCl$_{3}$ a $S=1$ spin-$1$ antiferromagnetic Heisenberg chain. Polarization analysis of inelastic neutron scattering data in the quantum-disordered phase did not allow for unambiguous identification of the multi-particle states similar to these observed in quantum-disordered phase of closely related CsNiCl$_{3}$. However, signatures of continuum scattering at the antiferromagnetic point of the intrachain dispersion were found in the three-dimensional ordered phase. The strength of the continuum does not agree with predictions of field-theory for spin-$1$ chains. It supports assumptions of frustration between the chains being possible source of observed phenomena. Linear spin-wave theory (LSWT) fit to the magnetic excitaitons at $T<T_{\text{N}}$ have revealed a set of discrepancies the calculated and measured spectra. These suggest strong influence of quantum fluctuations on the physics of the ordered state and exclude the feasibility of LSWT in this system. Some of the discrepancies were positively identified as phonon modes using the available results of DFT calculations