We report the detailed high-field and high-frequency electron spin resonance
(HF-ESR) spectroscopic study of the single-crystalline van der Waals compounds
Mn2P2S6 and MnNiP2S6. Analysis of magnetic excitations
shows that in comparison to Mn2P2S6 increasing the Ni content
yields a larger magnon gap in the ordered state and a larger g-factor value and
its anisotropy in the paramagnetic state. The studied compounds are found to be
strongly anisotropic having each the unique ground state and type of magnetic
order. Stronger deviation of the g-factor from the free electron value in the
samples containing Ni suggests that the anisotropy of the exchange is an
important contributor to the stabilization of a certain type of magnetic order
with particular anisotropy. At temperatures above the magnetic order, we have
analyzed the spin-spin correlations resulting in a development of slowly
fluctuating short-range order. They are much stronger pronounced in
MnNiP2S6 compared to Mn2P2S6. The enhanced spin
fluctuations in MnNiP2S6 are attributed to the competition of
different types of magnetic order. Finally, the analysis of the temperature
dependent critical behavior of the magnon gaps below the ordering temperature
in Mn2P2S6 suggests that the character of the spin wave
excitations in this compound undergoes a field induced crossover from a 3D-like
towards 2D XY regime