Indirect unitarity violation entangled with matter effects in reactor antineutrino oscillations

If finite but tiny masses of the three active neutrinos are generated via the canonical seesaw mechanism with three heavy sterile neutrinos, the 3\times 3 Pontecorvo-Maki-Nakagawa-Sakata neutrino mixing matrix V will not be exactly unitary. This kind of indirect unitarity violation can be probed in a precision reactor antineutrino oscillation experiment, but it may be entangled with terrestrial matter effects as both of them are very small. We calculate the probability of \overline{\nu}_e \to \overline{\nu}_e oscillations in a good analytical approximation, and find that, besides the zero-distance effect, the effect of unitarity violation is always smaller than matter effects, and their entanglement does not appear until the next-to-leading-order oscillating terms are taken into account. Given a 20-kiloton JUNO-like liquid scintillator detector, we reaffirm that terrestrial matter effects should not be neglected but indirect unitarity violation makes no difference, and demonstrate that the experimental sensitivities to the neutrino mass ordering and a precision measurement of \theta_{12} and \Delta_{21} \equiv m^2_2 - m^2_1 are robust.Comment: 21 pages, 6 figures, version to be published in PLB, more discussions adde