Top-quark spin correlations as a tool to distinguish pseudoscalar $A \to ZH$ and scalar $H \to ZA$ signatures in $Z t \bar t$ final states at the LHC

Abstract

Both ATLAS and CMS have recently performed the first searches for a heavy new spin-0 resonance decaying into a lighter new spin-0 resonance and a $Z$ boson, where the lighter spin-0 resonance subsequently decays into $t \bar t$ pairs. These searches are of particular interest to probe Two Higgs doublet model (2HDM) parameter space regions that predict a strong first-order electroweak phase transition. In the absence of CP violation, the investigated decay is possible if the lighter and the heavier spin-0 particles have opposite CP parities. The analysis techniques employed by ATLAS and CMS do not distinguish between the two possible signatures $A \to ZH$ and $H \to ZA$, where $A$ and $H$ denote CP-odd and CP-even Higgs bosons, respectively, if both signals are predicted to have the same total cross sections. We demonstrate the capability of angular variables that are sensitive to spin correlations of the top quarks to differentiate between $A \to ZH$ and $H \to ZA$ decays, even in scenarios where both signals possess identical total cross sections. Focusing on masses of 600 GeV and 800 GeV as a representative 2HDM benchmark, we find that a distinction between the two possible channels is possible with high significance with the anticipated data from the high-luminosity LHC, if the invariant mass distribution of the $t \bar t$ system is further binned in angular variables defined by the direction of flight of the leptons produced in the top-quark decays. Moreover, we find a moderate gain in experimental sensitivity due to the improved background rejection for both signals