While quantum mechanics imposes a fundamental limit on the precision of
interferometric measurements of mechanical motion due to measurement
backaction, the nonlinear nature of the coupling also leads to parametric
instabilities that place practical limits on the sensitivity by limiting the
power in the interferometer. Such instabilities have been extensively studied
in the context of gravitational wave detectors, and their presence has recently
been reported in Advanced LIGO. Here, we observe experimentally and describe
theoretically a new type of optomechanical instability that arises in two-tone
backaction-evading (BAE) measurements, designed to overcome the standard
quantum limit, and demonstrate the effect in the optical domain with a photonic
crystal nanobeam, and in the microwave domain with a micromechanical oscillator
coupled to a microwave resonator. In contrast to the well-known oscillatory
parametric instability that occurs in single-tone, blue-detuned pumping, which
is characterized by a vanishing effective mechanical damping, the parametric
instability in balanced two-tone optomechanics is exponential, and is a result
of small detuning errors in the two pump frequencies. Its origin can be
understood in a rotating frame as the vanishing of the effective mechanical
frequency due to an optical spring effect. Counterintuitively, the instability
occurs even in the presence of perfectly balanced intracavity fields, and can
occur for both signs of detuning. We find excellent quantitative agreement with
our theoretical predictions. Since the constraints on tuning accuracy become
stricter with increasing probe power, it imposes a fundamental limitation on
BAE measurements, as well as other two-tone schemes. In addition to introducing
a new limitation in two-tone BAE measurements, the results also introduce a new
type of nonlinear dynamics in cavity optomechanics