We study the optical bistability of an optomechanical system in which the
position of a mechanical oscillator modulates the cavity frequency. The
steady-state mean-field equation of the optical mode is identical to the one
for a Kerr medium, and thus we expect it to have the same characteristic
behavior with a lower, a middle, and an upper branch. However, the presence of
position fluctuations of the mechanical resonator leads to a new feature: the
upper branch will become unstable at sufficiently strong driving in certain
parameter regimes. We identify the appropriate parameter regime for the upper
branch to be stable, and we confirm, by numerical investigation of the quantum
steady state, that the mechanical mode indeed acts as a Kerr nonlinearity for
the optical mode in the low-temperature limit. This equivalence of the
optomechanical system and the Kerr medium will be important for future
applications of cavity optomechanics in quantum nonlinear optics and quantum
information science