In a companion paper, numerical models reveal that buoyant melting instabilities can occur beneath extending lithosphere for a sufficiently small mantle viscosity, extension rate, and rate of melt percolation. However, in some cases, instabilities do not develop during extension but only occur after extension slows or stops. These results are suggestive of a critical behavior in the onset of these kinds of instabilities and motivate a linear analysis to study the onset of instability in a partially melting, passively upwelling plane layer of mantle beneath extending lithosphere. The model we employ includes the effects of buoyancy arising from thermal expansion, the presence of a retained fraction of partial melt, and depletion of the solid by melt extraction. We find a critical behavior in the onset of instability controlled by melt retention buoyancy that is characterized by a “Rayleigh” number M, such that M must exceed some critical value M_(crit) which depends on the efficiency of Stokes rise of a partially molten body relative to the rate of background percolation. Comparison of this theory to the numerical results in the companion paper yields a close quantitative agreement. We also find that solid depletion buoyancy can either stabilize or destabilize a partially melting layer, depending upon both the distribution of preexisting depletion and the magnitude of density changes with depth. This theory is compared with previous studies of buoyant melting instabilities beneath mid‐ocean ridges where similar behavior was reported, and it suggests that the stability of passively upwelling, partially melting mantle underlying both narrow and wide rift settings is controlled by similar processes