This paper presents a novel method for the fault diagnosis of planetary gearboxes based on an accurate estimation of residual sidebands using a modulation signal bispectrum (MSB). The residual sideband resulting from the out-phase superposition of vibration waves from asymmetrical multiple meshing sources are much less influenced by gear errors than that of the in-phase sidebands. Therefore, with the accurate estimation by MSB they can produce accurate and consistent diagnosis, which are evaluated by both simulating and experimental studies. However, the commonly used in-phase sidebands have high amplitudes but include gear error effects, consequently leading to poor diagnostic results