To account for the phenomenon that the winding inter-turn short circuit fault of the turbine generator rotor leads to intensified vibration, a coupled nonlinear vibration model of the oil film force and electromagnetic force is established, and a dynamic stability analysis method is proposed. An improved shooting method for solving the periodic solutions of the bearing-rotor system with the unbalanced electromagnetic force is developed; and the system damping criterion used in examining the system stability margin is presented. The calculation on the nonlinear vibration for the 300 MW turbine generator is finished, and the vibration response characteristics and instability margin in a typical winding inter-turn short circuit fault are obtained. The results show that the winding inter-turn short circuit fault may result in the synchronized vortex phenomenon, and a comparison between the numerical results and the site test results is completed. The results will provide technical support and theoretical basis for directly predicting the winding inter-turn short-circuit fault via the vibration detection
