A nonlinear rotor-seal-bearing system model, including the Muszynska nonlinear seal force model and nonlinear oil-film force model based on short bearing assumption, is proposed in this paper. The non-dimensional dynamic motion equations of the system are also established and solved by fourth order Runge-kutta method. The nonlinear dynamic characteristics of the system are analyzed with the help of bifurcation diagrams, spectrum waterfall diagrams, axis orbit diagrams, Poincaré maps and amplitude spectrums. The effect of rotor speed, seal clearance, seal length and seal radius on the nonlinear characteristics of the system is also explored. The genetic algorithm (GA) is applied to optimize the stability of the system. The numerical results demonstrate that the rotor-seal-bearing system contains many motion forms, such as periodic, multi-periodic and quasi-periodic motions. Lower rotor speed, proper seal clearance and seal radius, larger seal length are of benefit to the stability of rotor-seal-bearing system. The minimum instability rotor speed changing from 1970 rad/s to 2110 rad/s indicates the GA is an effective optimization method of improving the stability of rotor-seal-bearing system
