In this paper, the problem of improving the stability of the power system with DFIG and in the presence of SSSC using nonlinear method is discussed. The nonlinear controller is designed by the multi-input backstepping method with a sliding mode observer. This controller is applied simultaneously to the excitation system of synchronous generators and the rotor side converter in DFIG and SSSC in a way that improves the stability of the power system compared to the linear and nonlinear methods described in this paper. The control coefficient matrices are adjusted using intelligent algorithms in such a way that the stability of the system is more optimized. Practical constraints on the system are considered in selecting the control inputs. The designed controller is robust to changes in the operating point and the location of the disturbance. The performance of the designed controller in a 39-bus, 10 machines NEW ENGLAND network including DFIG generators and in the presence of SSSC is simulated and investigated using MATLAB software
