When dealing with complex mechanical systems, the frictional contact is at the origin of significant changes in the dynamic behavior of systems. The presence of frictional contact can give rise to mode-coupling instabilities that produce harmonic "friction induced vibrations". Unstable vibrations can reach large amplitude that could compromise the structural integrity of the system and are often associated with annoying noise emission. The study of this kind of dynamic instability has been object of many studies ranging from both theoretical and numerical study of simple lumped models to numerical and experimental study on real mechanical systems, such as automotive brakes, typically affected by such issue. In this paper the numerical analysis of a lumped system constituted by several degrees of freedom in frictional contact with a slider is presented, where the introduction of friction gives rise to an unstable dynamic behavior. Two different approaches are used to investigate the effects of friction forces. The linear Complex Eigenvalue Analysis (CEA) allows for calculating of the complex eigenvalues of the system that can be characterized by a positive real part (i.e. negative apparent modal damping). The effects of the main parameters on the system stability are investigated. In the second approach a non linear model has been developed that takes into account the stick slip behavior at the interface to solve the time-history solution and analyze the unstable vibration. The mode selection mechanism occurring in transient nonlinear analysis, when several unstable modes are predicted by the linear CEA, and driving the selection of the frequency of the unstable vibrations, is investigated. Furthermore, by means of the transient analysis, the influence of the type of perturbation at the equilibrium position on the time history of the system vibrations is analyzed. Results comparison between the two different approaches highlights how nonlinearities affect the time-history solution and how stable and unstable behavior can be predicted by the linear CEA. The obtained results have been extended to the finite element model of a simple mechanical system
