We present an experimental and theoretical results of a study of a complex nonlinear polarization dynamics in a passively self-mode-locked erbium-doped fiber oscillator implemented in a ring configuration and operating near lasing threshold. The theoretical model consists of seven coupled non-linear equations and takes into account both orthogonal states of polarizations in the fiber. The experiment confirmed the existence of seven eigenfrequencies, predicted by the model due to polarization instability near lasing threshold. By adjusting the state of polarization of the pump and in-cavity birefringence we changed some eigenfrequencies from being different (non-degenerate state) to matching (degenerate state). The non-degenerate states of oscillator lead to the L-shaped probability distribution function and true rogue wave regime with a positive dominant Lyapunov exponent value between 1.4 and 2.6. Small detuning from partially degenerate case also leads to L-shaped probability distribution function with the tail trespassing eight standard deviations threshold, giving periodic patterns of pulses along with positive dominant Lyapunov exponent of a filtered signal between 0.6 and 3.2. The partial degeneration, in turn, guides to quasi-symmetric distribution and the value of dominant Lyapunov exponent of 42 which is a typical value for systems with a source of the strongly nonhomogeneous external noise
