Digital signal processing techniques for fiber nonlinearity compensation in coherent optical communication systems

The capacity of long-haul coherent optical communication systems is limited by the detrimental effects of fiber Kerr nonlinearity. The power-dependent nature of the Kerr nonlinearity restricts the maximum launch power into the fiber. That results in the reduction of the optical signal-to-noise ratio at the receiver; thereby, the maximum transmission reach is limited. Over the last few decades, several digital signal processing (DSP) techniques have been proposed to mitigate the effects of fiber nonlinearity, for example, digital back-propagation (DBP), perturbation based nonlinearity compensation (PB-NLC), and phase-conjugated twin wave (PCTW). However, low-complexity and spectrally efficient DSP-based fiber nonlinearity mitigation schemes for long-haul transmission systems are yet to be developed. In this thesis, we focus on the computationally efficient DSP-based techniques that can help to combat various sources of fiber nonlinearity in long-haul coherent optical communication systems. With this aim, we propose a linear time/polarization coded digital phase conjugation (DPC) technique for the mitigation of fiber nonlinearity that doubles the spectral efficiency obtained in the PCTW technique. In addition, we propose to investigate the impact of random polarization effects, like polarization-dependent loss and polarization mode dispersion, on the performance of the linear-coded DPC techniques. We also propose a joint technique that combines single-channel DBP with the PCTW technique. We show that the proposed scheme is computationally efficient and achieves similar performance as multi-channel DBP in wavelength division multiplexed superchannel systems. The regular perturbation (RP) series used to analytically approximate the solution of the nonlinear Schrödinger equation (NLSE) has a serious energy divergence problem when truncated to the first-order. Recent results on the transmission of high data-rate optical signals reveal that the nonlinearity compensation performance of the first-order PB-NLC technique decreases as the product of the transmission distance and launch power increases. The enhanced RP (ERP) method can improve the accuracy of the first-order RP approximation by partially solving the energy divergence problem. On this ground, we propose an ERP-based nonlinearity compensation technique to compensate for the fiber nonlinearity in a polarization-division multiplexed dispersion unmanaged optical communication system. Another possible solution to improve the accuracy of the PB-NLC technique is to increase the order of the RP solution. Based on this idea, we propose to extend the first-order solution of the NLSE to the second-order to improve the nonlinearity compensation performance of the PB-NLC technique. Following that, we investigate a few simplifying assumptions to reduce the implementation complexity of the proposed second-order PB-NLC technique