Design of neural network-based nonlinear equalisers for coherent optical communication systems

Abstract

Recent advancements in beyond 5G (B5G) networks and future communications demand ultra-high capacity and, therefore, impose stringent requirements on fibre-optic transmission infrastructures. Optical fibre channel impairments such as nonlinearities induced by the Kerr effect and their interactions with chromatic dispersion pose a significant challenge in achieving desirable transmission capacity in current coherent optical communication systems. Digital signal processing techniques, such as electronic dispersion compensation and digital backpropagation, may achieve suboptimal performance or require impractical high computing resources. Machine learning (ML) techniques offer a promising solution to the optical fibre nonlinearity equalisation problem due to their ability to exploit underlying features among the vast volume of digital information available in modern society. This thesis focuses on the design of nonlinear equalisers using machine learning, specifically neural networks (NNs), for coherent optical communication systems. A comprehensive study of machine learning algorithms, including unsupervised and supervised learning algorithms, applied in nonlinear equalisation is conducted, demonstrating that the bit error rate performance of these algorithms is superior to conventional symbol detection using mean square error. The results demonstrate the potential of machine learning algorithms, particularly NNs, for nonlinearity equalisation in coherent optical systems and provide a motivation for further exploration. The ML- and recurrent neural network (RNN)-based nonlinear equalisers with sequential symbol input are investigated. The results suggest the input sequences can provide relevant residual channel memory information for these equalisers to enhance the system performance after training, offering confidence in the design of low-complexity NN-based equalisers. Furthermore, an attention-aided partial bidirectional recurrent neural network (BRNN)-based nonlinear equaliser is proposed, successfully reducing complexity of ∼56.2% with the assistance of the attention mechanism, which also provides evidence of symbol-wise nonlinear memory. The contributions presented in this thesis demonstrate the potential of machine learning algorithms and NN-based equalisers, investigate and validate the feasibility of sequential input for them, and provide an effective evidence-based pruning process for the design of NN-based equalisers for optical transmission systems. xv