The optical fibres form a basis of the long-haul transmissions systems, and is a significant component of the connectivity infrastructure. Rapidly growing demand in data traffic requires instantaneous imperative actions with long-term effect to meet the future expansion of the digital economy. The current optical networks resources are overstretched, and the further extensive utilisation will ultimately constrain the development of other economic sectors. The intelligent and effective usage of the installed infrastructure can shift forward the existent limitations, keeping the cost low because of avoiding of the reinstallation. One of the principal constrain, which bounds the further optical fibre capacity grows, is the existence of undesirable nonlinear phenomena, the so-called Kerr nonlinearity, causing self-phase, cross-phase modulation and four-wave mixing. The combination of advanced achievements of mathematical physics, together with communication engineering and information theory allowed to implement the so-called nonlinear Fourier transform (NFT) approach to optical communication. In its paradigm, the fibre nonlinearity is considered as a valuable part of the model, and the NFT mapping effectively (de)composes the signal to naturally non-interacting modes. The NFT concept can be applied to the signal propagation model with either vanishing or periodic boundary condition, which involves the different structures of parameters for manipulation. In this thesis, I focused on the investigation of boundary condition cases, discovering analytical properties, available degrees of freedom, developing numerical methods, and coding approaches; then examining their performance via the simulation of optical transmission systems. The results allow us to conclude the existence of several technical limitations, which limit the achievable transmission quality and data-rate. These include: the deviation of the channel model from the purely integrable, nonlinear and not explicit coupling of the resulting signal parameters, numerical methods accuracy and amplifiers noise accumulation. In spite of those, the simulations demonstrate the considerable performance of NFT-based communication systems
