The continuous expansion of multimedia terminals, like smart-phones, besides the introduction of new multimedia services, like the high-definition streaming television, determines a constant increase in the demand of high-speed connectivity. This scenario requires a permanent update of the data capacity in local access networks. Nevertheless, a proper capacity upgrades plan, with new optical links deployment, has required for back-bone optical networks in order to avoid possible networks congestions. Such effect has theoretically forecast as probable in actual networks within next few tenths years, particularly in US if some network segments will reach their saturation. Nowadays, the most promising technology for future optical communication networks has proven to be coherent detection. Rather than intensity modulation direct detection systems, such solution enables to implement new multilevel transmission systems, allowing a more efficient use of the optical transmission bandwidth. These novel coherent optical communication systems have been made possible thanks to the availability of high-speed (gigabit-per-second) analog and digital opto-/electronic devices. Such new devices have allowed the introduction of real-time digital signal processing (DSP) in coherent receivers, so overcoming the practical limitations that were experienced in coherent optical system experiments in the early 1990s. In this thesis, four main topics have been investigated to assess the feasibility and behaviour of spectrally efficient multichannel coherent optical communication systems for the highest data channel capacity in ultra-long-haul links. More in details, Nyquist filtered multilevel optical modulation formats (QPSK, 8QAM and 16QAM) have been investigated as candidates for the efficient transmission of information (toward the Shannon limit). Wavelength division multiplexed (WDM) systems assembled with these modulation formats have proven to support high spectral efficiency for data transmission over distances ranging from 3000~km to 10000~km. Besides the report of state-of-the-art transmission experiments, some practical and theoretical limitations have been discussed to motivate choices for laboratory experimental solutions as well as devices characteristics, DSP parameters, etc. The introduction of a novel mathematical model, accounting for non-linear interference (NLI) accumulations in optical lines optimized for systems based on coherent detection, has required an experimental validation through ad-hoc Nyquist-WDM transmission experiments. Reliability of the NLI model in performance predictions, enables to derive analytical expressions of transmission quantities as: the Figure of Merit of optical fibers, the non-linear channel capacity, etc. Furthermore, a simulative study to investigate the potentiality of a digital back-propagation (DBP) technique has been performed. The application of a digital non-linear compensation technique to data of a Nyquist-WDM transmission experiment has been carried out. Such analysis confirmed a loss in efficiency of DBP based on single channel processing to improve transmission distance in ultra-dense WDM systems as Nyquist-WDM. Finally, a simulative study about the polarization dependent loss (PDL) impact on system performances has been reported. Results from this analysis have justified the introduction in the DSP of the coherent receiver algorithms for the continuous channel parameters monitor. This DSP upgrade allowed to improve and speed-up laboratory measurements. The thesis is organized as follows: in the first chapter, it is reported a brief review of motivations, advantages and drawbacks in reintroducing coherent detection based systems in optical networks. Thereafter, novel DSP-based coherent optical systems are briefly introduced with some references about DSP algorithms reported in Appendix A. Theoretical foundations about the application of Nyquist theory to achieve ultra-dense WDM systems are reported in the second chapter. Chapters three to six have been dedicated to separately report each of the four investigated topics mentioned above. The final chapter contains a conclusive analysis, and it has followed by a reference bibliography and a complete list of the papers published within the three-year PhD period (2010-2012