The rapid growth in interactive bandwidth-hungry services demands ever higher
capacity at various stages of the optical network, leading to a potential capacity exhaust,
termed the capacity crunch. The main aim of the research work described in this thesis
was to help solve the potential capacity crunch by exploring techniques to increase the
data rate, spectral efficiency and reach of optical fibre systems. The focus was on the
use of advanced signal modulation formats, including optical time-division multiplexing
(OTDM), quadrature phase shift keying (QPSK), and 16-state quadrature amplitude
modulation (QAM16). QPSK and QAM16 modulations formats were studied in
combination with coherent detection and digital signal processing (DSP) for the
compensation of transmission impairments. In addition, return-to-zero (RZ) pulses were
explored to increase the tolerance towards nonlinearity for coherently detected signals,
and nonlinearity compensation (NLC) through the DSP.
Initially, to maximise the bit-rate, research was focused on the study of OTDM
transmission at 80Gbit/s with the aim to optimise the phase difference between the
adjacent OTDM channels. A new technique to achieve bit-wise phase control using a
phase-stabilised fibre interferometer was proposed. Faced with a limited fibre capacity,
the need to maximise the spectral efficiency became paramount, and thus the need to
use phase, amplitude and polarisation domains for signal transmission. In combination
with coherent detection the research focused on the performance of optical fibre systems
using QPSK and QAM16 modulation formats, including their generation, transmission
and detection in single-channel and WDM regimes. This included the study of the
impact of pulse shapes, and the mitigation of linear and nonlinear transmission
impairments with receiver-based DSP at bit-rates ranging from 42.7 to 224Gbit/s. The
technique demonstrated for bit-wise phase control for OTDM was successfully used to
demonstrate a new method for QAM16 signal generation. Longest transmission
distances (up to 10160km in 112Gbit/s QPSK, 4240km in 112Gbit/s QAM16, and
2000km in 224Gbit/s QAM16) have been achieved with the use of NLC and RZ pulses.
The efficiency of these two techniques is explored through a comprehensive set of
experiments in both single-channel and WDM transmission experiments. The results
can be used in the design of future optical transmission systems