Fiber nonlinearity is the main factor limiting the transmission distance of coherent optical communications. We overview several modulation formats intrinsically tolerant to fiber nonlinearity. We recently proposed family of 4D modulation formats based on 2-ary amplitude 8-ary phase-shift keying (2A8PSK), covering the spectral efficiency of 5, 6, and 7 bits/4D symbol, which will be explained in detail in this chapter. These coded modulation formats fill the gap of spectral efficiency between DP-QPSK and DP-16QAM, showing superb performance both in linear and nonlinear regimes. Since these modulation formats share the same constellation and use different parity bit expressions only, digital signal processing can accommodate those multiple modulation formats with minimum additional complexity. Nonlinear transmission simulations indicate that these modulation formats outperform the conventional formats at each spectral efficiency. We also review DSP algorithms and experimental results. Their application to time-domain hybrid modulation for 4–8 bits/4D symbol is also reviewed. Furthermore, an overview of an eight-dimensional 2A8PSK-based modulation format based on a Grassmann code is also given. All these results indicate that the 4D-2A8PSK family show great promise of excellent linear and nonlinear performances in the spectral efficiency between 3.5 and 8 bits/4D symbol
