Coherent optical transmission systems have a four-dimensional (4-D) signal space (two quadratures in two polarizations). These four dimensions can be used to create modulation formats that have a better power efficiency (higher sensitivity) than the conventional binary phase shift keying/quadrature phase shift keying (BPSK/QPSK) signals. Several examples are given, with some emphasis on a 24-level format and an 8-level format, including descriptions of how they can be realized and expressions for their symbol and bit error probabilities. These formats are, respectively, an extension and a subset of the commonly used 16-level dual-polarization QPSK format. Sphere packing simulations in 2, 3, and 4 dimensions, up to 32 levels, are used to verify their optimality. The numerical results, as the number of levels increases, are shown to agree with lattice-theoretical results. Finally, we point out that the use of these constellations will lead to improved fundamental sensitivity limits for optical communication systems, and they may also be relevant as a way of reducing power demands and/or nonlinear influence. \ua9 2009, IEEE. All rights reserved
