The quantization of neural networks for the mitigation of the nonlinear and
components' distortions in dual-polarization optical fiber transmission is
studied. Two low-complexity neural network equalizers are applied in three
16-QAM 34.4 GBaud transmission experiments with different representative
fibers. A number of post-training quantization and quantization-aware training
algorithms are compared for casting the weights and activations of the neural
network in few bits, combined with the uniform, additive power-of-two, and
companding quantization. For quantization in the large bit-width regime of
≥5 bits, the quantization-aware training with the straight-through
estimation incurs a Q-factor penalty of less than 0.5 dB compared to the
unquantized neural network. For quantization in the low bit-width regime, an
algorithm dubbed companding successive alpha-blending quantization is
suggested. This method compensates for the quantization error aggressively by
successive grouping and retraining of the parameters, as well as an incremental
transition from the floating-point representations to the quantized values
within each group. The activations can be quantized at 8 bits and the weights
on average at 1.75 bits, with a penalty of ≤0.5~dB. If the activations
are quantized at 6 bits, the weights can be quantized at 3.75 bits with minimal
penalty. The computational complexity and required storage of the neural
networks are drastically reduced, typically by over 90\%. The results indicate
that low-complexity neural networks can mitigate nonlinearities in optical
fiber transmission.Comment: 15 pages, 9 figures, 5 table