Binary neural networks (BNNs) have attracted broad research interest due to
their efficient storage and computational ability. Nevertheless, a significant
challenge of BNNs lies in handling discrete constraints while ensuring bit
entropy maximization, which typically makes their weight optimization very
difficult. Existing methods relax the learning using the sign function, which
simply encodes positive weights into +1s, and -1s otherwise. Alternatively, we
formulate an angle alignment objective to constrain the weight binarization to
{0,+1} to solve the challenge. In this paper, we show that our weight
binarization provides an analytical solution by encoding high-magnitude weights
into +1s, and 0s otherwise. Therefore, a high-quality discrete solution is
established in a computationally efficient manner without the sign function. We
prove that the learned weights of binarized networks roughly follow a Laplacian
distribution that does not allow entropy maximization, and further demonstrate
that it can be effectively solved by simply removing the ℓ2​
regularization during network training. Our method, dubbed sign-to-magnitude
network binarization (SiMaN), is evaluated on CIFAR-10 and ImageNet,
demonstrating its superiority over the sign-based state-of-the-arts. Our source
code, experimental settings, training logs and binary models are available at
https://github.com/lmbxmu/SiMaN