Precise estimation of cross-correlation or similarity between two random
variables lies at the heart of signal detection, hyperdimensional computing,
associative memories, and neural networks. Although a vast literature exists on
different methods for estimating cross-correlations, the question what is the
best and simplest method to estimate cross-correlations using finite samples ?
is still not clear. In this paper, we first argue that the standard empirical
approach might not be the optimal method even though the estimator exhibits
uniform convergence to the true cross-correlation. Instead, we show that there
exists a large class of simple non-linear functions that can be used to
construct cross-correlators with a higher signal-to-noise ratio (SNR). To
demonstrate this, we first present a general mathematical framework using
Price's Theorem that allows us to analyze cross-correlators constructed using a
mixture of piece-wise linear functions. Using this framework and
high-dimensional embedding, we show that some of the most promising
cross-correlators are based on Huber's loss functions, margin-propagation (MP)
functions, and the log-sum-exp functions.Comment: 9 figure, 13 page