Patterns of Scalable Bayesian Inference

Datasets are growing not just in size but in complexity, creating a demand for rich models and quantification of uncertainty. Bayesian methods are an excellent fit for this demand, but scaling Bayesian inference is a challenge. In response to this challenge, there has been considerable recent work based on varying assumptions about model structure, underlying computational resources, and the importance of asymptotic correctness. As a result, there is a zoo of ideas with few clear overarching principles. In this paper, we seek to identify unifying principles, patterns, and intuitions for scaling Bayesian inference. We review existing work on utilizing modern computing resources with both MCMC and variational approximation techniques. From this taxonomy of ideas, we characterize the general principles that have proven successful for designing scalable inference procedures and comment on the path forward

Expectation-maximization for logistic regression

We present a family of expectation-maximization (EM) algorithms for binary and negative-binomial logistic regression, drawing a sharp connection with the variational-Bayes algorithm of Jaakkola and Jordan (2000). Indeed, our results allow a version of this variational-Bayes approach to be re-interpreted as a true EM algorithm. We study several interesting features of the algorithm, and of this previously unrecognized connection with variational Bayes. We also generalize the approach to sparsity-promoting priors, and to an online method whose convergence properties are easily established. This latter method compares favorably with stochastic-gradient descent in situations with marked collinearity

Morphological Network: How Far Can We Go with Morphological Neurons?

In recent years, the idea of using morphological operations as networks has received much attention. Mathematical morphology provides very efficient and useful image processing and image analysis tools based on basic operators like dilation and erosion, defined in terms of kernels. Many other morphological operations are built up using the dilation and erosion operations. Although the learning of structuring elements such as dilation or erosion using the backpropagation algorithm is not new, the order and the way these morphological operations are used is not standard. In this paper, we have theoretically analyzed the use of morphological operations for processing 1D feature vectors and shown that this gets extended to the 2D case in a simple manner. Our theoretical results show that a morphological block represents a sum of hinge functions. Hinge functions are used in many places for classification and regression tasks (Breiman (1993)). We have also proved a universal approximation theorem -- a stack of two morphological blocks can approximate any continuous function over arbitrary compact sets. To experimentally validate the efficacy of this network in real-life applications, we have evaluated its performance on satellite image classification datasets since morphological operations are very sensitive to geometrical shapes and structures. We have also shown results on a few tasks like segmentation of blood vessels from fundus images, segmentation of lungs from chest x-ray and image dehazing. The results are encouraging and further establishes the potential of morphological networks.Comment: 35 pages, 19 figures, 7 table

Binary Kummer Line

Gaudry and Lubicz introduced the idea of Kummer line in 2009, and Karati and Sarkar proposed three Kummer lines over prime fields in 2017. In this work, we explore the problem of secure and efficient scalar multiplications on binary field using Kummer line and investigate the possibilities of speedups using Kummer line compared to Koblitz curves, binary Edwards curve and Weierstrass curves. We propose a binary Kummer line $\mathsf{BKL}251$ over binary field $\mathbb{F}_{2^{251}}$ where the associated elliptic curve satisfies the required security conditions and offers 124.5-bit security which is the same as that of Binary Edwards curve $\mathsf{BEd251}$ and Weierstrass curve $\mathsf{CURVE2251}$. $\mathsf{BKL}251$ has small curve parameter and small base point. We implement our software of $\mathsf{BKL}l251$ using the instruction ${\tt PCLMULQDQ}$ of modern Intel processors and batch software $\mathsf{BBK251}$ using bitslicing technique. For fair comparison, we also implement the software $\mathsf{BEd}251$ for binary Edwards curve. In both the implementations, scalar multiplications take constant time which use Montgomery ladders. In case of left-to-right Montgomery ladder, both the Kummer line and Edwards curve have almost the same number of field operations. For right-to-left Montgomery ladder scalar multiplication, each ladder step of binary Kummer line needs less number of field operations compared to Edwards curve. Our experimental results show that left-to-right Montgomery scalar multiplications of $\mathsf{BKL}251$ are $9.63\%$ and $0.52\%$ faster than those of $\mathsf{BEd}251$ for fixed-base and variable-base, respectively. Left-to-right Montgomery scalar multiplication for variable-base of $\mathsf{BKL}251$ is 39.74\%, 23.25\% and 32.92\% faster than those of the curves $\mathsf{CURVE2251}$, K-283 and B-283 respectively. Using right-to-left Montgomery ladder with precomputation, $\mathsf{BKL}251$ achieves 17.84\% speedup over $\mathsf{BEd}251$ for fixed-base scalar multiplication. For batch computation, $\mathsf{BBK251}$ has comparatively the same (slightly faster) performance as $\mathsf{BBE251}$ and $\mathsf{sect283r1}$. Also it is clear from our experiments that scalar multiplications on $\mathsf{BKL}251$ and $\mathsf{BEd251}$ are (approximately) 65\% faster than one scalar multiplication (after scaling down) of batch software $\mathsf{BBK251}$ and $\mathsf{BBE251}$

Ranks of elliptic curves and deep neural networks

Determining the rank of an elliptic curve E/Q is a hard problem, and in some applications (e.g. when searching for curves of high rank) one has to rely on heuristics aimed at estimating the analytic rank (which is equal to the rank under the Birch and Swinnerton-Dyer conjecture). In this paper, we develop rank classification heuristics modeled by deep convolutional neural networks (CNN). Similarly to widely used Mestre-Nagao sums, it takes as an input the conductor of E and a sequence of normalized a_p-s (where a_p=p+1-#E(F_p) if p is a prime of good reduction) in some range (p<10^k for k=3,4,5), and tries to predict rank (or detect curves of ``high'' rank). The model has been trained and tested on two datasets: the LMFDB and a custom dataset consisting of elliptic curves with trivial torsion, conductor up to 10^30, and rank up to 10. For comparison, eight simple neural network models of Mestre-Nagao sums have also been developed. Experiments showed that CNN performed better than Mestre-Nagao sums on the LMFDB dataset (interestingly neural network that took as an input all Mestre-Nagao sums performed much better than each sum individually), while they were roughly equal on custom made dataset.Comment: 20 pages, 7 figures, comments welcome