Scalable and Sustainable Deep Learning via Randomized Hashing

Current deep learning architectures are growing larger in order to learn from complex datasets. These architectures require giant matrix multiplication operations to train millions of parameters. Conversely, there is another growing trend to bring deep learning to low-power, embedded devices. The matrix operations, associated with both training and testing of deep networks, are very expensive from a computational and energy standpoint. We present a novel hashing based technique to drastically reduce the amount of computation needed to train and test deep networks. Our approach combines recent ideas from adaptive dropouts and randomized hashing for maximum inner product search to select the nodes with the highest activation efficiently. Our new algorithm for deep learning reduces the overall computational cost of forward and back-propagation by operating on significantly fewer (sparse) nodes. As a consequence, our algorithm uses only 5% of the total multiplications, while keeping on average within 1% of the accuracy of the original model. A unique property of the proposed hashing based back-propagation is that the updates are always sparse. Due to the sparse gradient updates, our algorithm is ideally suited for asynchronous and parallel training leading to near linear speedup with increasing number of cores. We demonstrate the scalability and sustainability (energy efficiency) of our proposed algorithm via rigorous experimental evaluations on several real datasets

Approximate Least Squares

We present a novel iterative algorithm for approximating the linear least squares solution with low complexity. After a motivation of the algorithm we discuss the algorithm's properties including its complexity, and we present theoretical results as well as simulation based performance results. We describe the analysis of its convergence behavior and show that in the noise free case the algorithm converges to the least squares solution.Comment: Preprint of the paper submitted to IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 201

Eigenvalue-based Cyclostationary Spectrum Sensing Using Multiple Antennas

In this paper, we propose a signal-selective spectrum sensing method for cognitive radio networks and specifically targeted for receivers with multiple-antenna capability. This method is used for detecting the presence or absence of primary users based on the eigenvalues of the cyclic covariance matrix of received signals. In particular, the cyclic correlation significance test is used to detect a specific signal-of-interest by exploiting knowledge of its cyclic frequencies. The analytical threshold for achieving constant false alarm rate using this detection method is presented, verified through simulations, and shown to be independent of both the number of samples used and the noise variance, effectively eliminating the dependence on accurate noise estimation. The proposed method is also shown, through numerical simulations, to outperform existing multiple-antenna cyclostationary-based spectrum sensing algorithms under a quasi-static Rayleigh fading channel, in both spatially correlated and uncorrelated noise environments. The algorithm also has significantly lower computational complexity than these other approaches.Comment: 6 pages, 6 figures, accepted to IEEE GLOBECOM 201