Tradeoffs for language recognition on alternating machines

AbstractThe alternating machine having a separate input tape with k two-way, read-only heads, and a certain number of internal configurations, AM(k), is considered as a parallel computing model. For the complexity measure TIME · SPACE · PARALLELISM (TSP), the optimal lower bounds Ω(n2) and Ω(n3/2) respectively are proved for the recognition of specific languages on AM(1) and AM(k) respectively. For the complexity measure REVERSALS · SPACE · PARALLELISM (RSP), the lower bound Ω(n1/2) is established for the recognition of a specific language on AM(k). This result implies a polynomial lower bound on PARALLEL TIME · HARDWARE of parallel RAM's.Lower bounds on the complexity measures TIME · SPACE and REVERSALS · SPACE of nondeterministic machines are direct consequences of the result introduced above.All lower bounds obtained are substantially improved in the case that SPACE⩾ nɛ for 0<ɛ<1. Several strongest lower bounds for two-way and one-way alternating (deterministic, nondeterministic) multihead finite automata are obtained as direct consequences of these results. The hierarchies for the complexity measures TSP, RSP, TS and RS can be immediately achieved too

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

In order to fully utilize "big data", it is often required to use "big models". Such models tend to grow with the complexity and size of the training data, and do not make strong parametric assumptions upfront on the nature of the underlying statistical dependencies. Kernel methods fit this need well, as they constitute a versatile and principled statistical methodology for solving a wide range of non-parametric modelling problems. However, their high computational costs (in storage and time) pose a significant barrier to their widespread adoption in big data applications. We propose an algorithmic framework and high-performance implementation for massive-scale training of kernel-based statistical models, based on combining two key technical ingredients: (i) distributed general purpose convex optimization, and (ii) the use of randomization to improve the scalability of kernel methods. Our approach is based on a block-splitting variant of the Alternating Directions Method of Multipliers, carefully reconfigured to handle very large random feature matrices, while exploiting hybrid parallelism typically found in modern clusters of multicore machines. Our implementation supports a variety of statistical learning tasks by enabling several loss functions, regularization schemes, kernels, and layers of randomized approximations for both dense and sparse datasets, in a highly extensible framework. We evaluate the ability of our framework to learn models on data from applications, and provide a comparison against existing sequential and parallel libraries.Comment: Work presented at MMDS 2014 (June 2014) and JSM 201

Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations

Deep CCA is a recently proposed deep neural network extension to the traditional canonical correlation analysis (CCA), and has been successful for multi-view representation learning in several domains. However, stochastic optimization of the deep CCA objective is not straightforward, because it does not decouple over training examples. Previous optimizers for deep CCA are either batch-based algorithms or stochastic optimization using large minibatches, which can have high memory consumption. In this paper, we tackle the problem of stochastic optimization for deep CCA with small minibatches, based on an iterative solution to the CCA objective, and show that we can achieve as good performance as previous optimizers and thus alleviate the memory requirement.Comment: in 2015 Annual Allerton Conference on Communication, Control and Computin