Error-Resilient Machine Learning in Near Threshold Voltage via Classifier Ensemble

In this paper, we present the design of error-resilient machine learning architectures by employing a distributed machine learning framework referred to as classifier ensemble (CE). CE combines several simple classifiers to obtain a strong one. In contrast, centralized machine learning employs a single complex block. We compare the random forest (RF) and the support vector machine (SVM), which are representative techniques from the CE and centralized frameworks, respectively. Employing the dataset from UCI machine learning repository and architectural-level error models in a commercial 45 nm CMOS process, it is demonstrated that RF-based architectures are significantly more robust than SVM architectures in presence of timing errors due to process variations in near-threshold voltage (NTV) regions (0.3 V - 0.7 V). In particular, the RF architecture exhibits a detection accuracy (P_{det}) that varies by 3.2% while maintaining a median P_{det} > 0.9 at a gate level delay variation of 28.9% . In comparison, SVM exhibits a P_{det} that varies by 16.8%. Additionally, we propose an error weighted voting technique that incorporates the timing error statistics of the NTV circuit fabric to further enhance robustness. Simulation results confirm that the error weighted voting achieves a P_{det} that varies by only 1.4%, which is 12X lower compared to SVM

Optimal stochastic extragradient schemes for pseudomonotone stochastic variational inequality problems and their variants

We consider the stochastic variational inequality problem in which the map is expectation-valued in a component-wise sense. Much of the available convergence theory and rate statements for stochastic approximation schemes are limited to monotone maps. However, non-monotone stochastic variational inequality problems are not uncommon and are seen to arise from product pricing, fractional optimization problems, and subclasses of economic equilibrium problems. Motivated by the need to address a broader class of maps, we make the following contributions: (i) We present an extragradient-based stochastic approximation scheme and prove that the iterates converge to a solution of the original problem under either pseudomonotonicity requirements or a suitably defined acute angle condition. Such statements are shown to be generalizable to the stochastic mirror-prox framework; (ii) Under strong pseudomonotonicity, we show that the mean-squared error in the solution iterates produced by the extragradient SA scheme converges at the optimal rate of O(1/k) statements that were hitherto unavailable K in this regime. Notably, we optimize the initial steplength by obtaining an {\epsilon}-infimum of a discontinuous nonconvex function. Similar statements are derived for mirror-prox generalizations and can accommodate monotone SVIs under a weak-sharpness requirement. Finally, both the asymptotics and the empirical rates of the schemes are studied on a set of variational problems where it is seen that the theoretically specified initial steplength leads to significant performance benefits.Comment: Computational Optimization and Applications, 201

Moment properties of multivariate infinitely divisible laws and criteria for self-decomposability

Ramachandran (1969, Theorem 8) has shown that for any univariate infinitely divisible distribution and any positive real number $\alpha$, an absolute moment of order $\alpha$ relative to the distribution exists (as a finite number) if and only if this is so for a certain truncated version of the corresponding L$\acute{\rm e}$vy measure. A generalized version of this result in the case of multivariate infinitely divisible distributions, involving the concept of g-moments, is given by Sato (1999, Theorem 25.3). We extend Ramachandran's theorem to the multivariate case, keeping in mind the immediate requirements under appropriate assumptions of cumulant studies of the distributions referred to; the format of Sato's theorem just referred to obviously varies from ours and seems to be having a different agenda. Also, appealing to a further criterion based on the L$\acute{\rm e}$vy measure, we identify in a certain class of multivariate infinitely divisible distributions the distributions that are self-decomposable; this throws new light on structural aspects of certain multivariate distributions such as the multivariate generalized hyperbolic distributions studied by Barndorff-Nielsen (1977) and others. Various points of relevance to the study are also addressed through specific examples.Comment: 22 pages (To appear in: Journal of Multivariate Analysis

Tractable ADMM Schemes for Computing KKT Points and Local Minimizers for ℓ0\ell_0-Minimization Problems

We consider an $\ell_0$-minimization problem where $f(x) + \gamma \|x\|_0$ is minimized over a polyhedral set and the $\ell_0$-norm regularizer implicitly emphasizes sparsity of the solution. Such a setting captures a range of problems in image processing and statistical learning. Given the nonconvex and discontinuous nature of this norm, convex regularizers are often employed as substitutes. Therefore, far less is known about directly solving the $\ell_0$-minimization problem. Inspired by [19], we consider resolving an equivalent formulation of the $\ell_0$-minimization problem as a mathematical program with complementarity constraints (MPCC) and make the following contributions towards the characterization and computation of its KKT points: (i) First, we show that feasible points of this formulation satisfy the relatively weak Guignard constraint qualification. Furthermore, under the suitable convexity assumption on $f(x)$, an equivalence is derived between first-order KKT points and local minimizers of the MPCC formulation. (ii) Next, we apply two alternating direction method of multiplier (ADMM) algorithms to exploit special structure of the MPCC formulation: (ADMM$_{\rm cf}^{\mu, \alpha, \rho}$) and (ADMM$_{\rm cf}$). These two ADMM schemes both have tractable subproblems. Specifically, in spite of the overall nonconvexity, we show that the first of the ADMM updates can be effectively reduced to a closed-form expression by recognizing a hidden convexity property while the second necessitates solving a convex program. In (ADMM$_{\rm cf}^{\mu, \alpha, \rho}$), we prove subsequential convergence to a perturbed KKT point under mild assumptions. Our preliminary numerical experiments suggest that the tractable ADMM schemes are more scalable than their standard counterpart and ADMM$_{\rm cf}$ compares well with its competitors to solve the $\ell_0$-minimization problem.Comment: 47 pages, 3 table

On the resolution of misspecified convex optimization and monotone variational inequality problems

We consider a misspecified optimization problem that requires minimizing a function f(x;q*) over a closed and convex set X where q* is an unknown vector of parameters that may be learnt by a parallel learning process. In this context, We examine the development of coupled schemes that generate iterates {x_k,q_k} as k goes to infinity, then {x_k} converges x*, a minimizer of f(x;q*) over X and {q_k} converges to q*. In the first part of the paper, we consider the solution of problems where f is either smooth or nonsmooth under various convexity assumptions on function f. In addition, rate statements are also provided to quantify the degradation in rate resulted from learning process. In the second part of the paper, we consider the solution of misspecified monotone variational inequality problems to contend with more general equilibrium problems as well as the possibility of misspecification in the constraints. We first present a constant steplength misspecified extragradient scheme and prove its asymptotic convergence. This scheme is reliant on problem parameters (such as Lipschitz constants)and leads us to present a misspecified variant of iterative Tikhonov regularization. Numerics support the asymptotic and rate statements.Comment: 35 pages, 5 figure

Asynchronous Schemes for Stochastic and Misspecified Potential Games and Nonconvex Optimization

The distributed computation of equilibria and optima has seen growing interest in a broad collection of networked problems. We consider the computation of equilibria of convex stochastic Nash games characterized by a possibly nonconvex potential function. Our focus is on two classes of stochastic Nash games: (P1): A potential stochastic Nash game, in which each player solves a parameterized stochastic convex program; and (P2): A misspecified generalization, where the player-specific stochastic program is complicated by a parametric misspecification. In both settings, exact proximal BR solutions are generally unavailable in finite time since they necessitate solving parameterized stochastic programs. Consequently, we design two asynchronous inexact proximal BR schemes to solve the problems, where in each iteration a single player is randomly chosen to compute an inexact proximal BR solution with rivals' possibly outdated information. Yet, in the misspecified regime (P2), each player possesses an extra estimate of the misspecified parameter and updates its estimate by a projected stochastic gradient (SG) algorithm. By Since any stationary point of the potential function is a Nash equilibrium of the associated game, we believe this paper is amongst the first ones for stochastic nonconvex (but block convex) optimization problems equipped with almost-sure convergence guarantees. These statements can be extended to allow for accommodating weighted potential games and generalized potential games. Finally, we present preliminary numerics based on applying the proposed schemes to congestion control and Nash-Cournot games

On robust solutions to uncertain linear complementarity problems and their variants

A popular approach for addressing uncertainty in variational inequality problems is by solving the expected residual minimization (ERM) problem. This avenue necessitates distributional information associated with the uncertainty and requires minimizing nonconvex expectation-valued functions. We consider a distinctly different approach in the context of uncertain linear complementarity problems with a view towards obtaining robust solutions. Specifically, we define a robust solution to a complementarity problem as one that minimizes the worst-case of the gap function. In what we believe is amongst the first efforts to comprehensively address such problems in a distribution-free environment, we show that under specified assumptions on the uncertainty sets, the robust solutions to uncertain monotone linear complementarity problem can be tractably obtained through the solution of a single convex program. We also define uncertainty sets that ensure that robust solutions to non-monotone generalizations can also be obtained by solving convex programs. More generally, robust counterparts of uncertain non-monotone LCPs are proven to be low-dimensional nonconvex quadratically constrained quadratic programs. We show that these problems may be globally resolved by customizing an existing branching scheme. We further extend the tractability results to include uncertain affine variational inequality problems defined over uncertain polyhedral sets as well as to hierarchical regimes captured by mathematical programs with uncertain complementarity constraints. Preliminary numerics on uncertain linear complementarity and traffic equilibrium problems suggest that the presented avenues hold promise.Comment: 37 pages, 3 figures, 8 table

On the analysis of variance-reduced and randomized projection variants of single projection schemes for monotone stochastic variational inequality problems

Classical extragradient schemes and their stochastic counterpart represent a cornerstone for resolving monotone variational inequality problems. Yet, such schemes have a per-iteration complexity of two projections onto a convex set and require two evaluations of the map, the former of which could be relatively expensive if $X$ is a complicated set. We consider two related avenues where the per-iteration complexity is significantly reduced: (i) A stochastic projected reflected gradient method requiring a single evaluation of the map and a single projection; and (ii) A stochastic subgradient extragradient method that requires two evaluations of the map, a single projection onto $X$, and a significantly cheaper projection (onto a halfspace) computable in closed form. Under a variance-reduced framework reliant on a sample-average of the map based on an increasing batch-size, we prove almost sure (a.s.) convergence of the iterates to a random point in the solution set for both schemes. Additionally, both schemes display a non-asymptotic rate of $\mathcal{O}(1/K)$ where $K$ denotes the number of iterations; notably, both rates match those obtained in deterministic regimes. To address feasibility sets given by the intersection of a large number of convex constraints, we adapt both of the aforementioned schemes to a random projection framework. We then show that the random projection analogs of both schemes also display a.s. convergence under a weak-sharpness requirement; furthermore, without imposing the weak-sharpness requirement, both schemes are characterized by a provable rate of $\mathcal{O}(1/\sqrt{K})$ in terms of the gap function of the projection of the averaged sequence onto $X$ as well as the infeasibility of this sequence. Preliminary numerics support theoretical findings and the schemes outperform standard extragradient schemes in terms of the per-iteration complexity

Distributed Variable Sample-Size Gradient-response and Best-response Schemes for Stochastic Nash Equilibrium Problems over Graphs

This paper considers a stochastic Nash game in which each player minimizes an expectation valued composite objective. We make the following contributions. (I) Under suitable monotonicity assumptions on the concatenated gradient map, we derive optimal rate statements and oracle complexity bounds for the proposed variable sample-size proximal stochastic gradient-response (VS-PGR) scheme when the sample-size increases at a geometric rate. If the sample-size increases at a polynomial rate of degree $v > 0$, the mean-squared errordecays at a corresponding polynomial rate while the iteration and oracle complexities to obtain an $\epsilon$-NE are $\mathcal{O}(1/\epsilon^{1/v})$ and $\mathcal{O}(1/\epsilon^{1+1/v})$, respectively. (II) We then overlay (VS-PGR) with a consensus phase with a view towards developing distributed protocols for aggregative stochastic Nash games. In the resulting scheme, when the sample-size and the consensus steps grow at a geometric and linear rate, computing an $\epsilon$-NE requires similar iteration and oracle complexities to (VS-PGR) with a communication complexity of $\mathcal{O}(\ln^2(1/\epsilon))$; (III) Under a suitable contractive property associated with the proximal best-response (BR) map, we design a variable sample-size proximal BR (VS-PBR) scheme, where each player solves a sample-average BR problem. Akin to (I), we also give the rate statements, oracle and iteration complexity bounds. (IV) Akin to (II), the distributed variant achieves similar iteration and oracle complexities to the centralized (VS-PBR) with a communication complexity of $\mathcal{O}(\ln^2(1/\epsilon))$ when the communication rounds per iteration increase at a linear rate. Finally, we present some preliminary numerics to provide empirical support for the rate and complexity statements

SI-ADMM: A Stochastic Inexact ADMM Framework for Stochastic Convex Programs

We consider the structured stochastic convex program requiring the minimization of $\mathbb{E}[\tilde f(x,\xi)]+\mathbb{E}[\tilde g(y,\xi)]$ subject to the constraint $Ax + By = b$. Motivated by the need for decentralized schemes and structure, we propose a stochastic inexact ADMM (SI-ADMM) framework where subproblems are solved inexactly via stochastic approximation schemes. Based on this framework, we prove the following: (i) under suitable assumptions on the associated batch-size of samples utilized at each iteration, the SI-ADMM scheme produces a sequence that converges to the unique solution almost surely; (ii) If the number of gradient steps (or equivalently, the number of sampled gradients) utilized for solving the subproblems in each iteration increases at a geometric rate, the mean-squared error diminishes to zero at a prescribed geometric rate; (iii) The overall iteration complexity in terms of gradient steps (or equivalently samples) is found to be consistent with the canonical level of $\mathcal{O}(1/\epsilon)$. Preliminary applications on LASSO and distributed regression suggest that the scheme performs well compared to its competitors.Comment: 37 pages, 2 figures, 3 table