115 research outputs found
Tensor Computation: A New Framework for High-Dimensional Problems in EDA
Many critical EDA problems suffer from the curse of dimensionality, i.e. the
very fast-scaling computational burden produced by large number of parameters
and/or unknown variables. This phenomenon may be caused by multiple spatial or
temporal factors (e.g. 3-D field solvers discretizations and multi-rate circuit
simulation), nonlinearity of devices and circuits, large number of design or
optimization parameters (e.g. full-chip routing/placement and circuit sizing),
or extensive process variations (e.g. variability/reliability analysis and
design for manufacturability). The computational challenges generated by such
high dimensional problems are generally hard to handle efficiently with
traditional EDA core algorithms that are based on matrix and vector
computation. This paper presents "tensor computation" as an alternative general
framework for the development of efficient EDA algorithms and tools. A tensor
is a high-dimensional generalization of a matrix and a vector, and is a natural
choice for both storing and solving efficiently high-dimensional EDA problems.
This paper gives a basic tutorial on tensors, demonstrates some recent examples
of EDA applications (e.g., nonlinear circuit modeling and high-dimensional
uncertainty quantification), and suggests further open EDA problems where the
use of tensor computation could be of advantage.Comment: 14 figures. Accepted by IEEE Trans. CAD of Integrated Circuits and
System
Computing the polyadic decomposition of nonnegative third order tensors
International audienceComputing the minimal polyadic decomposition (also often referred to as canonical decomposition, or sometimes Parafac) amounts to finding the global minimum of a coercive polynomial in many variables. In the case of arrays with nonnegative entries, the low-rank approximation problem is well posed. In addition, due to the large dimension of the problem, the decomposition can be rather efficiently calculated with the help of preconditioned nonlinear conjugate gradient algorithms, as subsequently shown, if equipped with an algebraic calculation of the globally optimal stepsize in low dimension. Other algorithms are also studied (gradient and quasi-Newton approaches) for comparisons. Two versions of each algorithm are considered: the Enhanced Line Search version (ELS), and the backtracking version alternating with ELS. Computer simulations are provided and demonstrate the good behavior of these algorithms dedicated to nonnegative arrays, compared to others put forward in the literature. Finally, applications in the context of data analysis illustrate various algorithms. The main advantage of the suggested approach is to explicitly take into account the nonnegative nature of the loading matrices in the problem parameterization, instead of enforcing positive entries by projection. According to the experiments we have run, such an approach also happens to be more robust with respect to possible modeling errors
A quadratically convergent proximal algorithm for nonnegative tensor decomposition
The decomposition of tensors into simple rank-1 terms is key in a variety of
applications in signal processing, data analysis and machine learning. While
this canonical polyadic decomposition (CPD) is unique under mild conditions,
including prior knowledge such as nonnegativity can facilitate interpretation
of the components. Inspired by the effectiveness and efficiency of Gauss-Newton
(GN) for unconstrained CPD, we derive a proximal, semismooth GN type algorithm
for nonnegative tensor factorization. If the algorithm converges to the global
optimum, we show that -quadratic convergence can be obtained in the exact
case. Global convergence is achieved via backtracking on the forward-backward
envelope function. The -quadratic convergence is verified experimentally,
and we illustrate that using the GN step significantly reduces number of
(expensive) gradient computations compared to proximal gradient descent
Low Complexity Damped Gauss-Newton Algorithms for CANDECOMP/PARAFAC
The damped Gauss-Newton (dGN) algorithm for CANDECOMP/PARAFAC (CP)
decomposition can handle the challenges of collinearity of factors and
different magnitudes of factors; nevertheless, for factorization of an -D
tensor of size with rank , the algorithm is computationally
demanding due to construction of large approximate Hessian of size and its inversion where . In this paper, we propose a fast
implementation of the dGN algorithm which is based on novel expressions of the
inverse approximate Hessian in block form. The new implementation has lower
computational complexity, besides computation of the gradient (this part is
common to both methods), requiring the inversion of a matrix of size
, which is much smaller than the whole approximate Hessian, if
. In addition, the implementation has lower memory requirements,
because neither the Hessian nor its inverse never need to be stored in their
entirety. A variant of the algorithm working with complex valued data is
proposed as well. Complexity and performance of the proposed algorithm is
compared with those of dGN and ALS with line search on examples of difficult
benchmark tensors
Computing Large-Scale Matrix and Tensor Decomposition with Structured Factors: A Unified Nonconvex Optimization Perspective
The proposed article aims at offering a comprehensive tutorial for the
computational aspects of structured matrix and tensor factorization. Unlike
existing tutorials that mainly focus on {\it algorithmic procedures} for a
small set of problems, e.g., nonnegativity or sparsity-constrained
factorization, we take a {\it top-down} approach: we start with general
optimization theory (e.g., inexact and accelerated block coordinate descent,
stochastic optimization, and Gauss-Newton methods) that covers a wide range of
factorization problems with diverse constraints and regularization terms of
engineering interest. Then, we go `under the hood' to showcase specific
algorithm design under these introduced principles. We pay a particular
attention to recent algorithmic developments in structured tensor and matrix
factorization (e.g., random sketching and adaptive step size based stochastic
optimization and structure-exploiting second-order algorithms), which are the
state of the art---yet much less touched upon in the literature compared to
{\it block coordinate descent} (BCD)-based methods. We expect that the article
to have an educational values in the field of structured factorization and hope
to stimulate more research in this important and exciting direction.Comment: Final Version; to appear in IEEE Signal Processing Magazine; title
revised to comply with the journal's rul
Tensor Decomposition for Model Reduction in Neural Networks: A Review
Modern neural networks have revolutionized the fields of computer vision (CV)
and Natural Language Processing (NLP). They are widely used for solving complex
CV tasks and NLP tasks such as image classification, image generation, and
machine translation. Most state-of-the-art neural networks are
over-parameterized and require a high computational cost. One straightforward
solution is to replace the layers of the networks with their low-rank tensor
approximations using different tensor decomposition methods. This paper reviews
six tensor decomposition methods and illustrates their ability to compress
model parameters of convolutional neural networks (CNNs), recurrent neural
networks (RNNs) and Transformers. The accuracy of some compressed models can be
higher than the original versions. Evaluations indicate that tensor
decompositions can achieve significant reductions in model size, run-time and
energy consumption, and are well suited for implementing neural networks on
edge devices.Comment: IEEE Circuits and Systems Magazine, 202
- …