9 research outputs found
Approximation with Tensor Networks. Part III: Multivariate Approximation
We study the approximation of multivariate functions with tensor networks
(TNs). The main conclusion of this work is an answer to the following two
questions: "What are the approximation capabilities of TNs?" and "What is an
appropriate model class of functions that can be approximated with TNs?" To
answer the former: we show that TNs can (near to) optimally replicate
-uniform and -adaptive approximation, for any smoothness order of the
target function. Tensor networks thus exhibit universal expressivity w.r.t.
isotropic, anisotropic and mixed smoothness spaces that is comparable with more
general neural networks families such as deep rectified linear unit (ReLU)
networks. Put differently, TNs have the capacity to (near to) optimally
approximate many function classes -- without being adapted to the particular
class in question. To answer the latter: as a candidate model class we consider
approximation classes of TNs and show that these are (quasi-)Banach spaces,
that many types of classical smoothness spaces are continuously embedded into
said approximation classes and that TN approximation classes are themselves not
embedded in any classical smoothness space.Comment: For part I see arXiv:2007.00118, for part II see arXiv:2007.0012
Hierarchical adaptive low-rank format with applications to discretized partial differential equations
A novel framework for hierarchical low-rank matrices is proposed that combines an adaptive hierarchical partitioning of the matrix with low-rank approximation. One typical application is the approximation of discretized functions on rectangular domains; the flexibility of the format makes it possible to deal with functions that feature singularities in small, localized regions. To deal with time evolution and relocation of singularities, the partitioning can be dynamically adjusted based on features of the underlying data. Our format can be leveraged to efficiently solve linear systems with Kronecker product structure, as they arise from discretized partial differential equations (PDEs). For this purpose, these linear systems are rephrased as linear matrix equations and a recursive solver is derived from low-rank updates of such equations. We demonstrate the effectiveness of our framework for stationary and time-dependent, linear and nonlinear PDEs, including the Burgers' and Allen-Cahn equations
Approximation with Tensor Networks. Part II: Approximation Rates for Smoothness Classes
We study the approximation by tensor networks (TNs) of functions from
classical smoothness classes. The considered approximation tool combines a
tensorization of functions in , which allows to identify a
univariate function with a multivariate function (or tensor), and the use of
tree tensor networks (the tensor train format) for exploiting low-rank
structures of multivariate functions. The resulting tool can be interpreted as
a feed-forward neural network, with first layers implementing the
tensorization, interpreted as a particular featuring step, followed by a
sum-product network with sparse architecture. In part I of this work, we
presented several approximation classes associated with different measures of
complexity of tensor networks and studied their properties. In this work (part
II), we show how classical approximation tools, such as polynomials or splines
(with fixed or free knots), can be encoded as a tensor network with controlled
complexity. We use this to derive direct (Jackson) inequalities for the
approximation spaces of tensor networks. This is then utilized to show that
Besov spaces are continuously embedded into these approximation spaces. In
other words, we show that arbitrary Besov functions can be approximated with
optimal or near to optimal rate. We also show that an arbitrary function in the
approximation class possesses no Besov smoothness, unless one limits the depth
of the tensor network.Comment: For part I see arXiv:2007.00118, for part III see arXiv:2101.1193
Approximation with Tensor Networks. Part I: Approximation Spaces
We study the approximation of functions by tensor networks (TNs). We show
that Lebesgue -spaces in one dimension can be identified with tensor
product spaces of arbitrary order through tensorization. We use this tensor
product structure to define subsets of of rank-structured functions of
finite representation complexity. These subsets are then used to define
different approximation classes of tensor networks, associated with different
measures of complexity. These approximation classes are shown to be
quasi-normed linear spaces. We study some elementary properties and
relationships of said spaces. In part II of this work, we will show that
classical smoothness (Besov) spaces are continuously embedded into these
approximation classes. We will also show that functions in these approximation
classes do not possess any Besov smoothness, unless one restricts the depth of
the tensor networks. The results of this work are both an analysis of the
approximation spaces of TNs and a study of the expressivity of a particular
type of neural networks (NN) -- namely feed-forward sum-product networks with
sparse architecture. The input variables of this network result from the
tensorization step, interpreted as a particular featuring step which can also
be implemented with a neural network with a specific architecture. We point out
interesting parallels to recent results on the expressivity of rectified linear
unit (ReLU) networks -- currently one of the most popular type of NNs.Comment: For part II see arXiv:2007.00128, for part III see arXiv:2101.1193