5,493 research outputs found
Low Complexity Regularization of Linear Inverse Problems
Inverse problems and regularization theory is a central theme in contemporary
signal processing, where the goal is to reconstruct an unknown signal from
partial indirect, and possibly noisy, measurements of it. A now standard method
for recovering the unknown signal is to solve a convex optimization problem
that enforces some prior knowledge about its structure. This has proved
efficient in many problems routinely encountered in imaging sciences,
statistics and machine learning. This chapter delivers a review of recent
advances in the field where the regularization prior promotes solutions
conforming to some notion of simplicity/low-complexity. These priors encompass
as popular examples sparsity and group sparsity (to capture the compressibility
of natural signals and images), total variation and analysis sparsity (to
promote piecewise regularity), and low-rank (as natural extension of sparsity
to matrix-valued data). Our aim is to provide a unified treatment of all these
regularizations under a single umbrella, namely the theory of partial
smoothness. This framework is very general and accommodates all low-complexity
regularizers just mentioned, as well as many others. Partial smoothness turns
out to be the canonical way to encode low-dimensional models that can be linear
spaces or more general smooth manifolds. This review is intended to serve as a
one stop shop toward the understanding of the theoretical properties of the
so-regularized solutions. It covers a large spectrum including: (i) recovery
guarantees and stability to noise, both in terms of -stability and
model (manifold) identification; (ii) sensitivity analysis to perturbations of
the parameters involved (in particular the observations), with applications to
unbiased risk estimation ; (iii) convergence properties of the forward-backward
proximal splitting scheme, that is particularly well suited to solve the
corresponding large-scale regularized optimization problem
Private Incremental Regression
Data is continuously generated by modern data sources, and a recent challenge
in machine learning has been to develop techniques that perform well in an
incremental (streaming) setting. In this paper, we investigate the problem of
private machine learning, where as common in practice, the data is not given at
once, but rather arrives incrementally over time.
We introduce the problems of private incremental ERM and private incremental
regression where the general goal is to always maintain a good empirical risk
minimizer for the history observed under differential privacy. Our first
contribution is a generic transformation of private batch ERM mechanisms into
private incremental ERM mechanisms, based on a simple idea of invoking the
private batch ERM procedure at some regular time intervals. We take this
construction as a baseline for comparison. We then provide two mechanisms for
the private incremental regression problem. Our first mechanism is based on
privately constructing a noisy incremental gradient function, which is then
used in a modified projected gradient procedure at every timestep. This
mechanism has an excess empirical risk of , where is the
dimensionality of the data. While from the results of [Bassily et al. 2014]
this bound is tight in the worst-case, we show that certain geometric
properties of the input and constraint set can be used to derive significantly
better results for certain interesting regression problems.Comment: To appear in PODS 201
State-of-the-art in aerodynamic shape optimisation methods
Aerodynamic optimisation has become an indispensable component for any aerodynamic design over the past 60 years, with applications to aircraft, cars, trains, bridges, wind turbines, internal pipe flows, and cavities, among others, and is thus relevant in many facets of technology. With advancements in computational power, automated design optimisation procedures have become more competent, however, there is an ambiguity and bias throughout the literature with regards to relative performance of optimisation architectures and employed algorithms. This paper provides a well-balanced critical review of the dominant optimisation approaches that have been integrated with aerodynamic theory for the purpose of shape optimisation. A total of 229 papers, published in more than 120 journals and conference proceedings, have been classified into 6 different optimisation algorithm approaches. The material cited includes some of the most well-established authors and publications in the field of aerodynamic optimisation. This paper aims to eliminate bias toward certain algorithms by analysing the limitations, drawbacks, and the benefits of the most utilised optimisation approaches. This review provides comprehensive but straightforward insight for non-specialists and reference detailing the current state for specialist practitioners
Quadratically-Regularized Optimal Transport on Graphs
Optimal transportation provides a means of lifting distances between points
on a geometric domain to distances between signals over the domain, expressed
as probability distributions. On a graph, transportation problems can be used
to express challenging tasks involving matching supply to demand with minimal
shipment expense; in discrete language, these become minimum-cost network flow
problems. Regularization typically is needed to ensure uniqueness for the
linear ground distance case and to improve optimization convergence;
state-of-the-art techniques employ entropic regularization on the
transportation matrix. In this paper, we explore a quadratic alternative to
entropic regularization for transport over a graph. We theoretically analyze
the behavior of quadratically-regularized graph transport, characterizing how
regularization affects the structure of flows in the regime of small but
nonzero regularization. We further exploit elegant second-order structure in
the dual of this problem to derive an easily-implemented Newton-type
optimization algorithm.Comment: 27 page
Controlled switching in Kalman filtering and iterative learning controls
“Switching is not an uncommon phenomenon in practical systems and processes, for examples, power switches opening and closing, transmissions lifting from low gear to high gear, and air planes crossing different layers in air. Switching can be a disaster to a system since frequent switching between two asymptotically stable subsystems may result in unstable dynamics. On the contrary, switching can be a benefit to a system since controlled switching is sometimes imposed by the designers to achieve desired performance. This encourages the study of system dynamics and performance when undesired switching occurs or controlled switching is imposed. In this research, the controlled switching is applied to an estimation process and a multivariable Iterative Learning Control (ILC) system, and system stability as well as system performance under switching are investigated. The first article develops a controlled switching strategy for the estimation of a temporal shift in a Laser Tracker (LT). For some reason, the shift cannot be measured at all time. Therefore, a model-based predictor is adopted for estimation when the measurement is not available, and a Kalman Filter (KF) is used to update the estimate when the measurement is available. With the proposed method, the estimation uncertainty is always bounded within two predefined boundaries. The second article develops a controlled switching method for multivariable ILC systems where only partial outputs are measured at a time. Zero tracking error cannot be achieved for such systems using standard ILC due to incomplete knowledge of the outputs. With the developed controlled switching, all the outputs are measured in a sequential order, and, with each currently-measured output, the standard ILC is executed. Conditions under which zero convergent tracking error is accomplished with the proposed method are investigated. The proposed method is finally applied to solving a multi-agent coordination problem”--Abstract, page iv
Domain Agnostic Fourier Neural Operators
Fourier neural operators (FNOs) can learn highly nonlinear mappings between
function spaces, and have recently become a popular tool for learning responses
of complex physical systems. However, to achieve good accuracy and efficiency,
FNOs rely on the Fast Fourier transform (FFT), which is restricted to modeling
problems on rectangular domains. To lift such a restriction and permit FFT on
irregular geometries as well as topology changes, we introduce domain agnostic
Fourier neural operator (DAFNO), a novel neural operator architecture for
learning surrogates with irregular geometries and evolving domains. The key
idea is to incorporate a smoothed characteristic function in the integral layer
architecture of FNOs, and leverage FFT to achieve rapid computations, in such a
way that the geometric information is explicitly encoded in the architecture.
In our empirical evaluation, DAFNO has achieved state-of-the-art accuracy as
compared to baseline neural operator models on two benchmark datasets of
material modeling and airfoil simulation. To further demonstrate the capability
and generalizability of DAFNO in handling complex domains with topology
changes, we consider a brittle material fracture evolution problem. With only
one training crack simulation sample, DAFNO has achieved generalizability to
unseen loading scenarios and substantially different crack patterns from the
trained scenario. Our code and data accompanying this paper are available at
https://github.com/ningliu-iga/DAFNO
- …