84,913 research outputs found
Connections Between Adaptive Control and Optimization in Machine Learning
This paper demonstrates many immediate connections between adaptive control
and optimization methods commonly employed in machine learning. Starting from
common output error formulations, similarities in update law modifications are
examined. Concepts in stability, performance, and learning, common to both
fields are then discussed. Building on the similarities in update laws and
common concepts, new intersections and opportunities for improved algorithm
analysis are provided. In particular, a specific problem related to higher
order learning is solved through insights obtained from these intersections.Comment: 18 page
Recommended from our members
Generalized Stochastic Gradient Learning
We study the properties of generalized stochastic gradient (GSG) learning in forwardlooking models. We examine how the conditions for stability of standard stochastic gradient (SG) learning both di1er from and are related to E-stability, which governs stability under least squares learning. SG algorithms are sensitive to units of measurement and we show that there is a transformation of variables for which E-stability governs SG stability. GSG algorithms with constant gain have a deeper justification in terms of parameter drift, robustness and risk sensitivity
Parameter Estimation of Sigmoid Superpositions: Dynamical System Approach
Superposition of sigmoid function over a finite time interval is shown to be
equivalent to the linear combination of the solutions of a linearly
parameterized system of logistic differential equations. Due to the linearity
with respect to the parameters of the system, it is possible to design an
effective procedure for parameter adjustment. Stability properties of this
procedure are analyzed. Strategies shown in earlier studies to facilitate
learning such as randomization of a learning sequence and adding specially
designed disturbances during the learning phase are requirements for
guaranteeing convergence in the learning scheme proposed.Comment: 30 pages, 7 figure
- …