17,203 research outputs found
Locality in Network Optimization
In probability theory and statistics notions of correlation among random
variables, decay of correlation, and bias-variance trade-off are fundamental.
In this work we introduce analogous notions in optimization, and we show their
usefulness in a concrete setting. We propose a general notion of correlation
among variables in optimization procedures that is based on the sensitivity of
optimal points upon (possibly finite) perturbations. We present a canonical
instance in network optimization (the min-cost network flow problem) that
exhibits locality, i.e., a setting where the correlation decays as a function
of the graph-theoretical distance in the network. In the case of warm-start
reoptimization, we develop a general approach to localize a given optimization
routine in order to exploit locality. We show that the localization mechanism
is responsible for introducing a bias in the original algorithm, and that the
bias-variance trade-off that emerges can be exploited to minimize the
computational complexity required to reach a prescribed level of error
accuracy. We provide numerical evidence to support our claims
Adaptive Momentum for Neural Network Optimization
In this thesis, we develop a novel and efficient algorithm for optimizing neural networks inspired by a recently proposed geodesic optimization algorithm. Our algorithm, which we call Stochastic Geodesic Optimization (SGeO), utilizes an adaptive coefficient on top of Polyaks Heavy Ball method effectively controlling the amount of weight put on the previous update to the parameters based on the change of direction in the optimization path. Experimental results on strongly convex functions with Lipschitz gradients and deep Autoencoder benchmarks show that SGeO reaches lower errors than established first-order methods and competes well with lower or similar errors to a recent second-order method called K-FAC (Kronecker-Factored Approximate Curvature). We also incorporate Nesterov style lookahead gradient into our algorithm (SGeO-N) and observe notable improvements. We believe that our research will open up new directions for high-dimensional neural network optimization where combining the efficiency of first-order methods and the effectiveness of second-order methods proves a promising avenue to explore
Distributed Stochastic Optimization over Time-Varying Noisy Network
This paper is concerned with distributed stochastic multi-agent optimization
problem over a class of time-varying network with slowly decreasing
communication noise effects. This paper considers the problem in composite
optimization setting which is more general in noisy network optimization. It is
noteworthy that existing methods for noisy network optimization are Euclidean
projection based. We present two related different classes of non-Euclidean
methods and investigate their convergence behavior. One is distributed
stochastic composite mirror descent type method (DSCMD-N) which provides a more
general algorithm framework than former works in this literature. As a
counterpart, we also consider a composite dual averaging type method (DSCDA-N)
for noisy network optimization. Some main error bounds for DSCMD-N and DSCDA-N
are obtained. The trade-off among stepsizes, noise decreasing rates,
convergence rates of algorithm is analyzed in detail. To the best of our
knowledge, this is the first work to analyze and derive convergence rates of
optimization algorithm in noisy network optimization. We show that an optimal
rate of in nonsmooth convex optimization can be obtained for
proposed methods under appropriate communication noise condition. Moveover,
convergence rates in different orders are comprehensively derived in both
expectation convergence and high probability convergence sense.Comment: 27 page
Route Swarm: Wireless Network Optimization through Mobility
In this paper, we demonstrate a novel hybrid architecture for coordinating
networked robots in sensing and information routing applications. The proposed
INformation and Sensing driven PhysIcally REconfigurable robotic network
(INSPIRE), consists of a Physical Control Plane (PCP) which commands agent
position, and an Information Control Plane (ICP) which regulates information
flow towards communication/sensing objectives. We describe an instantiation
where a mobile robotic network is dynamically reconfigured to ensure high
quality routes between static wireless nodes, which act as source/destination
pairs for information flow. The ICP commands the robots towards evenly
distributed inter-flow allocations, with intra-flow configurations that
maximize route quality. The PCP then guides the robots via potential-based
control to reconfigure according to ICP commands. This formulation, deemed
Route Swarm, decouples information flow and physical control, generating a
feedback between routing and sensing needs and robotic configuration. We
demonstrate our propositions through simulation under a realistic wireless
network regime.Comment: 9 pages, 4 figures, submitted to the IEEE International Conference on
Intelligent Robots and Systems (IROS) 201
- …