703 research outputs found
Exploit Where Optimizer Explores via Residuals
In order to train the neural networks faster, many efforts have been devoted
to exploring a better solution trajectory, but few have been put into
exploiting the existing solution trajectory. To exploit the trajectory of
(momentum) stochastic gradient descent (SGD(m)) method, we propose a novel
method named SGD(m) with residuals (RSGD(m)), which leads to a performance
boost of both the convergence and generalization. Our new method can also be
applied to other optimizers such as ASGD and Adam. We provide theoretical
analysis to show that RSGD achieves a smaller growth rate of the generalization
error and the same (but empirically better) convergence rate compared with SGD.
Extensive deep learning experiments on image classification, language modeling
and graph convolutional neural networks show that the proposed algorithm is
faster than SGD(m)/Adam at the initial training stage, and similar to or better
than SGD(m) at the end of training with better generalization error
Robust Deep Compressive Sensing with Recurrent-Residual Structural Constraints
Existing deep compressive sensing (CS) methods either ignore adaptive online
optimization or depend on costly iterative optimizer during reconstruction.
This work explores a novel image CS framework with recurrent-residual
structural constraint, termed as RCS-NET. The RCS-NET first
progressively optimizes the acquired samplings through a novel recurrent neural
network. The cascaded residual convolutional network then fully reconstructs
the image from optimized latent representation. As the first deep CS framework
efficiently bridging adaptive online optimization, the RCS-NET integrates
the robustness of online optimization with the efficiency and nonlinear
capacity of deep learning methods. Signal correlation has been addressed
through the network architecture. The adaptive sensing nature further makes it
an ideal candidate for color image CS via leveraging channel correlation.
Numerical experiments verify the proposed recurrent latent optimization design
not only fulfills the adaptation motivation, but also outperforms classic long
short-term memory (LSTM) architecture in the same scenario. The overall
framework demonstrates hardware implementation feasibility, with leading
robustness and generalization capability among existing deep CS benchmarks
Reinterpreting Fundamental Plane Correlations with Machine Learning
This work explores the relationships between galaxy sizes and related
observable galaxy properties in a large volume cosmological hydrodynamical
simulation. The objectives of this work are to both develop a better
understanding of the correlations between galaxy properties and the influence
of environment on galaxy physics in order to build an improved model for the
galaxy sizes, building off of the {\it fundamental plane}. With an accurate
intrinsic galaxy size predictor, the residuals in the observed galaxy sizes can
potentially be used for multiple cosmological applications, including making
measurements of galaxy velocities in spectroscopic samples, estimating the rate
of cosmic expansion, and constraining the uncertainties in the photometric
redshifts of galaxies. Using projection pursuit regression, the model
accurately predicts intrinsic galaxy sizes and have residuals which have
limited correlation with galaxy properties. The model decreases the spatial
correlation of galaxy size residuals by a factor of 5 at small scales
compared to the baseline correlation when the mean size is used as a predictor.Comment: 16 pages, 12 figures, MNRA
Aero-Structural Design Optimization of Adaptive Shock Control Bumps
Shock control bumps (SCB) are a transonic flow control device that aim to reduce the overall drag due to a normal shock on a typical passenger jet at cruise. The concept of adaptive SCB which can be deployed for best use are investigated through an aero-structural design tool that produces optimal geometries. The optimizer uses a surface based performance metric to highlight the importance of the flow quality around the SCB as well as including a structural element that is required to provide the necessary flexibility to deform. The performance metric produces the target pressure distribution and successfully smears the shock. It is found that the structural constraint does not inhibit bump height and global airfoil performance is not significantly a↵ected, L/D varies < 0.6%. The aerodynamic pressure loading can be utilised to produce a new family of SCB geometries that are unachievable with mechanical actuation alone. The study shows that adaptive SCB that exploit the naturally occurring pressure field around an airfoil in a passive way are a feasible technology to mitigate the poor o↵-design performance of static SCB
- …