180 research outputs found
Convergence Theory of Learning Over-parameterized ResNet: A Full Characterization
ResNet structure has achieved great empirical success since its debut. Recent
work established the convergence of learning over-parameterized ResNet with a
scaling factor on the residual branch where is the network
depth. However, it is not clear how learning ResNet behaves for other values of
. In this paper, we fully characterize the convergence theory of gradient
descent for learning over-parameterized ResNet with different values of .
Specifically, with hiding logarithmic factor and constant coefficients, we show
that for gradient descent is guaranteed to converge to the
global minma, and especially when the convergence is irrelevant
of the network depth. Conversely, we show that for ,
the forward output grows at least with rate in expectation and then the
learning fails because of gradient explosion for large . This means the
bound is sharp for learning ResNet with arbitrary depth.
To the best of our knowledge, this is the first work that studies learning
ResNet with full range of .Comment: 31 page
Applying an extended prototype willingness model to predict back seat safety belt use in China
The risk of injury and death in traffic accidents for passengers in the back and front seats can be reduced by utilizing safety belts. However, passengers use back seatbelts far less frequently than those in the front. More investigation is therefore required into the psychological constructs that affect individuals\u27 attitudes toward using back seat belts. In this study, four models were used to analyze individual intentions and actual back seat belt use: the standard theory of planned behavior (TPB); the standard prototype willingness model (PWM); a model that integrates the TPB and PWM constructs; and a model that integrates the TPB construct, PWM constructs, descriptive norms and perceived law enforcement. The results showed that the standard PWM has much more explanatory power than the standard TPB in explaining the variance in behavioral intention and behavior. Incorporating perceived behavioral control (PBC) into the standard PWM did not improve the model fit considerably, while incorporating descriptive norms and perceived law enforcement moderately improved the model fit. Attitude greatly impacted behavioral intention and the use of back seat belts, followed by perceived law enforcement and descriptive norms, while subjective norms, prototype favorability, prototype similarity and PBC had no significant effect
Height estimation from single aerial images using a deep ordinal regression network
Understanding the 3D geometric structure of the Earth's surface has been an
active research topic in photogrammetry and remote sensing community for
decades, serving as an essential building block for various applications such
as 3D digital city modeling, change detection, and city management. Previous
researches have extensively studied the problem of height estimation from
aerial images based on stereo or multi-view image matching. These methods
require two or more images from different perspectives to reconstruct 3D
coordinates with camera information provided. In this paper, we deal with the
ambiguous and unsolved problem of height estimation from a single aerial image.
Driven by the great success of deep learning, especially deep convolution
neural networks (CNNs), some researches have proposed to estimate height
information from a single aerial image by training a deep CNN model with
large-scale annotated datasets. These methods treat height estimation as a
regression problem and directly use an encoder-decoder network to regress the
height values. In this paper, we proposed to divide height values into
spacing-increasing intervals and transform the regression problem into an
ordinal regression problem, using an ordinal loss for network training. To
enable multi-scale feature extraction, we further incorporate an Atrous Spatial
Pyramid Pooling (ASPP) module to extract features from multiple dilated
convolution layers. After that, a post-processing technique is designed to
transform the predicted height map of each patch into a seamless height map.
Finally, we conduct extensive experiments on ISPRS Vaihingen and Potsdam
datasets. Experimental results demonstrate significantly better performance of
our method compared to the state-of-the-art methods.Comment: 5 pages, 3 figure
A Full Characterization of Excess Risk via Empirical Risk Landscape
In this paper, we provide a unified analysis of the excess risk of the model
trained by a proper algorithm with both smooth convex and non-convex loss
functions. In contrast to the existing bounds in the literature that depends on
iteration steps, our bounds to the excess risk do not diverge with the number
of iterations. This underscores that, at least for smooth loss functions, the
excess risk can be guaranteed after training. To get the bounds to excess risk,
we develop a technique based on algorithmic stability and non-asymptotic
characterization of the empirical risk landscape. The model obtained by a
proper algorithm is proved to generalize with this technique. Specifically, for
non-convex loss, the conclusion is obtained via the technique and analyzing the
stability of a constructed auxiliary algorithm. Combining this with some
properties of the empirical risk landscape, we derive converged upper bounds to
the excess risk in both convex and non-convex regime with the help of some
classical optimization results.Comment: 38page
Towards Accelerating Training of Batch Normalization: A Manifold Perspective
Batch normalization (BN) has become a crucial component across diverse deep
neural networks. The network with BN is invariant to positively linear
re-scaling of weights, which makes there exist infinite functionally equivalent
networks with various scales of weights. However, optimizing these equivalent
networks with the first-order method such as stochastic gradient descent will
converge to different local optima owing to different gradients across
training. To alleviate this, we propose a quotient manifold \emph{PSI
manifold}, in which all the equivalent weights of the network with BN are
regarded as the same one element. Then, gradient descent and stochastic
gradient descent on the PSI manifold are also constructed. The two algorithms
guarantee that every group of equivalent weights (caused by positively
re-scaling) converge to the equivalent optima. Besides that, we give the
convergence rate of the proposed algorithms on PSI manifold and justify that
they accelerate training compared with the algorithms on the Euclidean weight
space. Empirical studies show that our algorithms can consistently achieve
better performances over various experimental settings
Improved OOD Generalization via Conditional Invariant Regularizer
Recently, generalization on out-of-distribution (OOD) data with correlation
shift has attracted great attention. The correlation shift is caused by the
spurious attributes that correlate to the class label, as the correlation
between them may vary in training and test data. For such a problem, we show
that given the class label, the conditionally independent models of spurious
attributes are OOD generalizable. Based on this, a metric Conditional Spurious
Variation (CSV) which controls OOD generalization error, is proposed to measure
such conditional independence. To improve the OOD generalization, we regularize
the training process with the proposed CSV. Under mild assumptions, our
training objective can be formulated as a nonconvex-concave mini-max problem.
An algorithm with provable convergence rate is proposed to solve the problem.
Extensive empirical results verify our algorithm's efficacy in improving OOD
generalization
- …