8 research outputs found
Regularization in neural network optimization via trimmed stochastic gradient descent with noisy label
Regularization is essential for avoiding over-fitting to training data in
neural network optimization, leading to better generalization of the trained
networks. The label noise provides a strong implicit regularization by
replacing the target ground truth labels of training examples by uniform random
labels. However, it may also cause undesirable misleading gradients due to the
large loss associated with incorrect labels. We propose a first-order
optimization method (Label-Noised Trim-SGD) which combines the label noise with
the example trimming in order to remove the outliers. The proposed algorithm
enables us to impose a large label noise and obtain a better regularization
effect than the original methods. The quantitative analysis is performed by
comparing the behavior of the label noise, the example trimming, and the
proposed algorithm. We also present empirical results that demonstrate the
effectiveness of our algorithm using the major benchmarks and the fundamental
networks, where our method has successfully outperformed the state-of-the-art
optimization methods
Learning Nanoscale Motion Patterns of Vesicles in Living Cells
Detecting and analyzing nanoscale motion patterns of vesicles, smaller than the microscope resolution (~250 nm), inside living biological cells is a challenging problem. State-of-the-art CV approaches based on detection, tracking, optical flow or deep learning perform poorly for this problem. We propose an integrative approach, built upon physics based simulations, nanoscopy algorithms, and shallow residual attention network to make it possible for the first time to analysis sub-resolution motion patterns in vesicles that may also be of sub-resolution diameter. Our results show state-of-the-art performance, 89% validation accuracy on simulated dataset and 82% testing accuracy on an experimental dataset of living heart muscle cells imaged under three different pathological conditions. We demonstrate automated analysis of the motion states and changed in them for over 9000 vesicles. Such analysis will enable large scale biological studies of vesicle transport and interaction in living cells in the future
Bayesian Optimization for Probabilistic Programs
We present the first general purpose framework for marginal maximum a
posteriori estimation of probabilistic program variables. By using a series of
code transformations, the evidence of any probabilistic program, and therefore
of any graphical model, can be optimized with respect to an arbitrary subset of
its sampled variables. To carry out this optimization, we develop the first
Bayesian optimization package to directly exploit the source code of its
target, leading to innovations in problem-independent hyperpriors, unbounded
optimization, and implicit constraint satisfaction; delivering significant
performance improvements over prominent existing packages. We present
applications of our method to a number of tasks including engineering design
and parameter optimization
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
Machine learning algorithms have been used widely in various applications and
areas. To fit a machine learning model into different problems, its
hyper-parameters must be tuned. Selecting the best hyper-parameter
configuration for machine learning models has a direct impact on the model's
performance. It often requires deep knowledge of machine learning algorithms
and appropriate hyper-parameter optimization techniques. Although several
automatic optimization techniques exist, they have different strengths and
drawbacks when applied to different types of problems. In this paper,
optimizing the hyper-parameters of common machine learning models is studied.
We introduce several state-of-the-art optimization techniques and discuss how
to apply them to machine learning algorithms. Many available libraries and
frameworks developed for hyper-parameter optimization problems are provided,
and some open challenges of hyper-parameter optimization research are also
discussed in this paper. Moreover, experiments are conducted on benchmark
datasets to compare the performance of different optimization methods and
provide practical examples of hyper-parameter optimization. This survey paper
will help industrial users, data analysts, and researchers to better develop
machine learning models by identifying the proper hyper-parameter
configurations effectively.Comment: 69 Pages, 10 tables, accepted in Neurocomputing, Elsevier. Github
link:
https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithm