10,248 research outputs found
The Lov\'asz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks
The Jaccard index, also referred to as the intersection-over-union score, is
commonly employed in the evaluation of image segmentation results given its
perceptual qualities, scale invariance - which lends appropriate relevance to
small objects, and appropriate counting of false negatives, in comparison to
per-pixel losses. We present a method for direct optimization of the mean
intersection-over-union loss in neural networks, in the context of semantic
image segmentation, based on the convex Lov\'asz extension of submodular
losses. The loss is shown to perform better with respect to the Jaccard index
measure than the traditionally used cross-entropy loss. We show quantitative
and qualitative differences between optimizing the Jaccard index per image
versus optimizing the Jaccard index taken over an entire dataset. We evaluate
the impact of our method in a semantic segmentation pipeline and show
substantially improved intersection-over-union segmentation scores on the
Pascal VOC and Cityscapes datasets using state-of-the-art deep learning
segmentation architectures.Comment: Accepted as a conference paper at CVPR 201
Numerical Investigation and Optimization of a Flushwall Injector for Scramjet Applications at Hypervelocity Flow Conditions
An investigation utilizing Reynolds-averaged simulations (RAS) was performed in order to find optimal designs for an interdigitated flushwall injector suitable for scramjet applications at hypervelocity conditions. The flight Mach number, duct height, spanwise width, and injection angle were the design variables selected to maximize two objective functions: the thrust potential and combustion efficiency. A Latin hypercube sampling design-of-experiments method was used to select design points for RAS. A methodology was developed that automated building geometries and generating grids for each design. The ensuing RAS analysis generated the performance database from which the two objective functions of interest were computed using a one-dimensional performance utility. The data were fitted using four surrogate models: an artificial neural network (ANN) model, a cubic polynomial, a quadratic polynomial, and a Kriging model. Variance-based decomposition showed that both objective functions were primarily driven by changes in the duct height. Multiobjective design optimization was performed for all four surrogate models via a genetic algorithm method. Optimal solutions were obtained at the upper and lower bounds of the flight Mach number range. The Kriging model obtained an optimal solution set that predicted high values for both objective functions. Additionally, three challenge points were selected to assess the designs on the Pareto fronts. Further sampling among the designs of the Pareto fronts are required in order to lower the errors and perform more accurate surrogate-based optimization.
sed optimization
Numerical Investigation and Optimization of a Flushwall Injector for Scramjet Applications at Hypervelocity Flow Conditions
An investigation utilizing Reynolds-averaged simulations (RAS) was performed in order to demonstrate the use of design and analysis of computer experiments (DACE) methods in Sandias DAKOTA software package for surrogate modeling and optimization. These methods were applied to a flow- path fueled with an interdigitated flushwall injector suitable for scramjet applications at hyper- velocity conditions and ascending along a constant dynamic pressure flight trajectory. The flight Mach number, duct height, spanwise width, and injection angle were the design variables selected to maximize two objective functions: the thrust potential and combustion efficiency. Because the RAS of this case are computationally expensive, surrogate models are used for optimization. To build a surrogate model a RAS database is created. The sequence of the design variables comprising the database were generated using a Latin hypercube sampling (LHS) method. A methodology was also developed to automatically build geometries and generate structured grids for each design point. The ensuing RAS analysis generated the simulation database from which the two objective functions were computed using a one-dimensionalization (1D) of the three-dimensional simulation data. The data were fitted using four surrogate models: an artificial neural network (ANN), a cubic polynomial, a quadratic polynomial, and a Kriging model. Variance-based decomposition showed that both objective functions were primarily driven by changes in the duct height. Multiobjective design optimization was performed for all four surrogate models via a genetic algorithm method. Optimal solutions were obtained at the upper and lower bounds of the flight Mach number range. The Kriging model predicted an optimal solution set that exhibited high values for both objective functions. Additionally, three challenge points were selected to assess the designs on the Pareto fronts. Further sampling among the designs of the Pareto fronts may be required to lower the surrogate model errors and perform more accurate surrogate-model-based optimization
Adaptive Simulation-based Training of AI Decision-makers using Bayesian Optimization
This work studies how an AI-controlled dog-fighting agent with tunable
decision-making parameters can learn to optimize performance against an
intelligent adversary, as measured by a stochastic objective function evaluated
on simulated combat engagements. Gaussian process Bayesian optimization (GPBO)
techniques are developed to automatically learn global Gaussian Process (GP)
surrogate models, which provide statistical performance predictions in both
explored and unexplored areas of the parameter space. This allows a learning
engine to sample full-combat simulations at parameter values that are most
likely to optimize performance and also provide highly informative data points
for improving future predictions. However, standard GPBO methods do not provide
a reliable surrogate model for the highly volatile objective functions found in
aerial combat, and thus do not reliably identify global maxima. These issues
are addressed by novel Repeat Sampling (RS) and Hybrid Repeat/Multi-point
Sampling (HRMS) techniques. Simulation studies show that HRMS improves the
accuracy of GP surrogate models, allowing AI decision-makers to more accurately
predict performance and efficiently tune parameters.Comment: submitted to JAIS for revie
Design of Computer Experiments for Optimization, Estimation of Function Contours, and Related Objectives
A computer code or simulator is a mathematical representation of a physical
system, for example a set of differential equations. Running the code with
given values of the vector of inputs, x, leads to an output y(x) or several
such outputs. For instance, one application we use for illustration simulates
the average tidal power, y, generated as a function of the turbine location, x
= (x1, x2), in the Bay of Fundy, Nova Scotia, Canada (Ranjan et al. 2011).
Performing scientific or engineering experiments via such a computer code is
often more time and cost effective than running a physical experiment.
Choosing new runs sequentially for optimization, moving y to a target, etc.
has been formalized using the concept of expected improvement (Jones et al.
1998). The next experimental run is made where the expected improvement in the
function of interest is largest. This expectation is with respect to the
predictive distribution of y from a statistical model relating y to x. By
considering a set of possible inputs x for the new run, we can choose that
which gives the largest expectation.Comment: 14 pages, 3 figures. in Chapter 7 - Statistics in Action: A Canadian
Outlook (ISBN 9781482236231 - CAT# K23109), Edited by Jerald F . Lawless
Chapman and Hall/CRC, 201
NNVA: Neural Network Assisted Visual Analysis of Yeast Cell Polarization Simulation
Complex computational models are often designed to simulate real-world
physical phenomena in many scientific disciplines. However, these simulation
models tend to be computationally very expensive and involve a large number of
simulation input parameters which need to be analyzed and properly calibrated
before the models can be applied for real scientific studies. We propose a
visual analysis system to facilitate interactive exploratory analysis of
high-dimensional input parameter space for a complex yeast cell polarization
simulation. The proposed system can assist the computational biologists, who
designed the simulation model, to visually calibrate the input parameters by
modifying the parameter values and immediately visualizing the predicted
simulation outcome without having the need to run the original expensive
simulation for every instance. Our proposed visual analysis system is driven by
a trained neural network-based surrogate model as the backend analysis
framework. Surrogate models are widely used in the field of simulation sciences
to efficiently analyze computationally expensive simulation models. In this
work, we demonstrate the advantage of using neural networks as surrogate models
for visual analysis by incorporating some of the recent advances in the field
of uncertainty quantification, interpretability and explainability of neural
network-based models. We utilize the trained network to perform interactive
parameter sensitivity analysis of the original simulation at multiple
levels-of-detail as well as recommend optimal parameter configurations using
the activation maximization framework of neural networks. We also facilitate
detail analysis of the trained network to extract useful insights about the
simulation model, learned by the network, during the training process.Comment: Published at IEEE Transactions on Visualization and Computer Graphic
Multi-Information Source Optimization
We consider Bayesian optimization of an expensive-to-evaluate black-box
objective function, where we also have access to cheaper approximations of the
objective. In general, such approximations arise in applications such as
reinforcement learning, engineering, and the natural sciences, and are subject
to an inherent, unknown bias. This model discrepancy is caused by an inadequate
internal model that deviates from reality and can vary over the domain, making
the utilization of these approximations a non-trivial task.
We present a novel algorithm that provides a rigorous mathematical treatment
of the uncertainties arising from model discrepancies and noisy observations.
Its optimization decisions rely on a value of information analysis that extends
the Knowledge Gradient factor to the setting of multiple information sources
that vary in cost: each sampling decision maximizes the predicted benefit per
unit cost.
We conduct an experimental evaluation that demonstrates that the method
consistently outperforms other state-of-the-art techniques: it finds designs of
considerably higher objective value and additionally inflicts less cost in the
exploration process.Comment: Added: benchmark logistic regression on MNIST/USPS, comparison to
MTBO/entropy search, estimation of hyper-parameter
Building accurate radio environment maps from multi-fidelity spectrum sensing data
In cognitive wireless networks, active monitoring of the wireless environment is often performed through advanced spectrum sensing and network sniffing. This leads to a set of spatially distributed measurements which are collected from different sensing devices. Nowadays, several interpolation methods (e.g., Kriging) are available and can be used to combine these measurements into a single globally accurate radio environment map that covers a certain geographical area. However, the calibration of multi-fidelity measurements from heterogeneous sensing devices, and the integration into a map is a challenging problem. In this paper, the auto-regressive co-Kriging model is proposed as a novel solution. The algorithm is applied to model measurements which are collected in a heterogeneous wireless testbed environment, and the effectiveness of the new methodology is validated
Finding Faster Configurations using FLASH
Finding good configurations for a software system is often challenging since
the number of configuration options can be large. Software engineers often make
poor choices about configuration or, even worse, they usually use a sub-optimal
configuration in production, which leads to inadequate performance. To assist
engineers in finding the (near) optimal configuration, this paper introduces
FLASH, a sequential model-based method, which sequentially explores the
configuration space by reflecting on the configurations evaluated so far to
determine the next best configuration to explore. FLASH scales up to software
systems that defeat the prior state of the art model-based methods in this
area. FLASH runs much faster than existing methods and can solve both
single-objective and multi-objective optimization problems. The central insight
of this paper is to use the prior knowledge (gained from prior runs) to choose
the next promising configuration. This strategy reduces the effort (i.e.,
number of measurements) required to find the (near) optimal configuration. We
evaluate FLASH using 30 scenarios based on 7 software systems to demonstrate
that FLASH saves effort in 100% and 80% of cases in single-objective and
multi-objective problems respectively by up to several orders of magnitude
compared to the state of the art techniques
- …