13,145 research outputs found
A non-parametric conditional factor regression model for high-dimensional input and response
In this paper, we propose a non-parametric conditional factor regression
(NCFR)model for domains with high-dimensional input and response. NCFR enhances
linear regression in two ways: a) introducing low-dimensional latent factors
leading to dimensionality reduction and b) integrating an Indian Buffet Process
as a prior for the latent factors to derive unlimited sparse dimensions.
Experimental results comparing NCRF to several alternatives give evidence to
remarkable prediction performance.Comment: 9 pages, 3 figures, NIPS submissio
Assessment of the lognormality assumption of seismic fragility curves using non-parametric representations
Fragility curves are commonly used in civil engineering to estimate the
vulnerability of structures to earthquakes. The probability of failure
associated with a failure criterion (e.g. the maximal inter-storey drift ratio
being greater than a prescribed threshold) is represented as a function of the
intensity of the earthquake ground motion (e.g. peak ground acceleration or
spectral acceleration). The classical approach consists in assuming a lognormal
shape of the fragility curves. In this paper, we introduce two non-parametric
approaches to establish the fragility curves without making any assumption,
namely the conditional Monte Carlo simulation and the kernel density
estimation. As an illustration, we compute the fragility curves of a 3-storey
steel structure, accounting for the nonlinear behavior of the system. The
curves obtained by the proposed approaches are compared with each other and
with those obtained using the classical lognormal assumption.Comment: 17 pages, 10 figures, submitted to Structural Safety journa
Stochastic Multi-objective Optimization on a Budget: Application to multi-pass wire drawing with quantified uncertainties
Design optimization of engineering systems with multiple competing objectives
is a painstakingly tedious process especially when the objective functions are
expensive-to-evaluate computer codes with parametric uncertainties. The
effectiveness of the state-of-the-art techniques is greatly diminished because
they require a large number of objective evaluations, which makes them
impractical for problems of the above kind. Bayesian global optimization (BGO),
has managed to deal with these challenges in solving single-objective
optimization problems and has recently been extended to multi-objective
optimization (MOO). BGO models the objectives via probabilistic surrogates and
uses the epistemic uncertainty to define an information acquisition function
(IAF) that quantifies the merit of evaluating the objective at new designs.
This iterative data acquisition process continues until a stopping criterion is
met. The most commonly used IAF for MOO is the expected improvement over the
dominated hypervolume (EIHV) which in its original form is unable to deal with
parametric uncertainties or measurement noise. In this work, we provide a
systematic reformulation of EIHV to deal with stochastic MOO problems. The
primary contribution of this paper lies in being able to filter out the noise
and reformulate the EIHV without having to observe or estimate the stochastic
parameters. An addendum of the probabilistic nature of our methodology is that
it enables us to characterize our confidence about the predicted Pareto front.
We verify and validate the proposed methodology by applying it to synthetic
test problems with known solutions. We demonstrate our approach on an
industrial problem of die pass design for a steel wire drawing process.Comment: 19 pages, 14 figure
Breiman's "Two Cultures" Revisited and Reconciled
In a landmark paper published in 2001, Leo Breiman described the tense
standoff between two cultures of data modeling: parametric statistical and
algorithmic machine learning. The cultural division between these two
statistical learning frameworks has been growing at a steady pace in recent
years. What is the way forward? It has become blatantly obvious that this
widening gap between "the two cultures" cannot be averted unless we find a way
to blend them into a coherent whole. This article presents a solution by
establishing a link between the two cultures. Through examples, we describe the
challenges and potential gains of this new integrated statistical thinking.Comment: This paper celebrates the 70th anniversary of Statistical Machine
Learning--- how far we've come, and how far we have to go. Keywords:
Integrated statistical learning theory, Exploratory machine learning,
Uncertainty prediction machine, ML-powered modern applied statistics,
Information theor
When Gaussian Process Meets Big Data: A Review of Scalable GPs
The vast quantity of information brought by big data as well as the evolving
computer hardware encourages success stories in the machine learning community.
In the meanwhile, it poses challenges for the Gaussian process (GP) regression,
a well-known non-parametric and interpretable Bayesian model, which suffers
from cubic complexity to data size. To improve the scalability while retaining
desirable prediction quality, a variety of scalable GPs have been presented.
But they have not yet been comprehensively reviewed and analyzed in order to be
well understood by both academia and industry. The review of scalable GPs in
the GP community is timely and important due to the explosion of data size. To
this end, this paper is devoted to the review on state-of-the-art scalable GPs
involving two main categories: global approximations which distillate the
entire data and local approximations which divide the data for subspace
learning. Particularly, for global approximations, we mainly focus on sparse
approximations comprising prior approximations which modify the prior but
perform exact inference, posterior approximations which retain exact prior but
perform approximate inference, and structured sparse approximations which
exploit specific structures in kernel matrix; for local approximations, we
highlight the mixture/product of experts that conducts model averaging from
multiple local experts to boost predictions. To present a complete review,
recent advances for improving the scalability and capability of scalable GPs
are reviewed. Finally, the extensions and open issues regarding the
implementation of scalable GPs in various scenarios are reviewed and discussed
to inspire novel ideas for future research avenues.Comment: 20 pages, 6 figure
Neural Likelihoods via Cumulative Distribution Functions
We leverage neural networks as universal approximators of monotonic functions
to build a parameterization of conditional cumulative distribution functions
(CDFs). By the application of automatic differentiation with respect to
response variables and then to parameters of this CDF representation, we are
able to build black box CDF and density estimators. A suite of families is
introduced as alternative constructions for the multivariate case. At one
extreme, the simplest construction is a competitive density estimator against
state-of-the-art deep learning methods, although it does not provide an easily
computable representation of multivariate CDFs. At the other extreme, we have a
flexible construction from which multivariate CDF evaluations and
marginalizations can be obtained by a simple forward pass in a deep neural net,
but where the computation of the likelihood scales exponentially with
dimensionality. Alternatives in between the extremes are discussed. We evaluate
the different representations empirically on a variety of tasks involving tail
area probabilities, tail dependence and (partial) density estimation.Comment: 10 page
Estimation Considerations in Contextual Bandits
Contextual bandit algorithms are sensitive to the estimation method of the
outcome model as well as the exploration method used, particularly in the
presence of rich heterogeneity or complex outcome models, which can lead to
difficult estimation problems along the path of learning. We study a
consideration for the exploration vs. exploitation framework that does not
arise in multi-armed bandits but is crucial in contextual bandits; the way
exploration and exploitation is conducted in the present affects the bias and
variance in the potential outcome model estimation in subsequent stages of
learning. We develop parametric and non-parametric contextual bandits that
integrate balancing methods from the causal inference literature in their
estimation to make it less prone to problems of estimation bias. We provide the
first regret bound analyses for contextual bandits with balancing in the domain
of linear contextual bandits that match the state of the art regret bounds. We
demonstrate the strong practical advantage of balanced contextual bandits on a
large number of supervised learning datasets and on a synthetic example that
simulates model mis-specification and prejudice in the initial training data.
Additionally, we develop contextual bandits with simpler assignment policies by
leveraging sparse model estimation methods from the econometrics literature and
demonstrate empirically that in the early stages they can improve the rate of
learning and decrease regret
A Contemporary Overview of Probabilistic Latent Variable Models
In this paper we provide a conceptual overview of latent variable models
within a probabilistic modeling framework, an overview that emphasizes the
compositional nature and the interconnectedness of the seemingly disparate
models commonly encountered in statistical practice
Sequential Design for Optimal Stopping Problems
We propose a new approach to solve optimal stopping problems via simulation.
Working within the backward dynamic programming/Snell envelope framework, we
augment the methodology of Longstaff-Schwartz that focuses on approximating the
stopping strategy. Namely, we introduce adaptive generation of the stochastic
grids anchoring the simulated sample paths of the underlying state process.
This allows for active learning of the classifiers partitioning the state space
into the continuation and stopping regions. To this end, we examine sequential
design schemes that adaptively place new design points close to the stopping
boundaries. We then discuss dynamic regression algorithms that can implement
such recursive estimation and local refinement of the classifiers. The new
algorithm is illustrated with a variety of numerical experiments, showing that
an order of magnitude savings in terms of design size can be achieved. We also
compare with existing benchmarks in the context of pricing multi-dimensional
Bermudan options.Comment: 24 page
- …