4,957 research outputs found
Learning from Data with Heterogeneous Noise using SGD
We consider learning from data of variable quality that may be obtained from
different heterogeneous sources. Addressing learning from heterogeneous data in
its full generality is a challenging problem. In this paper, we adopt instead a
model in which data is observed through heterogeneous noise, where the noise
level reflects the quality of the data source. We study how to use stochastic
gradient algorithms to learn in this model. Our study is motivated by two
concrete examples where this problem arises naturally: learning with local
differential privacy based on data from multiple sources with different privacy
requirements, and learning from data with labels of variable quality.
The main contribution of this paper is to identify how heterogeneous noise
impacts performance. We show that given two datasets with heterogeneous noise,
the order in which to use them in standard SGD depends on the learning rate. We
propose a method for changing the learning rate as a function of the
heterogeneity, and prove new regret bounds for our method in two cases of
interest. Experiments on real data show that our method performs better than
using a single learning rate and using only the less noisy of the two datasets
when the noise level is low to moderate
Physics-informed Neural Networks for Solving Inverse Problems of Nonlinear Biot's Equations: Batch Training
In biomedical engineering, earthquake prediction, and underground energy
harvesting, it is crucial to indirectly estimate the physical properties of
porous media since the direct measurement of those are usually
impractical/prohibitive. Here we apply the physics-informed neural networks to
solve the inverse problem with regard to the nonlinear Biot's equations.
Specifically, we consider batch training and explore the effect of different
batch sizes. The results show that training with small batch sizes, i.e., a few
examples per batch, provides better approximations (lower percentage error) of
the physical parameters than using large batches or the full batch. The
increased accuracy of the physical parameters, comes at the cost of longer
training time. Specifically, we find the size should not be too small since a
very small batch size requires a very long training time without a
corresponding improvement in estimation accuracy. We find that a batch size of
8 or 32 is a good compromise, which is also robust to additive noise in the
data. The learning rate also plays an important role and should be used as a
hyperparameter.Comment: arXiv admin note: text overlap with arXiv:2002.0823
Parameter-free locally differentially private stochastic subgradient descent
https://arxiv.org/pdf/1911.09564.pdfhttps://arxiv.org/pdf/1911.09564.pdfhttps://arxiv.org/pdf/1911.09564.pdfhttps://arxiv.org/pdf/1911.09564.pdfhttps://arxiv.org/pdf/1911.09564.pdfhttps://arxiv.org/pdf/1911.09564.pdfPublished versio
Enabling Explainable Fusion in Deep Learning with Fuzzy Integral Neural Networks
Information fusion is an essential part of numerous engineering systems and
biological functions, e.g., human cognition. Fusion occurs at many levels,
ranging from the low-level combination of signals to the high-level aggregation
of heterogeneous decision-making processes. While the last decade has witnessed
an explosion of research in deep learning, fusion in neural networks has not
observed the same revolution. Specifically, most neural fusion approaches are
ad hoc, are not understood, are distributed versus localized, and/or
explainability is low (if present at all). Herein, we prove that the fuzzy
Choquet integral (ChI), a powerful nonlinear aggregation function, can be
represented as a multi-layer network, referred to hereafter as ChIMP. We also
put forth an improved ChIMP (iChIMP) that leads to a stochastic gradient
descent-based optimization in light of the exponential number of ChI inequality
constraints. An additional benefit of ChIMP/iChIMP is that it enables
eXplainable AI (XAI). Synthetic validation experiments are provided and iChIMP
is applied to the fusion of a set of heterogeneous architecture deep models in
remote sensing. We show an improvement in model accuracy and our previously
established XAI indices shed light on the quality of our data, model, and its
decisions.Comment: IEEE Transactions on Fuzzy System
- …