72 research outputs found
Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts
Multivariate probabilistic time series forecasts are commonly evaluated via
proper scoring rules, i.e., functions that are minimal in expectation for the
ground-truth distribution. However, this property is not sufficient to
guarantee good discrimination in the non-asymptotic regime. In this paper, we
provide the first systematic finite-sample study of proper scoring rules for
time-series forecasting evaluation. Through a power analysis, we identify the
"region of reliability" of a scoring rule, i.e., the set of practical
conditions where it can be relied on to identify forecasting errors. We carry
out our analysis on a comprehensive synthetic benchmark, specifically designed
to test several key discrepancies between ground-truth and forecast
distributions, and we gauge the generalizability of our findings to real-world
tasks with an application to an electricity production problem. Our results
reveal critical shortcomings in the evaluation of multivariate probabilistic
forecasts as commonly performed in the literature.Comment: 37 pages, 28 figure
Invariant Causal Set Covering Machines
Rule-based models, such as decision trees, appeal to practitioners due to
their interpretable nature. However, the learning algorithms that produce such
models are often vulnerable to spurious associations and thus, they are not
guaranteed to extract causally-relevant insights. In this work, we build on
ideas from the invariant causal prediction literature to propose Invariant
Causal Set Covering Machines, an extension of the classical Set Covering
Machine algorithm for conjunctions/disjunctions of binary-valued rules that
provably avoids spurious associations. We demonstrate both theoretically and
empirically that our method can identify the causal parents of a variable of
interest in polynomial time
Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation
The practical utility of causality in decision-making is widely recognized,
with causal discovery and inference being inherently intertwined. Nevertheless,
a notable gap exists in the evaluation of causal discovery methods, where
insufficient emphasis is placed on downstream inference. To address this gap,
we evaluate six established baseline causal discovery methods and a newly
proposed method based on GFlowNets, on the downstream task of treatment effect
estimation. Through the implementation of a robust evaluation procedure, we
offer valuable insights into the efficacy of these causal discovery methods for
treatment effect estimation, considering both synthetic and real-world
scenarios, as well as low-data scenarios. Furthermore, the results of our study
demonstrate that GFlowNets possess the capability to effectively capture a wide
range of useful and diverse ATE modes.Comment: Peer-Reviewed and Accepted to ICML 2023 Workshop on Structured
Probabilistic Inference & Generative Modelin
Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning
In recent years, deep learning algorithms have become increasingly more
prominent for their unparalleled ability to automatically learn discriminant
features from large amounts of data. However, within the field of
electromyography-based gesture recognition, deep learning algorithms are seldom
employed as they require an unreasonable amount of effort from a single person,
to generate tens of thousands of examples.
This work's hypothesis is that general, informative features can be learned
from the large amounts of data generated by aggregating the signals of multiple
users, thus reducing the recording burden while enhancing gesture recognition.
Consequently, this paper proposes applying transfer learning on aggregated data
from multiple users, while leveraging the capacity of deep learning algorithms
to learn discriminant features from large datasets. Two datasets comprised of
19 and 17 able-bodied participants respectively (the first one is employed for
pre-training) were recorded for this work, using the Myo Armband. A third Myo
Armband dataset was taken from the NinaPro database and is comprised of 10
able-bodied participants. Three different deep learning networks employing
three different modalities as input (raw EMG, Spectrograms and Continuous
Wavelet Transform (CWT)) are tested on the second and third dataset. The
proposed transfer learning scheme is shown to systematically and significantly
enhance the performance for all three networks on the two datasets, achieving
an offline accuracy of 98.31% for 7 gestures over 17 participants for the
CWT-based ConvNet and 68.98% for 18 gestures over 10 participants for the raw
EMG-based ConvNet. Finally, a use-case study employing eight able-bodied
participants suggests that real-time feedback allows users to adapt their
muscle activation strategy which reduces the degradation in accuracy normally
experienced over time.Comment: Source code and datasets available:
https://github.com/Giguelingueling/MyoArmbandDatase
- …