3 research outputs found
XTSC-Bench: Quantitative Benchmarking for Explainers on Time Series Classification
Despite the growing body of work on explainable machine learning in time
series classification (TSC), it remains unclear how to evaluate different
explainability methods. Resorting to qualitative assessment and user studies to
evaluate explainers for TSC is difficult since humans have difficulties
understanding the underlying information contained in time series data.
Therefore, a systematic review and quantitative comparison of explanation
methods to confirm their correctness becomes crucial. While steps to
standardized evaluations were taken for tabular, image, and textual data,
benchmarking explainability methods on time series is challenging due to a)
traditional metrics not being directly applicable, b) implementation and
adaption of traditional metrics for time series in the literature vary, and c)
varying baseline implementations. This paper proposes XTSC-Bench, a
benchmarking tool providing standardized datasets, models, and metrics for
evaluating explanation methods on TSC. We analyze 3 perturbation-, 6 gradient-
and 2 example-based explanation methods to TSC showing that improvements in the
explainers' robustness and reliability are necessary, especially for
multivariate data.Comment: Accepted at ICMLA 202
Abschlussbericht des Forschungsprojekts "Broker fĂŒr Dynamische Produktionsnetzwerke"
Der Broker fĂŒr dynamische Produktionsnetzwerke (DPNB) ist ein vom Bundesministerium fĂŒr Bildung und Forschung (BMBF) gefördertes und durch den ProjekttrĂ€ger Karlsruhe (PTKA) betreutes Forschungsprojekt zwischen sieben Partnern aus Wissenschaft und Wirtschaft mit einer Laufzeit von Januar 2019 bis einschlieĂlich Dezember 2021. Ăber den Einsatz von Cloud Manufacturing sowie Hard- und Software-Komponenten bei den teilnehmenden Unternehmen, sollen KapazitĂ€tsanbieter mit KapazitĂ€tsnachfrager verbunden werden. Handelbare KapazitĂ€ten sind in diesem Falle Maschinen-, sowie Transport- und MontagekapazitĂ€ten, um Supply Chains anhand des Anwendungsfalls der Blechindustrie möglichst umfassend abzubilden. Der vorliegende Abschlussbericht fasst den Stand der Technik sowie die Erkenntnisse aus dem Projekt zusammen. AuĂerdem wird ein Ăberblick ĂŒber die Projektstruktur sowie die Projektpartner gegeben
Semantic Meaningfulness:Evaluating Counterfactual Approaches for Real-World Plausibility and Feasibility
Counterfactual explanations are rising in popularity when aiming to increase the explainability of machine learning models. This type of explanation is straightforward to understand and provides actionable feedback (i.e., how to change the model decision). One of the main challenges that remains is generating meaningful counterfactuals that are coherent with real-world relations. Multiple approaches incorporating real-world relations have been proposed in the past, e.g. by utilizing data distributions or structural causal models. However, evaluating whether the explanations from different counterfactual approaches fulfill known causal relationships is still an open issue. To fill this gap, this work proposes two metrics - Semantic Meaningful Output (SMO) and Semantic Meaningful Relations (SMR) - to measure the ability of counterfactual generation approaches to depict real-world relations. In addition, we provide multiple datasets with known structural causal models and leverage them to benchmark the semantic meaningfulness of new and existing counterfactual approaches. Finally, we evaluate the semantic meaningfulness of nine well-established counterfactual explanation approaches and conclude that none of the non-causal approaches were able to create semantically meaningful counterfactuals consistently.</p