1 research outputs found
ACCBench: A Framework for Comparing Causality Algorithms
Modern socio-technical systems are increasingly complex. A fundamental
problem is that the borders of such systems are often not well-defined
a-priori, which among other problems can lead to unwanted behavior during
runtime. Ideally, unwanted behavior should be prevented. If this is not
possible the system shall at least be able to help determine potential cause(s)
a-posterori, identify responsible parties and make them accountable for their
behavior. Recently, several algorithms addressing these concepts have been
proposed. However, the applicability of the corresponding approaches,
specifically their effectiveness and performance, is mostly unknown. Therefore,
in this paper, we propose ACCBench, a benchmark tool that allows to compare and
evaluate causality algorithms under a consistent setting. Furthermore, we
contribute an implementation of the two causality algorithms by G\"o{\ss}ler
and Metayer and G\"o{\ss}ler and Astefanoaei as well as of a policy compliance
approach based on some concepts of Main et al. Lastly, we conduct a case study
of an Intelligent Door Control System, which exposes concrete strengths and
weaknesses of all algorithms under different aspects. In the course of this, we
show that the effectiveness of the algorithms in terms of cause detection as
well as their performance differ to some extent. In addition, our analysis
reports on some qualitative aspects that should be considered when evaluating
each algorithm. For example, the human effort needed to configure the algorithm
and model the use case is analyzed.Comment: In Proceedings CREST 2017, arXiv:1710.0277