"Explain it in the Same Way!" -- Model-Agnostic Group Fairness of Counterfactual Explanations

Counterfactual explanations are a popular type of explanation for making the outcomes of a decision making system transparent to the user. Counterfactual explanations tell the user what to do in order to change the outcome of the system in a desirable way. However, it was recently discovered that the recommendations of what to do can differ significantly in their complexity between protected groups of individuals. Providing more difficult recommendations of actions to one group leads to a disadvantage of this group compared to other groups. In this work we propose a model-agnostic method for computing counterfactual explanations that do not differ significantly in their complexity between protected groups

Dynamisches Testen der Lesekompetenz. Theoretische Grundlagen, Konzeption und Testentwicklung. Projekt Dynamisches Testen

Dynamische Tests gelten insbesondere im Bildungssektor als Alternative zu herkömmlichen Tests. Für den Bereich der Lesekompetenz fehlen bislang jedoch gesicherte Erkenntnisse, die die diagnostische Gute eines dynamischen Lesekompetenztests belegen können. Der hier vorliegende Beitrag bietet einen theoretischen Überbau für die Entwicklung eines solchen dynamischen Lesekompetenztests und informiert zudem (nicht immer chronologisch) über grundlegende Schritte der Testentwicklung sowie erste Ergebnisse der Testkonstruktion. (DIPF/Orig.

For Better or Worse: The Impact of Counterfactual Explanations' Directionality on User Behavior in xAI

Counterfactual explanations (CFEs) are a popular approach in explainable artificial intelligence (xAI), highlighting changes to input data necessary for altering a model's output. A CFE can either describe a scenario that is better than the factual state (upward CFE), or a scenario that is worse than the factual state (downward CFE). However, potential benefits and drawbacks of the directionality of CFEs for user behavior in xAI remain unclear. The current user study (N=161) compares the impact of CFE directionality on behavior and experience of participants tasked to extract new knowledge from an automated system based on model predictions and CFEs. Results suggest that upward CFEs provide a significant performance advantage over other forms of counterfactual feedback. Moreover, the study highlights potential benefits of mixed CFEs improving user performance compared to downward CFEs or no explanations. In line with the performance results, users' explicit knowledge of the system is statistically higher after receiving upward CFEs compared to downward comparisons. These findings imply that the alignment between explanation and task at hand, the so-called regulatory fit, may play a crucial role in determining the effectiveness of model explanations, informing future research directions in xAI. To ensure reproducible research, the entire code, underlying models and user data of this study is openly available: https://github.com/ukuhl/DirectionalAlienZooComment: 22 pages, 3 figures This work has been accepted for presentation at the 1st World Conference on eXplainable Artificial Intelligence (xAI 2023), July 26-28, 2023 - Lisbon, Portuga

Setting a standard for low reading proficiency. A comparison of the bookmark procedure and constrained mixture Rasch model

In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach (i.e., Bookmark procedure) and a model-based one (i.e., constrained mixture Rasch model). This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade (N = 13,897) as well as adults (Ns = 5,335 and 3,145). The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample. (DIPF/Orig.

Metakognitives Wissen in der Sekundarstufe: Konstruktion und Evaluation domänenspezifischer Messverfahren. Projekt EWIKO

Deklaratives metakognitives Wissen stellt neben prozeduralen metakognitiven Fertigkeiten eine notwendige Voraussetzung zur Steuerung kognitiver Prozesse dar. Aufgrund psychometrischer Probleme stehen jedoch kaum Messverfahren zur Erfassung dieser Komponente metakognitiver Kompetenz zur Verfügung. In der vorliegenden Studie wurden domänenspezifische Verfahren zur Erfassung des deklarativen metakognitiven Wissens in den Domänen Lesen, Mathematik, Englisch und im selbstregulierten Lernen für den Einsatz zu Beginn der Sekundarstufe adaptiert bzw. neu konstruiert. Die vier Wissenstests basieren auf domänenspezifischen Szenarien und rekurrieren auf die Einschätzung der relativen Nützlichkeit von Strategien. Die Testkonstruktion wurde an einer Stichprobe von 798 bayerischen Schüler/innen der 5. und 6. Jahrgangsstufe aus den Schularten Hauptschule und Gymnasium validiert. Die interne Konsistenz (Cronbachs Alpha) der vier Verfahren ist zufrieden stellend bis gut. In allen vier Domänen zeigen sich deutliche Wissensunterschiede sowohl zwischen den untersuchten Jahrgangsstufen als auch zwischen den untersuchten Schularten. Die konstruierten Verfahren bilden das Leistungsspektrum am Beginn der Sekundarstufe reliabel und valide ab. Weitere, längsschnittliche Analysen zu den Entwicklungsmustern des deklarativen metakognitiven Wissens innerhalb und zwischen den Domänen sind notwendig. (DIPF/Autor