776 research outputs found
"Explain it in the Same Way!" -- Model-Agnostic Group Fairness of Counterfactual Explanations
Counterfactual explanations are a popular type of explanation for making the
outcomes of a decision making system transparent to the user. Counterfactual
explanations tell the user what to do in order to change the outcome of the
system in a desirable way. However, it was recently discovered that the
recommendations of what to do can differ significantly in their complexity
between protected groups of individuals. Providing more difficult
recommendations of actions to one group leads to a disadvantage of this group
compared to other groups.
In this work we propose a model-agnostic method for computing counterfactual
explanations that do not differ significantly in their complexity between
protected groups
Dynamisches Testen der Lesekompetenz. Theoretische Grundlagen, Konzeption und Testentwicklung. Projekt Dynamisches Testen
Dynamische Tests gelten insbesondere im Bildungssektor als Alternative zu herkömmlichen Tests. Für den Bereich der Lesekompetenz fehlen bislang jedoch gesicherte Erkenntnisse, die die diagnostische Gute eines dynamischen Lesekompetenztests belegen können. Der hier vorliegende Beitrag bietet einen theoretischen Überbau für die Entwicklung eines solchen dynamischen Lesekompetenztests und informiert zudem (nicht immer chronologisch) über grundlegende Schritte der Testentwicklung sowie erste Ergebnisse der Testkonstruktion. (DIPF/Orig.
For Better or Worse: The Impact of Counterfactual Explanations' Directionality on User Behavior in xAI
Counterfactual explanations (CFEs) are a popular approach in explainable
artificial intelligence (xAI), highlighting changes to input data necessary for
altering a model's output. A CFE can either describe a scenario that is better
than the factual state (upward CFE), or a scenario that is worse than the
factual state (downward CFE). However, potential benefits and drawbacks of the
directionality of CFEs for user behavior in xAI remain unclear. The current
user study (N=161) compares the impact of CFE directionality on behavior and
experience of participants tasked to extract new knowledge from an automated
system based on model predictions and CFEs. Results suggest that upward CFEs
provide a significant performance advantage over other forms of counterfactual
feedback. Moreover, the study highlights potential benefits of mixed CFEs
improving user performance compared to downward CFEs or no explanations. In
line with the performance results, users' explicit knowledge of the system is
statistically higher after receiving upward CFEs compared to downward
comparisons. These findings imply that the alignment between explanation and
task at hand, the so-called regulatory fit, may play a crucial role in
determining the effectiveness of model explanations, informing future research
directions in xAI. To ensure reproducible research, the entire code, underlying
models and user data of this study is openly available:
https://github.com/ukuhl/DirectionalAlienZooComment: 22 pages, 3 figures This work has been accepted for presentation at
the 1st World Conference on eXplainable Artificial Intelligence (xAI 2023),
July 26-28, 2023 - Lisbon, Portuga
Setting a standard for low reading proficiency. A comparison of the bookmark procedure and constrained mixture Rasch model
In order to draw pertinent conclusions about persons with low reading skills, it is essential to use validated standard-setting procedures by which they can be assigned to their appropriate level of proficiency. Since there is no standard-setting procedure without weaknesses, external validity studies are essential. Traditionally, studies have assessed validity by comparing different judgement-based standard-setting procedures. Only a few studies have used model-based approaches for validating judgement-based procedures. The present study addressed this shortcoming and compared agreement of the cut score placement between a judgement-based approach (i.e., Bookmark procedure) and a model-based one (i.e., constrained mixture Rasch model). This was performed by differentiating between individuals with low reading proficiency and those with a functional level of reading proficiency in three independent samples of the German National Educational Panel Study that included students from the ninth grade (N = 13,897) as well as adults (Ns = 5,335 and 3,145). The analyses showed quite similar mean cut scores for the two standard-setting procedures in two of the samples, whereas the third sample showed more pronounced differences. Importantly, these findings demonstrate that model-based approaches provide a valid and resource-efficient alternative for external validation, although they can be sensitive to the ability distribution within a sample. (DIPF/Orig.
Metakognitives Wissen in der Sekundarstufe: Konstruktion und Evaluation domänenspezifischer Messverfahren. Projekt EWIKO
Deklaratives metakognitives Wissen stellt neben prozeduralen metakognitiven Fertigkeiten eine notwendige Voraussetzung zur Steuerung kognitiver Prozesse dar. Aufgrund psychometrischer Probleme stehen jedoch kaum Messverfahren zur Erfassung dieser Komponente metakognitiver Kompetenz zur Verfügung. In der vorliegenden Studie wurden domänenspezifische Verfahren zur Erfassung des deklarativen metakognitiven Wissens in den Domänen Lesen, Mathematik, Englisch und im selbstregulierten Lernen für den Einsatz zu Beginn der Sekundarstufe adaptiert bzw. neu konstruiert. Die vier Wissenstests basieren auf domänenspezifischen Szenarien und rekurrieren auf die Einschätzung der relativen Nützlichkeit von Strategien. Die Testkonstruktion wurde an einer Stichprobe von 798 bayerischen Schüler/innen der 5. und 6. Jahrgangsstufe aus den Schularten Hauptschule und Gymnasium validiert. Die interne Konsistenz (Cronbachs Alpha) der vier Verfahren ist zufrieden stellend bis gut. In allen vier Domänen zeigen sich deutliche Wissensunterschiede sowohl zwischen den untersuchten Jahrgangsstufen als auch zwischen den untersuchten Schularten. Die konstruierten Verfahren bilden das Leistungsspektrum am Beginn der Sekundarstufe reliabel und valide ab. Weitere, längsschnittliche Analysen zu den Entwicklungsmustern des deklarativen metakognitiven Wissens innerhalb und zwischen den Domänen sind notwendig. (DIPF/Autor
Zur Vergleichbarkeit von Schülerleistungen bei Leseaufgaben unterschiedlichen sprachlichen Ursprungs
- …