Utilizing the Capabilities Offered by Eye-Tracking to Foster Novices' Comprehension of Business Porcess Models

Business process models constitute fundamental artifacts for enterprise architectures as well as for the engineering of processes and information systems. However, less experienced stakeholders (i.e., novices) face a wide range of issues when trying to read and comprehend these models. In particular, process model comprehension not only requires knowledge on process modeling notations, but also skills to visually and correctly interpret the models. In this context, many unresolved issues concerning the factors hindering process model comprehension exist and, hence, the identification of these factors becomes crucial. Using eye-tracking as an instrument, this paper presents the results obtained of a study, in which we analyzed eye-movements of novices and experts, while comprehending process models expressed in terms of the Business Process Model and Notation (BPMN) 2.0. Further, recorded eye-movements are visualized as scan paths to analyze the applied comprehension strategies. We learned that experts comprehend process models more effectively than novices. In addition, we observed particular patterns for eye-movements (e.g., back-and-forth saccade jumps) as well as different strategies of novices and experts in comprehending process models

Towards Measuring and Quantifying the Comprehensibility of Process Models -- The Process Model Comprehension Framework

Process models constitute crucial artifacts in modern information systems and, hence, the proper comprehension of these models is of utmost importance in the utilization of such systems. Generally, process models are considered from two different perspectives: process modelers and readers. Both perspectives share similarities and differences in the comprehension of process models (e.g., diverse experiences when working with process models). The literature proposed many rules and guidelines to ensure a proper comprehension of process models for both perspectives. As a novel contribution in this context, this paper introduces the Process Model Comprehension Framework (PMCF) as a first step towards the measurement and quantification of the perspectives of process modelers and readers as well as the interaction of both regarding the comprehension of process models. Therefore, the PMCF describes an Evaluation Theory Tree based on the Communication Theory as well as the Conceptual Modeling Quality Framework and considers a total of 96 quality metrics in order to quantify process model comprehension. Furthermore, the PMCF was evaluated in a survey with 131 participants and has been implemented as well as applied successfully in a practical case study including 33 participants. To conclude, the PMCF allows for the identification of pitfalls and provides related information about how to assist process modelers as well as readers in order to foster and enable a proper comprehension of process models.Comment: 16 pages, 5 Figures, 1 Table, additional materials in appendi

Towards the Applicability of Measuring the Electrodermal Activity in the Context of Process Model Comprehension: Feasibility Study

Process model comprehension is essential in order to understand the Five Ws (i.e., who, what, where, when, and why) pertaining to the processes of organizations. However, research in this context showed that a proper comprehension of process models often poses a challenge in practice. For this reason, a vast body of research exists studying the factors having an influence on process model comprehension. In order to point research towards a neuro-centric perspective in this context, the paper at hand evaluates the appropriateness of measuring the electrodermal activity (EDA) during the comprehension of process models. Therefore, a preliminary test run and a feasibility study were conducted relying on an EDA and physical activity sensor to record the EDA during process model comprehension. The insights obtained from the feasibility study demonstrated that process model comprehension leads to an increased activity in the EDA. Furthermore, EDA-related results indicated significantly that participants were confronted with a higher cognitive load during the comprehension of complex process models. In addition, the experiences and limitations we have learned in measuring the EDA during the comprehension of process models are discussed in this paper. In conclusion, the feasibility study demonstrated that the measurement of the EDA could be an appropriate method to obtain new insights in process model comprehension

Assessment of Visual Literacy – Contributions of Eye Tracking

Visual Literacy (VL) is defined as a set of competencies to understand and express oneself through visual imagery. An expansive model, the Common European Framework of Reference for Visual Literacy (CEFR-VL) (Wagner & Schönau, 2016), comprises 16 sub-competencies, including abilities such as analyzing, judging, experimenting with or aesthetically experiencing images. To empirically assess VL sub-competencies different visual tasks were presented to VL experts and novices. Problem-solving behavior and cognitive strategies involved in visual logical reasoning (Paper 1), Visual Search (Paper 2), and judgments of visual abstraction (Paper 3) were investigated. Eye tracking in combination with innovative statistical methods were used to uncover latent variables during task performance and to assess the possible effects of differences in expertise level. Furthermore, the relationship between students' self-reported visual abilities and their performance on VL assessment tasks is systematically explored. Results show how effects of perceptual skills of VL experts are less pronounced and more nuanced than implied by VL models. The comprehension of visual logical models does not seem to depend much on VL, as experts and novices did not differ in their solution strategies and eye movement indicators (Paper 1). In contrast, the visual search task on artworks revealed how experts were able to detect target regions with higher efficiency than novices revealed by higher precision of fixations on target regions. Furthermore, latent image features were detected by experts with more certainty (Paper 2). The assessment of perceived level of visual abstraction revealed how, contrary to our expectations, experts did not outperform novices but despite that were able to detect nuanced level of abstraction compared to student groups. Distribution of fixations indicate how attention is directed towards more ambiguous images (Paper 3). Students can be classified based on different levels of visual logical comprehension (Paper 1), on self-reported visual skills, and the time spent on the tasks (Paper 2, Paper 3). Self-reported visual art abilities of students (e.g., imagination) influences the visual search and the judgment of visual abstraction. Taken together the results show how VL skills are not determined solely by the number of correct responses, but rather by how visual tasks are solved and deconstructed; for example, experts are able to focus on less salient image regions during visual search and demonstrate a more nuanced interpretation of visual abstraction. Low-level perceptual abilities of experts and novices differ marginally, which is consistent with research on art expertise. Assessment of VL remains challenging, but new empirical methods are proposed to uncover the underlying components of VL

Training Effects of Adaptive Emotive Responses From Animated Agents in Simulated Environments

Humans are distinct from machines in their capacity to emote, stimulate, and express emotions. Because emotions play such an important role in human interactions, human-like agents used in pedagogical roles for simulation-based training should properly reflect emotions. Currently, research concerning the development of this type of agent focuses on basic agent interface characteristics, as well as character building qualities. However, human-like agents should provide emotion-like qualities that are clearly expressed, properly synchronized, and that simulate complex, real-time interactions through adaptive emotion systems. The research conducted for this dissertation was a quantitative investigation using 3 (within) x 2 (between) x 3 (within) factorial design. A total of 56 paid participants consented to complete the study. Independent variables included emotion intensity (i.e., low, moderate, and high emotion), levels of expertise (novice participant versus experienced participant), and number of trials. Dependent measures included visual attention, emotional response towards the animated agents, simulation performance score, and learners\u27 perception of the pedagogical agent persona while participants interacted with a pain assessment and management simulation. While no relationships were indicated between the levels of emotion intensity portrayed by the animated agents and the participants\u27 visual attention, emotional response towards the animated agent, and simulation performance score, there were significant relationships between the level of expertise of the participant and the visual attention, emotional responses, and performance outcomes. The results indicated that nursing students had higher visual attention during their interaction with the animated agents. Additionally, nursing students expressed more neutral facial expression whereas experienced nurses expressed more emotional facial expressions towards the animated agents. The results of the simulation performance scores indicated that nursing students obtained higher performance scores in the pain assessment and management task than experienced nurses. Both groups of participants had a positive perception of the animated agents persona

AN EXAMINATION OF THE IMPACT OF COMPUTER-BASED ANIMATIONS AND VISUALIZATION SEQUENCE ON LEARNERS' UNDERSTANDING OF HADLEY CELLS IN ATMOSPHERIC CIRCULATION

Research examining animation use for student learning has been conducted in the last two decades across a multitude of instructional environments and content areas. The extensive construction and implementation of animations in learning resulted from the availability of powerful computing systems and the perceived advantages the novel medium offered to deliver dynamic representations of complex systems beyond the human perceptual scale. Animations replaced or supplemented text and static diagrams of system functioning and were predicted to significantly improve learners' conceptual understanding of target systems. However, subsequent research has not consistently discovered affordances to understanding, and in some cases, has actually shown that animation use is detrimental to system understanding especially for content area novices (Lowe 2004; Mayer et al. 2005). This study sought to determine whether animation inclusion in an authentic learning context improved student understanding for an introductory earth science concept, Hadley Cell circulation. In addition, the study sought to determine whether the timing of animation examination improved conceptual understanding. A quasi-experimental pretest posttest design administered in an undergraduate science lecture and laboratory course compared four different learning conditions: text and static diagrams with no animation use, animation use prior to the examination of text and static diagrams, animation use following the examination of text and static diagrams, and animation use during the examination of text and static diagrams. Additionally, procedural data for a sample of three students in each condition were recorded and analyzed through the lens of self regulated learning (SRL) behaviors. The aim was to determine whether qualitative differences existed between cognitive processes employed. Results indicated that animation use did not improve understanding across all conditions. However learners able to employ animations while reading and examining the static diagrams and to a lesser extent, after reading the system description, showed evidence of higher levels of system understanding on posttest assessments. Procedural data found few differences between groups with one exception---learners given access to animations during the learning episode chose to examine and coordinate the representations more frequently. These results indicated a new finding from the use of animation, a sequence effect to improve understanding of Hadley Cells in atmospheric circulation

The Empirical Analysis of the Comprehensibility of Process Models created by Process Mining

Companies use process models to specify their operational processes. With the help of process models, the business processes in a company are analysed by process mining techniques to optimise them. The subdiscipline of process discovery identifies the actual state of business processes and enables them to be examined. Various tools and algorithms can be used, which lead to different process visualisations. The type of process visualisation has a major influence on the comprehensibility of process models. The objective of this thesis is to investigate the comprehensibility of process models generated by process mining. For this purpose, an exploratory eye-tracking study is conducted with fifteen participants. The study examines process models from two scenarios - a vaccination process and an insurance process. The corresponding process models are created manually, and event logs are generated from them using self-created applications. These event logs are loaded into the process mining tools Celonis Snap, Disco, ProM, Apromore and PM4Py and process models are generated from them. A selection of the resulting process models is then tested for comprehensibility in the user study. The analysis of variance (ANOVA) shows no significant differences between the different generated process models. Finally, with the Pearson correlation’s help, the participants’ subjective ranking is highly significantly related to the level of acceptability and cognitive load. The correlation between the time spent looking at the process models and the number of correctly answered comprehension questions is interesting. From this correlation, it can be concluded that understanding process models requires a certain amount of time. An astonishing result of the study is that the quality between manually created models and models generated by process mining is similarly high. Despite interesting results, further studies are needed, as the study is confronted with some limitations (particularly the number of participants). The results can be used as a basis for future studies to further explore this field of research

Proficiency-aware systems

In an increasingly digital world, technological developments such as data-driven algorithms and context-aware applications create opportunities for novel human-computer interaction (HCI). We argue that these systems have the latent potential to stimulate users and encourage personal growth. However, users increasingly rely on the intelligence of interactive systems. Thus, it remains a challenge to design for proficiency awareness, essentially demanding increased user attention whilst preserving user engagement. Designing and implementing systems that allow users to become aware of their own proficiency and encourage them to recognize learning benefits is the primary goal of this research. In this thesis, we introduce the concept of proficiency-aware systems as one solution. In our definition, proficiency-aware systems use estimates of the user's proficiency to tailor the interaction in a domain and facilitate a reflective understanding for this proficiency. We envision that proficiency-aware systems leverage collected data for learning benefit. Here, we see self-reflection as a key for users to become aware of necessary efforts to advance their proficiency. A key challenge for proficiency-aware systems is the fact that users often have a different self-perception of their proficiency. The benefits of personal growth and advancing one's repertoire might not necessarily be apparent to users, alienating them, and possibly leading to abandoning the system. To tackle this challenge, this work does not rely on learning strategies but rather focuses on the capabilities of interactive systems to provide users with the necessary means to reflect on their proficiency, such as showing calculated text difficulty to a newspaper editor or visualizing muscle activity to a passionate sportsperson. We first elaborate on how proficiency can be detected and quantified in the context of interactive systems using physiological sensing technologies. Through developing interaction scenarios, we demonstrate the feasibility of gaze- and electromyography-based proficiency-aware systems by utilizing machine learning algorithms that can estimate users' proficiency levels for stationary vision-dominant tasks (reading, information intake) and dynamic manual tasks (playing instruments, fitness exercises). Secondly, we show how to facilitate proficiency awareness for users, including design challenges on when and how to communicate proficiency. We complement this second part by highlighting the necessity of toolkits for sensing modalities to enable the implementation of proficiency-aware systems for a wide audience. In this thesis, we contribute a definition of proficiency-aware systems, which we illustrate by designing and implementing interactive systems. We derive technical requirements for real-time, objective proficiency assessment and identify design qualities of communicating proficiency through user reflection. We summarize our findings in a set of design and engineering guidelines for proficiency awareness in interactive systems, highlighting that proficiency feedback makes performance interpretable for the user.In einer zunehmend digitalen Welt schaffen technologische Entwicklungen - wie datengesteuerte Algorithmen und kontextabhängige Anwendungen - neuartige Interaktionsmöglichkeiten mit digitalen Geräten. Jedoch verlassen sich Nutzer oftmals auf die Intelligenz dieser Systeme, ohne dabei selbst auf eine persönliche Weiterentwicklung hinzuwirken. Wird ein solches Vorgehen angestrebt, verlangt dies seitens der Anwender eine erhöhte Aufmerksamkeit. Es ist daher herausfordernd, ein entsprechendes Design für Kompetenzbewusstsein (Proficiency Awareness) zu etablieren. Das primäre Ziel dieser Arbeit ist es, eine Methodik für das Design und die Implementierung von interaktiven Systemen aufzustellen, die Nutzer dabei unterstützen über ihre eigene Kompetenz zu reflektieren, um dadurch Lerneffekte implizit wahrnehmen können. Diese Arbeit stellt ein Konzept für fähigkeitsbewusste Systeme (proficiency-aware systems) vor, welche die Fähigkeiten von Nutzern abschätzen, die Interaktion entsprechend anpassen sowie das Bewusstsein der Nutzer über deren Fähigkeiten fördern. Hierzu sollten die Systeme gesammelte Daten von Nutzern einsetzen, um Lerneffekte sichtbar zu machen. Die Möglichkeit der Anwender zur Selbstreflexion ist hierbei als entscheidend anzusehen, um als Motivation zur Verbesserung der eigenen Fähigkeiten zu dienen. Eine zentrale Herausforderung solcher Systeme ist die Tatsache, dass Nutzer - im Vergleich zur Abschätzung des Systems - oft eine divergierende Selbstwahrnehmung ihrer Kompetenz haben. Im ersten Moment sind daher die Vorteile einer persönlichen Weiterentwicklung nicht unbedingt ersichtlich. Daher baut diese Forschungsarbeit nicht darauf auf, Nutzer über vorgegebene Lernstrategien zu unterrichten, sondern sie bedient sich der Möglichkeiten interaktiver Systeme, die Anwendern die notwendigen Hilfsmittel zur Verfügung stellen, damit diese selbst über ihre Fähigkeiten reflektieren können. Einem Zeitungseditor könnte beispielsweise die aktuelle Textschwierigkeit angezeigt werden, während einem passionierten Sportler dessen Muskelaktivität veranschaulicht wird. Zunächst wird herausgearbeitet, wie sich die Fähigkeiten der Nutzer mittels physiologischer Sensortechnologien erkennen und quantifizieren lassen. Die Evaluation von Interaktionsszenarien demonstriert die Umsetzbarkeit fähigkeitsbewusster Systeme, basierend auf der Analyse von Blickbewegungen und Muskelaktivität. Hierbei kommen Algorithmen des maschinellen Lernens zum Einsatz, die das Leistungsniveau der Anwender für verschiedene Tätigkeiten berechnen. Im Besonderen analysieren wir stationäre Aktivitäten, die hauptsächlich den Sehsinn ansprechen (Lesen, Aufnahme von Informationen), sowie dynamische Betätigungen, die die Motorik der Nutzer fordern (Spielen von Instrumenten, Fitnessübungen). Der zweite Teil zeigt auf, wie Systeme das Bewusstsein der Anwender für deren eigene Fähigkeiten fördern können, einschließlich der Designherausforderungen , wann und wie das System erkannte Fähigkeiten kommunizieren sollte. Abschließend wird die Notwendigkeit von Toolkits für Sensortechnologien hervorgehoben, um die Implementierung derartiger Systeme für ein breites Publikum zu ermöglichen. Die Forschungsarbeit beinhaltet eine Definition für fähigkeitsbewusste Systeme und veranschaulicht dieses Konzept durch den Entwurf und die Implementierung interaktiver Systeme. Ferner werden technische Anforderungen objektiver Echtzeitabschätzung von Nutzerfähigkeiten erforscht und Designqualitäten für die Kommunikation dieser Abschätzungen mittels Selbstreflexion identifiziert. Zusammengefasst sind die Erkenntnisse in einer Reihe von Design- und Entwicklungsrichtlinien für derartige Systeme. Insbesondere die Kommunikation, der vom System erkannten Kompetenz, hilft Anwendern, die eigene Leistung zu interpretieren

Comparative Analysis of Conventional Electronic and OZ Concept Displays for Aircraft Energy Management

A repeated-measures, within-subjects design was conducted on 58 participant pilots to assess mean differences on energy management situation awareness response time and response accuracy between a conventional electronic aircraft display, a primary flight display (PFD), and an ecological interface design aircraft display, the OZ concept display. Participants were associated with a small Midwestern aviation university, including student pilots, flight instructors, and faculty with piloting experience. Testing consisted of observing 15 static screenshots of each cockpit display type and then selecting applicable responses from 27 standardized responses for each screen. A paired samples t-test was computed comparing accuracy and response time for the two displays. There was no significant difference in means between PFD Response Time and OZ Response Time. On average, mean PFD Accuracy was significantly higher than mean OZ Accuracy (MDiff = 13.17, SDDiff = 20.96), t(57) = 4.78, p \u3c .001, d = 0.63. This finding showed operational potential for the OZ display, since even without first training to proficiency on the previously unseen OZ display, participant performance differences were not operationally remarkable. There was no significant correlation between PFD Response Time and PFD Accuracy, but there was a significant correlation between OZ Response Time and OZ Accuracy, r (58) = .353, p \u3c .01. These findings suggest the participant familiarity of the PFD resulted in accuracy scores unrelated to response time, compared to the participants unaccustomed with the OZ display where longer response times manifested in greater understanding of the OZ display. PFD Response Time and PFD Accuracy were not correlated with pilot flight hours, which was not expected. It was thought that increased experience would translate into faster and more accurate assessment of the aircraft stimuli. OZ Response Time and OZ Accuracy were also not correlated with pilot flight hours, but this was expected. This was consistent with previous research that observed novice operators performing as well as experienced professional pilots on dynamic flight tasks with the OZ display. A demographic questionnaire and a feedback survey were included in the trial. An equivalent three-quarters majority of participants rated the PFD as “easy” and the OZ as “confusing”, yet performance accuracy and response times between the two displays were not operationally different

Older Adults and Technology-Based Instruction: Optimizing Learning Outcomes and Transfer

The purpose of this paper is to provide an overview of the cognitive and socio-emotional changes associated with aging and to propose ways in which these changes can be accommodated in a technology-based training environment. We recommend that technology-based training for older adults should: 1) be highly structured, 2) provide feedback and adaptive guidance, 3) include meta-cognitive prompts, 4) incorporate principles derived from cognitive load theory and cognitive theory of multimedia learning, and 5) include a user interface that is simple and consistent throughout the course. With a focus on organizations as well as business schools, we then discuss contextual variables that are expected to enhance older learners\u27 training motivation or improve their transfer of training. Finally, we will recommend areas worthy of exploration that might reveal age-specific differences in TBI design