Gathering big data for teamwork evaluation with microworlds

info:eu-repo/semantics/publishedVersio

TELMA Cross Experiment Guidelines

Cerulli, M., Pedemonte, B., Robotti, E. (eds.). Internal Report, R.I. 01/07, I.T.D. - C.N.R., GenovaThis document contains the guidelines developed by members of TELMA as a means for planning, conducting, and analysing a cross experiment aimed at contributing to the construction of a shared research perspective among TELMA teams . This is the product of the PhD students and young researchers that brought forward the whole activity. The actual experimental phase was proceeded by a reflective phase in which an agreement was achieved on what research questions to address during the experiment. On this basis the first version of the guidelines document was built, containing all the research questions to be addressed, but also the experimental plans for each team. This included the employed didactical functionalities of the considered ICT tools, indications of the experimental settings, and the methods of data collection and analysis. During the whole experimental phase, the document was constantly updated, and shared among the involved persons which were periodically required to compare the different activities and reflections brought forward by all the teams

Cognitive modelling of complex problem solving behaviour

In the universe of problems humans face every day there is subset characterized by a salient dynamic component. The FireChief task (Omodei & Wearing 1995) is a fire-fighting computer simulation that can be characterized as the acquisition of interactive skills involving fast-paced actions cued by external information. This research describes the process followed to create a cognitive model of this complex dynamic task where full experimental control is not available. The cognitive model provides a detailed description of how cognition and perception interplay to produce the interactive skill of fighting the fire. Several artefacts were produced by this effort including a dynamic task fully compatible with ACT-R, a tool for analysing the data, and a cognitive model whose features enable the replication of several aspects of the empirical data. A key finding is that good performance is linked to an effective combination of strategic control with attention to changing task demands, reflecting time and care taken in informing and effecting action. The contributions of this work towards our understanding of complex problem solving are the methodological approach to the creation of the model, the design patterns embedded in the model (which are a reflection of the cognitive demands imposed by the nature of the task) and mainly an explanation of how skill, described in terms of strategy use, is acquired in complex scenarios. This study also provides a deeper understanding of the interactions observed in the Cañas et al. (2005) dataset, including a computational realisation of how cognitive inflexibility occurs

The Benefits and the Costs of Using Auditory Warning Messages in Dynamic Decision Making Settings

The failure to notice critical changes in both visual and auditory scenes may have important consequences for performance in complex dynamic environments, especially those related to security such as aviation, surveillance during major events, and command and control of emergency response. Previous work has shown that a significant number of situation changes remain undetected by operators in such environments. In the current study, we examined the impact of using auditory warning messages to support the detection of critical situation changes and to a broader extent the decision making required by the environment. Twenty-two participants performed a radar operator task involving multiple subtasks while detecting critical task-related events that were cued by a specific type of audio message. Results showed that about 22% of the critical changes remained undetected by participants, a percentage similar to that found in previous work using visual cues to support change detection. However, we found that audio messages tended to bias threat evaluation towards perceiving objects as more threatening than they were in reality. Such findings revealed both benefits and costs associated with using audio messages to support change detection in complex dynamic environments

Metafora: A Web-based Platform for Learning to Learn Together in Science and Mathematics

This paper presents Metafora, both a platform for integrated tools as well as an emerging pedagogy for supporting Learning to Learn Together in science and mathematics education. Our goal is to design technology that brings education to a higher level; a level where students not only learn a subject matter, but also gain a set of critical skills needed to engage in and self-regulate collaborative learning experiences in science and math education. We first discuss the core skills we hope students will gain as they learn to learn together. We then present our design and implementation that can achieve this goal; a platform and pedagogy we have developed to support the learning of these skills. Finally, we present an example use of our system based on results from pilot studies that demonstrates interaction with the platform, and potential benefits and limitations of the tools in promoting the associated skills

Modeling a Pandemic: Investigating Student Learning about Disease Spread in the Context of Agent-Based Modeling

The COVID-19 pandemic has highlighted a need for students to learn about public health issues, including the transmission of disease and methods for the prevention of epidemics. This study presents data from a project focused on developing computational microworlds to help middle school students learn about these topics. The microworld is designed to help students model and test their ideas about how a disease spreads through a population and how an epidemic can be prevented. I employed a lab-based case study approach to conduct one-on-one 1.5-hour interviews through Zoom with four middle-school students (ages 12-14). During the interview, the student was asked questions about the spread and prevention of disease and then invited to model and test their ideas in the microworld. This study presents an analysis of students’ pre and post instructional knowledge of disease spread and prevention, which they shared while constructing their initial and later models. I present student ideas in categories of disease transmission, recovery from disease, and disease protection strategies. The paper also analyzes students’ knowledge refinement through the building, testing, and debugging of a disease spread and prevention model. I model student refinement of thinking through steps of building initial models and predicting results, testing initial models, making sense of the results, debugging and retesting models, observing final models, and explaining results, resulting in three types of thinking shifts, and two types of thinking refinements. My findings suggest middle school students can learn about strategies for disease prevention through computational modeling

TURF for Teams: Considering Both the Team and I in the Work-Centered Design of Systems

Teams are an inherent part of many work domains, especially in the healthcare environment. Yet, most systems are often built with only the individual user in mind. How can we better incorporate the team, as a user, into the design of a system? By better understanding the team, through their user, task, representational, and functional needs, we can create more useful and helpful systems that match their work domain. For this research project, we utilize the TURF framework and expanded it further by also considering teams as a user, thus, creating the TURF for Teams framework. In addition, we chose to examine teams in the emergency department environment. We believe that designing a system with the team also fully incorporated and acknowledged in the work domain will be beneficial for supporting necessary team activities. Using TURF for Teams, we first conducted an observational field study in the emergency department to get a better understanding of the users, teams, tasks, workload, and interactions. We then identified the need for team communications to be better supported, especially in the management of interruptions, and further categorized the interruptions by their function in order to design a team tool that could help team members better manage their interruptions by focusing on the necessary, or domain, types of interruptions and more easily disregarding the unnecessary, or overhead, types of interruptions. We then administered some surveys and conducted a card sort and cognitive walkthrough with emergency clinician participants to help us better identify how to design interfaces for the team tool and simulation that would better match the needs of team communication behaviors observed and reported by emergency clinicians. After designing and developing the team tool and simulation, we conducted an evaluation of this system by having emergency medicine, medicine, and informatics graduate student teams go through the system and utilize the team tool and simulation as a team. Though we had a small sample size, we found that emergency medicine teams found the team tool and simulation to be very usable and they reacted favorably to its potential in helping them better understand and manage their team communications. In summary, we were able to utilize the TURF framework for incorporating teams into the design of systems, in this case a team communication tool and microworld simulation for the emergency department. Our findings suggest that TURF for Teams is a viable framework for designing useful and helpful team based systems for all work domains

A cognitive prosthesis for complex decision-making

While simple heuristics can be ecologically rational and effective in naturalistic decision making contexts, complex situations require analytical decision making strategies, hypothesis-testing and learning. Sub-optimal decision strategies – using simplified as opposed to analytic decision rules – have been reported in domains such as healthcare, military operational planning, and government policy making. We investigate the potential of a computational toolkit called “IMAGE” to improve decision-making by developing structural knowledge and increasing understanding of complex situations. IMAGE is tested within the context of a complex military convoy management task through (a) interactive simulations, and (b) visualization and knowledge representation capabilities. We assess the usefulness of two versions of IMAGE (desktop and immersive) compared to a baseline. Results suggest that the prosthesis helped analysts in making better decisions, but failed to increase their structural knowledge about the situation once the cognitive prosthesis is removed

A software framework for simulation studies of interaction models in agent teamwork.

This thesis proposes a new software framework that facilitates the study of agent interaction models in early development stages from a designer's perspective. Its purpose is to help reduced the design decision space through simulation experiments that provide early feedback on comparative performance of alternative solutions. This is achieved through interactive concurrent simulation of multiple teams in a representative microworld context. The generic simulator's architecture accommodates an open class of different microworlds and permits multiple communication mechanisms. It also supports interoperability with other software tools, distributed simulation, and various extensions. The framework was validated in the context of two different research projects on helpful behavior in agent teams: the Mutual Assistance Protocol, based on rational criteria for help, and the Empathic Help Model, based on a concept of empathy for artificial agents. The results show that the framework meets its design objectives and provides the flexibility needed for research experimentation. --Leaf i.The original print copy of this thesis may be available here: http://wizard.unbc.ca/record=b184472