Journal of Communication Pedagogy, Complete Volume 4, 2021

This is the complete volume 4 of the Journal of Communication Pedagogy

Partnering People with Deep Learning Systems: Human Cognitive Effects of Explanations

Advances in “deep learning” algorithms have led to intelligent systems that provide automated classifications of unstructured data. Until recently these systems could not provide the reasons behind a classification. This lack of “explainability” has led to resistance in applying these systems in some contexts. An intensive research and development effort to make such systems more transparent and interpretable has proposed and developed multiple types of explanation to address this challenge. Relatively little research has been conducted into how humans process these explanations. Theories and measures from areas of research in social cognition were selected to evaluate attribution of mental processes from intentional systems theory, measures of working memory demands from cognitive load theory, and self-efficacy from social cognition theory. Crowdsourced natural disaster damage assessment of aerial images was employed using a written assessment guideline as the task. The “Wizard of Oz” method was used to generate the damage assessment output of a simulated agent. The output and explanations contained errors consistent with transferring a deep learning system to a new disaster event. A between-subjects experiment was conducted where three types of natural language explanations were manipulated between conditions. Counterfactual explanations increased intrinsic cognitive load and made participants more aware of the challenges of the task. Explanations that described boundary conditions and failure modes (“hedging explanations”) decreased agreement with erroneous agent ratings without a detectable effect on cognitive load. However, these effects were not large enough to counteract decreases in self-efficacy and increases in erroneous agreement as a result of providing a causal explanation. The extraneous cognitive load generated by explanations had the strongest influence on self-efficacy in the task. Presenting all of the explanation types at the same time maximized cognitive load and agreement with erroneous simulated output. Perceived interdependence with the simulated agent was also associated with increases in self-efficacy; however, trust in the agent was not associated with differences in self-efficacy. These findings identify effects related to research areas which have developed methods to design tasks that may increase the effectiveness of explanations

An Approach to Modeling Simulated Military Human-agent Teaming

With the rise of human-agent teaming (HAT), a new cycle of scientific discovery commenced. Through scientific discovery, a number of theories of constructs in HAT were developed, however, an overarching model is lacking that elucidates the relative importance of these constructs in relation to human performance. The main objective of this research was to develop a model of simulated military HAT and to validate it against selected empirical data. Experimental data borrowed from four simulated military HAT studies were used to test the proposed Core model. The Core model was assumed to be directly affecting task performance and consisted of constructs related to Task Composition, Task Perception, and the qualities that each team member (Human/Agent Qualities) brings to the team. The available experimental data were tested against the null model: everything, within and between these Core sections, are equal contributors to hit rate. Furthermore, in order to validate the Core model, a validation approach was developed based on relative importance, wherein the outcome was a proportional value and followed a beta distribution (Ferrari & Cribari-Neto, 2004). This new modeling approach consisted of (1) application of dominance analysis (DA; Azen & Budescu, 2003; Budescu, 1993) to determine the most important contributors to task performance, (2) establishing robustness and generalizability of the dominance outcome through bootstrap procedures (Azen & Budescu, 2003; Efron, 1981), and (3) combining the dominant predictors into a full beta regression model to evaluate the fit and significance of the model (Ferrari & Cribari-Neto, 2004). DA of all four experimental studies examined in this research led to rejecting the null hypotheses. Constructs in the proposed Core model were not equally important to performance in these simulated military HAT studies. Results showed consistently similar yet different dominance patterns in relation to human performance. Attempts were made to elucidate the most important predictors of task performance. Analyses unveiled the importance of taking task difficulty into consideration when assessing the relative importance within the proposed Core model

Achieving Transparency in Human-Collective Systems

Collective robotic systems are biologically-inspired and exhibit behaviors found in spatial swarms (e.g., fish), colonies (e.g., ants), or a combination of both (e.g., bees). Collective robotic system popularity continues to increase due to their apparent global intelligence and emergent behaviors. Many applications can benefit from the incorporation of collectives, including environmental monitoring, disaster response missions, and infrastructure support. Human-collective system designers continue to debate how best to achieve transparency in human-collective systems in order to attain meaningful and insightful information exchanges between the operator and collective, enable positive operator influence on collectives, and improve the human-collective's performance. Few human-collective evaluations have been conducted, many of which have only assessed how embedding transparency into one system design element (e.g., models, visualizations, or control mechanisms) may impact human-collective behaviors, such as the human-collective performance. This dissertation developed a transparency definition for collective systems that was leveraged to assess how to achieve transparency in a single human-collective system. Multiple models and visualizations were evaluated for a sequential best-of-\textit{n} decision-making task with four collectives. Transparency was evaluated with respect to how the model and visualization impacted human operators who possess different capabilities, operator comprehension, system usability, and human-collective performance. Transparency design guidance was created in order to aid the design of future human-collective systems. One set of guidelines were inspired from the results and discussions of the single human-collective analyses and another set were based on a review of the biological literature. This dissertation can be used to aid designers achieve transparency in human-collective systems. The primary contributions are: 1. A transparency definition for human-collective systems that describes the process of identifying what factors affect and are influenced by transparency, why those factors are important, and how to design a system to achieve transparency. 2. An expansive set of metrics that successfully evaluated how transparency influenced operators with different individual capabilities, operator comprehension, system usability, and human-collective performance. 3. The recommendation that system transparency quantification requires evaluating the transparency embedded into the various system design elements in order to determine how they interact with one another and influence the human-collective interactions and performance. 4. Design guidance recommendations with respect to models, visualizations, and control mechanisms in order to inform designers how transparency can be achieved for human-collective systems

Transparency in human-agent teaming and its effect on complacent behavior

This study examined how transparency of an intelligent agent\u27s reasoning affected complacent behavior in a route selection task in a simulated environment. Also examined was how the information available to the operator affected those results. In two experiments, participants supervised a three-vehicle convoy as it traversed a simulated environment and re-routed the convoy when needed with the assistance of an intelligent agent, RoboLeader. Participants were randomly assigned to an Agent Reasoning Transparency condition. Participants received communications from a commander confirming either the presence or absence of activity in the area. They also received information regarding potential events along their route via icons that appeared on a map displaying the convoy route and surrounding area. Participants in Experiment 1 (low information setting) received information about their current route only; they did not receive any information about the suggested alternate route. Participants in Experiment 2 (high information setting) received information about both their current route and the agent recommended an alternative route. In the first experiment, access to agent reasoning was found to be an effective deterrent to complacent behavior when the operator has limited information about their task environment. However, the addition of information that created ambiguity for the operator encouraged complacency, resulting in reduced performance and poorer trust calibration. Agent reasoning did not increase response time or workload and appeared to have improved performance on the secondary task. These findings align with studies that have shown ambiguous information can increase workload and encourage complacency, as such, caution should be exercised when considering how transparent to make agent reasoning and what information should be included. In the second experiment, access to agent reasoning was found to have little effect on complacent behavior when the operator had complete information about the task environment. However, the addition of information that created ambiguity for the operator appeared to encourage complacency, as indicated by reduced performance and shorter decision times. Agent reasoning transparency did not increase overall workload, and operators reported higher satisfaction with their performance and reduced mental demand. Access to agent reasoning did not improve operators\u27 secondary task performance, situation awareness, or operator trust. However, when agent reasoning transparency included ambiguous information complacent behavior was again encouraged. Unlike the first experiment, there were notable differences in complacent behavior, performance, operator trust, and situation awareness due to individual difference factors. As such, these findings would suggest that when the operator has complete information regarding their task environment, access to agent reasoning may be beneficial, but not dramatically so. However, individual difference factors will greatly influence performance outcomes. The amount of information the operator has regarding the task environment has a profound effect on the proper use of the agent. Increased environmental information resulted in more rejections of the agent recommendation regardless of the transparency of agent reasoning. The addition of agent reasoning transparency appeared to be effective at keeping the operator engaged, while complacent behavior appeared to be encouraged both when agent reasoning was either not transparent or so transparent as to become ambiguous. Even so, operators reported lower trust and usability for the agent than when environmental information was limited. Situation awareness (SA2) scores were also higher in the high information environment when agent reasoning was either not transparent or so transparent as to become ambiguous, compared to the low information environment. However, when a moderate amount of agent reasoning was available to the operator, the amount of information available to the operator had no effect on the operators\u27 complacent behavior, subjective trust, or SA. These findings indicate that some negative outcomes resulting from the incongruous transparency of agent reasoning may be mitigated by increasing the information the operator has regarding the task environment

Compilation of Abstracts, December 2015

NPS Class of December 2015This quarter’s Compilation of Abstracts summarizes cutting-edge, security-related research conducted by NPS students and presented as theses, dissertations, and capstone reports. Each expands knowledge in its field.http://archive.org/details/compilationofabs109454828

A comparative study of the NCATE Standards and the Virginia Licensure Regulations for School Personnel with the perceptions of elementary principals and teachers

The purpose of this study was to compare the National Council for the Accreditation of Teacher Education (NCATE) 2000 Standards for Elementary Preparation and the Virginia Licensure Regulations for Elementary Personnel (1998) with the perceptions of principals and teachers in the elementary schools in Chesterfield County, Virginia. The methodology incorporated an exploratory design using qualitative data collection and analyses of surveys of school-based personnel, as well as the NCATE standards and Virginia Licensure Regulations. Specifically, themes were identified through a series of reductions of the terms and phrases contained within the documents and surveys.;The results of this study identified similarities and differences between the perceptions of the school-based individuals and the documents analyzed. Among others, some significant themes that emerged were knowledge of content and pedagogy, assessment, personal and professional skills, and authentic experiences

Talking about learning: the role of student-teacher dialogue in increasing authenticity and validity in assessment of student learning in secondary school drama

The intention of this work is to argue that if assessment in secondary school drama classes is to achieve any reasonable measure of authenticity and validity, student self-assessment and student/teacher dialogue must be a vital part of that assessment. The first four chapters comprise an overview covering five major concept areas: the current trends in assessment towards standardization and quantification and the problems inherent in those methods; the uniqueness of learning in the arts; definition of the various types of learning that occur during students' practice of drama and the difficulties of assessing them; an overview and analysis of recent practice in drama assessment; and a proposal for using self- and dialogic assessment including a literature review addressing the problems to be solved m utilizing those means for assessment. The fifth chapter details and defends the methodology by which the data were collected and analyzed. The data were collected through Action Research using my classroom as laboratory and my students as subjects. Data were collected through four separate methods detailed in Chapter Five. Chapter Six examines and analyzes the data. The chapter offers evidence of the various types of learning operationalized in Chapter Three, examines the language of self-assessment and the growth of students' self-assessment skills, and finally describes the effect of student/teacher and student/student dialogue in guiding and optimizing that self-assessment. In the concluding chapter, I suggest that the practice of self- and dialogic assessment may be useful in increasing the validity and authenticity of assessment across the curriculum and propose some areas in which further research concerning the use of self-assessment and dialogue could be useful