Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

We propose a computational model of situated language comprehension based on the Indexical Hypothesis that generates meaning representations by translating amodal linguistic symbols to modal representations of beliefs, knowledge, and experience external to the linguistic system. This Indexical Model incorporates multiple information sources, including perceptions, domain knowledge, and short-term and long-term experiences during comprehension. We show that exploiting diverse information sources can alleviate ambiguities that arise from contextual use of underspecific referring expressions and unexpressed argument alternations of verbs. The model is being used to support linguistic interactions in Rosie, an agent implemented in Soar that learns from instruction.Comment: Advances in Cognitive Systems 3 (2014

The Role of Human-Automation Consensus in Multiple Unmanned Vehicle Scheduling

Objective: This study examined the impact of increasing automation replanning rates on operator performance and workload when supervising a decentralized network of heterogeneous unmanned vehicles. Background: Futuristic unmanned vehicles systems will invert the operator-to-vehicle ratio so that one operator can control multiple dissimilar vehicles connected through a decentralized network. Significant human-automation collaboration will be needed because of automation brittleness, but such collaboration could cause high workload. Method: Three increasing levels of replanning were tested on an existing multiple unmanned vehicle simulation environment that leverages decentralized algorithms for vehicle routing and task allocation in conjunction with human supervision. Results: Rapid replanning can cause high operator workload, ultimately resulting in poorer overall system performance. Poor performance was associated with a lack of operator consensus for when to accept the automation’s suggested prompts for new plan consideration as well as negative attitudes toward unmanned aerial vehicles in general. Participants with video game experience tended to collaborate more with the automation, which resulted in better performance. Conclusion: In decentralized unmanned vehicle networks, operators who ignore the automation’s requests for new plan consideration and impose rapid replans both increase their own workload and reduce the ability of the vehicle network to operate at its maximum capacity. Application: These findings have implications for personnel selection and training for futuristic systems involving human collaboration with decentralized algorithms embedded in networks of autonomous systems.Aurora Flight Sciences Corp.United States. Office of Naval Researc

Flexible Task Execution and Cognitive Control in Human-Robot Interaction

A robotic system that interacts with humans is expected to flexibly execute structured cooperative tasks while reacting to unexpected events and behaviors. In this thesis, these issues are faced presenting a framework that integrates cognitive control, executive attention, structured task execution and learning. In the proposed approach, the execution of structured tasks is guided by top-down (task-oriented) and bottom-up (stimuli-driven) attentional processes that affect behavior selection and activation, while resolving conflicts and decisional impasses. Specifically, attention is here deployed to stimulate the activations of multiple hierarchical behaviors orienting them towards the execution of finalized and interactive activities. On the other hand, this framework allows a human to indirectly and smoothly influence the robotic task execution exploiting attention manipulation. We provide an overview of the overall system architecture discussing the framework at work in different applicative contexts. In particular, we show that multiple concurrent tasks/plans can be effectively orchestrated and interleaved in a flexible manner; moreover, in a human-robot interaction setting, we test and assess the effectiveness of attention manipulation and learning processes

Authority Management and Conflict Solving in Human-Machine Systems

This paper focuses on vehicle-embedded decision autonomy and the human operator’s role in so-called autonomous systems. Autonomy control and authority sharing are discussed, and the possible effects of authority conflicts on the human operator’s cognition and situation awareness are highlighted. As an illustration, an experiment conducted at ISAE (the French Aeronautical and Space Institute) shows that the occurrence of a conflict leads to a perseveration behavior and attentional tunneling of the operator. Formal methods are discussed to infer such attentional impairment from the monitoring of physiological and behavioral measures and some results are given

Goal-Directed Behavior under Variational Predictive Coding: Dynamic Organization of Visual Attention and Working Memory

Mental simulation is a critical cognitive function for goal-directed behavior because it is essential for assessing actions and their consequences. When a self-generated or externally specified goal is given, a sequence of actions that is most likely to attain that goal is selected among other candidates via mental simulation. Therefore, better mental simulation leads to better goal-directed action planning. However, developing a mental simulation model is challenging because it requires knowledge of self and the environment. The current paper studies how adequate goal-directed action plans of robots can be mentally generated by dynamically organizing top-down visual attention and visual working memory. For this purpose, we propose a neural network model based on variational Bayes predictive coding, where goal-directed action planning is formulated by Bayesian inference of latent intentional space. Our experimental results showed that cognitively meaningful competencies, such as autonomous top-down attention to the robot end effector (its hand) as well as dynamic organization of occlusion-free visual working memory, emerged. Furthermore, our analysis of comparative experiments indicated that introduction of visual working memory and the inference mechanism using variational Bayes predictive coding significantly improve the performance in planning adequate goal-directed actions

Attention Allocation for Human Multi-Robot Control: Cognitive Analysis based on Behavior Data and Hidden States

Human multi-robot interaction exploits both the human operator’s high-level decision-making skills and the robotic agents’ vigorous computing and motion abilities. While controlling multi-robot teams, an operator’s attention must constantly shift between individual robots to maintain sufficient situation awareness. To conserve an operator’s attentional resources, a robot with self reflect capability on its abnormal status can help an operator focus her attention on emergent tasks rather than unneeded routine checks. With the proposing self-reflect aids, the human-robot interaction becomes a queuing framework, where the robots act as the clients to request for interaction and an operator acts as the server to respond these job requests. This paper examined two types of queuing schemes, the self-paced Open-queue identifying all robots’ normal/abnormal conditions, whereas the forced-paced shortest-job-first (SJF) queue showing a single robot’s request at one time by following the SJF approach. As a robot may miscarry its experienced failures in various situations, the effects of imperfect automation were also investigated in this paper. The results suggest that the SJF attentional scheduling approach can provide stable performance in both primary (locate potential targets) and secondary (resolve robots’ failures) tasks, regardless of the system’s reliability levels. However, the conventional results (e.g., number of targets marked) only present little information about users’ underlying cognitive strategies and may fail to reflect the user’s true intent. As understanding users’ intentions is critical to providing appropriate cognitive aids to enhance task performance, a Hidden Markov Model (HMM) is used to examine operators’ underlying cognitive intent and identify the unobservable cognitive states. The HMM results demonstrate fundamental differences among the queuing mechanisms and reliability conditions. The findings suggest that HMM can be helpful in investigating the use of human cognitive resources under multitasking environments

Towards Multi-UAV and Human Interaction Driving System Exploiting Human Mental State Estimation

This paper addresses the growing human-multi-UAV interaction issue. Current active approaches towards a reliable multi-UAV system are reviewed. This brings us to the conclusion that the multiple Unmanned Aerial Vehicles (UAVs) control paradigm is segmented into two main scopes: i) autonomous control and coordination within the group of UAVs, and ii) a human centered approach with helping agents and overt behavior monitoring. Therefore, to move further with the future of human-multi-UAV interaction problem, a new perspective is put forth. In the following sections, a brief understanding of the system is provided, followed by the current state of multi-UAV research and how taking the human pilot's physiology into account could improve the interaction. This idea is developed first by detailing what physiological computing is, including mental states of interest and their associated physiological markers. Second, the article concludes with the proposed approach for Human-multi-UAV interaction control and future plans

Mitigation of Human Supervisory Control Wait Times through Automation Strategies

The application of network centric operations principles to human supervisory control (HSC) domains means that humans are increasingly being asked to manage multiple simultaneous HSC processes. However, increases in the number of available information sources, volume of information and operational tempo, all which place higher cognitive demands on operators, could become constraints limiting the success of network centric processes. In time-pressured scenarios typical of networked command and control scenarios, efficiently allocating attention between a set of dynamic tasks is crucial for mission success. Inefficient attention allocation leads to system wait times, which could eventually lead to critical events such as missed times on targets and degraded overall mission success. One potential solution to mitigating wait times is the introduction of automated decision support in order to relieve operator workload. However, it is not obvious what automated decision support is appropriate, as higher levels of automation may result in a situation awareness decrement and other problems typically associated with excessive automation such as automation bias. To assess the impact of increasing levels of automation on human and system performance in a time-critical HSC multiple task management context, an experiment was run in which an operator simultaneously managed four highly autonomous unmanned aerial vehicles (UAVs) executing an air tasking order, with the overall goal of destroying a pre-determined set of targets within a limited time period. Four increasing levels automated decision support were investigated as well as high and low operational replanning tempos. The highest level of automation, management-byexception, had the best performance across several metrics but had a greater number of catastrophic events during which a UAV erroneously destroyed a friendly target. Contrary to expectations, the collaborative level of decision support, which provided predictions for possible periods of task overload as well as possible courses of action to relieve the high workload, produced the worst performance. This is attributable to an unintended consequence of the automation where the graphical visualization of the computer’s predictions caused users to try to globally optimize the schedules for all UAVs instead of locally optimizing schedules in the immediate future, resulting in them being overwhelmed. Total system wait time across both experimental factors was dominated by wait time caused by lack of situation awareness, which is difficult to eliminate, implying that there will be a clear upper limit on the number of vehicles that any one person can supervise because of the need to stay cognitively aware of unfolding events.Prepared for Boeing, Phantom Work