Knowledge-based multimodal information fusion for role recognition and situation assessment by using mobile robot

Decision-making is the key for autonomous systems to achieve real intelligence and autonomy. This paper presents an integrated probabilistic decision framework for a robot to infer roles that humans fulfill in specific missions. The framework also enables the assessment of the situation and necessity of interaction with the person fulfilling the target role. The target role is the person who is distinctive in movement or holds a mission-critical object, where the object is pre-specified in the corresponding mission. The proposed framework associates prior knowledge with spatial relationships between the humans and objects as well as with their temporal changes. Distance-Based Inference (DBI) and Knowledge-Based Inference (KBI) support recognition of human roles. DBI deduces the role based on the relative distance between humans and the specified objects. KBI focuses on human actions and objects existence. The role is estimated using weighted fusion scheme based on the information entropy. The situation is assessed by analyzing the action of the person fulfilling the target role and relative position of this person to the mission-related entities, where the entity is something that has a particular function in the corresponding mission. This assessment determines the robot decision on what actions it should take. A series of experiments has proofed that the proposed framework provides a reasonable assessment of the situation. Moreover, it outperforms other approaches on accuracy, efficiency, and robustness.NRF (Natl Research Foundation, S’pore)Accepted versio

Software agents & human behavior

People make important decisions in emergencies. Often these decisions involve high stakes in terms of lives and property. Bhopal disaster (1984), Piper Alpha disaster (1988), Montara blowout (2009), and explosion on Deepwater Horizon (2010) are a few examples among many industrial incidents. In these incidents, those who were in-charge took critical decisions under various ental stressors such as time, fatigue, and panic. This thesis presents an application of naturalistic decision-making (NDM), which is a recent decision-making theory inspired by experts making decisions in real emergencies. This study develops an intelligent agent model that can be programed to make human-like decisions in emergencies. The agent model has three major components: (1) A spatial learning module, which the agent uses to learn escape routes that are designated routes in a facility for emergency evacuation, (2) a situation recognition module, which is used to recognize or distinguish among evolving emergency situations, and (3) a decision-support module, which exploits modules in (1) and (2), and implements an NDM based decision-logic for producing human-like decisions in emergencies. The spatial learning module comprises a generalized stochastic Petri net-based model of spatial learning. The model classifies routes into five classes based on landmarks, which are objects with salient spatial features. These classes deal with the question of how difficult a landmark turns out to be when an agent observes it the first time during a route traversal. An extension to the spatial learning model is also proposed where the question of how successive route traversals may impact retention of a route in the agent’s memory is investigated. The situation awareness module uses Markov logic network (MLN) to define different offshore emergency situations using First-order Logic (FOL) rules. The purpose of this module is to give the agent the necessary experience of dealing with emergencies. The potential of this module lies in the fact that different training samples can be used to produce agents having different experience or capability to deal with an emergency situation. To demonstrate this fact, two agents were developed and trained using two different sets of empirical observations. The two are found to be different in recognizing the prepare-to-abandon-platform alarm (PAPA ), and similar to each other in recognition of an emergency using other cues. Finally, the decision-support module is proposed as a union of spatial-learning module, situation awareness module, and NDM based decision-logic. The NDM-based decision-logic is inspired by Klein’s (1998) recognition primed decision-making (RPDM) model. The agent’s attitudes related to decision-making as per the RPDM are represented in the form of belief, desire, and intention (BDI). The decision-logic involves recognition of situations based on experience (as proposed in situation-recognition module), and recognition of situations based on classification, where ontological classification is used to guide the agent in cases where the agent’s experience about confronting a situation is inadequate. At the planning stage, the decision-logic exploits the agent’s spatial knowledge (as proposed in spatial-learning module) about the layout of the environment to make adjustments in the course of actions relevant to a decision that has already been made as a by-product of situation recognition. The proposed agent model has potential to be used to improve virtual training environment’s fidelity by adding agents that exhibit human-like intelligence in performing tasks related to emergency evacuation. Notwithstanding, the potential to exploit the basis provided here, in the form of an agent representing human fallibility, should not be ignored for fields like human reliability analysis

Desarrollo de un sistema computacional para el análisis de procesos emocionales a través de las técnicas de reconocimiento facial y de potenciales relacionados a eventos

Objetivo: Establecer una metodología basada en las técnicas de reconocimiento facial y de potenciales relacionados a eventos para los procesos de orientación vocacional de universitarios. Metodología: Para la realización de este trabajo se consideró la metodología de desarrollo de software Proceso Unificado Racional (RUP), la cual incluyó las fases de Inicio, Elaboración, Construcción y Transición. Estas fases contemplan los análisis de requerimientos, diseño y construcción de la herramienta para el análisis de emociones, las pruebas con estudiantes, el análisis de pruebas y la entrega final del software. El sistema de análisis de emociones se construyó a través de Reconocimiento Facial de Emociones RFE (Affectiva), evaluación de Electroencefalografía EEG (Emotiv), sistema de gestión de protocolos, aplicación en formato digital de la prueba de Kuder y evaluación automática de las respuestas brindadas por los encuestados según área de interés. Resultados: Se consolidó una metodología basada en RFE y EEG para el análisis de emociones en procesos de orientación vocacional de estudiantes universitarios. Se automatizó la aplicación de la prueba de Kuder. Se realizó el desarrollo de la herramienta computacional para la presentación de protocolos, el seguimiento de emociones con RFE y EEG. Se validó la herramienta con 25 sujetos de prueba, los cuales respondieron la prueba de Kuder y fueron evaluados mediante la herramienta mientras observaban protocolos de estimulación asociados con sus áreas de interés y de no interés. Se analizaron los datos adquiridos y se encontró la efectividad de la herramienta encontrándose un porcentaje de afinidad con la prueba de orientación vocacional con un acierto cercano al 85%, especificidad del 87% y sensibilidad del 88%, resultados de valor para un ambiente de alta variabilidad. Conclusiones: Fue posible consolidar una metodología basada en el análisis de RFE y EEG para complementar las pruebas de orientación vocacional. Se utilizó como medio de referencia la prueba de Kuder que permitió validar la capacidad del método a la hora de identificar emociones al tiempo que se observan protocolos de estimulación asociados con áreas de desempeño vocacional. La metodología propuesta puede ser usada como complemento en los procesos de orientación vocacional y como una prueba rápida para encontrar afinidad entre el evaluado y las diferentes áreas de interés.Objective: To establish a methodology based on facial recognition techniques and event-related potentials for the vocational orientation processes of university students. Methodology: For the realization of this work, the Rational Unified Process (RUP) software development methodology was considered, which included the phases of Initiation, Elaboration, Construction and Transition. These phases contemplate the requirements analysis, design and construction of the emotion analysis tool, student testing, test analysis and final delivery of the software. The emotion analysis system was built through Facial Recognition of Emotions RFE (Affectiva), Electroencephalography EEG evaluation (Emotiv), protocol management system and automatic Kuder test evaluation. Results: A methodology based on RFE and EEG was consolidated for the analysis of emotions in vocational orientation processes of university students. The application of the Kuder test was automated. A computational tool was developed for the presentation of protocols, monitoring of emotions with RFE and EEG. The tool was validated with 25 test subjects, who responded to the Kuder test and were evaluated using the tool while observing stimulation protocols associated with their areas of interest and non-interest. The acquired data were analyzed and the effectiveness of the tool was found, finding a percentage of affinity with the vocational orientation test with a hit rate close to 85%, specificity of 87% and sensitivity of 88%, results of value for an environment of high variability.Conclusions: It was possible to consolidate a methodology based on RFE and EEG analysis to complement vocational orientation tests. The Kuder test was used as a reference medium, which allowed validating the ability of the method to identify emotions while observing stimulation protocols associated with areas of vocational performance. The proposed methodology can be used as a complement in vocational orientation processes and as a quick test to find affinity between the evaluated and the different areas of interest