Personalizing Human-Robot Dialogue Interactions using Face and Name Recognition

Task-oriented dialogue systems are computer systems that aim to provide an interaction indistinguishable from ordinary human conversation with the goal of completing user- defined tasks. They are achieving this by analyzing the intents of users and choosing respective responses. Recent studies show that by personalizing the conversations with this systems one can positevely affect their perception and long-term acceptance. Personalised social robots have been widely applied in different fields to provide assistance. In this thesis we are working on development of a scientific conference assistant. The goal of this assistant is to provide the conference participants with conference information and inform about the activities for their spare time during conference. Moreover, to increase the engagement with the robot our team has worked on personalizing the human-robot interaction by means of face and name recognition. To achieve this personalisation, first the name recognition ability of available physical robot was improved, next by the concent of the participants their pictures were taken and used for memorization of returning users. As acquiring the consent for personal data storage is not an optimal solution, an alternative method for participants recognition using QR Codes on their badges was developed and compared to pre-trained model in terms of speed. Lastly, the personal details of each participant, as unviversity, country of origin, was acquired prior to conference or during the conversation and used in dialogues. The developed robot, called DAGFINN was displayed at two conferences happened this year in Stavanger, where the first time installment did not involve personalization feature. Hence, we conclude this thesis by discussing the influence of personalisation on dialogues with the robot and participants satisfaction with developed social robot

Conversational Agent: Developing a Model for Intelligent Agents with Transient Emotional States

The inclusion of human characteristics (i.e., emotions, personality) within an intelligent agent can often increase the effectiveness of information delivery and retrieval. Chat-bots offer a plethora of benefits within an eclectic range of disciplines (e.g., education, medicine, clinical and mental health). Hence, chatbots offer an effective way to observe, assess, and evaluate human communication patterns. Current research aims to develop a computational model for conversational agents with an emotional component to be applied to the army leadership training program that will allow for the examination of interpersonal skills in future research. Overall, the current research explores the application of the deep learning algorithm to the development of a generalized framework that will be based upon modeling empathetic conversation between an intelligent conversational agent (chatbot) and a human user in order to allow for higher level observation of interpersonal communication skills. Preliminary results demonstrate the promising potential of the seq2seq technique (e.g., through the use of Dialog Flow Chatbot platform) when applied to emotion-oriented conversational tasks. Both the classification and generative conversational modeling tasks demonstrate the promising potential of the current research for representing human to agent dialogue. However, this implementation may be extended by utilizing, a larger more high-quality dataset

Prototype of a Conversational Assistant for Satellite Mission Operations

The very first artificial satellite, Sputnik, was launched in 1957 marking a new era. Concurrently, satellite mission operations emerged. These start at launch and finish at the end of mission, when the spacecraft is decommissioned. Running a satellite mission requires the monitoring and control of telemetry data, to verify and maintain satellite health, reconfigure and command the spacecraft, detect, identify and resolve anomalies and perform launch and early orbit operations. The very first chatbot, ELIZA was created in 1966, and also marked a new era of Artificial Intelligence Systems. Said systems answer users’ questions in the most diverse domains, interpreting the human language input and responding in the same manner. Nowadays, these systems are everywhere, and the list of possible applications seems endless. The goal of the present master’s dissertation is to develop a prototype of a chatbot for mission operations. For this purpose implementing a Natural Language Processing (NLP) model for satellite missions allied to a dialogue flow model. The performance of the conversational assistant is evaluated with its implementation on a mission operated by the European Space Agency (ESA), implying the generation of the spacecraft’s Database Knowledge Graph (KG). Throughout the years, many tools have been developed and added to the systems used to monitor and control spacecrafts helping Flight Control Teams (FCT) either by maintaining a comprehensive overview of the spacecraft’s status and health, speeding up failure investigation, or allowing to easily correlate time series of telemetry data. However, despite all the advances made which facilitate the daily tasks, the teams still need to navigate through thousands of parameters and events spanning years of data, using purposely built user interfaces and relying on filters and time series plots. The solution presented in this dissertation and proposed by VisionSpace Technologies focuses on improving operational efficiency whilst dealing with the mission’s complex and extensive databases.O primeiro satélite artificial, Sputnik, foi lançado em 1957 e marcou o início de uma nova era. Simultaneamente, surgiram as operações de missão de satélites. Estas iniciam com o lançamento e terminam com desmantelamento do veículo espacial, que marca o fim da missão. A operação de satélites exige o acompanhamento e controlo de dados de telemetria, com o intuito de verificar e manter a saúde do satélite, reconfigurar e comandar o veículo, detetar, identificar e resolver anomalias e realizar o lançamento e as operações iniciais do satélite. Em 1966, o primeiro Chatbot foi criado, ELIZA, e também marcou uma nova era, de sistemas dotados de Inteligência Artificial. Tais sistemas respondem a perguntas nos mais diversos domínios, para tal interpretando linguagem humana e repondendo de forma similar. Hoje em dia, é muito comum encontrar estes sistemas e a lista de aplicações possíveis parece infindável. O objetivo da presente dissertação de mestrado consiste em desenvolver o protótipo de um Chatbot para operação de satélites. Para este proposito, criando um modelo de Processamento de Linguagem Natural (NLP) aplicado a missoões de satélites aliado a um modelo de fluxo de diálogo. O desempenho do assistente conversacional será avaliado com a sua implementação numa missão operada pela Agência Espacial Europeia (ESA), o que implica a elaboração do grafico de conhecimentos associado à base de dados da missão. Ao longo dos anos, várias ferramentas foram desenvolvidas e adicionadas aos sistemas que acompanham e controlam veículos espaciais, que colaboram com as equipas de controlo de missão, mantendo uma visão abrangente sobre a condição do satélite, acelerando a investigação de falhas, ou permitindo correlacionar séries temporais de dados de telemetria. No entanto, apesar de todos os progressos que facilitam as tarefas diárias, as equipas ainda necessitam de navegar por milhares de parametros e eventos que abrangem vários anos de recolha de dados, usando interfaces para esse fim e dependendo da utilização de filtros e gráficos de series temporais. A solução apresentada nesta dissertação e proposta pela VisionSpace Technologies tem como foco melhorar a eficiência operacional lidando simultaneamente com as suas complexas e extensas bases de dados

Development and evaluation of a microservice-based virtual assistant for chronic patients support

Los asistentes virtuales (también conocidos como chatbots) son programas que interactúan con los usuarios simulando una conversación humana a través de mensajes de texto o de voz. Los asistentes virtuales destinados al cuidado de la salud ofrecen servicios, herramientas, asesoramiento, ayuda, soporte y gestión de diferentes enfermedades. Los usuarios de este tipo de asistente virtual pueden ser, por ejemplo, pacientes, cuidadores y profesionales sanitarios, los cuales poseen diferentes necesidades y requerimientos. Los pacientes con enfermedades crónicas podrían beneficiarse de los asistentes virtuales que se encargan de realizar seguimientos de su condición, proporcionar información específica, fomentar la adherencia a la medicación, etc. Para realizar estas funciones, los asistentes virtuales necesitan una arquitectura de software adecuada. Esta tesis doctoral propone el diseño de una arquitectura específica para el desarrollo de asistentes virtuales destinados a proporcionar soporte a pacientes crónicos. Hoy en día, las personas interactúan entre sí diariamente utilizando plataformas de mensajería. Para alinear este tipo de interacción con la arquitectura del asistente virtual, proponemos el uso de plataformas de mensajería para la interacción asistente virtual-paciente, prestando especial atención a las cuestiones de seguridad y privacidad (es decir, el uso de plataformas de mensajería seguras con cifrado de extremo a extremo).Los asistentes virtuales pueden implementar sistemas conversacionales para que la interacción con los pacientes sea más natural. Los sistemas conversacionales en escenarios de atención médica complejos, como la gestión de enfermedades, deben ser capaces de poder comprender oraciones complejas utilizadas durante la interacción. La adaptación de nuevos métodos con el procesamiento de lenguaje natural (NLP, por su nombre en inglés, Natural Language Processing) puede aportar una mejora a la arquitectura del asistente virtual. Los word embeddings (incrustación de palabras) se han utilizado ampliamente en NLP como entrada en las redes neuronales. Tales word embeddings pueden ayudar a comprender el objetivo final y las palabras clave en una oración. Por ello, en esta tesis estudiamos el impacto de diferentes word embeddings entrenados con corpus generales y específicos utilizando el entendimiento del lenguaje natural conjunto (Joint NLU, por su nombre en inglés, Joint Natural Language Understanding) en el dominio de la medicación en español. Los datos para entrenar el modelo NLU conjunto se generan usando plantillas. Dicho modelo se utiliza para la detección de intenciones, así como para el slot filling (llenado de ranuras). En este estudio comparamos word2vec y fastText como word embeddings y ELMo y BERT como modelos de lenguaje. Para entrenar los embeddings utilizamos tres corpus diferentes: los datos de entrenamiento generados para este escenario, la Wikipedia en español como dominio general y la base de datos de medicamentos en español como datos especializados. El mejor resultado se obtuvo con el modelo ELMo entrenado con Wikipedia en español.Dotamos al asistente virtual de capacidades de gestión de medicamentos basadas en NLP. En consecuencia, se analiza el impacto del etiquetado de slots y la longitud de los datos de entrenamiento en modelos NLU conjuntos para escenarios de gestión de medicamentos utilizando asistentes virtuales en español. En este estudio definimos las intenciones (propósitos de las oraciones) para escenarios centrados en la administración de medicamentos y dos tipos de etiquetas de slots. Para entrenar el modelo, generamos cuatro conjuntos de datos, combinando oraciones largas o cortas con slots largos o cortos. Para el análisis comparativo, elegimos seis modelos NLU conjuntos (SlotRefine, stack-propagation framework, SF-ID network, capsule-NLU, slot-gated modeling y joint SLU-LM) de la literatura existente. Tras el análisis competitivo, se observa que el mejor resultado se obtuvo utilizando oraciones y slots cortos. Nuestros resultados sugirieron que los modelos NLU conjuntos entrenados con slots cortos produjeron mejores resultados que aquellos entrenados con slots largos para la tarea de slot filling.En definitiva, proponemos una arquitectura de microservicios genérica válida para cualquier tipo de gestión de enfermedades crónicas. El prototipo genérico ofrece un asistente virtual operativo para gestionar información básica y servir de base para futuras ampliaciones. Además, en esta tesis presentamos dos prototipos especializados con el objetivo de mostrar cómo esta nueva arquitectura permite cambiar, añadir o mejorar diferentes partes del asistente virtual de forma dinámica y flexible. El primer prototipo especializado tiene como objetivo ayudar en la gestión de la medicación del paciente. Este prototipo se encargará de recordar la ingesta de medicamentos a través de la creación de una comunidad de apoyo donde los pacientes, cuidadores y profesionales sanitarios interactúen con herramientas y servicios útiles ofrecidos por el asistente virtual. La implementación del segundo prototipo especializado está diseñada para una enfermedad crónica específica, la psoriasis. Este prototipo ofrece teleconsulta y almacenamiento de fotografías.Por último, esta tesis tiene como objetivo validar la eficacia del asistente virtual integrado en las plataformas de mensajería, destinado al cuidado de la salud. Por ello, esta tesis incluye la evaluación de los dos prototipos especializados. El primer estudio tiene como objetivo mejorar la adherencia a la medicación en pacientes con diabetes mellitus tipo 2 comórbida y trastorno depresivo. Para ello, se diseñó y posteriormente se realizó un estudio piloto de nueve meses. En el estudio analizamos la Tasa de Posesión de Medicamentos (MPR, por su nombre en inglés, Medication Possession Ratio), obtuvimos la puntuación del Cuestionario sobre la Salud del Paciente (PHQ-9, por su nombre en inglés, Patient Health Questionnaire) y medimos el nivel de hemoglobina glicosilada (HbA1c), en los pacientes antes y después del estudio. También realizamos entrevistas a todos los participantes. Un total de trece pacientes y cinco enfermeras utilizaron y evaluaron el asistente virtual propuesto. Los resultados mostraron que, en promedio, la adherencia a la medicación de los pacientes mejoró. El segundo estudio tiene como objetivo evaluar un año de uso entre el asistente virtual y pacientes con psoriasis y dermatólogos, y el impacto en su calidad de vida. Para ello se diseñó y realizó un estudio prospectivo de un año de duración con pacientes con psoriasis y dermatólogos. Para medir la mejora en la calidad de vida, en este estudio analizamos los cuestionarios de Calidad de Vida de los Pacientes con Psoriasis (PSOLIFE, por su nombre en inglés, Psoriasis Quality of Life) y el Índice de Calidad de Vida en Dermatología (DLQI, por su nombre en inglés, Dermatology Life Quality Index). Además, realizamos encuestas a todos los participantes y obtuvimos el número de consultas médicas realizadas a través del asistente virtual. Se incluyeron en el estudio un total de 34 participantes (30 pacientes diagnosticados con psoriasis moderada-grave y cuatro profesionales sanitarios). Los resultados mostraron que, en promedio, la calidad de vida mejoró.<br /

Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook

In recent years, reinforcement learning and bandits have transformed a wide range of real-world applications including healthcare, finance, recommendation systems, robotics, and last but not least, the speech and natural language processing. While most speech and language applications of reinforcement learning algorithms are centered around improving the training of deep neural networks with its flexible optimization properties, there are still many grounds to explore to utilize the benefits of reinforcement learning, such as its reward-driven adaptability, state representations, temporal structures and generalizability. In this survey, we present an overview of recent advancements of reinforcement learning and bandits, and discuss how they can be effectively employed to solve speech and natural language processing problems with models that are adaptive, interactive and scalable.Comment: To appear in Expert Systems with Applications. Accompanying INTERSPEECH 2022 Tutorial on the same topic. Including latest advancements in large language models (LLMs

Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors

In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding. This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models. The proposed method combines the back transcription procedure with a fine-grained technique for categorizing the errors that affect the performance of NLU models. The method relies on the usage of synthesized speech for NLU evaluation. We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in a significant way.Comment: Accepted to EMNLP 2023 main conferenc

Do (and say) as I say: Linguistic adaptation in human-computer dialogs

© Theodora Koulouri, Stanislao Lauria, and Robert D. Macredie. This article has been made available through the Brunel Open Access Publishing Fund.There is strong research evidence showing that people naturally align to each other’s vocabulary, sentence structure, and acoustic features in dialog, yet little is known about how the alignment mechanism operates in the interaction between users and computer systems let alone how it may be exploited to improve the efficiency of the interaction. This article provides an account of lexical alignment in human–computer dialogs, based on empirical data collected in a simulated human–computer interaction scenario. The results indicate that alignment is present, resulting in the gradual reduction and stabilization of the vocabulary-in-use, and that it is also reciprocal. Further, the results suggest that when system and user errors occur, the development of alignment is temporarily disrupted and users tend to introduce novel words to the dialog. The results also indicate that alignment in human–computer interaction may have a strong strategic component and is used as a resource to compensate for less optimal (visually impoverished) interaction conditions. Moreover, lower alignment is associated with less successful interaction, as measured by user perceptions. The article distills the results of the study into design recommendations for human–computer dialog systems and uses them to outline a model of dialog management that supports and exploits alignment through mechanisms for in-use adaptation of the system’s grammar and lexicon