Multimodal Fusion as Communicative Acts during Human-Robot Interaction

Research on dialog systems is a very active area in social robotics. During the last two decades, these systems have evolved from those based only on speech recognition and synthesis to the current and modern systems, which include new components and multimodality. By multimodal dialogue we mean the interchange of information among several interlocutors, not just using their voice as the mean of transmission but also all the available channels such as gestures, facial expressions, touch, sounds, etc. These channels add information to the message to be transmitted in every dialogue turn. The dialogue manager (IDiM) is one of the components of the robotic dialog system (RDS) and is in charge of managing the dialogue flow during the conversational turns. In order to do that, it is necessary to coherently treat the inputs and outputs of information that flow by different communication channels: audio, vision, radio frequency, touch, etc. In our approach, this multichannel input of information is temporarily fused into communicative acts (CAs). Each CA groups the information that flows through the different input channels into the same pack, transmitting a unique message or global idea. Therefore, this temporary fusion of information allows the IDiM to abstract from the channels used during the interaction, focusing only on the message, not on the way it is transmitted. This article presents the whole RDS and the description of how the multimodal fusion of information is made as CAs. Finally, several scenarios where the multimodal dialogue is used are presented.Comunidad de Madri

Processing and fusioning multiple heterogeneous information sources in multimodal dialog systems

Proceedings of: 17th International Conference on Information Fusion (FUSION 2014): Salamanca, Spain 7-10 July 2014.Context-aware dialog systems must be able to process very heterogeneous information sources and user input modes. In this paper we propose a method to fuse multimodal inputs into a unified representation. This representation allows the dialog manager of the system to find the best interaction strategy and also select the next system response. We show the applicability of our proposal by means of the implementation of a dialog system that considers spoken, tactile, and also information related to the context of the interaction with its users. Context information is related to the detection of user's intention during the dialog and their emotional state (internal context), and the user's location (external context).This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485).Publicad

Motivating the use of real-time multimodal travel planners: the role of users value, technophile and community resilience

Advanced traveler Information systems (ATIS) have the potential to improve travel experience, enhance personal mobility and productivity, since they allow travelers to make better informed travel decisions. However, understanding the motivators underlying individuals’ intentions to use the system is essential to evaluate the effectiveness of this solution for promoting sustainable travel trends. The aim of this paper is to find the drivers that influence decisions in using such systems on an individual level. Perceived values developed by using the new system, technophile and community resilience constructs are incorporated in the framework in order to better explain user-sided heterogeneity observed in individual behavior. The case-study focuses on a new ATIS in Copenhagen. It questions 822 Danish citizens by distributing a technology-use preference survey. Structural equation models revealed that the drivers are specific to individual users and depends on wide ranging factors that go above traditional economic and socio-demographic methods. The results show a) different value dimensions perceived by individuals for the new information system and different effects on its adoption b) a relation between different community resilience constructs and value dimensions c) a stronger appeal of using the system to individuals who are technophile and value the usefulness of the new information system to increase trip efficiency

Developing enhanced conversational agents for social virtual worlds

In This Paper, We Present A Methodology For The Development Of Embodied Conversational Agents For Social Virtual Worlds. The Agents Provide Multimodal Communication With Their Users In Which Speech Interaction Is Included. Our Proposal Combines Different Techniques Related To Artificial Intelligence, Natural Language Processing, Affective Computing, And User Modeling. A Statistical Methodology Has Been Developed To Model The System Conversational Behavior, Which Is Learned From An Initial Corpus And Improved With The Knowledge Acquired From The Successive Interactions. In Addition, The Selection Of The Next System Response Is Adapted Considering Information Stored Into User&#39 S Profiles And Also The Emotional Contents Detected In The User&#39 S Utterances. Our Proposal Has Been Evaluated With The Successful Development Of An Embodied Conversational Agent Which Has Been Placed In The Second Life Social Virtual World. The Avatar Includes The Different Models And Interacts With The Users Who Inhabit The Virtual World In Order To Provide Academic Information. The Experimental Results Show That The Agent&#39 S Conversational Behavior Adapts Successfully To The Specific Characteristics Of Users Interacting In Such Environments.Work partially supported by the Spanish CICyT Projects under grant TRA2015-63708-R and TRA2016-78886-C3-1-R

Integrating big data into a sustainable mobility policy 2.0 planning support system

It is estimated that each of us, on a daily basis, produces a bit more than 1 GB of digital content through our mobile phone and social networks activities, bank card payments, location-based positioning information, online activities, etc. However, the implementation of these large data amounts in city assets planning systems still remains a rather abstract idea for several reasons, including the fact that practical examples are still very strongly services-oriented, and are a largely unexplored and interdisciplinary field; hence, missing the cross-cutting dimension. In this paper, we describe the Policy 2.0 concept and integrate user generated content into Policy 2.0 platform for sustainable mobility planning. By means of a real-life example, we demonstrate the applicability of such a big data integration approach to smart cities planning process. Observed benefits range from improved timeliness of the data and reduced duration of the planning cycle to more informed and agile decision making, on both the citizens and the city planners end. The integration of big data into the planning process, at this stage, does not have uniform impact across all levels of decision making and planning process, therefore it should be performed gradually and with full awareness of existing limitations