14,299 research outputs found
Cognitive Principles in Robust Multimodal Interpretation
Multimodal conversational interfaces provide a natural means for users to
communicate with computer systems through multiple modalities such as speech
and gesture. To build effective multimodal interfaces, automated interpretation
of user multimodal inputs is important. Inspired by the previous investigation
on cognitive status in multimodal human machine interaction, we have developed
a greedy algorithm for interpreting user referring expressions (i.e.,
multimodal reference resolution). This algorithm incorporates the cognitive
principles of Conversational Implicature and Givenness Hierarchy and applies
constraints from various sources (e.g., temporal, semantic, and contextual) to
resolve references. Our empirical results have shown the advantage of this
algorithm in efficiently resolving a variety of user references. Because of its
simplicity and generality, this approach has the potential to improve the
robustness of multimodal input interpretation
Semantic Virtual Environments with Adaptive Multimodal Interfaces
We present a system for real-time configuration of multimodal interfaces to Virtual Environments (VE). The flexibility of our tool is supported by a semantics-based representation of VEs. Semantic descriptors are used to define interaction devices and virtual entities under control. We use portable (XML) descriptors to define the I/O channels of a variety of interaction devices. Semantic description of virtual objects turns them into reactive entities with whom the user can communicate in multiple ways. This article gives details on the semantics-based representation and presents some examples of multimodal interfaces created with our system, including gestures-based and PDA-based interfaces, amongst others
Sharing Human-Generated Observations by Integrating HMI and the Semantic Sensor Web
Current âInternet of Thingsâ concepts point to a future where connected objects gather meaningful information about their environment and share it with other objects and people. In particular, objects embedding Human Machine Interaction (HMI), such as mobile devices and, increasingly, connected vehicles, home appliances, urban interactive infrastructures, etc., may not only be conceived as sources of sensor information, but, through interaction with their users, they can also produce highly valuable context-aware human-generated observations. We believe that the great promise offered by combining and sharing all of the different sources of information available can be realized through the integration of HMI and Semantic Sensor Web technologies. This paper presents a technological framework that harmonizes two of the most influential HMI and Sensor Web initiatives: the W3Câs Multimodal Architecture and Interfaces (MMI) and the Open Geospatial Consortium (OGC) Sensor Web Enablement (SWE) with its semantic extension, respectively. Although the proposed framework is general enough to be applied in a variety of connected objects integrating HMI, a particular development is presented for a connected car scenario where driversâ observations about the traffic or their environment are shared across the Semantic Sensor Web. For implementation and evaluation purposes an on-board OSGi (Open Services Gateway Initiative) architecture was built, integrating several available HMI, Sensor Web and Semantic Web technologies. A technical performance test and a conceptual validation of the scenario with potential users are reported, with results suggesting the approach is soun
Multimodal agent interfaces and system architectures for health and fitness companions
Multimodal conversational spoken dialogues using physical and virtual agents provide a potential interface to motivate and support users in the domain of health and fitness. In this paper we present how such multimodal conversational Companions can be implemented to support their owners in various pervasive and mobile settings. In particular, we focus on different forms of multimodality and system architectures for such interfaces
Meetings and Meeting Modeling in Smart Environments
In this paper we survey our research on smart meeting rooms and its relevance for augmented reality meeting support and virtual reality generation of meetings in real time or off-line. The research reported here forms part of the European 5th and 6th framework programme projects multi-modal meeting manager (M4) and augmented multi-party interaction (AMI). Both projects aim at building a smart meeting environment that is able to collect multimodal captures of the activities and discussions in a meeting room, with the aim to use this information as input to tools that allow real-time support, browsing, retrieval and summarization of meetings. Our aim is to research (semantic) representations of what takes place during meetings in order to allow generation, e.g. in virtual reality, of meeting activities (discussions, presentations, voting, etc.). Being able to do so also allows us to look at tools that provide support during a meeting and at tools that allow those not able to be physically present during a meeting to take part in a virtual way. This may lead to situations where the differences between real meeting participants, human-controlled virtual participants and (semi-) autonomous virtual participants disappear
MIRIAM: A Multimodal Chat-Based Interface for Autonomous Systems
We present MIRIAM (Multimodal Intelligent inteRactIon for Autonomous
systeMs), a multimodal interface to support situation awareness of autonomous
vehicles through chat-based interaction. The user is able to chat about the
vehicle's plan, objectives, previous activities and mission progress. The
system is mixed initiative in that it pro-actively sends messages about key
events, such as fault warnings. We will demonstrate MIRIAM using SeeByte's
SeeTrack command and control interface and Neptune autonomy simulator.Comment: 2 pages, ICMI'17, 19th ACM International Conference on Multimodal
Interaction, November 13-17 2017, Glasgow, U
- âŠ