158,856 research outputs found
VEMI Lab 2021
The Virtual Environments and Multimodal Interaction (VEMI) Lab embodies an inclusive, collaborative, and multi-disciplinary approach to hands-on research and education. By bringing together students and faculty from more than a dozen majors and disciplines, VEMI is uniquely positioned to advance computing and STEM initiatives both here at the university as well as in broader communities throughout Maine and nationwide
Deep multimodal fusion : combining discrete events and continuous signals
Multimodal datasets often feature a combination of continuous signals and a series of discrete events. For instance, when
studying human behaviour it is common to annotate actions
performed by the participant over several other modalities
such as video recordings of the face or physiological signals.
These events are nominal, not frequent and are not sampled
at a continuous rate while signals are numeric and often
sampled at short fixed intervals. This fundamentally different nature complicates the analysis of the relation among
these modalities which is often studied after each modality
has been summarised or reduced.
This paper investigates a novel approach to model the
relation between such modality types bypassing the need
for summarising each modality independently of each other.
For that purpose, we introduce a deep learning model based
on convolutional neural networks that is adapted to process
multiple modalities at different time resolutions we name
deep multimodal fusion. Furthermore, we introduce and
compare three alternative methods (convolution, training
and pooling fusion) to integrate sequences of events with
continuous signals within this model. We evaluate deep multimodal fusion using a game user dataset where player physiological signals are recorded in parallel with game events.
Results suggest that the proposed architecture can appropriately capture multimodal information as it yields higher
prediction accuracies compared to single-modality models.
In addition, it appears that pooling fusion, based on a novel
filter-pooling method provides the more effective fusion approach for the investigated types of data.peer-reviewe
Introduction: Multimodal interaction
That human social interaction involves the intertwined cooperation of different modalities is uncontroversial. Researchers in several allied fields have, however, only recently begun to document the precise ways in which talk, gesture, gaze, and aspects of the material surround are brought together to form coherent courses of action. The papers in this volume are attempts to develop this line of inquiry. Although the authors draw on a range of analytic, theoretical, and methodological traditions (conversation analysis, ethnography, distributed cognition, and workplace studies), all are concerned to explore and illuminate the inherently multimodal character of social interaction. Recent studies, including those collected in this volume, suggest that different modalities work together not only to elaborate the semantic content of talk but also to constitute coherent courses of action. In this introduction we present evidence for this position. We begin by reviewing some select literature focusing primarily on communicative functions and interactive organizations of specific modalities before turning to consider the integration of distinct modalities in interaction
MIRIAM: A Multimodal Chat-Based Interface for Autonomous Systems
We present MIRIAM (Multimodal Intelligent inteRactIon for Autonomous
systeMs), a multimodal interface to support situation awareness of autonomous
vehicles through chat-based interaction. The user is able to chat about the
vehicle's plan, objectives, previous activities and mission progress. The
system is mixed initiative in that it pro-actively sends messages about key
events, such as fault warnings. We will demonstrate MIRIAM using SeeByte's
SeeTrack command and control interface and Neptune autonomy simulator.Comment: 2 pages, ICMI'17, 19th ACM International Conference on Multimodal
Interaction, November 13-17 2017, Glasgow, U
Staging Transformations for Multimodal Web Interaction Management
Multimodal interfaces are becoming increasingly ubiquitous with the advent of
mobile devices, accessibility considerations, and novel software technologies
that combine diverse interaction media. In addition to improving access and
delivery capabilities, such interfaces enable flexible and personalized dialogs
with websites, much like a conversation between humans. In this paper, we
present a software framework for multimodal web interaction management that
supports mixed-initiative dialogs between users and websites. A
mixed-initiative dialog is one where the user and the website take turns
changing the flow of interaction. The framework supports the functional
specification and realization of such dialogs using staging transformations --
a theory for representing and reasoning about dialogs based on partial input.
It supports multiple interaction interfaces, and offers sessioning, caching,
and co-ordination functions through the use of an interaction manager. Two case
studies are presented to illustrate the promise of this approach.Comment: Describes framework and software architecture for multimodal web
interaction managemen
Musical Multimodal Child Computer Interaction
In this project an interactive computer system is designed that envisions to contribute to young children's musical education. From literature, requirements for musical interaction were derived. In this paper these requirements and the design of the system are described
Multimodal Interaction in a Haptic Environment
In this paper we investigate the introduction of haptics in a multimodal tutoring environment. In this environment a haptic device is used to control a virtual piece of sterile cotton and a virtual injection needle. Speech input and output is provided to interact with a virtual tutor, available as a talking head, and a virtual patient. We introduce the haptic tasks and how different agents in the multi-agent system are made responsible for them. Notes are provided about the way we introduce an affective model in the tutor agent
- …