Improving Speech Interaction in Vehicles Using Context-Aware Information through A SCXML Framework

Speech Technologies can provide important benefits for the development of more usable and safe in-vehicle human-machine interactive systems (HMIs). However mainly due robustness issues, the use of spoken interaction can entail important distractions to the driver. In this challenging scenario, while speech technologies are evolving, further research is necessary to explore how they can be complemented with both other modalities (multimodality) and information from the increasing number of available sensors (context-awareness). The perceived quality of speech technologies can significantly be increased by implementing such policies, which simply try to make the best use of all the available resources; and the in vehicle scenario is an excellent test-bed for this kind of initiatives. In this contribution we propose an event-based HMI design framework which combines context modelling and multimodal interaction using a W3C XML language known as SCXML. SCXML provides a general process control mechanism that is being considered by W3C to improve both voice interaction (VoiceXML) and multimodal interaction (MMI). In our approach we try to anticipate and extend these initiatives presenting a flexible SCXML-based approach for the design of a wide range of multimodal context-aware HMI in-vehicle interfaces. The proposed framework for HMI design and specification has been implemented in an automotive OSGi service platform, and it is being used and tested in the Spanish research project MARTA for the development of several in-vehicle interactive applications

Exploring map-based interfaces for mobile solutions in emergency work

This thesis investigates challenges and requirements related to both domain area and usability principles through investigating task-support for incident commanders in the police. To this end, domain knowledge has been collected through field studies, interviews and observation, and a prototype combining these requirements with state of the art mobile technology has been developed. The prototype has been developed to support tasks related to resource allocations. It combines the use of a map-based interface with icons with lists and forms, and uses direct manipulation as a part of the interaction. Evaluations have been conducted with both usability and domain experts, and results from the evaluations are categorized, discussed and finally used to put forward design implications. The findings of this thesis include a set of design implications deduced from (1) careful investigation of the domain area, (2) usability theories and design guidelines, and (3) evaluations of a developed prototype. The study has proven that the uniqueness and characteristics of emergency situations does not allow us to rely on design theory alone, and a combination of usability and domain expert is essential. The results from the evaluations and the design implication put forward show that the work in this field is highly feasible, yet more knowledge about the domain area is required to further facilitate for added value when solving tasks. Results also confirm that state of the art mobile devices are well-suited for decision-support within emergency response. Furthermore, the challenges, requirements and alternative solutions presented in this thesis are highly transferrable to other emergency agencies

Para uma taxonomia da multimodalidade na interacção homem-computador : proposta aberta da classificação pluridimensional

Tese de doutoramento, Belas-Artes (Design de Comunicação), Universidade de Lisboa, Faculdade de Belas-Artes, 2015A Interacção Homem-Computador tem-se assumido como uma área interdisciplinar que integra contributos de diversas áreas científicas como a engenharia, a filosofia, as ciências sociais e humanas e o design. A presente investigação tem como principal objectivo o estudo e a classificação das Interacções Homem-Computador de carácter multimodal e procura contribuir para a produção de ferramentas teóricas de concepção, análise e construção de interfaces, compreendendo a forma como agentes humanos e informáticos interagem entre si. A primeira parte, integra o enquadramento conceptual e histórico sustentado pela revisão de literatura. A segunda, apresenta propostas de sistemas de classificação de modalidades de interacção, de processos interactivos e de interacções, baseados em reflexão académica e análise prévia do estado da arte, quando existente. É dada especial atenção ao paradigma de interacção natural pelo seu carácter historicamente dominante. O défice de análise académica registado no âmbito desta temática, contribuiu para o desenvolvimento de uma investigação de carácter tipológico e exploratório.Human-computer interaction has emerged as a interdisciplinary area that integrates contributions from different scientific territories such as engineering, philosophy, psychology, and design. This conveys a need for shared conceptual tools that promote interface invention, development, and analyses. This study aims to develop a pluridimensional classification system that encompasses the maximum diversity through the consideration of the multiple classification systems that have been put forward in the analyses of multimodality in human-computer interaction. The present investigation intends to contribute with a deeper understanding on the manners that interactions develop and/or may develop between human and computer agents. The investigation comprises two parts. The first, reviews conceptual and historic back- ground supported by literature. The second, pursuits investigation goals supporting it self on the different classification dimensions portrayed in academic classification systems on the analyses of multimodality in human-computer interaction. This leads to interaction modalities, interactive processes, and interactions classification systems presentation. The natural interaction paradigm receives special attention given its dominant and historical status. The absence of relevant academic analyses in some of the main classifications dimensions promotes a typological and exploratory approach

Embarking on Multimodal Interface Design

Designers are increasingly faced with the challenge of targeting multimodal applications, those that span heterogeneous devices and use multimodal input, but do not have tools to support them. We studied the early stage work practices of professional multimodal interaction designers. We noted the variety of different artifacts produced, such as design sketches and paper prototypes. Additionally, we observed Wizard of Oz techniques that are sometimes used to simulate an interactive application from these sketches. These studies have led to our development of a technique for interface designers to consider as they embark on creating multimodal applications