Speech-centric multimodal interaction for easy-to-access online services: A personal life assistant for the elderly

The PaeLife project is a European industry-academia collaboration whose goal is to provide the elderly with easy access to online services that make their life easier and encourage their continued participation in the society. To reach this goal, the project partners are developing a multimodal virtual personal life assistant (PLA) offering a wide range of services from weather information to social networking. This paper presents the multimodal architecture of the PLA, the services provided by the PLA, and the work done in the area of speech input and output modalities, which play a key role in the application.info:eu-repo/semantics/publishedVersio

Speech-centric multimodal interaction for easy-to-access online services: A personal life assistant for the elderly

The PaeLife project is a European industry-academia collaboration whose goal is to provide the elderly with easy access to online services that make their life easier and encourage their continued participation in the society. To reach this goal, the project partners are developing a multimodal virtual personal life assistant (PLA) offering a wide range of services from weather information to social networking. This paper presents the multimodal architecture of the PLA, the services provided by the PLA, and the work done in the area of speech input and output modalities, which play a key role in the application.info:eu-repo/semantics/publishedVersio

Multilingual speech recognition for the elderly: The AALFred personal life assistant

The PaeLife project is a European industry-academia collaboration in the framework of the Ambient Assisted Living Joint Programme (AAL JP), with a goal of developing a multimodal, multilingual virtual personal life assistant to help senior citizens remain active and socially integrated. Speech is one of the key interaction modalities of AALFred, the Windows application developed in the project; the application can be controlled using speech input in four European languages: French, Hungarian, Polish and Portuguese. This paper briefly presents the personal life assistant and then focuses on the speech-related achievements of the project. These include the collection, transcription and annotation of large corpora of elderly speech, the development of automatic speech recognisers optimised for elderly speakers, a speech modality component that can easily be reused in other applications, and an automatic grammar translation service that allows for fast expansion of the automatic speech recognition functionality to new languages.info:eu-repo/semantics/publishedVersio

“Read That Article”: Exploring synergies between gaze and speech interaction

Gaze information has the potential to benefit Human-Computer Interaction (HCI) tasks, particularly when combined with speech. Gaze can improve our understanding of the user intention, as a secondary input modality, or it can be used as the main input modality by users with some level of permanent or temporary impairments. In this paper we describe a multimodal HCI system prototype which supports speech, gaze and the combination of both. The system has been developed for Active Assisted Living scenarios.info:eu-repo/semantics/acceptedVersio

Value Co-Creation in Smart Services: A Functional Affordances Perspective on Smart Personal Assistants

In the realm of smart services, smart personal assistants (SPAs) have become a popular medium for value co-creation between service providers and users. The market success of SPAs is largely based on their innovative material properties, such as natural language user interfaces, machine learning-powered request handling and service provision, and anthropomorphism. In different combinations, these properties offer users entirely new ways to intuitively and interactively achieve their goals and thus co-create value with service providers. But how does the nature of the SPA shape value co-creation processes? In this paper, we look through a functional affordances lens to theorize about the effects of different types of SPAs (i.e., with different combinations of material properties) on users’ value co-creation processes. Specifically, we collected SPAs from research and practice by reviewing scientific literature and web resources, developed a taxonomy of SPAs’ material properties, and performed a cluster analysis to group SPAs of a similar nature. We then derived 2 general and 11 cluster-specific propositions on how different material properties of SPAs can yield different affordances for value co-creation. With our work, we point out that smart services require researchers and practitioners to fundamentally rethink value co-creation as well as revise affordances theory to address the dynamic nature of smart technology as a service counterpart

Interacção multimodal : contribuições para simplificar o desenvolvimento de aplicações

Doutoramento em Engenharia InformáticaA forma como interagimos com os dispositivos que nos rodeiam, no nosso diaa- dia, está a mudar constantemente, consequência do aparecimento de novas tecnologias e métodos que proporcionam melhores e mais aliciantes formas de interagir com as aplicações. No entanto, a integração destas tecnologias, para possibilitar a sua utilização alargada, coloca desafios significativos e requer, da parte de quem desenvolve, um conhecimento alargado das tecnologias envolvidas. Apesar de a literatura mais recente apresentar alguns avanços no suporte ao desenho e desenvolvimento de sistemas interactivos multimodais, vários aspectos chave têm ainda de ser resolvidos para que se atinja o seu real potencial. Entre estes aspectos, um exemplo relevante é o da dificuldade em desenvolver e integrar múltiplas modalidades de interacção. Neste trabalho, propomos, desenhamos e implementamos uma framework que permite um mais fácil desenvolvimento de interacção multimodal. A nossa proposta mantém as modalidades de interacção completamente separadas da aplicação, permitindo um desenvolvimento, independente de cada uma das partes. A framework proposta já inclui um conjunto de modalidades genéricas e módulos que podem ser usados em novas aplicações. De entre as modalidades genéricas, a modalidade de voz mereceu particular atenção, tendo em conta a relevância crescente da interacção por voz, por exemplo em cenários como AAL, e a complexidade associada ao seu desenvolvimento. Adicionalmente, a nossa proposta contempla ainda o suporte à gestão de aplicações multi-dispositivo e inclui um método e respectivo módulo para criar fusão entre eventos. O desenvolvimento da arquitectura e da framework ocorreu num contexto de I&D diversificado, incluindo vários projectos, cenários de aplicação e parceiros internacionais. A framework permitiu o desenho e desenvolvimento de um conjunto alargado de aplicações multimodais, sendo um exemplo digno de nota o assistente pessoal AALFred, do projecto PaeLife. Estas aplicações, por sua vez, serviram um contínuo melhoramento da framework, suportando a recolha iterativa de novos requisitos, e permitido demonstrar a sua versatilidade e capacidades.The way we interact with the devices around us, in everyday life, is constantly changing, boosted by emerging technologies and methods, providing better and more engaging ways to interact with applications. Nevertheless, the integration with these technologies, to enable their widespread use in current systems, presents a notable challenge and requires considerable knowhow from developers. While the recent literature has made some advances in supporting the design and development of multimodal interactive systems, several key aspects have yet to be addressed to enable its full potential. Among these, a relevant example is the difficulty to develop and integrate multiple interaction modalities. In this work, we propose, design and implement a framework enabling easier development of multimodal interaction. Our proposal fully decouples the interaction modalities from the application, allowing the separate development of each part. The proposed framework already includes a set of generic modalities and modules ready to be used in novel applications. Among the proposed generic modalities, the speech modality deserved particular attention, attending to the increasing relevance of speech interaction, for example in scenarios such as AAL, and the complexity behind its development. Additionally, our proposal also tackles the support for managing multi-device applications and includes a method and corresponding module to create fusion of events. The development of the architecture and framework profited from a rich R&D context including several projects, scenarios, and international partners. The framework successfully supported the design and development of a wide set of multimodal applications, a notable example being AALFred, the personal assistant of project PaeLife. These applications, in turn, served the continuous improvement of the framework by supporting the iterative collection of novel requirements, enabling the proposed framework to show its versatility and potential

Evolução de uma plataforma de interação multimodal e aplicações

Mestrado em Engenharia de Computadores e TelemáticaA investigação em Interação Humano-Computador (IHC) explora a criação de novos cenários e formas de utilização de dispositivos, permitindo que aqueles com menos capacidades ou deficiências possam também utilizar a tecnologia. Projetos como o European AAL PaeLife ou IRIS exploram o uso de múltiplas modalidades para atingir esse objetivo. Esta tese apresenta uma evolução da plataforma multimodal existente e utilizada nos projetos AAL anteriores com os seguintes três principais pontos: adição de um novo componente de modalidade à lista de modalidades já disponíveis que permite a interação com o olhar; a criação de um sistema de pesquisa para encontrar outros dispositivos que executam a mesma plataforma multimodal possibilitando a troca de contexto entre os dois dispositivos; permitir que os componentes de modalidade existentes possam ser usados em conjunto com a nova modalidade de olhar através da criação de um processo de fusão na plataforma. Estas melhorias foram apresentadas em cenários relacionados aos usados no PaeLife e IRIS, para os idosos e para uma criança com uma desordem do espectro autista.The research in Human-Computer Interaction (HCI) explores the creation of new scenarios and forms of using devices, enabling those with less capacities or impaired to also use technology. Projects such as the European AAL PaeLife or IRIS explore the use of multiple modalities to achieve that goal. This thesis presents an enhancement to the existing multimodal framework used on the previous AAL project with the main three points: addition of a new modality component to the list of available modalities that allow the interaction using gaze; the creation of a search system for finding other devices running the same multimodal framework and enable the exchange of context between the two devices; enable the existing modality components to be used together with the new gaze modality by adding a fusion process to the framework. These improvements were presented in scenarios related to the ones used on PaeLife and IRIS, for the elderly and for a child with an autistic spectrum disorder

A Review on Usability and User Experience of Assistive Social Robots for Older Persons

In the advancement of human-robot interaction technology, assistive social robots have been recognized as one of potential technologies that can provide physical and cognitive supports in older persons care. However, a major challenge faced by the designers is to develop an assistive social robot with prodigious usability and user experience for older persons who were known to have physical and cognitive limitations. A considerable number of published literatures was reporting on the technological design process of assistive social robots. However, only a small amount of attention has been paid to review the usability and user experience of the robots. The objective of this paper is to provide an overview of established researches in the literatures concerning usability and user experience issues faced by the older persons when interacting with assistive social robots. The authors searched relevant articles from the academic databases such as Google Scholar, Scopus and Web of Science as well as Google search for the publication period 2000 to 2021. Several search keywords were typed such as ‘older persons’ ‘elderly’, ‘senior citizens’, ‘assistive social robots’, ‘companion robots’, ‘personal robots’, ‘usability’ and ‘user experience’. This online search found a total of 215 articles which are related to assistive social robots in elderly care. Out of which, 54 articles identified as significant references, and they were examined thoroughly to prepare the main content of this paper. This paper reveals usability issues of 28 assistive social robots, and feedbacks of user experience based on 41 units of assistive social robots. Based on the research articles scrutinized, the authors concluded that the key elements in the design and development of assistive social robots to improve acceptance of older persons were determined by three factors: functionality, usability and users’ experience. Functionality refers to ability of robots to serve the older persons. Usability is ease of use of the robots. It is an indicator on how successful of interaction between the robots and the users. To improve usability, robot designers should consider the limitations of older persons such as vision, hearing, and cognition capabilities when interacting with the robots. User experience reflects to perceptions, preferences and behaviors of users that occur before, during and after use the robots. Combination of superior functionality and usability lead to a good user experience in using the robots which in the end achieves satisfaction of older persons