Interacção multimodal : contribuições para simplificar o desenvolvimento de aplicações

Doutoramento em Engenharia InformáticaA forma como interagimos com os dispositivos que nos rodeiam, no nosso diaa- dia, está a mudar constantemente, consequência do aparecimento de novas tecnologias e métodos que proporcionam melhores e mais aliciantes formas de interagir com as aplicações. No entanto, a integração destas tecnologias, para possibilitar a sua utilização alargada, coloca desafios significativos e requer, da parte de quem desenvolve, um conhecimento alargado das tecnologias envolvidas. Apesar de a literatura mais recente apresentar alguns avanços no suporte ao desenho e desenvolvimento de sistemas interactivos multimodais, vários aspectos chave têm ainda de ser resolvidos para que se atinja o seu real potencial. Entre estes aspectos, um exemplo relevante é o da dificuldade em desenvolver e integrar múltiplas modalidades de interacção. Neste trabalho, propomos, desenhamos e implementamos uma framework que permite um mais fácil desenvolvimento de interacção multimodal. A nossa proposta mantém as modalidades de interacção completamente separadas da aplicação, permitindo um desenvolvimento, independente de cada uma das partes. A framework proposta já inclui um conjunto de modalidades genéricas e módulos que podem ser usados em novas aplicações. De entre as modalidades genéricas, a modalidade de voz mereceu particular atenção, tendo em conta a relevância crescente da interacção por voz, por exemplo em cenários como AAL, e a complexidade associada ao seu desenvolvimento. Adicionalmente, a nossa proposta contempla ainda o suporte à gestão de aplicações multi-dispositivo e inclui um método e respectivo módulo para criar fusão entre eventos. O desenvolvimento da arquitectura e da framework ocorreu num contexto de I&D diversificado, incluindo vários projectos, cenários de aplicação e parceiros internacionais. A framework permitiu o desenho e desenvolvimento de um conjunto alargado de aplicações multimodais, sendo um exemplo digno de nota o assistente pessoal AALFred, do projecto PaeLife. Estas aplicações, por sua vez, serviram um contínuo melhoramento da framework, suportando a recolha iterativa de novos requisitos, e permitido demonstrar a sua versatilidade e capacidades.The way we interact with the devices around us, in everyday life, is constantly changing, boosted by emerging technologies and methods, providing better and more engaging ways to interact with applications. Nevertheless, the integration with these technologies, to enable their widespread use in current systems, presents a notable challenge and requires considerable knowhow from developers. While the recent literature has made some advances in supporting the design and development of multimodal interactive systems, several key aspects have yet to be addressed to enable its full potential. Among these, a relevant example is the difficulty to develop and integrate multiple interaction modalities. In this work, we propose, design and implement a framework enabling easier development of multimodal interaction. Our proposal fully decouples the interaction modalities from the application, allowing the separate development of each part. The proposed framework already includes a set of generic modalities and modules ready to be used in novel applications. Among the proposed generic modalities, the speech modality deserved particular attention, attending to the increasing relevance of speech interaction, for example in scenarios such as AAL, and the complexity behind its development. Additionally, our proposal also tackles the support for managing multi-device applications and includes a method and corresponding module to create fusion of events. The development of the architecture and framework profited from a rich R&D context including several projects, scenarios, and international partners. The framework successfully supported the design and development of a wide set of multimodal applications, a notable example being AALFred, the personal assistant of project PaeLife. These applications, in turn, served the continuous improvement of the framework by supporting the iterative collection of novel requirements, enabling the proposed framework to show its versatility and potential

Multilingual speech recognition for the elderly: The AALFred personal life assistant

The PaeLife project is a European industry-academia collaboration in the framework of the Ambient Assisted Living Joint Programme (AAL JP), with a goal of developing a multimodal, multilingual virtual personal life assistant to help senior citizens remain active and socially integrated. Speech is one of the key interaction modalities of AALFred, the Windows application developed in the project; the application can be controlled using speech input in four European languages: French, Hungarian, Polish and Portuguese. This paper briefly presents the personal life assistant and then focuses on the speech-related achievements of the project. These include the collection, transcription and annotation of large corpora of elderly speech, the development of automatic speech recognisers optimised for elderly speakers, a speech modality component that can easily be reused in other applications, and an automatic grammar translation service that allows for fast expansion of the automatic speech recognition functionality to new languages.info:eu-repo/semantics/publishedVersio

PRESTK : situation-aware presentation of messages and infotainment content for drivers

The amount of in-car information systems has dramatically increased over the last few years. These potentially mutually independent information systems presenting information to the driver increase the risk of driver distraction. In a first step, orchestrating these information systems using techniques from scheduling and presentation planning avoid conflicts when competing for scarce resources such as screen space. In a second step, the cognitive capacity of the driver as another scarce resource has to be considered. For the first step, an algorithm fulfilling the requirements of this situation is presented and evaluated. For the second step, I define the concept of System Situation Awareness (SSA) as an extension of Endsley’s Situation Awareness (SA) model. I claim that not only the driver needs to know what is happening in his environment, but also the system, e.g., the car. In order to achieve SSA, two paths of research have to be followed: (1) Assessment of cognitive load of the driver in an unobtrusive way. I propose to estimate this value using a model based on environmental data. (2) Developing model of cognitive complexity induced by messages presented by the system. Three experiments support the claims I make in my conceptual contribution to this field. A prototypical implementation of the situation-aware presentation management toolkit PRESTK is presented and shown in two demonstrators.In den letzten Jahren hat die Menge der informationsanzeigenden Systeme im Auto drastisch zugenommen. Da sie potenziell unabhängig voneinander ablaufen, erhöhen sie die Gefahr, die Aufmerksamkeit des Fahrers abzulenken. Konflikte entstehen, wenn zwei oder mehr Systeme zeitgleich auf limitierte Ressourcen wie z. B. den Bildschirmplatz zugreifen. Ein erster Schritt, diese Konflikte zu vermeiden, ist die Orchestrierung dieser Systeme mittels Techniken aus dem Bereich Scheduling und Präsentationsplanung. In einem zweiten Schritt sollte die kognitive Kapazität des Fahrers als ebenfalls limitierte Ressource berücksichtigt werden. Der Algorithmus, den ich zu Schritt 1 vorstelle und evaluiere, erfüllt alle diese Anforderungen. Zu Schritt 2 definiere ich das Konzept System Situation Awareness (SSA), basierend auf Endsley’s Konzept der Situation Awareness (SA). Dadurch wird erreicht, dass nicht nur der Fahrer sich seiner Umgebung bewusst ist, sondern auch das System (d.h. das Auto). Zu diesem Zweck m¨ussen zwei Bereiche untersucht werden: (1) Die kognitive Belastbarkeit des Fahrers unaufdringlich ermitteln. Dazu schlage ich ein Modell vor, das auf Umgebungsinformationen basiert. (2) Ein weiteres Modell soll die Komplexität der präsentierten Informationen bestimmen. Drei Experimente stützen die Behauptungen in meinem konzeptuellen Beitrag. Ein Prototyp des situationsbewussten Präsentationsmanagement-Toolkits PresTK wird vorgestellt und in zwei Demonstratoren gezeigt

Metadata for user-centred, inclusive access to digital resources: realising the theory of AccessForAll Accessibility

To be inclusive, the Web needs published resources to be matched to individual users’ needs and preferences for their perception and control. In a decade, this has not been achieved and many cannot make use of resources despite having appropriate facilities. This thesis argues that the necessary management of resources can be achieved with well-designed metadata. Demonstration and explanation of the accessibility problems, efforts to solve them and the current state of inaccessibility of Web resources, any resource that is available through the World Wide Web, is fundamental to the research. The author relies heavily on Dublin Core metadata as it is relatively easy to use; is probably the most populous metadata; can be managed with free software systems, and for commercial reasons. The research investigated what makes DC metadata, so apparently simple, powerful enough to be the most popular metadata because there is very little available that explains this. The thesis then documents the scientific view of metadata upon which effective use of metadata can be based in the context of accessibility. It argues, at a practical level, that metadata is essential and integral to any shift to an on-going process approach to accessibility. It contributes to the science of metadata in as much as it analyses, synthesizes, and articulates the characteristics of an essential infrastructure for a new approach to accessibility. The author argues in favour of an on-going process approach to accessibility of resources that supports continuous improvement of any given resource, not necessarily by the author of the resource, and not necessarily by design or with knowledge of the original author, by contributors who may be distributed globally. The thesis argues that the current dependence on production guidelines and post-production evaluation of resources as either universally accessible or otherwise, does not adequately provide for either the accessibility necessary for individuals or the continuous or evolutionary approach possible within the current Web environment. It argues that a distributed, social-networking view of the Web as interactive, combined with a social model of disability, given the management tools of machine-readable, interoperable AccessForAll metadata, as developed, can achieve the desired goals. It raises issues regarding its implementation in the distributed environment of the Web

Ubiquitous Computing

The aim of this book is to give a treatment of the actively developed domain of Ubiquitous computing. Originally proposed by Mark D. Weiser, the concept of Ubiquitous computing enables a real-time global sensing, context-aware informational retrieval, multi-modal interaction with the user and enhanced visualization capabilities. In effect, Ubiquitous computing environments give extremely new and futuristic abilities to look at and interact with our habitat at any time and from anywhere. In that domain, researchers are confronted with many foundational, technological and engineering issues which were not known before. Detailed cross-disciplinary coverage of these issues is really needed today for further progress and widening of application range. This book collects twelve original works of researchers from eleven countries, which are clustered into four sections: Foundations, Security and Privacy, Integration and Middleware, Practical Applications

Framework for ubiquitous and voice enabled web applicattions development.

RESUMEN La cantidad de dispositivos con capacidad de conexión a Internet crece rápidamente. En la actualidad se dispone de teléfonos móviles basados en tecnología WAP (Wireless Application Protocol) o I-Mode, Agendas Digitales Personales, Kioskos Internet, teléfonos convencionales basados en acceso a la Web por medio de la voz, dispositivos basados en televisión interactiva, electrodomésticos, entre otros. Desarrollar una versión de una aplicación web para cada uno de los dispositivos con conectividad a la Web resulta inviable. Por otra parte, desarrollar aplicaciones web que puedan ser visualizados en forma apropiada y aprovechando al máximo las capacidades del dispositivo se constituye en una tarea compleja. En esta tesis se propone un framework, entendido como un marco de trabajo genérico, que sirva como guía para el desarrollo de portales web pervasivos que puedan ser accedidos desde múltiples dispositivos, evitando el desarrollo de un portal por cada uno y teniendo en cuenta las grandes variaciones pueden existir en sus capacidades. Adicionalmente se ha planteado un modelo de agrupamiento de dispositivos, que permita definir una serie de grupos, así como las características asociadas a los mismos, en forma tal que puedan generarse posteriormente los formatos asociados a estos grupos de dispositivos y no a elementos individuales y se ha propuesto y validado una arquitectura de referencia para el desarrollo de aplicaciones pervasivas, que no genere dependencia de tecnologías de servidor, y que permita incorporar la solución de agrupamiento planteada previamente. ____________________________________________________________________________________________________The purpose of the Ubiquitous or Pervasive Computing - an emergent paradigm of personalized computation- is to obtain device interoperability under different conditions. The devices were designed for different purposes by different companies or from different technological generations. The ever increasing market of web enabled devices has brought up diverse challenges related to the difficulty of visualizing content in a unified form to diverse clients, while at the same time taking into account the great differences in the capacities of these devices. It is not feasible to develop a separate application for each of these devices, simply because the number of different devices is too high and still growing. In the analysis of existing proposals dealing with the modelling of ubiquitous web applications, the link that exists between the logical and conceptual modelling and the physical modelling of the applications is not clear enough, and the way in which the context aspects related to web access from these devices cannot be specified. On the other hand, the available commercial products are supplier-specific. Every future platform change would a costly and painstaking process In this thesis we present a proposal of a framework for the development of web applications that can be accessed from different types of devices, such as PCs, PDAs, mobile phones based on diverse technologies (like WAP and I-Mode) and conventional telephones that access the web through voice gateways and voice portals. The proposed framework serves as a guide for the development of this type of applications and it can be deployed to different server configurations and software development technologies. In order to obtain this objective, a description of diverse theoretical elements related to dynamic generation of information that can be acceded by devices has been made, as well as a description of involved technologies whose hardware, software and connectivity characteristics vary remarkably. The theoretical study was carried out in parallel with tests based on the different technologies used. A multilingual-ubiquitous traffic information portal was used to test the theory in an operational environment

The SEMAINE API : a component integration framework for a naturally interacting and emotionally competent embodied conversational agent

The present thesis addresses the topic area of Embodied Conversational Agents (ECAs) with capabilities for natural interaction with a human user and emotional competence with respect to the perception and generation of emotional expressivity. The focus is on the technological underpinnings that facilitate the implementation of a real-time system with these capabilities, built from re-usable components. The thesis comprises three main contributions. First, it describes a new component integration framework, the SEMAINE API, which makes it easy to build emotion-oriented systems from components which interact with one another using standard and pre-standard XML representations. Second, it presents a prepare-and-trigger system architecture which substantially speeds up the time to animation for system utterances that can be pre-planned. Third, it reports on the W3C Emotion Markup Language, an upcoming web standard for representing emotions in technological systems. We assess critical aspects of system performance, showing that the framework provides a good basis for implementing real-time interactive ECA systems, and illustrate by means of three examples that the SEMAINE API makes it is easy to build new emotion-oriented systems from new and existing components.Die vorliegende Dissertation behandelt das Thema der virtuellen Agenten mit Fähigkeiten zur natürlichen Benutzer-Interaktion sowie emotionaler Kompetenz bzgl. der Wahrnehmung und Generierung emotionalen Ausdrucks. Der Schwerpunkt der Arbeit liegt auf den technologischen Grundlagen für die Implementierung eines echtzeitfähigen Systems mit diesen Fähigkeiten, das aus wiederverwendbaren Komponenten erstellt werden kann. Die Arbeit umfasst drei Kernaspekte. Zum Einen beschreibt sie ein neues Framework zur Komponenten-Integration, die SEMAINE API: Diese erleichtert die Erstellung von Emotions-orientierten Systemen aus Komponenten, die untereinander mittels Standard- oder Prä-Standard-Repräsentationen kommunizieren. Zweitens wird eine Systemarchitektur vorgestellt, welche Vorbereitung und Auslösung von Systemverhalten entkoppelt und so zu einer substanziellen Beschleunigung der Generierungszeit führt, wenn Systemäußerungen im Voraus geplant werden können. Drittens beschreibt die Arbeit die W3C Emotion Markup Language, einen werdenden Web-Standard zur Repräsentation von Emotionen in technologischen Systemen. Es werden kritische Aspekte der Systemperformanz untersucht, wodurch gezeigt wird, dass das Framework eine gute Basis für die Implementierung echtzeitfähiger interaktiver Agentensysteme darstellt. Anhand von drei Beispielen wird illustriert, dass mit der SEMAINE API leicht neue Emotions-orientierte Systeme aus neuen und existierenden Komponenten erstellt werden können

The web as a runtime in mobile context

Web-teknologiat kehitettiin alun perin kuvaamaan staattisten web-sivujen sisältöä. Web-selainten suosion vuoksi samoja teknologioita hyödynnetään nykyisin myös sovellusten toteuttamiseen käyttäen web-selainta niiden suorittamiseen vuorovaikutteisesti. Web-teknologioiden suosiosta huolimatta ne sisältävät useita ongelmia ohjelmistojen toteuttamisen näkökulmasta. Lisäksi mobiililaitteiden rajoitukset tekevät kaikilla laitteilla toimivien sovellusten toteuttamisesta haasteellista. Tämän vuoksi uusia ohjelmistoalustoja on kehitetty ratkaisemaan web-selainten asettamia rajoituksia. Tässä diplomityössä koostetaan vaatimukset mobiililaitteissa toimiville web-sovelluksille. Tämän lisäksi web-sovelluksille luodaan luokitusjärjestelmä ja tärkeimmät web-sovelluskehitykseen liittyvät web-teknologiat ja ohjelmistoalustat esitellään. Esimerkkisovellus toteutetaan käyttäen web-teknologioita ja hyödyntäen ohjelmistoalustan ominaisuuksia. Lisäksi esimerkkisovellus arvioidaan määritettyä vaatimusmäärittelyä vastaan. Arviointi paljasti lukuisia haasteita, jotka liittyivät web-teknologioiden puutteelliseen ilmaisuvoimaan, yhteentoimivuuteen ja ohjelmistoalustan toiminnallisuuteen. Tästä huolimatta esimerkkisovellus toteutti sille asetetut vaatimukset työpöytäohjelmistotasoisesta toiminnallisuudesta.Web technologies were initially designed to facilitate the creation of static web pages. However, the ubiquity of the web browser has motivated the use of the same technologies as a basis for desktop-style applications which are executed within the web browser and have their characteristics such as high interactivity. Despite the popularity of web applications, there exists various problems due to the fact that established web technologies were not specified with applications in mind. In addition, the constraints introduced by mobile devices challenge the ubiquity of such applications. On this account, new platforms have emerged extending the capabilities of web browsers. In this thesis, the requirements for client-side web applications in mobile context are synthesized. Moreover, a taxonomy for web applications is drawn and client-side web technologies and major software platforms relevant to the client-side web applications are discussed. Furthermore, an application concept implementation developed using web technologies leveraging the capabilities of the major mobile platform is presented and evaluated against the defined requirements. The evaluation revealed various problems related to limited expressiveness of web technologies, interoperability and platform functionality. Regardless, the implementation provided a level of functionality comparable to that of native applications