Multimodale Interaktion in Virtueller und Erweiterter Realität

User interfaces that support the complementary use of at least two input modalities, e.g., speech and gestures, are called Multimodal Interfaces (MMIs). Psychological theories such as Gestalt, Cognitive Load, and Activity theories provide a comprehensive theoretical foundation that predicts various advantages of MMIs over unimodal solutions. Among these are increased expressiveness, mental efficiency, and flexibility. Major beneficiaries of such MMIs are applications that utilize Virtual-, Augmented-, and Mixed Reality technologies. These Extended Realities (XRs) are becoming increasingly relevant in several application fields, such as healthcare, education, and entertainment. However, XR's currently prevailing interaction paradigm is still (mostly) unimodal. There is a knowledge gap regarding the proper use of MMIs in practice, as evident by the scarcity of empirical research and the shortage of concrete design guidelines. This dissertation argues that this knowledge gap is caused by the lack of appropriate tool support exacerbating the inherently complex implementation effort of functional MMIs. The three research questions of this dissertation aim at identifying requirements for suitable tool support, designing and implementing a corresponding architecture, and investigating the suitability of the resulting tool support for conducting empirical research in XRs. A requirements analysis identifies five essential requirements: Respective tool support has to be able to perform (i) multimodal fusion and (ii) semantic integration, (iii) be compatible with commercially used XR platforms, (iv) support the rapid development of interfaces, and (v) provide a high degree of controllability over the created interface. Based on this analysis, this thesis presents the architecture and reference implementation of respective tool support featuring a specially designed descriptive multimodal fusion method, the concurrent Augmented Transition Network. The development of the tool support has been guided and validated through the continuous implementation of proof-of-concept XR applications and use in university teaching. Further, this dissertation presents three user studies in which MMIs created with the tool support are compared against commonly used unimodal interfaces. The studies do not only provide information about the suitability of the tool support but also show that the created MMIs perform on par or even better than their unimodal comparison partners in the examined XR use cases. A discussion of the presented results summarizes the experiences made while using the tool support concerning the postulated requirements, identifies limitations, and defines subsequent future work. The latter aims to improve developer usability. In conclusion, this dissertation shows how appropriate tool support can aid the conduct of empirical research and thus help to close the knowledge gap on MMI design, leading to better XR interfaces in the future.Benutzeroberflächen, die die komplementäre Verwendung von mindestens zwei Eingabemodalitäten unterstützen, z. B. Sprache und Gesten, werden als multimodale Schnittstellen (MMIs) bezeichnet. Psychologische Theorien wie die Gestalt-, Cognitive Load-, und Activity-Theorie bieten eine umfassende theoretische Fundierung, die verschiedene Vorteile von MMIs gegenüber unimodalen Lösungen prognostizieren. Dazu gehören eine erhöhte Ausdruckskraft, mentale Effizienz und Flexibilität. Am meisten profitieren von solchen MMIs Anwendungen, die Virtual-, Augmented- und Mixed-Reality-Technologien nutzen. Diese Extended Realities (XRs) gewinnen in verschiedenen Anwendungsbereichen wie Gesundheitswesen, Bildung und Unterhaltung zunehmend an Bedeutung. Das derzeit vorherrschende Interaktionsparadigma von XR ist jedoch (größtenteils) noch immer unimodal. Es besteht eine Wissenslücke hinsichtlich der richtigen Verwendung von MMIs in der Praxis, was sich in der Knappheit empirischer Forschung und dem Mangel an konkreten Designrichtlinien zeigt. Diese Dissertation argumentiert, dass diese Wissenslücke durch den Mangel an geeigneten Werkzeugunterstützungen verursacht wird, die den ohnehin schon komplexen Implementierungsaufwand funktionaler MMIs noch verschärfen. Die drei Forschungsfragen dieser Dissertation zielen darauf ab, die Anforderungen an eine geeignete Werkzeugunterstützung zu identifizieren, eine entsprechende Architektur zu entwerfen und zu implementieren und die Eignung der resultierenden Werkzeugunterstützung für die Durchführung empirischer Forschung in XRs zu untersuchen. Eine Anforderungsanalyse identifiziert fünf wesentliche Anforderungen: Die jeweilige Werkzeugunterstützung muss in der Lage sein, (i) multimodale Fusion und (ii) semantische Integration durchzuführen, (iii) mit kommerziell genutzten XR-Plattformen kompatibel sein, (iv) die schnelle Entwicklung von Schnittstellen unterstützen und (v) ein hohes Maß an Steuerbarkeit über die erstellte Schnittstelle bieten. Basierend auf dieser Analyse präsentiert diese Arbeit die Architektur und Referenzimplementierung der entsprechenden Werkzeugunterstützung mit einer speziell entwickelten deskriptiven multimodalen Fusionsmethode, dem Concurrent Augmented Transition Network. Die Entwicklung der Werkzeugunterstützung wurde durch die kontinuierliche Implementierung von Proof-of-Concept-XR-Anwendungen und den Einsatz in der Hochschullehre begleitet und validiert. Darüber hinaus präsentiert diese Dissertation drei Nutzungsstudien, in denen mit der Werkzeugunterstützung erstellte MMIs mit häufig verwendeten unimodalen Schnittstellen verglichen werden. Die Studien liefern nicht nur Informationen über die Eignung der Werkzeugunterstützung, sondern zeigen auch, dass die mit der Werkzeugunterstützung erstellten MMIs in den untersuchten XR-Anwendungsfällen gleichwertig oder sogar besser abschneiden als ihre unimodalen Vergleichspartner. Eine Diskussion der vorgestellten Ergebnisse fasst die Erfahrungen mit der Werkzeugunterstützung hinsichtlich der postulierten Anforderungen zusammen, identifiziert Einschränkungen und definiert weitere zukünftige Arbeiten. Letzteres zielt darauf ab, die Benutzerfreundlichkeit für den Entwicklungsprozess zu verbessern. Zusammenfassend zeigt diese Dissertation, wie eine angemessene Werkzeugunterstützung die Durchführung empirischer Forschung erleichtern und somit dazu beitragen kann, die Wissenslücke im Bereich MMI-Design zu schließen, was in Zukunft zu besseren XR-Schnittstellen führen wird

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept’s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality

Agrobiodiversity threats amid expanding woody monocultures and hopes nourished through farmer and food movements in the Mediterranean

International audienceThe high biodiversity of food and agriculture (agrobiodiversity) in the Mediterranean exists in rapidly changing landscapes and food systems. The first goal of this Commentary is to explain how agrobiodiverse Mediterranean food cereals and legumes are threatened by the accelerating expansion and intensification of monocultures of woody crops-principally olive, nut, grape, and citrus monocrops-in landscapes of the western Mediterranean (Spain, Morocco, and France). Its second goal is to explain the key countervailing force of specific food and farmer movements, organizations, and practices supporting agrobiodiversity. We argue this food agrobiodiversity support is timely and vital because of growing threats. Intensive woody monocultures have been promoted for climate change adaptation and policies, while the proposed agroecological alternatives to woody monocultures show a mixed record regarding the support of food agrobiodiversity. The Mediterranean's boom of woody monocultures relies on increased irrigation, including groundwater extraction, that undermines water sustainability. We engage with policy to explain how the timely support of food agrobiodiversity by farmer and food movements and practices demonstrates the production-consumption linkages that can strengthen sustainability, biodiversity conservation, and climate change adaptation/mitigation. Our policy arguments focus on the promising bridge of agrobiodiversity's production-consumption linkages to agroecology as an increasingly influential approach in these policy sectors. Finally, as a key complementary goal, we reflect on current agrobiodiversity-monoculture challenges by engaging the broad themes of rural-urban networks and urbanization in the Mediterranean, the land sparing versus land sharing debate, and the Plantationocene concept. Each thematic reflection enhances the understanding of food agrobiodiversity threats and support, landscapes of mixed agrobiodiversity and intensified woody monocultures (monoculture-agrobiodiversity landscapes), and relevant policy insight