Tangible interaction with anthropomorphic smart objects in instrumented environments

A major technological trend is to augment everyday objects with sensing, computing and actuation power in order to provide new services beyond the objects' traditional purpose, indicating that such smart objects might become an integral part of our daily lives. To be able to interact with smart object systems, users will obviously need appropriate interfaces that regard their distinctive characteristics. Concepts of tangible and anthropomorphic user interfaces are combined in this dissertation to create a novel paradigm for smart object interaction. This work provides an exploration of the design space, introduces design guidelines, and provides a prototyping framework to support the realisation of the proposed interface paradigm. Furthermore, novel methods for expressing personality and emotion by auditory means are introduced and elaborated, constituting essential building blocks for anthropomorphised smart objects. Two experimental user studies are presented, confirming the endeavours to reflect personality attributes through prosody-modelled synthetic speech and to express emotional states through synthesised affect bursts. The dissertation concludes with three example applications, demonstrating the potentials of the concepts and methodologies elaborated in this thesis.Die Integration von Informationstechnologie in Gebrauchsgegenstände ist ein gegenwärtiger technologischer Trend, welcher es Alltagsgegenständen ermöglicht, durch den Einsatz von Sensorik, Aktorik und drahtloser Kommunikation neue Dienste anzubieten, die über den ursprünglichen Zweck des Objekts hinausgehen. Die Nutzung dieser sogenannten Smart Objects erfordert neuartige Benutzerschnittstellen, welche die speziellen Eigenschaften und Anwendungsbereiche solcher Systeme berücksichtigen. Konzepte aus den Bereichen Tangible Interaction und Anthropomorphe Benutzerschnittstellen werden in dieser Dissertation vereint, um ein neues Interaktionsparadigma für Smart Objects zu entwickeln. Die vorliegende Arbeit untersucht dafür die Gestaltungsmöglichkeiten und zeigt relevante Aspekte aus verwandten Disziplinen auf. Darauf aufbauend werden Richtlinien eingeführt, welche den Entwurf von Benutzerschnittstellen nach dem hier vorgestellten Ansatz begleiten und unterstützen sollen. Für eine prototypische Implementierung solcher Benutzerschnittstellen wird eine Architektur vorgestellt, welche die Anforderungen von Smart Object Systemen in instrumentierten Umgebungen berücksichtigt. Ein wichtiger Bestandteil stellt dabei die Sensorverarbeitung dar, welche unter anderem eine Interaktionserkennung am Objekt und damit auch eine physikalische Eingabe ermöglicht. Des Weiteren werden neuartige Methoden für den auditiven Ausdruck von Emotion und Persönlichkeit entwickelt, welche essentielle Bausteine für anthropomorphisierte Smart Objects darstellen und in Benutzerstudien untersucht wurden. Die Dissertation schliesst mit der Beschreibung von drei Applikationen, welche im Rahmen der Arbeit entwickelt wurden und das Potential der hier erarbeiteten Konzepte und Methoden widerspiegeln

TOBE: Tangible Out-of-Body Experience

We propose a toolkit for creating Tangible Out-of-Body Experiences: exposing the inner states of users using physiological signals such as heart rate or brain activity. Tobe can take the form of a tangible avatar displaying live physiological readings to reflect on ourselves and others. Such a toolkit could be used by researchers and designers to create a multitude of potential tangible applications, including (but not limited to) educational tools about Science Technologies Engineering and Mathematics (STEM) and cognitive science, medical applications or entertainment and social experiences with one or several users or Tobes involved. Through a co-design approach, we investigated how everyday people picture their physiology and we validated the acceptability of Tobe in a scientific museum. We also give a practical example where two users relax together, with insights on how Tobe helped them to synchronize their signals and share a moment

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo

Multimodal interaction with mobile devices : fusing a broad spectrum of modality combinations

This dissertation presents a multimodal architecture for use in mobile scenarios such as shopping and navigation. It also analyses a wide range of feasible modality input combinations for these contexts. For this purpose, two interlinked demonstrators were designed for stand-alone use on mobile devices. Of particular importance was the design and implementation of a modality fusion module capable of combining input from a range of communication modes like speech, handwriting, and gesture. The implementation is able to account for confidence value biases arising within and between modalities and also provides a method for resolving semantically overlapped input. Tangible interaction with real-world objects and symmetric multimodality are two further themes addressed in this work. The work concludes with the results from two usability field studies that provide insight on user preference and modality intuition for different modality combinations, as well as user acceptance for anthropomorphized objects.Diese Dissertation präsentiert eine multimodale Architektur zum Gebrauch in mobilen Umständen wie z. B. Einkaufen und Navigation. Außerdem wird ein großes Gebiet von möglichen modalen Eingabekombinationen zu diesen Umständen analysiert. Um das in praktischer Weise zu demonstrieren, wurden zwei teilweise gekoppelte Vorführungsprogramme zum \u27stand-alone\u27; Gebrauch auf mobilen Geräten entworfen. Von spezieller Wichtigkeit war der Entwurf und die Ausführung eines Modalitäts-fusion Modul, das die Kombination einer Reihe von Kommunikationsarten wie Sprache, Handschrift und Gesten ermöglicht. Die Ausführung erlaubt die Veränderung von Zuverlässigkeitswerten innerhalb einzelner Modalitäten und außerdem ermöglicht eine Methode um die semantisch überlappten Eingaben auszuwerten. Wirklichkeitsnaher Dialog mit aktuellen Objekten und symmetrische Multimodalität sind zwei weitere Themen die in dieser Arbeit behandelt werden. Die Arbeit schließt mit Resultaten von zwei Feldstudien, die weitere Einsicht erlauben über die bevorzugte Art verschiedener Modalitätskombinationen, sowie auch über die Akzeptanz von anthropomorphisierten Objekten

Multimodal interaction with mobile devices : fusing a broad spectrum of modality combinations

This dissertation presents a multimodal architecture for use in mobile scenarios such as shopping and navigation. It also analyses a wide range of feasible modality input combinations for these contexts. For this purpose, two interlinked demonstrators were designed for stand-alone use on mobile devices. Of particular importance was the design and implementation of a modality fusion module capable of combining input from a range of communication modes like speech, handwriting, and gesture. The implementation is able to account for confidence value biases arising within and between modalities and also provides a method for resolving semantically overlapped input. Tangible interaction with real-world objects and symmetric multimodality are two further themes addressed in this work. The work concludes with the results from two usability field studies that provide insight on user preference and modality intuition for different modality combinations, as well as user acceptance for anthropomorphized objects.Diese Dissertation präsentiert eine multimodale Architektur zum Gebrauch in mobilen Umständen wie z. B. Einkaufen und Navigation. Außerdem wird ein großes Gebiet von möglichen modalen Eingabekombinationen zu diesen Umständen analysiert. Um das in praktischer Weise zu demonstrieren, wurden zwei teilweise gekoppelte Vorführungsprogramme zum 'stand-alone'; Gebrauch auf mobilen Geräten entworfen. Von spezieller Wichtigkeit war der Entwurf und die Ausführung eines Modalitäts-fusion Modul, das die Kombination einer Reihe von Kommunikationsarten wie Sprache, Handschrift und Gesten ermöglicht. Die Ausführung erlaubt die Veränderung von Zuverlässigkeitswerten innerhalb einzelner Modalitäten und außerdem ermöglicht eine Methode um die semantisch überlappten Eingaben auszuwerten. Wirklichkeitsnaher Dialog mit aktuellen Objekten und symmetrische Multimodalität sind zwei weitere Themen die in dieser Arbeit behandelt werden. Die Arbeit schließt mit Resultaten von zwei Feldstudien, die weitere Einsicht erlauben über die bevorzugte Art verschiedener Modalitätskombinationen, sowie auch über die Akzeptanz von anthropomorphisierten Objekten

Interactive ubiquitous displays based on steerable projection

The ongoing miniaturization of computers and their embedding into the physical environment require new means of visual output. In the area of Ubiquitous Computing, flexible and adaptable display options are needed in order to enable the presentation of visual content in the physical environment. In this dissertation, we introduce the concepts of Display Continuum and Virtual Displays as new means of human-computer interaction. In this context, we present a realization of a Display Continuum based on steerable projection, and we describe a number of different interaction methods for manipulating this Display Continuum and the Virtual Displays placed on it.Mit zunehmender Miniaturisierung der Computer und ihrer Einbettung in der physikalischen Umgebung werden neue Arten der visuellen Ausgabe notwendig. Im Bereich des Ubiquitous Computing (Rechnerallgegenwart) werden flexible und anpassungsfähige Displays benötigt, um eine Anzeige von visuellen Inhalten unmittelbar in der physikalischen Umgebung zu ermöglichen. In dieser Dissertation führen wir das Konzept des Display-Kontinuums und der Virtuellen Displays als Instrument der Mensch-Maschine-Interaktion ein. In diesem Zusammenhang präsentieren wir eine mögliche Display-Kontinuum-Realisierung, die auf der Verwendung steuerbarer Projektion basiert, und wir beschreiben mehrere verschiedene Interaktionsmethoden, mit denen man das Display-Kontinuum und die darauf platzierten Virtuellen Displays steuern kann

Interfaces for human-centered production and use of computer graphics assets

L'abstract è presente nell'allegato / the abstract is in the attachmen

Entangled ethnography : Towards a collective future understanding

In this work, we develop a vision for entangled ethnography, where constellations of people, artefacts, algorithms and data come together to collectively make sense of the relations between people and objects. This is grounded in New Materialism’s picture of a world understood through entanglement, through resonant constellations, through a multiplicity of unique individual viewpoints and their relationships. These perspectives are especially relevant for design ethnography, in particular for research around smart connected products, which collect data about their environment, the networks they are a part of, and the ways they are used. However, we are concerned about the current trend of many connected systems towards surveillance capitalism, as data is colonised, machinations are hidden, and a narrow definition of value is extracted. There is a key tension that while design, particularly of networked objects, attempts to go beyond human centeredness, the infrastructures that support it are moving towards a less than human perspective in their race to accumulate and dispossess. Our work tries to imagine the situations where participants in networked systems are richly engaged, rather than exploited. We hope for a future where human agency is central to a respectful and acceptable collaborative development of understanding

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics

Systemic Design for the innovation of home appliances The meaningfulness of data in designing sustainable systems

This work addressed the domestic environment considering this context as a complex system characterised by significant impacts in terms of resource consumption. Within the theoretical framework of Systemic Design (SD), this thesis focused on home appliances, in order to understand how to reduce the impact directly attributable to them, while optimising and simplifying daily tasks for the user. A design methodology towards environmental sustainability has been structured, by focusing on the use of data for design purposes and on creating value for the user through meaningful products. It considers the user, the product and the environment as central topics, by giving them the same relevance and the literature review is structured accordingly, investigating needs and requirements, ethical issues, but also current products and future scenarios. During my experience at TU Delft, I spent six months in the Department of Internet of Things at the Faculty of Industrial Design Engineering. Together with computer scientists, we developed a prototype to collect some missing data, establishing the importance of grounding the decision-making on reliable information. IoT and data gathering open a variety of possibilities in monitoring, accessing more precise knowledge of products and households useful for design purposes, up to understand how to fill the gap perceived by the user between needs and solutions. It considered the potential benefits of using IoT indicators to collect missing information about both the product, its use and its operating environment to address critical aspects in the design stage, thus extending products’ lifetime. This thesis highlighted the importance of building multidisciplinary design teams to investigate different classes of requirements, and the need for flexible tools to cope with complex and evolving requirements, the co-evolution of problem and solutions and investigating open-ended questions. This approach leaves room for addressing every step of the traditional life-cycle in a more circular way, shifting the focus from the life-cycle centrality of the previous century to a more complex vision about the product