Blending the Material and Digital World for Hybrid Interfaces

The development of digital technologies in the 21st century is progressing continuously and new device classes such as tablets, smartphones or smartwatches are finding their way into our everyday lives. However, this development also poses problems, as these prevailing touch and gestural interfaces often lack tangibility, take little account of haptic qualities and therefore require full attention from their users. Compared to traditional tools and analog interfaces, the human skills to experience and manipulate material in its natural environment and context remain unexploited. To combine the best of both, a key question is how it is possible to blend the material world and digital world to design and realize novel hybrid interfaces in a meaningful way. Research on Tangible User Interfaces (TUIs) investigates the coupling between physical objects and virtual data. In contrast, hybrid interfaces, which specifically aim to digitally enrich analog artifacts of everyday work, have not yet been sufficiently researched and systematically discussed. Therefore, this doctoral thesis rethinks how user interfaces can provide useful digital functionality while maintaining their physical properties and familiar patterns of use in the real world. However, the development of such hybrid interfaces raises overarching research questions about the design: Which kind of physical interfaces are worth exploring? What type of digital enhancement will improve existing interfaces? How can hybrid interfaces retain their physical properties while enabling new digital functions? What are suitable methods to explore different design? And how to support technology-enthusiast users in prototyping? For a systematic investigation, the thesis builds on a design-oriented, exploratory and iterative development process using digital fabrication methods and novel materials. As a main contribution, four specific research projects are presented that apply and discuss different visual and interactive augmentation principles along real-world applications. The applications range from digitally-enhanced paper, interactive cords over visual watch strap extensions to novel prototyping tools for smart garments. While almost all of them integrate visual feedback and haptic input, none of them are built on rigid, rectangular pixel screens or use standard input modalities, as they all aim to reveal new design approaches. The dissertation shows how valuable it can be to rethink familiar, analog applications while thoughtfully extending them digitally. Finally, this thesis’ extensive work of engineering versatile research platforms is accompanied by overarching conceptual work, user evaluations and technical experiments, as well as literature reviews.Die Durchdringung digitaler Technologien im 21. Jahrhundert schreitet stetig voran und neue Geräteklassen wie Tablets, Smartphones oder Smartwatches erobern unseren Alltag. Diese Entwicklung birgt aber auch Probleme, denn die vorherrschenden berührungsempfindlichen Oberflächen berücksichtigen kaum haptische Qualitäten und erfordern daher die volle Aufmerksamkeit ihrer Nutzer:innen. Im Vergleich zu traditionellen Werkzeugen und analogen Schnittstellen bleiben die menschlichen Fähigkeiten ungenutzt, die Umwelt mit allen Sinnen zu begreifen und wahrzunehmen. Um das Beste aus beiden Welten zu vereinen, stellt sich daher die Frage, wie neuartige hybride Schnittstellen sinnvoll gestaltet und realisiert werden können, um die materielle und die digitale Welt zu verschmelzen. In der Forschung zu Tangible User Interfaces (TUIs) wird die Verbindung zwischen physischen Objekten und virtuellen Daten untersucht. Noch nicht ausreichend erforscht wurden hingegen hybride Schnittstellen, die speziell darauf abzielen, physische Gegenstände des Alltags digital zu erweitern und anhand geeigneter Designparameter und Entwurfsräume systematisch zu untersuchen. In dieser Dissertation wird daher untersucht, wie Materialität und Digitalität nahtlos ineinander übergehen können. Es soll erforscht werden, wie künftige Benutzungsschnittstellen nützliche digitale Funktionen bereitstellen können, ohne ihre physischen Eigenschaften und vertrauten Nutzungsmuster in der realen Welt zu verlieren. Die Entwicklung solcher hybriden Ansätze wirft jedoch übergreifende Forschungsfragen zum Design auf: Welche Arten von physischen Schnittstellen sind es wert, betrachtet zu werden? Welche Art von digitaler Erweiterung verbessert das Bestehende? Wie können hybride Konzepte ihre physischen Eigenschaften beibehalten und gleichzeitig neue digitale Funktionen ermöglichen? Was sind geeignete Methoden, um verschiedene Designs zu erforschen? Wie kann man Technologiebegeisterte bei der Erstellung von Prototypen unterstützen? Für eine systematische Untersuchung stützt sich die Arbeit auf einen designorientierten, explorativen und iterativen Entwicklungsprozess unter Verwendung digitaler Fabrikationsmethoden und neuartiger Materialien. Im Hauptteil werden vier Forschungsprojekte vorgestellt, die verschiedene visuelle und interaktive Prinzipien entlang realer Anwendungen diskutieren. Die Szenarien reichen von digital angereichertem Papier, interaktiven Kordeln über visuelle Erweiterungen von Uhrarmbändern bis hin zu neuartigen Prototyping-Tools für intelligente Kleidungsstücke. Um neue Designansätze aufzuzeigen, integrieren nahezu alle visuelles Feedback und haptische Eingaben, um Alternativen zu Standard-Eingabemodalitäten auf starren Pixelbildschirmen zu schaffen. Die Dissertation hat gezeigt, wie wertvoll es sein kann, bekannte, analoge Anwendungen zu überdenken und sie dabei gleichzeitig mit Bedacht digital zu erweitern. Dabei umfasst die vorliegende Arbeit sowohl realisierte technische Forschungsplattformen als auch übergreifende konzeptionelle Arbeiten, Nutzerstudien und technische Experimente sowie die Analyse existierender Forschungsarbeiten

WristOrigami: Exploring foldable design for multi-display smartwatch

We present WristOrigami, an origami-inspired design concept and system extending the interaction with smartwatches through a foldable structure with multiple on-wrist displays. The current design provides extra affordances via folding, flipping, and elastic pulling actions on a multidisplay smartwatch. To motivate the design of WristOrigami, we developed a taxonomy that could be useful for analyzing and characterizing the origami-inspired multi-display smartwatch interaction. Through a participatory-design study with a set of prototypes with different levels of fidelity, we investigated users\u27 perception of WristOrigami in a wide range of applications with the presented features, and summarized a list of common shape configurations. We summarized our findings into seven design recommendations, to inform the future design of foldable smartwatch interactions. We further developed a set of application demonstrations as proofs-of-concept

Design of a Tracking Glove for use in Virtual Reality Training Environments

A thesis presented to the faculty of the College of Business and Technology at Morehead State University in partial fulfillment of the requirements for the degree Master of Science by Thomas A. Buteyn on April 25, 2022

Tabletop tangible maps and diagrams for visually impaired users

En dépit de leur omniprésence et de leur rôle essentiel dans nos vies professionnelles et personnelles, les représentations graphiques, qu'elles soient numériques ou sur papier, ne sont pas accessibles aux personnes déficientes visuelles car elles ne fournissent pas d'informations tactiles. Par ailleurs, les inégalités d'accès à ces représentations ne cessent de s'accroître ; grâce au développement de représentations graphiques dynamiques et disponibles en ligne, les personnes voyantes peuvent non seulement accéder à de grandes quantités de données, mais aussi interagir avec ces données par le biais de fonctionnalités avancées (changement d'échelle, sélection des données à afficher, etc.). En revanche, pour les personnes déficientes visuelles, les techniques actuellement utilisées pour rendre accessibles les cartes et les diagrammes nécessitent l'intervention de spécialistes et ne permettent pas la création de représentations interactives. Cependant, les récentes avancées dans le domaine de l'adaptation automatique de contenus laissent entrevoir, dans les prochaines années, une augmentation de la quantité de contenus adaptés. Cette augmentation doit aller de pair avec le développement de dispositifs utilisables et abordables en mesure de supporter l'affichage de représentations interactives et rapidement modifiables, tout en étant accessibles aux personnes déficientes visuelles. Certains prototypes de recherche s'appuient sur une représentation numérique seulement : ils peuvent être instantanément modifiés mais ne fournissent que très peu de retour tactile, ce qui rend leur exploration complexe d'un point de vue cognitif et impose de fortes contraintes sur le contenu. D'autres prototypes s'appuient sur une représentation numérique et physique : bien qu'ils puissent être explorés tactilement, ce qui est un réel avantage, ils nécessitent un support tactile qui empêche toute modification rapide. Quant aux dispositifs similaires à des tablettes Braille, mais avec des milliers de picots, leur coût est prohibitif. L'objectif de cette thèse est de pallier les limitations de ces approches en étudiant comment développer des cartes et diagrammes interactifs physiques, modifiables et abordables. Pour cela, nous nous appuyons sur un type d'interface qui a rarement été étudié pour des utilisateurs déficients visuels : les interfaces tangibles, et plus particulièrement les interfaces tangibles sur table. Dans ces interfaces, des objets physiques représentent des informations numériques et peuvent être manipulés par l'utilisateur pour interagir avec le système, ou par le système lui-même pour refléter un changement du modèle numérique - on parle alors d'interfaces tangibles sur tables animées, ou actuated. Grâce à la conception, au développement et à l'évaluation de trois interfaces tangibles sur table (les Tangible Reels, la Tangible Box et BotMap), nous proposons un ensemble de solutions techniques répondant aux spécificités des interfaces tangibles pour des personnes déficientes visuelles, ainsi que de nouvelles techniques d'interaction non-visuelles, notamment pour la reconstruction d'une carte ou d'un diagramme et l'exploration de cartes de type " Pan & Zoom ". D'un point de vue théorique, nous proposons aussi une nouvelle classification pour les dispositifs interactifs accessibles.Despite their omnipresence and essential role in our everyday lives, online and printed graphical representations are inaccessible to visually impaired people because they cannot be explored using the sense of touch. The gap between sighted and visually impaired people's access to graphical representations is constantly growing due to the increasing development and availability of online and dynamic representations that not only give sighted people the opportunity to access large amounts of data, but also to interact with them using advanced functionalities such as panning, zooming and filtering. In contrast, the techniques currently used to make maps and diagrams accessible to visually impaired people require the intervention of tactile graphics specialists and result in non-interactive tactile representations. However, based on recent advances in the automatic production of content, we can expect in the coming years a growth in the availability of adapted content, which must go hand-in-hand with the development of affordable and usable devices. In particular, these devices should make full use of visually impaired users' perceptual capacities and support the display of interactive and updatable representations. A number of research prototypes have already been developed. Some rely on digital representation only, and although they have the great advantage of being instantly updatable, they provide very limited tactile feedback, which makes their exploration cognitively demanding and imposes heavy restrictions on content. On the other hand, most prototypes that rely on digital and physical representations allow for a two-handed exploration that is both natural and efficient at retrieving and encoding spatial information, but they are physically limited by the use of a tactile overlay, making them impossible to update. Other alternatives are either extremely expensive (e.g. braille tablets) or offer a slow and limited way to update the representation (e.g. maps that are 3D-printed based on users' inputs). In this thesis, we propose to bridge the gap between these two approaches by investigating how to develop physical interactive maps and diagrams that support two-handed exploration, while at the same time being updatable and affordable. To do so, we build on previous research on Tangible User Interfaces (TUI) and particularly on (actuated) tabletop TUIs, two fields of research that have surprisingly received very little interest concerning visually impaired users. Based on the design, implementation and evaluation of three tabletop TUIs (the Tangible Reels, the Tangible Box and BotMap), we propose innovative non-visual interaction techniques and technical solutions that will hopefully serve as a basis for the design of future TUIs for visually impaired users, and encourage their development and use. We investigate how tangible maps and diagrams can support various tasks, ranging from the (re)construction of diagrams to the exploration of maps by panning and zooming. From a theoretical perspective we contribute to the research on accessible graphical representations by highlighting how research on maps can feed research on diagrams and vice-versa. We also propose a classification and comparison of existing prototypes to deliver a structured overview of current research

A model personal energy meter

Every day each of us consumes a significant amount of energy, both directly through transport, heating and use of appliances, and indirectly from our needs for the production of food, manufacture of goods and provision of services. This dissertation investigates a personal energy meter which can record and apportion an individual's energy usage in order to supply baseline information and incentives for reducing our environmental impact. If the energy costs of large shared resources are split evenly without regard for individual consumption each person minimises his own losses by taking advantage of others. Context awareness offers the potential to change this balance and apportion energy costs to those who cause them to be incurred. This dissertation explores how sensor systems installed in many buildings today can be used to apportion energy consumption between users, including an evaluation of a range of strategies in a case study and elaboration of the overriding principles that are generally applicable. It also shows how second-order estimators combined with location data can provide a proxy for fine-grained sensing. A key ingredient for apportionment mechanisms is data on energy usage. This may come from metering devices or buildings directly, or from profiling devices and using secondary indicators to infer their power state. A mechanism for profiling devices to determine the energy costs of specific activities, particularly applicable to shared programmable devices is presented which can make this process simpler and more accurate. By combining crowdsourced building-inventory information and a simple building energy model it is possible to estimate an individual's energy use disaggregated by device class with very little direct sensing. Contextual information provides crucial cues for apportioning the use and energy costs of resources, and one of the most valuable sources from which to infer context is location. A key ingredient for a personal energy meter is a low cost, low infrastructure location system that can be deployed on a truly global scale. This dissertation presents a description and evaluation of the new concept of inquiry-free Bluetooth tracking that has the potential to offer indoor location information with significantly less infrastructure and calibration than other systems. Finally, a suitable architecture for a personal energy meter on a global scale is demonstrated using a mobile phone application to aggregate energy feeds based on the case studies and technologies developed

An Interactive Belt-worn Badge with a Retractable String-based Input Mechanism

In this paper we explore a new type of wearable computing device, an interactive identity badge. An embedded LCD presents dynamic information to the wearer and interaction is facilitated by sensing movement of the retractable string which attaches the unit to the wearer’s belt. This formfactor makes it possible to interact using a single hand, providing lightweight and immediate access to a variety of information when it’s not convenient to pick up, unlock and interact directly with a device like a smartphone. In this paper we present our prototype interactive badge, demonstrate the underlying technology and describe a number of usage scenarios and interaction techniques

An interactive belt-worn badge with a retractable string-based input mechanism

In this paper we explore a new type of wearable computing device, an interactive identity badge. An embedded LCD presents dynamic information to the wearer and interaction is facilitated by sensing movement of the retractable string which attaches the unit to the wearer's belt. This form-factor makes it possible to interact using a single hand, providing lightweight and immediate access to a variety of information when it's not convenient to pick up, unlock and interact directly with a device like a smartphone. In this paper we present our prototype interactive badge, demonstrate the underlying technology and describe a number of usage scenarios and interaction techniques. Copyright © 2013 ACM

Science on a Deep-Ocean Shipwreck

Author Institution: Departments of Geological Sciences and Zoology, Museum of Biological Diversity, The Ohio state University ; Columbus-America Discovery GroupA five-year scientific investigation of a site on the North Atlantic seafloor, 270 km off Cape Fear, NC, at a depth of 2,200 m, was undertaken in conjunction with recovery operations on a nineteenth-century steamship (SS Central America which sank in an 1857 hurricane while carrying passengers and cargo en route to New York from the California gold fields). Activities in the disciplines of oceanography, marine geology, marine biology, materials science, and undersea archaeology were undertaken with the tele-directed submersible robot, Nemo. The study included field observations at the site (recorded in over 3,000 hours of videotape and 25,000 still photographs), examination of hundreds of deep-ocean specimens and artifacts, and analysis of several experiments deployed on the seafloor. Resting on a gentle slope of the Blake Ridge, the shipwreck environment was cold, lightless, oxygen-rich, and flushed by moderate currents. The sediments were a foraminiferal-pteropod ooze, deposited at a slow rate (1.7 cm/1,000 years). A diverse community of errant and sessile benthic invertebrates and benthopelagic fishes colonized the shipwreck deriving from it food, cover, and a place of attachment. This deep-ocean oasis supported a greater variety and concentration of animal life than did the surrounding ooze habitat. The timbers of the shipwreck were degraded by woodboring bivalves. The iron machinery was extensively corroded and mobilized into flow structures (rusticles) by iron-oxidizing bacteria. Passenger luggage recovered from the shipwreck contained artifacts which provided insight about the life styles of the voyagers during the Gold Rush. This project demonstrated that a holistic approach to a deep-ocean site of historic importance can provide understandings of the interrelated processes which affect cultural deposits on the abyssal seafloor and the marine life that they foster