Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

Physical Selection in Ubiquitous Computing

Jokapaikan tietotekniikassa (ubiquitous computing) tietotekniset laitteet sulautuvat fyysiseen ympäristöön siten että niiden käyttäjät voivat olla yhtä aikaa vuorovaikutuksessa näiden laitteiden kanssa ja toimia fyysisessä ympäristössään. Laitteet ovat yhteydessä toisiinsa, ne ovat eri kokoisia ja niillä on erilaisia syöttö- ja tulostusmahdollisuuksia tarkoituksestaan riippuen. Nämä jokapaikan tietotekniikan ominaisuudet luovat tarpeen vuorovaikutustavoille, jotka eroavat huomattavasti tavanomaisten työpöytätietokoneiden vuorovaikutustavoista. Fyysinen valinta (physical selection) on jokapaikan tietotekniikan vuorovaikutustehtävä, jota käytetään kertomaan käyttäjän kannettavalle päätelaitteelle minkä fyysisen esineen kanssa käyttäjä haluaa olla vuorovaikutuksessa. Fyysinen valinta perustuu tunnisteisiin (tag), jotka yksilöivät fyysiset esineet tai sisältävät fyysisen hyperlinkin digitaalisessa muodossa olevaan tietoon, joka liittyy esineeseen, johon kyseinen tunniste on liitetty. Käyttäjä valitsee fyysisen hyperlinkin koskettamalla, osoittamalla tai skannaamalla tunnistetta sopivalla lukulaitteella varustetulla päätelaitteellaan. Fyysinen valinta voidaan toteuttaa erilaisilla teknologioilla, kuten sähköisesti luettavilla tunnisteilla ja niiden lukijoilla, infrapunalähettimillä sekä optisesti luettavilla tunnisteilla ja matkapuhelinten kameroilla. Tässä väitöskirjassa analysoidaan fyysistä valintaa vuorovaikutustehtävänä ja toteutusteknisestä näkökulmasta sekä esitellään eri valintatavat ­ kosketus, osoitus ja skannaus. Koskeusta ja osoitusta on tutkittu toteuttamalla prototyyppi ja tutkimalla sen avulla valintatapoja kokeellisesti. Tämän väitöskirjan tuloksiin kuuluu fyysisen valinnan analysointi jokapaikan tietotekniikan kontekstissa, ehdotuksia fyysisten hyperlinkkien visualisoinnista sekä fyysisessä ympäristössä että päätelaitteessa, ja käyttäjävaatimuksia fyysiselle valinnalle osana jokapaikan tietotekniikan arkkitehtuuria.In ubiquitous computing, the computing devices are embedded into the physical environment so that the users can interact with the devices at the same time as they interact with the physical environment. The various devices are connected to each other, and have various sizes and input and output capabilities depending on their purpose. These features of ubiquitous computing create a need for interaction methods that are radically different from the desktop computer interactions. Physical selection is an interaction task for ubiquitous computing and it is used to tell the user s mobile terminal which physical object the user wants to interact with. It is based on tags that identify physical objects or store a physical hyperlink to digital information related to the object the tag is attached to. The user selects the physical hyperlink by touching, pointing or scanning the tag with the mobile terminal that is equipped with an appropriate reader. Physical selection has been implemented with various technologies, such as radio-frequency tags and readers, infrared transceivers, and optically readable tags and mobile phone cameras. In this dissertation, physical selection is analysed as a user interaction task, and from the implementation viewpoint. Different selection methods ­ touching, pointing and scanning ­ are presented. Touching and pointing have been studied by implementing a prototype and conducting user experiments with it. The contributions of this dissertation include an analysis of physical selection in the ubiquitous computing context, suggestions for visualising the physical hyperlinks in both the physical environment and in the mobile terminal, and user requirements for physical selection as a part of an ambient intelligence architecture

Physical Selection in Ubiquitous Computing

Jokapaikan tietotekniikassa (ubiquitous computing) tietotekniset laitteet sulautuvat fyysiseen ympäristöön siten että niiden käyttäjät voivat olla yhtä aikaa vuorovaikutuksessa näiden laitteiden kanssa ja toimia fyysisessä ympäristössään. Laitteet ovat yhteydessä toisiinsa, ne ovat eri kokoisia ja niillä on erilaisia syöttö- ja tulostusmahdollisuuksia tarkoituksestaan riippuen. Nämä jokapaikan tietotekniikan ominaisuudet luovat tarpeen vuorovaikutustavoille, jotka eroavat huomattavasti tavanomaisten työpöytätietokoneiden vuorovaikutustavoista. Fyysinen valinta (physical selection) on jokapaikan tietotekniikan vuorovaikutustehtävä, jota käytetään kertomaan käyttäjän kannettavalle päätelaitteelle minkä fyysisen esineen kanssa käyttäjä haluaa olla vuorovaikutuksessa. Fyysinen valinta perustuu tunnisteisiin (tag), jotka yksilöivät fyysiset esineet tai sisältävät fyysisen hyperlinkin digitaalisessa muodossa olevaan tietoon, joka liittyy esineeseen, johon kyseinen tunniste on liitetty. Käyttäjä valitsee fyysisen hyperlinkin koskettamalla, osoittamalla tai skannaamalla tunnistetta sopivalla lukulaitteella varustetulla päätelaitteellaan. Fyysinen valinta voidaan toteuttaa erilaisilla teknologioilla, kuten sähköisesti luettavilla tunnisteilla ja niiden lukijoilla, infrapunalähettimillä sekä optisesti luettavilla tunnisteilla ja matkapuhelinten kameroilla. Tässä väitöskirjassa analysoidaan fyysistä valintaa vuorovaikutustehtävänä ja toteutusteknisestä näkökulmasta sekä esitellään eri valintatavat ­ kosketus, osoitus ja skannaus. Koskeusta ja osoitusta on tutkittu toteuttamalla prototyyppi ja tutkimalla sen avulla valintatapoja kokeellisesti. Tämän väitöskirjan tuloksiin kuuluu fyysisen valinnan analysointi jokapaikan tietotekniikan kontekstissa, ehdotuksia fyysisten hyperlinkkien visualisoinnista sekä fyysisessä ympäristössä että päätelaitteessa, ja käyttäjävaatimuksia fyysiselle valinnalle osana jokapaikan tietotekniikan arkkitehtuuria.In ubiquitous computing, the computing devices are embedded into the physical environment so that the users can interact with the devices at the same time as they interact with the physical environment. The various devices are connected to each other, and have various sizes and input and output capabilities depending on their purpose. These features of ubiquitous computing create a need for interaction methods that are radically different from the desktop computer interactions. Physical selection is an interaction task for ubiquitous computing and it is used to tell the user s mobile terminal which physical object the user wants to interact with. It is based on tags that identify physical objects or store a physical hyperlink to digital information related to the object the tag is attached to. The user selects the physical hyperlink by touching, pointing or scanning the tag with the mobile terminal that is equipped with an appropriate reader. Physical selection has been implemented with various technologies, such as radio-frequency tags and readers, infrared transceivers, and optically readable tags and mobile phone cameras. In this dissertation, physical selection is analysed as a user interaction task, and from the implementation viewpoint. Different selection methods ­ touching, pointing and scanning ­ are presented. Touching and pointing have been studied by implementing a prototype and conducting user experiments with it. The contributions of this dissertation include an analysis of physical selection in the ubiquitous computing context, suggestions for visualising the physical hyperlinks in both the physical environment and in the mobile terminal, and user requirements for physical selection as a part of an ambient intelligence architecture

Arfid: A Reconfigurable Fabric of Input Devices for the Internet of Things

Abstract Low-cost, easily deployable, reconfigurable, movable input devices can enable adaptive workflows in commercial, industrial, and home environments. A key limitation of previous reconfigurable control systems is their high cost or maintenance burden (e.g., battery changes or wiring setup). Our poster presents Arfid, a "fabric" for reconfigurable input devices that connects low-cost, battery-free inputs to arbitrarily specified functions in their surroundings via a buildingwide network of RFID readers. Users can reassign controllers' functions using a simple web interface

Review: Development and technical design of tangible user interfaces in wide-field areas of application

A tangible user interface or TUI connects physical objects and digital interfaces. It is more interactive and interesting for users than a classic graphic user interface. This article presents a descriptive overview of TUI's real-world applications sorted into ten main application areas-teaching of traditional subjects, medicine and psychology, programming, database development, music and arts, modeling of 3D objects, modeling in architecture, literature and storytelling, adjustable TUI solutions, and commercial TUI smart toys. The paper focuses on TUI's technical solutions and a description of technical constructions that influences the applicability of TUIs in the real world. Based on the review, the technical concept was divided into two main approaches: the sensory technical concept and technology based on a computer vision algorithm. The sensory technical concept is processed to use wireless technology, sensors, and feedback possibilities in TUI applications. The image processing approach is processed to a marker and markerless approach for object recognition, the use of cameras, and the use of computer vision platforms for TUI applications.Web of Science2113art. no. 425

Beyond representations: towards an action-centric perspective on tangible interaction

In the light of theoretical as well as concrete technical development, we discuss a conceptual shift from an information-centric to an action-centric perspective on tangible interactive technology. We explicitly emphasise the qualities of shareable use, and the importance of designing tangibles that allow for meaningful manipulation and control of the digital material. This involves a broadened focus from studying properties of the interface, to instead aim for qualities of the activity of using a system, a general tendency towards designing for social and sharable use settings and an increased openness towards multiple and subjective interpretations. An effect of this is that tangibles are not designed as representations of data, but as resources for action. We discuss four ways that tangible artefacts work as resources for action: (1) for physical manipulation; (2) for referential, social and contextually oriented action; (3) for perception and sensory experience; (4) for digitally mediated action

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo