Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

Light on horizontal interactive surfaces: Input space for tabletop computing

In the last 25 years we have witnessed the rise and growth of interactive tabletop research, both in academic and in industrial settings. The rising demand for the digital support of human activities motivated the need to bring computational power to table surfaces. In this article, we review the state of the art of tabletop computing, highlighting core aspects that frame the input space of interactive tabletops: (a) developments in hardware technologies that have caused the proliferation of interactive horizontal surfaces and (b) issues related to new classes of interaction modalities (multitouch, tangible, and touchless). A classification is presented that aims to give a detailed view of the current development of this research area and define opportunities and challenges for novel touch- and gesture-based interactions between the human and the surrounding computational environment. © 2014 ACM.This work has been funded by Integra (Amper Sistemas and CDTI, Spanish Ministry of Science and Innovation) and TIPEx (TIN2010-19859-C03-01) projects and Programa de Becas y Ayudas para la Realización de Estudios Oficiales de Máster y Doctorado en la Universidad Carlos III de Madrid, 2010

Bringing the Physical to the Digital

This dissertation describes an exploration of digital tabletop interaction styles, with the ultimate goal of informing the design of a new model for tabletop interaction. In the context of this thesis the term digital tabletop refers to an emerging class of devices that afford many novel ways of interaction with the digital. Allowing users to directly touch information presented on large, horizontal displays. Being a relatively young field, many developments are in flux; hardware and software change at a fast pace and many interesting alternative approaches are available at the same time. In our research we are especially interested in systems that are capable of sensing multiple contacts (e.g., fingers) and richer information such as the outline of whole hands or other physical objects. New sensor hardware enable new ways to interact with the digital. When embarking into the research for this thesis, the question which interaction styles could be appropriate for this new class of devices was a open question, with many equally promising answers. Many everyday activities rely on our hands ability to skillfully control and manipulate physical objects. We seek to open up different possibilities to exploit our manual dexterity and provide users with richer interaction possibilities. This could be achieved through the use of physical objects as input mediators or through virtual interfaces that behave in a more realistic fashion. In order to gain a better understanding of the underlying design space we choose an approach organized into two phases. First, two different prototypes, each representing a specific interaction style – namely gesture-based interaction and tangible interaction – have been implemented. The flexibility of use afforded by the interface and the level of physicality afforded by the interface elements are introduced as criteria for evaluation. Each approaches’ suitability to support the highly dynamic and often unstructured interactions typical for digital tabletops is analyzed based on these criteria. In a second stage the learnings from these initial explorations are applied to inform the design of a novel model for digital tabletop interaction. This model is based on the combination of rich multi-touch sensing and a three dimensional environment enriched by a gaming physics simulation. The proposed approach enables users to interact with the virtual through richer quantities such as collision and friction. Enabling a variety of fine-grained interactions using multiple fingers, whole hands and physical objects. Our model makes digital tabletop interaction even more “natural”. However, because the interaction – the sensed input and the displayed output – is still bound to the surface, there is a fundamental limitation in manipulating objects using the third dimension. To address this issue, we present a technique that allows users to – conceptually – pick objects off the surface and control their position in 3D. Our goal has been to define a technique that completes our model for on-surface interaction and allows for “as-direct-as possible” interactions. We also present two hardware prototypes capable of sensing the users’ interactions beyond the table’s surface. Finally, we present visual feedback mechanisms to give the users the sense that they are actually lifting the objects off the surface. This thesis contributes on various levels. We present several novel prototypes that we built and evaluated. We use these prototypes to systematically explore the design space of digital tabletop interaction. The flexibility of use afforded by the interaction style is introduced as criterion alongside the user interface elements’ physicality. Each approaches’ suitability to support the highly dynamic and often unstructured interactions typical for digital tabletops are analyzed. We present a new model for tabletop interaction that increases the fidelity of interaction possible in such settings. Finally, we extend this model so to enable as direct as possible interactions with 3D data, interacting from above the table’s surface

Exploring the potential of physical visualizations

The goal of an external representation of abstract data is to provide insights and convey information about the structure of the underlying data, therefore helping people execute tasks and solve problems more effectively. Apart from the popular and well-studied digital visualization of abstract data there are other scarcely studied perceptual channels to represent data such as taste, sound or haptic. My thesis focuses on the latter and explores in which ways human knowledge and ability to sense and interact with the physical non-digital world can be used to enhance the way in which people analyze and explore abstract data. Emerging technological progress in digital fabrication allow an easy, fast and inexpensive production of physical objects. Machines such as laser cutters and 3D printers enable an accurate fabrication of physical visualizations with different form factors as well as materials. This creates, for the first time, the opportunity to study the potential of physical visualizations in a broad range. The thesis starts with the description of six prototypes of physical visualizations from static examples to digitally augmented variations to interactive artifacts. Based on these explorations, three promising areas of potential for physical visualizations were identified and investigated in more detail: perception & memorability, communication & collaboration, and motivation & self-reflection. The results of two studies in the area of information recall showed that participants who used a physical bar chart retained more information compared to the digital counterpart. Particularly facts about maximum and minimum values were be remembered more efficiently, when they were perceived from a physical visualization. Two explorative studies dealt with the potential of physical visualizations regarding communication and collaboration. The observations revealed the importance on the design and aesthetic of physical visualizations and indicated a great potential for their utilization by audiences with less interest in technology. The results also exposed the current limitations of physical visualizations, especially in contrast to their well-researched digital counterparts. In the area of motivation we present the design and evaluation of the Activity Sculptures project. We conducted a field study, in which we investigated physical visualizations of personal running activity. It was discovered that these sculptures generated curiosity and experimentation regarding the personal running behavior as well as evoked social dynamics such as discussions and competition. Based on the findings of the aforementioned studies this thesis concludes with two theoretical contributions on the design and potential of physical visualizations. On the one hand, it proposes a conceptual framework for material representations of personal data by describing a production and consumption lens. The goal is to encourage artists and designers working in the field of personal informatics to harness the interactive capabilities afforded by digital fabrication and the potential of material representations. On the other hand we give a first classification and performance rating of physical variables including 14 dimensions grouped into four categories. This complements the undertaking of providing researchers and designers with guidance and inspiration to uncover alternative strategies for representing data physically and building effective physical visualizations.Um aus abstrakten Daten konkrete Aussagen, komplexe Zusammenhänge oder überraschende Einsichten gewinnen zu können, müssen diese oftmals in eine, für den Menschen, anschauliche Form gebracht werden. Eine weitverbreitete und gut erforschte Möglichkeiten ist die Darstellung von Daten in visueller Form. Weniger erforschte Varianten sind das Verkörpern von Daten durch Geräusche, Gerüche oder physisch ertastbare Objekte und Formen. Diese Arbeit konzentriert sich auf die letztgenannte Variante und untersucht wie die menschlichen Fähigkeiten mit der physischenWelt zu interagieren dafür genutzt werden können, das Analysieren und Explorieren von Daten zu unterstützen. Der technische Fortschritt in der digitalen Fertigung vereinfacht und beschleunigt die Produktion von physischen Objekten und reduziert dabei deren Kosten. Lasercutter und 3D Drucker ermöglichen beispielsweise eine maßgerechte Fertigung physischer Visualisierungen verschiedenster Ausprägungen hinsichtlich Größe und Material. Dadurch ergibt sich zum ersten Mal die Gelegenheit, das Potenzial von physischen Visualisierungen in größerem Umfang zu erforschen. Der erste Teil der Arbeit skizziert insgesamt sechs Prototypen physischer Visualisierungen, wobei sowohl statische Beispiele beschrieben werden, als auch Exemplare die durch digital Inhalte erweitert werden oder dynamisch auf Interaktionen reagieren können. Basierend auf den Untersuchungen dieser Prototypen wurden drei vielversprechende Bereiche für das Potenzial physischer Visualisierungen ermittelt und genauer untersucht: Wahrnehmung & Einprägsamkeit, Kommunikation & Zusammenarbeit sowie Motivation & Selbstreflexion. Die Ergebnisse zweier Studien zur Wahrnehmung und Einprägsamkeit von Informationen zeigten, dass sich Teilnehmer mit einem physischen Balkendiagramm an deutlich mehr Informationen erinnern konnten, als Teilnehmer, die eine digitale Visualisierung nutzten. Insbesondere Fakten über Maximal- und Minimalwerte konnten besser im Gedächtnis behalten werden, wenn diese mit Hilfe einer physischen Visualisierung wahrgenommen wurden. Zwei explorative Studien untersuchten das Potenzial von physischen Visualisierungen im Bereich der Kommunikation mit Informationen sowie der Zusammenarbeit. Die Ergebnisse legten einerseits offen wie wichtig ein ausgereiftes Design und die Ästhetik von physischen Visualisierungen ist, deuteten anderseits aber auch darauf hin, dass Menschen mit geringem Interesse an neuen Technologien eine interessante Zielgruppe darstellen. Die Studien offenbarten allerdings auch die derzeitigen Grenzen von physischen Visualisierungen, insbesondere im Vergleich zu ihren gut erforschten digitalen Pendants. Im Bereich der Motivation und Selbstreflexion präsentieren wir die Entwicklung und Auswertung des Projekts Activity Sculptures. In einer Feldstudie über drei Wochen erforschten wir physische Visualisierungen, die persönliche Laufdaten repräsentieren. Unsere Beobachtungen und die Aussagen der Teilnehmer ließen darauf schließen, dass die Skulpturen Neugierde weckten und zum Experimentieren mit dem eigenen Laufverhalten einluden. Zudem konnten soziale Dynamiken entdeckt werden, die beispielsweise durch Diskussion aber auch Wettbewerbsgedanken zum Ausdruck kamen. Basierend auf den gewonnen Erkenntnissen durch die erwähnten Studien schließt diese Arbeit mit zwei theoretischen Beiträgen, hinsichtlich des Designs und des Potenzials von physischen Visualisierungen, ab. Zuerst wird ein konzeptionelles Framework vorgestellt, welches die Möglichkeiten und den Nutzen physischer Visualisierungen von persönlichen Daten veranschaulicht. Für Designer und Künstler kann dies zudem als Inspirationsquelle dienen, wie das Potenzial neuer Technologien, wie der digitalen Fabrikation, zur Darstellung persönlicher Daten in physischer Form genutzt werden kann. Des Weiteren wird eine initiale Klassifizierung von physischen Variablen vorgeschlagen mit insgesamt 14 Dimensionen, welche in vier Kategorien gruppiert sind. Damit vervollständigen wir unser Ziel, Forschern und Designern Inspiration und Orientierung zu bieten, um neuartige und effektvolle physische Visualisierungen zu erschaffen

Designing for Shareable Interfaces in the Wild

Despite excitement about the potential of interactive tabletops to support collaborative work, there have been few empirical demonstrations of their effectiveness (Marshall et al., 2011). In particular, while lab-based studies have explored the effects of individual design features, there has been a dearth of studies evaluating the success of systems in the wild. For this technology to be of value, designers and systems builders require a better understanding of how to develop and evaluate tabletop applications to be deployed in real world settings. This dissertation reports on two systems designed through a process that incorporated ethnography-style observations, iterative design and in the wild evaluation. The first study focused on collaborative learning in a medical setting. To address the fact that visitors to a hospital emergency ward were leaving with an incomplete understanding of their diagnosis and treatment, a system was prototyped in a working Emergency Room (ER) with doctors and patients. The system was found to be helpful but adoption issues hampered its impact. The second study focused on a planning application for visitors to a tourist information centre. Issues and opportunities for a successful, contextually-fitted system were addressed and it was found to be effective in supporting group planning activities by novice users, in particular, facilitating users’ first experiences, providing effective signage and offering assistance to guide the user through the application. This dissertation contributes to understanding of multi-user systems through literature review of tabletop systems, collaborative tasks, design frameworks and evaluation of prototypes. Some support was found for the claim that tabletops are a useful technology for collaboration, and several issues were discussed. Contributions to understanding in this field are delivered through design guidelines, heuristics, frameworks, and recommendations, in addition to the two case studies to help guide future tabletop system creators

Supporting Reflection and Classroom Orchestration with Tangible Tabletops

Tangible tabletop systems have been extensively proven to be able to enhance participation and engagement as well as enable many exciting activities, particularly in the education domain. However, it remains unclear as to whether students really benefit from using them for tasks that require a high level of reflection. Moreover, most existing tangible tabletops are designed as stand-alone systems or devices. Increasingly, this design assumption is no longer sufficient, especially in realistic learning settings. Due to the technological evolution in schools, multiple activities, resources, and constraints in the classroom ecosystem are now involved in the learning process. The way teachers manage technology-enhanced classrooms and the involved activities and constraints in real-time, also known as classroom orchestration, is a crucial aspect for the materialization of reflection and learning. This thesis aims to explore how educational tangible tabletop systems affect reflection, how reflection and orchestration are related, and how we can support reflection and orchestration to improve learning. It presents the design, implementation, and evaluations of three tangible tabletop systems – the DockLamp, the TinkerLamp, and the TinkerLamp 2.0 – in different learning contexts. Our experience with these systems, both inside and outside of the laboratory, results in an insightful understanding of the impacts of tangible tabletops on learning and the conditions for their effective use as well as deployment. These findings can be beneficial to the researchers and designers of learning environments using tangible tabletop and similar interfaces

The cockpit for the 21st century

Interactive surfaces are a growing trend in many domains. As one possible manifestation of Mark Weiser’s vision of ubiquitous and disappearing computers in everywhere objects, we see touchsensitive screens in many kinds of devices, such as smartphones, tablet computers and interactive tabletops. More advanced concepts of these have been an active research topic for many years. This has also influenced automotive cockpit development: concept cars and recent market releases show integrated touchscreens, growing in size. To meet the increasing information and interaction needs, interactive surfaces offer context-dependent functionality in combination with a direct input paradigm. However, interfaces in the car need to be operable while driving. Distraction, especially visual distraction from the driving task, can lead to critical situations if the sum of attentional demand emerging from both primary and secondary task overextends the available resources. So far, a touchscreen requires a lot of visual attention since its flat surface does not provide any haptic feedback. There have been approaches to make direct touch interaction accessible while driving for simple tasks. Outside the automotive domain, for example in office environments, concepts for sophisticated handling of large displays have already been introduced. Moreover, technological advances lead to new characteristics for interactive surfaces by enabling arbitrary surface shapes. In cars, two main characteristics for upcoming interactive surfaces are largeness and shape. On the one hand, spatial extension is not only increasing through larger displays, but also by taking objects in the surrounding into account for interaction. On the other hand, the flatness inherent in current screens can be overcome by upcoming technologies, and interactive surfaces can therefore provide haptically distinguishable surfaces. This thesis describes the systematic exploration of large and shaped interactive surfaces and analyzes their potential for interaction while driving. Therefore, different prototypes for each characteristic have been developed and evaluated in test settings suitable for their maturity level. Those prototypes were used to obtain subjective user feedback and objective data, to investigate effects on driving and glance behavior as well as usability and user experience. As a contribution, this thesis provides an analysis of the development of interactive surfaces in the car. Two characteristics, largeness and shape, are identified that can improve the interaction compared to conventional touchscreens. The presented studies show that large interactive surfaces can provide new and improved ways of interaction both in driver-only and driver-passenger situations. Furthermore, studies indicate a positive effect on visual distraction when additional static haptic feedback is provided by shaped interactive surfaces. Overall, various, non-exclusively applicable, interaction concepts prove the potential of interactive surfaces for the use in automotive cockpits, which is expected to be beneficial also in further environments where visual attention needs to be focused on additional tasks.Der Einsatz von interaktiven Oberflächen weitet sich mehr und mehr auf die unterschiedlichsten Lebensbereiche aus. Damit sind sie eine mögliche Ausprägung von Mark Weisers Vision der allgegenwärtigen Computer, die aus unserer direkten Wahrnehmung verschwinden. Bei einer Vielzahl von technischen Geräten des täglichen Lebens, wie Smartphones, Tablets oder interaktiven Tischen, sind berührungsempfindliche Oberflächen bereits heute in Benutzung. Schon seit vielen Jahren arbeiten Forscher an einer Weiterentwicklung der Technik, um ihre Vorteile auch in anderen Bereichen, wie beispielsweise der Interaktion zwischen Mensch und Automobil, nutzbar zu machen. Und das mit Erfolg: Interaktive Benutzeroberflächen werden mittlerweile serienmäßig in vielen Fahrzeugen eingesetzt. Der Einbau von immer größeren, in das Cockpit integrierten Touchscreens in Konzeptfahrzeuge zeigt, dass sich diese Entwicklung weiter in vollem Gange befindet. Interaktive Oberflächen ermöglichen das flexible Anzeigen von kontextsensitiven Inhalten und machen eine direkte Interaktion mit den Bildschirminhalten möglich. Auf diese Weise erfüllen sie die sich wandelnden Informations- und Interaktionsbedürfnisse in besonderem Maße. Beim Einsatz von Bedienschnittstellen im Fahrzeug ist die gefahrlose Benutzbarkeit während der Fahrt von besonderer Bedeutung. Insbesondere visuelle Ablenkung von der Fahraufgabe kann zu kritischen Situationen führen, wenn Primär- und Sekundäraufgaben mehr als die insgesamt verfügbare Aufmerksamkeit des Fahrers beanspruchen. Herkömmliche Touchscreens stellen dem Fahrer bisher lediglich eine flache Oberfläche bereit, die keinerlei haptische Rückmeldung bietet, weshalb deren Bedienung besonders viel visuelle Aufmerksamkeit erfordert. Verschiedene Ansätze ermöglichen dem Fahrer, direkte Touchinteraktion für einfache Aufgaben während der Fahrt zu nutzen. Außerhalb der Automobilindustrie, zum Beispiel für Büroarbeitsplätze, wurden bereits verschiedene Konzepte für eine komplexere Bedienung großer Bildschirme vorgestellt. Darüber hinaus führt der technologische Fortschritt zu neuen möglichen Ausprägungen interaktiver Oberflächen und erlaubt, diese beliebig zu formen. Für die nächste Generation von interaktiven Oberflächen im Fahrzeug wird vor allem an der Modifikation der Kategorien Größe und Form gearbeitet. Die Bedienschnittstelle wird nicht nur durch größere Bildschirme erweitert, sondern auch dadurch, dass Objekte wie Dekorleisten in die Interaktion einbezogen werden können. Andererseits heben aktuelle Technologieentwicklungen die Restriktion auf flache Oberflächen auf, so dass Touchscreens künftig ertastbare Strukturen aufweisen können. Diese Dissertation beschreibt die systematische Untersuchung großer und nicht-flacher interaktiver Oberflächen und analysiert ihr Potential für die Interaktion während der Fahrt. Dazu wurden für jede Charakteristik verschiedene Prototypen entwickelt und in Testumgebungen entsprechend ihres Reifegrads evaluiert. Auf diese Weise konnten subjektives Nutzerfeedback und objektive Daten erhoben, und die Effekte auf Fahr- und Blickverhalten sowie Nutzbarkeit untersucht werden. Diese Dissertation leistet den Beitrag einer Analyse der Entwicklung von interaktiven Oberflächen im Automobilbereich. Weiterhin werden die Aspekte Größe und Form untersucht, um mit ihrer Hilfe die Interaktion im Vergleich zu herkömmlichen Touchscreens zu verbessern. Die durchgeführten Studien belegen, dass große Flächen neue und verbesserte Bedienmöglichkeiten bieten können. Außerdem zeigt sich ein positiver Effekt auf die visuelle Ablenkung, wenn zusätzliches statisches, haptisches Feedback durch nicht-flache Oberflächen bereitgestellt wird. Zusammenfassend zeigen verschiedene, untereinander kombinierbare Interaktionskonzepte das Potential interaktiver Oberflächen für den automotiven Einsatz. Zudem können die Ergebnisse auch in anderen Bereichen Anwendung finden, in denen visuelle Aufmerksamkeit für andere Aufgaben benötigt wird

Exploring human-object interaction through force vector measurement

Thesis: S.M., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2019Cataloged from PDF version of thesis.Includes bibliographical references (pages 101-107).I introduce SCALE, a project aiming to further understand Human-Object Interaction through the real-time analysis of force vector signals, which I have defined as "Force-based Interaction" in this thesis. Force conveys fundamental information in Force-based Interaction, including force intensity, its direction, and object weight - information otherwise difficult to be accessed or inferred from other sensing modalities. To explore the design space of force-based interaction, I have developed the SCALE toolkit, which is composed of modularized 3d-axis force sensors and application APIs. In collaboration with big industry companies, this system has been applied to a variety of application domains and settings, including a retail store, a smart home and a farmers market. In this thesis, I have proposed a base system SCALE, and two additional advanced projects titled KI/OSK and DepthTouch, which build upon the SCALE project.by Takatoshi Yoshida.S.M.S.M. Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Science

Sistemas interativos tangíveis e processos de mediação tecnológica: hipóteses sobre agência, significação e cognição a partir da investigação do MIT Tangible Media Group

A presente dissertação toma a investigação em sistemas de interação tangível do MIT Tangible Media Group como objeto, a pretexto da sua inclusão na edição de 2016 do Festival Ars Electronica, sob o tema Radical Atoms: The Alchemists of Our Time. Pretende-se compreender quais os pontos de contato da investigação do grupo com os estudos dos media, de forma a localizar a sua relevância para a programação do festival. O enquadramento nos estudos dos media é feito pela localização de um conjunto de termos-chave no trabalho do grupo, os quais evocam questões afetas à fenomenologia, filosofia da tecnologia e mediação tecnológica. Conclui-se que estes sistemas de interação tangível ativam processos particulares de constituição de agência, significação e cognição. Na ausência de outros materiais que explorem estas relações no contexto do festival, a dissertação apresenta-se assim como complemento à leitura do tema Radical Atoms: The Alchemists of Our Time.This dissertation thesis takes the research of the MIT Tangible Media Group as its object, by occasion of its inclusion in the 2016 edition of Ars Electronica Festival under the theme Radical Atoms: The Alchemists of Our Time. The aim is to understand what are the common points between the group's research and media studies, in order to locate this object's relevance to the festival programming scope. The framing within media studies is done by surveying a set of keywords from the group's research, which evoke topics from phenomenology, philosophy of technology and technological mediation. It's concluded that these tangible interactive systems activate specific processes of agency, signification, and cognition. Given the lack of materials which explore these relationships within the context of the festival, the dissertation presents itself as a supplement to the reading of the Radical Atoms: The Alchemists of Our Time theme