Toward New Ecologies of Cyberphysical Representational Forms, Scales, and Modalities

Research on tangible user interfaces commonly focuses on tangible interfaces acting alone or in comparison with screen-based multi-touch or graphical interfaces. In contrast, hybrid approaches can be seen as the norm for established mainstream interaction paradigms. This dissertation describes interfaces that support complementary information mediations, representational forms, and scales toward an ecology of systems embodying hybrid interaction modalities. I investigate systems combining tangible and multi-touch, as well as systems combining tangible and virtual reality interaction. For each of them, I describe work focusing on design and fabrication aspects, as well as work focusing on reproducibility, engagement, legibility, and perception aspects

Kosketuskäyttöliittymän toteuttaminen olemassa olevaan ohjelmaan

The purpose of this work was to evaluate the migration steps of a windowing desktop application into a touch based input enabled software. The study was conducted on an already existing building information modelling software called Tekla BIMsight. The task was to retain all the functionality already in the software while making it possible to be used on touch-enabled devices, such as tablets or convertible laptops with a swivel display. Design and implementation of the system has been documented as part of the thesis, as well as most problematic issues during this period. The effects of the implementation are validated and tested with real users and the results from that study were documented. The usability study was conducted to obtain quantitative and qualitative metrics of the usability. The nature of the input mechanism, direct or indirect, affects the user experience greatly. The final system should be as responsive as possible to maintain a good level of perceived performance. Early prototyping and access to the target devices is critical to the success of a migration process. There are several common mistakes that should be avoided in the design and implementation phases. Not all the problems were critical, but many of them were identified as very cumbersome for the user that would affect the positive user experience of the software. With each new context for a user interface the problems need to be solved again and only experience from such solutions can help alleviate this task. The implemented touch support can be verified to meet the set requirements very well: It allows the system to be used on touch based input environments and all the major user interface elements support this.Työn tarkoituksena oli toteuttaa ja arvioida toimenpiteet ja. menetelmät joilla olemassa olevaan käyttöliittymään voidaan lisätä tuki kosketuskäytölle. Ominaisuudet lisättiin rakennusten tietomallinnuksen tarkasteluohjelmaan, Tekla BIMsight. Tehtävänä oli säilyttää kaikki aiemmat toiminnot ja tehdä ohjelmasta tehokkaasti käytettävä kosketuslaitteilla, kuten tableteilla ja kääntyvällä näytöllä varustetuilla kannettavilla. Suunnittelu ja toteutus järjestelmälle on dokumentoitu työssä ja kaikkein vaativimmat ongelmat. Toteutetun tuen vaikutuksia arvioitiin oikeiden käyttäjien kanssa tehdyssä käyttäjätutkimuksessa, jonka tulokset on esitetty. Käytettävyystutkimuksella hankittiin kvantitatiivista ja kvalitatiivista tietoa tuotteesta. Laite jolla ohjelmistoa käytetään vaikuttaa ohjelmasta saatuun käyttökokemukseen merkittävästi. Hyvän käyttökokemuksen saavuttamiseksi lopullisen järjestelmän käytön tulisi olla sujuvaa. Aikaisten prototyyppien kokeilu ja kohdelaitteiden saatavuus ovat tärkeitä tekijöitä siirtymäprosessin kannalta. Yleisiä ongelmatilanteita ja haasteita joita kohdattiin suunnittelu- ja toteutusvaiheissa on listattu työssä. Loppukäyttäjän kannalta useat ongelmat olivat rasittavia ja vaikuttaisivat käyttökokemukseen negatiivisesti jos niitä ei korjata. Uuden käyttöympäristön tuomat ongelmat joudutaan ratkaisemaan aina uudestaan. Vain kokemuksella vastaavista tilanteista on merkittävästi etua itse ratkaisujen löytämiselle. Toteutetun kosketuskäyttöliittymän tuen voidaan todeta vastaavan sille asetettuja tavoitteita ja vaatimuksia hyvin; se mahdollistaa ohjelman käyttämisen kosketuskäyttöliittymän omaavissa laitteissa ja kaikkein merkittävimmät käyttöliittymäelementit on tuettuina

Multimodal interaction: developing an interaction concept for a touchscreen incorporating tactile feedback

The touchscreen, as an alternative user interface for applications that normally require mice and keyboards, has become more and more commonplace, showing up on mobile devices, on vending machines, on ATMs and in the control panels of machines in industry, where conventional input devices cannot provide intuitive, rapid and accurate user interaction with the content of the display. The exponential growth in processing power on the PC, together with advances in understanding human communication channels, has had a significant effect on the design of usable, human-factored interfaces on touchscreens, and on the number and complexity of applications available on touchscreens. Although computer-driven touchscreen interfaces provide programmable and dynamic displays, the absence of the expected tactile cues on the hard and static surfaces of conventional touchscreens is challenging interface design and touchscreen usability, in particular for distracting, low-visibility environments. Current technology allows the human tactile modality to be used in touchscreens. While the visual channel converts graphics and text unidirectionally from the computer to the end user, tactile communication features a bidirectional information flow to and from the user as the user perceives and acts on the environment and the system responds to changing contextual information. Tactile sensations such as detents and pulses provide users with cues that make selecting and controlling a more intuitive process. Tactile features can compensate for deficiencies in some of the human senses, especially in tasks which carry a heavy visual or auditory burden. In this study, an interaction concept for tactile touchscreens is developed with a view to employing the key characteristics of the human sense of touch effectively and efficiently, especially in distracting environments where vision is impaired and hearing is overloaded. As a first step toward improving the usability of touchscreens through the integration of tactile effects, different mechanical solutions for producing motion in tactile touchscreens are investigated, to provide a basis for selecting suitable vibration directions when designing tactile displays. Building on these results, design know-how regarding tactile feedback patterns is further developed to enable dynamic simulation of UI controls, in order to give users a sense of perceiving real controls on a highly natural touch interface. To study the value of adding tactile properties to touchscreens, haptically enhanced UI controls are then further investigated with the aim of mapping haptic signals to different usage scenarios to perform primary and secondary tasks with touchscreens. The findings of the study are intended for consideration and discussion as a guide to further development of tactile stimuli, haptically enhanced user interfaces and touchscreen applications

The Use of Multiple Slate Devices to Support Active Reading Activities

Reading activities in the classroom and workplace occur predominantly on paper. Since existing electronic devices do not support these reading activities as well as paper, users have difficulty taking full advantage of the affordances of electronic documents. This dissertation makes three main contributions toward supporting active reading electronically. The first contribution is a comprehensive set of active reading requirements, drawn from three decades of research into reading processes. These requirements explain why existing devices are inadequate for supporting active reading activities. The second contribution is a multi-slate reading system that more completely supports the active reading requirements above. Researchers believe the suitability of paper for active reading is largely due to the fact it distributes content across different sheets of paper, which are capable of displaying information as well as capturing input. The multi-slate approach draws inspiration from the independent reading and writing surfaces that paper provides, to blend the beneficial features of e-book readers, tablets, PCs, and tabletop computers. The development of the multi-slate system began with the Dual-Display E-book, which used two screens to provide richer navigation capabilities than a single-screen device. Following the success of the Dual-Display E-book, the United Slates, a general-purpose reading system consisting of an extensible number of slates, was created. The United Slates consisted of custom slate hardware, specialized interactions that enabled the slates to be used cooperatively, and a cloud-based infrastructure that robustly integrated the slates with users' existing computing devices and workflow. The third contribution is a series of evaluations that characterized reading with multiple slates. A laboratory study with 12 participants compared the relative merits of paper and electronic reading surfaces. One month long in-situ deployments of the United Slates with graduate students in the humanities found the multi-slate configuration to be highly effective for reading. The United Slates system delivered desirable paper-like qualities that included enhanced reading engagement, ease of navigation, and peace-of-mind while also providing superior electronic functionality. The positive feedback suggests that the multi-slate configuration is a desirable method for supporting active reading activities

Sistemas interativos tangíveis e processos de mediação tecnológica: hipóteses sobre agência, significação e cognição a partir da investigação do MIT Tangible Media Group

A presente dissertação toma a investigação em sistemas de interação tangível do MIT Tangible Media Group como objeto, a pretexto da sua inclusão na edição de 2016 do Festival Ars Electronica, sob o tema Radical Atoms: The Alchemists of Our Time. Pretende-se compreender quais os pontos de contato da investigação do grupo com os estudos dos media, de forma a localizar a sua relevância para a programação do festival. O enquadramento nos estudos dos media é feito pela localização de um conjunto de termos-chave no trabalho do grupo, os quais evocam questões afetas à fenomenologia, filosofia da tecnologia e mediação tecnológica. Conclui-se que estes sistemas de interação tangível ativam processos particulares de constituição de agência, significação e cognição. Na ausência de outros materiais que explorem estas relações no contexto do festival, a dissertação apresenta-se assim como complemento à leitura do tema Radical Atoms: The Alchemists of Our Time.This dissertation thesis takes the research of the MIT Tangible Media Group as its object, by occasion of its inclusion in the 2016 edition of Ars Electronica Festival under the theme Radical Atoms: The Alchemists of Our Time. The aim is to understand what are the common points between the group's research and media studies, in order to locate this object's relevance to the festival programming scope. The framing within media studies is done by surveying a set of keywords from the group's research, which evoke topics from phenomenology, philosophy of technology and technological mediation. It's concluded that these tangible interactive systems activate specific processes of agency, signification, and cognition. Given the lack of materials which explore these relationships within the context of the festival, the dissertation presents itself as a supplement to the reading of the Radical Atoms: The Alchemists of Our Time theme

Designing Hybrid Interactions through an Understanding of the Affordances of Physical and Digital Technologies

Two recent technological advances have extended the diversity of domains and social contexts of Human-Computer Interaction: the embedding of computing capabilities into physical hand-held objects, and the emergence of large interactive surfaces, such as tabletops and wall boards. Both interactive surfaces and small computational devices usually allow for direct and space-multiplex input, i.e., for the spatial coincidence of physical action and digital output, in multiple points simultaneously. Such a powerful combination opens novel opportunities for the design of what are considered as hybrid interactions in this work. This thesis explores the affordances of physical interaction as resources for interface design of such hybrid interactions. The hybrid systems that are elaborated in this work are envisioned to support specific social and physical contexts, such as collaborative cooking in a domestic kitchen, or collaborative creativity in a design process. In particular, different aspects of physicality characteristic of those specific domains are explored, with the aim of promoting skill transfer across domains. irst, different approaches to the design of space-multiplex, function-specific interfaces are considered and investigated. Such design approaches build on related work on Graspable User Interfaces and extend the design space to direct touch interfaces such as touch-sensitive surfaces, in different sizes and orientations (i.e., tablets, interactive tabletops, and walls). These approaches are instantiated in the design of several experience prototypes: These are evaluated in different settings to assess the contextual implications of integrating aspects of physicality in the design of the interface. Such implications are observed both at the pragmatic level of interaction (i.e., patterns of users' behaviors on first contact with the interface), as well as on user' subjective response. The results indicate that the context of interaction affects the perception of the affordances of the system, and that some qualities of physicality such as the 3D space of manipulation and relative haptic feedback can affect the feeling of engagement and control. Building on these findings, two controlled studies are conducted to observe more systematically the implications of integrating some of the qualities of physical interaction into the design of hybrid ones. The results indicate that, despite the fact that several aspects of physical interaction are mimicked in the interface, the interaction with digital media is quite different and seems to reveal existing mental models and expectations resulting from previous experience with the WIMP paradigm on the desktop PC

Haptic Media Scenes

The aim of this thesis is to apply new media phenomenological and enactive embodied cognition approaches to explain the role of haptic sensitivity and communication in personal computer environments for productivity. Prior theory has given little attention to the role of haptic senses in influencing cognitive processes, and do not frame the richness of haptic communication in interaction design—as haptic interactivity in HCI has historically tended to be designed and analyzed from a perspective on communication as transmissions, sending and receiving haptic signals. The haptic sense may not only mediate contact confirmation and affirmation, but also rich semiotic and affective messages—yet this is a strong contrast between this inherent ability of haptic perception, and current day support for such haptic communication interfaces. I therefore ask: How do the haptic senses (touch and proprioception) impact our cognitive faculty when mediated through digital and sensor technologies? How may these insights be employed in interface design to facilitate rich haptic communication? To answer these questions, I use theoretical close readings that embrace two research fields, new media phenomenology and enactive embodied cognition. The theoretical discussion is supported by neuroscientific evidence, and tested empirically through case studies centered on digital art. I use these insights to develop the concept of the haptic figura, an analytical tool to frame the communicative qualities of haptic media. The concept gauges rich machine- mediated haptic interactivity and communication in systems with a material solution supporting active haptic perception, and the mediation of semiotic and affective messages that are understood and felt. As such the concept may function as a design tool for developers, but also for media critics evaluating haptic media. The tool is used to frame a discussion on opportunities and shortcomings of haptic interfaces for productivity, differentiating between media systems for the hand and the full body. The significance of this investigation is demonstrating that haptic communication is an underutilized element in personal computer environments for productivity and providing an analytical framework for a more nuanced understanding of haptic communication as enabling the mediation of a range of semiotic and affective messages, beyond notification and confirmation interactivity

Haptics Rendering and Applications

There has been significant progress in haptic technologies but the incorporation of haptics into virtual environments is still in its infancy. A wide range of the new society's human activities including communication, education, art, entertainment, commerce and science would forever change if we learned how to capture, manipulate and reproduce haptic sensory stimuli that are nearly indistinguishable from reality. For the field to move forward, many commercial and technological barriers need to be overcome. By rendering how objects feel through haptic technology, we communicate information that might reflect a desire to speak a physically- based language that has never been explored before. Due to constant improvement in haptics technology and increasing levels of research into and development of haptics-related algorithms, protocols and devices, there is a belief that haptics technology has a promising future

Augmented Reality Interfaces for Procedural Tasks

Procedural tasks involve people performing established sequences of activities while interacting with objects in the physical environment to accomplish particular goals. These tasks span almost all aspects of human life and vary greatly in their complexity. For some simple tasks, little cognitive assistance is required beyond an initial learning session in which a person follows one-time compact directions, or even intuition, to master a sequence of activities. In the case of complex tasks, procedural assistance may be continually required, even for the most experienced users. Approaches for rendering this assistance employ a wide range of written, audible, and computer-based technologies. This dissertation explores an approach in which procedural task assistance is rendered using augmented reality. Augmented reality integrates virtual content with a user's natural view of the environment, combining real and virtual objects interactively, and aligning them with each other. Our thesis is that an augmented reality interface can allow individuals to perform procedural tasks more quickly while exerting less effort and making fewer errors than other forms of assistance. This thesis is supported by several significant contributions yielded during the exploration of the following research themes: What aspects of AR are applicable and beneficial to the procedural task problem? In answering this question, we developed two prototype AR interfaces that improve procedural task accomplishment. The first prototype was designed to assist mechanics carrying out maintenance procedures under field conditions. An evaluation involving professional mechanics showed our prototype reduced the time required to locate procedural tasks and resulted in fewer head movements while transitioning between tasks. Following up on this work, we constructed another prototype that focuses on providing assistance in the underexplored psychomotor phases of procedural tasks. This prototype presents dynamic and prescriptive forms of instruction and was evaluated using a demanding and realistic alignment task. This evaluation revealed that the AR prototype allowed participants to complete the alignment more quickly and accurately than when using an enhanced version of currently employed documentation systems. How does the user interact with an AR application assisting with procedural tasks? The application of AR to the procedural task problem poses unique user interaction challenges. To meet these challenges, we present and evaluate a novel class of user interfaces that leverage naturally occurring and otherwise unused affordances in the native environment to provide a tangible user interface for augmented reality applications. This class of techniques, which we call Opportunistic Controls, combines hand gestures, overlaid virtual widgets, and passive haptics to form an interface that was proven effective and intuitive during quantitative evaluation. Our evaluation of these techniques includes a qualitative exploration of various preferences and heuristics for Opportunistic Control-based designs