Enabling mobile microinteractions

While much attention has been paid to the usability of desktop computers, mobile com- puters are quickly becoming the dominant platform. Because mobile computers may be used in nearly any situation--including while the user is actually in motion, or performing other tasks--interfaces designed for stationary use may be inappropriate, and alternative interfaces should be considered. In this dissertation I consider the idea of microinteractions--interactions with a device that take less than four seconds to initiate and complete. Microinteractions are desirable because they may minimize interruption; that is, they allow for a tiny burst of interaction with a device so that the user can quickly return to the task at hand. My research concentrates on methods for applying microinteractions through wrist- based interaction. I consider two modalities for this interaction: touchscreens and motion- based gestures. In the case of touchscreens, I consider the interface implications of making touchscreen watches usable with the finger, instead of the usual stylus, and investigate users' performance with a round touchscreen. For gesture-based interaction, I present a tool, MAGIC, for designing gesture-based interactive system, and detail the evaluation of the tool.Ph.D.Committee Chair: Starner, Thad; Committee Member: Abowd, Gregory; Committee Member: Isbell, Charles; Committee Member: Landay, james; Committee Member: McIntyre, Blai

Move, hold and touch: A framework for Tangible gesture interactive systems

© 2015 by the authors. Technology is spreading in our everyday world, and digital interaction beyond the screen, with real objects, allows taking advantage of our natural manipulative and communicative skills. Tangible gesture interaction takes advantage of these skills by bridging two popular domains in Human-Computer Interaction, tangible interaction and gestural interaction. In this paper, we present the Tangible Gesture Interaction Framework (TGIF) for classifying and guiding works in this field. We propose a classification of gestures according to three relationships with objects: move, hold and touch. Following this classification, we analyzed previous work in the literature to obtain guidelines and common practices for designing and building new tangible gesture interactive systems. We describe four interactive systems as application examples of the TGIF guidelines and we discuss the descriptive, evaluative and generative power of TGIF

Indutivo: Contact-Based, Object-Driven Interactions with Inductive Sensing

We present Indutivo, a contact-based inductive sensing technique for contextual interactions. Our technique recognizes conductive objects (metallic primarily) that are commonly found in households and daily environments, as well as their individual movements when placed against the sensor. These movements include sliding, hinging, and rotation. We describe our sensing principle and how we designed the size, shape, and layout of our sensor coils to optimize sensitivity, sensing range, recognition and tracking accuracy. Through several studies, we also demonstrated the performance of our proposed sensing technique in environments with varying levels of noise and interference conditions. We conclude by presenting demo applications on a smartwatch, as well as insights and lessons we learned from our experience

Making Spatial Information Accessible on Touchscreens for Users who are Blind and Visually Impaired

Touchscreens have become a de facto standard of input for mobile devices as they most optimally use the limited input and output space that is imposed by their form factor. In recent years, people who are blind and visually impaired have been increasing their usage of smartphones and touchscreens. Although basic access is available, there are still many accessibility issues left to deal with in order to bring full inclusion to this population. One of the important challenges lies in accessing and creating of spatial information on touchscreens. The work presented here provides three new techniques, using three different modalities, for accessing spatial information on touchscreens. The first system makes geometry and diagram creation accessible on a touchscreen through the use of text-to-speech and gestural input. This first study is informed by a qualitative study of how people who are blind and visually impaired currently access and create graphs and diagrams. The second system makes directions through maps accessible using multiple vibration sensors without any sound or visual output. The third system investigates the use of binaural sound on a touchscreen to make various types of applications accessible such as physics simulations, astronomy, and video games

Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

Design Principles of Mobile Information Systems in the Digital Transformation of the Workplace - Utilization of Smartwatch-based Information Systems in the Corporate Context

During the last decades, smartwatches emerged as an innovative and promising technology and hit the consumer market due to the accessibility of affordable devices and predominant acceptance caused by the considerable similarity to common wristwatches. With the unique characteristics of permanent availability, unobtrusiveness, and hands-free operation, they can provide additional value in the corporate context. Thus, this thesis analyzes use cases for smartwatches in companies, elaborates on the design of smartwatch-based information systems, and covers the usability of smartwatch applications during the development of smartwatch-based information systems. It is composed of three research complexes. The first research complex focuses on the digital assistance of (mobile) employees who have to execute manual work and have been excluded so far from the benefits of the digitalization since they cannot operate hand-held devices. The objective is to design smartwatch-based information systems to support workflows in the corporate context, facilitate the daily work of numerous employees, and make processes more efficient for companies. During a design science research approach, smartwatch-based software artifacts are designed and evaluated in use cases of production, support, security service, as well as logistics, and a nascent design theory is proposed to complement theory according to mobile information system research. The evaluation shows that, on the one hand, smartwatches have enormous potential to assist employees with a fast and ubiquitous exchange of information, instant notifications, collaboration, and workflow guidance while they can be operated incidentally during manual work. On the other hand, the design of smartwatch-based information systems is a crucial factor for successful long-term deployment in companies, and especially limitations according to the small form-factor, general conditions, acceptance of the employees, and legal regulations have to be addressed appropriately. The second research complex addresses smartwatch-based information systems at the office workplace. This broadens and complements the view on the utilization of smartwatches in the corporate context in addition to the mobile context described in the first research complex. Though smartwatches are devices constructed for mobile use, the utilization in low mobile or stationary scenarios also has benefits due they exhibit the characteristic of a wearable computer and are directly connected to the employee’s body. Various sensors can perceive employee-, environment- and therefore context-related information and demand the employees’ attention with proactive notifications that are accompanied by a vibration. Thus, a smartwatch-based and gamified information system for health promotion at the office workplace is designed and evaluated. Research complex three provides a closer look at the topic of usability concerning applications running on smartwatches since it is a crucial factor during the development cycle. As a supporting element for the studies within the first and second research complex, a framework for the usability analysis of smartwatch applications is developed. For research, this thesis contributes a systemization of the state-of-the-art of smartwatch utilization in the corporate context, enabling and inhibiting influence factors of the smartwatch adoption in companies, and design principles as well as a nascent design theory for smartwatch-based information systems to support mobile employees executing manual work. For practice, this thesis contributes possible use cases for smartwatches in companies, assistance in decision-making for the introduction of smartwatch-based information systems in the corporate context with the Smartwatch Applicability Framework, situated implementations of a smartwatch-based information system for typical use cases, design recommendations for smartwatch-based information systems, an implementation of a smartwatch-based information system for the support of mobile employees executing manual work, and a usability-framework for smartwatches to automatically access usability of existing applications providing suggestions for usability improvement

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo

Security and Privacy in Mobile Computing: Challenges and Solutions

abstract: Mobile devices are penetrating everyday life. According to a recent Cisco report [10], the number of mobile connected devices such as smartphones, tablets, laptops, eReaders, and Machine-to-Machine (M2M) modules will hit 11.6 billion by 2021, exceeding the world's projected population at that time (7.8 billion). The rapid development of mobile devices has brought a number of emerging security and privacy issues in mobile computing. This dissertation aims to address a number of challenging security and privacy issues in mobile computing. This dissertation makes fivefold contributions. The first and second parts study the security and privacy issues in Device-to-Device communications. Specifically, the first part develops a novel scheme to enable a new way of trust relationship called spatiotemporal matching in a privacy-preserving and efficient fashion. To enhance the secure communication among mobile users, the second part proposes a game-theoretical framework to stimulate the cooperative shared secret key generation among mobile users. The third and fourth parts investigate the security and privacy issues in mobile crowdsourcing. In particular, the third part presents a secure and privacy-preserving mobile crowdsourcing system which strikes a good balance among object security, user privacy, and system efficiency. The fourth part demonstrates a differentially private distributed stream monitoring system via mobile crowdsourcing. Finally, the fifth part proposes VISIBLE, a novel video-assisted keystroke inference framework that allows an attacker to infer a tablet user's typed inputs on the touchscreen by recording and analyzing the video of the tablet backside during the user's input process. Besides, some potential countermeasures to this attack are also discussed. This dissertation sheds the light on the state-of-the-art security and privacy issues in mobile computing.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201