122 research outputs found

    Intelligent Techniques to Accelerate Everyday Text Communication

    Get PDF
    People with some form of speech- or motor-impairments usually use a high-tech augmentative and alternative communication (AAC) device to communicate with other people in writing or in face-to-face conversations. Their text entry rate on these devices is slow due to their motor abilities. Making good letter or word predictions can help accelerate the communication of such users. In this dissertation, we investigated several approaches to accelerate input for AAC users. First, considering that an AAC user is participating in a face-to-face conversation, we investigated whether performing speech recognition on the speaking-side can improve next word predictions. We compared the accuracy of three plausible microphone deployment options and the accuracy of two commercial speech recognition engines. We found that despite recognition word error rates of 7-16%, our ensemble of n-gram and recurrent neural network language models made predictions nearly as good as when they used the reference transcripts. In a user study with 160 participants, we also found that increasing number of prediction slots in a keyboard interface does not necessarily correlate to improved performance. Second, typing every character in a text message may require an AAC user more time or effort than strictly necessary. Skipping spaces or other characters may be able to speed input and reduce an AAC user\u27s physical input effort. We designed a recognizer optimized for expanding noisy abbreviated input where users often omitted spaces and mid-word vowels. We showed using neural language models for selecting conversational-style training text and for rescoring the recognizer\u27s n-best sentences improved accuracy. We found accurate abbreviated input was possible even if a third of characters was omitted. In a study where users had to dwell for a second on each key, we found sentence abbreviated input was competitive with a conventional keyboard with word predictions. Finally, AAC keyboards rely on language modeling to auto-correct noisy typing and to offer word predictions. While today language models can be trained on huge amounts of text, pre-trained models may fail to capture the unique writing style and vocabulary of individual users. We demonstrated improved performance compared to a unigram cache by adapting to a user\u27s text via language models based on prediction by partial match (PPM) and recurrent neural networks. Our best model ensemble increased keystroke savings by 9.6%

    WearPut : Designing Dexterous Wearable Input based on the Characteristics of Human Finger Motions

    Get PDF
    Department of Biomedical Engineering (Human Factors Engineering)Powerful microchips for computing and networking allow a wide range of wearable devices to be miniaturized with high fidelity and availability. In particular, the commercially successful smartwatches placed on the wrist drive market growth by sharing the role of smartphones and health management. The emerging Head Mounted Displays (HMDs) for Augmented Reality (AR) and Virtual Reality (VR) also impact various application areas in video games, education, simulation, and productivity tools. However, these powerful wearables have challenges in interaction with the inevitably limited space for input and output due to the specialized form factors for fitting the body parts. To complement the constrained interaction experience, many wearable devices still rely on other large form factor devices (e.g., smartphones or hand-held controllers). Despite their usefulness, the additional devices for interaction can constrain the viability of wearable devices in many usage scenarios by tethering users' hands to the physical devices. This thesis argues that developing novel Human-Computer interaction techniques for the specialized wearable form factors is vital for wearables to be reliable standalone products. This thesis seeks to address the issue of constrained interaction experience with novel interaction techniques by exploring finger motions during input for the specialized form factors of wearable devices. The several characteristics of the finger input motions are promising to enable increases in the expressiveness of input on the physically limited input space of wearable devices. First, the input techniques with fingers are prevalent on many large form factor devices (e.g., touchscreen or physical keyboard) due to fast and accurate performance and high familiarity. Second, many commercial wearable products provide built-in sensors (e.g., touchscreen or hand tracking system) to detect finger motions. This enables the implementation of novel interaction systems without any additional sensors or devices. Third, the specialized form factors of wearable devices can create unique input contexts while the fingers approach their locations, shapes, and components. Finally, the dexterity of fingers with a distinctive appearance, high degrees of freedom, and high sensitivity of joint angle perception have the potential to widen the range of input available with various movement features on the surface and in the air. Accordingly, the general claim of this thesis is that understanding how users move their fingers during input will enable increases in the expressiveness of the interaction techniques we can create for resource-limited wearable devices. This thesis demonstrates the general claim by providing evidence in various wearable scenarios with smartwatches and HMDs. First, this thesis explored the comfort range of static and dynamic touch input with angles on the touchscreen of smartwatches. The results showed the specific comfort ranges on variations in fingers, finger regions, and poses due to the unique input context that the touching hand approaches a small and fixed touchscreen with a limited range of angles. Then, finger region-aware systems that recognize the flat and side of the finger were constructed based on the contact areas on the touchscreen to enhance the expressiveness of angle-based touch input. In the second scenario, this thesis revealed distinctive touch profiles of different fingers caused by the unique input context for the touchscreen of smartwatches. The results led to the implementation of finger identification systems for distinguishing two or three fingers. Two virtual keyboards with 12 and 16 keys showed the feasibility of touch-based finger identification that enables increases in the expressiveness of touch input techniques. In addition, this thesis supports the general claim with a range of wearable scenarios by exploring the finger input motions in the air. In the third scenario, this thesis investigated the motions of in-air finger stroking during unconstrained in-air typing for HMDs. The results of the observation study revealed details of in-air finger motions during fast sequential input, such as strategies, kinematics, correlated movements, inter-fingerstroke relationship, and individual in-air keys. The in-depth analysis led to a practical guideline for developing robust in-air typing systems with finger stroking. Lastly, this thesis examined the viable locations of in-air thumb touch input to the virtual targets above the palm. It was confirmed that fast and accurate sequential thumb touch can be achieved at a total of 8 key locations with the built-in hand tracking system in a commercial HMD. Final typing studies with a novel in-air thumb typing system verified increases in the expressiveness of virtual target selection on HMDs. This thesis argues that the objective and subjective results and novel interaction techniques in various wearable scenarios support the general claim that understanding how users move their fingers during input will enable increases in the expressiveness of the interaction techniques we can create for resource-limited wearable devices. Finally, this thesis concludes with thesis contributions, design considerations, and the scope of future research works, for future researchers and developers to implement robust finger-based interaction systems on various types of wearable devices.ope

    Integrating passive ubiquitous surfaces into human-computer interaction

    Get PDF
    Mobile technologies enable people to interact with computers ubiquitously. This dissertation investigates how ordinary, ubiquitous surfaces can be integrated into human-computer interaction to extend the interaction space beyond the edge of the display. It turns out that acoustic and tactile features generated during an interaction can be combined to identify input events, the user, and the surface. In addition, it is shown that a heterogeneous distribution of different surfaces is particularly suitable for realizing versatile interaction modalities. However, privacy concerns must be considered when selecting sensors, and context can be crucial in determining whether and what interaction to perform.Mobile Technologien ermöglichen den Menschen eine allgegenwĂ€rtige Interaktion mit Computern. Diese Dissertation untersucht, wie gewöhnliche, allgegenwĂ€rtige OberflĂ€chen in die Mensch-Computer-Interaktion integriert werden können, um den Interaktionsraum ĂŒber den Rand des Displays hinaus zu erweitern. Es stellt sich heraus, dass akustische und taktile Merkmale, die wĂ€hrend einer Interaktion erzeugt werden, kombiniert werden können, um Eingabeereignisse, den Benutzer und die OberflĂ€che zu identifizieren. DarĂŒber hinaus wird gezeigt, dass eine heterogene Verteilung verschiedener OberflĂ€chen besonders geeignet ist, um vielfĂ€ltige InteraktionsmodalitĂ€ten zu realisieren. Bei der Auswahl der Sensoren mĂŒssen jedoch Datenschutzaspekte berĂŒcksichtigt werden, und der Kontext kann entscheidend dafĂŒr sein, ob und welche Interaktion durchgefĂŒhrt werden soll

    Seamless Authentication for Ubiquitous Devices

    Get PDF
    User authentication is an integral part of our lives; we authenticate ourselves to personal computers and a variety of other things several times a day. Authentication is burdensome. When we wish to access to a computer or a resource, it is an additional task that we need to perform~-- an interruption in our workflow. In this dissertation, we study people\u27s authentication behavior and attempt to make authentication to desktops and smartphones less burdensome for users. First, we present the findings of a user study we conducted to understand people\u27s authentication behavior: things they authenticate to, how and when they authenticate, authentication errors they encounter and why, and their opinions about authentication. In our study, participants performed about 39 authentications per day on average; the majority of these authentications were to personal computers (desktop, laptop, smartphone, tablet) and with passwords, but the number of authentications to other things (e.g., car, door) was not insignificant. We saw a high failure rate for desktop and laptop authentication among our participants, affirming the need for a more usable authentication method. Overall, we found that authentication was a noticeable part of all our participants\u27 lives and burdensome for many participants, but they accepted it as cost of security, devising their own ways to cope with it. Second, we propose a new approach to authentication, called bilateral authentication, that leverages wrist-wearable technology to enable seamless authentication for things that people use with their hands, while wearing a smart wristband. In bilateral authentication two entities (e.g., user\u27s wristband and the user\u27s phone) share their knowledge (e.g., about user\u27s interaction with the phone) to verify the user\u27s identity. Using this approach, we developed a seamless authentication method for desktops and smartphones. Our authentication method offers quick and effortless authentication, continuous user verification while the desktop (or smartphone) is in use, and automatic deauthentication after use. We evaluated our authentication method through four in-lab user studies, evaluating the method\u27s usability and security from the system and the user\u27s perspective. Based on the evaluation, our authentication method shows promise for reducing users\u27 authentication burden for desktops and smartphones

    Data-Driven Evaluation of In-Vehicle Information Systems

    Get PDF
    Today’s In-Vehicle Information Systems (IVISs) are featurerich systems that provide the driver with numerous options for entertainment, information, comfort, and communication. Drivers can stream their favorite songs, read reviews of nearby restaurants, or change the ambient lighting to their liking. To do so, they interact with large center stack touchscreens that have become the main interface between the driver and IVISs. To interact with these systems, drivers must take their eyes off the road which can impair their driving performance. This makes IVIS evaluation critical not only to meet customer needs but also to ensure road safety. The growing number of features, the distraction caused by large touchscreens, and the impact of driving automation on driver behavior pose significant challenges for the design and evaluation of IVISs. Traditionally, IVISs are evaluated qualitatively or through small-scale user studies using driving simulators. However, these methods are not scalable to the growing number of features and the variety of driving scenarios that influence driver interaction behavior. We argue that data-driven methods can be a viable solution to these challenges and can assist automotive User Experience (UX) experts in evaluating IVISs. Therefore, we need to understand how data-driven methods can facilitate the design and evaluation of IVISs, how large amounts of usage data need to be visualized, and how drivers allocate their visual attention when interacting with center stack touchscreens. In Part I, we present the results of two empirical studies and create a comprehensive understanding of the role that data-driven methods currently play in the automotive UX design process. We found that automotive UX experts face two main conflicts: First, results from qualitative or small-scale empirical studies are often not valued in the decision-making process. Second, UX experts often do not have access to customer data and lack the means and tools to analyze it appropriately. As a result, design decisions are often not user-centered and are based on subjective judgments rather than evidence-based customer insights. Our results show that automotive UX experts need data-driven methods that leverage large amounts of telematics data collected from customer vehicles. They need tools to help them visualize and analyze customer usage data and computational methods to automatically evaluate IVIS designs. In Part II, we present ICEBOAT, an interactive user behavior analysis tool for automotive user interfaces. ICEBOAT processes interaction data, driving data, and glance data, collected over-the-air from customer vehicles and visualizes it on different levels of granularity. Leveraging our multi-level user behavior analysis framework, it enables UX experts to effectively and efficiently evaluate driver interactions with touchscreen-based IVISs concerning performance and safety-related metrics. In Part III, we investigate drivers’ multitasking behavior and visual attention allocation when interacting with center stack touchscreens while driving. We present the first naturalistic driving study to assess drivers’ tactical and operational self-regulation with center stack touchscreens. Our results show significant differences in drivers’ interaction and glance behavior in response to different levels of driving automation, vehicle speed, and road curvature. During automated driving, drivers perform more interactions per touchscreen sequence and increase the time spent looking at the center stack touchscreen. These results emphasize the importance of context-dependent driver distraction assessment of driver interactions with IVISs. Motivated by this we present a machine learning-based approach to predict and explain the visual demand of in-vehicle touchscreen interactions based on customer data. By predicting the visual demand of yet unseen touchscreen interactions, our method lays the foundation for automated data-driven evaluation of early-stage IVIS prototypes. The local and global explanations provide additional insights into how design artifacts and driving context affect drivers’ glance behavior. Overall, this thesis identifies current shortcomings in the evaluation of IVISs and proposes novel solutions based on visual analytics and statistical and computational modeling that generate insights into driver interaction behavior and assist UX experts in making user-centered design decisions

    An Activity-Centric Approach to Configuration Work in Distributed Interaction

    Get PDF
    The widespread introduction of new types of computing devices, such as smartphones, tablet computers, large interactive displays or even wearable devices, has led to setups in which users are interacting with a rich ecology of devices. These new device ecologies have the potential to introduce a whole new set of cross-device and cross-user interactions as well as to support seamless distributed workspaces that facilitate coordination and communication with other users. Because of the distributed nature of this paradigm, there is an intrinsic difficulty and overhead in managing and using these kind of complex device ecologies, which I refer to as configuration work. It is the effort required to set up, manage, communicate, understand and use information, applications and services that are distributed over all devices in use and people involved. Because current devices and their containing software are still document- and application-centric, they fail to capture and support the rich activities and context in which they are being used. This leaves users without a stable concept for cross-device information management, forcing them to perform a large amount of manual configuration work. In this dissertation, I explore an activity-centric approach to configuration work in distributed interaction. The central goal of this dissertation is to develop and apply concepts and ideas from Activity-Centric Computing to distributed interaction. Using the triangulation approach, I explore these concepts on a conceptual, empirical and technological level and present a framework and use cases for designing activitycentric configurations in multi-device information systems. The dissertation presents two major contributions: First, I introduce the term configuration work as an abstract analytical unit that describes and captures the problems and challenges of distributed interaction. Using both empirical data and related work, I argue that configuration work is composed of: curation work, task resumption lag, mobility work, physical handling and articulation work. Using configuration work as a problem description, I operationalize Activity Theory and Activity-Centric Computing to mitigate and reduce configuration work in distributed interaction. By allowing users to interact with computational representations of their real-world activities, creating complex multi-user device ecologies and switching between cross-device information configurations will be more efficient, more effective and provide better support for users’ mental model about a multi-user and multi-device environment. Using activity configuration as a central concept, I introduce a framework that describes how digital representations of human activity can be distributed, fragmented and used across multiple devices and users. Second, I present a technical infrastructure and four applications that apply the concepts of activity configuration. The infrastructure is a general purpose platform for the design, development and deployment of distributed activitycentric systems. The infrastructure simplifies the development of activity-centric systems as it presents complex distributed computing processes and services into high level activity system abstractions. Using this infrastructure and conceptual framework, I describe four fully working applications that explore multi-device interactions in two specific domains: office work and hospital work. The systems are evaluated and tested with end-users in a number of lab and field studies

    Understanding Mode and Modality Transfer in Unistroke Gesture Input

    Get PDF
    Unistroke gestures are an attractive input method with an extensive research history, but one challenge with their usage is that the gestures are not always self-revealing. To obtain expertise with these gestures, interaction designers often deploy a guided novice mode -- where users can rely on recognizing visual UI elements to perform a gestural command. Once a user knows the gesture and associated command, they can perform it without guidance; thus, relying on recall. The primary aim of my thesis is to obtain a comprehensive understanding of why, when, and how users transfer from guided modes or modalities to potentially more efficient, or novel, methods of interaction -- through symbolic-abstract unistroke gestures. The goal of my work is to not only study user behaviour from novice to more efficient interaction mechanisms, but also to expand upon the concept of intermodal transfer to different contexts. We garner this understanding by empirically evaluating three different use cases of mode and/or modality transitions. Leveraging marking menus, the first piece investigates whether or not designers should force expertise transfer by penalizing use of the guided mode, in an effort to encourage use of the recall mode. Second, we investigate how well users can transfer skills between modalities, particularly when it is impractical to present guidance in the target or recall modality. Lastly, we assess how well users' pre-existing spatial knowledge of an input method (the QWERTY keyboard layout), transfers to performance in a new modality. Applying lessons from these three assessments, we segment intermodal transfer into three possible characterizations -- beyond the traditional novice to expert contextualization. This is followed by a series of implications and potential areas of future exploration spawning from our work

    Consumer Contracts, Copyright Licensing, and Control over Data on the Internet of Things

    Get PDF
    This article presents our interdisciplinary analysis of end-user license agreements and privacy policies from a sample of 22 consumer goods/services connected to the Internet of Things (IoT). We gathered data in the form of legal documents and assessed them from legal and economic perspectives. We developed an original taxonomy of IoT-connected consumer goods/services, classified different business models built around them, and reviewed legal terms and conditions related to their use. Our analysis identifies copyright related restrictions and brings to light issues beyond copyright that merit consideration in the context of a review of copyright law and policy. First, we find that even obtaining legal information on smart products, including software license restrictions and other copyright limitations, is a difficult and time-consuming exercise. Second, our analysis of business models shows interoperability of platforms within an ecosystem of third-party devices and applications, but restrictions that limit interoperability across ecosystems. Third, terms and conditions of consumer use of smart devices in our sample are set up to allow for the collection and transfer of personal data, often sensitive data, in addition to all data collected by the companies from other sources such as social media. Fourth, our study shows that software licensing is now common practice among smart device manufacturers. Based on these findings, we make recommendations to address the issues of accessibility of legal information, data portability, interoperability of systems, and competition. We recommend that governments cooperate with industry, consumer, and public interest groups to: (1) promote labelling standards to help consumers locate and understand the terms on which they acquire and use IoT products and services; (2) support open standards and protocols to facilitate interoperability across platforms; (3) integrate data portability and related issues with ongoing discussions about not only copyright reform but also reforms to privacy laws and other digital rights; and (4) take seriously the relevant recommendations of the Parliamentary Committee for revision to the Copyright Act
    • 

    corecore