Latency guidelines for touchscreen virtual button feedback

Touchscreens are very widely used, especially in mobile phones. They feature many interaction methods, pressing a virtual button being one of the most popular ones. In addition to an inherent visual feedback, virtual button can provide audio and tactile feedback. Since mobile phones are essentially computers, the processing causes latencies in interaction. However, it has not been known, if the latency is an issue in mobile touchscreen virtual button interaction, and what the latency recommendations for visual, audio and tactile feedback are. The research in this thesis has investigated multimodal latency in mobile touchscreen virtual button interaction. For the first time, an affordable, but accurate tool was built to measure all three feedback latencies in touchscreens. For the first time, simultaneity perception of touch and feedback, as well as the effect of latency on virtual button perceived quality has been studied and thresholds found for both unimodal and bimodal feedback. The results from these studies were combined as latency guidelines for the first time. These guidelines enable interaction designers to establish requirements for mobile phone engineers to optimise the latencies on the right level. The latency measurement tool consisted of a high-speed camera, a microphone and an accelerometer for visual, audio and tactile feedback measurements. It was built with off-the-shelf components and, in addition, it was portable. Therefore, it could be copied at low cost or moved wherever needed. The tool enables touchscreen interaction designers to validate latencies in their experiments, making their results more valuable and accurate. The tool could benefit the touchscreen phone manufacturers, since it enables engineers to validate latencies during development of mobile phones. The tool has been used in mobile phone R&D within Nokia Corporation and for validation of a research device within the University of Glasgow. The guidelines established for unimodal feedback was as follows: visual feedback latency should be between 30 and 85 ms, audio between 20 and 70 ms and tactile between 5 and 50 ms. The guidelines were found to be different for bimodal feedback: visual feedback latency should be 95 and audio 70 ms when the feedback was visual-audio, visual 100 and tactile 55 ms when the feedback was visual-tactile and tactile 25 and audio 100 ms when the feedback was tactile-audio. These guidelines will help engineers and interaction designers to select and optimise latencies to be low enough, but not too low. Designers using these guidelines will make sure that most of the users will both perceive the feedback as simultaneous with their touch and experience high quality virtual buttons. The results from this thesis show that latency has a remarkable effect on touchscreen virtual buttons, and it is a key part of virtual button feedback design. The novel results enable researchers, designers and engineers to master the effect of latencies in research and development. This will lead to more accurate and reliable research results and help mobile phone manufacturers make better products

Effects of Local Latency on Games

Video games are a major type of entertainment for millions of people, and feature a wide variety genres. Many genres of video games require quick reactions, and in these games it is critical for player performance and player experience that the game is responsive. One of the major contributing factors that can make games less responsive is local latency — the total delay between input and a resulting change to the screen. Local latency is produced by a combination of delays from input devices, software processing, and displays. Due to latency, game companies spend considerable time and money play-testing their games to ensure the game is both responsive and that the in-game difficulty is reasonable. Past studies have made it clear that local latency negatively affects both player performance and experience, but there is still little knowledge about local latency’s exact effects on games. In this thesis, we address this problem by providing game designers with more knowledge about local latency’s effects. First, we performed a study to examine latency’s effects on performance and experience for popular pointing input devices used with games. Our results show significant differences between devices based on the task and the amount of latency. We then provide design guidelines based on our findings. Second, we performed a study to understand latency’s effects on ‘atoms’ of interaction in games. The study varied both latency and game speed, and found game speed to affect a task’s sensitivity to latency. Third, we used our findings to build a model to help designers quickly identify latency-sensitive game atoms, thus saving time during play-testing. We built and validated a model that predicts errors rates in a game atom based on latency and game speed. Our work helps game designers by providing new insight into latency’s varied effects and by modelling and predicting those effect

Eignung von virtueller Physik und Touch-Gesten in Touchscreen-Benutzerschnittstellen für kritische Aufgaben

The goal of this reasearch was to examine if modern touch screen interaction concepts that are established on consumer electronic devices like smartphones can be used in time-critical and safety-critical use cases like for machine control or healthcare appliances. Several prevalent interaction concepts with and without touch gestures and virtual physics were tested experimentally in common use cases to assess their efficiency, error rate and user satisfaction during task completion. Based on the results, design recommendations for list scrolling and horizontal dialog navigation are given.Das Ziel dieser Forschungsarbeit war es zu untersuchen, ob moderne Touchscreen-Interaktionskonzepte, die auf Consumer-Electronic-Geräten wie Smartphones etabliert sind, für zeit- und sicherheitskritische Anwendungsfälle wie Maschinensteuerung und Medizingeräte geeignet sind. Mehrere gebräuchliche Interaktionskonzepte mit und ohne Touch-Gesten und virtueller Physik wurden in häufigen Anwendungsfällen experimentell auf ihre Effizienz, Fehlerrate und Nutzerzufriedenheit bei der Aufgabenlösung untersucht. Basierend auf den Resultaten werden Empfehlungen für das Scrollen in Listen und dem horizontalen Navigieren in mehrseitigen Software-Dialogen ausgesprochen

Feel the Noise: Mid-Air Ultrasound Haptics as a Novel Human-Vehicle Interaction Paradigm

Focussed ultrasound can be used to create the sensation of touch in mid-air. Combined with gestures, this can provide haptic feedback to guide users, thereby overcoming the lack of agency associated with pure gestural interfaces, and reducing the need for vision – it is therefore particularly apropos of the driving domain. In a counter-balanced 2×2 driving simulator study, a traditional in-vehicle touchscreen was compared with a virtual mid-air gestural interface, both with and without ultrasound haptics. Forty-eight experienced drivers (28 male, 20 female) undertook representative in-vehicle tasks – discrete target selections and continuous slider-bar manipulations – whilst driving. Results show that haptifying gestures with ultrasound was particularly effective in reducing visual demand (number of long glances and mean off-road glance time), and increasing performance (shortest interaction times, highest number of correct responses and least ‘overshoots’) associated with continuous tasks. In contrast, for discrete, target-selections, the touchscreen enabled the highest accuracy and quickest responses, particularly when combined with haptic feedback to guide interactions, although this also increased visual demand. Subjectively, the gesture interfaces invited higher ratings of arousal compared to the more familiar touch-surface technology, and participants indicated the lowest levels of workload (highest performance, lowest frustration) associated with the gesture-haptics interface. In addition, gestures were preferred by participants for continuous tasks. The study shows practical utility and clear potential for the use of haptified gestures in the automotive domain

Vasteajan mittausjärjestelmän suunnittelu, toteutus ja testaus

A touchscreen is a commonly used medium for the interaction between a user and a device. Response to user's action is often indicated visually on the screen after a certain delay. This interface latency is inherent in any computer system. Studies indicate that the latency has a major contribution on how users perceive the interaction with the device. While modern commercial touchscreen devices manifest latencies ranging between 50 ms and 200 ms, research indicates that the user performance for tapping tasks deteriorates at considerably lower levels and users are able to discern the latency as low as 3 ms. In this Thesis we present a novel solution for Android operated mobile devices to expose factors behind the feedback latency of a tap event. We start by reviewing the main components of the Android operating system. Next we describe the internal system elements which partake in the interaction between the user's touch input event and its corresponding visual presentation on the screen of the device. Propelled by the obtained information, we implement an affordable, fully automated system that is capable of collecting both temporal and environmental data. The constructed measurement system provided revealing results. We discovered that most of the feedback latency on a mobile device is accumulated by the internal components which are involved in presenting the visual feedback to the user. We also identified two main user action patterns which impose a huge effect upon system's responsiveness. Firstly, the location of touch is reflected in the amount of feedback latency. Secondly, the interval between two consecutive touch events might cause even unexpected results. Our study demonstrated that the latency can vary a lot between different devices by ranging from no effect on one device to a five-fold difference on another device. The study concludes that, despite the feedback latency is affected by multiple factors, the latency can be measured very precisely with the system that can be built even by an average Joe.Kosketusnäyttö on yleisesti käytetty kanava käyttäjän ja laitteen välisessä vuorovaikutuksessa. Järjestelmän palaute käyttäjän antamaan syötteeseen esitetään usein visuaalisesti laitteen näytöllä. Vasteen tuottamisessa syntyy kuitenkin jonkin verran viivettä eli latenssia. Tutkimusten mukaan viiveellä on suuri vaikutus käyttäjäkokemukseen. Nykyisten kosketuslaitteiden latenssi vaihtelee yleensä 50 ja 200 millisekunnin välillä. Kosketuspohjaisten tapahtumien suorittamisen on todettu heikentyvät jo huomattavasti pienemmän viiveen johdosta ja jopa alle kolme millisekuntia kestävä viive on vielä havaittavissa. Tässä diplomityössä esitetään Android-pohjaisille mobiililaitteille luotu edullinen järjestelmä, jonka avulla pystytään mittaamaan käyttäjän näytölle luoman kosketuksen ja sitä vastaavan järjestelmän antaman visuaalisen palautteen välistä viivettä. Työssä esitetellään ensin Android-käyttöjärjestelmän komponentit, jotka osallistuvat tämän tapahtumaketjun suorittamiseksi vaadittaviin toimintoihin. Tietojen pohjalta luodaan järjestelmä, jolla voidaan kerätä automaattisesti dataa viiveen eri syntykohdista ja sen ympäristöön littyvistä seikoista. Datan avulla pystytään aiempaa paremmin arvioimaan viiveen syntyyn vaikuttavia tekijöitä. Saatua tietoa voidaan hyödyntää yleisesti viiveen hallitsemiseen tähtääviin toimenpiteisiin ja siten lopulta käyttäjäkokemuksen parantamiseen. Järjestelmällä mitatuista tuloksista selviää, että suurin osa tapahtumaketjun latenssista syntyy käyttäjälle esitettävän visuaalisen palautteen vaatimiin toimenpiteisiin. Lisäksi työ tuo esille kaksi käyttäjän syötteen antamiseen liittyvää toimintatapaa, joilla on suuri vaikutus latenssiin. Kosketuksen sijainti ruudulla ja kahden peräkkäisen kosketuksen välinen aika vaikuttavat vasteaikaan. Latenssi ei aina muodostu suoraviivaisesti ja se voi ilmentää jopa yllättäviä piirteitä eri laitteiden välillä: toimintatapa yhdessä laitteessa ei vaikuta tulokseen, mutta saattaa toisessa laitteessa näkyä moninkertaisena erona. Vaikka latenssin syntyyn vaikuttaa monta eri tekijää, sitä voidaan onneksi mitata erittäin tarkasti järjestelmällä, jonka jopa Matti Meikäläinen pystyy rakentamaan

A Utility Framework for Selecting Immersive Interactive Capability and Technology for Virtual Laboratories

There has been an increase in the use of virtual reality (VR) technology in the education community since VR is emerging as a potent educational tool that offers students with a rich source of educational material and makes learning exciting and interactive. With a rise of popularity and market expansion in VR technology in the past few years, a variety of consumer VR electronics have boosted educators and researchers’ interest in using these devices for practicing engineering and science laboratory experiments. However, little is known about how such devices may be well-suited for active learning in a laboratory environment. This research aims to address this gap by formulating a utility framework to help educators and decision-makers efficiently select a type of VR device that matches with their design and capability requirements for their virtual laboratory blueprint. Furthermore, a framework use case is demonstrated by not only surveying five types of VR devices ranging from low-immersive to full-immersive along with their capabilities (i.e., hardware specifications, cost, and availability) but also considering the interaction techniques in each VR device based on the desired laboratory task. To validate the framework, a research study is carried out to compare these five VR devices and investigate which device can provide an overall best-fit for a 3D virtual laboratory content that we implemented based on the interaction level, usability and performance effectiveness