Using Games to Practice Screen Reader Gestures

Tese de mestrado, Engenharia Informática (Engenharia de Software) Universidade de Lisboa, Faculdade de Ciências, 2021Nowadays, a smartphone is fundamental for multiple aspects of our lives. These have evolved from a basic tool to communicate to a multi-purpose tool that allows to communicate with colleagues and friends and to get any information or entertainment. Android and iOS, the most popular mobile operating systems, have developed built-in screen readers that make smartphones generally accessible to blind people using gestures and help them use more of their smartphones. However, users experience difficulties due to unfamiliarity with the gestures and due to lack of interaction with their touchscreens. One possible way to improve the accessibility of these technologies could be through games that can teach how to perform a gesture correctly and explain how it can be used, as lately there has been a growing interest on using video games as an innovative educational tool. We developed Games for Gestures, a set of accessible games to discover and learn the gestures Google Talkback offers and our goal is to explore whether it is possible for mobile accessible games to be used as a gesture discovery and practice method. Corda focuses on teaching how to navigate with Explore by Touch. Foguete focuses on directional swipes left and right and on teaching Swipe To Explore. Guarda Redes is focused on the more advanced gestures. To evaluate our games, we performed a study in which participants played our games for a period of 5 days. After that, we conducted audio-recorded remote interviews with questions about the games and their overall perception of gestures. Our results suggest that accessible games could be important in the process of learning gestures, as they offer a playful method of learning, particularly for less experienced users. This, in turn, would increase their autonomy and inclusion, as this process would become easier and more fun for them

Evaluation of the Accessibility of Touchscreens for Individuals who are Blind or have Low Vision: Where to go from here

Touchscreen devices are well integrated into daily life and can be found in both personal and public spaces, but the inclusion of accessible features and interfaces continues to lag behind technology’s exponential advancement. This thesis aims to explore the experiences of individuals who are blind or have low vision (BLV) while interacting with non-tactile touchscreens, such as smartphones, tablets, smartwatches, coffee machines, smart home devices, kiosks, ATM machines, and more. The goal of this research is to create a set of recommended guidelines that can be used in designing and developing either personal devices or shared public technologies with accessible touchscreens. This study consists of three phases, the first being an exploration of existing research related to accessibility of non-tactile touchscreens, followed by semi-structured interviews of 20 BLV individuals to address accessibility gaps in previous work, and finally a survey in order to get a better understanding of the experiences, thoughts, and barriers for BLV individuals while interacting with touchscreen devices. Some of the common themes found include: loss of independence, lack or uncertainty of accessibility features, and the need and desire for improvements. Common approaches for interaction were: the use of high markings, asking for sighted assistance, and avoiding touchscreen devices. These findings were used to create a set of recommended guidelines which include a universal feature setup, the setup of accessibility settings, universal headphone jack position, tactile feedback, ask for help button, situational lighting, and the consideration of time

Using audio cues to support motion gesture interaction on mobile devices

2015 Spring.Includes bibliographical references.Motion gestures are an underutilized input modality for mobile interaction despite numerous potential advantages. Negulescu et al. found that the lack of feedback on attempted motion gestures made it difficult for participants to diagnose and correct errors, resulting in poor recognition performance and user frustration. In this paper, we describe and evaluate a training and feedback technique, Glissando, which uses audio characteristics to provide feedback on the systems interpretation of user input. This technique enables feedback by verbally confirming correct gestures and notifying users of errors in addition to providing continuous feedback by mapping distinct musical notes to each of three axes and manipulating pitch to specify both spatial and temporal information

Optimizing Human Performance in Mobile Text Entry

Although text entry on mobile phones is abundant, research strives to achieve desktop typing performance "on the go". But how can researchers evaluate new and existing mobile text entry techniques? How can they ensure that evaluations are conducted in a consistent manner that facilitates comparison? What forms of input are possible on a mobile device? Do the audio and haptic feedback options with most touchscreen keyboards affect performance? What influences users' preference for one feedback or another? Can rearranging the characters and keys of a keyboard improve performance? This dissertation answers these questions and more. The developed TEMA software allows researchers to evaluate mobile text entry methods in an easy, detailed, and consistent manner. Many in academia and industry have adopted it. TEMA was used to evaluate a typical QWERTY keyboard with multiple options for audio and haptic feedback. Though feedback did not have a significant effect on performance, a survey revealed that users' choice of feedback is influenced by social and technical factors. Another study using TEMA showed that novice users entered text faster using a tapping technique than with a gesture or handwriting technique. This motivated rearranging the keys and characters to create a new keyboard, MIME, that would provide better performance for expert users. Data on character frequency and key selection times were gathered and used to design MIME. A longitudinal user study using TEMA revealed an entry speed of 17 wpm and a total error rate of 1.7% for MIME, compared to 23 wpm and 5.2% for QWERTY. Although MIME's entry speed did not surpass QWERTY's during the study, it is projected to do so after twelve hours of practice. MIME's error rate was consistently low and significantly lower than QWERTY's. In addition, participants found MIME more comfortable to use, with some reporting hand soreness after using QWERTY for extended periods

Making Spatial Information Accessible on Touchscreens for Users who are Blind and Visually Impaired

Touchscreens have become a de facto standard of input for mobile devices as they most optimally use the limited input and output space that is imposed by their form factor. In recent years, people who are blind and visually impaired have been increasing their usage of smartphones and touchscreens. Although basic access is available, there are still many accessibility issues left to deal with in order to bring full inclusion to this population. One of the important challenges lies in accessing and creating of spatial information on touchscreens. The work presented here provides three new techniques, using three different modalities, for accessing spatial information on touchscreens. The first system makes geometry and diagram creation accessible on a touchscreen through the use of text-to-speech and gestural input. This first study is informed by a qualitative study of how people who are blind and visually impaired currently access and create graphs and diagrams. The second system makes directions through maps accessible using multiple vibration sensors without any sound or visual output. The third system investigates the use of binaural sound on a touchscreen to make various types of applications accessible such as physics simulations, astronomy, and video games

Kindertivity: Usability and Communicability Strategies for Interactive Surfaces and Pre-Kindergarten Children

Tesis por compendio[ES] La tecnología multi-táctil se ha convertido en una de las más emergentes tras experimentar un enorme crecimiento desde sus pasos iniciales en los años ochenta hasta su amplia aceptación y uso en la actualidad. Por una parte, la tecnología multi-táctil se basa en el estilo de interacción de manipulación directa el cual proporciona a los usuarios la ventaja de ver los objetos y las acciones de interés, sustituir comandos escritos por acciones de señalado y, además, permite la realización de acciones rápidas, reversibles e incrementales evitando el uso de instrucciones complejas. Por otra parte, diversos trabajos han evaluado las virtudes derivadas de utilizar conjuntamente la manipulación directa con el toque directo mostrando que es posible evitar los problemas inherentes a otras técnicas de interacción como el ratón y el teclado. Por lo tanto, aprovechando la interacción natural e intuitiva proporcionada por la tecnología multi-táctil, ésta parece una forma ideal para dar soporte a la creación de escenarios educativos dirigidos a niños en edad preescolar. Sin embargo, a pesar de la existencia de diversos estudios que evalúan la idoneidad de utilizar el estilo de interacción de manipulación directa, existe una falta de trabajos abordando el uso dispositivos basados en superficies táctiles con niños de una temprana edad. Asimismo, en la actualidad existe una creciente tendencia a diseñar aplicaciones educativas y lúdicas dirigidas a niños en edad preescolar utilizando dispositivos multi-táctiles como los teléfonos inteligentes o las tabletas. Además, diversos informes señalan que los niños son usuarios frecuentes de este tipo de dispositivos y los utilizan incluso antes de ser capaces de hablar. Sin embargo, a pesar de este crecimiento en el uso de la tecnología multi-táctil y su aparente idoneidad para ser utilizado en el desarrollo de aplicaciones educativas para niños en edad preescolar, no existen unas interacciones universales y estandarizadas para preescolares a la hora de utilizar dispositivos táctiles ya que habitualmente sólo se utilizan dos gestos básicos (básicamente, el toque con un dedo para seleccionar y el arrastre con un dedo para el movimiento). Por lo tanto, existe una clara necesidad de llevar a cabo estudios empíricos para contribuir y avanzar en el diseño de aplicaciones que den un soporte adecuado y encaje con las habilidades de los niños en su temprano desarrollo. Por tanto, esta tesis propone, diseña y evalúa diversas estrategias de usabilidad y comunicabilidad adaptadas a los niños en edad preescolar para establecer la base para el diseño y desarrollo de futuras aplicaciones basadas en dispositivos táctiles dirigidas a preescolares. Estas estrategias llevarán a la adecuada definición de guías de diseño que permitirán a los niños aprovechar al máximo la tecnología multi-táctil, harán posible el desarrollo de nuevas y atractivas aplicaciones y, eventualmente, también podrán ayudar al desarrollo cognitivo y motor de los niños.[CA] La tecnologia multi-tàctil s'ha convertit en una de les més emergents després d'experimentar un enorme creixement des dels seus passos inicials als anys vuitanta fins l'actualitat on es àmpliament acceptada i utilitzada. D'una banda, la tecnologia multi-tàctil es basa en l'estil d'interacció de manipulació directa, el qual proporciona als usuaris l'avantatge de veure els objectes i les accions d'interès, substituir comandos escrits per accions d'assenyalament i, a més, permet la realització d'accions, ràpides, reversibles i incrementals evitant l'ús d'instruccions complexes. D'altra banda, diversos treballs han avaluat les virtuts derivades d'utilitzar conjuntament la manipulació directa amb el toc directe mostrant que és possible evitar els problemes inherents a altres tècniques d'interacció com el ratolí i el teclat. Per tant, aprofitant la interacció natural i intuïtiva proporcionada per la tecnologia multi-tàctil, aquesta sembla una forma ideal per donar suport a la creació d'escenaris educatius per a xiquets en edat preescolar. No obstant això, malgrat l'existència de diversos estudis que avaluen la idoneïtat d'utilitzar l'estil d'interacció de manipulació directa, existeix una manca de treballs abordant l'ús de dispositius basats en superfícies tàctils amb xiquets d'edat primerenca. Així mateix, en l'actualitat existeix una creixent tendència a dissenyar aplicacions educatives i lúdiques dirigides a xiquets en edat preescolar utilitzant dispositius tàctils com els telèfons intel¿ligents o les tauletes. A més, diversos informes assenyalen que els xiquets són usuaris freqüents d'aquests tipus de dispositius i els utilitzen fins i tot abans de ser capaços de parlar. Malgrat aquest creixement en l'ús de la tecnologia multi-tàctil i la seua aparent idoneïtat per a ser utilitzada en el desenvolupament d'aplicacions educatives per a xiquets en edat preescolar, no existeixen unes interaccions universals i estandarditzades per a preescolars a l'hora d'utilitzar dispositius tàctils ja que habitualment només s'utilitzen dos gestos bàsics (bàsicament, el toc amb un dit per a seleccionar i l'arrossegament amb un dit per al moviment). Per tant, hi ha una clara necessitat de dur a terme estudis empírics per a contribuir i avançar en el disseny d'aplicacions que donen un suport adequat i s'ajusten amb les habilitats dels xiquets en el seu primerenc desenvolupament. Per tant, la tesi proposa, dissenya i avalua diverses estratègies de usabilitat i comunicabilitat adaptades als xiquets en edat preescolar per tal d'establir la base per al disseny i desenvolupament de futures aplicacions basades en dispositius tàctils dirigides a preescolars. Aquestes estratègies portaran a l'adequada definició de guies de disseny que permetran als xiquets aprofitar al màxim la tecnologia multi-tàctil, faran possible el desenvolupament de noves i atractives aplicacions i, eventualment, podran també ajudar al desenvolupament cognitiu i motor dels xiquets.[EN] Multi-touch technology has become one of the most emergent technologies and has had an enormous growth since its initial steps in the eighties to be widespread accepted and used in the present. On the one hand, multi-touch technology relies on the direct manipulation interaction style which gives users the advantage to view the objects and actions of interest, replace typed commands by pointing actions and to perform rapid, reversible and incremental actions avoiding using complex instructions. On the other hand, several works have evaluated the virtues when joining direct manipulation with direct-touching showing that it solves the problems inherent in other interaction devices, such as those involving mouse or keyboard. Hence, taking advantage of the intuitive and natural interaction provided by multi-touch technology it seems an ideal way to support educational scenarios targeted to kindergarten children. Although several works have assessed the suitability of using the direct manipulation style with children, there is a lack of works addressing the use of touchscreen devices by this specific type of users. Moreover, there is a growing trend of designing educational and playful applications targeted to kindergarten children based on touchscreen devices such as smartphones and tablets. In addition, several reports point out that children use touchscreen devices even before they are able to speak and they are frequent users of devices such as smartphones and tablets. However, despite this growth in the use of multi-touch technology by children and its apparent suitability to be used to develop applications targeted to young children, there is a lack of standardized and universally accepted interactions for young children when using touchscreen devices since only two basic gestures are commonly used (basically, consisting of only one-finger touch for selection and one-finger drag for movement). Hence, there is a need of carrying out empirical studies to help and advance in the design of applications that adequately support and fit with children's development and skills. Therefore, this thesis proposes, designs and evaluates several usability and communicability strategies tailored to children in their early development stage to establish the design and development of future applications targeted to kindergarten children. These strategies will lead to define appropriate design strategies that enable infants to take full advantage of multi-touch technology, would make it possible to develop attractive new applications and, eventually, could also aid children's cognitive and motor development.Nácher Soler, VE. (2019). Kindertivity: Usability and Communicability Strategies for Interactive Surfaces and Pre-Kindergarten Children [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/116833TESISCompendi

Pictures in Your Mind: Using Interactive Gesture-Controlled Reliefs to Explore Art

Tactile reliefs offer many benefits over the more classic raised line drawings or tactile diagrams, as depth, 3D shape, and surface textures are directly perceivable. Although often created for blind and visually impaired (BVI) people, a wider range of people may benefit from such multimodal material. However, some reliefs are still difficult to understand without proper guidance or accompanying verbal descriptions, hindering autonomous exploration. In this work, we present a gesture-controlled interactive audio guide (IAG) based on recent low-cost depth cameras that can be operated directly with the hands on relief surfaces during tactile exploration. The interactively explorable, location-dependent verbal and captioned descriptions promise rapid tactile accessibility to 2.5D spatial information in a home or education setting, to online resources, or as a kiosk installation at public places. We present a working prototype, discuss design decisions, and present the results of two evaluation studies: the first with 13 BVI test users and the second follow-up study with 14 test users across a wide range of people with differences and difficulties associated with perception, memory, cognition, and communication. The participant-led research method of this latter study prompted new, significant and innovative developments

The Limited Effect of Graphic Elements in Video and Augmented Reality on Children’s Listening Comprehension

There is currently significant interest in the use of instructional strategies in learning environments thanks to the emergence of new multimedia systems that combine text, audio, graphics and video, such as augmented reality (AR). In this light, this study compares the effectiveness of AR and video for listening comprehension tasks. The sample consisted of thirty-two elementary school students with different reading comprehension. Firstly, the experience, instructions and objectives were introduced to all the students. Next, they were divided into two groups to perform activities—one group performed an activity involving watching an Educational Video Story of the Laika dog and her Space Journey available by mobile devices app Blue Planet Tales, while the other performed an activity involving the use of AR, whose contents of the same history were visualized by means of the app Augment Sales. Once the activities were completed participants answered a comprehension test. Results (p = 0.180) indicate there are no meaningful differences between the lesson format and test performance. But there are differences between the participants of the AR group according to their reading comprehension level. With respect to the time taken to perform the comprehension test, there is no significant difference between the two groups but there is a difference between participants with a high and low level of comprehension. To conclude SUS (System Usability Scale) questionnaire was used to establish the measure usability for the AR app on a smartphone. An average score of 77.5 out of 100 was obtained in this questionnaire, which indicates that the app has fairly good user-centered design

Human-powered smartphone assistance for blind people

Mobile devices are fundamental tools for inclusion and independence. Yet, there are still many open research issues in smartphone accessibility for blind people (Grussenmeyer and Folmer 2017). Currently, learning how to use a smartphone is non-trivial, especially when we consider that the need to learn new apps and accommodate to updates never ceases. When first transitioning from a basic feature-phone, people have to adapt to new paradigms of interaction. Where feature phones had a finite set of applications and functions, users can extend the possible functions and uses of a smartphone by installing new 3rd party applications. Moreover, the interconnectivity of these applications means that users can explore a seemingly endless set of workflows across applications. To that end, the fragmented nature of development on these devices results in users needing to create different mental models for each application. These characteristics make smartphone adoption a demanding task, as we found from our eight-week longitudinal study on smartphone adoption by blind people. We conducted multiple studies to characterize the smartphone challenges that blind people face, and found people often require synchronous, co-located assistance from family, peers, friends, and even strangers to overcome the different barriers they face. However, help is not always available, especially when we consider the disparity in each barrier, individual support network and current location. In this dissertation we investigated if and how in-context human-powered solutions can be leveraged to improve current smartphone accessibility and ease of use. Building on a comprehensive knowledge of the smartphone challenges faced and coping mechanisms employed by blind people, we explored how human-powered assistive technologies can facilitate use. The thesis of this dissertation is: Human-powered smartphone assistance by non-experts is effective and impacts perceptions of self-efficacy