Comparing Evaluation Methods for Encumbrance and Walking on Interaction with Touchscreen Mobile Devices

In this paper, two walking evaluation methods were compared to evaluate the effects of encumbrance while the preferred walking speed (PWS) is controlled. Users frequently carry cumbersome objects (e.g. shopping bags) and use mobile devices at the same time which can cause interaction difficulties and erroneous input. The two methods used to control the PWS were: walking on a treadmill and walking around a predefined route on the ground while following a pacesetter. The results from our target acquisition experiment showed that for ground walking at 100% of PWS, accuracy dropped to 36% when carrying a bag in the dominant hand while accuracy reduced to 34% for holding a box under the dominant arm. We also discuss the advantages and limitations of each evaluation method when examining encumbrance and suggest treadmill walking is not the most suitable approach to use if walking speed is an important factor in future mobile studies

The Performance and Preference of Different Fingers and Chords for Pointing, Dragging, and Object Transformation

International audienceThe development of robust methods to identify which finger is causing each touch point, called “finger identification,” will open up a new input space where interaction designers can associate system actions to different fingers. However, relatively little is known about the performance of specific fingers as single touch points or when used together in a “chord.” We present empirical results for accuracy, throughput, and subjective preference gathered in five experiments with 48 participants exploring all 10 fingers and 7 two-finger chords. Based on these results, we develop design guidelines for reasonable target sizes for specific fingers and two-finger chords, and a relative ranking of the suitability of fingers and two-finger chords for common multi-touch tasks. Our work contributes new knowledge regarding specific finger and chord performance and can inform the design of future interaction techniques and interfaces utilizing finger identification

EdgeGlass: Exploring Tapping Performance on Smart Glasses while Sitting and Walking

Department of Human Factors EngineeringCurrently, smart glasses allow only touch sensing area which supports front mounted touch pads. However, touches on top, front and bottom sides of glass mounted touchpad is not yet explored. We made a customized touch sensor (length: 5-6 cm, height: 1 cm, width: 0.5 cm) featuring the sensing on its top, front, and bottom surfaces. For doing that, we have used capacitive touch sensing technology (MPR121 chips) with an electrode size of ~4.5 mm square, which is typical in the modern touchscreens. We have created a hardware system which consists of a total of 48 separate touch sensors. We investigated the interaction technique by it for both the sitting and walking situation, using a single finger sequential tapping and a pair finger simultaneous tapping. We have divided each side into three equal target areas and this separation made a total of 36 combinations. Our quantitative result showed that pair finger simultaneous tapping touches were faster, less error-prone in walking condition, compared to single finger sequential tapping into walking condition. Whereas, single finger sequence tapping touches were slower, but less error-prone in sitting condition, compared to pair simultaneous tapping in sitting condition. However, single finger sequential tapping touches were slower, much less error-prone in sitting condition compared to walking. Interestingly, double finger tapping touches had similar performance result in terms of both, error rate and completion time, in both sitting and walking conditions. Mental, physical, performance, effort did not have any effect on any temporal tapping???s and body poses experience of workload. In case of the parameter of temporal demand, for single finger sequential tapping mean temporal (time pressure) workload demand was higher than pair finger simultaneous tapping but body poses did not affect temporal (time pressure) workload for both of the sequential and simultaneous tapping type. In case of the parameter of frustration, the result suggested that mean frustration workload was higher for single finger sequential tapping experienced by the participants compared to pair finger simultaneous tapping and among body poses, walking experienced higher frustration mean workload than sitting. The subjective measure of overall workload during the performance study showed no significant difference between both independent variable: body pose (sitting and walking) and temporal tapping (single finger sequential tapping and pair finger simultaneous tapping).ope

Efficient human-machine control with asymmetric marginal reliability input devices

Input devices such as motor-imagery brain-computer interfaces (BCIs) are often unreliable. In theory, channel coding can be used in the human-machine loop to robustly encapsulate intention through noisy input devices but standard feedforward error correction codes cannot be practically applied. We present a practical and general probabilistic user interface for binary input devices with very high noise levels. Our approach allows any level of robustness to be achieved, regardless of noise level, where reliable feedback such as a visual display is available. In particular, we show efficient zooming interfaces based on feedback channel codes for two-class binary problems with noise levels characteristic of modalities such as motor-imagery based BCI, with accuracy <75%. We outline general principles based on separating channel, line and source coding in human-machine loop design. We develop a novel selection mechanism which can achieve arbitrarily reliable selection with a noisy two-state button. We show automatic online adaptation to changing channel statistics, and operation without precise calibration of error rates. A range of visualisations are used to construct user interfaces which implicitly code for these channels in a way that it is transparent to users. We validate our approach with a set of Monte Carlo simulations, and empirical results from a human-in-the-loop experiment showing the approach operates effectively at 50-70% of the theoretical optimum across a range of channel conditions

Neuromotor Control of the Hand During Smartphone Manipulation

The primary focus of this dissertation was to understand the motor control strategy used by our neuromuscular system for the multi-layered motor tasks involved during smartphone manipulation. To understand this control strategy, we recorded the kinematics and multi-muscle activation pattern of the right limb during smartphone manipulation, including grasping with/out tapping, movement conditions (MCOND), and arm heights. In the first study (chapter 2), we examined the neuromuscular control strategy of the upper limb during grasping with/out tapping executed with a smartphone by evaluating muscle-activation patterns of the upper limb during different movement conditions (MCOND). There was a change in muscle activity for MCOND and segments. We concluded that our neuromuscular system generates the motor strategy that would allow smartphone manipulation involving grasping and tapping while maintaining MCOND by generating continuous and distinct multi-muscle activation patterns in the upper limb muscles. In the second study (chapter 3), we examined the muscle activity of the upper limb when the smartphone was manipulated at two arm heights: shoulder and abdomen to understand the influence of the arm height on the neuromuscular control strategy of the upper limb. Some muscles showed a significant effect for ABD, while some muscle showed a significant effect for SHD. We concluded that the motor control strategy was influenced by the arm height as there were changes in the shoulder and elbow joint angles along with the muscular activity of the upper limb. Further, shoulder position helped in holding the head upright while abdomen reduced the moment arm and moment and ultimately, muscle loading compared to the shoulder. Overall, our neuromuscular system generates motor command by activating a multi-muscle activation pattern in the upper limb, which would be dependent upon the task demands such as grasping with/out tapping, MCOND, and arm heights. Similarly, our neuromuscular system does not appear to increase muscle activation when there is a combined effect of MCOND and arm heights. Instead, it utilizes a simple control strategy that would select an appropriate muscle and activate them based on the levels of MCOND and arm heights

Extending mobile touchscreen interaction

Touchscreens have become a de facto interface for mobile devices, and are penetrating further beyond their core application domain of smartphones. This work presents a design space for extending touchscreen interaction, to which new solutions may be mapped. Specific touchscreen enhancements in the domains of manual input, visual output and haptic feedback are explored and quantitative and experiental findings reported. Particular areas covered are unintentional interaction, screen locking, stereoscopic displays and picoprojection. In addition, the novel interaction approaches of finger identification and onscreen physical guides are also explored. The use of touchscreens in the domains of car dashboards and smart handbags are evaluated as domain specific use cases. This work draws together solutions from the broad area of mobile touchscreen interaction. Fruitful directions for future research are identified, and information is provided for future researchers addressing those topics.Kosketusnäytöistä on muodostunut mobiililaitteiden pääasiallinen käyttöliittymä, ja ne ovat levinneet alkuperäiseltä ydinsovellusalueeltaan, matkapuhelimista, myös muihin laitteisiin. Työssä tutkitaan uusia vuorovaikutuksen, visualisoinnin ja käyttöliittymäpalautteen keinoja, jotka laajentavat perinteistä kosketusnäytön avulla tapahtuvaa vuorovaikutusta. Näihin liittyen väitöskirjassa esitetään sekä kvantitatiivisia tuloksia että uutta kartoittavia löydöksiä. Erityisesti työ tarkastelee tahatonta kosketusnäytön käyttöä, kosketusnäytön lukitusta, stereoskooppisia kosketusnäyttöjä ja pikoprojektoreiden hyödyntämistä. Lisäksi kartoitetaan uusia vuorovaikutustapoja, jotka liittyvät sormien identifioimiseen vuorovaikutuksen yhteydessä, ja fyysisiin, liikettä ohjaaviin rakenteisiin kosketusnäytöllä. Kosketusnäytön käyttöä autossa sekä osana älykästä käsilaukkua tarkastellaan esimerkkeinä käyttökonteksteista. Väitöskirjassa esitetään vuorovaikutussuunnittelun viitekehys, joka laajentaa kosketusnäyttöjen kautta tapahtuvaa vuorovaikutusta mobiililaitteen kanssa, ja johon työssä esitellyt, uudet vuorovaikutustavat voidaan sijoittaa. Väitöskirja yhdistää kosketusnäyttöihin liittyviä käyttöliittymäsuunnittelun ratkaisuja laajalta alueelta. Työ esittelee potentiaalisia suuntaviivoja tulevaisuuden tutkimuksille ja tuo uutta tutkimustietoa, jota mobiililaitteiden vuorovaikutuksen tutkijat ja käyttöliittymäsuunnittelijat voivat hyödyntää

Extending mobile touchscreen interaction

Touchscreens have become a de facto interface for mobile devices, and are penetrating further beyond their core application domain of smartphones. This work presents a design space for extending touchscreen interaction, to which new solutions may be mapped. Specific touchscreen enhancements in the domains of manual input, visual output and haptic feedback are explored and quantitative and experiental findings reported. Particular areas covered are unintentional interaction, screen locking, stereoscopic displays and picoprojection. In addition, the novel interaction approaches of finger identification and onscreen physical guides are also explored. The use of touchscreens in the domains of car dashboards and smart handbags are evaluated as domain specific use cases. This work draws together solutions from the broad area of mobile touchscreen interaction. Fruitful directions for future research are identified, and information is provided for future researchers addressing those topics.Kosketusnäytöistä on muodostunut mobiililaitteiden pääasiallinen käyttöliittymä, ja ne ovat levinneet alkuperäiseltä ydinsovellusalueeltaan, matkapuhelimista, myös muihin laitteisiin. Työssä tutkitaan uusia vuorovaikutuksen, visualisoinnin ja käyttöliittymäpalautteen keinoja, jotka laajentavat perinteistä kosketusnäytön avulla tapahtuvaa vuorovaikutusta. Näihin liittyen väitöskirjassa esitetään sekä kvantitatiivisia tuloksia että uutta kartoittavia löydöksiä. Erityisesti työ tarkastelee tahatonta kosketusnäytön käyttöä, kosketusnäytön lukitusta, stereoskooppisia kosketusnäyttöjä ja pikoprojektoreiden hyödyntämistä. Lisäksi kartoitetaan uusia vuorovaikutustapoja, jotka liittyvät sormien identifioimiseen vuorovaikutuksen yhteydessä, ja fyysisiin, liikettä ohjaaviin rakenteisiin kosketusnäytöllä. Kosketusnäytön käyttöä autossa sekä osana älykästä käsilaukkua tarkastellaan esimerkkeinä käyttökonteksteista. Väitöskirjassa esitetään vuorovaikutussuunnittelun viitekehys, joka laajentaa kosketusnäyttöjen kautta tapahtuvaa vuorovaikutusta mobiililaitteen kanssa, ja johon työssä esitellyt, uudet vuorovaikutustavat voidaan sijoittaa. Väitöskirja yhdistää kosketusnäyttöihin liittyviä käyttöliittymäsuunnittelun ratkaisuja laajalta alueelta. Työ esittelee potentiaalisia suuntaviivoja tulevaisuuden tutkimuksille ja tuo uutta tutkimustietoa, jota mobiililaitteiden vuorovaikutuksen tutkijat ja käyttöliittymäsuunnittelijat voivat hyödyntää

The effects of encumbrance and mobility on interactions with touchscreen mobile devices

Mobile handheld devices such as smartphones are now convenient as they allow users to make calls, reply to emails, find nearby services and many more. The increase in functionality and availability of mobile applications also allow mobile devices to be used in many different everyday situations (for example, while on the move and carrying items). While previous work has investigated the interaction difficulties in walking situations, there is a lack of empirical work in the literature on mobile input when users are physically constrained by other activities. As a result, how users input on touchscreen handheld devices in encumbered and mobile contexts is less well known and deserves more attention to examine the usability issues that are often ignored. This thesis investigates targeting performance on touchscreen mobile phones in one common encumbered situation - when users are carrying everyday objects while on the move. To identify the typical objects held during mobile interactions and define a set of common encumbrance scenarios to evaluate in subsequent user studies, Chapter 3 describes an observational study that examined users in different public locations. The results showed that people carried different types of bags and boxes the most frequently. To measure how much tapping performance on touchscreen mobile phones is affected, Chapter 4 examines a range of encumbrance scenarios, which includes holding a bag in-hand or a box underarm, either on the dominant or non-dominant side, during target selections on a mobile phone. Users are likely to switch to a more effective input posture when encumbered and on the move, so Chapter 5 investigates one- and two- handed encumbered interactions and evaluates situations where both hands are occupied with multiple objects. Touchscreen devices afford various multi-touch input types, so Chapter 6 compares the performance of four main one- and two- finger gesture inputs: tapping, dragging, spreading & pinching and rotating, while walking and encumbered. Several main evaluation approaches have been used in previous walking studies, but more attention is required when the effects of encumbrance is also being examined. Chapter 7 examines the appropriateness of two methods (ground and treadmill walking) for encumbered and walking studies, justifies the need to control walking speed and examines the effects of varying walking speed (i.e. walking slower or faster than normal) on encumbered targeting performance. The studies all showed a reduction in targeting performance when users were walking and encumbered, so Chapter 8 explores two ways to improve target selections. The first approach defines a target size, based on the results collected from earlier studies, to increase tapping accuracy and subsequently, a novel interface arrangement was designed which optimises screen space more effectively. The second approach evaluates a benchmark pointing technique, which has shown to improve the selection of small targets, to see if it is useful in walking and encumbered contexts

Embodied interaction with guitars: instruments, embodied practices and ecologies

In this thesis I investigate the embodied performance preparation practices of guitarists to design and develop tools to support them. To do so, I employ a series of human-centred design methodologies such as design ethnography, participatory design, and soma design. The initial ethnographic study I conducted involved observing guitarists preparing to perform individually and with their bands in their habitual places of practice. I also interviewed these musicians on their preparation activities. Findings of this study allowed me to chart an ecology of tools and resources employed in the process, as well as pinpoint a series of design opportunities for augmenting guitars, namely supporting (1) encumbered interactions, (2) contextual interactions, and (3) connected interactions. Going forward with the design process I focused on remediating encumbered interactions that emerge during performance preparation with multimedia devices, particularly during instrumental transcription. I then prepared and ran a series of hands-on co-design workshops with guitarists to discuss five media controller prototypes, namely, instrument-mounted controls, pedal-based controls, voice-based controls, gesture-based controls, and “music-based” controls. This study highlighted the value that guitarists give to their guitars and to their existing practice spaces, tools, and resources by critically reflecting on how these interaction modalities would support or disturb their existing embodied preparation practices with the instrument. In parallel with this study, I had the opportunity to participate in a soma design workshop (and then prepare my own) in which I harnessed my first-person perspective of guitar playing to guide the design process. By exploring a series of embodied ideation and somatic methods, as well as materials and sensors across several points of contact between our bodies and the guitar, we collaboratively ideated a series of design concepts for guitar across both workshops, such as a series of breathing guitars, stretchy straps, and soft pedals. I then continued to develop and refine the Stretchy Strap concept into a guitar strap augmented with electronic textile stretch sensors to harness it as an embodied media controller to remediate encumbered interaction during musical transcription with guitar when using secondary multimedia resources. The device was subsequently evaluated by guitarists at a home practicing space, providing insights on nuanced aspects of its embodied use, such as how certain media control actions like play and pause are better supported by the bodily gestures enacted with the strap, whilst other actions, like rewinding the play back or setting in and out points for a loop are better supported by existing peripherals like keyboards and mice, as these activities do not necessarily happen in the flow of the embodied practice of musical transcription. Reflecting on the overall design process, a series of considerations are extracted for designing embodied interactions with guitars, namely, (1) considering the instrument and its potential for augmentation, i.e., considering the shape of the guitar, its material and its cultural identity, (2) considering the embodied practices with the instrument, i.e., the body and the subjective felt experience of the guitarist during their skilled embodied practices with the instrument and how these determine its expert use according to a particular instrumental tradition and/or musical practice, and (3) considering the practice ecology of the guitarist, i.e., the tools, resources, and spaces they use according to their practice