Browse-to-search

This demonstration presents a novel interactive online shopping application based on visual search technologies. When users want to buy something on a shopping site, they usually have the requirement of looking for related information from other web sites. Therefore users need to switch between the web page being browsed and other websites that provide search results. The proposed application enables users to naturally search products of interest when they browse a web page, and make their even causal purchase intent easily satisfied. The interactive shopping experience is characterized by: 1) in session - it allows users to specify the purchase intent in the browsing session, instead of leaving the current page and navigating to other websites; 2) in context - -the browsed web page provides implicit context information which helps infer user purchase preferences; 3) in focus - users easily specify their search interest using gesture on touch devices and do not need to formulate queries in search box; 4) natural-gesture inputs and visual-based search provides users a natural shopping experience. The system is evaluated against a data set consisting of several millions commercial product images. © 2012 Authors

A Person-Centric Design Framework for At-Home Motor Learning in Serious Games

abstract: In motor learning, real-time multi-modal feedback is a critical element in guided training. Serious games have been introduced as a platform for at-home motor training due to their highly interactive and multi-modal nature. This dissertation explores the design of a multimodal environment for at-home training in which an autonomous system observes and guides the user in the place of a live trainer, providing real-time assessment, feedback and difficulty adaptation as the subject masters a motor skill. After an in-depth review of the latest solutions in this field, this dissertation proposes a person-centric approach to the design of this environment, in contrast to the standard techniques implemented in related work, to address many of the limitations of these approaches. The unique advantages and restrictions of this approach are presented in the form of a case study in which a system entitled the "Autonomous Training Assistant" consisting of both hardware and software for guided at-home motor learning is designed and adapted for a specific individual and trainer. In this work, the design of an autonomous motor learning environment is approached from three areas: motor assessment, multimodal feedback, and serious game design. For motor assessment, a 3-dimensional assessment framework is proposed which comprises of 2 spatial (posture, progression) and 1 temporal (pacing) domains of real-time motor assessment. For multimodal feedback, a rod-shaped device called the "Intelligent Stick" is combined with an audio-visual interface to provide feedback to the subject in three domains (audio, visual, haptic). Feedback domains are mapped to modalities and feedback is provided whenever the user's performance deviates from the ideal performance level by an adaptive threshold. Approaches for multi-modal integration and feedback fading are discussed. Finally, a novel approach for stealth adaptation in serious game design is presented. This approach allows serious games to incorporate motor tasks in a more natural way, facilitating self-assessment by the subject. An evaluation of three different stealth adaptation approaches are presented and evaluated using the flow-state ratio metric. The dissertation concludes with directions for future work in the integration of stealth adaptation techniques across the field of exergames.Dissertation/ThesisDoctoral Dissertation Computer Science 201

In Touch with the Wild: Exploring Real-time Feedback for Learning to Play the Violin

Real-time feedback has great potential for enhancing learning complex motor-skills by enabling people to correct their mistakes as they go. Multimodal real-time cues could provide reinforcement to inform players whether they are making the correct or incorrect movements at a given time. However, little is known about how best to communicate information in real-time so that people can readily perceive and apply it to improving their movement while learning complex motor-skills. This thesis addresses this gap in knowledge by investigating how real-time feedback can enhance learning to play the violin. It explores how haptic and visual feedback are perceived, understood and acted upon in real-time when engaged in the primary task of playing the violin. Prototypes were built with sensors to measure movement and either vibrations on the body or visual signals as feedback. Three in-the-wild user studies were conducted: one comparing visual and vibrotactile feedback for individual practice; one investigating shared feedback at a musical summer school; and one examining real-time feedback as part of a programme of learning at a high school. In-the-wild studies investigate users interacting with technology in a naturalistic setting, with all the demands that this entails. The findings show real-time feedback is effective at improving violin technique and can support learning in other ways such as encouraging mutual support between learners. The positive learning outcomes, however, need to be understood with respect to the complex interplay between the technology, demands of the setting and characteristics of individual learners. A conceptual framework is provided that outlines these interdependent factors. The findings are discussed regarding their applicability to learning other physical skills and the challenges and insights of using an in-the-wild methodology. The contribution of this thesis is to demonstrate empirically and theoretically how real-time vibrotactile and visual feedback can enhance learning a complex motor-skill