Vision systems with the human in the loop

The emerging cognitive vision paradigm deals with vision systems that apply machine learning and automatic reasoning in order to learn from what they perceive. Cognitive vision systems can rate the relevance and consistency of newly acquired knowledge, they can adapt to their environment and thus will exhibit high robustness. This contribution presents vision systems that aim at flexibility and robustness. One is tailored for content-based image retrieval, the others are cognitive vision systems that constitute prototypes of visual active memories which evaluate, gather, and integrate contextual knowledge for visual analysis. All three systems are designed to interact with human users. After we will have discussed adaptive content-based image retrieval and object and action recognition in an office environment, the issue of assessing cognitive systems will be raised. Experiences from psychologically evaluated human-machine interactions will be reported and the promising potential of psychologically-based usability experiments will be stressed

Pictures in Your Mind: Using Interactive Gesture-Controlled Reliefs to Explore Art

Tactile reliefs offer many benefits over the more classic raised line drawings or tactile diagrams, as depth, 3D shape, and surface textures are directly perceivable. Although often created for blind and visually impaired (BVI) people, a wider range of people may benefit from such multimodal material. However, some reliefs are still difficult to understand without proper guidance or accompanying verbal descriptions, hindering autonomous exploration. In this work, we present a gesture-controlled interactive audio guide (IAG) based on recent low-cost depth cameras that can be operated directly with the hands on relief surfaces during tactile exploration. The interactively explorable, location-dependent verbal and captioned descriptions promise rapid tactile accessibility to 2.5D spatial information in a home or education setting, to online resources, or as a kiosk installation at public places. We present a working prototype, discuss design decisions, and present the results of two evaluation studies: the first with 13 BVI test users and the second follow-up study with 14 test users across a wide range of people with differences and difficulties associated with perception, memory, cognition, and communication. The participant-led research method of this latter study prompted new, significant and innovative developments

Action-based effects on music perception

The classical, disembodied approach to music cognition conceptualizes action and perception as separate, peripheral processes. In contrast, embodied accounts of music cognition emphasize the central role of the close coupling of action and perception. It is a commonly established fact that perception spurs action tendencies. We present a theoretical framework that captures the ways in which the human motor system and its actions can reciprocally influence the perception of music. The cornerstone of this framework is the common coding theory, postulating a representational overlap in the brain between the planning, the execution, and the perception of movement. The integration of action and perception in so-called internal models is explained as a result of associative learning processes. Characteristic of internal models is that they allow intended or perceived sensory states to be transferred into corresponding motor commands (inverse modeling), and vice versa, to predict the sensory outcomes of planned actions (forward modeling). Embodied accounts typically refer to inverse modeling to explain action effects on music perception (Leman, 2007). We extend this account by pinpointing forward modeling as an alternative mechanism by which action can modulate perception. We provide an extensive overview of recent empirical evidence in support of this idea. Additionally, we demonstrate that motor dysfunctions can cause perceptual disabilities, supporting the main idea of the paper that the human motor system plays a functional role in auditory perception. The finding that music perception is shaped by the human motor system and its actions suggests that the musical mind is highly embodied. However, we advocate for a more radical approach to embodied (music) cognition in the sense that it needs to be considered as a dynamical process, in which aspects of action, perception, introspection, and social interaction are of crucial importance

Development of a Virtual Laparoscopic Trainer using Accelerometer Augmented Tools to Assess Performance in Surgical training

Previous research suggests that virtual reality (VR) may supplement conventional training in laparoscopy. It may prove useful in the selection of surgical trainees in terms of their dexterity and spatial awareness skills in the near future. Current VR training solutions provide levels of realism and in some instances, haptic feedback, but they are cumbersome by being tethered and not ergonomically close to the actual surgical instruments for weight and freedom of use factors. In addition, they are expensive hence making them less accessible to departments than conventional box trainers. The box trainers in comparison, although more economical, lack tangible feedback and realism for handling delicate tissue structures. We have previously reported on the development of a modified digitally enhanced surgical instrument for laparoscopic training, named the Parkar Tool. This tool contains wireless accelerometer and gyroscopic sensors integrated into actual laparoscopic instruments. By design, it alleviates the need for both tethered and physically different shaped tools thereby enhancing the realism when performing surgical procedures. Additionally the software (Valhalla) has the ability to digitally record surgical motions, thereby enabling it to remotely capture surgical training data to analyse and objectively evaluate performance. We have adapted and further developed our initial single training tool method as used with a laparoscopic pyloromyotomy scenario, to an enhanced method using multiple Parkar wireless tools simultaneously, for use in several different case scenarios. This allows the use and measurement of right and left handed dexterity with the benefit of using several tasks of differing complexity. The development of a 3D tissue-surface deformations solution written in OpenGL gives us several different virtual surgical training scenario approximations to use with the instruments. The trainee can start with learning simple tasks e.g. incising tissue, grasping, squeezing and stretching tissue, to more complex procedures such as suturing, herniotomies, bowel anastomoses, as well as the original pyloromyotomy as used in the first model

Toward a model of computational attention based on expressive behavior: applications to cultural heritage scenarios

Our project goals consisted in the development of attention-based analysis of human expressive behavior and the implementation of real-time algorithm in EyesWeb XMI in order to improve naturalness of human-computer interaction and context-based monitoring of human behavior. To this aim, perceptual-model that mimic human attentional processes was developed for expressivity analysis and modeled by entropy. Museum scenarios were selected as an ecological test-bed to elaborate three experiments that focus on visitor profiling and visitors flow regulation

A Study of Educational Simulations Part I - Engagement and Learning

Interactive computer simulations with complex representations and sophisticated graphics are a relatively new addition to the classroom, and research in this area is limited. We have conducted over 200 individual student interviews during which the students described what they were thinking as they interacted with simulations. These interviews were conducted as part of the research and design of simulations for the Physics Education Technology (PhET) project. PhET is an ongoing project that has developed over 60 simulations for use in teaching physics, chemistry, and physical science. These interviews are a rich source of information about how students interact with computer simulations and what makes an educationally effective simulation. We have observed that simulations can be highly engaging and educationally effective, but only if the student's interaction with the simulation is directed by the student's own questioning. Here we describe our design process, what features are effective for engaging students in educationally productive interactions and the underlying principles which support our empirically developed guidelines. In a companion paper we describe in detail the design features used to create an intuitive simulation for students to use

Mobile Learning Revolution: Implications for Language Pedagogy

Mobile technologies including cell phones and tablets are a pervasive feature of everyday life with potential impact on teaching and learning. “Mobile pedagogy” may seem like a contradiction in terms, since mobile learning often takes place physically beyond the teacher's reach, outside the walls of the classroom. While pedagogy implies careful planning, mobility exposes learners to the unexpected. A thoughtful pedagogical response to this reality involves new conceptualizations of what is to be learned and new activity designs. This approach recognizes that learners may act in more self-determined ways beyond the classroom walls, where online interactions and mobile encounters influence their target language communication needs and interests. The chapter sets out a range of opportunities for out-of-class mobile language learning that give learners an active role and promote communication. It then considers the implications of these developments for language content and curricula and the evolving roles and competences of teachers