Video interaction using pen-based technology

Dissertação para obtenção do Grau de Doutor em InformáticaVideo can be considered one of the most complete and complex media and its manipulating is still a difficult and tedious task. This research applies pen-based technology to video manipulation, with the goal to improve this interaction. Even though the human familiarity with pen-based devices, how they can be used on video interaction, in order to improve it, making it more natural and at the same time fostering the user’s creativity is an open question. Two types of interaction with video were considered in this work: video annotation and video editing. Each interaction type allows the study of one of the interaction modes of using pen-based technology: indirectly, through digital ink, or directly, trough pen gestures or pressure. This research contributes with two approaches for pen-based video interaction: pen-based video annotations and video as ink. The first uses pen-based annotations combined with motion tracking algorithms, in order to augment video content with sketches or handwritten notes. It aims to study how pen-based technology can be used to annotate a moving objects and how to maintain the association between a pen-based annotations and the annotated moving object The second concept replaces digital ink by video content, studding how pen gestures and pressure can be used on video editing and what kind of changes are needed in the interface, in order to provide a more familiar and creative interaction in this usage context.This work was partially funded by the UTAustin-Portugal, Digital Media, Program (Ph.D. grant: SFRH/BD/42662/2007 - FCT/MCTES); by the HP Technology for Teaching Grant Initiative 2006; by the project "TKB - A Transmedia Knowledge Base for contemporary dance" (PTDC/EAT/AVP/098220/2008 funded by FCT/MCTES); and by CITI/DI/FCT/UNL (PEst-OE/EEI/UI0527/2011

Computer-Assisted Interactive Documentary and Performance Arts in Illimitable Space

This major component of the research described in this thesis is 3D computer graphics, specifically the realistic physics-based softbody simulation and haptic responsive environments. Minor components include advanced human-computer interaction environments, non-linear documentary storytelling, and theatre performance. The journey of this research has been unusual because it requires a researcher with solid knowledge and background in multiple disciplines; who also has to be creative and sensitive in order to combine the possible areas into a new research direction. [...] It focuses on the advanced computer graphics and emerges from experimental cinematic works and theatrical artistic practices. Some development content and installations are completed to prove and evaluate the described concepts and to be convincing. [...] To summarize, the resulting work involves not only artistic creativity, but solving or combining technological hurdles in motion tracking, pattern recognition, force feedback control, etc., with the available documentary footage on film, video, or images, and text via a variety of devices [....] and programming, and installing all the needed interfaces such that it all works in real-time. Thus, the contribution to the knowledge advancement is in solving these interfacing problems and the real-time aspects of the interaction that have uses in film industry, fashion industry, new age interactive theatre, computer games, and web-based technologies and services for entertainment and education. It also includes building up on this experience to integrate Kinect- and haptic-based interaction, artistic scenery rendering, and other forms of control. This research work connects all the research disciplines, seemingly disjoint fields of research, such as computer graphics, documentary film, interactive media, and theatre performance together.Comment: PhD thesis copy; 272 pages, 83 figures, 6 algorithm

The Impact Of Platform On Global Video Game Sales

This paper examines video game sales by platform in the global market from a period spanning 2006 through 2011.  As the home video game industry has rapidly matured and become established as a forefront facet of interactive entertainment in the home, we seek to determine what aspects of the video game market have the greatest impact on sales.  This question is particularly poignant, as the maturation of the video game industry has witnessed efforts at both vertical integration and horizontal expansion on the part of the top game publishers and developers in hopes of solidly grounding the industry.  This study employs a Kruskal-Wallis test to compare eight different gaming platforms.  The results indicate Nintendo’s Wii was the top selling global platform; Nintendo DS was the second tier; Xbox 360, Sony PlayStation 3, and the personal computer (PC) are in the third tier; the fourth tier consists of Sony PlayStation 2 and Sony PSP; and the retired sixth generation Nintendo GameCube is the lowest sales tier

Interfaces for television content sharing and annotation

Dissertação para obtenção do Grau de Mestre em Engenharia InformáticaThe ways of television consumption and production are changing significantly, with the viewers moving away from the traditional linear model. The various devices for accessing content have a significant role in these changes and suggest new paradigms of access. Social experience has also changed and takes on new forms molded by technology. Content sharing and production from users are some of the trends that globally influence how we relate to audiovisual content. Therefore the aim is to develop ways to access television content, that allow commenting and sharing, through multimodal annotations. These annotations include text, sketches, handwriting and images. Our solution provides users a way to watch and annotate television content, in real-time and in a collaborative environment. Using a mobile device, users can annotate content together with other users, while watching both content and annotations on a TV. These annotations can also be shared through social networks or saved on other platforms. Finally, the system also uses content provided by the users to search and link to television content.Fundação para a Ciência e a Tecnologia - UTA-Est/MAI/0010/2009 - project ImTV (On-Demand Immersive-TV for Communities of Media Producers and Consumers

Augmenting patient therapies with video game technology

PhD ThesisThere is an increasing body of work showing that video games can be used for more than just entertainment, but can also facilitate positive physical and mental changes. For people suffering debilitating side-effects from illnesses such as stroke, there is need to deliver and monitor effective rehabilitative physical therapies; video game technologies could potentially deliver an effective alternative to traditional rehabilitative physical therapy, and alleviate the need for direct therapist oversight. Most existing research into video game therapies has focussed on the use of offthe- shelf games to augment a patient’s ongoing therapy. There has currently been little progress into how best to design bespoke software capable of integrating with traditional therapy, or how to replicate common therapies and medical measurements in software. This thesis investigates the ability for video games to be applied to stroke rehabilitation, using modern gaming peripherals for input. The work presents a quantitative measurement of motion detection quality afforded by such hardware. An extendible game development framework capable of high quality movement data output is also presented, affording detailed analysis of player responsiveness to a video game delivered therapy for acute stroke. Finally, a system by which therapists can interactively create complex physical movements for their patients to replicate in a video game environment is detailed, enabling bespoke therapies to be developed, and providing the means by which rehabilitative games for stroke can provide an assessment of patient ability similar to that afforded by traditional therapies

3D Information Technologies in Cultural Heritage Preservation and Popularisation

This Special Issue of the journal Applied Sciences presents recent advances and developments in the use of digital 3D technologies to protect and preserve cultural heritage. While most of the articles focus on aspects of 3D scanning, modeling, and presenting in VR of cultural heritage objects from buildings to small artifacts and clothing, part of the issue is devoted to 3D sound utilization in the cultural heritage field

Sonic Interaction Design in Immersive Theatre

Immersive theatre has two vital features: immersion and interactivity. Immersion has two types: one is physical immersion, and the other one is mental immersion. Interactivity happens between audiences and immersive theatre production, such as props, installations and actors. Audiences have chances to interact with sets, installations and actors. Besides, immersion and interactivity are not two distinct features. Interactivity can help lower the barriers to the immersive experience, but it cannot guarantee immersion in the audience. This thesis shows the result of three professional sound designers’ interviews and a case study of immersive theatre – Sleep No More in Shanghai to analyse sound design methods. From interviews and the case study, we can learn that sound is an important component in immersive theatre, as in other theatre performances. Sound design approaches have some similarities and differences when compared to traditional theatre. Looping sound is a main technique for the sound designer. Besides, they will also create a different audio system for immersive theatre, which happens in different spaces. In this thesis, a sonic interactive system flowchart was provided. It compares two types of gesture recognition technology, analyses the possible ways users can manipulate this system by using software design methods and uses the audio engine to design the system flowchart. Considering the concept of the sound zone in Sleep No More, we can combine scopes of camera-based gesture recognition devices with the sound zone in immersive theatre

FLOW. INTERACTIVE SONIC ART: THE CREATION AND USE OF RESPONSIVE STRATEGIES TO RE-IMAGINE THE PERFORMER/SPECTATOR RELATIONSHIP AND CREATE VISITOR INCLUSIVE SONIC ENVIRONMENTS.

FLOW operates on two levels, firstly as an engaging live performance environment and secondly as a vehicle to discuss a number of philosophical ideas relating to sound as art. As a performance piece FLOW exists to provide an inclusive interactive environment for musicians and casual visitors alike. A series of sensors allow those who enter the arena to make interventions in an immersive soundscape through their movements, opening up possibilities for the exploration of sound and gestural action within the space. The piece challenges the conventional roles of performer and spectator and offers interactive technology as a means of uniting the two. The artist creates a re-imagination of the performance paradigm based on active engagement rather than passive observance through the establishment of a circular discourse between human and computer. The following paper will also examine the nature of sound as art, suggesting that the poststructural ideas of Derrida and Deleuze and Guattari can be used as a conduit to define sonic emergences and morphologies within a Human/computer discourse, both in terms of timbral nature and spatial diffusion. Central to this is the concept that suggests the relationship between man and machine in interactive sonic art is one of energy transfer from organic fluidity to digital regulation and back to energy in the form of processed sound, according to the processes put in place. This leads into a final discussion of the nature of experimental compositional process, the choice between the determinate and the stochastic and the compromises between these that may need to be made to retain artistic coherence

Interaction Design for Digital Musical Instruments

The thesis aims to elucidate the process of designing interactive systems for musical performance that combine software and hardware in an intuitive and elegant fashion. The original contribution to knowledge consists of: (1) a critical assessment of recent trends in digital musical instrument design, (2) a descriptive model of interaction design for the digital musician and (3) a highly customisable multi-touch performance system that was designed in accordance with the model. Digital musical instruments are composed of a separate control interface and a sound generation system that exchange information. When designing the way in which a digital musical instrument responds to the actions of a performer, we are creating a layer of interactive behaviour that is abstracted from the physical controls. Often, the structure of this layer depends heavily upon: 1. The accepted design conventions of the hardware in use 2. Established musical systems, acoustic or digital 3. The physical configuration of the hardware devices and the grouping of controls that such configuration suggests This thesis proposes an alternate way to approach the design of digital musical instrument behaviour – examining the implicit characteristics of its composite devices. When we separate the conversational ability of a particular sensor type from its hardware body, we can look in a new way at the actual communication tools at the heart of the device. We can subsequently combine these separate pieces using a series of generic interaction strategies in order to create rich interactive experiences that are not immediately obvious or directly inspired by the physical properties of the hardware. This research ultimately aims to enhance and clarify the existing toolkit of interaction design for the digital musician

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD