Bi & tri dimensional scene description and composition in the MPEG-4 standard

MPEG-4 is a new ISO/IEC standard being developed by MPEG (Moving Picture Experts Group). The standard is to be released in November 1998 and version 1 will be an International Standard in January 1999 The MPEG-4 standard addresses the new demands that arise in a world in which more and more audio-visual material is exchanged in digital form MPEG-4 addresses the coding of objects of various types. Not only traditional video and audio frames, but also natural video and audio objects as well as textures, text, 2- and 3-dimensional graphic primitives, and synthetic music and sound effects. Using MPEG-4 to reconstruct an audio-visual scene at a terminal, it is hence no longer sufficient to encode the raw audio-visual data and transmit it, as MPEG-2 does m order to synchronize video and audio. In MPEG-4, all objects are multiplexed together at the encoder and transported to the terminal Once de-multiplexed, these objects are composed at the terminal to construct and present to the end user a meaningful audio-visual scene. The placement of these elementary audio-visual objects in space and time is described in the scene description of a scene. While the action of putting these objects together in the same representation space is the composition of audio-visual objects. My research was concerned with the scene description and composition of the audio-visual objects that are defined in an audio-visual scene Scene descriptions are coded independently irom sticams related to primitive audio-visual objects. The set of parameters belonging to the scene description are differentiated from the parameters that are used to improve the coding efficiency of an object. While the independent coding of different objects may achieve a higher compression rate, it also brings the ability to manipulate content at the terminal. This allows the modification of the scene description parameters without having to decode the primitive audio-visual objects themselves. This approach allows the development of a syntax that describes the spatio-temporal relationships of audio-visual scene objects. The behaviours of objects and their response to user inputs can thus also be represented in the scene description, allowing richer audio-visual content to be delivered as an MPEG-4 stream

MPEG-4's BIFS-Anim protocol: using MPEG-4 for streaming of 3D animations

This thesis explores issues related to the generation and animation of synthetic objects within the context of MPEG-4. MPEG-4 was designed to provide a standard that will deliver rich multimedia content on many different platforms and networks. MPEG-4 should be viewed as a toolbox rather than as a monolithic standard as each implementer of the standard will pick the necessary tools adequate to their needs, likely to be a small subset of the available tools. The subset of MPEG-4 that will be examined here are the tools relating to the generation of 3D scenes and to the animation of those scenes. A comparison with the most popular 3D standard, Virtual Reality Modeling Language (VRML) will be included. An overview of the MPEG-4 standard will be given, describing the basic concepts. MPEG-4 uses a scene description language called Binary Format for Scene (BIFS) for the composition of scenes, this description language will be described. The potential for the technology used in BIFS to provide low bitrate streaming 3D animations will be analysed and some examples of the possible uses of this technology will be given. A tool for the encoding of streaming 3D animations will be described and results will be shown that MPEG-4 provides a more efficient way of encoding 3D data when compared to VRML. Finally a look will be taken at the future of 3D content on the Internet

Paralympic VR Game Immersive Game using Virtual Reality Technology

Throughout the years virtual reality has been used for a wide range of applications, and several types of research have been made in order to improve its techniques and technology. In the last few years, the interest in virtual reality has been increasing partially due to the emergence of cheaper and more accessible hardware, and the increase in content available. One of the possible applications for virtual reality is to lead people into seeing situations from a different perspective, which can help change opinions. This thesis uses virtual reality to help people better understand paralympic sports by allowing them to experience the sports’ world from the athletes’ perspective. For the creation of the virtual environment, both computer-generated elements and 360 video are used. The integration of these two components presented a challenge to explore. This thesis focused on wheelchair basketball, and a simulator of this sport was created resorting to the use of a game engine (Unity 3D). For the development of this simulator, computer-generated elements were built, and the interaction with them implemented. Besides allowing the users to play the sport as if they are in the athlete’s shoes, users can also watch 360 videos in which explanations of the modality (rules and classification) are presented. They are also capable of interacting with some of these videos through virtual elements that are placed over the videos. User studies were conducted to evaluate the sense of presence, motion sickness and usability of the system developed. The results were positive although there are still some aspects that should be improved

Merging the Real and the Virtual: An Exploration of Interaction Methods to Blend Realities

We investigate, build, and design interaction methods to merge the real with the virtual. An initial investigation looks at spatial augmented reality (SAR) and its effects on pointing with a real mobile phone. A study reveals a set of trade-offs between the raycast, viewport, and direct pointing techniques. To further investigate the manipulation of virtual content within a SAR environment, we design an interaction technique that utilizes the distance that a user holds mobile phone away from their body. Our technique enables pushing virtual content from a mobile phone to an external SAR environment, interact with that content, rotate-scale-translate it, and pull the content back into the mobile phone. This is all done in a way that ensures seamless transitions between the real environment of the mobile phone and the virtual SAR environment. To investigate the issues that occur when the physical environment is hidden by a fully immersive virtual reality (VR) HMD, we design and investigate a system that merges a realtime 3D reconstruction of the real world with a virtual environment. This allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical reality without losing their sense of immersion or presence inside a virtual world. A study with VR users demonstrates the affordances provided by the system and how it can be used to enhance current VR experiences. We then move to AR, to investigate the limitations of optical see-through HMDs and the problem of communicating the internal state of the virtual world with unaugmented users. To address these issues and enable new ways to visualize, manipulate, and share virtual content, we propose a system that combines a wearable SAR projector. Demonstrations showcase ways to utilize the projected and head-mounted displays together, such as expanding field of view, distributing content across depth surfaces, and enabling bystander collaboration. We then turn to videogames to investigate how spectatorship of these virtual environments can be enhanced through expanded video rendering techniques. We extract and combine additional data to form a cumulative 3D representation of the live game environment for spectators, which enables each spectator to individually control a personal view into the stream while in VR. A study shows that users prefer spectating in VR when compared with a comparable desktop rendering

Augmented reality as a telemedicine platform for remote procedural training

Traditionally, rural areas in many countries are limited by a lack of access to health care due to the inherent challenges associated with recruitment and retention of healthcare professionals. Telemedicine, which uses communication technology to deliver medical services over distance, is an economical and potentially effective way to address this problem. In this research, we develop a new telepresence application using an augmented reality (AR) system. We explore the use of the Microsoft HoloLens to facilitate and enhance remote medical training. Intrinsic advantages of AR systems enable remote learners to perform complex medical procedures such as Point of Care Ultrasound (PoCUS) without visual interference. This research uses the HoloLens to capture the first-person view of a simulated rural emergency room (ER) through mixed reality capture (MRC) and serves as a novel telemedicine platform with remote pointing capabilities. The mentor's hand gestures are captured using a Leap Motion and virtually displayed in the AR space of the HoloLens. To explore the feasibility of the developed platform, twelve novice medical trainees were guided by a mentor through a simulated ultrasound exploration in a trauma scenario, as part of a pilot user study. The study explores the utility of the system from the trainees, mentor, and objective observers' perspectives and compares the findings to that of a more traditional multi-camera telemedicine solution. The results obtained provide valuable insight and guidance for the development of an AR-supported telemedicine platform

Utilização da Norma JPEG2000 para codificar proteger e comercializar Produtos de Observação Terrestre

Applications like, change detection, global monitoring, disaster detection and management have emerging requirements that need the availability of large amounts of data. This data is currently being capture by a multiplicity of instruments and EO (Earth Observation) sensors originating large volumes of data that needs to be stored, processed and accessed in order to be useful – as an example, ENVISAT accumulates, in a yearly basis, several hundred terabytes of data. This need to recover, store, process and access brings some interesting challenges, like storage space, processing power, bandwidth and security, just to mention a few. These challenges are still very important on today’s technological world. If we take a look for example at the number of subscribers of ISP (Internet Service Providers) broadband services on the developed world today, one can notice that broadband services are still far from being common and dominant. On the underdeveloped countries the picture is even dimmer, not only from a bandwidth point of view but also in all other aspects regarding information and communication technologies (ICTs). All this challenges need to be taken into account if a service is to reach the broadest audience possible. Obviously protection and securing of services and contents is an extra asset that helps on the preservation of possible business values, especially if we consider such a costly business as the space industry. This thesis presents and describes a system which allows, not only the encoding and decoding of several EO products into a JPEG2000 format, but also supports some of the security requirements identified previously that allows ESA (European Space Agency) and related EO services to define and apply efficient EO data access security policies and even to exploit new ways to commerce EO products over the Internet.Aplicações como, detecção de mudanças no terreno, monitorização planetária, detecção e gestão de desastres, têm necessidades prementes que necessitam de vastas quantidades de dados. Estes dados estão presentemente a ser capturados por uma multiplicidade de instrumentos e sensores de observação terrestre, que originam uma enormidade de dados que necessitam de ser armazenados processados e acedidos de forma a se tornarem úteis – por exemplo, a ENVISAT acumula anualmente varias centenas de terabytes de dados. Esta necessidade de recuperar, armazenar, processar e aceder introduz alguns desafios interessantes como o espaço de armazenamento, poder de processamento, largura de banda e segurança dos dados só para mencionar alguns. Estes desafios são muito importantes no mundo tecnológico de hoje. Se olharmos, por exemplo, ao número actual de subscritores de ISP (Internet Service Providers) de banda larga nos países desenvolvidos podemos ficar surpreendidos com o facto do número de subscritores desses serviços ainda não ser uma maioria da população ou dos agregados familiares. Nos países subdesenvolvidos o quadro é ainda mais negro não só do ponto de vista da largura de banda mas também de todos os outros aspectos relacionados com Tecnologias da Informação e Comunicação (TICs). Todos estes aspectos devem ser levados em consideração se se pretende que um serviço se torne o mais abrangente possível em termos de audiências. Obviamente a protecção e segurança dos conteúdos é um factor extra que ajuda a preservar possíveis valores de negócio, especialmente considerando industrias tão onerosas como a Industria Espacial. Esta tese apresenta e descreve um sistema que permite, não só a codificação e descodificação de diversos produtos de observação terrestre para formato JPEG2000 mas também o suporte de alguns requisitos de segurança identificados previamente que permitem, á Agência Espacial Europeia e a outros serviços relacionados com observação terrestre, a aplicação de politicas eficientes de acesso seguro a produtos de observação terrestre, permitindo até o aparecimento de novas forma de comercialização de produtos de observação terrestre através da Internet

Designing sound : procedural audio research based on the book by Andy Farnell

In procedural media, data normally acquired by measuring something, commonly described as sampling, is replaced by a set of computational rules (procedure) that defines the typical structure and/or behaviour of that thing. Here, a general approach to sound as a definable process, rather than a recording, is developed. By analysis of their physical and perceptual qualities, natural objects or processes that produce sound are modelled by digital Sounding Objects for use in arts and entertainments. This Thesis discusses different aspects of Procedural Audio introducing several new approaches and solutions to this emerging field of Sound Design.Em Media Procedimental, os dados os dados normalmente adquiridos através da medição de algo habitualmente designado como amostragem, são substituídos por um conjunto de regras computacionais (procedimento) que definem a estrutura típica, ou comportamento, desse elemento. Neste caso é desenvolvida uma abordagem ao som definível como um procedimento em vez de uma gravação. Através da análise das suas características físicas e perceptuais , objetos naturais ou processos que produzem som, são modelados como objetos sonoros digitais para utilização nas Artes e Entretenimento. Nesta Tese são discutidos diferentes aspectos de Áudio Procedimental, sendo introduzidas várias novas abordagens e soluções para o campo emergente do Design Sonoro