Correlated multi-streaming in distributed interactive multimedia systems

Distributed Interactive Multimedia Environments (DIMEs) enable geographically distributed people to interact with each other in a joint media-rich virtual environment for a wide range of activities, such as art performance, medical consultation, sport training, etc. The real-time collaboration is made possible by exchanging a set of multi-modal sensory streams over the network in real time. The characterization and evaluation of such multi-stream interactive environments is challenging because the traditional Quality of Service metrics (e.g., delay, jitter) are limited to a per stream basis. In this work, we present a novel ???Bundle of Streams??? concept to de???ne correlated multi-streams in DIMEs and present new cyber-physical, spatio-temporal QoS metrics to measure QoS over bundle of streams. We realize Bundle of Streams concept by presenting a novel paradigm of Bundle Streaming as a Service (SAS). We propose and develop SAS Kernel, a generic, distributed, modular and highly ???exible streaming kernel realizing SAS concept. We validate the Bundle of Streams model by comparing the QoS performance of bundle of streams over different transport protocols in a 3D tele-immersive testbed. Also, further experiments demonstrate that the SAS Kernel incurs low overhead in delay, CPU, and bandwidth demands

Multi-Stream Management for Supporting Multi-Party 3D Tele-Immersive Environments

Three-dimensional tele-immersive (3DTI) environments have great potential to promote collaborative work among geographically distributed participants. However, extensive application of 3DTI environments is still hindered by problems pertaining to scalability, manageability and reliance of special-purpose components. Thus, one critical question is how to organize the acquisition, transmission and display of large volume real-time 3D visual data over commercially available computing and networking infrastructures so that .everybody. would be able to install and enjoy 3DTI environments for high quality tele-collaboration. In the thesis, we explore the design space from the angle of multi-stream Quality-of-Service (QoS) management to support multi-party 3DTI communication. In 3DTI environments, multiple correlated 3D video streams are deployed to provide a comprehensive representation of the physical scene. Traditional QoS approach in 2D and single-stream scenario has become inadequate. On the other hand, the existence of multiple streams provides unique opportunity for QoS provisioning. We propose an innovative cross-layer hierarchical and distributed multi-stream management middleware framework for QoS provisioning to fully enable multi-party 3DTI communication over general delivery infrastructure. The major contributions are as follows. First, we introduce the view model for representing the user interest in the application layer. The design revolves around the concept of view-aware multi-stream coordination, which leverages the central role of view semantics in 3D video systems. Second, in the stream differentiation layer we present the design of view to stream mapping, where a subset of relevant streams are selected based on the relative importance of each stream to the current view. Conventional streaming controllers focus on a fixed set of streams specified by the application. Different from all the others, in our management framework the application layer only specifies the view information while the underlying controller dynamically determines the set of streams to be managed. Third, in the stream coordination layer we present two designs applicable in different situations. In the case of end-to-end 3DTI communication, a learning-based controller is embedded which provides bandwidth allocation for relevant streams. In the case of multi-party 3DTI communication, we propose a novel ViewCast protocol to coordinate the multi-stream content dissemination upon an end-system overlay network

Métodos de representação virtual e visualização para informação arquitetónica e contextual em sítios arqueológicos

This work seeks to outline some guidelines in order to improve the use of 3D visualization applied to archaeological data of diverse nature and at different scales. One difficulty found in this process is related to the still frequent two-dimensional representation of the three-dimensional archaeological reality. Aware that the existence of data of two-dimensional nature is fundamental in the archaeological process and that they result, on the one hand, from the manual archaeological recording processes and, on the other hand, from the intense analysis and interpretation activity of the archaeological investigation team, we seek to ensure an adequate 3D representation based on 3D acquisition methods mostly available to the archaeology teams. Archaeological visualization in three-dimensional support is an increasingly frequent and necessary practice, but it continues to show some difficulties. These are substantiated in the reduced number of visualization techniques used, the use of visualization tools that are not very customized for the archaeological needs and the privileged use of visual features of the models during the archaeological process phases. Thus, the main objective of this work is to design and evaluate appropriate methods for visualizing archaeological data. To determine which visualization methods are most used during the phases of the archaeological process, an online user-survey was carried out, which allowed consolidating the 3D representation methodologies used, as well as to propose a visualization model that also categorizes the appropriate visualization techniques which increase the visual perception and understanding of the archaeological elements. Three prototypes are defined according to the different 3D data acquisition methodologies presented and visualization methodologies are designed in order to, on the one hand, take into account the scale and diversity of the archaeological elements and, on the other hand, to account for the need to ensure visualization methods which are easily assimilated by archaeologists. Each prototype was evaluated by two archaeologists with different professional background. They were proposed, through a set of previously determined tasks, to assess the interaction with 3D models and with the visualization methods and the satisfaction of the visualization results regarding the archaeological needs. The evaluation of the prototypes allowed to conclude that the presented visualization methods increase the perception of 3D models which represent archaeological elements. In addition, it was also possible to produce new objects that reveal elements of archaeological interest. It is suggested to make these methodologies available on a web-based application and on mobile platforms.Este trabalho procura esboçar algumas diretrizes no sentido de melhorar a utilização da visualização 3D aplicada aos dados arqueológicos de natureza diversa e a escalas distintas. Uma dificuldade encontrada neste processo prende-se com a, ainda frequente, representação bidimensional da realidade arqueológica tridimensional. Ciente de que a existência de dados de natureza bidimensional são fundamentais no processo arqueológico e que resultam, por um lado, dos processos manuais de registo arqueológicos e, por outro, da intensa atividade de análise e interpretação da equipa de investigação arqueológica, procuramos assegurar uma representação 3D adequada, com base em metodologias de aquisição de dados 3D geralmente disponíveis às equipas de arqueologia. A visualização arqueológica em suporte tridimensional é uma prática cada vez mais frequente e necessária, mas que continua a evidenciar algumas dificuldades. Estas substanciam-se no reduzido número de técnicas de visualização usadas, na utilização de ferramentas de visualização pouco adaptadas às necessidades arqueológicas e na utilização preferencial de características visuais dos modelos durante as fases do processo arqueológico. Assim, o objetivo primordial deste trabalho é desenhar e avaliar métodos adequados à visualização de dados arqueológicos. Para determinar que métodos de visualização são mais utilizados durante as fases do processo arqueológico realizou-se um questionário online que permitiu consolidar as metodologias de representação 3D usadas, bem como propor um modelo de visualização que também categoriza as técnicas de visualização adequadas para aumentar a perceção e a compreensão visual dos elementos arqueológicos. Definem-se três protótipos de acordo com as distintas metodologias de aquisição de dados 3D apresentados e são desenhadas metodologias de visualização que, por um lado, têm em conta a escala e a diversidade dos elementos arqueológicos e, por outro, a necessidade de assegurar métodos de visualização facilmente assimilados pelos arqueólogos. Cada protótipo foi avaliado por dois arqueólogos com experiências profissionais distintas. O que lhes foi proposto, através de um conjunto de tarefas previamente estabelecidas, foi aferir da facilidade de interação com os modelos 3D e com os métodos de visualização e adequação dos resultados de visualização às necessidades dos arqueólogos. A avaliação dos protótipos permitiu concluir que os métodos de visualização apresentados aumentam a perceção dos modelos 3D que representam elementos arqueológicos. Para além disso foi possível produzir também novos objetos que revelam elementos com interesse arqueológico. É sugerida a disponibilização destas metodologias em ambiente web e plataformas móveis.Programa Doutoral em Informátic

Web-based Stereoscopic Collaboration for Medical Visualization

Medizinische Volumenvisualisierung ist ein wertvolles Werkzeug zur Betrachtung von Volumen- daten in der medizinischen Praxis und Lehre. Eine interaktive, stereoskopische und kollaborative Darstellung in Echtzeit ist notwendig, um die Daten vollständig und im Detail verstehen zu können. Solche Visualisierung von hochauflösenden Daten ist jedoch wegen hoher Hardware- Anforderungen fast nur an speziellen Visualisierungssystemen möglich. Remote-Visualisierung wird verwendet, um solche Visualisierung peripher nutzen zu können. Dies benötigt jedoch fast immer komplexe Software-Deployments, wodurch eine universelle ad-hoc Nutzbarkeit erschwert wird. Aus diesem Sachverhalt ergibt sich folgende Hypothese: Ein hoch performantes Remote- Visualisierungssystem, welches für Stereoskopie und einfache Benutzbarkeit spezialisiert ist, kann für interaktive, stereoskopische und kollaborative medizinische Volumenvisualisierung genutzt werden. Die neueste Literatur über Remote-Visualisierung beschreibt Anwendungen, welche nur reine Webbrowser benötigen. Allerdings wird bei diesen kein besonderer Schwerpunkt auf die perfor- mante Nutzbarkeit von jedem Teilnehmer gesetzt, noch die notwendige Funktion bereitgestellt, um mehrere stereoskopische Präsentationssysteme zu bedienen. Durch die Bekanntheit von Web- browsern, deren einfach Nutzbarkeit und weite Verbreitung hat sich folgende spezifische Frage ergeben: Können wir ein System entwickeln, welches alle Aspekte unterstützt, aber nur einen reinen Webbrowser ohne zusätzliche Software als Client benötigt? Ein Proof of Concept wurde durchgeführt um die Hypothese zu verifizieren. Dazu gehörte eine Prototyp-Entwicklung, deren praktische Anwendung, deren Performanzmessung und -vergleich. Der resultierende Prototyp (CoWebViz) ist eines der ersten Webbrowser basierten Systeme, welches flüssige und interaktive Remote-Visualisierung in Realzeit und ohne zusätzliche Soft- ware ermöglicht. Tests und Vergleiche zeigen, dass der Ansatz eine bessere Performanz hat als andere ähnliche getestete Systeme. Die simultane Nutzung verschiedener stereoskopischer Präsen- tationssysteme mit so einem einfachen Remote-Visualisierungssystem ist zur Zeit einzigartig. Die Nutzung für die normalerweise sehr ressourcen-intensive stereoskopische und kollaborative Anatomieausbildung, gemeinsam mit interkontinentalen Teilnehmern, zeigt die Machbarkeit und den vereinfachenden Charakter des Ansatzes. Die Machbarkeit des Ansatzes wurde auch durch die erfolgreiche Nutzung für andere Anwendungsfälle gezeigt, wie z.B. im Grid-computing und in der Chirurgie

Semantics-aware content delivery framework for 3D Tele-immersion

3D Tele-immersion (3DTI) technology allows full-body, multimodal interaction among geographically dispersed users, which opens a variety of possibilities in cyber collaborative applications such as art performance, exergaming, and physical rehabilitation. However, with its great potential, the resource and quality demands of 3DTI rise inevitably, especially when some advanced applications target resource-limited computing environments with stringent scalability demands. Under these circumstances, the tradeoffs between 1) resource requirements, 2) content complexity, and 3) user satisfaction in delivery of 3DTI services are magnified. In this dissertation, we argue that these tradeoffs of 3DTI systems are actually avoidable when the underlying delivery framework of 3DTI takes the semantic information into consideration. We introduce the concept of semantic information into 3DTI, which encompasses information about the three factors: environment, activity, and user role in 3DTI applications. With semantic information, 3DTI systems are able to 1) identify the characteristics of its computing environment to allocate computing power and bandwidth to delivery of prioritized contents, 2) pinpoint and discard the dispensable content in activity capturing according to properties of target application, and 3) differentiate contents by their contributions on fulfilling the objectives and expectation of user’s role in the application so that the adaptation module can allocate resource budget accordingly. With these capabilities we can change the tradeoffs into synergy between resource requirements, content complexity, and user satisfaction. We implement semantics-aware 3DTI systems to verify the performance gain on the three phases in 3DTI systems’ delivery chain: capturing phase, dissemination phase, and receiving phase. By introducing semantics information to distinct 3DTI systems, the efficiency improvements brought by our semantics-aware content delivery framework are validated under different application requirements, different scalability bottlenecks, and different user and application models. To sum up, in this dissertation we aim to change the tradeoff between requirements, complexity, and satisfaction in 3DTI services by exploiting the semantic information about the computing environment, the activity, and the user role upon the underlying delivery systems of 3DTI. The devised mechanisms will enhance the efficiency of 3DTI systems targeting on serving different purposes and 3DTI applications with different computation and scalability requirements

Video based reconstruction system for mixed reality environments supporting contextualised non-verbal communication and its study

This Thesis presents a system to capture, reconstruct and render the three-dimensional form of people and objects of interest in such detail that the spatial and visual aspects of non-verbal behaviour can be communicated.The system supports live distribution and simultaneous rendering in multiple locations enabling the apparent teleportation of people and objects. Additionally, the system allows for the recording of live sessions and their playback in natural time with free-viewpoint.It utilises components of a video based reconstruction and a distributed video implementation to create an end-to-end system that can operate in real-time and on commodity hardware.The research addresses the specific challenges of spatial and colour calibration, segmentation and overall system architecture to overcome technical barriers, the requirement of domain specific knowledge to setup and generate avatars to a consistent high quality.Applications of the system include, but are not limited to, telepresence, where the computer generated avatars used in Immersive Collaborative Virtual Environments can be replaced with ones that are faithful of the people they represent and supporting researchers in their study of human communication such as gaze, inter-personal distance and facial expression.The system has been adopted in other research projects and is integrated with a mixed reality application where, during a live linkup, a three-dimensional avatar is streamed to multiple end-points across different countries

Multiple-view product representation and development using augmented reality technology

Ph.DDOCTOR OF PHILOSOPH

ASTRAL PROJECTION: THEORIES OF METAPHOR, PHILOSOPHIES OF SCIENCE, AND THE ART O F SCIENTIFIC VISUALIZATION

This thesis provides an intellectual context for my work in computational scientific visualization for large-scale public outreach in venues such as digitaldome planetarium shows and high-definition public television documentaries. In my associated practicum, a DVD that provides video excerpts, 1 focus especially on work I have created with my Advanced Visualization Laboratory team at the National Center for Supercomputing Applications (Champaign, Illinois) from 2002-2007. 1 make three main contributions to knowledge within the field of computational scientific visualization. Firstly, I share the unique process 1 have pioneered for collaboratively producing and exhibiting this data-driven art when aimed at popular science education. The message of the art complements its means of production: Renaissance Team collaborations enact a cooperative paradigm of evolutionary sympathetic adaptation and co-creation. Secondly, 1 open up a positive, new space within computational scientific visualization's practice for artistic expression—especially in providing a theory of digi-epistemology that accounts for how this is possible given the limitations imposed by the demands of mapping numerical data and the computational models derived from them onto visual forms. I am concerned not only with liberating artists to enrich audience's aesthetic experiences of scientific visualization, to contribute their own vision, but also with conceiving of audiences as co-creators of the aesthetic significance of the work, to re-envision and re-circulate what they encounter there. Even more commonly than in the age of traditional media, on-line social computing and digital tools have empowered the public to capture and repurpose visual metaphors, circulating them within new contexts and telling new stories with them. Thirdly, I demonstrate the creative power of visaphors (see footnote, p. 1) to provide novel embodied experiences through my practicum as well as my thesis discussion. Specifically, I describe how the visaphors my Renaissance Teams and I create enrich the Environmentalist Story of Science, essentially promoting a counter-narrative to the Enlightenment Story of Science through articulating how humanity participates in an evolving universal consciousness through our embodied interaction and cooperative interdependence within nested, self-producing (autopoetic) systems, from the micro- to the macroscopic. This contemporary account of the natural world, its inter-related systems, and their dynamics may be understood as expressing a creative and generative energy—a kind of consciousness-that transcends the human yet also encompasses it

Archaeology of Digital Environments: Tools, Methods, and Approaches

Digital archaeologists use digital tools for conducting archaeological work, but their potential also lies in applying archaeological thinking and methods to understanding digital built environments (i.e., software) as contemporary examples of human settlement, use, and abandonment. This thesis argues for digital spaces as archaeological artifacts, sites, and landscapes that can be investigated in both traditional and non-traditional ways. At the core of my research is the fundamental argument that human-occupied digital spaces can be studied archaeologically with existing and modified theory, tools, and methods to reveal that human occupation and use of synthetic worlds is similar to how people behave in the natural world. Working digitally adds new avenues of investigation into human behavior in relation to the things people make, modify, and inhabit. In order to investigate this argument, the thesis focuses on three video game case studies, each using different kinds of archaeology specifically chosen to help understand the software environments being researched: 1) epigraphy, stylometry, and text analysis for the code-artifact of Colossal Cave Adventure; 2) photogrammetry, 3D printing, GIS mapping, phenomenology, and landscape archaeology within the designed, digital heritage virtual reality game-site of Skyrim VR; 3) actual survey and excavation of 30 heritage sites for a community of displaced human players in the synthetic landscape of No Man’s Sky. My conclusions include a blended approach to conducting future archaeological fieldwork in digital built environments, one that modifies traditional approaches to archaeological sites and material in a post/transhuman landscape. As humanity continues trending towards constant digital engagement, archaeologists need to be prepared to study how digital places are settled, used, and abandoned. This thesis takes a step in that direction using the vernacular of games as a starting point