Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments

Despite the impressive progress of telepresence systems for room-scale scenes with static and dynamic scene entities, expanding their capabilities to scenarios with larger dynamic environments beyond a fixed size of a few square-meters remains challenging. In this paper, we aim at sharing 3D live-telepresence experiences in large-scale environments beyond room scale with both static and dynamic scene entities at practical bandwidth requirements only based on light-weight scene capture with a single moving consumer-grade RGB-D camera. To this end, we present a system which is built upon a novel hybrid volumetric scene representation in terms of the combination of a voxel-based scene representation for the static contents, that not only stores the reconstructed surface geometry but also contains information about the object semantics as well as their accumulated dynamic movement over time, and a point-cloud-based representation for dynamic scene parts, where the respective separation from static parts is achieved based on semantic and instance information extracted for the input frames. With an independent yet simultaneous streaming of both static and dynamic content, where we seamlessly integrate potentially moving but currently static scene entities in the static model until they are becoming dynamic again, as well as the fusion of static and dynamic data at the remote client, our system is able to achieve VR-based live-telepresence at close to real-time rates. Our evaluation demonstrates the potential of our novel approach in terms of visual quality, performance, and ablation studies regarding involved design choices

ISMCR 1994: Topical Workshop on Virtual Reality. Proceedings of the Fourth International Symposium on Measurement and Control in Robotics

This symposium on measurement and control in robotics included sessions on: (1) rendering, including tactile perception and applied virtual reality; (2) applications in simulated medical procedures and telerobotics; (3) tracking sensors in a virtual environment; (4) displays for virtual reality applications; (5) sensory feedback including a virtual environment application with partial gravity simulation; and (6) applications in education, entertainment, technical writing, and animation

Latency and Distortion compensation in Augmented Environments using Electromagnetic trackers

Augmented reality (AR) systems are often used to superimpose virtual objects or information on a scene to improve situational awareness. Delays in the display system or inaccurate registration of objects destroy the sense of immersion a user experiences when using AR systems. AC electromagnetic trackers are ideally for these applications when combined with head orientation prediction to compensate for display system delays. Unfortunately, these trackers do not perform well in environments that contain conductive or ferrous materials due to magnetic field distortion without expensive calibration techniques. In our work we focus on both the prediction and distortion compensation aspects of this application, developing a “small footprint” predictive filter for display lag compensation and a simplified calibration system for AC magnetic trackers. In the first phase of our study we presented a novel method of tracking angular head velocity from quaternion orientation using an Extended Kalman Filter in both single model (DQEKF) and multiple model (MMDQ) implementations. In the second phase of our work we have developed a new method of mapping the magnetic field generated by the tracker without high precision measurement equipment. This method uses simple fixtures with multiple sensors in a rigid geometry to collect magnetic field data in the tracking volume. We have developed a new algorithm to process the collected data and generate a map of the magnetic field distortion that can be used to compensation distorted measurement data

Enabling Real-Time Shared Environments on Mobile Head-Mounted Displays

Head-Mounted Displays (HMDs) are becoming more prevalent consumer devices, allowing users to experience scenes and environments from a point of view naturally controlled by their movement. However there is limited application of this experiential paradigm to telecommunications -- that is, where a HMD user can 'call' a mobile phone user and begin to look around in their environment. In this thesis we present a telepresence system for connecting mobile phone users with people wearing HMDs, allowing the HMD user to experience the environment of the mobile user in real-time. We developed an Android application that supports generating and transmitting high quality spherical panorama based environments in real-time, and a companion application for HMDs to view those environments live. This thesis focusses on the technical challenges involved with creating panoramic environments of sufficient quality to be suitable for viewing inside a HMD, given the constraints that arise from using mobile phones. We present computer vision techniques optimised for these constrained conditions, justifying the trade-offs made between speed and quality. We conclude by comparing our solution to conceptually similar past research along the metrics of computation speed and output quality

A survey on Quality of Experience of Virtual and Augmented Reality environments

Οι εφαρμογές εικονικής και επαυξημένης πραγματικότητας αποτελούν μια νέα και πολλά υποσχόμενη τεχνολογία, με εφαρμογές στην ιατρική, την εκπαίδευση, τα βιντεοπαιχνίδια, το ηλεκτρονικό εμπόριο και πολλές άλλες. Αυτή η τεχνολογία θέτει μια νέα πρόκληση για τους σχεδιαστές εφαρμογών, καθώς και για τους παρόχους υπηρεσιών δικτύου, επειδή είναι εντατική ως προς τους απαιτούμενους πόρους ώστε να είναι σε θέση να παρέχει συναρπαστική εμπειρία στους χρήστες της. Αυτή η πρόκληση γίνεται ακόμη πιο δύσκολη στα δίκτυα κινητής τηλεφωνίας, λόγω παραγόντων που είναι δύσκολο να μοντελοποιηθούν και να προβλεφθούν, όπως η κινητικότητα, η στρατηγική μεταβίβασης και η κατανομή πόρων. Αυτή η διπλωματική φιλοδοξεί να παράσχει μια ανασκόπηση των τεχνικών και μεθόδων εκτίμησης της ποιότητας της εμπειρίας (QoE) και των μεθόδων που έχουν αναπτυχθεί γύρω από αυτές τις εφαρμογές. Στην πρώτη ενότητα, εξετάζουμε τις στρατηγικές παροχής QoE για εφαρμογές εικονικής πραγματικότητας. Αυτή η ενότητα εξετάζει αρκετές περιπτώσεις εφαρμογών εικονικής πραγματικότητας, όπως ένα λογισμικό προσομοίωσης βαρέων μηχανημάτων, μια εκπαιδευτική εφαρμογή και άλλες ψηφιακές εφαρμογές εμβύθισης. Το εύρος και η ποικιλία των εφαρμογών και των μεθόδων εκτίμησης της ποιότητας της εμπειρίας οδηγούν σε αντικρουόμενα συμπεράσματα σχετικά με τις μεθόδους αξιολόγησης QoE. Στην επόμενη ενότητα αναφερόμαστε σε εφαρμογές επαυξημένης πραγματικότητας, και πάλι με μια αναφορά σε μια μεγάλη ποικιλία εφαρμογών, όπως ένας βοηθός επαυξημένης πραγματικότητας, βιντεοπαιχνίδια επαυξημένης πραγματικότητας και άλλες ψηφιακές εφαρμογές. Τα συμπεράσματα σε αυτήν την ενότητα είναι πιο ισχυρά και τα συναισθήματα των ανθρώπων μπορούν να σχηματίσουν πιο ουσιαστικά πορίσματα. Στην τελευταία ενότητα διερευνούμε την QoE σε εφαρμογές εικονικής και επαυξημένης πραγματικότητας για κινητές συσκευές και κινητά δίκτυα. Σε αυτό το μέρος ασχολούμαστε με πιο τεχνικές πτυχές όπως η διαχείριση της κινητικότητας, οι στρατηγικές μεταβίβασης και οι αλγόριθμοι κατανομής πόρων και ο αντίκτυπος που έχουν αυτοί στην εμπειρία των χρηστών.Virtual and augmented reality applications constitute a new and promising technology, with applications in medicine, education, gaming, e-commerce and many more. This technology poses a new challenge to application designers, as well as network service providers, because it is resource intensive in order to be able to provide the immersive experience to its users. This task becomes even more challenging in mobile networks, due to factors that are difficult to be modeled and predicted, such as mobility, handoff strategy and resource allocation. This thesis aspires to provide a review of Quality of Experience (QoE) estimation and provision techniques and methods that have been developed around these applications. In the first section, we review QoE provisioning strategies for virtual reality applications. This section examines some corner cases of augmented reality applications, such as a heavy machinery simulation software, an educational application, and many digital immersive applications. The scope and diversity of applications and implementation methods lead to some conflicting conclusions in relation to QoE evaluation methods. In the next section we refer to augmented reality applications, again with a reference to a wide variety of applications such as an augmented reality task assistant, augmented reality video games and digital immersive applications. The conclusions in this section are more robust and peoples’ feelings can form more meaningful aggregations. In the last section we investigate the QoE in virtual and augmented reality applications when these applications are implemented in mobile devices. Τhis part is concerned with more technical aspects such as mobility management, handoff strategies and resource allocation algorithms and their impact on users’ experience

Videos in Context for Telecommunication and Spatial Browsing

The research presented in this thesis explores the use of videos embedded in panoramic imagery to transmit spatial and temporal information describing remote environments and their dynamics. Virtual environments (VEs) through which users can explore remote locations are rapidly emerging as a popular medium of presence and remote collaboration. However, capturing visual representation of locations to be used in VEs is usually a tedious process that requires either manual modelling of environments or the employment of specific hardware. Capturing environment dynamics is not straightforward either, and it is usually performed through specific tracking hardware. Similarly, browsing large unstructured video-collections with available tools is difficult, as the abundance of spatial and temporal information makes them hard to comprehend. At the same time, on a spectrum between 3D VEs and 2D images, panoramas lie in between, as they offer the same 2D images accessibility while preserving 3D virtual environments surrounding representation. For this reason, panoramas are an attractive basis for videoconferencing and browsing tools as they can relate several videos temporally and spatially. This research explores methods to acquire, fuse, render and stream data coming from heterogeneous cameras, with the help of panoramic imagery. Three distinct but interrelated questions are addressed. First, the thesis considers how spatially localised video can be used to increase the spatial information transmitted during video mediated communication, and if this improves quality of communication. Second, the research asks whether videos in panoramic context can be used to convey spatial and temporal information of a remote place and the dynamics within, and if this improves users' performance in tasks that require spatio-temporal thinking. Finally, the thesis considers whether there is an impact of display type on reasoning about events within videos in panoramic context. These research questions were investigated over three experiments, covering scenarios common to computer-supported cooperative work and video browsing. To support the investigation, two distinct video+context systems were developed. The first telecommunication experiment compared our videos in context interface with fully-panoramic video and conventional webcam video conferencing in an object placement scenario. The second experiment investigated the impact of videos in panoramic context on quality of spatio-temporal thinking during localization tasks. To support the experiment, a novel interface to video-collection in panoramic context was developed and compared with common video-browsing tools. The final experimental study investigated the impact of display type on reasoning about events. The study explored three adaptations of our video-collection interface to three display types. The overall conclusion is that videos in panoramic context offer a valid solution to spatio-temporal exploration of remote locations. Our approach presents a richer visual representation in terms of space and time than standard tools, showing that providing panoramic contexts to video collections makes spatio-temporal tasks easier. To this end, videos in context are suitable alternative to more difficult, and often expensive solutions. These findings are beneficial to many applications, including teleconferencing, virtual tourism and remote assistance

Contributions to the Development of Objective Techniques for Presence Measurement in Virtual Environments by means of Brain Activity Analysis

En esta tesis, se propone el uso de la técnica de Doppler transcraneal (DTC) para monitorizar la actividad cerebral durante la exposición a entornos virtuales (EV) y así poder analizar los correlatos cerebrales del sentido de presencia. Las hipótesis de partida son las siguientes: 1) DTC se podrá utilizar fácilmente en combinación con sistemas de realidad virtual. 2) Los datos de velocidad de flujo sanguíneo medidos por DTC se podrán utilizar para analizar cambios de actividad cerebral durante la exposición a EV. 3) Habrá diferencias en la velocidad del flujo sanguíneo asociadas a distintos niveles de presencia. 4) Habrá correlación entre el grado de presencia medido por cuestionarios y parámetros de la velocidad de flujo sanguíneo. 5) Cada componente de la experiencia virtual tendrá una influencia en las variaciones de velocidad observadas. Para analizar las hipótesis planteadas, se realizaron cuatro experimentos distintos, en los que se analizó la velocidad del flujo sanguíneo durante: 1) distintas condiciones de navegación, 2) distintas condiciones de inmersión, 3) una tarea de percepción visual y 4) tareas motoras para manejo de un joystick. Durante la tesis, se han propuesto distintas técnicas de procesado de señal basadas en análisis espectral y en la obtención parámetros no lineales de la señal, que no habían sido utilizadas previamente en experimentos psicofisiológicos con DTC. Se ha observado que existe un incremento en la velocidad del flujo sanguíneo durante la exposición a un EV, el cual puede deberse a distintos factores que intervienen en la experiencia: tareas de interacción visuoespacial, tareas de atención, la creación y ejecución de un plan motor, cambios emocionales Los análisis han mostrado que existen correlaciones significativas entre la velocidad media de flujo sanguíneo en las arterias cerebrales medias durante la exposición al EV y respuestas a los cuestionarios de presencia utilizados.Rey Solaz, B. (2010). Contributions to the Development of Objective Techniques for Presence Measurement in Virtual Environments by means of Brain Activity Analysis [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/8505Palanci

Haptic data reduction through dynamic perceptual analysis and event-based communication

This research presents an adjustable and flexible framework for haptic data compression and communication that can be used in a robotic teleoperation session. The framework contains a customized event-driven transmission control protocol, several dynamically adaptive perceptual and prediction methods for haptic sample reduction, and last but not the least, an architecture for the data flow

Améliorer les interactions homme-machine et la présence sociale avec l’informatique physiologique

This thesis explores how physiological computing can contribute to human-computer interaction (HCI) and foster new communication channels among the general public. We investigated how physiological sensors, such as electroencephalography (EEG), could be employed to assess the mental state of the users and how they relate to other evaluation methods. We created the first brain-computer interface that could sense visual comfort during the viewing of stereoscopic images and shaped a framework that could help to assess the over all user experience by monitoring workload, attention and error recognition.To lower the barrier between end users and physiological sensors,we participated in the software integration of a low-cost and open hardware EEG device; used off-the shelf webcams to measure heart rate remotely, crafted we arables that can quickly equip users so that electrocardiography, electrodermal activity or EEG may be measured during public exhibitions. We envisioned new usages for our sensors, that would increase social presence. In a study about human-agent interaction, participants tended to prefer virtual avatars that were mirroring their own internal state. A follow-up study focused on interactions between users to describe how physiological monitoringcould alter our relationships. Advances in HCI enabled us to seam lesslyintegrate biofeedback to the physical world. We developped Teegi, apuppet that lets novices discover by themselves about their brain activity. Finally, with Tobe, a toolkit that encompasses more sensors and give more freedom about their visualizations, we explored how such proxy shifts our representations, about our selves as well as about the others.Cette thèse explore comment l’informatique physiologique peut contribuer aux interactions homme-machine (IHM) et encourager l’apparition de nouveaux canaux de communication parmi le grand public. Nous avons examiné comment des capteurs physiologiques,tels que l’électroencéphalographie (EEG), pourraient être utilisés afin d’estimer l’état mental des utilisateurs et comment ils se positionnent par rapport à d’autres méthodes d’évaluation. Nous avons créé la première interface cerveau-ordinateur capable de discriminer le confort visuel pendant le visionnage d’images stéréoscopiques et nous avons esquissé un système qui peux aider à estimer l’expérience utilisateur dans son ensemble, en mesurant charge mentale, attention et reconnaissance d’erreur. Pour abaisser la barrière entre utilisateurs finaux et capteurs physiologiques, nous avons participé à l’intégration logicielle d’un appareil EEG bon marché et libre, nous avons utilisé des webcams du commerce pour mesurer le rythme cardiaque à distance, nous avons confectionné des wearables dont les utilisateurs peuvent rapidement s’équiper afin qu’électrocardiographie, activité électrodermale et EEG puissent être mesurées lors de manifestations publiques. Nous avons imaginé de nouveaux usages pour nos capteurs, qui augmenteraient la présence sociale. Dans une étude autour de l’interaction humain agent,les participants avaient tendance à préférer les avatars virtuels répliquant leurs propres états internes. Une étude ultérieure s’est concentrée sur l’interaction entre utilisateurs, profitant d’un jeu de plateau pour décrire comment l’examen de la physiologie pourrait changer nos rapports. Des avancées en IHM ont permis d’intégrer de manière transparente du biofeedback au monde physique. Nous avons développé Teegi, une poupée qui permet aux novices d’en découvrir plus sur leur activité cérébrale, par eux-mêmes. Enfin avec Tobe, un toolkit qui comprend plus de capteurs et donne plus de liberté quant à leurs visualisations, nous avons exploré comment un tel proxy décalenos représentations, tant de nous-mêmes que des autres

Image-Guided Robot-Assisted Techniques with Applications in Minimally Invasive Therapy and Cell Biology

There are several situations where tasks can be performed better robotically rather than manually. Among these are situations (a) where high accuracy and robustness are required, (b) where difficult or hazardous working conditions exist, and (c) where very large or very small motions or forces are involved. Recent advances in technology have resulted in smaller size robots with higher accuracy and reliability. As a result, robotics is fi nding more and more applications in Biomedical Engineering. Medical Robotics and Cell Micro-Manipulation are two of these applications involving interaction with delicate living organs at very di fferent scales.Availability of a wide range of imaging modalities from ultrasound and X-ray fluoroscopy to high magni cation optical microscopes, makes it possible to use imaging as a powerful means to guide and control robot manipulators. This thesis includes three parts focusing on three applications of Image-Guided Robotics in biomedical engineering, including: Vascular Catheterization: a robotic system was developed to insert a catheter through the vasculature and guide it to a desired point via visual servoing. The system provides shared control with the operator to perform a task semi-automatically or through master-slave control. The system provides control of a catheter tip with high accuracy while reducing X-ray exposure to the clinicians and providing a more ergonomic situation for the cardiologists. Cardiac Catheterization: a master-slave robotic system was developed to perform accurate control of a steerable catheter to touch and ablate faulty regions on the inner walls of a beating heart in order to treat arrhythmia. The system facilitates touching and making contact with a target point in a beating heart chamber through master-slave control with coordinated visual feedback. Live Neuron Micro-Manipulation: a microscope image-guided robotic system was developed to provide shared control over multiple micro-manipulators to touch cell membranes in order to perform patch clamp electrophysiology. Image-guided robot-assisted techniques with master-slave control were implemented for each case to provide shared control between a human operator and a robot. The results show increased accuracy and reduced operation time in all three cases