Estudi de percepció de Heightmaps en un entorn de Realitat Virtual

Un Heatmap és una representació visual bidimensional de dades amb colors, on tots els colors representen valors diferents. Els Heightmaps es poden considerar com una extensió dels Heatmaps però utilitzant la tercera dimensió per codificar els valors mitjançant alçada, a més de codificar-los per color. Aquest tipus de tècniques de visualització serveixen per descriure la intensitat de les variables i la variància. Són especialment útils per a visualitzar patrons, i fins i tot per identificar anomalies. Davant de la falta d'informació i estudis rellevants, aquest projecte pretén implementar una extensió d'un software existent que permeti crear Heatmaps i Heightmaps en un entorn de realitat virtual. S'avaluarà la viabilitat de la solució i es farà un estudi de percepció.A Heatmap is a two-dimensional visual representation of data with colors, where all colors represent different values. Heightmaps can be thought of as an extension of Heatmaps but using the third dimension to encode values by height, in addition to encoding them by color. These types of visualization techniques serve to describe the intensity of the variables and the variance. They are particularly useful for visualizing patterns, and even for identifying anomalies. Given the lack of relevant information and studies, this project aims to implement an extension of an existing software that allows creating Heatmaps and Heightmaps in a virtual reality environment. The feasibility of the solution will be assessed and a perception study will be carried out

Bridging Vision and Dynamic Legged Locomotion

Legged robots have demonstrated remarkable advances regarding robustness and versatility in the past decades. The questions that need to be addressed in this field are increasingly focusing on reasoning about the environment and autonomy rather than locomotion only. To answer some of these questions visual information is essential. If a robot has information about the terrain it can plan and take preventive actions against potential risks. However, building a model of the terrain is often computationally costly, mainly because of the dense nature of visual data. On top of the mapping problem, robots need feasible body trajectories and contact sequences to traverse the terrain safely, which may also require heavy computations. This computational cost has limited the use of visual feedback to contexts that guarantee (quasi-) static stability, or resort to planning schemes where contact sequences and body trajectories are computed before starting to execute motions. In this thesis we propose a set of algorithms that reduces the gap between visual processing and dynamic locomotion. We use machine learning to speed up visual data processing and model predictive control to achieve locomotion robustness. In particular, we devise a novel foothold adaptation strategy that uses a map of the terrain built from on-board vision sensors. This map is sent to a foothold classifier based on a convolutional neural network that allows the robot to adjust the landing position of the feet in a fast and continuous fashion. We then use the convolutional neural network-based classifier to provide safe future contact sequences to a model predictive controller that optimizes target ground reaction forces in order to track a desired center of mass trajectory. We perform simulations and experiments on the hydraulic quadruped robots HyQ and HyQReal. For all experiments the contact sequences, the foothold adaptations, the control inputs and the map are computed and processed entirely on-board. The various tests show that the robot is able to leverage the visual terrain information to handle complex scenarios in a safe, robust and reliable manner

Técnicas de coste reducido para el posicionamiento del paciente en radioterapia percutánea utilizando un sistema de imágenes ópticas

Patient positioning is an important part of radiation therapy which is one of the main solutions for the treatment of malignant tissue in the human body. Currently, the most common patient positioning methods expose healthy tissue of the patient's body to extra dangerous radiations. Other non-invasive positioning methods are either not very accurate or are very costly for an average hospital. In this thesis, we explore the possibility of developing a system comprised of affordable hardware and advanced computer vision algorithms that facilitates patient positioning. Our algorithms are based on the usage of affordable RGB-D sensors, image features, ArUco planar markers, and other geometry registration methods. Furthermore, we take advantage of consumer-level computing hardware to make our systems widely accessible. More specifically, we avoid the usage of approaches that need to take advantage of dedicated GPU hardware for general-purpose computing since they are more costly. In different publications, we explore the usage of the mentioned tools to increase the accuracy of reconstruction/localization of the patient in its pose. We also take into account the visualization of the patient's target position with respect to their current position in order to assist the person who performs patient positioning. Furthermore, we make usage of augmented reality in conjunction with a real-time 3D tracking algorithm for better interaction between the program and the operator. We also solve more fundamental problems about ArUco markers that could be used in the future to improve our systems. These include highquality multi-camera calibration and mapping using ArUco markers plus detection of these markers in event cameras which are very useful in the presence of fast camera movement. In the end, we conclude that it is possible to increase the accuracy of 3D reconstruction and localization by combining current computer vision algorithms with fiducial planar markers with RGB-D sensors. This is reflected in the low amount of error we have achieved in our experiments for patient positioning, pushing forward the state of the art for this application.En el tratamiento de tumores malignos en el cuerpo, el posicionamiento del paciente en las sesiones de radioterapia es una cuestión crucial. Actualmente, los métodos más comunes de posicionamiento del paciente exponen tejido sano del mismo a radiaciones peligrosas debido a que no es posible asegurar que la posición del paciente siempre sea la misma que la que tuvo cuando se planificó la zona a radiar. Los métodos que se usan actualmente, o no son precisos o tienen costes que los hacen inasequibles para ser usados en hospitales con financiación limitada. En esta Tesis hemos analizado la posibilidad de desarrollar un sistema compuesto por hardware de bajo coste y métodos avanzados de visión por ordenador que ayuden a que el posicionamiento del paciente sea el mismo en las diferentes sesiones de radioterapia, con respecto a su pose cuando fue se planificó la zona a radiar. La solución propuesta como resultado de la Tesis se basa en el uso de sensores RGB-D, características extraídas de la imagen, marcadores cuadrados denominados ArUco y métodos de registro de la geometría en la imagen. Además, en la solución propuesta, se aprovecha la existencia de hardware convencional de bajo coste para hacer nuestro sistema ampliamente accesible. Más específicamente, evitamos el uso de enfoques que necesitan aprovechar GPU, de mayores costes, para computación de propósito general. Se han obtenido diferentes publicaciones para conseguir el objetivo final. Las mismas describen métodos para aumentar la precisión de la reconstrucción y la localización del paciente en su pose, teniendo en cuenta la visualización de la posición ideal del paciente con respecto a su posición actual, para ayudar al profesional que realiza la colocación del paciente. También se han propuesto métodos de realidad aumentada junto con algoritmos para seguimiento 3D en tiempo real para conseguir una mejor interacción entre el sistema ideado y el profesional que debe realizar esa labor. De forma añadida, también se han propuesto soluciones para problemas fundamentales relacionados con el uso de marcadores cuadrados que han sido utilizados para conseguir el objetivo de la Tesis. Las soluciones propuestas pueden ser empleadas en el futuro para mejorar otros sistemas. Los problemas citados incluyen la calibración y el mapeo multicámara de alta calidad utilizando los marcadores y la detección de estos marcadores en cámaras de eventos, que son muy útiles en presencia de movimientos rápidos de la cámara. Al final, concluimos que es posible aumentar la precisión de la reconstrucción y localización en 3D combinando los actuales algoritmos de visión por ordenador, que usan marcadores cuadrados de referencia, con sensores RGB-D. Los resultados obtenidos con respecto al error que el sistema obtiene al reproducir el posicionamiento del paciente suponen un importante avance en el estado del arte de este tópico

Collaborative Interaction Techniques in Virtual Reality for Emergency Management

Virtual Reality (VR) technology has had many interesting applications over the last decades. It can be seen in a multitude of industries: entertainment, education, tourism to crisis management among others. Many of them, feature collaborative uses of VR technology. This thesis presents the design, development and evaluation of a multi-user VR system, aimed at collaborative usage focused on a crisis scenario based on real-life wildfire as the use case. The system also features a dual-map interface to display geographical information, providing both two-dimensional and three-dimensional views over the region and data relevant to the scenario. The main goals of this thesis are to understand how people can collaborate in VR, test which interface is preferred, as well as what kinds of notification mechanisms are more user friendly. The Virtual Environment (VE) displays relevant geo-located information, such as roads, towns, vehicles and the wildfire itself, in a dual-map setup, in two and three dimensions. Users are able to share the environment and, simultaneously, use available tools to interact with the maps and communicate with each other, while controlling the wildfire playback time to understand how it propagates. Actions such as drawing, measuring distances, directing vehicles and notifying other users are available. Users can propose actions that can then be accepted or denied. Eighteen subjects took part in a user study to evaluate the application. Participants were asked to perform several tasks, using the tools available, while sharing that same environment with the researcher. Upon analyzing data from the testing sessions, it is possible to state that most users agree they would be able to use the system to collaborate. The results also support the presence of both types of map interfaces, two-dimensional and three-dimensional, as they are objectively better suited for different tasks; users, subjectively, affirmed preference for both of them, depending on the task at hand.A Realidade Virtual (RV) tem demonstrado ter várias aplicações interessantes ao longo das últimas décadas. Faz parte de múltiplas indústrias, tais como entertenimento, educação, turismo, gestão de crises, entre outras. Muitas delas usam a tecnologia num contexto colaborativo. Nesta tese é apresentado o design, desenvolvimento e avaliação de um sistema multiutilizador de RV, dedicado ao uso colaborativo durante um cenário de crise baseado num fogo real. É também implementada uma interface dual-map que visualiza informação geográfica, providenciando duas vistas (2D e 3D) sobre a região e dados relevantes ao cenário descrito. Perceber como podem as pessoas colaborar em RV, testar qual a interface preferida e quais os tipos de mecanismos de notificação preferíveis são os objectivos principais desta tese. O Ambiente Virtual (AV) apresenta informação geo-referenciada relevante, como estradas, povoações, veículos e o próprio incêndio, através da interface dual. Utilizadores podem partilhar o ambiente e, simultaneamente, usar as ferramentas disponíveis para interagir com os mapas e comunicar entre si, enquanto controlam o progresso do incêndio para melhor entender como se propaga. Ações como desenhar, medir distâncias, direcionar veículos e notificar outros utilizadores estão disponíveis. Utilizadores podem também propor ações que serão aceites ou recusadas. Dezoito pessoas fizeram parte do estudo de utilizador para avaliar a aplicação. Os participantes executaram múltiplas tarefas, usando as ferramentas disponíveis, enquanto partilhavam o mesmo AV que o investigador. Após análise dos dados gerados, é possível afirmar que a maioria dos participantes consideram que seriam capazes de usar o sistema para colaborar. Os resultados também suportam a presença de ambos os tipos de mapas, 2D e 3D, pois ambos são objectivamente melhores para tarefas distintas; participantes, subjectivamente, afirmam preferência por ambas, dependendo da tarefa a executar

3D terrain generation using neural networks

With the increase in computation power, coupled with the advancements in the field in the form of GANs and cGANs, Neural Networks have become an attractive proposition for content generation. This opened opportunities for Procedural Content Generation algorithms (PCG) to tap Neural Networks generative power to create tools that allow developers to remove part of creative and developmental burden imposed throughout the gaming industry, be it from investors looking for a return on their investment and from consumers that want more and better content, fast. This dissertation sets out to develop a PCG mixed-initiative tool, leveraging cGANs, to create authored 3D terrains, allowing users to directly influence the resulting generated content without the need for formal training on terrain generation or complex interactions with the tool to influence the generative output, as opposed to state of the art generative algorithms that only allow for random content generation or are needlessly complex. Testing done to 113 people online, as well as in-person testing done to 30 people, revealed that it is indeed possible to develop a tool that allows users from any level of terrain creation knowledge, and minimal tool training, to easily create a 3D terrain that is more realistic looking than those generated by state-of-the-art solutions such as Perlin Noise.Com o aumento do poder de computação, juntamente com os avanços neste campo na forma de GANs e cGANs, as Redes Neurais tornaram-se numa proposta atrativa para a geração de conteúdos. Graças a estes avanços, abriram-se oportunidades para os algoritmos de Geração de Conteúdos Procedimentais(PCG) explorarem o poder generativo das Redes Neurais para a criação de ferramentas que permitam aos programadores remover parte da carga criativa e de desenvolvimento imposta em toda a indústria dos jogos, seja por parte dos investidores que procuram um retorno do seu investimento ou por parte dos consumidores que querem mais e melhor conteúdo, o mais rápido possível. Esta dissertação pretende desenvolver uma ferramenta de iniciativa mista PCG, alavancando cGANs, para criar terrenos 3D cocriados, permitindo aos utilizadores influenciarem diretamente o conteúdo gerado sem necessidade de terem formação formal sobre a criação de terrenos 3D ou interações complexas com a ferramenta para influenciar a produção generativa, opondo-se assim a algoritmos generativos comummente utilizados, que apenas permitem a geração de conteúdo aleatório ou que são desnecessariamente complexos. Um conjunto de testes feitos a 113 pessoas online e a 30 pessoas presencialmente, revelaram que é de facto possível desenvolver uma ferramenta que permita aos utilizadores, de qualquer nível de conhecimento sobre criação de terrenos, e com uma formação mínima na ferramenta, criar um terreno 3D mais realista do que os terrenos gerados a partir da solução de estado da arte, como o Perlin Noise, e de uma forma fácil

Context-consistent generation of indoor virtual environments based on geometry constraints

In this paper, we propose a system that can automatically generate immersive and interactive virtual reality (VR) scenes by taking real-world geometric constraints into account. Our system can not only help users avoid real-world obstacles in virtual reality experiences, but also provide context-consistent contents to preserve their sense of presence. To do so, our system first identifies the positions and bounding boxes of scene objects as well as a set of interactive planes from 3D scans. Then context-compatible virtual objects that have similar geometric properties to the real ones can be automatically selected and placed into the virtual scene, based on learned object association relations and layout patterns from large amounts of indoor scene configurations. We regard virtual object replacement as a combinatorial optimization problem, considering both geometric and contextual consistency constraints. Quantitative and qualitative results show that our system can generate plausible interactive virtual scenes that highly resemble real environments, and have the ability to keep the sense of presence for users in their VR experiences

Immersive analytics with abstract 3D visualizations: A survey

After a long period of scepticism, more and more publications describe basic research but also practical approaches to how abstract data can be presented in immersive environments for effective and efficient data understanding. Central aspects of this important research question in immersive analytics research are concerned with the use of 3D for visualization, the embedding in the immersive space, the combination with spatial data, suitable interaction paradigms and the evaluation of use cases. We provide a characterization that facilitates the comparison and categorization of published works and present a survey of publications that gives an overview of the state of the art, current trends, and gaps and challenges in current research

Real-time transition texture synthesis for terrains.

Depicting the transitions where differing material textures meet on a terrain surface presents a particularly unique set of challenges in the field of real-time rendering. Natural landscapes are inherently irregular and composed of complex interactions between many different material types of effectively endless detail and variation. Although consumer grade graphics hardware is becoming ever increasingly powerful with each successive generation, terrain texturing remains a trade-off between realism and the computational resources available. Technological constraints aside, there is still the challenge of generating the texture resources to represent terrain surfaces which can often span many hundreds or even thousands of square kilometres. To produce such textures by hand is often impractical when operating on a restricted budget of time and funding. This thesis presents two novel algorithms for generating texture transitions in realtime using automated processes. The first algorithm, Feature-Based Probability Blending (FBPB), automates the task of generating transitions between material textures containing salient features. As such features protrude through the terrain surface FBPB ensures that the topography of these features is maintained at transitions in a realistic manner. The transitions themselves are generated using a probabilistic process that also dynamically adds wear and tear to introduce high frequency detail and irregularity at the transition contour. The second algorithm, Dynamic Patch Transitions (DPT), extends FBPB by applying the probabilistic transition approach to material textures that contain no salient features. By breaking up texture space into a series of layered patches that are either rendered or discarded on a probabilistic basis, the contour of the transition is greatly increased in resolution and irregularity. When used in conjunction with high frequency detail techniques, such as alpha masking, DPT is capable of producing endless, detailed, irregular transitions without the need for artistic input