Haptic Interaction with 3D oriented point clouds on the GPU

Real-time point-based rendering and interaction with virtual objects is gaining popularity and importance as di�erent haptic devices and technologies increasingly provide the basis for realistic interaction. Haptic Interaction is being used for a wide range of applications such as medical training, remote robot operators, tactile displays and video games. Virtual object visualization and interaction using haptic devices is the main focus; this process involves several steps such as: Data Acquisition, Graphic Rendering, Haptic Interaction and Data Modi�cation. This work presents a framework for Haptic Interaction using the GPU as a hardware accelerator, and includes an approach for enabling the modi�cation of data during interaction. The results demonstrate the limits and capabilities of these techniques in the context of volume rendering for haptic applications. Also, the use of dynamic parallelism as a technique to scale the number of threads needed from the accelerator according to the interaction requirements is studied allowing the editing of data sets of up to one million points at interactive haptic frame rates

The State of the Art of Spatial Interfaces for 3D Visualization

International audienceWe survey the state of the art of spatial interfaces for 3D visualization. Interaction techniques are crucial to data visualization processes and the visualization research community has been calling for more research on interaction for years. Yet, research papers focusing on interaction techniques, in particular for 3D visualization purposes, are not always published in visualization venues, sometimes making it challenging to synthesize the latest interaction and visualization results. We therefore introduce a taxonomy of interaction technique for 3D visualization. The taxonomy is organized along two axes: the primary source of input on the one hand and the visualization task they support on the other hand. Surveying the state of the art allows us to highlight specific challenges and missed opportunities for research in 3D visualization. In particular, we call for additional research in: (1) controlling 3D visualization widgets to help scientists better understand their data, (2) 3D interaction techniques for dissemination, which are under-explored yet show great promise for helping museum and science centers in their mission to share recent knowledge, and (3) developing new measures that move beyond traditional time and errors metrics for evaluating visualizations that include spatial interaction

Enhancing detailed haptic relief for real-time interaction

The present document exposes a different approach for haptic rendering, defined as the simulation of force interactions to reproduce the sensation of surface relief in dense models. Current research shows open issues in timely haptic interaction involving large meshes, with several problems affecting performance and fidelity, and without a dominant technique to treat these issues properly. Relying in pure geometric collisions when rendering highly dense mesh models (hundreds of thousands of triangles) sensibly degrades haptic rates due to the sheer number of collisions that must be tracked between the mesh's faces and a haptic probe. Several bottlenecks were identified in order to enhance haptic performance: software architecture and data structures, collision detection, and accurate rendering of surface relief. To account for overall software architecture and data structures, it was derived a complete component framework for transforming standalone VR applications into full-fledged multi-threaded Collaborative Virtual Reality Environments (CVREs), after characterizing existing implementations into a feature-rich superset. Enhancements include: a scalable arbitrated peer-to-peer topology for scene sharing; multi-threaded components for graphics rendering, user interaction and network communications; a collaborative user interface model for session handling; and interchangeable user roles with multi-camera perspectives, avatar awareness and shared annotations. We validate the framework by converting the existing ALICE VR Navigator into a complete CVRE, showing good performance in collaborative manipulation of complex models. To specifically address collision detection computation, we derive a conformal algebra treatment for collisions among points, segments, areas, and volumes, based on collision detection in conformal R{4,1} (5D) space, and implemented in GPU for faster parallel queries. Results show orders of magnitude time reductions in collisions computations, allowing interactive rates. Finally, the main core of the research is the haptic rendering of surface mesostructure in large meshes. Initially, a method for surface haptic rendering was proposed, using image-based Hybrid Rugosity Mesostructures (HRMs) of per-face heightfield displacements and normalmaps layered on top of a simpler mesh, adding greater surface detail than actually present. Haptic perception is achieved modulating the haptic probe's force response using the HRM coat. A usability testbed framework was built to measure experimental performance with a common set tests, meshes and HRMs. Trial results show the goodness of the proposed technique, rendering accurate 3D surface detail at high sampling rates. This local per-face method is extended into a fast global approach for haptic rendering, building a mesostructure-based atlas of depth/normal textures (HyRMA), computed out of surface differences of the same mesh object at two different resolutions: original and simplified. For each triangle in the simplified mesh, an irregular prism is considered defined by the triangle's vertices and their normals. This prism completely covers the original mesh relief over the triangle. Depth distances and surfaces normals within each prism are warped from object volume space to orthogonal tangent space, by means of a novel and fast method for computing barycentric coordinates at the prism, and storing normals and relief in a sorted atlas. Haptic rendering is effected by colliding the probe against the atlas, and effecting a modulated force response at the haptic probe. The method is validated numerically, statistically and perceptually in user testing controlled trials, achieving accurate haptic sensation of large meshes' fine features at interactive rendering rates, with some minute loss of mesostructure detail.En aquesta tesi es presenta un novedós enfocament per a la percepció hàptica del relleu de models virtuals complexes mitjançant la simulació de les forces d'interacció entre la superfície i un element de contacte. La proposta contribueix a l'estat de l'art de la recerca en aquesta àrea incrementant l'eficiència i la fidelitat de la interacció hàptica amb grans malles de triangles. La detecció de col·lisions amb malles denses (centenars de milers de triangles) limita la velocitat de resposta hàptica degut al gran nombre d'avaluacions d'intersecció cara-dispositiu hàptic que s'han de realitzar. Es van identificar diferents alternatives per a incrementar el rendiment hàptic: arquitectures de software i estructures de dades específiques, algorismes de detecció de col·lisions i reproducció hàptica de relleu superficial. En aquesta tesi es presenten contribucions en alguns d'aquests aspectes. S'ha proposat una estructura completa de components per a transformar aplicacions de Realitat Virtual en Ambients Col·laboratius de Realitat Virtual (CRVEs) multithread en xarxa. L'arquitectura proposada inclou: una topologia escalable punt a punt per a compartir escenes; components multithread per a visualització gràfica, interacció amb usuaris i comunicació en xarxa; un model d'interfície d'usuari col·laboratiu per a la gestió de sessions; i rols intercanviables de l'usuari amb perspectives de múltiples càmeres, presència d'avatars i anotacions compartides. L'estructura s'ha validat convertint el navegador ALICE en un CVRE completament funcional, mostrant un bon rendiment en la manipulació col·laborativa de models complexes. Per a incrementar l'eficiència del càlcul de col·lisions, s'ha proposat un algorisme que treballa en un espai conforme R{4,1} (5D) que permet detectar col·lisions entre punts, segments, triangles i volums. Aquest algorisme s'ha implementat en GPU per obtenir una execució paral·lela més ràpida. Els resultats mostren reduccions en el temps de càlcul de col·lisions permetent interactivitat. Per a la percepció hàptica de malles complexes que modelen objectes rugosos, s'han proposat diferents algorismes i estructures de dades. Les denominades Mesoestructures Híbrides de Rugositat (HRM) permeten substituir els detalls geomètrics d'una cara (rugositats) per dues textures: de normals i d'alçades. La percepció hàptica s'aconsegueix modulant la força de resposta entre el dispositiu hàptic i la HRM. Els tests per avaluar experimentalment l'eficiència del càlcul de col·lisions i la percepció hàptica utilitzant HRM respecte a modelar les rugositats amb geometria, van mostrar que la tècnica proposada va ser encertada, permetent percebre detalls 3D correctes a altes tases de mostreig. El mètode es va estendre per a representar rugositats d'objectes. Es proposa substituir l'objecte per un model simplificat i un atles de mesoestructures en el que s'usen textures de normals i de relleus (HyRMA). Aquest atles s'obté a partir de la diferència en el detall de la superfície entre dos malles del mateix objecte: l'original i la simplificada. A partir d'un triangle de la malla simplificada es construeix un prisma, definit pels vèrtexs del triangle i les seves normals, que engloba el relleu de la malla original sobre el triangle. Les alçades i normals dins del prisma es transformen des de l'espai de volum a l'espai ortogonal tangent, amb mètode novedós i eficient que calcula les coordenades baricèntriques relatives al prisma, per a guardar el mapa de textures transformat en un atles ordenat. La percepció hàptica s'assoleix detectant les col·lisions entre el dispositiu hàptic i l'atles, i modulant la força de resposta d'acord al resultat de la col·lisió. El mètode s'ha validat numèricament, estadística i perceptual en tests amb usuaris, aconseguint una correcta i interactiva sensació tàctil dels objectes simulats mitjançant la mesoestructura de les mallesEn esta tesis se presenta un enfoque novedoso para la percepción háptica del relieve de modelos virtuales complejos mediante la simulación de las fuerzas de interacción entre la superficie y un elemento de contacto. La propuesta contribuye al estado del arte de investigación en este área incrementando la eficiencia y fidelidad de interacción háptica con grandes mallas de triángulos. La detección de colisiones con mallas geométricas densas (cientos de miles de triángulos) limita la velocidad de respuesta háptica debido al elevado número de evaluaciones de intersección cara-dispositivo háptico que deben realizarse. Se identificaron diferentes alternativas para incrementar el rendimiento háptico: arquitecturas de software y estructuras de datos específicas, algoritmos de detección de colisiones y reproducción háptica de relieve superficial. En esta tesis se presentan contribuciones en algunos de estos aspectos. Se ha propuesto una estructura completa de componentes para transformar aplicaciones aisladas de Realidad Virtual en Ambientes Colaborativos de Realidad Virtual (CRVEs) multithread en red. La arquitectura propuesta incluye: una topología escalable punto a punto para compartir escenas; componentes multithread para visualización gráfica, interacción con usuarios y comunicación en red; un modelo de interfaz de usuario colaborativo para la gestión de sesiones; y roles intercambiables del usuario con perspectivas de múltiples cámaras, presencia de avatares y anotaciones compartidas. La estructura se ha validado convirtiendo el navegador ALICE en un CVRE completamente funcional, mostrando un buen rendimiento en la manipulación colaborativa de modelos complejos. Para incrementar la eficiencia del cálculo de colisiones, se ha propuesto un algoritmo que trabaja en un espacio conforme R4,1 (5D) que permite detectar colisiones entre puntos, segmentos, triángulos y volúmenes. Este algoritmo se ha implementado en GPU a efectos de obtener una ejecución paralelamás rápida. Los resultadosmuestran reducciones en el tiempo de cálculo de colisiones permitiendo respuesta interactiva. Para la percepción háptica de mallas complejas que modelan objetos rugosos, se han propuesto diferentes algoritmos y estructuras de datos. Las denominadasMesoestructuras Híbridas de Rugosidad (HRM) permiten substituir los detalles geométricos de una cara (rugosidades) por una textura de normales y otra de alturas. La percepción háptica se consigue modulando la fuerza de respuesta entre el dispositivo háptico y la HRM. Los tests realizados para evaluar experimentalmente la eficiencia del cálculo de colisiones y la percepción háptica utilizando HRM respecto a modelar las rugosidades con geometría, mostraron que la técnica propuesta fue acertada, permitiendo percibir detalles 3D correctos a altas tasas de muestreo. Este método anterior es extendido a un procedimiento global para representar rugosidades de objetos. Para hacerlo se propone sustituir el objeto por un modelo simplificado y un atlas de mesostructuras usando texturas de normales y relieves (HyRMA). Este atlas se obtiene de la diferencia en detalle de superficie entre dos mallas del mismo objeto: la original y la simplificada. A partir de un triángulo de la malla simplificada se construye un prisma definido por los vértices del triángulo a lo largo de sus normales, que engloba completamente el relieve de la malla original sobre este triángulo. Las alturas y normales dentro de cada prisma se transforman del espacio de volumen al espacio ortoganal tangente, usando un método novedoso y eficiente que calcula las coordenadas baricéntricas relativas a cada prisma para guardar el mapa de texturas transformado en un atlas ordenado. La percepción háptica se consigue detectando directamente las colisiones entre el dispositivo háptico y el atlas, y modulando la fuerza de respuesta de acuerdo al resultado de la colisión. El procedmiento se ha validado numérica, estadística y perceptualmente en ensayos con usuarios, consiguiendo a tasas interactivas la correcta sensación táctil de los objetos simulados mediante la mesoestructura de las mallas, con alguna pérdida muy puntual de detall

VR Technologies in Cultural Heritage

This open access book constitutes the refereed proceedings of the First International Conference on VR Technologies in Cultural Heritage, VRTCH 2018, held in Brasov, Romania in May 2018. The 13 revised full papers along with the 5 short papers presented were carefully reviewed and selected from 21 submissions. The papers of this volume are organized in topical sections on data acquisition and modelling, visualization methods / audio, sensors and actuators, data management, restoration and digitization, cultural tourism

Haptic Media Scenes

The aim of this thesis is to apply new media phenomenological and enactive embodied cognition approaches to explain the role of haptic sensitivity and communication in personal computer environments for productivity. Prior theory has given little attention to the role of haptic senses in influencing cognitive processes, and do not frame the richness of haptic communication in interaction design—as haptic interactivity in HCI has historically tended to be designed and analyzed from a perspective on communication as transmissions, sending and receiving haptic signals. The haptic sense may not only mediate contact confirmation and affirmation, but also rich semiotic and affective messages—yet this is a strong contrast between this inherent ability of haptic perception, and current day support for such haptic communication interfaces. I therefore ask: How do the haptic senses (touch and proprioception) impact our cognitive faculty when mediated through digital and sensor technologies? How may these insights be employed in interface design to facilitate rich haptic communication? To answer these questions, I use theoretical close readings that embrace two research fields, new media phenomenology and enactive embodied cognition. The theoretical discussion is supported by neuroscientific evidence, and tested empirically through case studies centered on digital art. I use these insights to develop the concept of the haptic figura, an analytical tool to frame the communicative qualities of haptic media. The concept gauges rich machine- mediated haptic interactivity and communication in systems with a material solution supporting active haptic perception, and the mediation of semiotic and affective messages that are understood and felt. As such the concept may function as a design tool for developers, but also for media critics evaluating haptic media. The tool is used to frame a discussion on opportunities and shortcomings of haptic interfaces for productivity, differentiating between media systems for the hand and the full body. The significance of this investigation is demonstrating that haptic communication is an underutilized element in personal computer environments for productivity and providing an analytical framework for a more nuanced understanding of haptic communication as enabling the mediation of a range of semiotic and affective messages, beyond notification and confirmation interactivity

VR Technologies in Cultural Heritage

This open access book constitutes the refereed proceedings of the First International Conference on VR Technologies in Cultural Heritage, VRTCH 2018, held in Brasov, Romania in May 2018. The 13 revised full papers along with the 5 short papers presented were carefully reviewed and selected from 21 submissions. The papers of this volume are organized in topical sections on data acquisition and modelling, visualization methods / audio, sensors and actuators, data management, restoration and digitization, cultural tourism

Interfaces for human-centered production and use of computer graphics assets

L'abstract è presente nell'allegato / the abstract is in the attachmen

The Application of Mixed Reality Within Civil Nuclear Manufacturing and Operational Environments

This thesis documents the design and application of Mixed Reality (MR) within a nuclear manufacturing cell through the creation of a Digitally Assisted Assembly Cell (DAAC). The DAAC is a proof of concept system, combining full body tracking within a room sized environment and bi-directional feedback mechanism to allow communication between users within the Virtual Environment (VE) and a manufacturing cell. This allows for training, remote assistance, delivery of work instructions, and data capture within a manufacturing cell. The research underpinning the DAAC encompasses four main areas; the nuclear industry, Virtual Reality (VR) and MR technology, MR within manufacturing, and finally the 4 th Industrial Revolution (IR4.0). Using an array of Kinect sensors, the DAAC was designed to capture user movements within a real manufacturing cell, which can be transferred in real time to a VE, creating a digital twin of the real cell. Users can interact with each other via digital assets and laser pointers projected into the cell, accompanied by a built-in Voice over Internet Protocol (VoIP) system. This allows for the capture of implicit knowledge from operators within the real manufacturing cell, as well as transfer of that knowledge to future operators. Additionally, users can connect to the VE from anywhere in the world. In this way, experts are able to communicate with the users in the real manufacturing cell and assist with their training. The human tracking data fills an identified gap in the IR4.0 network of Cyber Physical System (CPS), and could allow for future optimisations within manufacturing systems, Material Resource Planning (MRP) and Enterprise Resource Planning (ERP). This project is a demonstration of how MR could prove valuable within nuclear manufacture. The DAAC is designed to be low cost. It is hoped this will allow for its use by groups who have traditionally been priced out of MR technology. This could help Small to Medium Enterprises (SMEs) close the double digital divide between themselves and larger global corporations. For larger corporations it offers the benefit of being low cost, and, is consequently, easier to roll out across the value chain. Skills developed in one area can also be transferred to others across the internet, as users from one manufacturing cell can watch and communicate with those in another. However, as a proof of concept, the DAAC is at Technology Readiness Level (TRL) five or six and, prior to its wider application, further testing is required to asses and improve the technology. The work was patented in both the UK (S. R EDDISH et al., 2017a), the US (S. R EDDISH et al., 2017b) and China (S. R EDDISH et al., 2017c). The patents are owned by Rolls-Royce and cover the methods of bi-directional feedback from which users can interact from the digital to the real and vice versa. Stephen Reddish Mixed Mode Realities in Nuclear Manufacturing Key words: Mixed Mode Reality, Virtual Reality, Augmented Reality, Nuclear, Manufacture, Digital Twin, Cyber Physical Syste

Computational interaction techniques for 3D selection, manipulation and navigation in immersive VR

3D interaction provides a natural interplay for HCI. Many techniques involving diverse sets of hardware and software components have been proposed, which has generated an explosion of Interaction Techniques (ITes), Interactive Tasks (ITas) and input devices, increasing thus the heterogeneity of tools in 3D User Interfaces (3DUIs). Moreover, most of those techniques are based on general formulations that fail in fully exploiting human capabilities for interaction. This is because while 3D interaction enables naturalness, it also produces complexity and limitations when using 3DUIs. In this thesis, we aim to generate approaches that better exploit the high potential human capabilities for interaction by combining human factors, mathematical formalizations and computational methods. Our approach is focussed on the exploration of the close coupling between specific ITes and ITas while addressing common issues of 3D interactions. We specifically focused on the stages of interaction within Basic Interaction Tasks (BITas) i.e., data input, manipulation, navigation and selection. Common limitations of these tasks are: (1) the complexity of mapping generation for input devices, (2) fatigue in mid-air object manipulation, (3) space constraints in VR navigation; and (4) low accuracy in 3D mid-air selection. Along with two chapters of introduction and background, this thesis presents five main works. Chapter 3 focusses on the design of mid-air gesture mappings based on human tacit knowledge. Chapter 4 presents a solution to address user fatigue in mid-air object manipulation. Chapter 5 is focused on addressing space limitations in VR navigation. Chapter 6 describes an analysis and a correction method to address Drift effects involved in scale-adaptive VR navigation; and Chapter 7 presents a hybrid technique 3D/2D that allows for precise selection of virtual objects in highly dense environments (e.g., point clouds). Finally, we conclude discussing how the contributions obtained from this exploration, provide techniques and guidelines to design more natural 3DUIs