Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Current `dry lab' surgical phantom simulators are a valuable tool for surgeons which allows them to improve their dexterity and skill with surgical instruments. These phantoms mimic the haptic and shape of organs of interest, but lack a realistic visual appearance. In this work, we present an innovative application in which representations learned from real intraoperative endoscopic sequences are transferred to a surgical phantom scenario. The term hyperrealism is introduced in this field, which we regard as a novel subform of surgical augmented reality for approaches that involve real-time object transfigurations. For related tasks in the computer vision community, unpaired cycle-consistent Generative Adversarial Networks (GANs) have shown excellent results on still RGB images. Though, application of this approach to continuous video frames can result in flickering, which turned out to be especially prominent for this application. Therefore, we propose an extension of cycle-consistent GANs, named tempCycleGAN, to improve temporal consistency.The novel method is evaluated on captures of a silicone phantom for training endoscopic reconstructive mitral valve procedures. Synthesized videos show highly realistic results with regard to 1) replacement of the silicone appearance of the phantom valve by intraoperative tissue texture, while 2) explicitly keeping crucial features in the scene, such as instruments, sutures and prostheses. Compared to the original CycleGAN approach, tempCycleGAN efficiently removes flickering between frames. The overall approach is expected to change the future design of surgical training simulators since the generated sequences clearly demonstrate the feasibility to enable a considerably more realistic training experience for minimally-invasive procedures.Comment: 8 pages, accepted at MICCAI 2018, supplemental material at https://youtu.be/qugAYpK-Z4

Multiscale multiagent architecture validation by virtual instruments in molecular dynamics experiments

International audienceA multiagent architecture is proposed in order to build a physics laboratory in virtual reality. Thermodynamics experiments are used for its validation. Classical numerical resolution of thermodynamics problems comes up against the number and variability of boundary conditions. Based on molecular dynamics, a multiagent approach is proposed, resting upon agent spatial and temporal autonomy. This approach grants each particle a capacity to identify its environment using both its own clock and perceptive area. Individual molecular properties are injected into thermal and mechanical models, and macroscopic gas behaviors can be detected and quantified by 3D virtual instruments, created in order to involve the user into the simulation. In order to assess its ability to simulate thermodynamic experiments, our method is applied to classical situations, such as the Joule-Gay Lussac experiment or the maxwellian relaxation in a hard-sphere gas. The simulated gas behavior is in good agreement with theoretical results for gases without interaction. Taking into account the volume of the molecules, our method also allows to quantify the mean free path and the average collision time for Neon and Xenon hard-sphere gases at equilibrium. Dynamic speed relaxation from uniform to maxwellian distribution is simulated successfully, and molecular covolume are also measured with such virtual gases

Educational Uses of Augmented Reality (AR): Experiences in Educational Science

Augmented Reality (AR) is an emerging technology that is gaining greater influence on teaching every day. AR, together with mobile technology, is defined as one of the most efficient pairs for supporting significant and ubiquitous learning. Purpose of the study: the Instructional Material Motivational Survey (IMMS), by Keller, was used to determine the degree of motivation possessed by the Pedagogy students on the utilization of the notes enriched with AR in the classroom, available for their didactic use through mobile devices. Methods: through an app designed for the courses Education Technology (ET) and Information and Communication Technologies (ICT) Applied to Education, the motivation gained when participating in this experience, and how it influences the improvement of academic performance, was evaluated. Results and conclusions: the most notable main result was finding a strong relationship between the motivation of the students when using the enriched notes and the increase of performance in the academic subject where it was used. Likewise, it was proved that the use of Augmented Reality benefited the learning process itself

Virtual reality training and assessment in laparoscopic rectum surgery

Background: Virtual-reality (VR) based simulation techniques offer an efficient and low cost alternative to conventional surgery training. This article describes a VR training and assessment system in laparoscopic rectum surgery. Methods: To give a realistic visual performance of interaction between membrane tissue and surgery tools, a generalized cylinder based collision detection and a multi-layer mass-spring model are presented. A dynamic assessment model is also designed for hierarchy training evaluation. Results: With this simulator, trainees can operate on the virtual rectum with both visual and haptic sensation feedback simultaneously. The system also offers surgeons instructions in real time when improper manipulation happens. The simulator has been tested and evaluated by ten subjects. Conclusions: This prototype system has been verified by colorectal surgeons through a pilot study. They believe the visual performance and the tactile feedback are realistic. It exhibits the potential to effectively improve the surgical skills of trainee surgeons and significantly shorten their learning curve. © 2014 John Wiley & Sons, Ltd

3D sound for simulation of arthroscopic surgery

Arthroscopic surgery offers many advantages compared to traditional surgery. Nevertheless, the required skills to practice this kind of surgery need specific training. Surgery simulators are used to train surgeon apprentices to practice specific gestures. In this paper, we present a study showing the contribution of 3D sound in assisting the triangulation gesture in arthroscopic surgery simulation. This ability refers to the capacity of the subject to manipulate the instruments while having a modified and limited view provided by the video camera of the simulator. Our approach, based on the use of 3D sound metaphors, provides interaction cues to the subjects about the real position of the instrument. The paper reports a performance evaluation study based on the perception of 3D sound integrated in the process of training of surgical task. Despite the fact that 3D sound cueing was not shown useful to all subjects in terms of execution time, the results of the study revealed that the majority of subjects who participated to the experiment confirmed the added value of 3D sound in terms of ease of use

Gulliver project: performers and visitors

This paper discusses two projects in our research environment. The Gulliver project, an ambitious project conceived by some artists connected to our research efforts, and the Aveiro-project, as well ambitious, but with goals that can be achieved beause of technological developments, rather than be dependent on artistic and 'political' (read: financial) sources. Both projects are on virtual and augmented reality. The main goal is to design inhabited environments, where 'inhabited' refers to autonomous agents and agents that represent humans, realtime or off-line, visiting the virtual environment and interacting with other agents. The Gulliver environment has been designed by two artists: Matjaz Stuk and Alena Hudcovicova. The Aveiro project is a research effort of a group of researchers trying to design models of intelligence and interaction underlying the behavior of (groups of) agents inhabiting virtual worlds. In this paper we survey the current state of both projects and we discuss current and future attempts to have music performances by virtual and real performers in these environments

Evaluating LAB@FUTURE, a collaborative e-learning Laboratory experiments platform

This paper presents Lab@Future, an advanced e-learning platform that uses novel Information and Communication Technologies to support and expand laboratory teaching practices. For this purpose, Lab@Future uses real and computer generated objects that are interfaced using mechatronic systems, augmented reality, mobile technologies and 3D multi user environments. The main aim is to develop and demonstrate technological support for practical experiments in the following focused disciplines namely: Fluid Dynamics - Science subject in Germany, Geometry - Mathematics subject in Austria, History and Environmental Awareness – Arts and Humanities subjects in Greece and Slovenia. In order to pedagogically enhance the design and functional aspects of this e-learning technology, we are investigating the dialogical operationalisation of learning theories so as to leverage our understanding of teaching and learning practices in the targeted context of deployment. To be able to evaluate the lab@future system in its entire complexity an evaluation methodology including several phases has been developed, performing formative as well as summative evaluations

Entanglement and Quantum Nonlocality Demystified

Quantum nonlocality is presented often as the most remarkable and inexplicable phenomenon known to modern science which was confirmed in the experiments proving the violation of Bell Inequalities (BI). It has been known already for a long time that the probabilistic models used to prove BI for spin polarization correlation experiments (SPCE) are incompatible with the experimental protocols of SPCE. In particular these models use a common probability space together with joint probability distributions for various incompatible coincidence experiments and/or conditional independence (Bell's locality). Strangely enough these results are not known or simply neglected. Therefore so called Bell's or quantum nonlocality has nothing to do with the common notion of the non-locality and it should be rather called quantum non-Kolmogorovness or quantum contextuality. We quickly explain the true meaning of various Bell's locality assumptions and show that if local variables describing the measuring instruments are correctly taken into consideration then BI can no longer be proven. In order to demystify even further the entanglement and quantum long range correlations we give an example of a macroscopic entangled "two qubit state". Namely we show that one can prepare two macroscopic systems in such a way that simple realizable local experiments on these systems violate BI. Of course we do not question the usefulness of the long range correlations characterizing the entangled physical systems in the domain of Quantum Information. However one should not forget that the anti-correlations cannot be perfect, that the wave function should not be treated as an attribute of the individual quantum system which can be change instantaneously and that the unperformed experiments have no results.Comment: Corrected typos and omissions, added one reference, eliminated jumps in the reference list and numbering adjuste

The LAB@FUTURE Project - Moving Towards the Future of E-Learning

This paper presents Lab@Future, an advanced e-learning platform that uses novel Information and Communication Technologies to support and expand laboratory teaching practices. For this purpose, Lab@Future uses real and computer-generated objects that are interfaced using mechatronic systems, augmented reality, mobile technologies and 3D multi user environments. The main aim is to develop and demonstrate technological support for practical experiments in the following focused subjects namely: Fluid Dynamics - Science subject in Germany, Geometry - Mathematics subject in Austria, History and Environmental Awareness – Arts and Humanities subjects in Greece and Slovenia. In order to pedagogically enhance the design and functional aspects of this e-learning technology, we are investigating the dialogical operationalisation of learning theories so as to leverage our understanding of teaching and learning practices in the targeted context of deployment