Augmented reality with tangible Auto-Fabricated models for molecular biology applications

Abstract: The evolving technology of computer auto-fabrication ("3-D printing") now makes it possible to produce physical models for complex biological molecules and assemblies. We report on an application that demonstrates the use of auto-fabricated tangible models and augmented reality for research and education in molecular biology, and for enhancing the scientific environment for collaboration and exploration. We have adapted an augmented reality system to allows virtual 3-D representations (generated by the Python Molecular Viewer) to be overlaid onto a tangible molecular model. Users can easily change the overlaid information, switching between different representations of the molecule, displays of molecular properties such as electrostatics, or dynamic information. The physical model provides a powerful, intuitive interface for manipulating the computer models, streamlining the interface between human intent, the physical model, and the computational activity. INTRODUCTION With the prevalence of structural and genomic data, molecular biology has become a human-guided, computer-assisted endeavor. The computer assists the essential human function in two ways: in exploration of scientific data, searching for and testing scientific hypotheses; and in collaboration between two or more scientists, to share knowledge and expertise. As databases grow, as our structure and process models become more complex, and as software methods become more diverse, access and manipulation of digital information is increasingly a critical issue for research in molecular biology. Currently, exploratory research in structural molecular biology is dominated by 3-D representations via computer graphics. Collaboration, both remote and local, is aided by shared viewing of these interactive visual representations of molecular data. Yet, recent advances in the field of human-computer interfaces have not been applied to the technology used by molecular biologists --most work in biomolecular structure and genomics is performed in front of a workstation using a mouse and keyboard as input devices. The tactile and kinesthetic senses provide key perceptual cues to our ability to understand 3-D form and to perform physical manipulations, but are currently under-utilized in molecular biology. Early structure research relied heavily on physical models: Pauling used his newly-invented spacefilling models to predict the basic folding units of protein structures [1] and Watson and Crick used brass-wire molecular models to help them determine the structure of DNA [2], which reconciled decades of genetic data. These researchers "thought with their hands" to produce important scientific results. Current research in molecular biology now focuses on larger assemblies and more complex interactions, for which traditional atomic models are inadequate. Merging physical and virtual objects into an "augmented reality" (AR) environment The evolving technology of computer auto-fabrication ("3D printing") now makes it possible to produce physical models for complex molecular assemblies. In this paper we report on an application that demonstrates the use of auto-fabricated tangible models and AR for research in molecular biology to enhance the scientific environment for collaboration and exploration. The physical models are integrated into an augmented reality environment to streamline the interface between human intent, the physical model, and the computational activity. We have developed an AR system that allows virtual 3-D representations generated by our Python Molecular Viewer (PMV) [5] to be overlaid on an auto-fabricated model of the molecule. The precise registration of the virtual objects with the real world is done using the ARToolKit library developed at the University of Washington We will first describe how we create 3D tangible models of a molecular structure from a known atomic structure, then explain the integration of ARToolKit in our Python framework, and finally present some examples. DESIGN OF PHYSICAL MODELS We use PMV October 10-15

MoleculARweb: A Web Site for Chemistry and Structural Biology Education through Interactive Augmented Reality out of the Box in Commodity Devices

Augmented/virtual realities (ARs/VRs) promise to revolutionize STEM education. However, most easy-to-use tools are limited to static visualizations, which limits the approachable content, whereas more interactive and dynamic alternatives require costly hardware, preventing large-scale use and evaluation of pedagogical effects. Here, we introduce https://MoleculARweb.epfl.ch, a free, open-source web site with interactive AR webpage-based apps that work out-of-the-box in laptops, tablets, and smartphones, where students and teachers can naturally handle virtual objects to explore molecular structure, reactivity, dynamics, and interactions, covering topics from inorganic, organic, and biological chemistry. With these web apps, teachers and science communicators can develop interactive material for their lessons and hands-on activities for their students and target public, in person or online, as we exemplify. Thousands of accesses to moleculARweb attest to the ease of use; teacher feedback attests to the utility in online teaching and homework during a pandemic; and in-class plus online surveys show that users find AR engaging and useful for teaching and learning chemistry. These observations support the potential of AR in future education and show the large impact that modern web technologies have in democratizing access to digital learning tools, providing the possibility to mass-test the pedagogical effect of these technologies in STEM education.Fil: Rodríguez, Fabio Cortés. École Polytechnique Fédérale de Lausanne; Suiza. Swiss Institute of Bioinformatics; SuizaFil: Frattini, Gianfranco. Universidad Nacional de Rosario. Facultad de Ciencias Bioquímicas y Farmacéuticas; ArgentinaFil: Krapp, Lucien F.. Ecole Polytechnique Federale de Lausanne; Francia. Swiss Institute of Bioinformatics; SuizaFil: Martinez Hung, Hassan. Universidad de Oriente; VenezuelaFil: Moreno, Diego Martin. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Rosario. Instituto de Química Rosario. Universidad Nacional de Rosario. Facultad de Ciencias Bioquímicas y Farmacéuticas. Instituto de Química Rosario; ArgentinaFil: Roldán, Mariana. Provincia de Córdoba. Instituto Colbert; ArgentinaFil: Salomón, Jorge Eduardo. Provincia de Buenos Aires. Escuela de Educación Técnica Nro. 4; ArgentinaFil: Stemkoski, Lee. Adelphi University; Estados UnidosFil: Traeger, Sylvain. École Polytechnique Fédérale de Lausanne; Suiza. Swiss Institute of Bioinformatics; SuizaFil: Dal Peraro, Matteo. École Polytechnique Fédérale de Lausanne; Suiza. Swiss Institute of Bioinformatics; SuizaFil: Abriata, Luciano Andres. École Polytechnique Fédérale de Lausanne; Suiza. Swiss Institute of Bioinformatics; Suiz

A data physicalization pipeline enhanced with augmented reality

Data visualization is an indispensable methodology for interpretation of information. The key purpose of traditional data visualization methods is to convert observed records into meaningful visuals to ease the cognition of trends. This virtual, passive technique on a display o↵ers flexibility to create wide range of di↵erent visualization designs utilizing, however, only visual perception. Physical visualizations, on the other hand, enable sensations other than mere visual input, thus enhancing the experience and the impact. Although physical visualizations have some certain proven benefits over traditional visualizations, generating them is not as e↵ective and quick. In that regard, need of physical models shaped around well-defined design rules are a prerequisite. Moreover, digital construction of the designed solid models for manufacturing is the next step to be achieved. However, even for a small set of data, constructing several models becomes a discouraging and highly time consuming task. This main problem is covered in this thesis by the implementation of an authoring tool. The introduced tool alleviates the burden of physical model generation process. Predefined models under design rules are generated in accordance with both the data input and adjusted parameters by the user. Utilization of digital fabrication techniques that are nowadays becoming widespread and easy to access is the key for physicalization. In order for an ”Overview first, detail on demand” approach, an augmented reality tool is also introduced to work with designed models so as to retain the physicality while presenting more detailed information such as exact values of data points along with augmented graphics if desired

Immersive analytics for oncology patient cohorts

This thesis proposes a novel interactive immersive analytics tool and methods to interrogate the cancer patient cohort in an immersive virtual environment, namely Virtual Reality to Observe Oncology data Models (VROOM). The overall objective is to develop an immersive analytics platform, which includes a data analytics pipeline from raw gene expression data to immersive visualisation on virtual and augmented reality platforms utilising a game engine. Unity3D has been used to implement the visualisation. Work in this thesis could provide oncologists and clinicians with an interactive visualisation and visual analytics platform that helps them to drive their analysis in treatment efficacy and achieve the goal of evidence-based personalised medicine. The thesis integrates the latest discovery and development in cancer patients’ prognoses, immersive technologies, machine learning, decision support system and interactive visualisation to form an immersive analytics platform of complex genomic data. For this thesis, the experimental paradigm that will be followed is in understanding transcriptomics in cancer samples. This thesis specifically investigates gene expression data to determine the biological similarity revealed by the patient's tumour samples' transcriptomic profiles revealing the active genes in different patients. In summary, the thesis contributes to i) a novel immersive analytics platform for patient cohort data interrogation in similarity space where the similarity space is based on the patient's biological and genomic similarity; ii) an effective immersive environment optimisation design based on the usability study of exocentric and egocentric visualisation, audio and sound design optimisation; iii) an integration of trusted and familiar 2D biomedical visual analytics methods into the immersive environment; iv) novel use of the game theory as the decision-making system engine to help the analytics process, and application of the optimal transport theory in missing data imputation to ensure the preservation of data distribution; and v) case studies to showcase the real-world application of the visualisation and its effectiveness

Craniofacial Growth Series Volume 56

https://deepblue.lib.umich.edu/bitstream/2027.42/153991/1/56th volume CF growth series FINAL 02262020.pdfDescription of 56th volume CF growth series FINAL 02262020.pdf : Proceedings of the 46th Annual Moyers Symposium and 44th Moyers Presymposiu

Progress in Scientific Visualization

Visualization of observed data or simulation output is important to science and engineering. I have been particularly interested in visualizing 3-D structures, and report here my personal impressions on progress in the last 20 years in visualizing molecules, scalar fields, and vector fields and their associated flows. I have tried to keep the survey and list of references manageable, so apologize to those authors whose techniques I have not mentioned, or have described without a reference citation

AUGMENTED REALITY SYSTEMS AND USER INTERACTION TECHNIQUES FOR STEM LEARNING

Learning practices and crosscutting concepts in science, technology, engineering, andmathematics (STEM) subjects pose challenges to young learners. Without external support to foster long-term interest and scaffold learning, children might lose interest in STEM subjects. While prior research has investigated how Augmented Reality (AR) may enhance learning of scientific concepts and increase student engagement, only a few considered young children who require developmentally appropriate approaches. The primary goal of my dissertation is to design, develop, and evaluate AR learning systems to engage children (ages 5-11) with STEM experiences. Leveraging advanced computer vision, machine learning, and sensing technologies, my dissertation explores novel user interaction techniques. The proposed techniques can give learners chance to investigate STEM ideas in their own setting, what educators call contextual learning, and lower barriers for STEM learning practices. Using the systems, my research further investigates Human-Artificial Intelligence (AI) interaction—how children understand, use, and react to the intelligent systems. Specifically, there are four major objectives in my research including: (i) gathering design ideas of AR applications to promote children’s STEM learning; (ii) exploring AR user interaction techniques that utilize personally meaningful material for learning; (iii) developing and evaluating AR learning systems and learning applications; and (iv) building design implications for AR systems for education

A comparison of processing techniques for producing prototype injection moulding inserts.

This project involves the investigation of processing techniques for producing low-cost moulding inserts used in the particulate injection moulding (PIM) process. Prototype moulds were made from both additive and subtractive processes as well as a combination of the two. The general motivation for this was to reduce the entry cost of users when considering PIM. PIM cavity inserts were first made by conventional machining from a polymer block using the pocket NC desktop mill. PIM cavity inserts were also made by fused filament deposition modelling using the Tiertime UP plus 3D printer. The injection moulding trials manifested in surface finish and part removal defects. The feedstock was a titanium metal blend which is brittle in comparison to commodity polymers. That in combination with the mesoscale features, small cross-sections and complex geometries were considered the main problems. For both processing methods, fixes were identified and made to test the theory. These consisted of a blended approach that saw a combination of both the additive and subtractive processes being used. The parts produced from the three processing methods are investigated and their respective merits and issues are discussed