Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?

After being collected for patient care, Observational Health Data (OHD) can further benefit patient well-being by sustaining the development of health informatics and medical research. Vast potential is unexploited because of the fiercely private nature of patient-related data and regulations to protect it. Generative Adversarial Networks (GANs) have recently emerged as a groundbreaking way to learn generative models that produce realistic synthetic data. They have revolutionized practices in multiple domains such as self-driving cars, fraud detection, digital twin simulations in industrial sectors, and medical imaging. The digital twin concept could readily apply to modelling and quantifying disease progression. In addition, GANs posses many capabilities relevant to common problems in healthcare: lack of data, class imbalance, rare diseases, and preserving privacy. Unlocking open access to privacy-preserving OHD could be transformative for scientific research. In the midst of COVID-19, the healthcare system is facing unprecedented challenges, many of which of are data related for the reasons stated above. Considering these facts, publications concerning GAN applied to OHD seemed to be severely lacking. To uncover the reasons for this slow adoption, we broadly reviewed the published literature on the subject. Our findings show that the properties of OHD were initially challenging for the existing GAN algorithms (unlike medical imaging, for which state-of-the-art model were directly transferable) and the evaluation synthetic data lacked clear metrics. We find more publications on the subject than expected, starting slowly in 2017, and since then at an increasing rate. The difficulties of OHD remain, and we discuss issues relating to evaluation, consistency, benchmarking, data modelling, and reproducibility.Comment: 31 pages (10 in previous version), not including references and glossary, 51 in total. Inclusion of a large number of recent publications and expansion of the discussion accordingl

Research on real-time physics-based deformation for haptic-enabled medical simulation

This study developed a multiple effective visuo-haptic surgical engine to handle a variety of surgical manipulations in real-time. Soft tissue models are based on biomechanical experiment and continuum mechanics for greater accuracy. Such models will increase the realism of future training systems and the VR/AR/MR implementations for the operating room

Development of a Realistic Crowd Simulation Environment for Fine-grained Validation of People Tracking Methods

Generally, crowd datasets can be collected or generated from real or synthetic sources. Real data is generated by using infrastructure-based sensors (such as static cameras or other sensors). The use of simulation tools can significantly reduce the time required to generate scenario-specific crowd datasets, facilitate data-driven research, and next build functional machine learning models. The main goal of this work was to develop an extension of crowd simulation (named CrowdSim2) and prove its usability in the application of people-tracking algorithms. The simulator is developed using the very popular Unity 3D engine with particular emphasis on the aspects of realism in the environment, weather conditions, traffic, and the movement and models of individual agents. Finally, three methods of tracking were used to validate generated dataset: IOU-Tracker, Deep-Sort, and Deep-TAMA

Research and Education in Computational Science and Engineering

Over the past two decades the field of computational science and engineering (CSE) has penetrated both basic and applied research in academia, industry, and laboratories to advance discovery, optimize systems, support decision-makers, and educate the scientific and engineering workforce. Informed by centuries of theory and experiment, CSE performs computational experiments to answer questions that neither theory nor experiment alone is equipped to answer. CSE provides scientists and engineers of all persuasions with algorithmic inventions and software systems that transcend disciplines and scales. Carried on a wave of digital technology, CSE brings the power of parallelism to bear on troves of data. Mathematics-based advanced computing has become a prevalent means of discovery and innovation in essentially all areas of science, engineering, technology, and society; and the CSE community is at the core of this transformation. However, a combination of disruptive developments---including the architectural complexity of extreme-scale computing, the data revolution that engulfs the planet, and the specialization required to follow the applications to new frontiers---is redefining the scope and reach of the CSE endeavor. This report describes the rapid expansion of CSE and the challenges to sustaining its bold advances. The report also presents strategies and directions for CSE research and education for the next decade.Comment: Major revision, to appear in SIAM Revie

OpenSim-Based Musculoskeletal Modeling: Foundation for Interactive Obstetric Simulator

The use of mathematical and computational models to understand complex biological systems, such as the human birth process, is a rapidly growing field in medicine. These models can be used to optimize and personalize medical treatments for individual patients, enhance training, and aid in educational efforts. While recent advancements in healthcare, particularly in obstetrics, have improved care for mothers and babies, studies and government reports indicate a rising rate of maternal mortality in the United States. Despite this rising trend, there is a lack of detailed studies concerning the use of modeling and simulation to develop an interactive obstetrics simulator that can aid both practitioners and patients. This research builds upon a novel template for developing an interactive obstetric simulator and aims to replicate an onerous finite element vaginal delivery simulation with an interactive, patient-specific simulator that emphasizes musculoskeletal dynamics. The study utilizes the open-source platform of OpenSim and inverse-kinematic solutions to develop fetal and maternal musculoskeletal models and simulate birth on the musculoskeletal level

Virtual Reality Simulator for Training in Myringotomy with Tube Placement

Myringotomy refers to a surgical incision in the eardrum, and it is often followed by ventilation tube placement to treat middle-ear infections. The procedure is difficult to learn; hence, the objectives of this work were to develop a virtual-reality training simulator, assess its face and content validity, and implement quantitative performance metrics and assess construct validity. A commercial digital gaming engine (Unity3D) was used to implement the simulator with support for 3D visualization of digital ear models and support for major surgical tasks. A haptic arm co-located with the stereo scene was used to manipulate virtual surgical tools and to provide force feedback. A questionnaire was developed with 14 face validity questions focusing on realism and 6 content validity questions focusing on training potential. Twelve participants from the Department of Otolaryngology were recruited for the study. Responses to 12 of the 14 face validity questions were positive. One concern was with contact modeling related to tube insertion into the eardrum, and the second was with movement of the blade and forceps. The former could be resolved by using a higher resolution digital model for the eardrum to improve contact localization. The latter could be resolved by using a higher fidelity haptic device. With regard to content validity, 64% of the responses were positive, 21% were neutral, and 15% were negative. In the final phase of this work, automated performance metrics were programmed and a construct validity study was conducted with 11 participants: 4 senior Otolaryngology consultants and 7 junior Otolaryngology residents. Each participant performed 10 procedures on the simulator and metrics were automatically collected. Senior Otolaryngologists took significantly less time to completion compared to junior residents. Junior residents had 2.8 times more errors as compared to experienced surgeons. The senior surgeons also had significantly longer incision lengths, more accurate incision angles, and lower magnification keeping both the umbo and annulus in view. All metrics were able to discriminate senior Otolaryngologists from junior residents with a significance of p \u3c 0.002. The simulator has sufficient realism, training potential and performance discrimination ability to warrant a more resource intensive skills transference study

Effectiveness analysis of traditional and mixed reality simulations in medical training: a methodological approach for the assessment of stress, cognitive load and performance

La simulazione nell'educazione in medicina è considerata un metodo di formazione in grado di migliorare le competenze cliniche e il comportamento degli operatori sanitari e, di conseguenza, la qualità dell'assistenza per il paziente. Inoltre, l'utilizzo di nuove tecnologie come la Realtà Aumentata, offre ai discenti l'opportunità di esercitarsi in un ambiente immersivo. L'opportunità di sperimentare questo innovativo metodo didattico è efficace non solo nel ridurre il rischio di errori e approcci sbagliati ma anche nel provare ansia e stress simili a quelli avvertiti nella pratica reale. La sfida sta nel trovare il giusto equilibrio. I discenti devono infatti provare lo stesso stress che avvertirebbero lavorando ad un vero caso clinico ma, allo stesso tempo, devono essere controllati ed evitati possibili disturbi da stress post-traumatico, verificabili soprattutto nel campo della gestione delle emergenze (pronto soccorso). Inoltre, è fondamentale anche ottenere alte prestazioni e un apprendimento adeguato, evitando sovraccarichi cognitivi che influenzerebbero negativamente l’apprendimento. Tuttavia, ad oggi mancano ancora studi approfonditi sull'impatto che le simulazioni mediche hanno su stress, frustrazione, carico cognitivo e apprendimento dei discenti. Per questo motivo, l'obiettivo principale di questo studio è valutare l'efficacia del training tramite simulazione, analizzando prestazioni, ansia, stress e carico cognitivo durante simulazioni cliniche tradizionali (con manichino) ed avanzate (in realtà mista). A questo scopo, è stato sviluppato un approccio metodologico strutturato e completo per valutare le prestazioni, le condizioni emotive e cognitive degli studenti. Questo comprende l'acquisizione e l'analisi di parametri psicologici (valutazione soggettiva), segnali biometrici (valutazione oggettiva) e prestazioni. Questa indagine consente di evidenziare i punti deboli delle simulazioni e offre l'opportunità di definire utili linee guida per la riprogettazione e l'ottimizzazione delle stesse. La metodologia è stata applicata su tre casi studio: il primo si riferisce a simulazioni ad alta fedeltà per la gestione del paziente in pronto soccorso, il secondo si riferisce a simulazioni a bassa fedeltà per la pratica della rachicentesi. Per il terzo caso studio, è stato progettato e sviluppato un prototipo di simulatore in realtà mista per la rachicentesi, con l'obiettivo di migliorare il senso di realismo e immersione della simulazione a bassa fedeltà. 148 studenti sono stati coinvolti nei primi due casi studio osservazionali, mentre soltanto 36 studenti hanno preso parte allo studio pilota sulla simulazione in realtà mista. In tutti i casi di studio sono state effettuate analisi descrittive delle prestazioni, degli stati cognitivi ed emotivi. Per le simulazioni ad alta e bassa fedeltà, le analisi di regressione statistica hanno evidenziato quali variabili influenzano le prestazioni, lo stress e il carico cognitivo degli studenti. Per lo studio pilota sulla realtà mista, l'analisi della user experience ha sottolineato i limiti tecnici della nuova tecnologia.Simulation in medical education is considered a training method capable of improving clinical competence and practitioners’ behaviour, and, consequently quality of care and patient’s outcome. Moreover, the use of new technologies, such as augmented reality, offers to the learners the opportunity to engage themselves in an immersive environment. The opportunity to experiment with this innovative instructional method is effective not only in reducing the risk of errors and wrong approaches but also in experiencing anxiety and stress as in real practice. The challenge is to find the right stress balance: learners have to feel as if they were practicing in the real stressful clinical case, and, at the same time, post-traumatic stress disorders, verifiable especially in the emergency field, must be controlled and avoided. Moreover, it is fundamental also to obtain high performance and learning, thus avoiding cognitive overloads. However, extensive researches about the impact of medical simulations on students’ stress, frustration, cognitive load, and learning are still lacking. For this reason, the main objective of this study is to assess simulation training effectiveness by analysing performance, anxiety, stress, and cognitive load during traditional (with manikin) and advanced (with augmented reality) clinical simulations. A structured and comprehensive methodological approach to assess performance, emotional and cognitive conditions of students has been developed. It includes the acquisition and analysis of psychological parameters (subjective assessment), biometric signals (objective assessment), and task performance. This investigation allows to point out simulations’ weaknesses and offers the opportunity to define useful optimisation guidelines. The methodology has been applied to three case studies: the first one refers to high-fidelity simulations, for the patient management in the emergency room, the second one refers to low-fidelity simulation for rachicentesis. For the third case study, a prototype of a mixed reality simulator for the rachicentesis practice has been designed and developed aiming at improving the sense of realism and immersion of the low-fidelity simulation. While 148 students have been enrolled in the first two case studies, only 36 students have taken part in the pilot study about mixed reality simulation. Descriptive analysis about performance, cognitive and emotional states have been done in all the case studies. For the high-fidelity and low-fidelity simulations, the statistical regression analysis has pointed out which variables affect students’ performance, stress, and cognitive load. For the pilot study about mixed reality, the user experience analysis highlighted the technical limitations of the new technology

Degenerative adversarial neuroimage nets for brain scan simulations: Application in ageing and dementia

© 2021 The Author(s). Published by Elsevier B.V. This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY), https://creativecommons.org/licenses/by/4.0/Accurate and realistic simulation of high-dimensional medical images has become an important research area relevant to many AI-enabled healthcare applications. However, current state-of-the-art approaches lack the ability to produce satisfactory high-resolution and accurate subject-specific images. In this work, we present a deep learning framework, namely 4D-Degenerative Adversarial NeuroImage Net (4D-DANI-Net), to generate high-resolution, longitudinal MRI scans that mimic subject-specific neurodegeneration in ageing and dementia. 4D-DANI-Net is a modular framework based on adversarial training and a set of novel spatiotemporal, biologically-informed constraints. To ensure efficient training and overcome memory limitations affecting such high-dimensional problems, we rely on three key technological advances: i) a new 3D training consistency mechanism called Profile Weight Functions (PWFs), ii) a 3D super-resolution module and iii) a transfer learning strategy to fine-tune the system for a given individual. To evaluate our approach, we trained the framework on 9852 T1-weighted MRI scans from 876 participants in the Alzheimer's Disease Neuroimaging Initiative dataset and held out a separate test set of 1283 MRI scans from 170 participants for quantitative and qualitative assessment of the personalised time series of synthetic images. We performed three evaluations: i) image quality assessment; ii) quantifying the accuracy of regional brain volumes over and above benchmark models; and iii) quantifying visual perception of the synthetic images by medical experts. Overall, both quantitative and qualitative results show that 4D-DANI-Net produces realistic, low-artefact, personalised time series of synthetic T1 MRI that outperforms benchmark models.Peer reviewe

Degenerative adversarial neuroimage nets for brain scan simulations: Application in ageing and dementia

Accurate and realistic simulation of high-dimensional medical images has become an important research area relevant to many AI-enabled healthcare applications. However, current state-of-the-art approaches lack the ability to produce satisfactory high-resolution and accurate subject-specific images. In this work, we present a deep learning framework, namely 4D-Degenerative Adversarial NeuroImage Net (4D-DANI-Net), to generate high-resolution, longitudinal MRI scans that mimic subject-specific neurodegeneration in ageing and dementia. 4D-DANI-Net is a modular framework based on adversarial training and a set of novel spatiotemporal, biologically-informed constraints. To ensure efficient training and overcome memory limitations affecting such high-dimensional problems, we rely on three key technological advances: i) a new 3D training consistency mechanism called Profile Weight Functions (PWFs), ii) a 3D super-resolution module and iii) a transfer learning strategy to fine-tune the system for a given individual. To evaluate our approach, we trained the framework on 9852 T1-weighted MRI scans from 876 participants in the Alzheimer's Disease Neuroimaging Initiative dataset and held out a separate test set of 1283 MRI scans from 170 participants for quantitative and qualitative assessment of the personalised time series of synthetic images. We performed three evaluations: i) image quality assessment; ii) quantifying the accuracy of regional brain volumes over and above benchmark models; and iii) quantifying visual perception of the synthetic images by medical experts. Overall, both quantitative and qualitative results show that 4D-DANI-Net produces realistic, low-artefact, personalised time series of synthetic T1 MRI that outperforms benchmark models

A Virtual University Infrastructure For Orthopaedic Surgical Training With Integrated Simulation

This thesis pivots around the fulcrum of surgical, educational and technological factors. Whilst there is no single conclusion drawn, it is a multidisciplinary thesis exploring the juxtaposition of different academic domains that have a significant influence upon each other. The relationship centres on the engineering and computer science factors in learning technologies for surgery. Following a brief introduction to previous efforts developing surgical simulation, this thesis considers education and learning in orthopaedics, the design and building of a simulator for shoulder surgery. The thesis considers the assessment of such tools and embedding into a virtual learning environment. It explains how the performed experiments clarified issues and their actual significance. This leads to discussion of the work and conclusions are drawn regarding the progress of integration of distributed simulation within the healthcare environment, suggesting how future work can proceed