A computational medical XR discipline

Computational medical XR (extended reality) brings together life sciences and neuroscience with mathematics, engineering, and computer science. It unifies computational science (scientific computing) with intelligent extended reality and spatial computing for the medical field. It significantly extends previous Clinical XR, by integrating computational methods from neural simulation to computational geometry, computational vision and computer graphics up to theoretical computer science to solve hard problems in medicine and neuroscience: from low-code/no-code authoring medical XR platforms to deep learning systems for diagnostics, therapeutics, rehabilitation and from surgical planning to real-time operative navigation in XR

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.Comment: Review, 21 pages, 8 figures and 5 boxe

Adaptive and Topological Deep Learning with applications to Neuroscience

Deep Learning and neuroscience have developed a two way relationship with each informing the other. Neural networks, the main tools at the heart of Deep Learning, were originally inspired by connectivity in the brain and have now proven to be critical to state-of-the-art computational neuroscience methods. This dissertation explores this relationship, first, by developing an adaptive sampling method for a neural network-based partial different equation solver and then by developing a topological deep learning framework for neural spike decoding. We demonstrate that our adaptive scheme is convergent and more accurate than DGM -- as long as the residual mirrors the local error -- at the same number of training steps and using the same or less number of training points. We present a multitude of tests applied to selected PDEs discussing the robustness of our scheme. Next, we further illustrate the partnership between deep learning and neuroscience by decoding neural activity using a novel neural network architecture developed to exploit the underlying connectivity of the data by employing tools from Topological Data Analysis. Neurons encode information like external stimuli or allocentric location by generating firing patterns where specific ensembles of neurons fire simultaneously for one value. Understanding, representing, and decoding these neural structures require models that encompass higher order connectivity than traditional graph-based models may provide. Our framework combines unsupervised simplicial complex discovery with the power of deep learning via a new architecture we develop herein called a simplicial convolutional recurrent neural network (SCRNN). Simplicial complexes, topological spaces that use not only vertices and edges but also higher-dimensional objects, naturally generalize graphs and capture more than just pairwise relationships. The effectiveness and versatility of the SCRNN is demonstrated on head direction data to test its performance and then applied to grid cell datasets with the task to automatically predict trajectories