Interactive Visualization of Multimodal Brain Connectivity: Applications in Clinical and Cognitive Neuroscience

Magnetic resonance imaging (MRI) has become a readily available prognostic and diagnostic method, providing invaluable information for the clinical treatment of neurological diseases. Multimodal neuroimaging allows integration of complementary data from various aspects such as functional and anatomical properties; thus, it has the potential to overcome the limitations of each individual modality. Specifically, functional and diffusion MRI are two non-invasive neuroimaging techniques customized to capture brain activity and microstructural properties, respectively. Data from these two modalities is inherently complex, and interactive visualization can assist with data comprehension. The current thesis presents the design, development, and validation of visualization and computation approaches that address the need for integration of brain connectivity from functional and structural domains. Two contexts were considered to develop these approaches: neuroscience exploration and minimally invasive neurosurgical planning. The goal was to provide novel visualization algorithms and gain new insights into big and complex data (e.g., brain networks) by visual analytics. This goal was achieved through three steps: 3D Graphical Collision Detection: One of the primary challenges was the timely rendering of grey matter (GM) regions and white matter (WM) fibers based on their 3D spatial maps. This challenge necessitated pre-scanning those objects to generate a memory array containing their intersections with memory units. This process helped faster retrieval of GM and WM virtual models during the user interactions. Neuroscience Enquiry (MultiXplore): A software interface was developed to display and react to user inputs by means of a connectivity matrix. This matrix displays connectivity information and is capable to accept selections from users and display the relevant ones in 3D anatomical view (with associated anatomical elements). In addition, this package can load multiple matrices from dynamic connectivity methods and annotate brain fibers. Neurosurgical Planning (NeuroPathPlan): A computational method was provided to map the network measures to GM and WM; thus, subject-specific eloquence metric can be derived from related resting state networks and used in objective assessment of cortical and subcortical tissue. This metric was later compared to apriori knowledge based decisions from neurosurgeons. Preliminary results show that eloquence metric has significant similarities with expert decisions

Immersive Virtual Reality Tool for Connectome Visualization and Analysis

The human brain is a complex organ made up of billions of neurons that are interconnected through a vast network of synapses. This network of connections enables the brain to perform a wide range of cognitive and motor functions. Studying and analyzing these brain networks is important for understanding how different regions of the brain communicate and work together to carry out specific tasks and how neurological disorders such as Alzheimer’s disease, Parkinson’s disease, or schizophrenia impact brain connectivity contributing to the development of these disorders. Virtual reality technology has proven to be a versatile tool for learning, exploration, and analysis. It can expand the user’s senses, provide a more detailed and immersive view of the subject matter, encourage active learning and exploration, and facilitate global analysis of complex data. In this dissertation, we present VRNConnect, a virtual reality system for interactively exploring brain connectivity data. VRNConnect enables users to analyze brain networks using either structural or functional connectivity matrices. By visualizing the 3D brain connectome network as a graph, users can interact with various regions using hand gestures or controllers to access network analysis metrics and information about Regions of Interest (ROIs). The system includes features such as colour coding of nodes and edges, thresholding, and shortest path calculation to enhance usability. Moreover, VRNConnect has the ability to be tailored to specific needs, allowing for the importation of connectivity data from various modalities. Our platform was designed with flexibility in mind, making it easy to incorporate additional features as needed. In order to evaluate the usability and cognitive workload associated with using our system, we conducted a study with 16 participants. Our findings suggest that VRNConnect could serve as an effective academic and analytical tool

Visual Exploration And Information Analytics Of High-Dimensional Medical Images

Data visualization has transformed how we analyze increasingly large and complex data sets. Advanced visual tools logically represent data in a way that communicates the most important information inherent within it and culminate the analysis with an insightful conclusion. Automated analysis disciplines - such as data mining, machine learning, and statistics - have traditionally been the most dominant fields for data analysis. It has been complemented with a near-ubiquitous adoption of specialized hardware and software environments that handle the storage, retrieval, and pre- and postprocessing of digital data. The addition of interactive visualization tools allows an active human participant in the model creation process. The advantage is a data-driven approach where the constraints and assumptions of the model can be explored and chosen based on human insight and confirmed on demand by the analytic system. This translates to a better understanding of data and a more effective knowledge discovery. This trend has become very popular across various domains, not limited to machine learning, simulation, computer vision, genetics, stock market, data mining, and geography. In this dissertation, we highlight the role of visualization within the context of medical image analysis in the field of neuroimaging. The analysis of brain images has uncovered amazing traits about its underlying dynamics. Multiple image modalities capture qualitatively different internal brain mechanisms and abstract it within the information space of that modality. Computational studies based on these modalities help correlate the high-level brain function measurements with abnormal human behavior. These functional maps are easily projected in the physical space through accurate 3-D brain reconstructions and visualized in excellent detail from different anatomical vantage points. Statistical models built for comparative analysis across subject groups test for significant variance within the features and localize abnormal behaviors contextualizing the high-level brain activity. Currently, the task of identifying the features is based on empirical evidence, and preparing data for testing is time-consuming. Correlations among features are usually ignored due to lack of insight. With a multitude of features available and with new emerging modalities appearing, the process of identifying the salient features and their interdependencies becomes more difficult to perceive. This limits the analysis only to certain discernible features, thus limiting human judgments regarding the most important process that governs the symptom and hinders prediction. These shortcomings can be addressed using an analytical system that leverages data-driven techniques for guiding the user toward discovering relevant hypotheses. The research contributions within this dissertation encompass multidisciplinary fields of study not limited to geometry processing, computer vision, and 3-D visualization. However, the principal achievement of this research is the design and development of an interactive system for multimodality integration of medical images. The research proceeds in various stages, which are important to reach the desired goal. The different stages are briefly described as follows: First, we develop a rigorous geometry computation framework for brain surface matching. The brain is a highly convoluted structure of closed topology. Surface parameterization explicitly captures the non-Euclidean geometry of the cortical surface and helps derive a more accurate registration of brain surfaces. We describe a technique based on conformal parameterization that creates a bijective mapping to the canonical domain, where surface operations can be performed with improved efficiency and feasibility. Subdividing the brain into a finite set of anatomical elements provides the structural basis for a categorical division of anatomical view points and a spatial context for statistical analysis. We present statistically significant results of our analysis into functional and morphological features for a variety of brain disorders. Second, we design and develop an intelligent and interactive system for visual analysis of brain disorders by utilizing the complete feature space across all modalities. Each subdivided anatomical unit is specialized by a vector of features that overlap within that element. The analytical framework provides the necessary interactivity for exploration of salient features and discovering relevant hypotheses. It provides visualization tools for confirming model results and an easy-to-use interface for manipulating parameters for feature selection and filtering. It provides coordinated display views for visualizing multiple features across multiple subject groups, visual representations for highlighting interdependencies and correlations between features, and an efficient data-management solution for maintaining provenance and issuing formal data queries to the back end

Establishing a Framework for the development of Multimodal Virtual Reality Interfaces with Applicability in Education and Clinical Practice

The development of Virtual Reality (VR) and Augmented Reality (AR) content with multiple sources of both input and output has led to countless contributions in a great many number of fields, among which medicine and education. Nevertheless, the actual process of integrating the existing VR/AR media and subsequently setting it to purpose is yet a highly scattered and esoteric undertaking. Moreover, seldom do the architectures that derive from such ventures comprise haptic feedback in their implementation, which in turn deprives users from relying on one of the paramount aspects of human interaction, their sense of touch. Determined to circumvent these issues, the present dissertation proposes a centralized albeit modularized framework that thus enables the conception of multimodal VR/AR applications in a novel and straightforward manner. In order to accomplish this, the aforesaid framework makes use of a stereoscopic VR Head Mounted Display (HMD) from Oculus Rift©, a hand tracking controller from Leap Motion©, a custom-made VR mount that allows for the assemblage of the two preceding peripherals and a wearable device of our own design. The latter is a glove that encompasses two core modules in its innings, one that is able to convey haptic feedback to its wearer and another that deals with the non-intrusive acquisition, processing and registering of his/her Electrocardiogram (ECG), Electromyogram (EMG) and Electrodermal Activity (EDA). The software elements of the aforementioned features were all interfaced through Unity3D©, a powerful game engine whose popularity in academic and scientific endeavors is evermore increasing. Upon completion of our system, it was time to substantiate our initial claim with thoroughly developed experiences that would attest to its worth. With this premise in mind, we devised a comprehensive repository of interfaces, amid which three merit special consideration: Brain Connectivity Leap (BCL), Ode to Passive Haptic Learning (PHL) and a Surgical Simulator

Advanced Visualization and Intuitive User Interface Systems for Biomedical Applications

Modern scientific research produces data at rates that far outpace our ability to comprehend and analyze it. Such sources include medical imaging data and computer simulations, where technological advancements and spatiotemporal resolution generate increasing amounts of data from each scan or simulation. A bottleneck has developed whereby medical professionals and researchers are unable to fully use the advanced information available to them. By integrating computer science, computer graphics, artistic ability and medical expertise, scientific visualization of medical data has become a new field of study. The objective of this thesis is to develop two visualization systems that use advanced visualization, natural user interface technologies and the large amount of biomedical data available to produce results that are of clinical utility and overcome the data bottleneck that has developed. Computational Fluid Dynamics (CFD) is a tool used to study the quantities associated with the movement of blood by computer simulation. We developed methods of processing spatiotemporal CFD data and displaying it in stereoscopic 3D with the ability to spatially navigate through the data. We used this method with two sets of display hardware: a full-scale visualization environment and a small-scale desktop system. The advanced display and data navigation abilities provide the user with the means to better understand the relationship between the vessel\u27s form and function. Low-cost 3D, depth-sensing cameras capture and process user body motion to recognize motions and gestures. Such devices allow users to use hand motions as an intuitive interface to computer applications. We developed algorithms to process and prepare the biomedical and scientific data for use with a custom control application. The application interprets user gestures as commands to a visualization tool and allows the user to control the visualization of multi-dimensional data. The intuitive interface allows the user to control the visualization of data without manual contact with an interaction device. In developing these methods and software tools we have leveraged recent trends in advanced visualization and intuitive interfaces in order to efficiently visualize biomedical data in such a way that provides meaningful information that can be used to further appreciate it

A formal architecture-centric and model driven approach for the engineering of science gateways

From n-Tier client/server applications, to more complex academic Grids, or even the most recent and promising industrial Clouds, the last decade has witnessed significant developments in distributed computing. In spite of this conceptual heterogeneity, Service-Oriented Architecture (SOA) seems to have emerged as the common and underlying abstraction paradigm, even though different standards and technologies are applied across application domains. Suitable access to data and algorithms resident in SOAs via so-called ‘Science Gateways’ has thus become a pressing need in order to realize the benefits of distributed computing infrastructures.In an attempt to inform service-oriented systems design and developments in Grid-based biomedical research infrastructures, the applicant has consolidated work from three complementary experiences in European projects, which have developed and deployed large-scale production quality infrastructures and more recently Science Gateways to support research in breast cancer, pediatric diseases and neurodegenerative pathologies respectively. In analyzing the requirements from these biomedical applications the applicant was able to elaborate on commonly faced issues in Grid development and deployment, while proposing an adapted and extensible engineering framework. Grids implement a number of protocols, applications, standards and attempt to virtualize and harmonize accesses to them. Most Grid implementations therefore are instantiated as superposed software layers, often resulting in a low quality of services and quality of applications, thus making design and development increasingly complex, and rendering classical software engineering approaches unsuitable for Grid developments.The applicant proposes the application of a formal Model-Driven Engineering (MDE) approach to service-oriented developments, making it possible to define Grid-based architectures and Science Gateways that satisfy quality of service requirements, execution platform and distribution criteria at design time. An novel investigation is thus presented on the applicability of the resulting grid MDE (gMDE) to specific examples and conclusions are drawn on the benefits of this approach and its possible application to other areas, in particular that of Distributed Computing Infrastructures (DCI) interoperability, Science Gateways and Cloud architectures developments

On pattern recognition of brain connectivity in resting-state functional MRI

Dissertação de mestrado integrado em Biomedical Engineering (specialization on Medical Informatics)The human urge and pursuit for information have led to the development of increasingly complex technologies, and new means to study and understand the most advanced and intricate biological system: the human brain. Large-scale neuronal communication within the brain, and how it relates to human behaviour can be inferred by delving into the brain network, and searching for patterns in connectivity. Functional connectivity is a steady characteristic of the brain, and it has been proved to be very useful for examining how mental disorders affect connections within the brain. The detection of abnormal behaviour in brain networks is performed by experts, such as physicians, who limit the process with human subjectivity, and unwittingly introduce errors in the interpretation. The continuous search for alternatives to obtain faster and robuster results have put Machine Learning and Deep Learning in the leading position of computer vision, as they enable the extraction of meaningful patterns, some beyond human perception. The aim of this dissertation is to design and develop an experiment setup to analyse functional connectivity at a voxel level, in order to find functional patterns. For the purpose, a pipeline was outlined to include steps from data download to data analysis, resulting in four methods: Data Download, Data Preprocessing, Dimensionality Reduction, and Analysis. The proposed experiment setup was modeled using as materials resting state fMRI data from two sources: Life and Health Sciences Research Institute (Portugal), and Human Connectome Project (USA). To evaluate its performance, a case study was performed using the In-House data for concerning a smaller number of subjects to study. The pipeline was successful at delivering results, although limitations concerning the memory of the machine used restricted some aspects of this experiment setup’s testing. With appropriate resources, this experiment setup may support the process of analysing and extracting patterns from any resting state functional connectivity data, and aid in the detection of mental disorders.O desejo e a busca intensos do ser humano por informação levaram ao desenvolvimento de tecnologias cada vez mais complexas e novos meios para estudar e entender o sistema biológico mais avançado e intrincado: o cérebro humano. A comunicação neuronal em larga escala no cérebro, e como ela se relaciona com o comportamento humano, pode ser inferida investigando a rede neuronal cerebral e procurando por padrões de conectividade. A conectividade funcional é uma característica constante do cérebro e provou ser muito útil para examinar como os distúrbios mentais afetam as conexões cerebrais. A deteção de anormalidades em imagens de ressonância magnética é realizada por especialistas, como médicos, que limitam o processo com a subjetividade humana e, inadvertidamente, introduzem erros na interpretação. A busca contínua de alternativas para obter resultados mais rápidos e robustos colocou as técnicas de machine learning e deep learning na posição de liderança de visão computacional, pois permitem a extração de padrões significativos e alguns deles para além da percepção humana. O objetivo desta dissertação é projetar e desenvolver uma configuração experimental para analisar a conectividade funcional ao nível do voxel, a fim de encontrar padrões funcionais. Nesse sentido, foi delineado um pipeline para incluir etapas a começar no download de dados até à análise desses mesmos dados, resultando assim em quatro métodos: Download de Dados, Pré-processamento de Dados, Redução de Dimensionalidade e Análise. A configuração experimental proposta foi modelada usando dados de ressonância magnética funcional de resting-state de duas fontes: Instituto de Ciências da Vida e Saúde (Portugal) e Human Connectome Project (EUA). Para avaliar o seu desempenho, foi realizado um estudo de caso usando os dados internos por considerar um número menor de participantes a serem estudados. O pipeline foi bem-sucedido em fornecer resultados, embora limitações relacionadas com a memória da máquina usada tenham restringido alguns aspetos do teste desta configuração experimental. Com recursos apropriados, esta configuração experimental poderá servir de suporte para o processo de análise e extração de padrões de qualquer conjunto de dados de conectividade funcional em resting-state e auxiliar na deteção de transtornos mentais

AI in Medical Imaging Informatics: Current Challenges and Future Directions

This paper reviews state-of-the-art research solutions across the spectrum of medical imaging informatics, discusses clinical translation, and provides future directions for advancing clinical practice. More specifically, it summarizes advances in medical imaging acquisition technologies for different modalities, highlighting the necessity for efficient medical data management strategies in the context of AI in big healthcare data analytics. It then provides a synopsis of contemporary and emerging algorithmic methods for disease classification and organ/ tissue segmentation, focusing on AI and deep learning architectures that have already become the de facto approach. The clinical benefits of in-silico modelling advances linked with evolving 3D reconstruction and visualization applications are further documented. Concluding, integrative analytics approaches driven by associate research branches highlighted in this study promise to revolutionize imaging informatics as known today across the healthcare continuum for both radiology and digital pathology applications. The latter, is projected to enable informed, more accurate diagnosis, timely prognosis, and effective treatment planning, underpinning precision medicine