Interactive in situ visualization of large volume data

Three-dimensional volume data is routinely produced, at increasingly high spatial resolution, in computer simulations and image acquisition tasks. In-situ visualization, the visualization of an experiment or simulation while it is running, enables new modes of interaction, including simulation steering and experiment control. These can provide the scientist a deeper understanding of the underlying phenomena, but require interactive visualization with smooth viewpoint changes and zooming to convey depth perception and spatial understanding. As the size of the volume data increases, however, it is increasingly challenging to achieve interactive visualization with smooth viewpoint changes. This thesis presents an end-to-end solution for interactive in-situ visualization based on novel extensions proposed to the Volumetric Depth Image (VDI) representation. VDIs are view-dependent, compact representations of volume data than can be rendered faster than the original data. Novel methods are proposed in this thesis for generating VDIs on large data and for rendering them faster. Together, they enable interactive in situ visualization with smooth viewpoint changes and zooming for large volume data. The generation of VDIs involves decomposing the volume rendering integral along rays into segments that store composited color and opacity, forming a representation much smaller than the volume data. This thesis introduces a technique to automatically determine the sensitivity parameter that governs the decomposition of rays, eliminating the need for manual parameter tuning in the generation of a VDI. Further, a method is proposed for sort-last parallel generation and compositing of VDIs on distributed computers, enabling their in situ generation with distributed numerical simulations. A low latency architecture is proposed for the sharing of data and hardware resources with a running simulation. The resulting VDI can be streamed for interactive visualization. A novel raycasting method is proposed for rendering VDIs. Properties of perspective projection are exploited to simplify the intersection of rays with the view-dependent segments contained within the VDI. Spatial smoothness in volume data is leveraged to minimize memory accesses. Benchmarks are performed showing that the method significantly outperforms existing methods for rendering the VDI, and achieves responsive frame rates for High Definition (HD) display resolutions near the viewpoint of generation. Further, a method is proposed to subsample the VDI for preview rendering, maintaining high frame rates even for large viewpoint deviations. The quality and performance of the approach are analyzed on multiple datasets, and the contributions are provided as extensions of established open-source tools. The thesis concludes with a discussion on the strengths, limitations, and future directions for the proposed approach

Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers

Autonomous systems, such as self-driving cars and drones, have made significant strides in recent years by leveraging visual inputs and machine learning for decision-making and control. Despite their impressive performance, these vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. Such errors can cascade to catastrophic system failures and compromise system safety. In this work, we introduce a run-time anomaly monitor to detect and mitigate such closed-loop, system-level failures. Specifically, we leverage a reachability-based framework to stress-test the vision-based controller offline and mine its system-level failures. This data is then used to train a classifier that is leveraged online to flag inputs that might cause system breakdowns. The anomaly detector highlights issues that transcend individual modules and pertain to the safety of the overall system. We also design a fallback controller that robustly handles these detected anomalies to preserve system safety. We validate the proposed approach on an autonomous aircraft taxiing system that uses a vision-based controller for taxiing. Our results show the efficacy of the proposed approach in identifying and handling system-level anomalies, outperforming methods such as prediction error-based detection, and ensembling, thereby enhancing the overall safety and robustness of autonomous systems

Optimizing Electric Vehicle Efficiency with Real-Time Telemetry using Machine Learning

In the contemporary world with degrading natural resources, the urgency of energy efficiency has become imperative due to the conservation and environmental safeguarding. Therefore, it's crucial to look for advanced technology to minimize energy consumption. This research focuses on the optimization of battery-electric city style vehicles through the use of a real-time in-car telemetry system that communicates between components through the robust Controller Area Network (CAN) protocol. By harnessing real-time data from various sensors embedded within vehicles, our driving assistance system provides the driver with visual and haptic actionable feedback that guides the driver on using the optimum driving style to minimize power consumed by the vehicle. To develop the pace feedback mechanism for the driver, real-time data is collected through a Shell Eco Marathon Urban Concept vehicle platform and after pre-processing, it is analyzed using the novel machine learning algorithm TEMSL, that outperforms the existing baseline approaches across various performance metrics. This innovative method after numerous experimentation has proven effective in enhancing energy efficiency, guiding the driver along the track, and reducing human errors. The driving-assistance system offers a range of utilities, from cost savings and extended vehicle lifespan to significant contributions to environmental conservation and sustainable driving practices

scenery: Flexible Virtual Reality Visualization on the Java VM

Life science today involves computational analysis of a large amount and variety of data, such as volumetric data acquired by state-of-the-art microscopes, or mesh data from analysis of such data or simulations. Visualization is often the first step in making sense of data, and a crucial part of building and debugging analysis pipelines. It is therefore important that visualizations can be quickly prototyped, as well as developed or embedded into full applications. In order to better judge spatiotemporal relationships, immersive hardware, such as Virtual or Augmented Reality (VR/AR) headsets and associated controllers are becoming invaluable tools. In this work we introduce scenery, a flexible VR/AR visualization framework for the Java VM that can handle mesh and large volumetric data, containing multiple views, timepoints, and color channels. scenery is free and open-source software, works on all major platforms, and uses the Vulkan or OpenGL rendering APIs. We introduce scenery's main features and example applications, such as its use in VR for microscopy, in the biomedical image analysis software Fiji, or for visualizing agent-based simulations.Comment: Added IEEE DOI, version published at VIS 201

Raloxifene improves bone mechanical properties in mice previously treated with zoledronate

Bisphosphonates represent the gold-standard pharmaceutical agent for reducing fracture risk. Long-term treatment with bisphosphonates can result in tissue brittleness which in rare clinical cases manifests as atypical femoral fracture. Although this has led to an increasing call for bisphosphonate cessation, few studies have investigated therapeutic options for follow-up treatment. The goal of this study was to test the hypothesis that treatment with raloxifene, a drug that has cell-independent effects on bone mechanical material properties, could reverse the compromised mechanical properties that occur following zoledronate treatment. Skeletally mature male C57Bl/6J mice were treated with vehicle (VEH), zoledronate (ZOL), or ZOL followed by raloxifene (RAL; 2 different doses). At the conclusion of 8 weeks of treatment, femora were collected and assessed with microCT and mechanical testing. Trabecular BV/TV was significantly higher in all treated animals compared to VEH with both RAL groups having significantly higher BV/TV compared to ZOL (+21%). All three drug-treated groups had significantly more cortical bone area, higher cortical thickness, and greater moment of inertia at the femoral mid-diaphysis compared to VEH with no difference among the three treated groups. All three drug-treated groups had significantly higher ultimate load compared to VEH-treated animals (+14 to 18%). Both doses of RAL resulted in significantly higher displacement values compared to ZOL-treated animals (+25 to +50%). In conclusion, the current work shows beneficial effects of raloxifene in animals previously treated with zoledronate. The higher mechanical properties of raloxifene-treated animals, combined with similar cortical bone geometry compared to animals treated with zoledronate, suggest that the raloxifene treatment is enhancing mechanical material properties of the tissue

Robust estimation of bacterial cell count from optical density

Optical density (OD) is widely used to estimate the density of cells in liquid culture, but cannot be compared between instruments without a standardized calibration protocol and is challenging to relate to actual cell count. We address this with an interlaboratory study comparing three simple, low-cost, and highly accessible OD calibration protocols across 244 laboratories, applied to eight strains of constitutive GFP-expressing E. coli. Based on our results, we recommend calibrating OD to estimated cell count using serial dilution of silica microspheres, which produces highly precise calibration (95.5% of residuals <1.2-fold), is easily assessed for quality control, also assesses instrument effective linear range, and can be combined with fluorescence calibration to obtain units of Molecules of Equivalent Fluorescein (MEFL) per cell, allowing direct comparison and data fusion with flow cytometry measurements: in our study, fluorescence per cell measurements showed only a 1.07-fold mean difference between plate reader and flow cytometry data

Interactive in situ visualization of large volume data

Three-dimensional volume data is routinely produced, at increasingly high spatial resolution, in computer simulations and image acquisition tasks. In-situ visualization, the visualization of an experiment or simulation while it is running, enables new modes of interaction, including simulation steering and experiment control. These can provide the scientist a deeper understanding of the underlying phenomena, but require interactive visualization with smooth viewpoint changes and zooming to convey depth perception and spatial understanding. As the size of the volume data increases, however, it is increasingly challenging to achieve interactive visualization with smooth viewpoint changes. This thesis presents an end-to-end solution for interactive in-situ visualization based on novel extensions proposed to the Volumetric Depth Image (VDI) representation. VDIs are view-dependent, compact representations of volume data than can be rendered faster than the original data. Novel methods are proposed in this thesis for generating VDIs on large data and for rendering them faster. Together, they enable interactive in situ visualization with smooth viewpoint changes and zooming for large volume data. The generation of VDIs involves decomposing the volume rendering integral along rays into segments that store composited color and opacity, forming a representation much smaller than the volume data. This thesis introduces a technique to automatically determine the sensitivity parameter that governs the decomposition of rays, eliminating the need for manual parameter tuning in the generation of a VDI. Further, a method is proposed for sort-last parallel generation and compositing of VDIs on distributed computers, enabling their in situ generation with distributed numerical simulations. A low latency architecture is proposed for the sharing of data and hardware resources with a running simulation. The resulting VDI can be streamed for interactive visualization. A novel raycasting method is proposed for rendering VDIs. Properties of perspective projection are exploited to simplify the intersection of rays with the view-dependent segments contained within the VDI. Spatial smoothness in volume data is leveraged to minimize memory accesses. Benchmarks are performed showing that the method significantly outperforms existing methods for rendering the VDI, and achieves responsive frame rates for High Definition (HD) display resolutions near the viewpoint of generation. Further, a method is proposed to subsample the VDI for preview rendering, maintaining high frame rates even for large viewpoint deviations. The quality and performance of the approach are analyzed on multiple datasets, and the contributions are provided as extensions of established open-source tools. The thesis concludes with a discussion on the strengths, limitations, and future directions for the proposed approach

Interactive in situ visualization of large volume data

Three-dimensional volume data is routinely produced, at increasingly high spatial resolution, in computer simulations and image acquisition tasks. In-situ visualization, the visualization of an experiment or simulation while it is running, enables new modes of interaction, including simulation steering and experiment control. These can provide the scientist a deeper understanding of the underlying phenomena, but require interactive visualization with smooth viewpoint changes and zooming to convey depth perception and spatial understanding. As the size of the volume data increases, however, it is increasingly challenging to achieve interactive visualization with smooth viewpoint changes. This thesis presents an end-to-end solution for interactive in-situ visualization based on novel extensions proposed to the Volumetric Depth Image (VDI) representation. VDIs are view-dependent, compact representations of volume data than can be rendered faster than the original data. Novel methods are proposed in this thesis for generating VDIs on large data and for rendering them faster. Together, they enable interactive in situ visualization with smooth viewpoint changes and zooming for large volume data. The generation of VDIs involves decomposing the volume rendering integral along rays into segments that store composited color and opacity, forming a representation much smaller than the volume data. This thesis introduces a technique to automatically determine the sensitivity parameter that governs the decomposition of rays, eliminating the need for manual parameter tuning in the generation of a VDI. Further, a method is proposed for sort-last parallel generation and compositing of VDIs on distributed computers, enabling their in situ generation with distributed numerical simulations. A low latency architecture is proposed for the sharing of data and hardware resources with a running simulation. The resulting VDI can be streamed for interactive visualization. A novel raycasting method is proposed for rendering VDIs. Properties of perspective projection are exploited to simplify the intersection of rays with the view-dependent segments contained within the VDI. Spatial smoothness in volume data is leveraged to minimize memory accesses. Benchmarks are performed showing that the method significantly outperforms existing methods for rendering the VDI, and achieves responsive frame rates for High Definition (HD) display resolutions near the viewpoint of generation. Further, a method is proposed to subsample the VDI for preview rendering, maintaining high frame rates even for large viewpoint deviations. The quality and performance of the approach are analyzed on multiple datasets, and the contributions are provided as extensions of established open-source tools. The thesis concludes with a discussion on the strengths, limitations, and future directions for the proposed approach