Imaging ductal carcinoma using a hyperspectral imaging system

Hyperspectral Imaging (HSI) is a non-invasive optical imaging modality that shows the potential to aid pathologists in breast cancer diagnoses cases. In this study, breast cancer tissues from different patients were imaged by a hyperspectral system to detect spectral differences between normal and breast cancer tissues, as well as early and late stages of breast cancer. If the spectral differences in these tissue types can be measured, automated systems can be developed to help the pathologist identify suspect biopsy samples, which will improve sample throughput and assist in making critical treatment decisions. Tissue samples from ten different patients were provided by the WVU Pathology Department. The samples from each patient included both normal and ductal carcinoma tissue, both stained and unstained. These cells were imaged using a snapshot HSI system, and the spectral reflectances were evaluated to see if there was a measurable spectral difference between the various cell types. Analysis of the spectral reflectance values indicated that wavelengths near 550nm show the best differentiation between tissue types. This information was used to train image processing algorithms using supervised and unsupervised data. K-Means and Support Vector Machine (SVM) approaches were applied to the hyperspectral data cubes, and successfully detected spectral tissue differences with sensitivity of 85.45%, and specificity of 94.64% with TNR of 95.8%, and FPR of 4.2%. These results were verified by ground truth marking of the tissue samples by a pathologist. This interdisciplinary work will build a bridge between pathology and hyperspectral optical diagnostic imaging in order to reduce time and workload on the pathologist, which can lead to benefit of lead reducing time, and increasing the accuracy of diagnoses

Framework for Hyperspectral Image Processing and Quantification for Cancer Detection During Animal Tumor Surgery

Hyperspectral imaging (HSI) is an imaging modality that holds strong potential for rapid cancer detection during image-guided surgery. But the data from HSI often needs to be processed appropriately in order to extract the maximum useful information that differentiates cancer from normal tissue. We proposed a framework for hyperspectral image processing and quantification, which includes a set of steps including image preprocessing, glare removal, feature extraction, and ultimately image classification. The framework has been tested on images from mice with head and neck cancer, using spectra from 450- to 900-nm wavelength. The image analysis computed Fourier coefficients, normalized reflectance, mean, and spectral derivatives for improved accuracy. The experimental results demonstrated the feasibility of the hyperspectral image processing and quantification framework for cancer detection during animal tumor surgery, in a challenging setting where sensitivity can be low due to a modest number of features present, but potential for fast image classification can be high. This HSI approach may have potential application in tumor margin assessment during image-guided surgery, where speed of assessment may be the dominant factor

Surgical spectral imaging

Recent technological developments have resulted in the availability of miniaturised spectral imaging sensors capable of operating in the multi- (MSI) and hyperspectral imaging (HSI) regimes. Simultaneous advances in image-processing techniques and artificial intelligence (AI), especially in machine learning and deep learning, have made these data-rich modalities highly attractive as a means of extracting biological information non-destructively. Surgery in particular is poised to benefit from this, as spectrally-resolved tissue optical properties can offer enhanced contrast as well as diagnostic and guidance information during interventions. This is particularly relevant for procedures where inherent contrast is low under standard white light visualisation. This review summarises recent work in surgical spectral imaging (SSI) techniques, taken from Pubmed, Google Scholar and arXiv searches spanning the period 2013–2019. New hardware, optimised for use in both open and minimally-invasive surgery (MIS), is described, and recent commercial activity is summarised. Computational approaches to extract spectral information from conventional colour images are reviewed, as tip-mounted cameras become more commonplace in MIS. Model-based and machine learning methods of data analysis are discussed in addition to simulation, phantom and clinical validation experiments. A wide variety of surgical pilot studies are reported but it is apparent that further work is needed to quantify the clinical value of MSI/HSI. The current trend toward data-driven analysis emphasises the importance of widely-available, standardised spectral imaging datasets, which will aid understanding of variability across organs and patients, and drive clinical translation

Determination of the transection margin during colorectal resection with hyperspectral imaging (HSI)

Abstract Purpose: This study evaluated the use of hyperspectral imaging for the determination of the resection margin during colorectal resections instead of clinical macroscopic assessment. Methods: The used hyperspectral camera is able to record light spectra from 500 to 1000 nm and provides information about physiologic parameters of the recorded tissue area intraoperatively (e.g., tissue oxygenation and perfusion). We performed an open-label, single-arm, and non-randomized intervention clinical trial to compare clinical assessment and hyperspectral measurement to define the resection margin in 24 patients before and after separation of the marginal artery over 15 min; HSI was performed each minute to assess the parameters mentioned above. Results: The false color images calculated from the hyperspectral data visualized the margin of perfusion in 20 out of 24 patients precisely. In the other four patients, the perfusion difference could be displayed with additional evaluation software. In all cases, there was a deviation between the transection line planed by the surgeon and the border line visualized by HSI (median 1 mm; range - 13 to 13 mm). Tissue perfusion dropped up to 12% within the first 10 mm distal to the border line. Therefore, the resection area was corrected proximally in five cases due to HSI record. The biggest drop in perfusion took place in less than 2 min after devascularization. Conclusion: Determination of the resection margin by HSI provides the surgeon with an objective decision aid for assessment of the best possible perfusion and ideal anastomotic area in colorectal surgery.:Inhaltsverzeichnis Inhaltsverzeichnis................................................................. I 1 Einführung............................................................................. 1 1.1 Anastomoseninsuffizienz...................................................1 1.2 Methodik Hyperspectral Imaging (HSI)............................. 3 1.3 Einsatzbereiche der Hyperspektral-Kamera..................... 5 1.4 Chirurgische Technik........................................................ 6 1.5 Studienplanung................................................................. 7 1.6 Vergleich der HSI-Technik mit weiteren Messmethoden...8 2 Publikation...............................................................................11 3 Zusammenfassung der Arbeit............................................... 21 4 Literaturverzeichnis............................................................... 26 5 Anhang.................................................................................... 30 Darstellung des eigenen Beitrags.........................................34 Eigenständigkeitserklärung...................................................35 Lebenslauf.............................................................................. 36 Danksagung........................................................................... 3

A Broadly Tunable Surface Plasmon-Coupled Wavelength Filter for Visible and Near Infrared Hyperspectral Imaging

Hyperspectral imaging is a set of techniques that has contributed to the study of advanced materials, pharmaceuticals, semiconductors, ceramics, polymers, biological specimens, and geological samples. Its use for remote sensing has advanced our understanding of agriculture, forestry, the Earth, environmental science, and the universe. The development of ultra-compact handheld hyperspectral imagers has been impeded by the scarcity of small widefield tunable wavelength filters. The widefield modality is preferred for handheld imaging applications in which image registration can be performed to counter scene shift caused by irregular user motions that would thwart scanning approaches. In the work presented here an electronically tunable widefield wavelength filter has been developed for hyperspectral imaging applications in the visible and near-infrared region. Conventional electronically tunable widefield imaging filter technologies include liquid crystal-based filters, acousto-optic tunable filters, and electronically tuned etalons; each having its own set of advantages and disadvantages. The construction of tunable filters is often complex and requires elaborate optical assemblies and electronic control circuits. I introduce in the work presented here is a novel widefield tunable filter, the surface plasmon coupled tunable filter (SPCTF), for visible and near infrared imaging. The SPCTF is based on surface plasmon coupling and has simple optical design that can be miniaturized without sacrificing performance. The SPCTF provides diffraction limited spatial resolution with a moderately narrow nominal passband (\u3c10 \u3enm) and a large spurious free spectral range (450 nm-1000 nm). The SPCTF employs surface plasmon coupling of the π-polarized component of incident light in metal films separated by a tunable dielectric layer. Acting on the π-polarized component, the device is limited to transmitting 50 percent of unpolarized incident light. This is higher than the throughput of comparable Lyot-based liquid crystal tunable filters that employ a series of linear polarizers. In addition, the SPCTF is not susceptible to the unwanted harmonic bands that lead to spurious diffraction in Bragg-based devices. Hence its spurious free spectral range covers a broad region from the blue through near infrared wavelengths. The compact design and rugged optical assembly make it suitable for hand-held hyperspectral imagers. The underlying theory and SPCTF design are presented along with a comparison of its performance to calculated estimates of transmittance, spectral resolution, and spectral range. In addition, widefield hyperspectral imaging using the SPCTF is demonstrated on model sample

Integration of Spatial and Spectral Information for Hyperspectral Image Classification

Hyperspectral imaging has become a powerful tool in biomedical and agriculture fields in the recent years and the interest amongst researchers has increased immensely. Hyperspectral imaging combines conventional imaging and spectroscopy to acquire both spatial and spectral information from an object. Consequently, a hyperspectral image data contains not only spectral information of objects, but also the spatial arrangement of objects. Information captured in neighboring locations may provide useful supplementary knowledge for analysis. Therefore, this dissertation investigates the integration of information from both the spectral and spatial domains to enhance hyperspectral image classification performance. The major impediment to the combined spatial and spectral approach is that most spatial methods were only developed for single image band. Based on the traditional singleimage based local Geary measure, this dissertation successfully proposes a Multidimensional Local Spatial Autocorrelation (MLSA) for hyperspectral image data. Based on the proposed spatial measure, this research work develops a collaborative band selection strategy that combines both the spectral separability measure (divergence) and spatial homogeneity measure (MLSA) for hyperspectral band selection task. In order to calculate the divergence more efficiently, a set of recursive equations for the calculation of divergence with an additional band is derived to overcome the computational restrictions. Moreover, this dissertation proposes a collaborative classification method which integrates the spectral distance and spatial autocorrelation during the decision-making process. Therefore, this method fully utilizes the spatial-spectral relationships inherent in the data, and thus improves the classification performance. In addition, the usefulness of the proposed band selection and classification method is evaluated with four case studies. The case studies include detection and identification of tumor on poultry carcasses, fecal on apple surface, cancer on mouse skin and crop in agricultural filed using hyperspectral imagery. Through the case studies, the performances of the proposed methods are assessed. It clearly shows the necessity and efficiency of integrating spatial information for hyperspectral image processing

Medical vision: web and mobile medical image retrieval system based on google cloud vision

The application of information technology is rapidly utilized in the medical system. There is also a massive development in the automatic method for recognizing and detecting objects in the real world. In this study, we present a system called Medical Vision which is designed for people who has no expertise in medical. Medical Vision is a web and mobile-based application to give an initial knowledge in a medical image. This system has 5 features; object detection, web detection, object labeling, safe search, and image properties. These features are run by embedding Google Vision API in the system. We evaluate this system by observing the result of some medical images which inputted into the system. The results showed that our system presents a promising performance and able to give relevant information related to the given image

Development of deep learning methods for head and neck cancer detection in hyperspectral imaging and digital pathology for surgical guidance

Surgeons performing routine cancer resections utilize palpation and visual inspection, along with time-consuming microscopic tissue analysis, to ensure removal of cancer. Despite this, inadequate surgical cancer margins are reported for up to 10-20% of head and neck squamous cell carcinoma (SCC) operations. There exists a need for surgical guidance with optical imaging to ensure complete cancer resection in the operating room. The objective of this dissertation is to evaluate hyperspectral imaging (HSI) as a non-contact, label-free optical imaging modality to provide intraoperative diagnostic information. For comparison of different optical methods, autofluorescence, RGB composite images synthesized from HSI, and two fluorescent dyes are also acquired and investigated for head and neck cancer detection. A novel and comprehensive dataset was obtained of 585 excised tissue specimens from 204 patients undergoing routine head and neck cancer surgeries. The first aim was to use SCC tissue specimens to determine the potential of HSI for surgical guidance in the challenging task of head and neck SCC detection. It is hypothesized that HSI could reduce time and provide quantitative cancer predictions. State-of-the-art deep learning algorithms were developed for SCC detection in 102 patients and compared to other optical methods. HSI detected SCC with a median AUC score of 85%, and several anatomical locations demonstrated good SCC detection, such as the larynx, oropharynx, hypopharynx, and nasal cavity. To understand the ability of HSI for SCC detection, the most important spectral features were calculated and correlated with known cancer physiology signals, notably oxygenated and deoxygenated hemoglobin. The second aim was to evaluate HSI for tumor detection in thyroid and salivary glands, and RGB images were synthesized using the spectral response curves of the human eye for comparison. Using deep learning, HSI detected thyroid tumors with 86% average AUC score, which outperformed fluorescent dyes and autofluorescence, but HSI-synthesized RGB imagery performed with 90% AUC score. The last aim was to develop deep learning algorithms for head and neck cancer detection in hundreds of digitized histology slides. Slides containing SCC or thyroid carcinoma can be distinguished from normal slides with 94% and 99% AUC scores, respectively, and SCC and thyroid carcinoma can be localized within whole-slide images with 92% and 95% AUC scores, respectively. In conclusion, the outcomes of this thesis work demonstrate that HSI and deep learning methods could aid surgeons and pathologists in detecting head and neck cancers.Ph.D

Translational Functional Imaging in Surgery Enabled by Deep Learning

Many clinical applications currently rely on several imaging modalities such as Positron Emission Tomography (PET), Magnetic Resonance Imaging (MRI), Computed Tomography (CT), etc. All such modalities provide valuable patient data to the clinical staff to aid clinical decision-making and patient care. Despite the undeniable success of such modalities, most of them are limited to preoperative scans and focus on morphology analysis, e.g. tumor segmentation, radiation treatment planning, anomaly detection, etc. Even though the assessment of different functional properties such as perfusion is crucial in many surgical procedures, it remains highly challenging via simple visual inspection. Functional imaging techniques such as Spectral Imaging (SI) link the unique optical properties of different tissue types with metabolism changes, blood flow, chemical composition, etc. As such, SI is capable of providing much richer information that can improve patient treatment and care. In particular, perfusion assessment with functional imaging has become more relevant due to its involvement in the treatment and development of several diseases such as cardiovascular diseases. Current clinical practice relies on Indocyanine Green (ICG) injection to assess perfusion. Unfortunately, this method can only be used once per surgery and has been shown to trigger deadly complications in some patients (e.g. anaphylactic shock). This thesis addressed common roadblocks in the path to translating optical functional imaging modalities to clinical practice. The main challenges that were tackled are related to a) the slow recording and processing speed that SI devices suffer from, b) the errors introduced in functional parameter estimations under changing illumination conditions, c) the lack of medical data, and d) the high tissue inter-patient heterogeneity that is commonly overlooked. This framework follows a natural path to translation that starts with hardware optimization. To overcome the limitation that the lack of labeled clinical data and current slow SI devices impose, a domain- and task-specific band selection component was introduced. The implementation of such component resulted in a reduction of the amount of data needed to monitor perfusion. Moreover, this method leverages large amounts of synthetic data, which paired with unlabeled in vivo data is capable of generating highly accurate simulations of a wide range of domains. This approach was validated in vivo in a head and neck rat model, and showed higher oxygenation contrast between normal and cancerous tissue, in comparison to a baseline using all available bands. The need for translation to open surgical procedures was met by the implementation of an automatic light source estimation component. This method extracts specular reflections from low exposure spectral images, and processes them to obtain an estimate of the light source spectrum that generated such reflections. The benefits of light source estimation were demonstrated in silico, in ex vivo pig liver, and in vivo human lips, where the oxygenation estimation error was reduced when utilizing the correct light source estimated with this method. These experiments also showed that the performance of the approach proposed in this thesis surpass the performance of other baseline approaches. Video-rate functional property estimation was achieved by two main components: a regression and an Out-of-Distribution (OoD) component. At the core of both components is a compact SI camera that is paired with state-of-the-art deep learning models to achieve real time functional estimations. The first of such components features a deep learning model based on a Convolutional Neural Network (CNN) architecture that was trained on highly accurate physics-based simulations of light-tissue interactions. By doing this, the challenge of lack of in vivo labeled data was overcome. This approach was validated in the task of perfusion monitoring in pig brain and in a clinical study involving human skin. It was shown that this approach is capable of monitoring subtle perfusion changes in human skin in an arm clamping experiment. Even more, this approach was capable of monitoring Spreading Depolarizations (SDs) (deoxygenation waves) in the surface of a pig brain. Even though this method is well suited for perfusion monitoring in domains that are well represented with the physics-based simulations on which it was trained, its performance cannot be guaranteed for outlier domains. To handle outlier domains, the task of ischemia monitoring was rephrased as an OoD detection task. This new functional estimation component comprises an ensemble of Invertible Neural Networks (INNs) that only requires perfused tissue data from individual patients to detect ischemic tissue as outliers. The first ever clinical study involving a video-rate capable SI camera in laparoscopic partial nephrectomy was designed to validate this approach. Such study revealed particularly high inter-patient tissue heterogeneity under the presence of pathologies (cancer). Moreover, it demonstrated that this personalized approach is now capable of monitoring ischemia at video-rate with SI during laparoscopic surgery. In conclusion, this thesis addressed challenges related to slow image recording and processing during surgery. It also proposed a method for light source estimation to facilitate translation to open surgical procedures. Moreover, the methodology proposed in this thesis was validated in a wide range of domains: in silico, rat head and neck, pig liver and brain, and human skin and kidney. In particular, the first clinical trial with spectral imaging in minimally invasive surgery demonstrated that video-rate ischemia monitoring is now possible with deep learning