Context-free hyperspectral image enhancement for wide-field optical biomarker visualization

Many well-known algorithms for the color enhancement of hyperspectral measurements in biomedical imaging are based on statistical assumptions that vary greatly with respect to the proportions of different pixels that appear in a given image, and thus may thwart their application in a surgical environment. This article attempts to explain why this occurs with SVD-based enhancement methods, and proposes the separation of spectral enhancement from analysis. The resulting method, termed affinity-based color enhancement, or ACE for short, achieves multi- and hyperspectral image coloring and contrast based on current spectral affinity metrics that can physically relate spectral data to a particular biomarker. This produces tunable, real-time results which are analogous to the current state-of-the-art algorithms, without suffering any of their inherent context-dependent limitations. Two applications of this method are shown as application examples: vein contrast enhancement and high-precision chromophore concentration estimation.Spanish Ministry of Science, Innovation and Universities (FIS2010-19860, TEC2016-76021-C2-2-R); Spanish Ministry of Economy, Industry and Competitiveness and Instituto de Salud Carlos III (DTS15-00238, DTS17-00055); Instituto de Investigación Valdecilla (IDIVAL) (INNVAL16/02, INNVAL18/23); Spanish Ministry of Education, Culture, and Sports (FPU16/05705

Affinity-based color enhancement methods for contrast enhancement in hyperspectral and multimodal imaging

This work proposes separating data analysis from hyperspectral enhancement or editing, providing a robust, context-independent, fully-tunable framework for biomarker-based contrast in wide-field imaging with a series of reliable properties that could enable its use in guided surgery. Some applications of this method powered by deep learning diagnostics will be discussed and shown.Spanish Ministry of Science, Innovation and Universities (FIS2010-19860, TEC2016-76021-C2-2-R), Spanish Ministry of Economy, Industry and Competitiveness and Instituto de Salud Carlos III (DTS17-00055, DTS15- 00238), Instituto de Investigación Valdecilla (INNVAL16/02, INNVAL18/23), Spanish Ministry of Education, Culture, and Sports (FPU16/05705)

Hyper-Skin: A Hyperspectral Dataset for Reconstructing Facial Skin-Spectra from RGB Images

We introduce Hyper-Skin, a hyperspectral dataset covering wide range of wavelengths from visible (VIS) spectrum (400nm - 700nm) to near-infrared (NIR) spectrum (700nm - 1000nm), uniquely designed to facilitate research on facial skin-spectra reconstruction. By reconstructing skin spectra from RGB images, our dataset enables the study of hyperspectral skin analysis, such as melanin and hemoglobin concentrations, directly on the consumer device. Overcoming limitations of existing datasets, Hyper-Skin consists of diverse facial skin data collected with a pushbroom hyperspectral camera. With 330 hyperspectral cubes from 51 subjects, the dataset covers the facial skin from different angles and facial poses. Each hyperspectral cube has dimensions of 1024$\times$1024$\times$448, resulting in millions of spectra vectors per image. The dataset, carefully curated in adherence to ethical guidelines, includes paired hyperspectral images and synthetic RGB images generated using real camera responses. We demonstrate the efficacy of our dataset by showcasing skin spectra reconstruction using state-of-the-art models on 31 bands of hyperspectral data resampled in the VIS and NIR spectrum. This Hyper-Skin dataset would be a valuable resource to NeurIPS community, encouraging the development of novel algorithms for skin spectral reconstruction while fostering interdisciplinary collaboration in hyperspectral skin analysis related to cosmetology and skin's well-being. Instructions to request the data and the related benchmarking codes are publicly available at: \url{https://github.com/hyperspectral-skin/Hyper-Skin-2023}.Comment: Skin spectral datase

Surgical Guidance for Removal of Cholesteatoma Using a Multispectral 3D-Endoscope

We develop a stereo-multispectral endoscopic prototype in which a filter-wheel is used for surgical guidance to remove cholesteatoma tissue in the middle ear. Cholesteatoma is a destructive proliferating tissue. The only treatment for this disease is surgery. Removal is a very demanding task, even for experienced surgeons. It is very difficult to distinguish between bone and cholesteatoma. In addition, it can even reoccur if not all tissue particles of the cholesteatoma are removed, which leads to undesirable follow-up operations. Therefore, we propose an image-based method that combines multispectral tissue classification and 3D reconstruction to identify all parts of the removed tissue and determine their metric dimensions intraoperatively. The designed multispectral filter-wheel 3D-endoscope prototype can switch between narrow-band spectral and broad-band white illumination, which is technically evaluated in terms of optical system properties. Further, it is tested and evaluated on three patients. The wavelengths 400 nm and 420 nm are identified as most suitable for the differentiation task. The stereoscopic image acquisition allows accurate 3D surface reconstruction of the enhanced image information. The first results are promising, as the cholesteatoma can be easily highlighted, correctly identified, and visualized as a true-to-scale 3D model showing the patient-specific anatomy.Peer Reviewe

Perioperative Hyperspectral Imaging to Assess Mastectomy Skin Flap and DIEP Flap Perfusion in Immediate Autologous Breast Reconstruction: A Pilot Study.

Mastectomy skin flap necrosis (MSFN) and partial DIEP (deep inferior epigastric artery perforator) flap loss represent two frequently reported complications in immediate autologous breast reconstruction. These complications could be prevented when areas of insufficient tissue perfusion are detected intraoperatively. Hyperspectral imaging (HSI) is a relatively novel, non-invasive imaging technique, which could be used to objectively assess tissue perfusion through analysis of tissue oxygenation patterns (StO2%), near-infrared (NIR%), tissue hemoglobin (THI%), and tissue water (TWI%) perfusion indices. This prospective clinical pilot study aimed to evaluate the efficacy of HSI for tissue perfusion assessment and to identify a cut-off value for flap necrosis. Ten patients with a mean age of 55.4 years underwent immediate unilateral autologous breast reconstruction. Prior, during and up to 72 h after surgery, a total of 19 hyperspectral images per patient were acquired. MSFN was observed in 3 out of 10 patients. No DIEP flap necrosis was observed. In all MSFN cases, an increased THI% and decreased StO2%, NIR%, and TWI% were observed when compared to the vital group. StO2% was found to be the most sensitive parameter to detect MSFN with a statistically significant lower mean StO2% (51% in the vital group versus 32% in the necrosis group, p < 0.0001) and a cut-off value of 36.29% for flap necrosis. HSI has the potential to accurately assess mastectomy skin flap perfusion and discriminate between vital and necrotic skin flap during the early postoperative period prior to clinical observation. Although the results should be confirmed in future studies, including DIEP flap necrosis specifically, these findings suggest that HSI can aid clinicians in postoperative mastectomy skin flap and DIEP flap monitoring

An investigation into the complementary capabilities of X-ray computed tomography and hyperspectral imaging of drill core in geometallurgy

The mining industry is faced with the challenge of mining and processing low grade, heterogeneous, and complex ores, a phenomenon known as ore variability. These ores need to be managed at an early operational stage, ideally during drill core exploration, to avoid risks during the project phase (such as project delays and failure) and operational phases (such as plant instabilities), ultimately affecting the cash flow. The discipline of geometallurgy has arisen to manage the risks associated with ore variability by acquiring upfront knowledge of the mineral assemblage and texture before mining and processing. As we head towards the fourth industrial revolution (4IR), machine learning, intensive and automated data derived from drill cores are becoming more common. In this case, using non-destructive, rapid, and inexpensive automated scanning techniques such as 2D hyperspectral imaging (HSI) and 3D Xray computed tomography (XCT) have the potential to be incorporated into the machine learning dataset. Hyperspectral imaging is a critical component of continuous drill core scanning in geometallurgy for identifying problematic minerals in downstream mineral processing, such as the phyllosilicates (e.g., kaolinite, serpentine and talc). However, it only provides 2D imaging of the core, and its mineral identification is limited to minerals that show a definitive spectral response. On the other hand, XCT provides 3D imaging of drill cores, but is more routinely used in research applications and does not independently give the mineral assemblage. Mineral identification and discrimination for XCT is limited and requires prior mineralogical knowledge and sufficient mineral density and attenuation coefficient variation greater than 6%. No systematic study to date appears to have explored how the results from these two techniques can be integrated using a local South African magmatic nickel-copper-platinum group element (Ni-Cu-PGE) ore case study. This opened an opportunity to couple the two techniques to address and emphasize the image scanning techniques for drill core in geometallurgy and to provide further knowledge on the practicality of the HSI and XCT in drill core from image acquisition to processing. Ultimately, the aim is to investigate how well the techniques complement each other for mineral and texture identifications and, if combined, will produce additional mineralogical and textural information. The objective of this study was achieved by moving HSI cores to smaller samples than standard practice to produce 25 mm diameter mini cores instead of standard cores (e.g., 50 mm in diameter). For accurate mineral assemblage and textural characterisation of the drill cores, manual core logging, quantitative evaluation of minerals by scanning electron microscopy (QEMSCAN) and quantitative X-ray diffraction (QXRD) were used as supporting techniques. The results showed HSI scanning on the magmatic Ni-Cu-PGE drill core to be challenging because of pervasive mineral alteration and the nature of the rock types (mafic and ultra-mafic rocks) - providing limited information on the mineral assemblage and texture due to low scanning resolution and pervasive alteration (serpentinisation and chloritization) in the rocks. The limited mineral identification includes mixed-phases (such as serpentine-olivine in visible-shortwave infrared and plagioclase-chlorite in the longwave infrared) and unclassified minerals in the core. The resultant mineral assemblage was comparable to QEMSCAN and QXRD in terms of minerals present with generally similar abundances. However, useful information on the alteration mineralogy can still be extracted, such as the presence of serpentine, chlorite and talc and their association with other silicate minerals. Other parameters such as mineral grades and grain sizes were quantified on MATLAB using specially developed scripts. The interconnected grains could not be separated due to invisible boundaries on the HSI maps. Therefore, only a small number of grains were generated with larger grain size values, likely underestimating the real grain numbers. XCT provided information on valuable high-density minerals (including possible platinum-group minerals (PGMs)) and mineral texture in the cores. Due to extensive alteration in the rocks, discrimination between grey values was, however, challenging. Grey level segmentation into the different mineral groups was also noted to be dependent on the rock type. For example, plagioclase and orthopyroxene were more easily discriminated in the less altered rocks (feldspathic pyroxenite and anorthosite) than the more altered rocks (altered harzburgite and pegmatoidal pyroxenite). The high scanning resolution allowed for the extraction of mineral texture, such as mineral association and grain size distribution (GSD). The 3D XCT derived GSD was slightly coarser than the 2D QEMSCAN derived GSD. The differences in GSD are attributed to a combination of both stereological and sampling effects. However, sufficient information on ore variability can be obtained when using the pertinent scanning parameters and careful segmentation processes. These two techniques provide variable information on the mineral assemblage and texture, such as the identification of silicate minerals (particularly alteration minerals) in HSI and high-density minerals in XCT and good textural information on XCT than HSI. With the information provided, possible image overlapping scenarios of the two techniques were identified: (1) using XCT for high-density minerals, and HSI for silicate identification, (2) using XCT data with good mineral and texture discrimination (silicate associated with sulphides) to map unclassified areas in HSI, (3) is the opposite of the second scenario. Ultimately, the two scanning techniques will likely offer complementary information, although the application of this combined technique for routine work will be limited in practicality. Additionally, more work needs to be carried out with revised scanning and processing to improve the sustainability of the techniques in geometallurgy

Advances in automated tongue diagnosis techniques

This paper reviews the recent advances in a significant constituent of traditional oriental medicinal technology, called tongue diagnosis. Tongue diagnosis can be an effective, noninvasive method to perform an auxiliary diagnosis any time anywhere, which can support the global need in the primary healthcare system. This work explores the literature to evaluate the works done on the various aspects of computerized tongue diagnosis, namely preprocessing, tongue detection, segmentation, feature extraction, tongue analysis, especially in traditional Chinese medicine (TCM). In spite of huge volume of work done on automatic tongue diagnosis (ATD), there is a lack of adequate survey, especially to combine it with the current diagnosis trends. This paper studies the merits, capabilities, and associated research gaps in current works on ATD systems. After exploring the algorithms used in tongue diagnosis, the current trend and global requirements in health domain motivates us to propose a conceptual framework for the automated tongue diagnostic system on mobile enabled platform. This framework will be able to connect tongue diagnosis with the future point-of-care health system

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin