Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging

Hyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are physically impossible to acquire in intraoperative settings. In this work, we present a fully unsupervised hyperspectral image demosaicking algorithm which only requires exemplar snapshot images for training purposes. We regard hyperspectral demosaicking as an ill-posed linear inverse problem which we solve using a deep neural network. We take advantage of the spectral correlation occurring in natural scenes to design a novel inter spectral band regularisation term based on spatial gradient consistency. By combining our proposed term with standard regularisation techniques and exploiting a standard data fidelity term, we obtain an unsupervised loss function for training deep neural networks, which allows us to achieve real-time hyperspectral image demosaicking. Quantitative results on hyperspetral image datasets show that our unsupervised demosaicking approach can achieve similar performance to its supervised counter-part, and significantly outperform linear demosaicking. A qualitative user study on real snapshot hyperspectral surgical images confirms the results from the quantitative analysis. Our results suggest that the proposed unsupervised algorithm can achieve promising hyperspectral demosaicking in real-time thus advancing the suitability of the modality for intraoperative use

Multispectral Imaging for the Analysis of Materials and Pathologies in Civil Engineering, Constructions and Natural Spaces

Tesis por compendio de publicaciones[EN] Multispectral imaging is a non-destructive technique that combines imaging and spectroscopy to analyse the spectral behaviour of materials and land covers through the use of geospatial sensors. These sensors collect both spatial and spectral information for a given scenario and a spectral range, so that, their graphical representation elements (pixels or points) store the spectral properties of the radiation reflected by the material sample or land cover. The term multispectral imaging is commonly associated with satellite imaging, but the application range extends to other scales as close-range photogrammetry through the use of sensors on board of airborne systems (gliders, trikes, drones, etc.) or through their use at ground level. Its usefulness has been proved in a variety of disciplines from topography, geology, atmospheric science to forestry or agriculture. The present thesis is framed within close-range remote sensing applied to the civil engineering, cultural heritage and natural resources fields via multispectral image analysis. Specifically, the main goal of this research work is to study and analyse the radiometric behaviour of different natural and artificial covers by combining several sensors recording data in the visible and infrared ranges of the spectrum. The research lines have not been limited to the 2D data analysis, but in some cases 3D intensity data have been integrated with 2D data from active (terrestrial laser scanners) and passive (multispectral digital cameras) sensors in order to analyse different materials and possible associated pathologies, getting more comprehensive products due to the metric that 3D brings to 2D data. Works began with the radiometric calibration of the active and passive sensors used by the vicarious calibration method. The calibrations were carried out through MULRACS, a multispectral radiometric calibration software developed for this purpose (see Appendix B). After the calibration process, active and passive sensors were used together for the discretization of sedimentary rocks and detecting pathologies, as moisture, in façades and in civil structures. Finally, the Doctoral Thesis concludes with a theoretical book chapter in which all the know-how and expertise arising during this research stage have been compiled.[ES]Las imágenes multiespectrales se constituyen como técnica no destructiva que combina imagen y espectroscopía para analizar el comportamiento espectral de distintos materiales y superficies terrestres a través del uso de sensores geoespaciales. Estos sensores adquieren tanto información espacial como espectral para un escenario y un rango espectral dados de tal forma sus unidades de representación gráfica (ya sean píxeles o puntos) registran las propiedades de la radiación reflejada para cada material o cobertura a estudiar y longitud de onda. Las imágenes multiespectrales no solo se limitan a las observaciones satelitales a las que tradicionalmente se vinculan, sino que tienen un campo de aplicación más amplio gracias a los estudios de rango cercano realizados a través del uso de sensores tanto embarcados en sistemas aéreos (planeadores, paramotores, drones, etc.) como a nivel terreno. Su utilidad ha sido demostrada en multitud de disciplinas; desde la topografía, geología, aerología, hasta la ingeniería forestal o la agricultura entre otros. La presente tesis se enmarca dentro de la teledetección de rango cercano aplicada a la ingeniería civil, el patrimonio cultural y los recursos naturales a través del análisis multiespectral de imágenes. Concretamente, el principal objetivo de este trabajo de investigación consiste en el estudio y análisis del comportamiento radiométrico de distintas coberturas naturales y artificiales mediante el uso combinado de distintos sensores que registran información espectral en los rangos visible e infrarrojo del espectro electromagnético. Las líneas de investigación no se han limitado al análisis de datos bidimensionales (imágenes) sino que en algunos casos se han integrado datos de intensidad registrados en 3D a través de sensores activos (láser escáner terrestres) con datos 2D capturados con sensores pasivos (cámaras digitales convencionales y multiespectrales) con el objetivo de analizar diferentes materiales y posibles patologías asociadas a los mismos ofreciendo resultados más completos gracias a la métrica que los datos 3D aportan a los datos 2D. Los trabajos comenzaron con la calibración radiométrica de los sensores por el método de calibración vicario. Las calibraciones fueron resueltas gracias al uso del software MULRACS, un software para la calibración radiométrica multiespectral desarrollado durante este periodo para tal fin (ver Apéndice B). Tras el proceso de calibración, se combinó el uso de sensores activos y pasivos para la diferenciación de distintos tipos de rocas sedimentarias y la detección de patologías, como humedades, en fachadas de edificios históricos y en estructuras de ingeniería civil. Finalmente, la Tesis Doctoral concluye con un capítulo teórico de libro en el cual se recopilan todos los conocimientos y experiencias adquiridos durante este periodo de investigación

Real-time multispectral fluorescence and reflectance imaging for intraoperative applications

Fluorescence guided surgery supports doctors by making unrecognizable anatomical or pathological structures become recognizable. For instance, cancer cells can be targeted with one fluorescent dye whereas muscular tissue, nerves or blood vessels can be targeted by other dyes to allow distinction beyond conventional color vision. Consequently, intraoperative imaging devices should combine multispectral fluorescence with conventional reflectance color imaging over the entire visible and near-infrared spectral range at video rate, which remains a challenge. In this work, the requirements for such a fluorescence imaging device are analyzed in detail. A concept based on temporal and spectral multiplexing is developed, and a prototype system is build. Experiments and numerical simulations show that the prototype fulfills the design requirements and suggest future improvements. The multispectral fluorescence image stream is processed to present fluorescent dye images to the surgeon using linear unmixing. However, artifacts in the unmixed images may not be noticed by the surgeon. A tool is developed in this work to indicate unmixing inconsistencies on a per pixel and per frame basis. In-silico optimization and a critical review suggest future improvements and provide insight for clinical translation