Numerical Demultiplexing of Color Image Sensor Measurements via Non-linear Random Forest Modeling

Due to recent advancements in technology, consumer digital cameras are becoming cheaper and easier to use. These consumer digital cameras, with Bayer color filter arrays (CFAs), allow for simultaneous capture of the red, green and blue (RGB) channels. To achieve higher spectral resolution, multispectral imaging systems use methods such as filter wheels and tunable filters to capture data in a sequential manner. However, in order to capture transient phenomena, one would need to capture spectral information of a 2D scene in a simultaneous manner. Therefore, there has been an on-going trend towards creating a simultaneous multispectral imaging system that uses a conventional consumer digital camera with a Bayer CFA. Such a system allows for a effective imaging of transient or dynamic phenomena with a low-cost and compact system. Currently, the main method to accomplish this is known as Wiener estimation which uses statistical assumptions of the relationship between the incoming spectra and the RGB measurements. However, these assumptions limit the ability to accurately predict the incoming spectra. Therefore, we leverage a comprehensive framework based on numerical demultiplexing of sensor measurements via spectral characterization of the image sensor CFA and non-linear random forest modeling. To create this numerical demultiplexing system we create a forward model from the spectral sensitivity of the imaging system, which is accomplished with a monochrometer. This forward model is then used to create a mapping of 10,000 randomly generated spectra to their corresponding RGB values. This mapping acts as our training set for our non-linear inverse model which utilizes the random forest modeling framework. Having constructed the numerical demultiplexer, we test the performance against the state-of-the-art Wiener estimation for both quantitative and qualitative experiments. In the first set of experiments, we performed a quantitative performance assessment of the proposed framework within a controlled simulation environment. The second set of experiments, validated the observations made from the first set of controlled simulation experiments within a real-world setting. More specifically, we used an icon with different colors as well as a scene of different color flowers to perform quantitative analysis. In these experiments, we show that the proposed numerical demultiplexer outperforms the state-of-the art and is a more robust and reliable way to infer higher spectra from RGB measurements. Having validated the numerical demultiplexer, we use it for two applications which are photoplethysmogrpahic imaging and multispectral microscopy. For photoplethysmogrpahic imaging we found that decomposing the RGB camera measurements into narrow-band spectral information can noticeably improve the prediction of heart rate estimation. In addition, we used the numerical demultiplexer for both a bright-field multispectral microscope as well as a dark-field fluorescence multispectral microscope, which illustrates its potential as a low-cost, portable, point-of-care system

Skimager for the objective erythema estimation in atopic dogs

In this study, the severity of canine skin erythema was assessed objectively for the first time. Atopic dermatitis (AD) is a common canine inflammatory and pruritic skin disease associated with an allergic reaction to exogenous allergens. The monitoring of skin erythema over time with lesion severity scales like the CADESI-4 is an essential diagnostic and research tool, especially for clinical trials. Currently, the erythema assessment is subjective due to visual estimation. In our study, we calculated the erythema index (EI) in 14 atopic dogs based on the analysis of multispectral skin images taken with the Skimager device. The relationship between the EI and a visual erythema estimation was modeled by linear regression with the first-order polynomial. The coefficient of determination (r squared) reached 0.81. Based on such high correlation, we conclude that optical measurements could replace the visual estimation of erythema in atopic dogs and, thus, improve the validity of skin lesion severity scales in dogs

Remote Assessment of the Cardiovascular Function Using Camera-Based Photoplethysmography

Camera-based photoplethysmography (cbPPG) is a novel measurement technique that allows the continuous monitoring of vital signs by using common video cameras. In the last decade, the technology has attracted a lot of attention as it is easy to set up, operates remotely, and offers new diagnostic opportunities. Despite the growing interest, cbPPG is not completely established yet and is still primarily the object of research. There are a variety of reasons for this lack of development including that reliable and autonomous hardware setups are missing, that robust processing algorithms are needed, that application fields are still limited, and that it is not completely understood which physiological factors impact the captured signal. In this thesis, these issues will be addressed. A new and innovative measuring system for cbPPG was developed. In the course of three large studies conducted in clinical and non-clinical environments, the system’s great flexibility, autonomy, user-friendliness, and integrability could be successfully proven. Furthermore, it was investigated what value optical polarization filtration adds to cbPPG. The results show that a perpendicular filter setting can significantly enhance the signal quality. In addition, the performed analyses were used to draw conclusions about the origin of cbPPG signals: Blood volume changes are most likely the defining element for the signal's modulation. Besides the hardware-related topics, the software topic was addressed. A new method for the selection of regions of interest (ROIs) in cbPPG videos was developed. Choosing valid ROIs is one of the most important steps in the processing chain of cbPPG software. The new method has the advantage of being fully automated, more independent, and universally applicable. Moreover, it suppresses ballistocardiographic artifacts by utilizing a level-set-based approach. The suitability of the ROI selection method was demonstrated on a large and challenging data set. In the last part of the work, a potentially new application field for cbPPG was explored. It was investigated how cbPPG can be used to assess autonomic reactions of the nervous system at the cutaneous vasculature. The results show that changes in the vasomotor tone, i.e. vasodilation and vasoconstriction, reflect in the pulsation strength of cbPPG signals. These characteristics also shed more light on the origin problem. Similar to the polarization analyses, they support the classic blood volume theory. In conclusion, this thesis tackles relevant issues regarding the application of cbPPG. The proposed solutions pave the way for cbPPG to become an established and widely accepted technology

Quantitative multispectral imaging differentiates melanoma from seborrheic keratosis

Funding Information: This work was supported by grants from the EFOP-3.6.3-VEKOP-16-2017-00009 (S.B., P.A.) EFOP-3.6.3-VEKOP-16 (S.B.) the ÚNKP-20-4-II-SE-7 (N.K.) and ÚNKP-20-3-I-SE-24 (S.Z.) New National Excellence Program of the Ministry For Innovation and Technology from the source of the National Research, Development and Innovation Fund of Hungary and the European Regional Development Fund projects “Time-resolved autofluorescence methodology for non-invasive skin cancer diagnostics” [No. 1.1.1.2/16/I/001, agreement No. 1.1.1.2/VIAA/1/16/014 (A.L.)] and “Development and clinical validation of a novel cost effective multi-modal methodology for early diagnostics of skin cancers” [No. 1.1.1.2/16/I/001 agreement No. 1.1.1.2/VIAA/1/16/052 (I.L.)] and the National Research, Development and Innovation Office of Hungary—NKFIH (FK_131916, 2019 (Semmelweis University, M.M.)). Publisher Copyright: © 2021 by the authors.Melanoma is a melanocytic tumor that is responsible for the most skin cancer-related deaths. By contrast, seborrheic keratosis (SK) is a very common benign lesion with a clinical picture that may resemble melanoma. We used a multispectral imaging device to distinguish these two entities, with the use of autofluorescence imaging with 405 nm and diffuse reflectance imaging with 525 and 660 narrow-band LED illumination. We analyzed intensity descriptors of the acquired images. These included ratios of intensity values of different channels, standard deviation and minimum/maximum values of intensity of the lesions. The pattern of the lesions was also assessed with the use of particle analysis. We found significantly higher intensity values in SKs compared with melanoma, especially with the use of the autofluorescence channel. Moreover, we found a significantly higher number of particles with high fluorescence in SKs. We created a parameter, the SK index, using these values to differentiate melanoma from SK with a sensitivity of 91.9% and specificity of 57.0%. In conclusion, this imaging technique is potentially applicable to distinguish melanoma from SK based on the analysis of various quantitative parameters. For this application, multispectral imaging could be used as a screening tool by general physicians and non-experts in the everyday practice.publishersversionPeer reviewe

Hyperspectral imaging of human skin aided by artificial neural networks

We developed a compact, hand-held hyperspectral imaging system for 2D neural network-based visualization of skin chromophores and blood oxygenation. State-of-the-art micro-optic multichannel matrix sensor combined with the tunable Fabry-Perot micro interferometer enables a portable diagnostic device sensitive to the changes of the oxygen saturation as well as the variations of blood volume fraction of human skin. Generalized object-oriented Monte Carlo model is used extensively for the training of an artificial neural network utilized for the hyperspectral image processing. In addition, the results are verified and validated via actual experiments with tissue phantoms and human skin in vivo. The proposed approach enables a tool combining both the speed of an artificial neural network processing and the accuracy and flexibility of advanced Monte Carlo modeling. Finally, the results of the feasibility studies and the experimental tests on biotissue phantoms and healthy volunteers are presented

Blind Source Separation for the Processing of Contact-Less Biosignals

(Spatio-temporale) Blind Source Separation (BSS) eignet sich für die Verarbeitung von Multikanal-Messungen im Bereich der kontaktlosen Biosignalerfassung. Ziel der BSS ist dabei die Trennung von (z.B. kardialen) Nutzsignalen und Störsignalen typisch für die kontaktlosen Messtechniken. Das Potential der BSS kann praktisch nur ausgeschöpft werden, wenn (1) ein geeignetes BSS-Modell verwendet wird, welches der Komplexität der Multikanal-Messung gerecht wird und (2) die unbestimmte Permutation unter den BSS-Ausgangssignalen gelöst wird, d.h. das Nutzsignal praktisch automatisiert identifiziert werden kann. Die vorliegende Arbeit entwirft ein Framework, mit dessen Hilfe die Effizienz von BSS-Algorithmen im Kontext des kamera-basierten Photoplethysmogramms bewertet werden kann. Empfehlungen zur Auswahl bestimmter Algorithmen im Zusammenhang mit spezifischen Signal-Charakteristiken werden abgeleitet. Außerdem werden im Rahmen der Arbeit Konzepte für die automatisierte Kanalauswahl nach BSS im Bereich der kontaktlosen Messung des Elektrokardiogramms entwickelt und bewertet. Neuartige Algorithmen basierend auf Sparse Coding erwiesen sich dabei als besonders effizient im Vergleich zu Standard-Methoden.(Spatio-temporal) Blind Source Separation (BSS) provides a large potential to process distorted multichannel biosignal measurements in the context of novel contact-less recording techniques for separating distortions from the cardiac signal of interest. This potential can only be practically utilized (1) if a BSS model is applied that matches the complexity of the measurement, i.e. the signal mixture and (2) if permutation indeterminacy is solved among the BSS output components, i.e the component of interest can be practically selected. The present work, first, designs a framework to assess the efficacy of BSS algorithms in the context of the camera-based photoplethysmogram (cbPPG) and characterizes multiple BSS algorithms, accordingly. Algorithm selection recommendations for certain mixture characteristics are derived. Second, the present work develops and evaluates concepts to solve permutation indeterminacy for BSS outputs of contact-less electrocardiogram (ECG) recordings. The novel approach based on sparse coding is shown to outperform the existing concepts of higher order moments and frequency-domain features

Optical Methods in Sensing and Imaging for Medical and Biological Applications

The recent advances in optical sources and detectors have opened up new opportunities for sensing and imaging techniques which can be successfully used in biomedical and healthcare applications. This book, entitled ‘Optical Methods in Sensing and Imaging for Medical and Biological Applications’, focuses on various aspects of the research and development related to these areas. The book will be a valuable source of information presenting the recent advances in optical methods and novel techniques, as well as their applications in the fields of biomedicine and healthcare, to anyone interested in this subject

Lightfield hyperspectral imaging in neuro-oncology surgery: an IDEAL 0 and 1 study

IntroductionHyperspectral imaging (HSI) has shown promise in the field of intra-operative imaging and tissue differentiation as it carries the capability to provide real-time information invisible to the naked eye whilst remaining label free. Previous iterations of intra-operative HSI systems have shown limitations, either due to carrying a large footprint limiting ease of use within the confines of a neurosurgical theater environment, having a slow image acquisition time, or by compromising spatial/spectral resolution in favor of improvements to the surgical workflow. Lightfield hyperspectral imaging is a novel technique that has the potential to facilitate video rate image acquisition whilst maintaining a high spectral resolution. Our pre-clinical and first-in-human studies (IDEAL 0 and 1, respectively) demonstrate the necessary steps leading to the first in-vivo use of a real-time lightfield hyperspectral system in neuro-oncology surgery.MethodsA lightfield hyperspectral camera (Cubert Ultris ×50) was integrated in a bespoke imaging system setup so that it could be safely adopted into the open neurosurgical workflow whilst maintaining sterility. Our system allowed the surgeon to capture in-vivo hyperspectral data (155 bands, 350–1,000 nm) at 1.5 Hz. Following successful implementation in a pre-clinical setup (IDEAL 0), our system was evaluated during brain tumor surgery in a single patient to remove a posterior fossa meningioma (IDEAL 1). Feedback from the theater team was analyzed and incorporated in a follow-up design aimed at implementing an IDEAL 2a study.ResultsFocusing on our IDEAL 1 study results, hyperspectral information was acquired from the cerebellum and associated meningioma with minimal disruption to the neurosurgical workflow. To the best of our knowledge, this is the first demonstration of HSI acquisition with 100+ spectral bands at a frame rate over 1Hz in surgery.DiscussionThis work demonstrated that a lightfield hyperspectral imaging system not only meets the design criteria and specifications outlined in an IDEAL-0 (pre-clinical) study, but also that it can translate into clinical practice as illustrated by a successful first in human study (IDEAL 1). This opens doors for further development and optimisation, given the increasing evidence that hyperspectral imaging can provide live, wide-field, and label-free intra-operative imaging and tissue differentiation

Camera-Based Heart Rate Extraction in Noisy Environments

Remote photoplethysmography (rPPG) is a non-invasive technique that benefits from video to measure vital signs such as the heart rate (HR). In rPPG estimation, noise can introduce artifacts that distort rPPG signal and jeopardize accurate HR measurement. Considering that most rPPG studies occurred in lab-controlled environments, the issue of noise in realistic conditions remains open. This thesis aims to examine the challenges of noise in rPPG estimation in realistic scenarios, specifically investigating the effect of noise arising from illumination variation and motion artifacts on the predicted rPPG HR. To mitigate the impact of noise, a modular rPPG measurement framework, comprising data preprocessing, region of interest, signal extraction, preparation, processing, and HR extraction is developed. The proposed pipeline is tested on the LGI-PPGI-Face-Video-Database public dataset, hosting four different candidates and real-life scenarios. In the RoI module, raw rPPG signals were extracted from the dataset using three machine learning-based face detectors, namely Haarcascade, Dlib, and MediaPipe, in parallel. Subsequently, the collected signals underwent preprocessing, independent component analysis, denoising, and frequency domain conversion for peak detection. Overall, the Dlib face detector leads to the most successful HR for the majority of scenarios. In 50% of all scenarios and candidates, the average predicted HR for Dlib is either in line or very close to the average reference HR. The extracted HRs from the Haarcascade and MediaPipe architectures make up 31.25% and 18.75% of plausible results, respectively. The analysis highlighted the importance of fixated facial landmarks in collecting quality raw data and reducing noise

Simultaneous Multispectral Imaging: Using Multiview Computational Compressive Sensing

Multispectral imaging is traditionally performed using a combination of an imaging device with a filter bank such as a filter wheel or a form of tunable filter, or a combination of many imaging devices with various spectral beam splitting optics. The complexity and size of these devices seem to be the limiting factor of their adoption and use in various fields that could potentially benefit from this imaging modality. With the advent of nanophotonics, there has been a surge in single camera, snapshot, multispectral imaging exploiting the capabilities of nanotechnology to devise pixel-based spectral filters. This new form of sensing, which can be classified as compressive sensing, has its limitations. One example is the laborious process of fabricating the filter bank and installing it into a detector since the detector fabrication process is completely removed from the filter fabrication process. The work presented here will describe an optical design that would enable a single-camera, simultaneous multispectral imaging via multiview computational compressive sensing. A number of points-of-view (POVs) of the field-of-view (FOV) of the camera are generated and directed through an assortment of spectral pre-filters en route to the camera. The image of each of the POVs is then captured on a different spatial location on the detector. With the spectral response of the detector pixels well characterized, spatial and spectral compressive sensing is performed as the images are recorded. Various computational techniques are used in this work which would: register the images captured from multiple views resulting in even more sparsely sensed images; perform spatial interpolation of the sparsely sampled spectral images; implement hyper-focusing of the images from all POVs captured as some defocusing will happen as the result of the discrepancy in the optical paths in each view; execute numerical dimensionality reduction analysis to extract information from the multispectral images. The spectral imaging capabilities of the device are tested with a collection of fluorescent microspheres. The spectral sensing capability of the device is examined by measuring the fluorescent spectra of adulterated edible oils and demonstrating the ability of the imaging system to differentiate between various types of oil as well as various levels of contamination. Lastly, the system is used to scrutinize samples of black ink from different pen manufacturers, and is able to discriminate between the different inks