Infrared Image Super-Resolution: Systematic Review, and Future Trends

Image Super-Resolution (SR) is essential for a wide range of computer vision and image processing tasks. Investigating infrared (IR) image (or thermal images) super-resolution is a continuing concern within the development of deep learning. This survey aims to provide a comprehensive perspective of IR image super-resolution, including its applications, hardware imaging system dilemmas, and taxonomy of image processing methodologies. In addition, the datasets and evaluation metrics in IR image super-resolution tasks are also discussed. Furthermore, the deficiencies in current technologies and possible promising directions for the community to explore are highlighted. To cope with the rapid development in this field, we intend to regularly update the relevant excellent work at \url{https://github.com/yongsongH/Infrared_Image_SR_SurveyComment: Submitted to IEEE TNNL

Visible and NIR Image Fusion Algorithm Based on Information Complementarity

Visible and near-infrared(NIR) band sensors provide images that capture complementary spectral radiations from a scene. And the fusion of the visible and NIR image aims at utilizing their spectrum properties to enhance image quality. However, currently visible and NIR fusion algorithms cannot well take advantage of spectrum properties, as well as lack information complementarity, which results in color distortion and artifacts. Therefore, this paper designs a complementary fusion model from the level of physical signals. First, in order to distinguish between noise and useful information, we use two layers of the weight-guided filter and guided filter to obtain texture and edge layers, respectively. Second, to generate the initial visible-NIR complementarity weight map, the difference maps of visible and NIR are filtered by the extend-DoG filter. After that, the significant region of NIR night-time compensation guides the initial complementarity weight map by the arctanI function. Finally, the fusion images can be generated by the complementarity weight maps of visible and NIR images, respectively. The experimental results demonstrate that the proposed algorithm can not only well take advantage of the spectrum properties and the information complementarity, but also avoid color unnatural while maintaining naturalness, which outperforms the state-of-the-art

A Fast Near-Infrared Image Colorization Deep Learning Mode

Near-infrared(NIR) image colorization is the main research content in the field of current near-infrared image application. It has a wide range of application value. For the problem of image colorization, such as diffuse color and even color error, and can not be automated, A fast near-infrared image colorization model consisting of a lightweight image recognition network module and an image colorization CNN module with a fusion layer, firstly using a lightweight image recognition network for image recognition of near-infrared images, and then selecting from the IamgeNet image library The image of the same class as the scene is used as the training set of the colorized network. After training with the colored CNN module with the fusion layer, the near-infrared image is input as the testing set for colorization. The experimental results show that the color is colored by the algorithm. The image details are clear, the color transfer effect is good and the running speed is fast

Resolution enhancement of thermal infrared images via high resolution class-map and statistical methods

Remote sensing from long stand-off distances offers numerous advantages. As our ability to extract information from data has increased, so has the need for high spatial resolution. Such results are often not available due to technological or financial limitations on the detectors which scan the scene and produce the imagery. Therefore, for many years to come, spatial resolution enhancement using additional data from a variety of sources shall remain popular and cost-effective. Work has been on-going at the Rochester Institute of Technology\u27s Center for Imaging Science in the spatial resolution enhancement of thermal infrared imagery. Background thermal imaging theory is presented and the most recent work by the Digital Imaging and Remote Sensing group is reviewed. A literature search of materials published on the topic since 1985 is included: numerous methods and techniques are presented. Based upon these concepts several areas of study were carried out. All investigations undertaken were confined to cases that ensure radiometric fidelity across image processing operations, since derivation of accurate temperature or emissivity maps necessitate this requirement. Given a low spatial resolution thermal band, these methods produced a high resolution estimate thereof based on enhancement using: (1) a single panchromatic band, (2) a high resolution class-map derived from multi-spectral bands and (3) a statistically based combination of multi-spectral bands

Hard-Hearted Scrolls: A Noninvasive Method for Reading the Herculaneum Papyri

The Herculaneum scrolls were buried and carbonized by the eruption of Mount Vesuvius in A.D. 79 and represent the only classical library discovered in situ. Charred by the heat of the eruption, the scrolls are extremely fragile. Since their discovery two centuries ago, some scrolls have been physically opened, leading to some textual recovery but also widespread damage. Many other scrolls remain in rolled form, with unknown contents. More recently, various noninvasive methods have been attempted to reveal the hidden contents of these scrolls using advanced imaging. Unfortunately, their complex internal structure and lack of clear ink contrast has prevented these efforts from successfully revealing their contents. This work presents a machine learning-based method to reveal the hidden contents of the Herculaneum scrolls, trained using a novel geometric framework linking 3D X-ray CT images with 2D surface imagery of scroll fragments. The method is verified against known ground truth using scroll fragments with exposed text. Some results are also presented of hidden characters revealed using this method, the first to be revealed noninvasively from this collection. Extensions to the method, generalizing the machine learning component to other multimodal transformations, are presented. These are capable not only of revealing the hidden ink, but also of generating rendered images of scroll interiors as if they were photographed in color prior to their damage two thousand years ago. The application of these methods to other domains is discussed, and an additional chapter discusses the Vesuvius Challenge, a $1,000,000+ open research contest based on the dataset built as a part of this work

The fate of colors in the 20th - 21st centuries: preserving the organic colorants in plastic artifacts

Objectos modernos e contemporâneos feitos de plástico são amplamente encontrados no património cultural. Presentemente, a sua preservação levanta questões críticas aos conservadores e cientistas uma vez que estes objectos podem facilmente sofrer degradação num curto espaço de tempo. Um dos fenómenos que pode alterar significativamente a aparência de objectos em plástico é a alteração de cor (descoloração). De um modo geral, a descoloração é habitualmente associada à degradação dos polímeros, contudo, os pigmentos, que são parte integrante das formulações do plástico, também podem desvanecer devido à exposição à luz. A identificação de objetos de plástico com pigmentos sensíveis à luz é um exercício bastante exigente devido à sensibilidade dos mesmos a alterações na cor. A caracterização dos corantes nos plásticos é normalmente realizada através de amostragem, métodos de extração e testagem destrutiva. Como alternativa, esta tese apresenta uma abordagem inovadora e multi- analítica baseada em espectroscopias que foi desenvolvida para a identificação in situ dos pigmentos em plásticos históricos. Esta metodologia compreendeu a utilização de microscopia ótica (MO), microespectrometria por fluorescência de raios X dispersiva de energias (μ-EDXRF), espectroscopia UV-Vis-NIR de reflectância, fotoluminescência (PL) e micro-espectroscopia de Raman (μ-Raman) na análise de obras de arte, objetos industriais e de uso diário, datados de 1950-2000s e pertencentes a coleções Portuguesas. Deste estudo resultou a identificação dos pigmentos comumente presentes na paleta de cor dos coloristas da indústria dos plásticos portuguesa: óxido de ferro (PR 101, α-Fe2O3), molibdato de cromato de chumbo (PR 104, Pb(Cr,Mo,S)O4), vermelho de cádmio (PR 108, Cd(S,Se); PR 113, (Cd,Hg)S), amarelo de cádmio (PY 37, CdS; PY 35; (Cd,Zn)S), branco de titânio (PW 6, TiO2 ambos rutilo e anátase), oxicloreto de bismuto (PW 14, BiOCl) e lacas do pigmento orgânico β-naftol (PR 48, PR 49, PR 53). Adicionalmente, foi também identificado um pigmento fora do comum, o pigmento perlascente plumbonacrite Pb5(CO3)3O(OH)2. Para todos os casos de estudo, μ-Raman foi a ferramenta chave para a caracterização dos pigmentos nos objetos de plástico, aportando dados conclusivos para a identificação dos mesmos. A impressão digital vibracional dos pigmentos orgânicos e inorgânicos foi adquirida com sucesso recorrendo à focagem do laser na superfície das partículas. A aquisição de dados espectrais de pigmentos com concentrações muito baixas (0.1 % a 5%, aproximadamente) à escala micro foi possível através de microscopia confocal, que faz parte do sistema do equipamento de μ-Raman. Adicionalmente, foi também possível obter informação sobre o polímero base (principalmente termoplásticos) e cargas. Os métodos analíticos desenvolvidos neste estudo deverão, em trabalhos futuros, facilitar a obtenção de informação complementar sobre estes objetos de plástico e permitir uma melhor identificação e avaliação do seu estado de conservação. Esta tese foca particularmente objectos de plástico vermelhos visto que estes foram identificados como os mais severamente afetados por alterações de cor. O estado avançado de desvanecimento identificado no pigmento β-naftol PR 53 mostrou a sua fraca estabilidade à luz em formulações de plástico. Esta situação, junto com as alterações de cor descritas em literatura para o pigmento PR 48 em objetos de plástico, sugere uma sensibilidade dos pigmentos vermelhos da família dos β-naftol ao desvanecimento. O PR 53 e os pigmentos vermelhos da família dos β-naftóis são pigmentos históricos facilmente encontrados em objetos do património cultural. No entanto, o conhecimento acerca da sua estabilidade a longo prazo e resistência à foto-degradação é limitado, especialmente para os casos onde os mesmos se encontram em polímeros, sendo que este conhecimento é essencial para a sua preservação. Neste estudo, a quantificação da foto-estabilidade para uma série de pigmentos vermelhos da família dos β-naftol foi realizada pela primeira vez, através do cálculo do rendimento quântico de fotodegradação (ΦR). Os valores obtidos variaram entre 3x10-6 e 4x10-5, indicativo de uma estabilidade relativamente boa à luz por parte das moléculas. Tendo em consideração que a estabilidade dos pigmentos não se limita exclusivamente ao pigmento em si, mas também à sua interação com o meio envolvente, foram realizados ensaios de envelhecimento por exposição à luz (λ>300 nm) do pigmento em solução, em pó e incorporado em polímeros de modo a avaliar o papel do meio na estabilidade à luz dos pigmentos e as vias pelas quais estes se degradam. Verificou-se que o ligante tem um impacto significativo na estabilidade do pigmento uma vez que se foi detetada uma maior sensibilidade à luz dos pigmentos PR 48 e PR 53 quando incorporados nos plásticos, comparativamente ao ensaio do pigmento em pó. Este novo conhecimento irá contribuir para o desenvolvimento de novas estratégias na conservação dos plásticos com estes pigmentos vermelhos fotossensíveis através da previsão do desvanecimento. Espectrometrias de massa (MS) por cromatografia em fase líquida e gasosa foram utilizadas na caracterização dos principais subprodutos da degradação. Observou-se uma fotodegradação significativa e a formação de compostos ftálicos e ftalatos nos pigmentos em solução e em pó.Modern and contemporary objects made of plastics are widely found in cultural heritage. Today, their preser- vation poses critical issues to conservators and scientists, as they can suffer from extensive degradation in a short time period. Color change (discoloration) is one of the alteration phenomena that can significantly affect their appearance. Discoloration is commonly associated with the degradation of polymers. However, pigments within plastics can also fade due to exposure to light. The identification of objects that contain light-sensitive pigments is demanding because of the sensitivity of plastics to color change. Normally sampling, extraction methods and destructive testing are required for the characterization of colorants in plastics. In this work, an innovative multi-analytical spectroscopic approach for the in situ identification of pigments in historical plastics was developed. Optical microscopy (MO), micro-energy dispersive X-ray fluo- rescence (μ-EDXRF), UV-Vis-NIR reflectance, photoluminescence (PL) and Raman microscopy (μ-Raman), were used for the analysis of artworks, industrial and daily objects dated from 1950s-2000s from Portuguese collections. A common colorists’ palette within the Portuguese plastics industry was identified: iron oxide (PR 101, α-Fe2O3), lead chromate molybdate (PR 104, Pb(Cr,Mo,S)O4), cadmium red (PR 108, Cd(S,Se); PR 113, (Cd,Hg)S) and cadmium yellow (PY 37, CdS; PY 35; (Cd,Zn)S) pigments, titanium whites (PW 6, TiO2 both rutile and anatase), bismuth oxychloride (PW 14, BiOCl) and organic β-naphthol lakes (PR 48, PR 49, PR 53). An exceptional pigment found was the pearlescent plumbonacrite pigment Pb5(CO3)3O(OH)2. In all the case studies, μ-Raman was the key analytical tool for pigment characterization in the plastic objects, providing conclusive data for their identification. The vibrational fingerprint of both inorganic and organic pigments was successfully recorded by focusing the laser beam on particle surfaces. The confocal microscopy system used in μ-Raman enabled the collection of spectral data from low concentrations of pigments (ap- proximately 0.1%-5%) on the micro-scale. In addition to pigments, information on the base polymer (mainly thermoplastics) and fillers was obtained. The analytical methods developed will facilitate the acquisition of complementary data from plastics allowing material identification and condition assessment in the future. This thesis focused on red pigmented plastic artifacts, as they were found to be severely faded among the studied objects. The identification of β-naphthol pigment lake PR 53 as a faded pigment highlighted its poor fastness in plastics, that together with the color change of PR 48 in plastic objects, reported in literature, suggests the particular susceptibility of β-naphthol red lakes to fading. PR 53, and the other β-naphthol reds, are historical pigments widely found in cultural heritage. However, little is known about their photodegradation and stability, especially when they are found in polymer media, and this knowledge is essential for their long- term preservation. For the first time, photodegradation quantum yields (ΦR) were calculated for a series of red pigments based on β-naphthol in order to quantify their photo-stability. ΦR values ranging from 3x10-6 to 4x10-5 were obtained, indicating relatively light-stable molecules. Bearing in mind that pigment fastness is not only related to the pigment itself, but also to its interaction with the confined environment, light-aging experiments (λ>300 nm) were conducted in solution, on powders, and in polymers to assess the role of the medium on the lightfastness of the pigments and their degradation pathways. A significant impact of the binder on their stability was found. Indeed, a higher sensitivity to light of PR 48 and PR 53 pigments, when incorporated in plastics than in powder, was observed. This new knowledge will contribute to the prediction of plastic fading and inform effective preventive conservation strategies for objects containing light- sensitive β-naphthol red pigments. Liquid- and gas-chromatography mass spectrometry (MS) were used for the characterization of the main degradation products. Extensive photodegradation was observed with the formation of phthalic compounds and phthalates in both solution and powder phases

How to build a 2d and 3d aerial multispectral map?—all steps deeply explained

UIDB/04111/2020 PCIF/SSI/0102/2017 IF/00325/2015 UIDB/00066/2020The increased development of camera resolution, processing power, and aerial platforms helped to create more cost-efficient approaches to capture and generate point clouds to assist in scientific fields. The continuous development of methods to produce three-dimensional models based on two-dimensional images such as Structure from Motion (SfM) and Multi-View Stereopsis (MVS) allowed to improve the resolution of the produced models by a significant amount. By taking inspiration from the free and accessible workflow made available by OpenDroneMap, a detailed analysis of the processes is displayed in this paper. As of the writing of this paper, no literature was found that described in detail the necessary steps and processes that would allow the creation of digital models in two or three dimensions based on aerial images. With this, and based on the workflow of OpenDroneMap, a detailed study was performed. The digital model reconstruction process takes the initial aerial images obtained from the field survey and passes them through a series of stages. From each stage, a product is acquired and used for the following stage, for example, at the end of the initial stage a sparse reconstruction is produced, obtained by extracting features of the images and matching them, which is used in the following step, to increase its resolution. Additionally, from the analysis of the workflow, adaptations were made to the standard workflow in order to increase the compatibility of the developed system to different types of image sets. Particularly, adaptations focused on thermal imagery were made. Due to the low presence of strong features and therefore difficulty to match features across thermal images, a modification was implemented, so thermal models could be produced alongside the already implemented processes for multispectral and RGB image sets.publishersversionpublishe

Testing the Robustness of Solution Force Fields for MD Simulations on Gaseous Protein Ions.

It is believed that electrosprayed proteins and protein complexes can retain solution-like conformations in the gas phase. However, the lack of high-resolution structure determination methods for gaseous protein ions implies that their properties remain poorly understood. Many practitioners tackle this difficulty by complementing mass spectrometry-based experiments with molecular dynamics (MD) simulations. It is a potential problem that the standard MD force fields used for this purpose (such as OPLS-AA/L and CHARMM) were optimized for solution conditions. The question whether these force fields produce meaningful gas-phase data has received surprisingly little attention. Standard force fields are overpolarized to account for an aqueous environment, i.e., atomic charges and intramolecular dipole moments are ∼20% larger than predicted by gas-phase ab initio methods. Here, we examined the implications of this overpolarization by conducting a series of MD simulations on electrosprayed proteins. Force fields were modified via a charge scaling factor (CSF), while ensuring that the net protein charge remained unchanged. CSF = 0.8 should roughly eliminate water-associated overpolarization. Gas-phase CHARMM simulations on myoglobin with CSF = 0.8 and with unmodified parameters (CSF = 1) yielded similar results, preserving a compact structure that was consistent with ion mobility experiments. Major structural changes caused by weakened charge-dipole and dipole-dipole contacts occurred only when lowering CSF to physically unreasonable values (0.5 and 0.1). Similar results were obtained in mobile-proton OPLS-AA/L simulations on the collision-induced dissociation of transthyretin. Our data support the view that gas-phase MD simulations with standard (solution) force fields are suitable for modeling gaseous protein ions in a semiquantitative manner. Although this is welcome news for the mass spectrometry community, it is hoped that dedicated gas-phase MD force fields will become available in the near future

X-Ray Image Processing and Visualization for Remote Assistance of Airport Luggage Screeners

X-ray technology is widely used for airport luggage inspection nowadays. However, the ever-increasing sophistication of threat-concealment measures and types of threats, together with the natural complexity, inherent to the content of each individual luggage make x-ray raw images obtained directly from inspection systems unsuitable to clearly show various luggage and threat items, particularly low-density objects, which poses a great challenge for airport screeners. This thesis presents efforts spent in improving the rate of threat detection using image processing and visualization technologies. The principles of x-ray imaging for airport luggage inspection and the characteristics of single-energy and dual-energy x-ray data are first introduced. The image processing and visualization algorithms, selected and proposed for improving single energy and dual energy x-ray images, are then presented in four categories: (1) gray-level enhancement, (2) image segmentation, (3) pseudo coloring, and (4) image fusion. The major contributions of this research include identification of optimum combinations of common segmentation and enhancement methods, HSI based color-coding approaches and dual-energy image fusion algorithms —spatial information-based and wavelet-based image fusions. Experimental results generated with these image processing and visualization algorithms are shown and compared. Objective image quality measures are also explored in an effort to reduce the overhead of human subjective assessments and to provide more reliable evaluation results. Two application software are developed − an x-ray image processing application (XIP) and a wireless tablet PC-based remote supervision system (RSS). In XIP, we implemented in a user-friendly GUI the preceding image processing and visualization algorithms. In RSS, we ported available image processing and visualization methods to a wireless mobile supervisory station for screener assistance and supervision. Quantitative and on-site qualitative evaluations for various processed and fused x-ray luggage images demonstrate that using the proposed algorithms of image processing and visualization constitutes an effective and feasible means for improving airport luggage inspection

A comparative analysis of near-infrared image colorization methods for low-power NVIDIA Jetson embedded systems

The near-infrared (NIR) image obtained by an NIR camera is a grayscale image that is inconsistent with the human visual spectrum. It can be difficult to perceive the details of a scene from an NIR scene; thus, a method is required to convert them to visible images, providing color and texture information. In addition, a camera produces so much video data that it increases the pressure on the cloud server. Image processing can be done on an edge device, but the computing resources of edge devices are limited, and their power consumption constraints need to be considered. Graphics Processing Unit (GPU)-based NVIDIA Jetson embedded systems offer a considerable advantage over Central Processing Unit (CPU)-based embedded devices in inference speed. For this study, we designed an evaluation system that uses image quality, resource occupancy, and energy consumption metrics to verify the performance of different NIR image colorization methods on low-power NVIDIA Jetson embedded systems for practical applications. The performance of 11 image colorization methods on NIR image datasets was tested on three different configurations of NVIDIA Jetson boards. The experimental results indicate that the Pix2Pix method performs best, with a rate of 27 frames per second on the Jetson Xavier NX. This performance is sufficient to meet the requirements of real-time NIR image colorization