29 research outputs found
CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion
Infrared and visible image fusion targets to provide an informative image by
combining complementary information from different sensors. Existing
learning-based fusion approaches attempt to construct various loss functions to
preserve complementary features from both modalities, while neglecting to
discover the inter-relationship between the two modalities, leading to
redundant or even invalid information on the fusion results. To alleviate these
issues, we propose a coupled contrastive learning network, dubbed CoCoNet, to
realize infrared and visible image fusion in an end-to-end manner. Concretely,
to simultaneously retain typical features from both modalities and remove
unwanted information emerging on the fused result, we develop a coupled
contrastive constraint in our loss function.In a fused imge, its foreground
target/background detail part is pulled close to the infrared/visible source
and pushed far away from the visible/infrared source in the representation
space. We further exploit image characteristics to provide data-sensitive
weights, which allows our loss function to build a more reliable relationship
with source images. Furthermore, to learn rich hierarchical feature
representation and comprehensively transfer features in the fusion process, a
multi-level attention module is established. In addition, we also apply the
proposed CoCoNet on medical image fusion of different types, e.g., magnetic
resonance image and positron emission tomography image, magnetic resonance
image and single photon emission computed tomography image. Extensive
experiments demonstrate that our method achieves the state-of-the-art (SOTA)
performance under both subjective and objective evaluation, especially in
preserving prominent targets and recovering vital textural details.Comment: 25 pages, 16 figure
Image Simulation in Remote Sensing
Remote sensing is being actively researched in the fields of environment, military and urban planning through technologies such as monitoring of natural climate phenomena on the earth, land cover classification, and object detection. Recently, satellites equipped with observation cameras of various resolutions were launched, and remote sensing images are acquired by various observation methods including cluster satellites. However, the atmospheric and environmental conditions present in the observed scene degrade the quality of images or interrupt the capture of the Earth's surface information. One method to overcome this is by generating synthetic images through image simulation. Synthetic images can be generated by using statistical or knowledge-based models or by using spectral and optic-based models to create a simulated image in place of the unobtained image at a required time. Various proposed methodologies will provide economical utility in the generation of image learning materials and time series data through image simulation. The 6 published articles cover various topics and applications central to Remote sensing image simulation. Although submission to this Special Issue is now closed, the need for further in-depth research and development related to image simulation of High-spatial and spectral resolution, sensor fusion and colorization remains.I would like to take this opportunity to express my most profound appreciation to the MDPI Book staff, the editorial team of Applied Sciences journal, especially Ms. Nimo Lang, the assistant editor of this Special Issue, talented authors, and professional reviewers
Structural similarity loss for learning to fuse multi-focus images
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. Convolutional neural networks have recently been used for multi-focus image fusion. However, some existing methods have resorted to adding Gaussian blur to focused images, to simulate defocus, thereby generating data (with ground-truth) for supervised learning. Moreover, they classify pixels as ‘focused’ or ‘defocused’, and use the classified results to construct the fusion weight maps. This then necessitates a series of post-processing steps. In this paper, we present an end-to-end learning approach for directly predicting the fully focused output image from multi-focus input image pairs. The suggested approach uses a CNN architecture trained to perform fusion, without the need for ground truth fused images. The CNN exploits the image structural similarity (SSIM) to calculate the loss, a metric that is widely accepted for fused image quality evaluation. What is more, we also use the standard deviation of a local window of the image to automatically estimate the importance of the source images in the final fused image when designing the loss function. Our network can accept images of variable sizes and hence, we are able to utilize real benchmark datasets, instead of simulated ones, to train our network. The model is a feed-forward, fully convolutional neural network that can process images of variable sizes during test time. Extensive evaluation on benchmark datasets show that our method outperforms, or is comparable with, existing state-of-the-art techniques on both objective and subjective benchmarks
MDLatLRR: A novel decomposition method for infrared and visible image fusion
Image decomposition is crucial for many image processing tasks, as it allows
to extract salient features from source images. A good image decomposition
method could lead to a better performance, especially in image fusion tasks. We
propose a multi-level image decomposition method based on latent low-rank
representation(LatLRR), which is called MDLatLRR. This decomposition method is
applicable to many image processing fields. In this paper, we focus on the
image fusion task. We develop a novel image fusion framework based on MDLatLRR,
which is used to decompose source images into detail parts(salient features)
and base parts. A nuclear-norm based fusion strategy is used to fuse the detail
parts, and the base parts are fused by an averaging strategy. Compared with
other state-of-the-art fusion methods, the proposed algorithm exhibits better
fusion performance in both subjective and objective evaluation.Comment: IEEE Trans. Image Processing 2020, 14 pages, 17 figures, 3 table
A novel multispectral and 2.5D/3D image fusion camera system for enhanced face recognition
The fusion of images from the visible and long-wave infrared (thermal) portions of the spectrum
produces images that have improved face recognition performance under varying lighting conditions.
This is because long-wave infrared images are the result of emitted, rather than reflected,
light and are therefore less sensitive to changes in ambient light. Similarly, 3D and 2.5D images
have also improved face recognition under varying pose and lighting. The opacity of glass to
long-wave infrared light, however, means that the presence of eyeglasses in a face image reduces
the recognition performance.
This thesis presents the design and performance evaluation of a novel camera system which is
capable of capturing spatially registered visible, near-infrared, long-wave infrared and 2.5D depth
video images via a common optical path requiring no spatial registration between sensors beyond
scaling for differences in sensor sizes. Experiments using a range of established face recognition
methods and multi-class SVM classifiers show that the fused output from our camera system not
only outperforms the single modality images for face recognition, but that the adaptive fusion
methods used produce consistent increases in recognition accuracy under varying pose, lighting
and with the presence of eyeglasses