10 research outputs found
Recaptured Raw Screen Image and Video Demoir\'eing via Channel and Spatial Modulations
Capturing screen contents by smartphone cameras has become a common way for
information sharing. However, these images and videos are often degraded by
moir\'e patterns, which are caused by frequency aliasing between the camera
filter array and digital display grids. We observe that the moir\'e patterns in
raw domain is simpler than those in sRGB domain, and the moir\'e patterns in
raw color channels have different properties. Therefore, we propose an image
and video demoir\'eing network tailored for raw inputs. We introduce a
color-separated feature branch, and it is fused with the traditional
feature-mixed branch via channel and spatial modulations. Specifically, the
channel modulation utilizes modulated color-separated features to enhance the
color-mixed features. The spatial modulation utilizes the feature with large
receptive field to modulate the feature with small receptive field. In
addition, we build the first well-aligned raw video demoir\'eing
(RawVDemoir\'e) dataset and propose an efficient temporal alignment method by
inserting alternating patterns. Experiments demonstrate that our method
achieves state-of-the-art performance for both image and video demori\'eing. We
have released the code and dataset in https://github.com/tju-chengyijia/VD_raw
Image and Video Forensics
Nowadays, images and videos have become the main modalities of information being exchanged in everyday life, and their pervasiveness has led the image forensics community to question their reliability, integrity, confidentiality, and security. Multimedia contents are generated in many different ways through the use of consumer electronics and high-quality digital imaging devices, such as smartphones, digital cameras, tablets, and wearable and IoT devices. The ever-increasing convenience of image acquisition has facilitated instant distribution and sharing of digital images on digital social platforms, determining a great amount of exchange data. Moreover, the pervasiveness of powerful image editing tools has allowed the manipulation of digital images for malicious or criminal ends, up to the creation of synthesized images and videos with the use of deep learning techniques. In response to these threats, the multimedia forensics community has produced major research efforts regarding the identification of the source and the detection of manipulation. In all cases (e.g., forensic investigations, fake news debunking, information warfare, and cyberattacks) where images and videos serve as critical evidence, forensic technologies that help to determine the origin, authenticity, and integrity of multimedia content can become essential tools. This book aims to collect a diverse and complementary set of articles that demonstrate new developments and applications in image and video forensics to tackle new and serious challenges to ensure media authenticity
Medical Image Modality Synthesis and Resolution Enhancement Based on Machine Learning Techniques
To achieve satisfactory performance from automatic medical image analysis
algorithms such as registration or segmentation, medical imaging data with
the desired modality/contrast and high isotropic resolution are preferred, yet
they are not always available. We addressed this problem in this thesis using
1) image modality synthesis and 2) resolution enhancement.
The first contribution of this thesis is computed tomography (CT)-tomagnetic
resonance imaging (MRI) image synthesis method, which was developed
to provide MRI when CT is the only modality that is acquired. The
main challenges are that CT has poor contrast as well as high noise in soft
tissues and that the CT-to-MR mapping is highly nonlinear. To overcome these
challenges, we developed a convolutional neural network (CNN) which is a
modified U-net. With this deep network for synthesis, we developed the first
segmentation method that provides detailed grey matter anatomical labels on
CT neuroimages using synthetic MRI.
The second contribution is a method for resolution enhancement for a
common type of acquisition in clinical and research practice, one in which
there is high resolution (HR) in the in-plane directions and low resolution (LR)
in the through-plane direction. The challenge of improving the through-plane resolution for such acquisitions is that the state-of-art convolutional neural
network (CNN)-based super-resolution methods are sometimes not applicable
due to lack of external LR/HR paired training data. To address this challenge,
we developed a self super-resolution algorithm called SMORE and its iterative
version called iSMORE, which are CNN-based yet do not require LR/HR
paired training data other than the subject image itself. SMORE/iSMORE
create training data from the HR in-plane slices of the subject image itself, then
train and apply CNNs to through-plane slices to improve spatial resolution
and remove aliasing. In this thesis, we perform SMORE/iSMORE on multiple
simulated and real datasets to demonstrate their accuracy and generalizability.
Also, SMORE as a preprocessing step is shown to improve segmentation
accuracy.
In summary, CT-to-MR synthesis, SMORE, and iSMORE were demonstrated
in this thesis to be effective preprocessing algorithms for visual quality
and other automatic medical image analysis such as registration or segmentation
Image Quality Improvement of Medical Images using Deep Learning for Computer-aided Diagnosis
Retina image analysis is an important screening tool for early detection of multiple dis eases such as diabetic retinopathy which greatly impairs visual function. Image analy sis and pathology detection can be accomplished both by ophthalmologists and by the
use of computer-aided diagnosis systems. Advancements in hardware technology led to
more portable and less expensive imaging devices for medical image acquisition. This
promotes large scale remote diagnosis by clinicians as well as the implementation of
computer-aided diagnosis systems for local routine disease screening. However, lower cost equipment generally results in inferior quality images. This may jeopardize the
reliability of the acquired images and thus hinder the overall performance of the diagnos tic tool. To solve this open challenge, we carried out an in-depth study on using different
deep learning-based frameworks for improving retina image quality while maintaining
the underlying morphological information for the diagnosis. Our results demonstrate
that using a Cycle Generative Adversarial Network for unpaired image-to-image trans lation leads to successful transformations of retina images from a low- to a high-quality
domain. The visual evidence of this improvement was quantitatively affirmed by the two
proposed validation methods. The first used a retina image quality classifier to confirm a
significant prediction label shift towards a quality enhance. On average, a 50% increase
of images being classified as high-quality was verified. The second analysed the perfor mance modifications of a diabetic retinopathy detection algorithm upon being trained
with the quality-improved images. The latter led to strong evidence that the proposed
solution satisfies the requirement of maintaining the images’ original information for
diagnosis, and that it assures a pathology-assessment more sensitive to the presence of
pathological signs. These experimental results confirm the potential effectiveness of our
solution in improving retina image quality for diagnosis. Along with the addressed con tributions, we analysed how the construction of the data sets representing the low-quality
domain impacts the quality translation efficiency. Our findings suggest that by tackling
the problem more selectively, that is, constructing data sets more homogeneous in terms
of their image defects, we can obtain more accentuated quality transformations
Review : Deep learning in electron microscopy
Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy