256 research outputs found
Content Authentication for Neural Imaging Pipelines: End-to-end Optimization of Photo Provenance in Complex Distribution Channels
Forensic analysis of digital photo provenance relies on intrinsic traces left
in the photograph at the time of its acquisition. Such analysis becomes
unreliable after heavy post-processing, such as down-sampling and
re-compression applied upon distribution in the Web. This paper explores
end-to-end optimization of the entire image acquisition and distribution
workflow to facilitate reliable forensic analysis at the end of the
distribution channel. We demonstrate that neural imaging pipelines can be
trained to replace the internals of digital cameras, and jointly optimized for
high-fidelity photo development and reliable provenance analysis. In our
experiments, the proposed approach increased image manipulation detection
accuracy from 45% to over 90%. The findings encourage further research towards
building more reliable imaging pipelines with explicit provenance-guaranteeing
properties.Comment: Camera ready + supplement, CVPR'1
Neural Imaging Pipelines - the Scourge or Hope of Forensics?
Forensic analysis of digital photographs relies on intrinsic statistical
traces introduced at the time of their acquisition or subsequent editing. Such
traces are often removed by post-processing (e.g., down-sampling and
re-compression applied upon distribution in the Web) which inhibits reliable
provenance analysis. Increasing adoption of computational methods within
digital cameras further complicates the process and renders explicit
mathematical modeling infeasible. While this trend challenges forensic analysis
even in near-acquisition conditions, it also creates new opportunities. This
paper explores end-to-end optimization of the entire image acquisition and
distribution workflow to facilitate reliable forensic analysis at the end of
the distribution channel, where state-of-the-art forensic techniques fail. We
demonstrate that a neural network can be trained to replace the entire photo
development pipeline, and jointly optimized for high-fidelity photo rendering
and reliable provenance analysis. Such optimized neural imaging pipeline
allowed us to increase image manipulation detection accuracy from approx. 45%
to over 90%. The network learns to introduce carefully crafted artifacts, akin
to digital watermarks, which facilitate subsequent manipulation detection.
Analysis of performance trade-offs indicates that most of the gains can be
obtained with only minor distortion. The findings encourage further research
towards building more reliable imaging pipelines with explicit
provenance-guaranteeing properties.Comment: Manuscript + supplement; currently under review; compressed figures
to minimize file size. arXiv admin note: text overlap with arXiv:1812.0151
Handheld Multi-Frame Super-Resolution
Compared to DSLR cameras, smartphone cameras have smaller sensors, which
limits their spatial resolution; smaller apertures, which limits their light
gathering ability; and smaller pixels, which reduces their signal-to noise
ratio. The use of color filter arrays (CFAs) requires demosaicing, which
further degrades resolution. In this paper, we supplant the use of traditional
demosaicing in single-frame and burst photography pipelines with a multiframe
super-resolution algorithm that creates a complete RGB image directly from a
burst of CFA raw images. We harness natural hand tremor, typical in handheld
photography, to acquire a burst of raw frames with small offsets. These frames
are then aligned and merged to form a single image with red, green, and blue
values at every pixel site. This approach, which includes no explicit
demosaicing step, serves to both increase image resolution and boost signal to
noise ratio. Our algorithm is robust to challenging scene conditions: local
motion, occlusion, or scene changes. It runs at 100 milliseconds per
12-megapixel RAW input burst frame on mass-produced mobile phones.
Specifically, the algorithm is the basis of the Super-Res Zoom feature, as well
as the default merge method in Night Sight mode (whether zooming or not) on
Google's flagship phone.Comment: 24 pages, accepted to Siggraph 2019 Technical Papers progra
Megapixel Photon-Counting Color Imaging using Quanta Image Sensor
Quanta Image Sensor (QIS) is a single-photon detector designed for extremely
low light imaging conditions. Majority of the existing QIS prototypes are
monochrome based on single-photon avalanche diodes (SPAD). Passive color
imaging has not been demonstrated with single-photon detectors due to the
intrinsic difficulty of shrinking the pixel size and increasing the spatial
resolution while maintaining acceptable intra-pixel cross-talk. In this paper,
we present image reconstruction of the first color QIS with a resolution of
pixels, supporting both single-bit and multi-bit photon
counting capability. Our color image reconstruction is enabled by a customized
joint demosaicing-denoising algorithm, leveraging truncated Poisson statistics
and variance stabilizing transforms. Experimental results of the new sensor and
algorithm demonstrate superior color imaging performance for very low-light
conditions with a mean exposure of as low as a few photons per pixel in both
real and simulated images
Joint optimization of multispectral filter arrays and demosaicking for pathological images
A capturing system with multispectral filter array (MSFA) technology is
proposed for shortening the capture time and reducing costs. Therein, a
mosaicked image captured using an MSFA is demosaicked to reconstruct
multispectral images (MSIs). Joint optimization of the spectral sensitivity of
the MSFAs and demosaicking is considered, and pathology-specific multispectral
imaging is proposed. This optimizes the MSFA and the demosaicking matrix by
minimizing the reconstruction error between the training data of a hematoxylin
and eosin-stained pathological tissue and a demosaicked MSI using a cost
function. Initially, the spectral sensitivity of the filter array is set
randomly and the mosaicked image is obtained from the training data.
Subsequently, a reconstructed image is obtained using Wiener estimation. To
minimize the reconstruction error, the spectral sensitivity of the filter array
and the Wiener estimation matrix are optimized iteratively through an
interior-point approach. The effectiveness of the proposed MSFA and
demosaicking is demonstrated by comparing the recovered spectrum and RGB image
with those obtained using a conventional method
Rendering Nighttime Image Via Cascaded Color and Brightness Compensation
Image signal processing (ISP) is crucial for camera imaging, and neural
networks (NN) solutions are extensively deployed for daytime scenes. The lack
of sufficient nighttime image dataset and insights on nighttime illumination
characteristics poses a great challenge for high-quality rendering using
existing NN ISPs. To tackle it, we first built a high-resolution nighttime
RAW-RGB (NR2R) dataset with white balance and tone mapping annotated by expert
professionals. Meanwhile, to best capture the characteristics of nighttime
illumination light sources, we develop the CBUnet, a two-stage NN ISP to
cascade the compensation of color and brightness attributes. Experiments show
that our method has better visual quality compared to traditional ISP pipeline,
and is ranked at the second place in the NTIRE 2022 Night Photography Rendering
Challenge for two tracks by respective People's and Professional Photographer's
choices. The code and relevant materials are avaiable on our website:
https://njuvision.github.io/CBUnet.Comment: Accepted by NTIRE 2022 (CVPR Workshop
Smart Cameras
We review camera architecture in the age of artificial intelligence. Modern
cameras use physical components and software to capture, compress and display
image data. Over the past 5 years, deep learning solutions have become superior
to traditional algorithms for each of these functions. Deep learning enables
10-100x reduction in electrical sensor power per pixel, 10x improvement in
depth of field and dynamic range and 10-100x improvement in image pixel count.
Deep learning enables multiframe and multiaperture solutions that fundamentally
shift the goals of physical camera design. Here we review the state of the art
of deep learning in camera operations and consider the impact of AI on the
physical design of cameras
Mobile Computational Photography: A Tour
The first mobile camera phone was sold only 20 years ago, when taking
pictures with one's phone was an oddity, and sharing pictures online was
unheard of. Today, the smartphone is more camera than phone. How did this
happen? This transformation was enabled by advances in computational
photography -the science and engineering of making great images from small form
factor, mobile cameras. Modern algorithmic and computing advances, including
machine learning, have changed the rules of photography, bringing to it new
modes of capture, post-processing, storage, and sharing. In this paper, we give
a brief history of mobile computational photography and describe some of the
key technological components, including burst photography, noise reduction, and
super-resolution. At each step, we may draw naive parallels to the human visual
system
Learning the image processing pipeline
Many creative ideas are being proposed for image sensor designs, and these
may be useful in applications ranging from consumer photography to computer
vision. To understand and evaluate each new design, we must create a
corresponding image processing pipeline that transforms the sensor data into a
form that is appropriate for the application. The need to design and optimize
these pipelines is time-consuming and costly. We explain a method that combines
machine learning and image systems simulation that automates the pipeline
design. The approach is based on a new way of thinking of the image processing
pipeline as a large collection of local linear filters. We illustrate how the
method has been used to design pipelines for novel sensor architectures in
consumer photography applications
Digital Image Forensics vs. Image Composition: An Indirect Arms Race
The field of image composition is constantly trying to improve the ways in
which an image can be altered and enhanced. While this is usually done in the
name of aesthetics and practicality, it also provides tools that can be used to
maliciously alter images. In this sense, the field of digital image forensics
has to be prepared to deal with the influx of new technology, in a constant
arms-race. In this paper, the current state of this arms-race is analyzed,
surveying the state-of-the-art and providing means to compare both sides. A
novel scale to classify image forensics assessments is proposed, and
experiments are performed to test composition techniques in regards to
different forensics traces. We show that even though research in forensics
seems unaware of the advanced forms of image composition, it possesses the
basic tools to detect it
- …