6 research outputs found
Quanta Burst Photography
Single-photon avalanche diodes (SPADs) are an emerging sensor technology
capable of detecting individual incident photons, and capturing their
time-of-arrival with high timing precision. While these sensors were limited to
single-pixel or low-resolution devices in the past, recently, large (up to 1
MPixel) SPAD arrays have been developed. These single-photon cameras (SPCs) are
capable of capturing high-speed sequences of binary single-photon images with
no read noise. We present quanta burst photography, a computational photography
technique that leverages SPCs as passive imaging devices for photography in
challenging conditions, including ultra low-light and fast motion. Inspired by
recent success of conventional burst photography, we design algorithms that
align and merge binary sequences captured by SPCs into intensity images with
minimal motion blur and artifacts, high signal-to-noise ratio (SNR), and high
dynamic range. We theoretically analyze the SNR and dynamic range of quanta
burst photography, and identify the imaging regimes where it provides
significant benefits. We demonstrate, via a recently developed SPAD array, that
the proposed method is able to generate high-quality images for scenes with
challenging lighting, complex geometries, high dynamic range and moving
objects. With the ongoing development of SPAD arrays, we envision quanta burst
photography finding applications in both consumer and scientific photography.Comment: A version with better-quality images can be found on the project
webpage: http://wisionlab.cs.wisc.edu/project/quanta-burst-photography
SoDaCam: Software-defined Cameras via Single-Photon Imaging
Reinterpretable cameras are defined by their post-processing capabilities
that exceed traditional imaging. We present "SoDaCam" that provides
reinterpretable cameras at the granularity of photons, from photon-cubes
acquired by single-photon devices. Photon-cubes represent the spatio-temporal
detections of photons as a sequence of binary frames, at frame-rates as high as
100 kHz. We show that simple transformations of the photon-cube, or photon-cube
projections, provide the functionality of numerous imaging systems including:
exposure bracketing, flutter shutter cameras, video compressive systems, event
cameras, and even cameras that move during exposure. Our photon-cube
projections offer the flexibility of being software-defined constructs that are
only limited by what is computable, and shot-noise. We exploit this flexibility
to provide new capabilities for the emulated cameras. As an added benefit, our
projections provide camera-dependent compression of photon-cubes, which we
demonstrate using an implementation of our projections on a novel compute
architecture that is designed for single-photon imaging.Comment: Accepted at ICCV 2023 (oral). Project webpage can be found at
https://wisionlab.com/project/sodacam
Robotic Burst Imaging for Light-Constrained 3D Reconstruction
This thesis proposes a novel input scheme, robotic burst, to improve vision-based 3D reconstruction for robots operating in low-light conditions, where existing state-of-the-art robotic vision algorithms struggle due to low signal-to-noise ratio in low-light images. We aim to improve the correspondence search stage of feature-based reconstruction using robotic burst imaging, including burst-merged images, a burst feature finder, and an end-to-end learning-based feature extractor. Firstly, we establish the use of robotic burst imaging to compute burst-merged images for feature-based reconstruction. We then develop a burst feature finder that locates features with well-defined scale and apparent motion on a burst to deal with limitations of burst-merged images such as misalignment at strong noise. To improve feature matches in burst-based reconstruction, we also present an end-to-end learning-based feature extractor that finds well-defined scale features directly on light-constrained bursts.
We evaluate our methods against state-of-the-art reconstruction methods for conventional imaging that uses both classical and learning-based feature extractors. We validate our novel input scheme using burst imagery captured on a robotic arm and drones. We demonstrate progressive improvements in low-light reconstruction using our burst-based methods against conventional approaches and overall, converging 90% of all scenes captured in millilux conditions that otherwise converge with 10% success rate using conventional methods. This work opens up new avenues for applications, including autonomous driving and drone delivery at night, mining, and behavioral studies on nocturnal animals