Layered depth images

In this paper we present a set of efficient image based rendering methods capable of rendering multiple frames per second on a PC. The first method warps Sprites with Depth representing smooth surfaces without the gaps found in other techniques. A second method for more general scenes performs warping from an intermediate representation called a Layered Depth Image (LDI). An LDI is a view of the scene from a single input camera view, but with multiple pixels along each line of sight. The size of the representation grows only linearly with the observed depth complexity in the scene. Moreover, because the LDI data are represented in a single image coordinate system, McMillan's warp ordering algorithm can be successfully adapted. As a result, pixels are drawn in the output image in back-to-front order. No z-buffer is required, so alpha-compositing can be done efficiently without depth sorting. This makes splatting an efficient solution to the resampling problem.Engineering and Applied Science

Fast Poisson blending using multi-splines.

Abstract We present a technique for fast Poisson blending and gradient domain compositing. Instead of using a single piecewise-smooth offset map to perform the blending, we associate a separate map with each input source image. Each individual offset map is itself smoothly varying and can therefore be represented using a low-dimensional spline. The resulting linear system is much smaller than either the original Poisson system or the quadtree spline approximation of a single (unified) offset map. We demonstrate the speed and memory improvements available with our system and apply it to large panoramas. We also show how robustly modeling the multiplicative gain rather than the offset between overlapping images leads to improved results, and how adding a small amount of Laplacian pyramid blending improves the results in areas of inconsistent texture

This is not an apple! Benefits and challenges of applying computer vision to museum collections

The application of computer vision on museum collection data is at an experimental stage with predictions that it will grow in significance and use in the coming years. This research, based on the analysis of five case studies and semi-structured interviews with museum professionals, examined the opportunities and challenges of these technologies, the resources and funding required, and the ethical implications that arise during these initiatives. The case studies examined in this paper are drawn from: The Metropolitan Museum of Art (USA), Princeton University Art Museum (USA), Museum of Modern Art (USA), Harvard Art Museums (USA), Science Museum Group (UK). The research findings highlight the possibilities of computer vision to offer new ways to analyze, describe and present museum collections. However, their actual implementation on digital products is currently very limited due to the lack of resources and the inaccuracies created by algorithms. This research adds to the rapidly evolving field of computer vision within the museum sector and provides recommendations to operationalize the usage of these technologies, increase the transparency on their application, create ethics playbooks to manage potential bias and collaborate across the museum sector

Accidental Light Probes

Recovering lighting in a scene from a single image is a fundamental problem in computer vision. While a mirror ball light probe can capture omnidirectional lighting, light probes are generally unavailable in everyday images. In this work, we study recovering lighting from accidental light probes (ALPs) -- common, shiny objects like Coke cans, which often accidentally appear in daily scenes. We propose a physically-based approach to model ALPs and estimate lighting from their appearances in single images. The main idea is to model the appearance of ALPs by photogrammetrically principled shading and to invert this process via differentiable rendering to recover incidental illumination. We demonstrate that we can put an ALP into a scene to allow high-fidelity lighting estimation. Our model can also recover lighting for existing images that happen to contain an ALP.Comment: CVPR2023. Project website: https://kovenyu.com/ALP

Image Restoration by Matching Gradient Distributions

The restoration of a blurry or noisy image is commonly performed with a MAP estimator, which maximizes a posterior probability to reconstruct a clean image from a degraded image. A MAP estimator, when used with a sparse gradient image prior, reconstructs piecewise smooth images and typically removes textures that are important for visual realism. We present an alternative deconvolution method called iterative distribution reweighting (IDR) which imposes a global constraint on gradients so that a reconstructed image should have a gradient distribution similar to a reference distribution. In natural images, a reference distribution not only varies from one image to another, but also within an image depending on texture. We estimate a reference distribution directly from an input image for each texture segment. Our algorithm is able to restore rich mid-frequency textures. A large-scale user study supports the conclusion that our algorithm improves the visual realism of reconstructed images compared to those of MAP estimators