Intrinsic Images by Clustering

International audienceDecomposing an input image into its intrinsic shading and reflectance components is a long-standing ill-posed problem. We present a novel algorithm that requires no user strokes and works on a single image. Based on simple assumptions about its reflectance and luminance, we first find clusters of similar reflectance in the image, and build a linear system describing the connections and relations between them. Our assumptions are less restrictive than widely-adopted Retinex-based approaches, and can be further relaxed in conflicting situations. The resulting system is robust even in the presence of areas where our assumptions do not hold. We show a wide variety of results, including natural images, objects from the MIT dataset and texture images, along with several applications, proving the versatility of our method

Practical intrinsic image decomposition

El conocimiento previo de las luces y los materiales que componen una escena es el primer paso para su total captura y reconstrucción. Sin embargo, obtener esta información a partir de una sencilla fotografía no es una tarea fácil. Cuando capturamos una imagen del mundo real, toda la información de color, geometría e iluminación se integra en el sensor de nuestra cámara dando como resultado un conjunto de píxeles RGB. Estos valores carecen de toda la información geométrica de la imagen que nos permitiría realizar tareas como reiluminación o cambio de materiales. El objetivo de la presente Tesis Fin de Máster ha sido estudiar y resolver este problema que comúnmente se conoce como descomposición de una imagen en sus componentes intrínsecas, y que consiste en obtener, para una única imagen, la parte correspondiente a iluminación y la que corresponde con reflectancia (textura, color). Actualmente, la mayoría de los métodos que resuelven este problema requieren excesiva interacción del usuario. De este modo, un usuario inexperto o la ausencia de información pueden dar lugar a malas descomposiciones. En este trabajo se ha tratado de obtener una solución eficiente, con resultados de alta calidad y robustos, partiendo de una única imagen de la escena a descomponer. En particular, se han estudiado dos soluciones distintas. La primera solución propuesta, denominada Intrinsic Images by Clustering, ha sido publicada en la revista Computer Graphics Forum cuyo JCR 2011 index es 35/83 (Q2) en la categoría de Computer Science, Software Engineering, con un índice de impacto de 5 años de 1.634. El método propuesto requiere una única imagen para funcionar y se basa en la detección en la imagen de zonas de la misma reflectancia. Con esta información se construye un sistema de ecuaciones lineales donde se describen las conexiones y las relaciones entre ellas. Este algoritmo constituye el actual estado del arte en métodos de separación en imágenes intrínsecas a partir de una sola imagen de entrada. La segunda solución planteada ha sido desarrollada en colaboración con la empresa Adobe Systems Inc. bajo la supervisión del Dr. Sunil Hadap. El nuevo método se basa en la observación de que los gradientes de reflectancia de la imagen siguen una dirección invariante y relativa a la fuente de luz. De este modo, estimando la dirección invariante a partir de la información de color de la imagen, podríamos ser capaces de desambiguar los cambios debidos a reflectancia y los cambios debidos sombreado. Los resultados obtenidos de este primer estudio del algoritmo bajo un entorno controlado concluyen que el algoritmo tiene mucho potencial, y se abre una interesante vía para futuras investigaciones mediante la combinación con otras técnicas complementarias que aporten nueva información de la escena. Descomponer una imagen en sus componentes intrínsecas es todavía un problema abierto con múltiples aplicaciones potenciales. Con esta investigación se ha contribuido con un paso más hacia la solución global y óptima. Además, se concluye que futuras investigaciones deberían enfocarse a obtener un algoritmo que requiera la menor interacción posible, ya que debido a la complejidad del problema es matemáticamente imposible obtener una solución única y sin interacción para todos los escenarios

CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering

Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire. We explore the use of synthetic data for training CNN-based intrinsic image decomposition models, then applying these learned models to real-world images. To that end, we present \ICG, a new, large-scale dataset of physically-based rendered images of scenes with full ground truth decompositions. The rendering process we use is carefully designed to yield high-quality, realistic images, which we find to be crucial for this problem domain. We also propose a new end-to-end training method that learns better decompositions by leveraging \ICG, and optionally IIW and SAW, two recent datasets of sparse annotations on real-world images. Surprisingly, we find that a decomposition network trained solely on our synthetic data outperforms the state-of-the-art on both IIW and SAW, and performance improves even further when IIW and SAW data is added during training. Our work demonstrates the suprising effectiveness of carefully-rendered synthetic data for the intrinsic images task.Comment: Paper for 'CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering' published in ECCV, 201

Reflectance Adaptive Filtering Improves Intrinsic Image Estimation

Separating an image into reflectance and shading layers poses a challenge for learning approaches because no large corpus of precise and realistic ground truth decompositions exists. The Intrinsic Images in the Wild~(IIW) dataset provides a sparse set of relative human reflectance judgments, which serves as a standard benchmark for intrinsic images. A number of methods use IIW to learn statistical dependencies between the images and their reflectance layer. Although learning plays an important role for high performance, we show that a standard signal processing technique achieves performance on par with current state-of-the-art. We propose a loss function for CNN learning of dense reflectance predictions. Our results show a simple pixel-wise decision, without any context or prior knowledge, is sufficient to provide a strong baseline on IIW. This sets a competitive baseline which only two other approaches surpass. We then develop a joint bilateral filtering method that implements strong prior knowledge about reflectance constancy. This filtering operation can be applied to any intrinsic image algorithm and we improve several previous results achieving a new state-of-the-art on IIW. Our findings suggest that the effect of learning-based approaches may have been over-estimated so far. Explicit prior knowledge is still at least as important to obtain high performance in intrinsic image decompositions.Comment: CVPR 201

Unsupervised Deep Single-Image Intrinsic Decomposition using Illumination-Varying Image Sequences

Machine learning based Single Image Intrinsic Decomposition (SIID) methods decompose a captured scene into its albedo and shading images by using the knowledge of a large set of known and realistic ground truth decompositions. Collecting and annotating such a dataset is an approach that cannot scale to sufficient variety and realism. We free ourselves from this limitation by training on unannotated images. Our method leverages the observation that two images of the same scene but with different lighting provide useful information on their intrinsic properties: by definition, albedo is invariant to lighting conditions, and cross-combining the estimated albedo of a first image with the estimated shading of a second one should lead back to the second one's input image. We transcribe this relationship into a siamese training scheme for a deep convolutional neural network that decomposes a single image into albedo and shading. The siamese setting allows us to introduce a new loss function including such cross-combinations, and to train solely on (time-lapse) images, discarding the need for any ground truth annotations. As a result, our method has the good properties of i) taking advantage of the time-varying information of image sequences in the (pre-computed) training step, ii) not requiring ground truth data to train on, and iii) being able to decompose single images of unseen scenes at runtime. To demonstrate and evaluate our work, we additionally propose a new rendered dataset containing illumination-varying scenes and a set of quantitative metrics to evaluate SIID algorithms. Despite its unsupervised nature, our results compete with state of the art methods, including supervised and non data-driven methods.Comment: To appear in Pacific Graphics 201