ImageSpirit: Verbal Guided Image Parsing

Humans describe images in terms of nouns and adjectives while algorithms operate on images represented as sets of pixels. Bridging this gap between how humans would like to access images versus their typical representation is the goal of image parsing, which involves assigning object and attribute labels to pixel. In this paper we propose treating nouns as object labels and adjectives as visual attribute labels. This allows us to formulate the image parsing problem as one of jointly estimating per-pixel object and attribute labels from a set of training images. We propose an efficient (interactive time) solution. Using the extracted labels as handles, our system empowers a user to verbally refine the results. This enables hands-free parsing of an image into pixel-wise object/attribute labels that correspond to human semantics. Verbally selecting objects of interests enables a novel and natural interaction modality that can possibly be used to interact with new generation devices (e.g. smart phones, Google Glass, living room devices). We demonstrate our system on a large number of real-world images with varying complexity. To help understand the tradeoffs compared to traditional mouse based interactions, results are reported for both a large scale quantitative evaluation and a user study.Comment: http://mmcheng.net/imagespirit

Geometric and Textural Augmentation for Domain Gap Reduction

Research has shown that convolutional neural networks for object recognition are vulnerable to changes in depiction because learning is biased towards the low-level statistics of texture patches. Recent works concentrate on improving robustness by applying style transfer to training examples to mitigate against over-fitting to one depiction style. These new approaches improve performance, but they ignore the geometric variations in object shape that real art exhibits: artists deform and warp objects for artistic effect. Motivated by this observation, we propose a method to reduce bias by jointly increasing the texture and geometry diversities of the training data. In effect, we extend the visual object class to include examples with shape changes that artists use. Specifically, we learn the distribution of warps that cover each given object class. Together with augmenting textures based on a broad distribution of styles, we show by experiments that our method improves performance on several cross-domain benchmarks

Perceptual 3D rendering based on principles of analytical cubism

Cataloged from PDF version of article.Cubism, pioneered by Pablo Picasso and Georges Braque, was a breakthrough in art, influencing artists to abandon existing traditions. In this paper, we present a novel approach for cubist rendering of 3D synthetic environments. Rather than merely imitating cubist paintings, we apply the main principles of analytical cubism to 3D graphics rendering. In this respect, we develop a new cubist camera providing an extended view, and a perceptually based spatial imprecision technique that keeps the important regions of the scene within a certain area of the output. Additionally, several methods to provide a painterly style are applied. We demonstrate the effectiveness of our extending view method by comparing the visible face counts in the images rendered by the cubist camera model and the traditional perspective camera. Besides, we give an overall discussion of final results and apply user tests in which users compare our results very well with analytical cubist paintings but not synthetic cubist paintings. (c) 2012 Elsevier Ltd. All rights reserved

Geometric and Textural Augmentation for Domain Gap Reduction

Research has shown that convolutional neural networks for object recognition are vulnerable to changes in depiction because learning is biased towards the low-level statistics of texture patches. Recent works concentrate on improving robustness by applying style transfer to training examples to mitigate against over-fitting to one depiction style. These new approaches improve performance, but they ignore the geometric variations in object shape that real art exhibits: artists deform and warp objects for artistic effect. Motivated by this observation, we propose a method to reduce bias by jointly increasing the texture and geometry diversities of the training data. In effect, we extend the visual object class to include examples with shape changes that artists use. Specifically, we learn the distribution of warps that cover each given object class. Together with augmenting textures based on a broad distribution of styles, we show by experiments that our method improves performance on several cross-domain benchmarks

Video sculpture:spatio-temporal warping

In this thesis the concept behind our notion of video sculpture is to imagine an image sequence or movie as a three dimensional volume. We then also imagine that there is a frameset that traverses the image sequence to give us what we commonly think of as a video or movie. In the ordinary sense this frameset moves through an image sequence in a completely timeparallel linear fashion. In video sculpture, we free the frameset from these bounds so that we can sample space and time in completely unorthodox ways. We can view the whenwhere in previously unforeseen perspectives. Slices of the video environment can simultaneously reveal both past and future actions within a single frame. Building on this free representation of video spacetime, we then wrest the frame once more from the present constraints of topography and/or topology. The frame can bend and twist and jump and dive. The view of a fading quadratic surface cutting through two scenes makes for a beautiful curtain transition. We present a framework and an implementation for modeling the frame as it passes through the image sequence volume object

Shadow art

"To them, I said, the truth would be literally nothing but the shadows of the images." - Plato, The Republic Shadow art is a unique form of sculptural art where the 2D shadows cast by a 3D sculpture are essential for the artistic effect. We introduce computational tools for the creation of shadow art and propose a design process where the user can directly specify the desired shadows by providing a set of binary images and corresponding projection information. Since multiple shadow images often contradict each other, we present a geometric optimization that computes a 3D shadow volume whose shadows best approximate the provided input images. Our analysis shows that this optimization is essential for obtaining physically realizable 3D sculptures. The resulting shadow volume can then be modified with a set of interactive editing tools that automatically respect the often intricate shadow constraints. We demonstrate the potential of our system with a number of complex 3D shadow art sculptures that go beyond what is seen in contemporary art pieces. © 2009 ACM