Painterly rendering using human vision

Painterly rendering has been linked to computer vision, but we propose to link it to human vision because perception and painting are two processes that are interwoven. Recent progress in developing computational models allows to establish this link. We show that completely automatic rendering can be obtained by applying four image representations in the visual system: (1) colour constancy can be used to correct colours, (2) coarse background brightness in combination with colour coding in cytochrome-oxidase blobs can be used to create a background with a big brush, (3) the multi-scale line and edge representation provides a very natural way to render fi ner brush strokes, and (4) the multi-scale keypoint representation serves to create saliency maps for Focus-of-Attention, and FoA can be used to render important structures. Basic processes are described, renderings are shown, and important ideas for future research are discussed

Automatic painting with economized strokes

Journal ArticleWe present a method that takes a raster image as input and produces a painting-like image composed of strokes rather than pixels. Unlike previous automatic painting methods, we attempt to use very few brush-strokes. This is accomplished by first segmenting the image into features, finding the medial axes points of these features, converting the medial axes points into ordered lists of image tokens, and finally rendering these lists as brush strokes. Our process creates images reminiscent of modern realist painters who often want an abstract or sketchy quality in their work

Artistic vision: painterly rendering using computer vision techniques

Journal ArticleWe present a method that takes a raster image as input and produces a painting-like image composed of strokes rather than pixels. Unlike previous automatic painting methods, we attempt to keep the number of brush-stroke small. This is accomplished by first segmenting the image into features, finding the medial axes points of these features, converting the medial axes points into ordered lists of image tokens, and finally rendering these lists as brush strokes. Our process creates images reminiscent of modern realist painters who often want an abstract or sketchy quality in their work

Image morphology: from perception to rendering

Complete image ontology can be obtained by formalising a top-down meta-language wich must address all possibilities, from global message and composition to objects and local surface properties

Accurate and discernible photocollages

There currently exist several techniques for selecting and combining images from a digital image library into a single image so that the result meets certain prespecified visual criteria. Image mosaic methods, first explored by Connors and Trivedi[18], arrange library images according to some tiling arrangement, often a regular grid, so that the combination of images, when viewed as a whole, resembles some input target image. Other techniques, such as Autocollage of Rother et al.[78], seek only to combine images in an interesting and visually pleasing manner, according to certain composition principles, without attempting to approximate any target image. Each of these techniques provide a myriad of creative options for artists who wish to combine several levels of meaning into a single image or who wish to exploit the meaning and symbolism contained in each of a large set of images through an efficient and easy process. We first examine the most notable and successful of these methods, and summarize the advantages and limitations of each. We then formulate a set of goals for an image collage system that combines the advantages of these methods while addressing and mitigating the drawbacks. Particularly, we propose a system for creating photocollages that approximate a target image as an aggregation of smaller images, chosen from a large library, so that interesting visual correspondences between images are exploited. In this way, we allow users to create collages in which multiple layers of meaning are encoded, with meaningful visual links between each layer. In service of this goal, we ensure that the images used are as large as possible and are combined in such a way that boundaries between images are not immediately apparent, as in Autocollage. This has required us to apply a multiscale approach to searching and comparing images from a large database, which achieves both speed and accuracy. We also propose a new framework for color post-processing, and propose novel techniques for decomposing images according to object and texture information

Integrated multi-scale architecture of the cortex with application to computer vision

Tese de dout., Engenharia Electrónica e de Computadores, Faculdade de Ciência e Tecnologia, Universidade do Algarve, 2007The main goal of this thesis is to try to understand the functioning of the visual cortex through the development of computational models. In the input layer V1 of the visual cortex there are simple, complex and endstopped cells. These provide a multi-scale representation of objects and scene in terms of lines, edges and keypoints. In this thesis we combine recent progress concerning the development of computational models of these and other cells with processes in higher cortical areas V2 and V4 etc. Three pertinent challenges are discussed: (i) object recognition embedded in a cortical architecture; (ii) brightness perception, and (iii) painterly rendering based on human vision. Specific aspects are Focusof- Attention by means of keypoint-based saliency maps, the dynamic routing of features from V1 through higher cortical areas in order to obtain translation, rotation and size invariance, and the construction of normalized object templates with canonical views in visual memory. Our simulations show that the multi-scale representations can be integrated into a cortical architecture in order to model subsequent processing steps: from segregation, via different categorization levels, until final object recognition is obtained. As for real cortical processing, the system starts with coarse-scale information, refines categorization by using mediumscale information, and employs all scales in recognition. We also show that a 2D brightness model can be based on the multi-scale symbolic representation of lines and edges, with an additional low-pass channel and nonlinear amplitude transfer functions, such that object recognition and brightness perception are combined processes based on the same information. The brightness model can predict many different effects such as Mach bands, grating induction, the Craik-O’Brien-Cornsweet illusion and brightness induction, i.e. the opposite effects of assimilation (White effect) and simultaneous brightness contrast. Finally, a novel application is introduced: painterly rendering has been linked to computer vision, but we propose to link it to human vision because perception and painting are two processes which are strongly interwoven

Stylisation d'objets éclairés par des cartes d'environnement HDR

National audienceDans cet article, nous introduisons un pipeline de rendu permettant de styliser de manière interactive des objets éclairés par des cartes d'environnement à grande dynamique (High-Dynamic Range ou HDR). L'utilisation d'images HDR permet d'améliorer la qualité de certains traitements, comme les segmentations ou l'extraction des détails. De plus, cette architecture permet de combiner facilement des stylisations 2D (sur la carte d'environnement et sur les images) et 3D (sur les objets). Les nouveaux styles que nous présentons sont basés sur ce pipeline et illustrent la flexibilité de notre approche

Colour videos with depth : acquisition, processing and evaluation

The human visual system lets us perceive the world around us in three dimensions by integrating evidence from depth cues into a coherent visual model of the world. The equivalent in computer vision and computer graphics are geometric models, which provide a wealth of information about represented objects, such as depth and surface normals. Videos do not contain this information, but only provide per-pixel colour information. In this dissertation, I hence investigate a combination of videos and geometric models: videos with per-pixel depth (also known as RGBZ videos). I consider the full life cycle of these videos: from their acquisition, via filtering and processing, to stereoscopic display. I propose two approaches to capture videos with depth. The first is a spatiotemporal stereo matching approach based on the dual-cross-bilateral grid – a novel real-time technique derived by accelerating a reformulation of an existing stereo matching approach. This is the basis for an extension which incorporates temporal evidence in real time, resulting in increased temporal coherence of disparity maps – particularly in the presence of image noise. The second acquisition approach is a sensor fusion system which combines data from a noisy, low-resolution time-of-flight camera and a high-resolution colour video camera into a coherent, noise-free video with depth. The system consists of a three-step pipeline that aligns the video streams, efficiently removes and fills invalid and noisy geometry, and finally uses a spatiotemporal filter to increase the spatial resolution of the depth data and strongly reduce depth measurement noise. I show that these videos with depth empower a range of video processing effects that are not achievable using colour video alone. These effects critically rely on the geometric information, like a proposed video relighting technique which requires high-quality surface normals to produce plausible results. In addition, I demonstrate enhanced non-photorealistic rendering techniques and the ability to synthesise stereoscopic videos, which allows these effects to be applied stereoscopically. These stereoscopic renderings inspired me to study stereoscopic viewing discomfort. The result of this is a surprisingly simple computational model that predicts the visual comfort of stereoscopic images. I validated this model using a perceptual study, which showed that it correlates strongly with human comfort ratings. This makes it ideal for automatic comfort assessment, without the need for costly and lengthy perceptual studies

Dataremix: Aesthetic Experiences of Big Data and Data Abstraction

This PhD by published work expands on the contribution to knowledge in two recent large-scale transdisciplinary artistic research projects: ATLAS in silico and INSTRUMENT | One Antarctic Night and their exhibited and published outputs. The thesis reflects upon this practice-based artistic research that interrogates data abstraction: the digitization, datafication and abstraction of culture and nature, as vast and abstract digital data. The research is situated in digital arts practices that engage a combination of big (scientific) data as artistic material, embodied interaction in virtual environments, and poetic recombination. A transdisciplinary and collaborative artistic practice, x-resonance, provides a framework for the hybrid processes, outcomes, and contributions to knowledge from the research. These are purposefully and productively situated at the objective | subjective interface, have potential to convey multiple meanings simultaneously to a variety of audiences and resist disciplinary definition. In the course of the research, a novel methodology emerges, dataremix, which is employed and iteratively evolved through artistic practice to address the research questions: 1) How can a visceral and poetic experience of data abstraction be created? and 2) How would one go about generating an artistically-informed (scientific) discovery? Several interconnected contributions to knowledge arise through the first research question: creation of representational elements for artistic visualization of big (scientific) data that includes four new forms (genomic calligraphy, algorithmic objects as natural specimens, scalable auditory data signatures, and signal objects); an aesthetic of slowness that contributes an extension to the operative forces in Jevbratt’s inverted sublime of looking down and in to also include looking fast and slow; an extension of Corby’s objective and subjective image consisting of “informational and aesthetic components” to novel virtual environments created from big 3 (scientific) data that extend Davies’ poetic virtual spatiality to poetic objective | subjective generative virtual spaces; and an extension of Seaman’s embodied interactive recombinant poetics through embodied interaction in virtual environments as a recapitulation of scientific (objective) and algorithmic processes through aesthetic (subjective) physical gestures. These contributions holistically combine in the artworks ATLAS in silico and INSTRUMENT | One Antarctic Night to create visceral poetic experiences of big data abstraction. Contributions to knowledge from the first research question develop artworks that are visceral and poetic experiences of data abstraction, and which manifest the objective | subjective through art. Contributions to knowledge from the second research question occur through the process of the artworks functioning as experimental systems in which experiments using analytical tools from the scientific domain are enacted within the process of creation of the artwork. The results are “returned” into the artwork. These contributions are: elucidating differences in DNA helix bending and curvature along regions of gene sequences specified as either introns or exons, revealing nuanced differences in BLAST results in relation to genomics sequence metadata, and cross-correlation of astronomical data to identify putative variable signals from astronomical objects for further scientific evaluation