Interactive Style Transfer for Data Visualization and Data Art

This thesis discusses Data Brushes, an interactive web application to explore neural style transfer using models trained on artistic data visualizations. The application invites casual creators to engage with deep convolutional neural networks to co-create custom artworks with a focus on style transfer networks created from canonical and contemporary works of data visualization and data art to demonstrate the versatility and flexibility of the algorithm. In addition to enabling a novel creative workflow, the process of interactively modifying an image via multiple style transfer networks reveals meaningful features encoded within the networks, and provides insight into the effects particular networks have on different images, or different regions within a single image. To evaluate Data Brushes, we gathered expert feedback from participants of a data science symposium and ran an observational study, finding that our application facilitates the creative exploration of neural style transfer for data art and enhances user intuition regarding the expressive range of style transfer features. This thesis explores both the practical uses of such tools for artists as Data Brushes and the interpretive uses of creating such venues for accessibility to computational art, remixing the purpose of data visualizations to be more than just graphical representations of information

Deep Photo Style Transfer

This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style. Our approach builds upon the recent work on painterly transfer that separates style from the content of an image by considering different layers of a neural network. However, as is, this approach is not suitable for photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom fully differentiable energy term. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits

Matching neural paths: transfer from recognition to correspondence search

Many machine learning tasks require finding per-part correspondences between objects. In this work we focus on low-level correspondences - a highly ambiguous matching problem. We propose to use a hierarchical semantic representation of the objects, coming from a convolutional neural network, to solve this ambiguity. Training it for low-level correspondence prediction directly might not be an option in some domains where the ground-truth correspondences are hard to obtain. We show how transfer from recognition can be used to avoid such training. Our idea is to mark parts as "matching" if their features are close to each other at all the levels of convolutional feature hierarchy (neural paths). Although the overall number of such paths is exponential in the number of layers, we propose a polynomial algorithm for aggregating all of them in a single backward pass. The empirical validation is done on the task of stereo correspondence and demonstrates that we achieve competitive results among the methods which do not use labeled target domain data.Comment: Accepted at NIPS 201