Calipso: Physics-based Image and Video Editing through CAD Model Proxies

We present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. Running these simulations allows us to apply new, unseen forces to move or deform selected objects, change physical parameters such as mass or elasticity, or even add entire new objects that interact with the rest of the underlying scene. In Calipso, the user makes edits directly in 3D; these edits are processed by the simulation and then transfered to the target 2D content using shape-to-image correspondences in a photo-realistic rendering process. To align the CAD models, we introduce an efficient CAD-to-image alignment procedure that jointly minimizes for rigid and non-rigid alignment while preserving the high-level structure of the input shape. Moreover, the user can choose to exploit image flow to estimate scene motion, producing coherent physical behavior with ambient dynamics. We demonstrate Calipso's physics-based editing on a wide range of examples producing myriad physical behavior while preserving geometric and visual consistency.Comment: 11 page

Efficient Representations of Object Geometry for Reinforcement Learning of Interactive Grasping Policies

Grasping objects of different shapes and sizes - a foundational, effortless skill for humans - remains a challenging task in robotics. Although model-based approaches can predict stable grasp configurations for known object models, they struggle to generalize to novel objects and often operate in a non-interactive open-loop manner. In this work, we present a reinforcement learning framework that learns the interactive grasping of various geometrically distinct real-world objects by continuously controlling an anthropomorphic robotic hand. We explore several explicit representations of object geometry as input to the policy. Moreover, we propose to inform the policy implicitly through signed distances and show that this is naturally suited to guide the search through a shaped reward component. Finally, we demonstrate that the proposed framework is able to learn even in more challenging conditions, such as targeted grasping from a cluttered bin. Necessary pre-grasping behaviors such as object reorientation and utilization of environmental constraints emerge in this case. Videos of learned interactive policies are available at https://maltemosbach.github. io/geometry_aware_grasping_policies

Searching force-closure optimal grasps of articulated 2D objects with n links

This paper proposes a method that finds a locally optimal grasp of an articulated 2D object with n links considering frictionless contacts. The surface of each link of the object is represented by a finite set of points, thus it may have any shape. The proposed approach finds, first, an initial force-closure grasp and from it starts an iterative search of a local optimum grasp. The quality measure considered in this work is the largest perturbation wrench that a grasp can resist with independence of the direction of the perturbation. The approach has been implemented and some illustrative examples are included in the article.Postprint (published version

Articulation-aware Canonical Surface Mapping

We tackle the tasks of: 1) predicting a Canonical Surface Mapping (CSM) that indicates the mapping from 2D pixels to corresponding points on a canonical template shape, and 2) inferring the articulation and pose of the template corresponding to the input image. While previous approaches rely on keypoint supervision for learning, we present an approach that can learn without such annotations. Our key insight is that these tasks are geometrically related, and we can obtain supervisory signal via enforcing consistency among the predictions. We present results across a diverse set of animal object categories, showing that our method can learn articulation and CSM prediction from image collections using only foreground mask labels for training. We empirically show that allowing articulation helps learn more accurate CSM prediction, and that enforcing the consistency with predicted CSM is similarly critical for learning meaningful articulation.Comment: To appear at CVPR 2020, project page https://nileshkulkarni.github.io/acsm

Spatial Transfiguration: Anamorphic Mixed-Reality in the Virtual Reality Panorama

Spatial illusion and immersion was achieved in Renaissance painting through the manipulation of linear perspective’s pictorial conventions and painterly technique. The perceptual success of a painted trompe l’œil, its ability to fool the observer into believing they were viewing a real three-dimensional scene, was constrained by the limited immersive capacity of the two-dimensional painted canvas. During the baroque period however, artists began to experiment with the amalgamation of the ‘real’ space occupied by the observer together with the pictorial space enveloped by the painting’s picture plane: real and pictorial space combined into one pictorial composition resulting in a hybridised ‘mixed-reality’. Today, the way architects, and designers generally, use the QuickTime Virtual Reality panorama to represent spaces of increasing visual density have much to learn from the way in which Renaissance and baroque artists manipulated the three-dimensional characteristics of the picture plane in order to offer more convincing spatial illusions. This paper outlines the conceptual development of the QuickTime VR panorama by Ken Turkowski and the Apple Advanced Technology Group during the late 1980s. Further, it charts the technical methods of the Virtual Reality panorama’s creation in order to reflect upon the VR panorama’s geometric construction and range and effectiveness of spatial illusion. Finally, through a brief analysis of Hans Holbein’s Ambassadors [1533] and Andrea Pozzo’s nave painting in Sant ‘Ignazio [1691-94] this paper proposes an alternative conceptual model for the pictorial construction of the VR panorama that is innovatively based upon an anamorphic ‘mixed-reality’

Learning in a Landscape: Simulation-building as Reflexive Intervention

This article makes a dual contribution to scholarship in science and technology studies (STS) on simulation-building. It both documents a specific simulation-building project, and demonstrates a concrete contribution to interdisciplinary work of STS insights. The article analyses the struggles that arise in the course of determining what counts as theory, as model and even as a simulation. Such debates are especially decisive when working across disciplinary boundaries, and their resolution is an important part of the work involved in building simulations. In particular, we show how ontological arguments about the value of simulations tend to determine the direction of simulation-building. This dynamic makes it difficult to maintain an interest in the heterogeneity of simulations and a view of simulations as unfolding scientific objects. As an outcome of our analysis of the process and reflections about interdisciplinary work around simulations, we propose a chart, as a tool to facilitate discussions about simulations. This chart can be a means to create common ground among actors in a simulation-building project, and a support for discussions that address other features of simulations besides their ontological status. Rather than foregrounding the chart's classificatory potential, we stress its (past and potential) role in discussing and reflecting on simulation-building as interdisciplinary endeavor. This chart is a concrete instance of the kinds of contributions that STS can make to better, more reflexive practice of simulation-building.Comment: 37 page