Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.Comment: CVPR 2018. The first two authors contributed equally to this work. Project page: http://pix3d.csail.mit.ed

TREE-D-SEEK: A Framework for Retrieving Three-Dimensional Scenes

In this dissertation, a strategy and framework for retrieving 3D scenes is proposed. The strategy is to retrieve 3D scenes based on a unified approach for indexing content from disparate information sources and information levels. The TREE-D-SEEK framework implements the proposed strategy for retrieving 3D scenes and is capable of indexing content from a variety of corpora at distinct information levels. A semantic annotation model for indexing 3D scenes in the TREE-D-SEEK framework is also proposed. The semantic annotation model is based on an ontology for rapid prototyping of 3D virtual worlds. With ongoing improvements in computer hardware and 3D technology, the cost associated with the acquisition, production and deployment of 3D scenes is decreasing. As a consequence, there is a need for efficient 3D retrieval systems for the increasing number of 3D scenes in corpora. An efficient 3D retrieval system provides several benefits such as enhanced sharing and reuse of 3D scenes and 3D content. Existing 3D retrieval systems are closed systems and provide search solutions based on a predefined set of indexing and matching algorithms Existing 3D search systems and search solutions cannot be customized for specific requirements, type of information source and information level. In this research, TREE-D-SEEK—an open, extensible framework for retrieving 3D scenes—is proposed. The TREE-D-SEEK framework is capable of retrieving 3D scenes based on indexing low level content to high-level semantic metadata. The TREE-D-SEEK framework is discussed from a software architecture perspective. The architecture is based on a common process flow derived from indexing disparate information sources. Several indexing and matching algorithms are implemented. Experiments are conducted to evaluate the usability and performance of the framework. Retrieval performance of the framework is evaluated using benchmarks and manually collected corpora. A generic, semantic annotation model is proposed for indexing a 3D scene. The primary objective of using the semantic annotation model in the TREE-D-SEEK framework is to improve retrieval relevance and to support richer queries within a 3D scene. The semantic annotation model is driven by an ontology. The ontology is derived from a 3D rapid prototyping framework. The TREE-D-SEEK framework supports querying by example, keyword based and semantic annotation based query types for retrieving 3D scenes

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.Comment: CVPR 2018. The first two authors contributed equally to this work. Project page: http://pix3d.csail.mit.ed

An Exploration of Digital Sketch Mapping, Interview and Qualitative Analysis to Document a Therapeutic Landscape in Whatcom County

Recent literature cites interest toward utilizing new technologies to unify methods within geography. One area showing promise towards fulfilling this goal is qualitative GIS (QGIS), which combines the methods of social/cultural and spatial/analytical geographers. QGIS research combines sketch maps with GIS and qualitative research methods to uncover “hidden geographies” found within the individual geo-narratives of individuals and within groups of individuals. This thesis explores the merits of using newly developed technology for digital sketch maps acquisition, computer assisted qualitative data analysis (CAQDAS) and qualitative geographic information system (QGIS) analysis for the discovery of “hidden geographies”. The case study demonstrates the utility of touchscreen technology to collect sketch maps and the complementary effect of combining social/cultural and spatial/analytical methods to visualize the hidden geography within the therapeutic landscape of student veterans in Whatcom County, Washington. This exploration also suggests direction for further research using digital sketch map acquisition for gaining insights into other socio-spatial processes that are not captured by traditional geographical analysis methods

CloudJet4BigData: Streamlining Big Data via an Accelerated Socket Interface

Big data needs to feed users with fresh processing results and cloud platforms can be used to speed up big data applications. This paper describes a new data communication protocol (CloudJet) for long distance and large volume big data accessing operations to alleviate the large latencies encountered in sharing big data resources in the clouds. It encapsulates a dynamic multi-stream/multi-path engine at the socket level, which conforms to Portable Operating System Interface (POSIX) and thereby can accelerate any POSIX-compatible applications across IP based networks. It was demonstrated that CloudJet accelerates typical big data applications such as very large database (VLDB), data mining, media streaming and office applications by up to tenfold in real-world tests

Extracting invariant characteristics of sketch maps: Towards place query-by-sketch

In geography, invariant aspects of sketches are essential to study because they reflect the human perception of real-world places. A person's perception of a place can be ex-pressed in sketches. In this article, we quantitatively and qualitatively analyzed the characteristics of single objects and characteristics among objects in sketches and the real world to find reliable invariants that can be used to establish references/correspondences between sketch and world in a matching process. These characteristics include category, shape, name, and relative size of each object. Moreover, quantity and spatial relationships—such as topological, or-dering, and location relationships—among all objects are also analyzed to assess consistency between sketched and actual places. The approach presented in this study extracts the reliable invariants for query-by-sketch and prioritizes their relevance for a sketch-map matching process

LOCATIVE MEDIA, AUGMENTED REALITIES AND THE ORDINARY AMERICAN LANDSCAPE

This dissertation investigates the role of annotative locative media in mediating experiences of place. The overarching impetus motivating this research is the need to bring to bear the theoretical and substantive concerns of cultural landscape studies on the development of a methodological framework for interrogating the ways in which annotative locative media reconfigure experiences of urban landscapes. I take as my empirical cases i) Google Maps with its associated Street View and locational placemark interface, and ii) Layar, an augmented reality platform combining digital mapping and real-time locational augmentation. In the spirit of landscape studies’ longstanding and renewed interest in what may be termed “ordinary” residential landscapes, and reflecting the increasing imbrication of locative media technologies in everyday lives, the empirical research is based in Kenwick, a middleclass, urban residential neighborhood in Lexington, Kentucky. Overall, I present an argument about the need to consider the digital, code (i.e. software), and specifically locative media, in the intellectual context of critical geographies in general and cultural landscape studies in particular