Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View

We present Im2Pano3D, a convolutional neural network that generates a dense prediction of 3D structure and a probability distribution of semantic labels for a full 360 panoramic view of an indoor scene when given only a partial observation (<= 50%) in the form of an RGB-D image. To make this possible, Im2Pano3D leverages strong contextual priors learned from large-scale synthetic and real-world indoor scenes. To ease the prediction of 3D structure, we propose to parameterize 3D surfaces with their plane equations and train the model to predict these parameters directly. To provide meaningful training supervision, we use multiple loss functions that consider both pixel level accuracy and global context consistency. Experiments demon- strate that Im2Pano3D is able to predict the semantics and 3D structure of the unobserved scene with more than 56% pixel accuracy and less than 0.52m average distance error, which is significantly better than alternative approaches.Comment: Video summary: https://youtu.be/Au3GmktK-S

Evaluating 3D Shape Analysis Methods for Robustness to Rotation Invariance

This paper analyzes the robustness of recent 3D shape descriptors to SO(3) rotations, something that is fundamental to shape modeling. Specifically, we formulate the task of rotated 3D object instance detection. To do so, we consider a database of 3D indoor scenes, where objects occur in different orientations. We benchmark different methods for feature extraction and classification in the context of this task. We systematically contrast different choices in a variety of experimental settings investigating the impact on the performance of different rotation distributions, different degrees of partial observations on the object, and the different levels of difficulty of negative pairs. Our study, on a synthetic dataset of 3D scenes where objects instances occur in different orientations, reveals that deep learning-based rotation invariant methods are effective for relatively easy settings with easy-to-distinguish pairs. However, their performance decreases significantly when the difference in rotations on the input pair is large, or when the degree of observation of input objects is reduced, or the difficulty level of input pair is increased. Finally, we connect feature encodings designed for rotation-invariant methods to 3D geometry that enable them to acquire the property of rotation invariance.Comment: 20th Conference on Robots and Vision (CRV) 202

The Repo homeodomain transcription factor suppresses hematopoiesis in Drosophila and preserves the glial fate

Despite their different origins, Drosophila glia and hemocytes are related cell populations that provide an immune function. Drosophila hemocytes patrol the body cavity and act as macrophages outside the nervous system whereas glia originate from the neuroepithelium and provide the scavenger population of the nervous system. Drosophila glia are hence the functional orthologs of vertebrate microglia, even though the latter are cells of immune origin that subsequently move into the brain during development. Interestingly, the Drosophila immune cells within (glia) and outside the nervous system (hemocytes) require the same transcription factor Glide/Gcm for their development. This raises the issue of how do glia specifically differentiate in the nervous system and hemocytes in the procephalic mesoderm. The Repo homeodomain transcription factor and pan-glial direct target of Glide/Gcm is known to ensure glial terminal differentiation. Here we show that Repo also takes center stage in the process that discriminates between glia and hemocytes. First, Repo expression is repressed in the hemocyte anlagen by mesoderm-specific factors. Second, Repo ectopic activation in the procephalic mesoderm is sufficient to repress the expression of hemocyte-specific genes. Third, the lack of Repo triggers the expression of hemocyte markers in glia. Thus, a complex network of tissue-specific cues biases the potential of Glide/Gcm. These data allow us to revise the concept of fate determinants and help us understand the bases of cell specification. Both sexes were analyzed.SIGNIFICANCE STATEMENTDistinct cell types often require the same pioneer transcription factor, raising the issue of how does one factor trigger different fates. In Drosophila, glia and hemocytes provide a scavenger activity within and outside the nervous system, respectively. While they both require the Glide/Gcm transcription factor, glia originate from the ectoderm, hemocytes from the mesoderm. Here we show that tissue-specific factors inhibit the gliogenic potential of Glide/Gcm in the mesoderm by repressing the expression of the homeodomain protein Repo, a major glial-specific target of Glide/Gcm. Repo expression in turn inhibits the expression of hemocyte-specific genes in the nervous system. These cell-specific networks secure the establishment of the glial fate only in the nervous system and allow cell diversification

Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects

Single-view 3D shape retrieval is a challenging task that is increasingly important with the growth of available 3D data. Prior work that has studied this task has not focused on evaluating how realistic occlusions impact performance, and how shape retrieval methods generalize to scenarios where either the target 3D shape database contains unseen shapes, or the input image contains unseen objects. In this paper, we systematically evaluate single-view 3D shape retrieval along three different axes: the presence of object occlusions and truncations, generalization to unseen 3D shape data, and generalization to unseen objects in the input images. We standardize two existing datasets of real images and propose a dataset generation pipeline to produce a synthetic dataset of scenes with multiple objects exhibiting realistic occlusions. Our experiments show that training on occlusion-free data as was commonly done in prior work leads to significant performance degradation for inputs with occlusion. We find that that by first pretraining on our synthetic dataset with occlusions and then finetuning on real data, we can significantly outperform models from prior work and demonstrate robustness to both unseen 3D shapes and unseen objects

MOPA: Modular Object Navigation with PointGoal Agents

We propose a simple but effective modular approach MOPA (Modular ObjectNav with PointGoal agents) to systematically investigate the inherent modularity of the object navigation task in Embodied AI. MOPA consists of four modules: (a) an object detection module trained to identify objects from RGB images, (b) a map building module to build a semantic map of the observed objects, (c) an exploration module enabling the agent to explore the environment, and (d) a navigation module to move to identified target objects. We show that we can effectively reuse a pretrained PointGoal agent as the navigation model instead of learning to navigate from scratch, thus saving time and compute. We also compare various exploration strategies for MOPA and find that a simple uniform strategy significantly outperforms more advanced exploration methods