Scene Reconstruction Beyond Structure-from-Motion and Multi-View Stereo

Image-based 3D reconstruction has become a robust technology for recovering accurate and realistic models of real-world objects and scenes. A common pipeline for 3D reconstruction is to first apply Structure-from-Motion (SfM), which recovers relative poses for the input images and sparse geometry for the scene, and then apply Multi-view Stereo (MVS), which estimates a dense depthmap for each image. While this two-stage process is quite effective in many 3D modeling scenarios, there are limits to what can be reconstructed. This dissertation focuses on three particular scenarios where the SfM+MVS pipeline fails and introduces new approaches to accomplish each reconstruction task. First, I introduce a novel method to recover dense surface reconstructions of endoscopic video. In this setting, SfM can generally provide sparse surface structure, but the lack of surface texture as well as complex, changing illumination often causes MVS to fail. To overcome these difficulties, I introduce a method that utilizes SfM both to guide surface reflectance estimation and to regularize shading-based depth reconstruction. I also introduce models of reflectance and illumination that improve the final result. Second, I introduce an approach for augmenting 3D reconstructions from large-scale Internet photo-collections by recovering the 3D position of transient objects --- specifically, people --- in the input imagery. Since no two images can be assumed to capture the same person in the same location, the typical triangulation constraints enjoyed by SfM and MVS cannot be directly applied. I introduce an alternative method to approximately triangulate people who stood in similar locations, aided by a height distribution prior and visibility constraints provided by SfM. The scale of the scene, gravity direction, and per-person ground-surface normals are also recovered. Finally, I introduce the concept of using crowd-sourced imagery to create living 3D reconstructions --- visualizations of real places that include dynamic representations of transient objects. A key difficulty here is that SfM+MVS pipelines often poorly reconstruct ground surfaces given Internet images. To address this, I introduce a volumetric reconstruction approach that leverages scene scale and person placements. Crowd simulation is then employed to add virtual pedestrians to the space and bring the reconstruction "to life."Doctor of Philosoph

Deformable segmentation of 3D MR prostate images via distributed discriminative dictionary and ensemble learning

Automatic prostate segmentation from MR images is an important task in various clinical applications such as prostate cancer staging and MR-guided radiotherapy planning. However, the large appearance and shape variations of the prostate in MR images make the segmentation problem difficult to solve. Traditional Active Shape/Appearance Model (ASM/AAM) has limited accuracy on this problem, since its basic assumption, i.e., both shape and appearance of the targeted organ follow Gaussian distributions, is invalid in prostate MR images. To this end, the authors propose a sparse dictionary learning method to model the image appearance in a nonparametric fashion and further integrate the appearance model into a deformable segmentation framework for prostate MR segmentation

Deep Blending for Free-Viewpoint Image-Based Rendering

International audienceIBR Algorithm Real surface Reconstruction Fig. 1. Image-based rendering blends contributions from different input images to synthesize a novel view. (a) This blending operation is complex for a variety of reasons. For example, incorrect visibility (e.g., for the non-reconstructed green surface in input view (1)) may result in wrong projections of an image into the novel view. Blending must also account for, e.g., differences in projected resolution ((2) and (3)). Previous methods have used hand-crafted heuristics to overcome these issues. (b) Our method generates a set of ranked contributions (mosaics) from the input images and uses predicted blending weights from a CNN to perform deep blending and synthesize (c) novel views. Our solution significantly reduces visible artifacts compared to (d) previous methods. Free-viewpoint image-based rendering (IBR) is a standing challenge. IBR methods combine warped versions of input photos to synthesize a novel view. The image quality of this combination is directly affected by geometric inaccuracies of multi-view stereo (MVS) reconstruction and by view-and image-dependent effects that produce artifacts when contributions from different input views are blended. We present a new deep learning approach to blending for IBR, in which we use held-out real image data to learn blending weights to combine input photo contributions. Our Deep Blending method requires us to address several challenges to achieve our goal of interactive free-viewpoint IBR navigation. We first need to provide sufficiently accurate geometry so the Convolutional Neural Network (CNN) can succeed in finding correct blending weights. We do this by combining two different MVS reconstructions with complementary accuracy vs. completeness tradeoffs. To tightly integrate learning in an interactive IBR system, we need to adapt our rendering algorithm to produce a fixed number of input layers that can then be blended by the CNN. We generate training data with a variety of captured scenes, using each input photo as ground truth in a held-out approach. We also design the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts. Our results demonstrate free-viewpoint IBR in a wide variety of scenes, clearly surpassing previous methods in visual quality, especially when moving far from the input cameras

High-Resolution Genome-Wide Dissection of the Two Rules of Speciation in Drosophila

Postzygotic reproductive isolation is characterized by two striking empirical patterns. The first is Haldane's rule—the preferential inviability or sterility of species hybrids of the heterogametic (XY) sex. The second is the so-called large X effect—substitution of one species's X chromosome for another's has a disproportionately large effect on hybrid fitness compared to similar substitution of an autosome. Although the first rule has been well-established, the second rule remains controversial. Here, we dissect the genetic causes of these two rules using a genome-wide introgression analysis of Drosophila mauritiana chromosome segments in an otherwise D. sechellia genetic background. We find that recessive hybrid incompatibilities outnumber dominant ones and that hybrid male steriles outnumber all other types of incompatibility, consistent with the dominance and faster-male theories of Haldane's rule, respectively. We also find that, although X-linked and autosomal introgressions are of similar size, most X-linked introgressions cause hybrid male sterility (60%) whereas few autosomal introgressions do (18%). Our results thus confirm the large X effect and identify its proximate cause: incompatibilities causing hybrid male sterility have a higher density on the X chromosome than on the autosomes. We evaluate several hypotheses for the evolutionary cause of this excess of X-linked hybrid male sterility

On the occurrence of cytochrome P450 in viruses

Author Posting. © The Author(s), 2019. This is the author's version of the work. It is posted here by permission of National Academy of Sciences for personal use, not for redistribution. The definitive version was published in Proceedings of the National Academy of Sciences of the United States of America 116(25), (2019):12343-12352, doi:10.1073/pnas.1901080116.Genes encoding cytochrome P450 (CYP; P450) enzymes occur widely in the Archaea, Bacteria, and Eukarya, where they play important roles in metabolism of endogenous regulatory molecules and exogenous chemicals. We now report that genes for multiple and unique P450s occur commonly in giant viruses in the Mimiviridae, Pandoraviridae, and other families in the proposed order Megavirales. P450 genes were also identified in a herpesvirus (Ranid herpesvirus 3) and a phage (Mycobacterium phage Adler). The Adler phage P450 was classified as CYP102L1, and the crystal structure of the open form was solved at 2.5 Å. Genes encoding known redox partners for P450s (cytochrome P450 reductase, ferredoxin and ferredoxin reductase, and flavodoxin and flavodoxin reductase) were not found in any viral genome so far described, implying that host redox partners may drive viral P450 activities. Giant virus P450 proteins share no more than 25% identity with the P450 gene products we identified in Acanthamoeba castellanii, an amoeba host for many giant viruses. Thus, the origin of the unique P450 genes in giant viruses remains unknown. If giant virus P450 genes were acquired from a host, we suggest it could have been from an as yet unknown and possibly ancient host. These studies expand the horizon in the evolution and diversity of the enormously important P450 superfamily. Determining the origin and function of P450s in giant viruses may help to discern the origin of the giant viruses themselves.We thank Dr. David Nes (Texas Tech University) for providing sterols and Dr. Matthieu Legendre and Dr. Chantal Abergel (CNRS, Marseille) for access to the P. celtis sequences. Drs. Irina Arkhipova, Mark Hahn, Judith Luborsky, and Ann Bucklin commented on the manuscript. The research was supported by a USA-UK Fulbright Scholarship and a Royal Society grant (to D.C.L.), the Boston University Superfund Research Program [NIH Grant 5P42ES007381 (to J.J.S. and J.V.G.) and NIH Grant 5U41HG003345 (to J.V.G.)], the European Regional Development Fund and Welsh Government Project BEACON (S.L.K.), the Woods Hole Center for Oceans and Human Health [NIH Grant P01ES021923 and National Science Foundation Grant OCE-1314642 (to J.J.S.)], and NIH Grant R01GM53753 (to T.L.P.).2019-12-0

Mitosis Phase Enrichment with Identification of Mitotic Centromere-Associated Kinesin As a Therapeutic Target in Castration-Resistant Prostate Cancer

The recently described transcriptomic switch to a mitosis program in castration-resistant prostate cancer (CRPC) suggests that mitotic proteins may be rationally targeted at this lethal stage of the disease. In this study, we showed upregulation of the mitosis-phase at the protein level in our cohort of 51 clinical CRPC cases and found centrosomal aberrations to also occur preferentially in CRPC compared with untreated, high Gleason–grade hormone-sensitive prostate cancer (P<0.0001). Expression profiling of chemotherapy-resistant CRPC samples (n = 25) was performed, and the results were compared with data from primary chemotherapy-naïve CRPC (n = 10) and hormone-sensitive prostate cancer cases (n = 108). Our results showed enrichment of mitosis-phase genes and pathways, with progression to both castration-resistant and chemotherapy-resistant disease. The mitotic centromere-associated kinesin (MCAK) was identified as a novel mitosis-phase target in prostate cancer that was overexpressed in multiple CRPC gene-expression datasets. We found concordant gene expression of MCAK between our parent and murine CRPC xenograft pairs and increased MCAK protein expression with clinical progression of prostate cancer to a castration-resistant disease stage. Knockdown of MCAK arrested the growth of prostate cancer cells suggesting its utility as a potential therapeutic target

Observation of associated near-side and away-side long-range correlations in √sNN=5.02 TeV proton-lead collisions with the ATLAS detector

Two-particle correlations in relative azimuthal angle (Δϕ) and pseudorapidity (Δη) are measured in √sNN=5.02  TeV p+Pb collisions using the ATLAS detector at the LHC. The measurements are performed using approximately 1  μb-1 of data as a function of transverse momentum (pT) and the transverse energy (ΣETPb) summed over 3.1<η<4.9 in the direction of the Pb beam. The correlation function, constructed from charged particles, exhibits a long-range (2<|Δη|<5) “near-side” (Δϕ∼0) correlation that grows rapidly with increasing ΣETPb. A long-range “away-side” (Δϕ∼π) correlation, obtained by subtracting the expected contributions from recoiling dijets and other sources estimated using events with small ΣETPb, is found to match the near-side correlation in magnitude, shape (in Δη and Δϕ) and ΣETPb dependence. The resultant Δϕ correlation is approximately symmetric about π/2, and is consistent with a dominant cos⁡2Δϕ modulation for all ΣETPb ranges and particle pT

Measurement of χ c1 and χ c2 production with s√ = 7 TeV pp collisions at ATLAS

The prompt and non-prompt production cross-sections for the χ c1 and χ c2 charmonium states are measured in pp collisions at s√ = 7 TeV with the ATLAS detector at the LHC using 4.5 fb−1 of integrated luminosity. The χ c states are reconstructed through the radiative decay χ c → J/ψγ (with J/ψ → μ + μ −) where photons are reconstructed from γ → e + e − conversions. The production rate of the χ c2 state relative to the χ c1 state is measured for prompt and non-prompt χ c as a function of J/ψ transverse momentum. The prompt χ c cross-sections are combined with existing measurements of prompt J/ψ production to derive the fraction of prompt J/ψ produced in feed-down from χ c decays. The fractions of χ c1 and χ c2 produced in b-hadron decays are also measured