Learning Aligned Cross-Modal Representations from Weakly Aligned Data

People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize cross-modal scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for cross-modal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.Comment: Conference paper at CVPR 201

Matterport3D: Learning from RGB-D Data in Indoor Environments

Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification

Reconstructing dynamical networks via feature ranking

Empirical data on real complex systems are becoming increasingly available. Parallel to this is the need for new methods of reconstructing (inferring) the topology of networks from time-resolved observations of their node-dynamics. The methods based on physical insights often rely on strong assumptions about the properties and dynamics of the scrutinized network. Here, we use the insights from machine learning to design a new method of network reconstruction that essentially makes no such assumptions. Specifically, we interpret the available trajectories (data) as features, and use two independent feature ranking approaches -- Random forest and RReliefF -- to rank the importance of each node for predicting the value of each other node, which yields the reconstructed adjacency matrix. We show that our method is fairly robust to coupling strength, system size, trajectory length and noise. We also find that the reconstruction quality strongly depends on the dynamical regime

A 60 yr record of atmospheric carbon monoxide reconstructed from Greenland firn air

We present the first reconstruction of the Northern Hemisphere (NH) high latitude atmospheric carbon monoxide (CO) mole fraction from Greenland firn air. Firn air samples were collected at three deep ice core sites in Greenland (NGRIP in 2001, Summit in 2006 and NEEM in 2008). CO records from the three sites agree well with each other as well as with recent atmospheric measurements, indicating that CO is well preserved in the firn at these sites. CO atmospheric history was reconstructed back to the year 1950 from the measurements using a combination of two forward models of gas transport in firn and an inverse model. The reconstructed history suggests that Arctic CO in 1950 was 140–150 nmol mol-1, which is higher than today's values. CO mole fractions rose by 10–15 nmol mol-1 from 1950 to the 1970s and peaked in the 1970s or early 1980s, followed by a ˜ 30 nmol mol-1 decline to today's levels. We compare the CO history with the atmospheric histories of methane, light hydrocarbons, molecular hydrogen, CO stable isotopes and hydroxyl radicals (OH), as well as with published CO emission inventories and results of a historical run from a chemistry-transport model. We find that the reconstructed Greenland CO history cannot be reconciled with available emission inventories unless unrealistically large changes in OH are assumed. We argue that the available CO emission inventories strongly underestimate historical NH emissions, and fail to capture the emission decline starting in the late 1970s, which was most likely due to reduced emissions from road transportation in North America and Europe

Robustness of proxy-based climate field reconstruction methods

We present results from continued investigations into the fidelity of covariance-based climate field reconstruction (CFR) approaches used in proxy-based climate reconstruction. Our experiments employ synthetic “pseudoproxy” data derived from simulations of forced climate changes over the past millennium. Using networks of these pseudoproxy data, we investigate the sensitivity of CFR performance to signal-to-noise ratios, the noise spectrum, the spatial sampling of pseudoproxy locations, the statistical representation of predictors used, and the diagnostic used to quantify reconstruction skill. Our results reinforce previous conclusions that CFR methods, correctly implemented and applied to suitable networks of proxy data, should yield reliable reconstructions of past climate histories within estimated uncertainties. Our results also demonstrate the deleterious impact of a linear detrending procedure performed recently in certain CFR studies and illustrate flaws in some previously proposed metrics of reconstruction skill

High-density diffuse optical tomography for imaging human brain function

This review describes the unique opportunities and challenges for noninvasive optical mapping of human brain function. Diffuse optical methods offer safe, portable, and radiation free alternatives to traditional technologies like positron emission tomography or functional magnetic resonance imaging (fMRI). Recent developments in high-density diffuse optical tomography (HD-DOT) have demonstrated capabilities for mapping human cortical brain function over an extended field of view with image quality approaching that of fMRI. In this review, we cover fundamental principles of the diffusion of near infrared light in biological tissue. We discuss the challenges involved in the HD-DOT system design and implementation that must be overcome to acquire the signal-to-noise necessary to measure and locate brain function at the depth of the cortex. We discuss strategies for validation of the sensitivity, specificity, and reliability of HD-DOT acquired maps of cortical brain function. We then provide a brief overview of some clinical applications of HD-DOT. Though diffuse optical measurements of neurophysiology have existed for several decades, tremendous opportunity remains to advance optical imaging of brain function to address a crucial niche in basic and clinical neuroscience: that of bedside and minimally constrained high fidelity imaging of brain function