Temporal Relational Reasoning in Videos

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.Comment: camera-ready version for ECCV'1

New Hardware and Software Innovations (for Volumetric Modeling)

19 pages (includes illustrations and maps)

Continuum Variability of Deeply Embedded Protostars as a Probe of Envelope Structure

Stars may be assembled in large growth spurts, however the evidence for this hypothesis is circumstantial. Directly studying the accretion at the earliest phases of stellar growth is challenging because young stars are deeply embedded in optically thick envelopes, which have spectral energy distributions that peak in the far-IR, where observations are difficult. In this paper, we consider the feasibility of detecting accretion outbursts from these younger stars by investigating the timescales for how the protostellar envelope responds to changes in the emission properties of the central source. The envelope heats up in response to an outburst, brightening at all wavelengths and with the emission peak moving to shorter wavelengths. The timescale for this change depends on the time for dust grains to heat and re-emit photons and the time required for the energy to escape the inner, optically-thick portion of the envelope. We find that the dust response time is much shorter than the photon propagation time and thus the timescale over which the emission varies is set by time delays imposed by geometry. These times are hours to days near the peak of the spectral energy distribution and weeks to months in the sub-mm. The ideal location to quickly detect continuum variability is therefore in the mid- to far-IR, near the peak of the spectral energy distribution, where the change in emission amplitude is largest. Searching for variability in sub-mm continuum emission is also feasible, though with a longer time separation and a weaker relationship between the amount of detected emission amplitude and change in central source luminosity. Such observations would constrain accretion histories of protostars and would help to trace the disk/envelope instabilities that lead to stellar growth.Comment: 25 pages, 6 figures, accepted for publication in the Astrophysical Journa