Learning Latent Super-Events to Detect Multiple Activities in Videos

In this paper, we introduce the concept of learning latent super-events from activity videos, and present how it benefits activity detection in continuous videos. We define a super-event as a set of multiple events occurring together in videos with a particular temporal organization; it is the opposite concept of sub-events. Real-world videos contain multiple activities and are rarely segmented (e.g., surveillance videos), and learning latent super-events allows the model to capture how the events are temporally related in videos. We design temporal structure filters that enable the model to focus on particular sub-intervals of the videos, and use them together with a soft attention mechanism to learn representations of latent super-events. Super-event representations are combined with per-frame or per-segment CNNs to provide frame-level annotations. Our approach is designed to be fully differentiable, enabling end-to-end learning of latent super-event representations jointly with the activity detector using them. Our experiments with multiple public video datasets confirm that the proposed concept of latent super-event learning significantly benefits activity detection, advancing the state-of-the-arts.Comment: CVPR 201

Predicting Deeper into the Future of Semantic Segmentation

The ability to predict and therefore to anticipate the future is an important attribute of intelligence. It is also of utmost importance in real-time systems, e.g. in robotics or autonomous driving, which depend on visual scene understanding for decision making. While prediction of the raw RGB pixel values in future video frames has been studied in previous work, here we introduce the novel task of predicting semantic segmentations of future frames. Given a sequence of video frames, our goal is to predict segmentation maps of not yet observed video frames that lie up to a second or further in the future. We develop an autoregressive convolutional neural network that learns to iteratively generate multiple frames. Our results on the Cityscapes dataset show that directly predicting future segmentations is substantially better than predicting and then segmenting future RGB frames. Prediction results up to half a second in the future are visually convincing and are much more accurate than those of a baseline based on warping semantic segmentations using optical flow.Comment: Accepted to ICCV 2017. Supplementary material available on the authors' webpage

Learning activity progression in LSTMs for activity detection and early detection

In this work we improve training of temporal deep models to better learn activity progression for activity detection and early detection tasks. Conventionally, when training a Recurrent Neural Network, specifically a Long Short Term Memory (LSTM) model, the training loss only considers classification error. However, we argue that the detection score of the correct activity category, or the detection score margin between the correct and incorrect categories, should be monotonically non-decreasing as the model observes more of the activity. We design novel ranking losses that directly penalize the model on violation of such monotonicities, which are used together with classification loss in training of LSTM models. Evaluation on ActivityNet shows significant benefits of the proposed ranking losses in both activity detection and early detection tasks.https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Ma_Learning_Activity_Progression_CVPR_2016_paper.htmlPublished versio

Dynamic Analysis of Vascular Morphogenesis Using Transgenic Quail Embryos

Background: One of the least understood and most central questions confronting biologists is how initially simple clusters or sheet-like cell collectives can assemble into highly complex three-dimensional functional tissues and organs. Due to the limits of oxygen diffusion, blood vessels are an essential and ubiquitous presence in all amniote tissues and organs. Vasculogenesis, the de novo self-assembly of endothelial cell (EC) precursors into endothelial tubes, is the first step in blood vessel formation [1]. Static imaging and in vitro models are wholly inadequate to capture many aspects of vascular pattern formation in vivo, because vasculogenesis involves dynamic changes of the endothelial cells and of the forming blood vessels, in an embryo that is changing size and shape. Methodology/Principal Findings: We have generated Tie1 transgenic quail lines Tg(tie1:H2B-eYFP) that express H2B-eYFP in all of their endothelial cells which permit investigations into early embryonic vascular morphogenesis with unprecedented clarity and insight. By combining the power of molecular genetics with the elegance of dynamic imaging, we follow the precise patterning of endothelial cells in space and time. We show that during vasculogenesis within the vascular plexus, ECs move independently to form the rudiments of blood vessels, all while collectively moving with gastrulating tissues that flow toward the embryo midline. The aortae are a composite of somatic derived ECs forming its dorsal regions and the splanchnic derived ECs forming its ventral region. The ECs in the dorsal regions of the forming aortae exhibit variable mediolateral motions as they move rostrally; those in more ventral regions show significant lateral-to-medial movement as they course rostrally. Conclusions/Significance: The present results offer a powerful approach to the major challenge of studying the relative role(s) of the mechanical, molecular, and cellular mechanisms of vascular development. In past studies, the advantages of the molecular genetic tools available in mouse were counterbalanced by the limited experimental accessibility needed for imaging and perturbation studies. Avian embryos provide the needed accessibility, but few genetic resources. The creation of transgenic quail with labeled endothelia builds upon the important roles that avian embryos have played in previous studies of vascular development