Sequence-to-Sequence Alignment

Abstract

This thesis studies the problem of spatio-temporal alignment of video sequences, i.e., establishing correspondences in time and in space between two di erent video sequences of the same dynamic scene. It shows that temporal variations between image frames such as moving objects, changes in scene illumination, or camera ego-motion, are powerful cues for alignment. Such temporal variations cannot be exploited by standard image-to-image alignment techniques, as they are not captured by a single image, but only by a sequence of images. We show that by folding these new temporal cues and known spatial cues into a single alignment framework, situations which are inherently ambiguous for traditional image-to-image alignment methods are often uniquely resolved by sequence-to-sequence alignment. This gives rise to a wide range of new video applications. These are discussed in this thesis. The thesis investigates the cases where thesequences are recorded byuncalibrated video cameras with xed internal and relative external parameters. However, the notion of sequence-to-sequence alignment/matching is more general, and is not restricted to those cases alone.