Towards Space-Time Semantics In Two Frames

Abstract. We present a novel, low-level scheme to analyze spatial and temporal change within a local support region. Assuming available region correspondences between two adjacent frames, we divide each region into a regular grid of patches. Depending on the change of an image function inside the patch over time, each patch is assigned weights for the following four labels: “C ” for a constant patch, “O ” when new information originates from outside the support region, “I ” for “inner ” changes, and “N ” for information from neighboring patches. Our method goes beyond optical flow, as it provides an additional semantic level of understanding the changes in space-time. We demonstrate how our novel “COIN” scheme can be used to categorize local space-time events in image pairs, including locally planar support regions, 3D discontinuities, and virtual vs. real crossings of 3D structures.