Video Segmentation with Motion Smoothness

本論文探討並實作了一個基於圖分割演算法的互動式影片分割系。近來,基於圖分割演算法的圖像分割、影片分割於電腦圖學與電視覺研究界甚為普遍。然而,絕大多數的關連研究僅僅使用了影片身的色彩資訊,作為主要的分割依據。這在前景與背景有部分區域色彩上甚為相似的狀況下,容易產生錯誤。而不幸地,這樣的條件不罕見,特別是當拍攝對象並非在棚拍等人工環境之下拍攝,而是日常場景作為背景之時。因此,在本論文之中,我們提出了除了色之外的依據進行影片分割的演算法。我們觀察到前景的動向與背景常是相當不同的,因此,選擇結合色彩以及動向資訊共同進行影片割。此外,本系統尚且擴充了原本採用於圖片分割領域的漸進式分,使其能夠用於影片分割。最後,我們將本系統的結果與關連研究行了比較,以實例證實了本系統的效能確實優於既往研究。In this thesis, we present an interactive graph cut based video segmenta-ion system. Recently, graph cut based segmentation tools become prevelantor image/video segmentation problem. However, most of the previous workseal with color information only. Such systems could fail under the conditionhat there are regions similar in color between foreground and background.nfortunately, it is usutally hard to avoid. Especially when the objects areilmed under a natural environment. To make it more pratical to use, weropose criterion other than color to conduct the segmentation. Through ourbservation, motion is a natural choice, since it is usually the case that fore-round and background has different motion pattern. Moreover, we also ex-end the Progressive Cut to the temporal-spatial video volume. Experimentshows that by combining color and motion information, our system outper-orms the previous works.中文摘要 ibstract iii Introduction 1 1.1 Background 1 1.2 Problem Statement 3 1.3 Thesis Organization 3 Related work 5 2.1 Traditional Approaches 5 2.1.1 Chroma Keying 5 2.1.2 Difference Matte 5 2.1.3 Silhouette Tracking 6 2.2 Modern Approaches 6 2.2.1 Interactive approaches in image domain 6 2.2.2 Graph cuts approaches in video domain 7 2.2.3 Video matting 7 2.2.4 Segmentation that handles the occlusion condition 8 2.3 Graph Cuts 8 2.4 Optical Flow 9 Graph Cuts 11 3.1 Video Segmentation and Graph Cuts 11 3.2 Problem Statement 12 3.3 Algorithm Summary 12 3.3.1 Local Minimum in Large Moves 12 3.3.2 Encode the Energy Function Specification in Graphs 13 3.4 Implementation Detail 14 Video Segmentation with Motion Smoothness 17 4.1 System Overview 17 4.2 Interactive User Interface 18 4.3 Optical Flow Calculation and Refinement 19 4.4 3D Graph Cut Segmentation 20 4.4.1 3D Graph Construction 20 4.4.2 Encode Data Term 20 4.4.3 Encode Color Smoothness Term 21 4.4.4 Encode Temporal Smoothness Term 21 4.4.5 Encode Motion Smoothness Term 22 4.4.6 3D Graph Cut Optimization 22 4.5 Local Refinement by 3D Progressive Cut 23 4.5.1 Some Observation from User Strokes 23 4.5.2 User Term 25 Results 27 5.1 Experiment Results 27 5.1.1 Results that Can be Segmented Properly by Color Term Only 27 5.1.2 Results that Better with Motion Smoothness Term 29 5.2 Applications 41 5.3 Limitation 41 5.4 Implementation Details 43 Conclusion and Future Work 45 6.1 Conclusion 45 6.2 Future Work 45ibliography 4