Enhanced Soft 3D Reconstruction Method with an Iterative Matching Cost Update Using Object Surface Consensus

In this paper, we propose a multi-view stereo matching method, EnSoft3D (Enhanced Soft 3D Reconstruction) to obtain dense and high-quality depth images. Multi-view stereo is one of the high-interest research areas and has wide applications. Motivated by the Soft3D reconstruction method, we introduce a new multi-view stereo matching scheme. The original Soft3D method is introduced for novel view synthesis, while occlusion-aware depth is also reconstructed by integrating the matching costs of the Plane Sweep Stereo (PSS) and soft visibility volumes. However, the Soft3D method has an inherent limitation because the erroneous PSS matching costs are not updated. To overcome this limitation, the proposed scheme introduces an update process of the PSS matching costs. From the object surface consensus volume, an inverse consensus kernel is derived, and the PSS matching costs are iteratively updated using the kernel. The proposed EnSoft3D method reconstructs a highly accurate 3D depth image because both the multi-view matching cost and soft visibility are updated simultaneously. The performance of the proposed method is evaluated by using structured and unstructured benchmark datasets. Disparity error is measured to verify 3D reconstruction accuracy, and both PSNR and SSIM are measured to verify the simultaneous enhancement of view synthesis

Simultaneous Video Retrieval and Alignment

With the growth of the video streaming industry, video retrieval and video alignment are facing high levels of demand. Several studies have demonstrated the feasibility of these methods for various problems related to video retrieval and alignment independently, but testing in a unified framework has never been done. However, in real-world applications, it is also simultaneously necessary not only to find which video pairs are similar (video retrieval), but also to align the positions of the pairs that are related (video alignment). In this paper, we present a novel task: simultaneous video retrieval and alignment. As a solution to this task, a Simultaneous video Retrieval and Alignment framework, abbreviated as SRA, is proposed, which is a two-stage approach consisting of a foreground proposal stage and a downstream stage to efficiently process untrimmed videos. Furthermore, two criteria are suggested to support the new task: a metric mAP@J assessing how highly related videos are ranked and how well relevant positions are assigned in those videos, and a dataset FIVR+A that includes video-level relationships and hierarchical segment-level annotations. Finally, we conduct multi-pronged analyses to assess how our approach handles the new task in various experiments