Real-time self-adaptive deep stereo

Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set, e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective training/tuning in any target domain, thus making this setup impractical for many applications. Instead, we propose to perform unsupervised and continuous online adaptation of a deep stereo network, which allows for preserving its accuracy in any environment. However, this strategy is extremely computationally demanding and thus prevents real-time inference. We address this issue introducing a new lightweight, yet effective, deep stereo architecture, Modularly ADaptive Network (MADNet) and developing a Modular ADaptation (MAD) algorithm, which independently trains sub-portions of the network. By deploying MADNet together with MAD we introduce the first real-time self-adaptive deep stereo system enabling competitive performance on heterogeneous datasets.Comment: Accepted at CVPR2019 as oral presentation. Code Available https://github.com/CVLAB-Unibo/Real-time-self-adaptive-deep-stere

Guided Stereo Matching

Stereo is a prominent technique to infer dense depth maps from images, and deep learning further pushed forward the state-of-the-art, making end-to-end architectures unrivaled when enough data is available for training. However, deep networks suffer from significant drops in accuracy when dealing with new environments. Therefore, in this paper, we introduce Guided Stereo Matching, a novel paradigm leveraging a small amount of sparse, yet reliable depth measurements retrieved from an external source enabling to ameliorate this weakness. The additional sparse cues required by our method can be obtained with any strategy (e.g., a LiDAR) and used to enhance features linked to corresponding disparity hypotheses. Our formulation is general and fully differentiable, thus enabling to exploit the additional sparse inputs in pre-trained deep stereo networks as well as for training a new instance from scratch. Extensive experiments on three standard datasets and two state-of-the-art deep architectures show that even with a small set of sparse input cues, i) the proposed paradigm enables significant improvements to pre-trained networks. Moreover, ii) training from scratch notably increases accuracy and robustness to domain shifts. Finally, iii) it is suited and effective even with traditional stereo algorithms such as SGM.Comment: CVPR 201

Learning monocular depth estimation with unsupervised trinocular assumptions

Obtaining accurate depth measurements out of a single image represents a fascinating solution to 3D sensing. CNNs led to considerable improvements in this field, and recent trends replaced the need for ground-truth labels with geometry-guided image reconstruction signals enabling unsupervised training. Currently, for this purpose, state-of-the-art techniques rely on images acquired with a binocular stereo rig to predict inverse depth (i.e., disparity) according to the aforementioned supervision principle. However, these methods suffer from well-known problems near occlusions, left image border, etc inherited from the stereo setup. Therefore, in this paper, we tackle these issues by moving to a trinocular domain for training. Assuming the central image as the reference, we train a CNN to infer disparity representations pairing such image with frames on its left and right side. This strategy allows obtaining depth maps not affected by typical stereo artifacts. Moreover, being trinocular datasets seldom available, we introduce a novel interleaved training procedure enabling to enforce the trinocular assumption outlined from current binocular datasets. Exhaustive experimental results on the KITTI dataset confirm that our proposal outperforms state-of-the-art methods for unsupervised monocular depth estimation trained on binocular stereo pairs as well as any known methods relying on other cues.Comment: 14 pages, 7 figures, 4 tables. Accepted to 3DV 201

TemporalStereo: Efficient Spatial-Temporal Stereo Matching Network

We present TemporalStereo, a coarse-to-fine based online stereo matching network which is highly efficient, and able to effectively exploit the past geometry and context information to boost the matching accuracy. Our network leverages sparse cost volume and proves to be effective when a single stereo pair is given, however, its peculiar ability to use spatio-temporal information across frames allows TemporalStereo to alleviate problems such as occlusions and reflective regions while enjoying high efficiency also in the case of stereo sequences. Notably our model trained, once with stereo videos, can run in both single-pair and temporal ways seamlessly. Experiments show that our network relying on camera motion is even robust to dynamic objects when running on videos. We validate TemporalStereo through extensive experiments on synthetic (SceneFlow, TartanAir) and real (KITTI 2012, KITTI 2015) datasets. Detailed results show that our model achieves state-of-the-art performance on any of these datasets. Code is available at \url{https://github.com/youmi-zym/TemporalStereo.git}

Learning optical flow from still images

This paper deals with the scarcity of data for training optical flow networks, highlighting the limitations of existing sources such as labeled synthetic datasets or unlabeled real videos. Specifically, we introduce a framework to generate accurate ground-truth optical flow annotations quickly and in large amounts from any readily available single real picture. Given an image, we use an off-the-shelf monocular depth estimation network to build a plausible point cloud for the observed scene. Then, we virtually move the camera in the reconstructed environment with known motion vectors and rotation angles, allowing us to synthesize both a novel view and the corresponding optical flow field connecting each pixel in the input image to the one in the new frame. When trained with our data, state-of-the-art optical flow networks achieve superior generalization to unseen real data compared to the same models trained either on annotated synthetic datasets or unlabeled videos, and better specialization if combined with synthetic images.Comment: CVPR 2021. Project page with supplementary and code: https://mattpoggi.github.io/projects/cvpr2021aleotti

Excitation of travelling multibreathers in anharmonic chains

We study the dynamics of the "externally" forced and damped Fermi-Pasta-Ulam (FPU) 1D lattice. The forcing has the spatial symmetry of the Fourier mode with wavenumber p and oscillates sinusoidally in time with the frequency omega. When omega is in the phonon band, the p-mode becomes modulationally unstable above a critical forcing, which we determine analytically in terms of the parameters of the system. For omega above the phonon band, the instability of the p-mode leads to the formation of a travelling multibreather, that, in the low-amplitude limit could be described in terms of soliton solutions of a suitable driven-damped nonlinear Schroedinger (NLS) equation. Similar mechanisms of instability could show up in easy-axis magnetic structures, that are governed by such NLS equations.Comment: To appear in Physica D (2002

Utopía, mística, revolución. A los inicios de la tragedia alemana del siglo XX

Como se deduce por el subtítulo, esta reflexión no se ocupa de la relación entre la utopía, mística y revolución desde un punto de vista general. Entre los tres conceptos – cada uno de los cuales ha sido tratado por una literatura infinita- los nexos siguen estando muy vinculados y en absoluto casuales. Dichos vínculos han sido estudiados muy ampliamente, remontándose a la historia de las ideas filosóficas, teológicas y políticas. Sin embargo, me voy a enfocar en nuestra época y en un momento muy peculiar y importante de la historia de Europa. Voy a examinar algunos aspectos de la relación entre la utopía, la mística y la revolución en la Alemania de los primeras dos décadas del siglo XX. Para esto es imprescindibile un prólogo, breve, pero tampoco tanto.Como se deduce por el subtítulo, esta reflexión no se ocupa de la relación entre la utopía, mística y revolución desde un punto de vista general. Entre los tres conceptos – cada uno de los cuales ha sido tratado por una literatura infinita- los nexos siguen estando muy vinculados y en absoluto casuales. Dichos vínculos han sido estudiados muy ampliamente, remontándose a la historia de las ideas filosóficas, teológicas y políticas. Sin embargo, me voy a enfocar en nuestra época y en un momento muy peculiar y importante de la historia de Europa. Voy a examinar algunos aspectos de la relación entre la utopía, la mística y la revolución en la Alemania de los primeras dos décadas del siglo XX. Para esto es imprescindibile un prólogo, breve, pero tampoco tanto