A variational approach to video registration with subspace constraints

This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multi-frame optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.Ravi Garg, Anastasios Roussos, Lourdes Agapit

Variational methods for direct and indirect tracking in dynamic imaging

Das Thema Tracking ist von großer Bedeutung für eine Vielzahl von Anwendungen. Wir analysieren Ansätze zur Bewegungsschätzung bei unterschiedlichen Anforderungen von Daten. Zunächst betrachten wir Modelle für direktes Tracking, bei denen Bewegungsschätzung ohne Vorverarbeitung möglich ist. Wir erweitern bekannte Modelle zum optischen Fluss um deren Performance bei komplexen Bewegungen zu verbessern. Wir analysieren und testen diese Modelle anhand synthetischer und realer Datensätze. Als Anwendung untersuchen wir Mikroskopie-Bilder von Zellen und erläutern die erforderlichen mathematischen Aufgaben. Für den zweiten Teil betrachten wir Daten, bei denen zunächst Bilder rekonstruiert werden müssen, um Bewegungen zu schätzen. Wir präsentieren ein Modell für gleichzeitige Rekonstruktion und Bewegungsschätzung. Dieses Modell wenden wir auf dynamische CT-Daten an. Hierbei betrachten wir sich bewegende Objekte unter Verwendung von nur einem oder zwei Messwinkeln pro Zeitschritt.The topic of estimating motion is a major issue for a wide range of applications. We study techniques for tracking given different external requirements. We start by studying models for direct tracking where estimating motion is possible without preprocessing. We extend established optical flow models in order to improve their performance in connection with complex motion. We analyze these models and check their performance based on different synthetic and real data sets. As a specific example we study microscopy data of migrating cells and illustrate the required mathematical tasks. For the second theoretical part we consider types of data that require reconstruction in order to obtain images that are needed to estimate motion. We present a model for simultaneous reconstruction and motion estimation. We apply this model to dynamic X-ray tomography data. We consider moving objects measured from only one or two different angles per time step

Reasoning about Geometric Object Interactions in 3D for Manipulation Action Understanding

In order to efficiently interact with human users, intelligent agents and autonomous systems need the ability of interpreting human actions. We focus our attention on manipulation actions, wherein an agent typically grasps an object and moves it, possibly altering its physical state. Agent-object and object-object interactions during a manipulation are a defining part of the performed action itself. In this thesis, we focus on extracting semantic cues, derived from geometric object interactions in 3D space during a manipulation, that are useful for action understanding at the cognitive level. First, we introduce a simple grounding model for the most common pairwise spatial relations between objects and investigate the descriptive power of their temporal evolution for action characterization. We propose a compact, abstract action descriptor that encodes the geometric object interactions during action execution, as captured by the spatial relation dynamics. Our experiments on a diverse dataset confirm both the validity and effectiveness of our spatial relation models and the discriminative power of our representation with respect to the underlying action semantics. Second, we model and detect lower level interactions, namely object contacts and separations, viewing them as topological scene changes within a dense motion estimation setting. In addition to improving motion estimation accuracy in the challenging case of motion boundaries induced by these events, our approach shows promising performance in the explicit detection and classification of the latter. Building upon dense motion estimation and using detected contact events as an attention mechanism, we propose a bottom-up pipeline for the guided segmentation and rigid motion extraction of manipulated objects. Finally, in addition to our methodological contributions, we introduce a new open-source software library for point cloud data processing, developed for the needs of this thesis, which aims at providing an easy to use, flexible, and efficient framework for the rapid development of performant software for a range of 3D perception tasks

Schätzung dichter Korrespondenzfelder unter Verwendung mehrerer Bilder

Most optical flow algorithms assume pairs of images that are acquired with an ideal, short exposure time. We present two approaches, that use additional images of a scene to estimate highly accurate, dense correspondence fields. In our first approach we consider video sequences that are acquired with alternating exposure times so that a short-exposure image is followed by a long-exposure image that exhibits motion-blur. With the help of the two enframing short-exposure images, we can decipher not only the motion information encoded in the long-exposure image, but also estimate occlusion timings, which are a basis for artifact-free frame interpolation. In our second approach we consider the data modality of multi-view video sequences, as it commonly occurs, e.g., in stereoscopic video. As several images capture nearly the same data of a scene, this redundancy can be used to establish more robust and consistent correspondence fields than the consideration of two images permits.Die meisten Verfahren zur Schätzung des optischen Flusses verwenden zwei Bilder, die mit einer optimalen, kurzen Belichtungszeit aufgenommen wurden. Wir präsentieren zwei Methoden, die zusätzliche Bilder zur Schätzung von hochgenauen, dichten Korrespondenzfeldern verwenden. Die erste Methode betrachtet Videosequenzen, die mit alternierender Belichtungsdauer aufgenommen werden, so dass auf eine Kurzzeitbelichtung eine Langzeitbelichtung folgt, die Bewegungsunschärfe enthält. Mit der Hilfe von zwei benachbarten Kurzzeitbelichtungen können wir nicht nur die Bewegung schätzen, die in der Bewegungsunschärfe der Langzeitbelichtung verschlüsselt ist, sondern zusätzlich auch Verdeckungszeiten schätzen, die sich bei der Interpolation von Zwischenbildern als große Hilfe erweisen. Die zweite Methode betrachtet Videos, die eine Szene aus mehreren Ansichten aufzeichnen, wie z.B. Stereovideos. Dabei enthalten mehrere Bilder fast dieselbe Information über die Szene. Wir nutzen diese Redundanz aus, um konsistentere und robustere Bewegungsfelder zu bestimmen, als es mit zwei Bildern möglich ist

Partial Differential Equation-Constrained Diffeomorphic Registration from Sum of Squared Differences to Normalized Cross-Correlation, Normalized Gradient Fields, and Mutual Information: A Unifying Framework; 35632143

This work proposes a unifying framework for extending PDE-constrained Large Deformation Diffeomorphic Metric Mapping (PDE-LDDMM) with the sum of squared differences (SSD) to PDE-LDDMM with different image similarity metrics. We focused on the two best-performing variants of PDE-LDDMM with the spatial and band-limited parameterizations of diffeomorphisms. We derived the equations for gradient-descent and Gauss-Newton-Krylov (GNK) optimization with Normalized Cross-Correlation (NCC), its local version (lNCC), Normalized Gradient Fields (NGFs), and Mutual Information (MI). PDE-LDDMM with GNK was successfully implemented for NCC and lNCC, substantially improving the registration results of SSD. For these metrics, GNK optimization outperformed gradient-descent. However, for NGFs, GNK optimization was not able to overpass the performance of gradient-descent. For MI, GNK optimization involved the product of huge dense matrices, requesting an unaffordable memory load. The extensive evaluation reported the band-limited version of PDE-LDDMM based on the deformation state equation with NCC and lNCC image similarities among the best performing PDE-LDDMM methods. In comparison with benchmark deep learning-based methods, our proposal reached or surpassed the accuracy of the best-performing models. In NIREP16, several configurations of PDE-LDDMM outperformed ANTS-lNCC, the best benchmark method. Although NGFs and MI usually underperformed the other metrics in our evaluation, these metrics showed potentially competitive results in a multimodal deformable experiment. We believe that our proposed image similarity extension over PDE-LDDMM will promote the use of physically meaningful diffeomorphisms in a wide variety of clinical applications depending on deformable image registration

International Conference on Continuous Optimization (ICCOPT) 2019 Conference Book

The Sixth International Conference on Continuous Optimization took place on the campus of the Technical University of Berlin, August 3-8, 2019. The ICCOPT is a flagship conference of the Mathematical Optimization Society (MOS), organized every three years. ICCOPT 2019 was hosted by the Weierstrass Institute for Applied Analysis and Stochastics (WIAS) Berlin. It included a Summer School and a Conference with a series of plenary and semi-plenary talks, organized and contributed sessions, and poster sessions. This book comprises the full conference program. It contains, in particular, the scientific program in survey style as well as with all details, and information on the social program, the venue, special meetings, and more

Reconstruction Methods for Free-Breathing Dynamic Contrast-Enhanced MRI

Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) is a valuable diagnostic tool due to the combination of anatomical and physiological information it provides. However, the sequential sampling of MRI presents an inherent tradeoff between spatial and temporal resolution. Compressed Sensing (CS) methods have been applied to undersampled MRI to reconstruct full-resolution images at sub-Nyquist sampling rates. In exchange for shorter data acquisition times, CS-MRI requires more computationally intensive iterative reconstruction methods. We present several model-based image reconstruction (MBIR) methods to improve the spatial and temporal resolution of MR images and/or the computational time for multi-coil MRI reconstruction. We propose efficient variable splitting (VS) methods for support-constrained MRI reconstruction, image reconstruction and denoising with non-circulant boundary conditions, and improved temporal regularization for breast DCE-MRI. These proposed VS algorithms decouple the system model and sparsity terms of the convex optimization problem. By leveraging matrix structures in the system model and sparsifying operator, we perform alternating minimization over a list of auxiliary variables, each of which can be performed efficiently. We demonstrate the computational benefits of our proposed VS algorithms compared to similar proposed methods. We also demonstrate convergence guarantees for two proposed methods, ADMM-tridiag and ADMM-FP-tridiag. With simulation experiments, we demonstrate lower error in spatial and temporal dimensions for these VS methods compared to other object models. We also propose a method for indirect motion compensation in 5D liver DCE-MRI. 5D MRI separates temporal changes due to contrast from anatomical changes due to respiratory motion into two distinct dimensions. This work applies a pre-computed motion model to perform motion-compensated regularization across the respiratory dimension and improve the conditioning of this highly sparse 5D reconstruction problem. We demonstrate a proof of concept using a digital phantom with contrast and respiratory changes, and we show preliminary results for motion model-informed regularization on in vivo patient data.PHDElectrical Engineering: SystemsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/138498/1/mtle_1.pd

Biological image analysis

In biological research images are extensively used to monitor growth, dynamics and changes in biological specimen, such as cells or plants. Many of these images are used solely for observation or are manually annotated by an expert. In this dissertation we discuss several methods to automate the annotating and analysis of bio-images. Two large clusters of methods have been investigated and developed. A first set of methods focuses on the automatic delineation of relevant objects in bio-images, such as individual cells in microscopic images. Since these methods should be useful for many different applications, e.g. to detect and delineate different objects (cells, plants, leafs, ...) in different types of images (different types of microscopes, regular colour photographs, ...), the methods should be easy to adjust. Therefore we developed a methodology relying on probability theory, where all required parameters can easily be estimated by a biologist, without requiring any knowledge on the techniques used in the actual software. A second cluster of investigated techniques focuses on the analysis of shapes. By defining new features that describe shapes, we are able to automatically classify shapes, retrieve similar shapes from a database and even analyse how an object deforms through time