Joint Optical Flow and Temporally Consistent Semantic Segmentation

The importance and demands of visual scene understanding have been steadily increasing along with the active development of autonomous systems. Consequently, there has been a large amount of research dedicated to semantic segmentation and dense motion estimation. In this paper, we propose a method for jointly estimating optical flow and temporally consistent semantic segmentation, which closely connects these two problem domains and leverages each other. Semantic segmentation provides information on plausible physical motion to its associated pixels, and accurate pixel-level temporal correspondences enhance the accuracy of semantic segmentation in the temporal domain. We demonstrate the benefits of our approach on the KITTI benchmark, where we observe performance gains for flow and segmentation. We achieve state-of-the-art optical flow results, and outperform all published algorithms by a large margin on challenging, but crucial dynamic objects.Comment: 14 pages, Accepted for CVRSUAD workshop at ECCV 201

Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentation

We propose a new deep learning method for tumour segmentation when dealing with missing imaging modalities. Instead of producing one network for each possible subset of observed modalities or using arithmetic operations to combine feature maps, our hetero-modal variational 3D encoder-decoder independently embeds all observed modalities into a shared latent representation. Missing data and tumour segmentation can be then generated from this embedding. In our scenario, the input is a random subset of modalities. We demonstrate that the optimisation problem can be seen as a mixture sampling. In addition to this, we introduce a new network architecture building upon both the 3D U-Net and the Multi-Modal Variational Auto-Encoder (MVAE). Finally, we evaluate our method on BraTS2018 using subsets of the imaging modalities as input. Our model outperforms the current state-of-the-art method for dealing with missing modalities and achieves similar performance to the subset-specific equivalent networks.Comment: Accepted at MICCAI 201

Identification of Disulfide Bond Formation between MitoNEET and Glutamate Dehydrogenase 1

MitoNEET is a protein that was identified as a drug target for diabetes, but its cellular function as well as its role in diabetes remains elusive. Protein pull-down experiments identified glutamate dehydrogenase 1 (GDH1) as a potential binding partner. GDH1 is a key metabolic enzyme with emerging roles in insulin regulation. MitoNEET forms a covalent complex with GDH1 through disulfide bond formation and acts as an activator. Proteomic analysis identified the specific cysteine residues that participate in the disulfide bond. This is the first report that effectively links mitoNEET to activation of the insulin regulator GDH1

Joint 3D estimation of vehicles and scene flow

driving. While much progress has been made in recent years, imaging conditions in natural outdoor environments are still very challenging for current reconstruction and recognition methods. In this paper, we propose a novel unified approach which reasons jointly about 3D scene flow as well as the pose, shape and motion of vehicles in the scene. Towards this goal, we incorporate a deformable CAD model into a slanted-plane conditional random field for scene flow estimation and enforce shape consistency between the rendered 3D models and the parameters of all superpixels in the image. The association of superpixels to objects is established by an index variable which implicitly enables model selection. We evaluate our approach on the challenging KITTI scene flow dataset in terms of object and scene flow estimation. Our results provide a prove of concept and demonstrate the usefulness of our method. © 2015 Copernicus GmbH. All Rights Reserved

FLOT: Scene Flow on Point Clouds Guided by Optimal Transport

We propose and study a method called FLOT that estimates scene flow on point clouds. We start the design of FLOT by noticing that scene flow estimation on point clouds reduces to estimating a permutation matrix in a perfect world. Inspired by recent works on graph matching, we build a method to find these correspondences by borrowing tools from optimal transport. Then, we relax the transport constraints to take into account real-world imperfections. The transport cost between two points is given by the pairwise similarity between deep features extracted by a neural network trained under full supervision using synthetic datasets. Our main finding is that FLOT can perform as well as the best existing methods on synthetic and real-world datasets while requiring much less parameters and without using multiscale analysis. Our second finding is that, on the training datasets considered, most of the performance can be explained by the learned transport cost. This yields a simpler method, FLOT$_0$, which is obtained using a particular choice of optimal transport parameters and performs nearly as well as FLOT.Comment: Accepted at ECCV2

Identification of Disulfide Bond Formation between MitoNEET and Glutamate Dehydrogenase 1

MitoNEET is a protein that was identified as a drug target for diabetes, but its cellular function as well as its role in diabetes remains elusive. Protein pull-down experiments identified glutamate dehydrogenase 1 (GDH1) as a potential binding partner. GDH1 is a key metabolic enzyme with emerging roles in insulin regulation. MitoNEET forms a covalent complex with GDH1 through disulfide bond formation and acts as an activator. Proteomic analysis identified the specific cysteine residues that participate in the disulfide bond. This is the first report that effectively links mitoNEET to activation of the insulin regulator GDH1

Neural parameters estimation for brain tumor growth modeling

Understanding the dynamics of brain tumor progression is essential for optimal treatment planning. Cast in a mathematical formulation, it is typically viewed as evaluation of a system of partial differential equations, wherein the physiological processes that govern the growth of the tumor are considered. To personalize the model, i.e. find a relevant set of parameters, with respect to the tumor dynamics of a particular patient, the model is informed from empirical data, e.g., medical images obtained from diagnostic modalities, such as magnetic-resonance imaging. Existing model-observation coupling schemes require a large number of forward integrations of the biophysical model and rely on simplifying assumption on the functional form, linking the output of the model with the image information. In this work, we propose a learning-based technique for the estimation of tumor growth model parameters from medical scans. The technique allows for explicit evaluation of the posterior distribution of the parameters by sequentially training a mixture-density network, relaxing the constraint on the functional form and reducing the number of samples necessary to propagate through the forward model for the estimation. We test the method on synthetic and real scans of rats injected with brain tumors to calibrate the model and to predict tumor progression

Deep learning-based parameter mapping for joint relaxation and diffusion tensor MR Fingerprinting

Magnetic Resonance Fingerprinting (MRF) enables the simultaneous quantification of multiple properties of biological tissues. It relies on a pseudo-random acquisition and the matching of acquired signal evolutions to a precomputed dictionary. However, the dictionary is not scalable to higher-parametric spaces, limiting MRF to the simultaneous mapping of only a small number of parameters (proton density, T1 and T2 in general). Inspired by diffusion-weighted SSFP imaging, we present a proof-of-concept of a novel MRF sequence with embedded diffusion-encoding gradients along all three axes to efficiently encode orientational diffusion and T1 and T2 relaxation. We take advantage of a convolutional neural network (CNN) to reconstruct multiple quantitative maps from this single, highly undersampled acquisition. We bypass expensive dictionary matching by learning the implicit physical relationships between the spatiotemporal MRF data and the T1, T2 and diffusion tensor parameters. The predicted parameter maps and the derived scalar diffusion metrics agree well with state-of-the-art reference protocols. Orientational diffusion information is captured as seen from the estimated primary diffusion directions. In addition to this, the joint acquisition and reconstruction framework proves capable of preserving tissue abnormalities in multiple sclerosis lesions

Selection on dispersal drives evolution of metabolic capacities for energy production in female wing-polymorphic sand field crickets, Gryllus firmus

Life history and metabolism covary, but the mechanisms and individual traits responsible for these linkages remain unresolved. Dispersal capability is a critical component of life histories that is constrained by metabolic capacities for energy production. Conflicting relationships between metabolism and life histories may be explained by accounting for variation in dispersal and maximal metabolic rates. We used female wing-polymorphic sand field crickets, Gryllus firmus, selected either for long wings (LW) and flight-capability or short wings (SW) and high early lifetime fecundity to test the hypothesis that selection on dispersal capability drives the evolution of metabolic capacities. While resting metabolic rates were similar, long-winged crickets reached higher maximal metabolic rates than short-winged crickets, resulting in improved running performance. We further provided insight into the mechanisms responsible for covariation between life history and metabolism by comparing mitochondrial content of tissues involved in powering locomotion and assessing function of mitochondria isolated from long- and short-winged crickets. This demonstrated that larger metabolic capacities in long-winged crickets were underpinned by increases in mitochondrial content of dorsoventral flight muscle and enhanced bioenergetic capacities of mitochondria within the fat body, a tissue responsible for fuel storage and mobilization. Thus, selection on flight-capability remodels metabolism in a trait and tissue-specific manner to enlarge metabolic capacities necessary for dispersal

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance. On KITTI, RAFT achieves an F1-all error of 5.10%, a 16% error reduction from the best published result (6.10%). On Sintel (final pass), RAFT obtains an end-point-error of 2.855 pixels, a 30% error reduction from the best published result (4.098 pixels). In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count. Code is available at https://github.com/princeton-vl/RAFT.Comment: fixed a formatting issue, Eq 7. no change in conten