130 research outputs found
How worms move in 3D
Animals that live in the sky, underwater or underground display unique three dimensional behaviours made possible by their ability to generate movement in all directions. As animals explore their environment, they constantly adapt their locomotion strategies to balance factors such as distance travelled, speed, and energy expenditure. While exploration strategies have been widely studied across a variety of species, how animals explore 3D space remains an open problem. The nematode Caenorhabditis elegans presents an ideal candidate for the study of 3D exploration as it is naturally found in complex fluid and granular environments and is well sized (~1mm long) for the simultaneous capture of individual postures and long term trajectories using a fixed imaging setup. However, until recently C. elegans has been studied almost exclusively in planar environments and in 3D neither its modes of locomotion nor its exploration strategies are known. Here we present methods for reconstructing microscopic postures and tracking macroscopic trajectories from a large corpus of triaxial recordings of worms freely exploring complex gelatinous fluids. To account for the constantly changing optical properties of these gels we develop a novel differentiable renderer to construct images from 3D postures for direct comparison with the recorded images. The method is robust to interference such as air bubbles and dirt trapped in the gel, stays consistent through complex sequences of postures and recovers reliable estimates from low-resolution, blurry images. Using this approach we generate a large dataset of 3D exploratory trajectories (over 6 hours) and midline postures (over 4 hours). We find that C. elegans explore 3D space through the composition of quasi-planar regions separated by turns and variable-length runs. To achieve this, C. elegans use locomotion gaits and complex manoeuvres that differ from those previously observed on an agar surface. We show that the associated costs of locomotion increase with non-planarity and we develop a mathematical model to probe the implications of this connection. We find that quasi-planar strategies (such as we find in the data) yield the largest volumes explored as they provide a balance between 3D coverage and trajectory distance. Taken together, our results link locomotion primitives with exploration strategies in the context of short term volumetric foraging to provide a first integrated study into how worms move in 3D
NR-SLAM: Non-Rigid Monocular SLAM
In this paper we present NR-SLAM, a novel non-rigid monocular SLAM system
founded on the combination of a Dynamic Deformation Graph with a Visco-Elastic
deformation model. The former enables our system to represent the dynamics of
the deforming environment as the camera explores, while the later allows us to
model general deformations in a simple way. The presented system is able to
automatically initialize and extend a map modeled by a sparse point cloud in
deforming environments, that is refined with a sliding-window Deformable Bundle
Adjustment. This map serves as base for the estimation of the camera motion and
deformation and enables us to represent arbitrary surface topologies,
overcoming the limitations of previous methods. To assess the performance of
our system in challenging deforming scenarios, we evaluate it in several
representative medical datasets. In our experiments, NR-SLAM outperforms
previous deformable SLAM systems, achieving millimeter reconstruction accuracy
and bringing automated medical intervention closer. For the benefit of the
community, we make the source code public.Comment: 12 pages, 7 figures, submited to the IEEE Transactions on Robotics
(T-RO
Deep Structured Layers for Instance-Level Optimization in 2D and 3D Vision
The approach we present in this thesis is that of integrating optimization problems
as layers in deep neural networks. Optimization-based modeling provides an additional set of tools enabling the design of powerful neural networks for a wide
battery of computer vision tasks. This thesis shows formulations and experiments
for vision tasks ranging from image reconstruction to 3D reconstruction.
We first propose an unrolled optimization method with implicit regularization
properties for reconstructing images from noisy camera readings. The method resembles an unrolled majorization minimization framework with convolutional neural networks acting as regularizers. We report state-of-the-art performance in image
reconstruction on both noisy and noise-free evaluation setups across many datasets.
We further focus on the task of monocular 3D reconstruction of articulated objects using video self-supervision. The proposed method uses a structured layer for
accurate object deformation that controls a 3D surface by displacing a small number
of learnable handles. While relying on a small set of training data per category for
self-supervision, the method obtains state-of-the-art reconstruction accuracy with
diverse shapes and viewpoints for multiple articulated objects.
We finally address the shortcomings of the previous method that revolve
around regressing the camera pose using multiple hypotheses. We propose a method
that recovers a 3D shape from a 2D image by relying solely on 3D-2D correspondences regressed from a convolutional neural network. These correspondences are
used in conjunction with an optimization problem to estimate per sample the camera pose and deformation. We quantitatively show the effectiveness of the proposed
method on self-supervised 3D reconstruction on multiple categories without the need for multiple hypotheses
Planning Framework for Robotic Pizza Dough Stretching with a Rolling Pin
Stretching a pizza dough with a rolling pin is a nonprehensile manipulation. Since the object is deformable, force closure cannot be established, and the manipulation is carried out in a nonprehensile way. The framework of this pizza dough stretching application that is explained in this chapter consists of four sub-procedures: (i) recognition of the pizza dough on a plate, (ii) planning the necessary steps to shape the pizza dough to the desired form, (iii) path generation for a rolling pin to execute the output of the pizza dough planner, and (iv) inverse kinematics for the bi-manual robot to grasp and control the rolling pin properly. Using the deformable object model described in Chap. 3, each sub-procedure of the proposed framework is explained sequentially
Einstein vs. Bergson
On 6 April 1922, Einstein met Bergson to debate the nature of time: is the time the physicist calculates the same time the philosopher reflects on? Einstein claimed that only scientific time is real, while Bergson argued that scientific time always presupposes a living and perceiving subject. On that day, nearly 100 years ago, conflict was inevitable. Is it still inevitable today? How many kinds of time are there
Deep Shape-from-Template: Single-image quasi-isometric deformable registration and reconstruction
Shape-from-Template (SfT) solves 3D vision from a single image and a deformable 3D object model, called a template. Concretely, SfT computes registration (the correspondence between the template and the image) and reconstruction (the depth in camera frame). It constrains the object deformation to quasi-isometry. Real-time and automatic SfT represents an open problem for complex objects and imaging conditions. We present four contributions to address core unmet challenges to realise SfT with a Deep Neural Network (DNN). First, we propose a novel DNN called DeepSfT, which encodes the template in its weights and hence copes with highly complex templates. Second, we propose a semi-supervised training procedure to exploit real data. This is a practical solution to overcome the render gap that occurs when training only with simulated data. Third, we propose a geometry adaptation module to deal with different cameras at training and inference. Fourth, we combine statistical learning with physics-based reasoning. DeepSfT runs automatically and in real-time and we show with numerous experiments and an ablation study that it consistently achieves a lower 3D error than previous work. It outperforms in generalisation and achieves great performance in terms of reconstruction and registration error with wide-baseline, occlusions, illumination changes, weak texture and blur.Agencia Estatal de InvestigaciónMinisterio de Educació
Recommended from our members
Harnessing Simulated Data with Graphs
Physically accurate simulations allow for unlimited exploration of arbitrarily crafted environments. From a scientific perspective, digital representations of the real world are useful because they make it easy validate ideas. Virtual sandboxes allow observations to be collected at-will, without intricate setting up for measurements or needing to wait on the manufacturing, shipping, and assembly of physical resources. Simulation techniques can also be utilized over and over again to test the problem without expending costly materials or producing any waste.
Remarkably, this freedom to both experiment and generate data becomes even more powerful when considering the rising adoption of data-driven techniques across engineering disciplines. These are systems that aggregate over available samples to model behavior, and thus are better informed when exposed to more data. Naturally, the ability to synthesize limitless data promises to make approaches that benefit from datasets all the more robust and desirable.
However, the ability to readily and endlessly produce synthetic examples also introduces several new challenges. Data must be collected in an adaptive format that can capture the complete diversity of states achievable in arbitrary simulated configurations while too remaining amenable to downstream applications. The quantity and zoology of observations must also straddle a range which prevents overfitting but is descriptive enough to produce a robust approach. Pipelines that naively measure virtual scenarios can easily be overwhelmed by trying to sample an infinite set of available configurations. Variations observed across multiple dimensions can quickly lead to a daunting expansion of states, all of which must be processed and solved. These and several other concerns must first be addressed in order to safely leverage the potential of boundless simulated data.
In response to these challenges, this thesis proposes to wield graphs in order to instill structure over digitally captured data, and curb the growth of variables. The paradigm of pairing data with graphs introduced in this dissertation serves to enforce consistency, localize operators, and crucially factor out any combinatorial explosion of states. Results demonstrate the effectiveness of this methodology in three distinct areas, each individually offering unique challenges and practical constraints, and together showcasing the generality of the approach. Namely, studies observing state-of-the-art contributions in design for additive manufacturing, side-channel security threats, and large-scale physics based contact simulations are collectively achieved by harnessing simulated datasets with graph algorithms
State of the Art in Dense Monocular Non-Rigid 3D Reconstruction
3D reconstruction of deformable (or non-rigid) scenes from a set of monocular2D image observations is a long-standing and actively researched area ofcomputer vision and graphics. It is an ill-posed inverse problem,since--without additional prior assumptions--it permits infinitely manysolutions leading to accurate projection to the input 2D images. Non-rigidreconstruction is a foundational building block for downstream applicationslike robotics, AR/VR, or visual content creation. The key advantage of usingmonocular cameras is their omnipresence and availability to the end users aswell as their ease of use compared to more sophisticated camera set-ups such asstereo or multi-view systems. This survey focuses on state-of-the-art methodsfor dense non-rigid 3D reconstruction of various deformable objects andcomposite scenes from monocular videos or sets of monocular views. It reviewsthe fundamentals of 3D reconstruction and deformation modeling from 2D imageobservations. We then start from general methods--that handle arbitrary scenesand make only a few prior assumptions--and proceed towards techniques makingstronger assumptions about the observed objects and types of deformations (e.g.human faces, bodies, hands, and animals). A significant part of this STAR isalso devoted to classification and a high-level comparison of the methods, aswell as an overview of the datasets for training and evaluation of thediscussed techniques. We conclude by discussing open challenges in the fieldand the social aspects associated with the usage of the reviewed methods.<br
Vehicle and Traffic Safety
The book is devoted to contemporary issues regarding the safety of motor vehicles and road traffic. It presents the achievements of scientists, specialists, and industry representatives in the following selected areas of road transport safety and automotive engineering: active and passive vehicle safety, vehicle dynamics and stability, testing of vehicles (and their assemblies), including electric cars as well as autonomous vehicles. Selected issues from the area of accident analysis and reconstruction are discussed. The impact on road safety of aspects such as traffic control systems, road infrastructure, and human factors is also considered
State of the Art in Dense Monocular Non-Rigid 3D Reconstruction
3D reconstruction of deformable (or non-rigid) scenes from a set of monocular
2D image observations is a long-standing and actively researched area of
computer vision and graphics. It is an ill-posed inverse problem,
since--without additional prior assumptions--it permits infinitely many
solutions leading to accurate projection to the input 2D images. Non-rigid
reconstruction is a foundational building block for downstream applications
like robotics, AR/VR, or visual content creation. The key advantage of using
monocular cameras is their omnipresence and availability to the end users as
well as their ease of use compared to more sophisticated camera set-ups such as
stereo or multi-view systems. This survey focuses on state-of-the-art methods
for dense non-rigid 3D reconstruction of various deformable objects and
composite scenes from monocular videos or sets of monocular views. It reviews
the fundamentals of 3D reconstruction and deformation modeling from 2D image
observations. We then start from general methods--that handle arbitrary scenes
and make only a few prior assumptions--and proceed towards techniques making
stronger assumptions about the observed objects and types of deformations (e.g.
human faces, bodies, hands, and animals). A significant part of this STAR is
also devoted to classification and a high-level comparison of the methods, as
well as an overview of the datasets for training and evaluation of the
discussed techniques. We conclude by discussing open challenges in the field
and the social aspects associated with the usage of the reviewed methods.Comment: 25 page
- …