Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid

In modern computer vision, images are typically represented as a fixed uniform grid with some stride and processed via a deep convolutional neural network. We argue that deforming the grid to better align with the high-frequency image content is a more effective strategy. We introduce \emph{Deformable Grid} DefGrid, a learnable neural network module that predicts location offsets of vertices of a 2-dimensional triangular grid, such that the edges of the deformed grid align with image boundaries. We showcase our DefGrid in a variety of use cases, i.e., by inserting it as a module at various levels of processing. We utilize DefGrid as an end-to-end \emph{learnable geometric downsampling} layer that replaces standard pooling methods for reducing feature resolution when feeding images into a deep CNN. We show significantly improved results at the same grid resolution compared to using CNNs on uniform grids for the task of semantic segmentation. We also utilize DefGrid at the output layers for the task of object mask annotation, and show that reasoning about object boundaries on our predicted polygonal grid leads to more accurate results over existing pixel-wise and curve-based approaches. We finally showcase DefGrid as a standalone module for unsupervised image partitioning, showing superior performance over existing approaches. Project website: http://www.cs.toronto.edu/~jungao/def-gridComment: ECCV 202

An Intestinal Surgery Simulator: Real-Time Collision Processing and Visualization

International audienceThis research work is aimed towards the development of a VR-based trainer for colon cancer removal. It enables the surgeons to interactively view and manipulate the concerned virtual organs as during a real surgery. First, we present a method for animating the small intestine and the mesentery (the tissue that connects it to the main vessels) in real-time, thus enabling user-interaction through virtual surgical tools during the simulation. We present a stochastic approach for fast collision detection in highly deformable, self-colliding objects. A simple and efficient response to collisions is also introduced in order to reduce the overall animation complexity. Secondly, we describe a new method based on generalized cylinders for fast rendering of the intestine. An efficient curvature detection method, along with an adaptive sampling algorithm is presented. This approach, while providing improved tessellation without the classical self-intersection problem, also allows for high-performance rendering, thanks to the new 3D skinning feature available in recent GPUs. The rendering algorithm is also designed to ensure a guaranteed frame rate. Finally, we present the quantitative results of the simulations and describe the qualitative feedback obtained from the surgeons

Precise Hausdorff distance computation for freeform surfaces based on computations with osculating toroidal patches

We present an efficient algorithm for computing the precise Hausdorff Distance (HD) between two freeform surfaces. The algorithm is based on a hybrid Bounding Volume Hierarchy (BVH), where osculating toroidal patches (stored in the leaf nodes) provide geometric properties essential for the HD computation in high precision. Intrinsic features from the osculating geometry resolve computational issues in handling the cross-boundary problem for composite surfaces, which leads to the acceleration of HD algorithm with a solution (within machine precision) to the exact HD. The HD computation for general freeform surfaces is discussed, where we focus on the computational issues in handling the local geometry across surface boundaries or around surface corners that appear as the result of gluing multiple patches together in the modeling of generic composite surfaces. We also discuss how to switch from an approximation stage to the final step of computing the precise HD using numerical improvements and confirming the correctness of the HD computation result. The main advantage of our algorithm is in the high precision of HD computation result. As the best cases of the proposed torus-based approach, we also consider the acceleration of HD computation for freeform surfaces of revolution and linear extrusion, where we can support real-time computation even for deformable surfaces. The acceleration is mainly due to a fast biarc approximation to the planar profile curves of the simple surfaces, each generated by rotating or translating a planar curve. We demonstrate the effectiveness of the proposed approach using experimental results

Generating anatomical substructures for physically-based facial animation.

Physically-based facial animation techniques are capable of producing realistic facial deformations, but have failed to find meaningful use outside the academic community because they are notoriously difficult to create, reuse, and art-direct, in comparison to other methods of facial animation. This thesis addresses these shortcomings and presents a series of methods for automatically generating a skull, the superficial musculoaponeurotic system (SMAS – a layer of fascia investing and interlinking the mimic muscle system), and mimic muscles for any given 3D face model. This is done toward (the goal of) a production-viable framework or rig-builder for physically-based facial animation. This workflow consists of three major steps. First, a generic skull is fitted to a given head model using thin-plate splines computed from the correspondence between landmarks placed on both models. Second, the SMAS is constructed as a variational implicit or radial basis function surface in the interface between the head model and the generic skull fitted to it. Lastly, muscle fibres are generated as boundary-value straightest geodesics, connecting muscle attachment regions defined on the surface of the SMAS. Each step of this workflow is developed with speed, realism and reusability in mind

Markerless deformation capture of hoverfly wings using multiple calibrated cameras

This thesis introduces an algorithm for the automated deformation capture of hoverfly wings from multiple camera image sequences. The algorithm is capable of extracting dense surface measurements, without the aid of fiducial markers, over an arbitrary number of wingbeats of hovering flight and requires limited manual initialisation. A novel motion prediction method, called the ‘normalised stroke model’, makes use of the similarity of adjacent wing strokes to predict wing keypoint locations, which are then iteratively refined in a stereo image registration procedure. Outlier removal, wing fitting and further refinement using independently reconstructed boundary points complete the algorithm. It was tested on two hovering data sets, as well as a challenging flight manoeuvre. By comparing the 3-d positions of keypoints extracted from these surfaces with those resulting from manual identification, the accuracy of the algorithm is shown to approach that of a fully manual approach. In particular, half of the algorithm-extracted keypoints were within 0.17mm of manually identified keypoints, approximately equal to the error of the manual identification process. This algorithm is unique among purely image based flapping flight studies in the level of automation it achieves, and its generality would make it applicable to wing tracking of other insects

Virtual human modelling and animation for real-time sign language visualisation

>Magister Scientiae - MScThis thesis investigates the modelling and animation of virtual humans for real-time sign language visualisation. Sign languages are fully developed natural languages used by Deaf communities all over the world. These languages are communicated in a visual-gestural modality by the use of manual and non-manual gestures and are completely di erent from spoken languages. Manual gestures include the use of hand shapes, hand movements, hand locations and orientations of the palm in space. Non-manual gestures include the use of facial expressions, eye-gazes, head and upper body movements. Both manual and nonmanual gestures must be performed for sign languages to be correctly understood and interpreted. To e ectively visualise sign languages, a virtual human system must have models of adequate quality and be able to perform both manual and non-manual gesture animations in real-time. Our goal was to develop a methodology and establish an open framework by using various standards and open technologies to model and animate virtual humans of adequate quality to e ectively visualise sign languages. This open framework is to be used in a Machine Translation system that translates from a verbal language such as English to any sign language. Standards and technologies we employed include H-Anim, MakeHuman, Blender, Python and SignWriting. We found it necessary to adapt and extend H-Anim to e ectively visualise sign languages. The adaptations and extensions we made to H-Anim include imposing joint rotational limits, developing exible hands and the addition of facial bones based on the MPEG-4 Facial De nition Parameters facial feature points for facial animation. By using these standards and technologies, we found that we could circumvent a few di cult problems, such as: modelling high quality virtual humans; adapting and extending H-Anim; creating a sign language animation action vocabulary; blending between animations in an action vocabulary; sharing animation action data between our virtual humans; and e ectively visualising South African Sign Language.South Afric

Computer Vision Problems in 3D Plant Phenotyping

In recent years, there has been significant progress in Computer Vision based plant phenotyping (quantitative analysis of biological properties of plants) technologies. Traditional methods of plant phenotyping are destructive, manual and error prone. Due to non-invasiveness and non-contact properties as well as increased accuracy, imaging techniques are becoming state-of-the-art in plant phenotyping. Among several parameters of plant phenotyping, growth analysis is very important for biological inference. Automating the growth analysis can result in accelerating the throughput in crop production. This thesis contributes to the automation of plant growth analysis. First, we present a novel system for automated and non-invasive/non-contact plant growth measurement. We exploit the recent advancements of sophisticated robotic technologies and near infrared laser scanners to build a 3D imaging system and use state-of-the-art Computer Vision algorithms to fully automate growth measurement. We have set up a gantry robot system having 7 degrees of freedom hanging from the roof of a growth chamber. The payload is a range scanner, which can measure dense depth maps (raw 3D coordinate points in mm) on the surface of an object (the plant). The scanner can be moved around the plant to scan from different viewpoints by programming the robot with a specific trajectory. The sequence of overlapping images can be aligned to obtain a full 3D structure of the plant in raw point cloud format, which can be triangulated to obtain a smooth surface (triangular mesh), enclosing the original plant. We show the capability of the system to capture the well known diurnal pattern of plant growth computed from the surface area and volume of the plant meshes for a number of plant species. Second, we propose a technique to detect branch junctions in plant point cloud data. We demonstrate that using these junctions as feature points, the correspondence estimation can be formulated as a subgraph matching problem, and better matching results than state-of-the-art can be achieved. Also, this idea removes the requirement of a priori knowledge about rotational angles between adjacent scanning viewpoints imposed by the original registration algorithm for complex plant data. Before, this angle information had to be approximately known. Third, we present an algorithm to classify partially occluded leaves by their contours. In general, partial contour matching is a NP-hard problem. We propose a suboptimal matching solution and show that our method outperforms state-of-the-art on 3 public leaf datasets. We anticipate using this algorithm to track growing segmented leaves in our plant range data, even when a leaf becomes partially occluded by other plant matter over time. Finally, we perform some experiments to demonstrate the capability and limitations of the system and highlight the future research directions for Computer Vision based plant phenotyping

Techniques for Realtime Viewing and Manipulation of Volumetric Data

Visualizing and manipulating volumetric data is a major component in many areas including anatomical registration in biomedical fields, seismic data analysis in the oil industry, machine part design in computer-aided geometric design, character animation in the movie industry, and fluid simulation. These industries have to meet the demands of the times and be able to make meaningful assertions about the data they generate. The shear size of this data presents many challenges to facilitating realtime interaction. In the recent decade, graphics hardware has become increasingly powerful and more sophisticated which has introduced a new realm of possibilities for processing volumetric data. This thesis focuses on a suite of techniques for viewing and editing volumetric data that efficiently use the processing power of central processing units (CPUs) as well as the large processing power of the graphics hardware (GPUs). This work begins with an algorithm to improve the efficiency of a texture-based volume rendering. We continue with a framework for performing realtime constructive solid geometry (CSG) with complex shapes and smoothing operations on watertight meshes based on a variation of Depth Peeling. We then move to an intuitive technique for deforming volumetric data using a collection of control points. Finally, we apply this technique to image registration of 3-dimensional computed tomography (CT) images used for lung cancel treatment, planning

Computer Vision Problems in 3D Plant Phenotyping

In recent years, there has been significant progress in Computer Vision based plant phenotyping (quantitative analysis of biological properties of plants) technologies. Traditional methods of plant phenotyping are destructive, manual and error prone. Due to non-invasiveness and non-contact properties as well as increased accuracy, imaging techniques are becoming state-of-the-art in plant phenotyping. Among several parameters of plant phenotyping, growth analysis is very important for biological inference. Automating the growth analysis can result in accelerating the throughput in crop production. This thesis contributes to the automation of plant growth analysis. First, we present a novel system for automated and non-invasive/non-contact plant growth measurement. We exploit the recent advancements of sophisticated robotic technologies and near infrared laser scanners to build a 3D imaging system and use state-of-the-art Computer Vision algorithms to fully automate growth measurement. We have set up a gantry robot system having 7 degrees of freedom hanging from the roof of a growth chamber. The payload is a range scanner, which can measure dense depth maps (raw 3D coordinate points in mm) on the surface of an object (the plant). The scanner can be moved around the plant to scan from different viewpoints by programming the robot with a specific trajectory. The sequence of overlapping images can be aligned to obtain a full 3D structure of the plant in raw point cloud format, which can be triangulated to obtain a smooth surface (triangular mesh), enclosing the original plant. We show the capability of the system to capture the well known diurnal pattern of plant growth computed from the surface area and volume of the plant meshes for a number of plant species. Second, we propose a technique to detect branch junctions in plant point cloud data. We demonstrate that using these junctions as feature points, the correspondence estimation can be formulated as a subgraph matching problem, and better matching results than state-of-the-art can be achieved. Also, this idea removes the requirement of a priori knowledge about rotational angles between adjacent scanning viewpoints imposed by the original registration algorithm for complex plant data. Before, this angle information had to be approximately known. Third, we present an algorithm to classify partially occluded leaves by their contours. In general, partial contour matching is a NP-hard problem. We propose a suboptimal matching solution and show that our method outperforms state-of-the-art on 3 public leaf datasets. We anticipate using this algorithm to track growing segmented leaves in our plant range data, even when a leaf becomes partially occluded by other plant matter over time. Finally, we perform some experiments to demonstrate the capability and limitations of the system and highlight the future research directions for Computer Vision based plant phenotyping

On Triangular Splines:CAD and Quadrature

The standard representation of CAD (computer aided design) models is based on the boundary representation (B-reps) with trimmed and (topologically) stitched tensor-product NURBS patches. Due to trimming, this leads to gaps and overlaps in the models. While these can be made arbitrarily small for visualisation and manufacturing purposes, they still pose problems in downstream applications such as (isogeometric) analysis and 3D printing. It is therefore worthwhile to investigate conversion methods which (necessarily approximately) convert these models into water-tight or even smooth representations. After briefly surveying existing conversion methods, we will focus on techniques that convert CAD models into triangular spline surfaces of various levels of continuity. In the second part, we will investigate efficient quadrature rules for triangular spline space