SFNet: Learning Object-aware Semantic Correspondence

We address the problem of semantic correspondence, that is, establishing a dense flow field between images depicting different instances of the same object or scene category. We propose to use images annotated with binary foreground masks and subjected to synthetic geometric deformations to train a convolutional neural network (CNN) for this task. Using these masks as part of the supervisory signal offers a good compromise between semantic flow methods, where the amount of training data is limited by the cost of manually selecting point correspondences, and semantic alignment ones, where the regression of a single global geometric transformation between images may be sensitive to image-specific details such as background clutter. We propose a new CNN architecture, dubbed SFNet, which implements this idea. It leverages a new and differentiable version of the argmax function for end-to-end training, with a loss that combines mask and flow consistency with smoothness terms. Experimental results demonstrate the effectiveness of our approach, which significantly outperforms the state of the art on standard benchmarks.Comment: cvpr 2019 oral pape

Label-driven weakly-supervised learning for multimodal deformable image registration

Spatially aligning medical images from different modalities remains a challenging task, especially for intraoperative applications that require fast and robust algorithms. We propose a weakly-supervised, label-driven formulation for learning 3D voxel correspondence from higher-level label correspondence, thereby bypassing classical intensity-based image similarity measures. During training, a convolutional neural network is optimised by outputting a dense displacement field (DDF) that warps a set of available anatomical labels from the moving image to match their corresponding counterparts in the fixed image. These label pairs, including solid organs, ducts, vessels, point landmarks and other ad hoc structures, are only required at training time and can be spatially aligned by minimising a cross-entropy function of the warped moving label and the fixed label. During inference, the trained network takes a new image pair to predict an optimal DDF, resulting in a fully-automatic, label-free, real-time and deformable registration. For interventional applications where large global transformation prevails, we also propose a neural network architecture to jointly optimise the global- and local displacements. Experiment results are presented based on cross-validating registrations of 111 pairs of T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients with a total of over 4000 anatomical labels, yielding a median target registration error of 4.2 mm on landmark centroids and a median Dice of 0.88 on prostate glands.Comment: Accepted to ISBI 201

Alakzatok lineáris deformációinak becslése és orvosi alkalmazásai = Estimation of Linear Shape Deformations and its Medical Applications

A projekt fő eredménye egy általánosan használható, teljesen automatikus alakzat regisztrációs módszer, amely az alábbi tulajdonságokkal rendelkezik: • nincs szükség pontmegfeleltetésekre illetve iteratív optimalizáló algoritmusokra; • képes 2D lineáris és (invertálható) projektív deformációk, valamint 3D affin deformációk meghatározására; • robusztus a geometriai és szegmentálási hibákra; • lineáris időkomplexitású, ami lehetővé teszi nagy felbontású képek közel valós idejű illesztését. Publikusan elérhetővé tettünk 3 demó programot, amelyek a 2D és 3D affin, valamint síkhomográfia regisztrációs algoritmusainkat implementálják. Továbbá kifejlesztettünk egy prototípus szoftvert csípőprotézis röntgenképek illesztésére, amit átadtunk a projektben közreműködő radiológusoknak további felhasználásra. Az eredményeinket a terület vezető konferenciáin ( pl. ICCV, ECCV) illetve vezető folyóiratokban (pl. IEEE Transactions on Pattern Analysis and Machine Intelligence, Pattern Recognition). A projekten dolgozó egyik MSc hallgató második helyezést ért el az OTDK-n. Domokos Csaba PhD fokozatot szerzett, továbbá munkáját Kuba Attila díjjal ismerte el a Képfeldolgozók és Alakfelismerők Társasága. A projekt eredményeiről részletesebb információ a projekt honlapokon található: • http://www.inf.u-szeged.hu/ipcg/projects/AFFSHAPE.html • http://www.inf.u-szeged.hu/ipcg/projects/AffinePuzzle.html • http://www.inf.u-szeged.hu/ipcg/projects/diffeoshape.html | The main achievement of the project is a fully functional automatic shape registration method with the following properties: • it doesn’t need established point correspondences nor the use of iterative optimization algorithms; • capable of recovering 2D linear and (invertible) projective shape deformations as well as affine distortions of 3D shapes; • robust in the presence of geometric noise and segmentation errors; • has a linear time complexity allowing near real-time registration of high resolution images. 3 demo programs are publicly available implementing our affine 2D, 3D and planar homography registration algorithms. Furthermore, we have developed a prototype software for aligning hip prosthesis X-ray images, which has been transfered to collaborating radiologists for further exploitation. Our results have been presented at top conferences (e.g. ICCV, ECCV) and in leading journals (e.g. IEEE Trans. on Patt. Anal. & Mach. Intell., Patt. Rec.). An MSc student working on the project received the second price of the National Scientific Student Conference. Csaba Domokos obtained his PhD degree and his work has been awarded the Attila Kuba Prize of the Hungarian Association for Image Processing and Pattern Recognition. More details about our results can be found at: • http://www.inf.u-szeged.hu/ipcg/projects/AFFSHAPE.html • http://www.inf.u-szeged.hu/ipcg/projects/AffinePuzzle.html • http://www.inf.u-szeged.hu/ipcg/projects/diffeoshape.htm

Deformable GANs for Pose-based Human Image Generation

In this paper we address the problem of generating person images conditioned on a given pose. Specifically, given an image of a person and a target pose, we synthesize a new image of that person in the novel pose. In order to deal with pixel-to-pixel misalignments caused by the pose differences, we introduce deformable skip connections in the generator of our Generative Adversarial Network. Moreover, a nearest-neighbour loss is proposed instead of the common L1 and L2 losses in order to match the details of the generated image with the target image. We test our approach using photos of persons in different poses and we compare our method with previous work in this area showing state-of-the-art results in two benchmarks. Our method can be applied to the wider field of deformable object generation, provided that the pose of the articulated object can be extracted using a keypoint detector.Comment: CVPR 2018 versio

Computerized Analysis of Magnetic Resonance Images to Study Cerebral Anatomy in Developing Neonates

The study of cerebral anatomy in developing neonates is of great importance for the understanding of brain development during the early period of life. This dissertation therefore focuses on three challenges in the modelling of cerebral anatomy in neonates during brain development. The methods that have been developed all use Magnetic Resonance Images (MRI) as source data. To facilitate study of vascular development in the neonatal period, a set of image analysis algorithms are developed to automatically extract and model cerebral vessel trees. The whole process consists of cerebral vessel tracking from automatically placed seed points, vessel tree generation, and vasculature registration and matching. These algorithms have been tested on clinical Time-of- Flight (TOF) MR angiographic datasets. To facilitate study of the neonatal cortex a complete cerebral cortex segmentation and reconstruction pipeline has been developed. Segmentation of the neonatal cortex is not effectively done by existing algorithms designed for the adult brain because the contrast between grey and white matter is reversed. This causes pixels containing tissue mixtures to be incorrectly labelled by conventional methods. The neonatal cortical segmentation method that has been developed is based on a novel expectation-maximization (EM) method with explicit correction for mislabelled partial volume voxels. Based on the resulting cortical segmentation, an implicit surface evolution technique is adopted for the reconstruction of the cortex in neonates. The performance of the method is investigated by performing a detailed landmark study. To facilitate study of cortical development, a cortical surface registration algorithm for aligning the cortical surface is developed. The method first inflates extracted cortical surfaces and then performs a non-rigid surface registration using free-form deformations (FFDs) to remove residual alignment. Validation experiments using data labelled by an expert observer demonstrate that the method can capture local changes and follow the growth of specific sulcus