The Parallel Distributed Image Search Engine (ParaDISE)

Image retrieval is a complex task that differs according to the context and the user requirements in any specific field, for example in a medical environment. Search by text is often not possible or optimal and retrieval by the visual content does not always succeed in modelling high-level concepts that a user is looking for. Modern image retrieval techniques consists of multiple steps and aim to retrieve information from large–scale datasets and not only based on global image appearance but local features and if possible in a connection between visual features and text or semantics. This paper presents the Parallel Distributed Image Search Engine (ParaDISE), an image retrieval system that combines visual search with text–based retrieval and that is available as open source and free of charge. The main design concepts of ParaDISE are flexibility, expandability, scalability and interoperability. These concepts constitute the system, able to be used both in real–world applications and as an image retrieval research platform. Apart from the architecture and the implementation of the system, two use cases are described, an application of ParaDISE in retrieval of images from the medical literature and a visual feature evaluation for medical image retrieval. Future steps include the creation of an open source community that will contribute and expand this platform based on the existing parts

Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition

This is an electronic version of an article published inComputer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualizationon 13 August 2015, by Taylor & Francis, DOI: 10.1080/21681163.2015.1061448.Available online at: http://www.tandfonline.com/10.1080/21681163.2015.1061448.International audienceIn this paper, we propose a convolutional neural network-based method to automatically retrieve missing or noisy cardiac acquisition plane information from magnetic resonance imaging and predict the five most common cardiac views. We fine-tune a convolutional neural network (CNN) initially trained on a large natural image recognition data-set (Imagenet ILSVRC2012) and transfer the learnt feature representations to cardiac view recognition. We contrast this approach with a previously introduced method using classification forests and an augmented set of image miniatures, with prediction using off the shelf CNN features, and with CNNs learnt from scratch. We validate this algorithm on two different cardiac studies with 200 patients and 15 healthy volunteers, respectively. We show that there is value in fine-tuning a model trained for natural images to transfer it to medical images. Our approach achieves an average F1 score of 97.66% and significantly improves the state-of-the-art of image-based cardiac view recognition. This is an important building block to organise and filter large collections of cardiac data prior to further analysis. It allows us to merge studies from multiple centres, to perform smarter image filtering, to select the most appropriate image processing algorithm, and to enhance visualisation of cardiac data-sets in content-based image retrieval

Apprentissage automatique pour simplifier l’utilisation de banques d’images cardiaques

The recent growth of data in cardiac databases has been phenomenal. Cleveruse of these databases could help find supporting evidence for better diagnosis and treatment planning. In addition to the challenges inherent to the large quantity of data, the databases are difficult to use in their current state. Data coming from multiple sources are often unstructured, the image content is variable and the metadata are not standardised. The objective of this thesis is therefore to simplify the use of large databases for cardiology specialists withautomated image processing, analysis and interpretation tools. The proposed tools are largely based on supervised machine learning techniques, i.e. algorithms which can learn from large quantities of cardiac images with groundtruth annotations and which automatically find the best representations. First, the inconsistent metadata are cleaned, interpretation and visualisation of images is improved by automatically recognising commonly used cardiac magnetic resonance imaging views from image content. The method is based on decision forests and convolutional neural networks trained on a large image dataset. Second, the thesis explores ways to use machine learning for extraction of relevant clinical measures (e.g. volumes and masses) from3D and 3D+t cardiac images. New spatio-temporal image features are designed andclassification forests are trained to learn how to automatically segment the main cardiac structures (left ventricle and left atrium) from voxel-wise label maps. Third, a web interface is designed to collect pairwise image comparisons and to learn how to describe the hearts with semantic attributes (e.g. dilation, kineticity). In the last part of the thesis, a forest-based machinelearning technique is used to map cardiac images to establish distances and neighborhoods between images. One application is retrieval of the most similar images.L'explosion récente de données d'imagerie cardiaque a été phénoménale. L'utilisation intelligente des grandes bases de données annotées pourrait constituer une aide précieuse au diagnostic et à la planification de thérapie. En plus des défis inhérents à la grande taille de ces banques de données, elles sont difficilement utilisables en l'état. Les données ne sont pas structurées, le contenu des images est variable et mal indexé, et les métadonnées ne sont pas standardisées. L'objectif de cette thèse est donc le traitement, l'analyse et l'interprétation automatique de ces bases de données afin de faciliter leur utilisation par les spécialistes de cardiologie. Dans ce but, la thèse explore les outils d'apprentissage automatique supervisé, ce qui aide à exploiter ces grandes quantités d'images cardiaques et trouver de meilleures représentations. Tout d'abord, la visualisation et l'interprétation d'images est améliorée en développant une méthode de reconnaissance automatique des plans d'acquisition couramment utilisés en imagerie cardiaque. La méthode se base sur l'apprentissage par forêts aléatoires et par réseaux de neurones à convolution, en utilisant des larges banques d'images, où des types de vues cardiaques sont préalablement établies. La thèse s'attache dans un deuxième temps au traitement automatique des images cardiaques, avec en perspective l'extraction d'indices cliniques pertinents. La segmentation des structures cardiaques est une étape clé de ce processus. A cet effet une méthode basée sur les forêts aléatoires qui exploite des attributs spatio-temporels originaux pour la segmentation automatique dans des images 3Det 3D+t est proposée. En troisième partie, l'apprentissage supervisé de sémantique cardiaque est enrichi grâce à une méthode de collecte en ligne d'annotations d'usagers. Enfin, la dernière partie utilise l'apprentissage automatique basé sur les forêts aléatoires pour cartographier des banques d'images cardiaques, tout en établissant les notions de distance et de voisinage d'images. Une application est proposée afin de retrouver dans une banque de données, les images les plus similaires à celle d'un nouveau patient

Strategies for image visualisation and browsing

PhDThe exploration of large information spaces has remained a challenging task even though the proliferation of database management systems and the state-of-the art retrieval algorithms is becoming pervasive. Signi cant research attention in the multimedia domain is focused on nding automatic algorithms for organising digital image collections into meaningful structures and providing high-semantic image indices. On the other hand, utilisation of graphical and interactive methods from information visualisation domain, provide promising direction for creating e cient user-oriented systems for image management. Methods such as exploratory browsing and query, as well as intuitive visual overviews of image collection, can assist the users in nding patterns and developing the understanding of structures and content in complex image data-sets. The focus of the thesis is combining the features of automatic data processing algorithms with information visualisation. The rst part of this thesis focuses on the layout method for displaying the collection of images indexed by low-level visual descriptors. The proposed solution generates graphical overview of the data-set as a combination of similarity based visualisation and random layout approach. Second part of the thesis deals with problem of visualisation and exploration for hierarchical organisation of images. Due to the absence of the semantic information, images are considered the only source of high-level information. The content preview and display of hierarchical structure are combined in order to support image retrieval. In addition to this, novel exploration and navigation methods are proposed to enable the user to nd the way through database structure and retrieve the content. On the other hand, semantic information is available in cases where automatic or semi-automatic image classi ers are employed. The automatic annotation of image items provides what is referred to as higher-level information. This type of information is a cornerstone of multi-concept visualisation framework which is developed as a third part of this thesis. This solution enables dynamic generation of user-queries by combining semantic concepts, supported by content overview and information ltering. Comparative analysis and user tests, performed for the evaluation of the proposed solutions, focus on the ways information visualisation a ects the image content exploration and retrieval; how e cient and comfortable are the users when using di erent interaction methods and the ways users seek for information through di erent types of database organisation

Planification de l’ablation radiofréquence des arythmies cardiaques en combinant modélisation et apprentissage automatique

Cardiac arrhythmias are heart rhythm disruptions which can lead to sudden cardiac death. They require a deeper understanding for appropriate treatment planning. In this thesis, we integrate personalized structural and functional data into a 3D tetrahedral mesh of the biventricular myocardium. Next, the Mitchell-Schaeffer (MS) simplified biophysical model is used to study the spatial heterogeneity of electrophysiological (EP) tissue properties and their role in arrhythmogenesis. Radiofrequency ablation (RFA) with the elimination of local abnormal ventricular activities (LAVA) has recently arisen as a potentially curative treatment for ventricular tachycardia but the EP studies required to locate LAVA are lengthy and invasive. LAVA are commonly found within the heterogeneous scar, which can be imaged non-invasively with 3D delayed enhanced magnetic resonance imaging (DE-MRI). We evaluate the use of advanced image features in a random forest machine learning framework to identify areas of LAVA-inducing tissue. Furthermore, we detail the dataset’s inherent error sources and their formal integration in the training process. Finally, we construct MRI-based structural patient-specific heart models and couple them with the MS model. We model a recording catheter using a dipole approach and generate distinct normal and LAVA-like electrograms at locations where they have been found in clinics. This enriches our predictions of the locations of LAVA-inducing tissue obtained through image-based learning. Confidence maps can be generated and analyzed prior to RFA to guide the intervention. These contributions have led to promising results and proofs of concepts.Les arythmies sont des perturbations du rythme cardiaque qui peuvent entrainer la mort subite et requièrent une meilleure compréhension pour planifier leur traitement. Dans cette thèse, nous intégrons des données structurelles et fonctionnelles à un maillage 3D tétraédrique biventriculaire. Le modèle biophysique simplifié de Mitchell-Schaeffer (MS) est utilisé pour étudier l’hétérogénéité des propriétés électrophysiologiques (EP) du tissu et leur rôle sur l’arythmogénèse. L’ablation par radiofréquence (ARF) en éliminant les activités ventriculaires anormales locales (LAVA) est un traitement potentiellement curatif pour la tachycardie ventriculaire, mais les études EP requises pour localiser les LAVA sont longues et invasives. Les LAVA se trouvent autour de cicatrices hétérogènes qui peuvent être imagées de façon non-invasive par IRM à rehaussement tardif. Nous utilisons des caractéristiques d’image dans un contexte d’apprentissage automatique avec des forêts aléatoires pour identifier des aires de tissu qui induisent des LAVA. Nous détaillons les sources d’erreur inhérentes aux données et leur intégration dans le processus d’apprentissage. Finalement, nous couplons le modèle MS avec des géométries du coeur spécifiques aux patients et nous modélisons le cathéter avec une approche par un dipôle pour générer des électrogrammes normaux et des LAVA aux endroits où ils ont été localisés en clinique. Cela améliore la prédiction de localisation du tissu induisant des LAVA obtenue par apprentissage sur l’image. Des cartes de confiance sont générées et peuvent être utilisées avant une ARF pour guider l’intervention. Les contributions de cette thèse ont conduit à des résultats et des preuves de concepts prometteurs

Two Dimensional (2D) Visual Tracking in Construction Scenarios

The tracking of construction resources (e.g. workforce and equipment) in videos, i.e., two-dimensional (2D) visual tracking, has gained significant interests in the construction industries. There exist lots of research studies that relied on 2D visual tracking methods to support the surveillance of construction productivity, safety, and project progress. However, few efforts have been put on evaluating the accuracy and robustness of these tracking methods in the construction scenarios. Meanwhile, it is noticed that state-of-art tracking methods have not shown reliable performance in tracking articulated equipment, such as excavators, backhoes, and dozers etc. The main objective of this research is to fill these knowledge gaps. First, a total of fifth (15) 2D visual tracking methods were selected here due to their excellent performances identified in the computer vision field. Then, the methods were tested with twenty (20) videos captured from multiple construction job sites at day and night. The videos contain construction resources, including but not limited to excavators, backhoes, and compactors. Also, they were characterized by the attributes, such as occlusions, scale variation, and background clutter, in order to provide a comprehensive evaluation. The tracking results were evaluated with the sequence overlap score, center error ratio, and tracking length ratio respectively. According to the quantitative comparison of tracking methods, two improvements were further conducted. One is to fuse the tracking results of individual tracking methods based on the non-maximum suppression. The other is to track the articulated equipment by proposing the idea of tracking the equipment parts respectively. The test results from this research study indicated that 1) the methods built on the local sparse representation were more effective; 2) the generative tracking strategy typically outperformed the discriminative one, when being adopted to track the equipment and workforce in the construction scenarios; 3) the fusion of the results from different tracking methods increased the tracking performance by 10% in accuracy; and 4) the part-based tracking methods improved the tracking performance in both accuracy and robustness, when being used to track the articulated equipment

Virtuaalse proovikabiini 3D kehakujude ja roboti juhtimisalgoritmide uurimine

Väitekirja elektrooniline versioon ei sisalda publikatsiooneVirtuaalne riiete proovimine on üks põhilistest teenustest, mille pakkumine võib suurendada rõivapoodide edukust, sest tänu sellele lahendusele väheneb füüsilise töö vajadus proovimise faasis ning riiete proovimine muutub kasutaja jaoks mugavamaks. Samas pole enamikel varem välja pakutud masinnägemise ja graafika meetoditel õnnestunud inimkeha realistlik modelleerimine, eriti terve keha 3D modelleerimine, mis vajab suurt kogust andmeid ja palju arvutuslikku ressurssi. Varasemad katsed on ebaõnnestunud põhiliselt seetõttu, et ei ole suudetud korralikult arvesse võtta samaaegseid muutusi keha pinnal. Lisaks pole varasemad meetodid enamasti suutnud kujutiste liikumisi realistlikult reaalajas visualiseerida. Käesolev projekt kavatseb kõrvaldada eelmainitud puudused nii, et rahuldada virtuaalse proovikabiini vajadusi. Välja pakutud meetod seisneb nii kasutaja keha kui ka riiete skaneerimises, analüüsimises, modelleerimises, mõõtmete arvutamises, orientiiride paigutamises, mannekeenidelt võetud 3D visuaalsete andmete segmenteerimises ning riiete mudeli paigutamises ja visualiseerimises kasutaja kehal. Selle projekti käigus koguti visuaalseid andmeid kasutades 3D laserskannerit ja Kinecti optilist kaamerat ning koostati nendest andmebaas. Neid andmeid kasutati välja töötatud algoritmide testimiseks, mis peamiselt tegelevad riiete realistliku visuaalse kujutamisega inimkehal ja suuruse pakkumise süsteemi täiendamisega virtuaalse proovikabiini kontekstis.Virtual fitting constitutes a fundamental element of the developments expected to rise the commercial prosperity of online garment retailers to a new level, as it is expected to reduce the load of the manual labor and physical efforts required. Nevertheless, most of the previously proposed computer vision and graphics methods have failed to accurately and realistically model the human body, especially, when it comes to the 3D modeling of the whole human body. The failure is largely related to the huge data and calculations required, which in reality is caused mainly by inability to properly account for the simultaneous variations in the body surface. In addition, most of the foregoing techniques cannot render realistic movement representations in real-time. This project intends to overcome the aforementioned shortcomings so as to satisfy the requirements of a virtual fitting room. The proposed methodology consists in scanning and performing some specific analyses of both the user's body and the prospective garment to be virtually fitted, modeling, extracting measurements and assigning reference points on them, and segmenting the 3D visual data imported from the mannequins. Finally, superimposing, adopting and depicting the resulting garment model on the user's body. The project is intended to gather sufficient amounts of visual data using a 3D laser scanner and the Kinect optical camera, to manage it in form of a usable database, in order to experimentally implement the algorithms devised. The latter will provide a realistic visual representation of the garment on the body, and enhance the size-advisor system in the context of the virtual fitting room under study