Developing a Semantic-Driven Hybrid Segmentation Method for Point Clouds of 3D Shapes

With the rapid development of point cloud processing technologies and the availability of a wide range of 3D capturing devices, a geometric object from the real world can be directly represented digitally as a dense and fine point cloud. Decomposing a 3D shape represented in point cloud into meaningful parts has very important practical implications in the fields of computer graphics, virtual reality and mixed reality. In this paper, a semantic-driven automated hybrid segmentation method is proposed for 3D point cloud shapes. Our method consists of three stages: semantic clustering, variational merging, and region remerging. In the first stage, a new feature of point cloud, called Local Concave-Convex Histogram, is introduced to first extract saddle regions complying with the semantic boundary feature. All other types of regions are then aggregated according to this extracted feature. This stage often leads to multiple over-segmentation convex regions, which are then remerged by a variational method established based on the narrow-band theory. Finally, in order to recombine the regions with the approximate shapes, order relation is introduced to improve the weighting forms in calculating the conventional Shape Diameter Function. We have conducted extensive experiments with the Princeton Dataset. The results show that the proposed algorithm outperforms the state-of-the-art algorithms in this area. We have also applied the proposed algorithm to process the point cloud data acquired directly from the real 3D objects. It achieves excellent results too. These results demonstrate that the method proposed in this paper is effective and universal

A multi-scale imaging approach to understand osteoarthritis development

X-ray phase-contrast imaging is an innovative and advanced imaging method. Contrary to conventional radiology, where the image contrast is primarily determined by X-ray attenuation, phase-contrast images contain additional information generated by the phase shifts or refraction of the X-rays passing through matter. The refractive effect on tissue samples is orders of magnitude higher than the absorption effect in the X-ray energy range used in biomedical imaging. This technique makes it possible to produce excellent and enhanced image contrast, particularly when examining soft biological tissues or features with similar X-ray attenuation properties. In combination with high spatial resolution detector technology and computer tomography, X-ray phase-contrast imaging has been proved to be a powerful method to examine tissue morphology and the evolution of pathologies three-dimensionally, with great detail and without the need of contrast agents. This Thesis work has focused on developing an accurate, multi-scale X-ray-based methodology for imaging and characterizing the early stages of osteoarthritis. X-ray phase-contrast images acquired at different spatial resolutions provide unprecedented insights into cartilage and the development of its degeneration, i.e., osteoarthritis. Other types of X-ray phase-contrast imaging techniques and setups using spatial resolutions ranging from micrometer down to nanometer were applied. Lower spatial resolutions allow large sample coverage and comprehensive representations, while the nanoscale analysis provides a precise depiction of anatomical details and pathological signs. X-ray phase-contrast results are correlated to data obtained, on the same specimens, by standard laboratory methods, such as histology and transmission electron microscopy. Furthermore, X-ray phase-contrast images of cartilage were acquired using different X-ray sources and results were compared in terms of image quality. It was shown that with the use of synchrotron radiation, more detailed images and much faster data acquisitions could be achieved. A second focus in this Thesis work has been the investigation of the reaction of healthy and degenerated cartilage under different physical pressures, simulating the different levels of stress to which the tissue is subject during daily movements. A specifically designed setup was used to dynamically study cartilage response to varying pressures with X-ray phase-contrast micro-computed tomography, and a fully volumetric and quantitative methodology to accurately describe the tissue morphological variations. This study revealed changes in the behavior of the cartilage cell structure, which differ between normal and osteoarthritic cartilage tissues. The third focus of this Thesis is the realization of an automated evaluation procedure for the discrimination of healthy and cartilage images with osteoarthritis. In recent years, developments in neural networks have shown that they are excellently suited for image classification tasks. The transfer learning method was applied, in which a pre-trained neural network with cartilage images is further trained and then used for classification. This enables a fast, robust and automated grouping of images with pathological findings. A neural network constructed in this way could be used as a supporting instrument in pathology. X-ray phase-contrast imaging computed tomography can provide a powerful tool for a fully 3D, highly accurate and quantitative depiction and characterization of healthy and early stage-osteoarthritic cartilage, supporting the understanding of the development of osteoarthritis.Röntgen-Phasenkontrast-Bildgebung ist eine innovative und weiterführende Bildgebungsmethode. Im Gegensatz zu herkömlichen Absorptions-Röntgenaufnahmen, wie sie in der Radiologie verwendet werden, wird der Kontrast bei dieser Methode aus dem Effekt der Phasenverschiebung oder auch Brechung der Röngtenstrahlen gebildet. Der Brechungseffekt bei Gewebeproben ist um ein Vielfaches höher als der Absorptionseffekt des elektromagnetischen Spektrums der Röntgenstrahlen. Diese Methode ermöglicht die Darstellung von großen Kontraste im Gewebe. Unter Verwendung eines hochauflösenden Detektors und in Kombination mit der Computer-Tomographie, ist Phasenkontrast-Bildgebung eine sehr gute Methode um Knorpelgewebe und Arthrose im Knorpel zu untersuchen. Diese Arbeit beschreibt primär ein Verfahren zur Darstellung arthrotischen Knorpels im Anfangsstadium. Die mit verschiedenen Auflösungen und 3D-Phasen-Kontrast-Methoden produzierten Aufnahmen ermöglichen einen noch nie dagewesenen Einblick in den Knorpel und die Entwicklung von Arthrose im Anfangsstadium. Hierbei kam die propagationsbasierte Phasenkontrastmethode mit einer Auflösung im mikrometer Bereich und die (Nano)-Holotomographie-Methode mit einer Auflösung im Submicrometer Bereich zum Einsatz. Durch Auflösung im mikrometer Bereich kann ein großes Volumen im Knorpel gescannt werden, während die Nano-Holotomographie Methode eine sehr große Detailauflösung aufweißt. Die Phasenkontrast-Aufnahmen werden mit zwei anderen wissenschaftlichen Methoden verglichen: mikroskopische Abbildungen histologisch aufgearbeiteter Knorpelproben und Aufnahmen eines Transmissionselektroskop zeigen sehr große Übereinstimmungen zur Röntgen-Phasenkontrast-Bildgebung. Desweiteren wurden Phasenkontrast-Aufnahmen von Knorpel aus unterschiedlichen Röntgenquellen verglichen. Hierbei zeigte sich, dass mit Hilfe des Teilchenbeschleunigers (Synchrotron) detailreichere und schnellere Aufnahmen erzielt werden können. Bilder aus Flüssig-Metall-Quellen zeigen sich durchaus von guter Qualität, erfordern jedoch sehr lange Aufnahmezeiten. In dieser Arbeit wird zudem das Verhalten von Knorpelgewebe, welches ein Anfangsstadium von Arthrose aufweist, unter physikalischem Druck untersucht. Hierfür wurden 3D-Computertomographie-Aufnahmen von komprimiertem Knorpelgewebe angefertig und mit Aufnahmen ohne Komprimierung verglichen. Ein quantitativer Vergleich machte Veränderungen des Verhaltens der Knorpelzellstruktur (Chondronen) sichtbar. Es konnte gezeigt werden, dass Chondrone bei arthrotischem Knorpel ein verändertes Kompressionsverhalten haben. Der dritte Fokus dieser Arbeit liegt auf der automatisierten Auswertung von Aufnahmen gesunden und arthrotischen Knorpelgewebes. Die Entwicklungen im Bereich der Neuronale Netze zeigten in den letzten Jahren, dass diese sich hervoragend für Bildklassifizierungsaufgaben eignen. Es wurde die Methode des transferierenden Lernens angewandt, bei der ein vortrainiertes Neuronales Netz mit Knorpelbildern weitertrainiert und anschließend zur Klassifizierung eingesetzt wird. Dadurch ist eine schnelle, robuste und automatisierte Gruppierung von Bildern mit pathologischen Befunden möglich. Ein derart konstruiertes Neuronales Netz könnte als unterstützendes Instrument in der Pathologie angewandt werden. Röntgen-Phasenkontrast-CT kann ein leistungsstarkes Werkzeug für eine umfassende, hochpräzise und quantitative 3D-Darstellung und Charakterisierung von gesundem Knorpel und athrotischem Knorpel im Frühstadium bieten, um das Verständnis der Entwicklung von Osteoarthritis zu erweitern

Similarity reasoning for local surface analysis and recognition

This thesis addresses the similarity assessment of digital shapes, contributing to the analysis of surface characteristics that are independent of the global shape but are crucial to identify a model as belonging to the same manufacture, the same origin/culture or the same typology (color, common decorations, common feature elements, compatible style elements, etc.). To face this problem, the interpretation of the local surface properties is crucial. We go beyond the retrieval of models or surface patches in a collection of models, facing the recognition of geometric patterns across digital models with different overall shape. To address this challenging problem, the use of both engineered and learning-based descriptions are investigated, building one of the first contributions towards the localization and identification of geometric patterns on digital surfaces. Finally, the recognition of patterns adds a further perspective in the exploration of (large) 3D data collections, especially in the cultural heritage domain. Our work contributes to the definition of methods able to locally characterize the geometric and colorimetric surface decorations. Moreover, we showcase our benchmarking activity carried out in recent years on the identification of geometric features and the retrieval of digital models completely characterized by geometric or colorimetric patterns

Automatic Structural Scene Digitalization

In this paper, we present an automatic system for the analysis and labeling of structural scenes, floor plan drawings in Computer-aided Design (CAD) format. The proposed system applies a fusion strategy to detect and recognize various components of CAD floor plans, such as walls, doors, windows and other ambiguous assets. Technically, a general rule-based filter parsing method is fist adopted to extract effective information from the original floor plan. Then, an image-processing based recovery method is employed to correct information extracted in the first step. Our proposed method is fully automatic and real-time. Such analysis system provides high accuracy and is also evaluated on a public website that, on average, archives more than ten thousands effective uses per day and reaches a relatively high satisfaction rate.Comment: paper submitted to PloS On

Shape segmentation and retrieval based on the skeleton cut space

3D vormverzamelingen groeien snel in veel toepassingsgebieden. Om deze effectief te kunnen gebruiken bij modelleren, simuleren, of 3D contentontwikkeling moet men 3D vormen verwerken. Voorbeelden hiervan zijn het snijden van een vorm in zijn natuurlijke onderdelen (ook bekend als segmentatie), en het vinden van vormen die lijken op een gegeven model in een grote vormverzameling (ook bekend als opvraging). Dit proefschrift presenteert nieuwe methodes voor 3D vormsegmentatie en vormopvraging die gebaseerd zijn op het zogenaamde oppervlakskelet van een 3D vorm. Hoewel allang bekend, dergelijke skeletten kunnen alleen sinds kort snel, robuust, en bijna automatisch berekend worden. Deze ontwikkelingen stellen ons in staat om oppervlakskeletten te gebruiken om vormen te karakteriseren en analyseren zodat operaties zoals segmentatie en opvraging snel en automatisch gedaan kunnen worden. We vergelijken onze nieuwe methodes met moderne methodes voor dezelfde doeleinden en laten zien dat ons aanpak kwalitatief betere resultaten kan produceren. Ten slotte presenteren wij een nieuwe methode om oppervlakskeletten te extraheren die is veel simpeler dan, en heeft vergelijkbare snelheid met, de beste technieken in zijn klasse. Samenvattend, dit proefschrift laat zien hoe men een complete workflow kan implementeren voor het segmenteren en opvragen van 3D vormen gebruik makend van oppervlakskeletten alleen

Analysis of 3D human gait reconstructed with a depth camera and mirrors

L'évaluation de la démarche humaine est l'une des composantes essentielles dans les soins de santé. Les systèmes à base de marqueurs avec plusieurs caméras sont largement utilisés pour faire cette analyse. Cependant, ces systèmes nécessitent généralement des équipements spécifiques à prix élevé et/ou des moyens de calcul intensif. Afin de réduire le coût de ces dispositifs, nous nous concentrons sur un système d'analyse de la marche qui utilise une seule caméra de profondeur. Le principe de notre travail est similaire aux systèmes multi-caméras, mais l'ensemble de caméras est remplacé par un seul capteur de profondeur et des miroirs. Chaque miroir dans notre configuration joue le rôle d'une caméra qui capture la scène sous un point de vue différent. Puisque nous n'utilisons qu'une seule caméra, il est ainsi possible d'éviter l'étape de synchronisation et également de réduire le coût de l'appareillage. Notre thèse peut être divisée en deux sections: reconstruction 3D et analyse de la marche. Le résultat de la première section est utilisé comme entrée de la seconde. Notre système pour la reconstruction 3D est constitué d'une caméra de profondeur et deux miroirs. Deux types de capteurs de profondeur, qui se distinguent sur la base du mécanisme d'estimation de profondeur, ont été utilisés dans nos travaux. Avec la technique de lumière structurée (SL) intégrée dans le capteur Kinect 1, nous effectuons la reconstruction 3D à partir des principes de l'optique géométrique. Pour augmenter le niveau des détails du modèle reconstruit en 3D, la Kinect 2 qui estime la profondeur par temps de vol (ToF), est ensuite utilisée pour l'acquisition d'images. Cependant, en raison de réflections multiples sur les miroirs, il se produit une distorsion de la profondeur dans notre système. Nous proposons donc une approche simple pour réduire cette distorsion avant d'appliquer les techniques d'optique géométrique pour reconstruire un nuage de points de l'objet 3D. Pour l'analyse de la démarche, nous proposons diverses alternatives centrées sur la normalité de la marche et la mesure de sa symétrie. Cela devrait être utile lors de traitements cliniques pour évaluer, par exemple, la récupération du patient après une intervention chirurgicale. Ces méthodes se composent d'approches avec ou sans modèle qui ont des inconvénients et avantages différents. Dans cette thèse, nous présentons 3 méthodes qui traitent directement les nuages de points reconstruits dans la section précédente. La première utilise la corrélation croisée des demi-corps gauche et droit pour évaluer la symétrie de la démarche, tandis que les deux autres methodes utilisent des autoencodeurs issus de l'apprentissage profond pour mesurer la normalité de la démarche.The problem of assessing human gaits has received a great attention in the literature since gait analysis is one of key components in healthcare. Marker-based and multi-camera systems are widely employed to deal with this problem. However, such systems usually require specific equipments with high price and/or high computational cost. In order to reduce the cost of devices, we focus on a system of gait analysis which employs only one depth sensor. The principle of our work is similar to multi-camera systems, but the collection of cameras is replaced by one depth sensor and mirrors. Each mirror in our setup plays the role of a camera which captures the scene at a different viewpoint. Since we use only one camera, the step of synchronization can thus be avoided and the cost of devices is also reduced. Our studies can be separated into two categories: 3D reconstruction and gait analysis. The result of the former category is used as the input of the latter one. Our system for 3D reconstruction is built with a depth camera and two mirrors. Two types of depth sensor, which are distinguished based on the scheme of depth estimation, have been employed in our works. With the structured light (SL) technique integrated into the Kinect 1, we perform the 3D reconstruction based on geometrical optics. In order to increase the level of details of the 3D reconstructed model, the Kinect 2 with time-of-flight (ToF) depth measurement is used for image acquisition instead of the previous generation. However, due to multiple reflections on the mirrors, depth distortion occurs in our setup. We thus propose a simple approach for reducing such distortion before applying geometrical optics to reconstruct a point cloud of the 3D object. For the task of gait analysis, we propose various alternative approaches focusing on the problem of gait normality/symmetry measurement. They are expected to be useful for clinical treatments such as monitoring patient's recovery after surgery. These methods consist of model-free and model-based approaches that have different cons and pros. In this dissertation, we present 3 methods that directly process point clouds reconstructed from the previous work. The first one uses cross-correlation of left and right half-bodies to assess gait symmetry while the other ones employ deep auto-encoders to measure gait normality

Symbolic and Visual Retrieval of Mathematical Notation using Formula Graph Symbol Pair Matching and Structural Alignment

Large data collections containing millions of math formulae in different formats are available on-line. Retrieving math expressions from these collections is challenging. We propose a framework for retrieval of mathematical notation using symbol pairs extracted from visual and semantic representations of mathematical expressions on the symbolic domain for retrieval of text documents. We further adapt our model for retrieval of mathematical notation on images and lecture videos. Graph-based representations are used on each modality to describe math formulas. For symbolic formula retrieval, where the structure is known, we use symbol layout trees and operator trees. For image-based formula retrieval, since the structure is unknown we use a more general Line of Sight graph representation. Paths of these graphs define symbol pairs tuples that are used as the entries for our inverted index of mathematical notation. Our retrieval framework uses a three-stage approach with a fast selection of candidates as the first layer, a more detailed matching algorithm with similarity metric computation in the second stage, and finally when relevance assessments are available, we use an optional third layer with linear regression for estimation of relevance using multiple similarity scores for final re-ranking. Our model has been evaluated using large collections of documents, and preliminary results are presented for videos and cross-modal search. The proposed framework can be adapted for other domains like chemistry or technical diagrams where two visually similar elements from a collection are usually related to each other

Shape Retrieval Methods for Architectural 3D Models

This thesis introduces new methods for content-based retrieval of architecture-related 3D models. We thereby consider two different overall types of architectural 3D models. The first type consists of context objects that are used for detailed design and decoration of 3D building model drafts. This includes e.g. furnishing for interior design or barriers and fences for forming the exterior environment. The second type consists of actual building models. To enable efficient content-based retrieval for both model types that is tailored to the user requirements of the architectural domain, type-specific algorithms must be developed. On the one hand, context objects like furnishing that provide similar functions (e.g. seating furniture) often share a similar shape. Nevertheless they might be considered to belong to different object classes from an architectural point of view (e.g. armchair, elbow chair, swivel chair). The differentiation is due to small geometric details and is sometimes only obvious to an expert from the domain. Building models on the other hand are often distinguished according to the underlying floor- and room plans. Topological floor plan properties for example serve as a starting point for telling apart residential and commercial buildings. The first contribution of this thesis is a new meta descriptor for 3D retrieval that combines different types of local shape descriptors using a supervised learning approach. The approach enables the differentiation of object classes according to small geometric details and at the same time integrates expert knowledge from the field of architecture. We evaluate our approach using a database containing arbitrary 3D models as well as on one that only consists of models from the architectural domain. We then further extend our approach by adding a sophisticated shape descriptor localization strategy. Additionally, we exploit knowledge about the spatial relationship of object components to further enhance the retrieval performance. In the second part of the thesis we introduce attributed room connectivity graphs (RCGs) as a means to characterize a 3D building model according to the structure of its underlying floor plans. We first describe how RCGs are inferred from a given building model and discuss how substructures of this graph can be queried efficiently. We then introduce a new descriptor denoted as Bag-of-Attributed-Subgraphs that transforms attributed graphs into a vector-based representation using subgraph embeddings. We finally evaluate the retrieval performance of this new method on a database consisting of building models with different floor plan types. All methods presented in this thesis are aimed at an as automated as possible workflow for indexing and retrieval such that only minimum human interaction is required. Accordingly, only polygon soups are required as inputs which do not need to be manually repaired or structured. Human effort is only needed for offline groundtruth generation to enable supervised learning and for providing information about the orientation of building models and the unit of measurement used for modeling

Computation applications in archaeology

This thesis is a critical analysis of the use which has been made of the computer in archaeology up to the year 1972. The main chapters cover the applications in archaeology of Statistics, Information Retrieval, Graphics, Pottery Classification and Survey Reduction. A large body of Miscellaneous Applications, including Pollen Analysis, are also examined. The majority of computer applications have been in Statistics. These applications include Numerical Taxonomy, Matrix Manipulation and Seriation, the generation of hypotheses and models, MUltidimensional Scaling, Cumulative Percentage Graphs and Trend Surface Analysis. It is worthwhile to note that for small sets of data several manual methods give comparable results to complex computer analyses and at far less cost. Computer Information Retrieval is examined in the light of its use for large bodies of specialist archaeological information, for museum cataloguing, and for the compilation of a site excavation record using a remote terminal. The use of Computer Graphics in the production of archaeological maps, plans and diagrams is examined. Facilities include the production of dot-density plots, distribution maps, histograms, piecharts, pottery diagrams, site block diagrams with 3D rotation and perspective, sections, pit outlines and projectile point classification by Fourier analysis. The use of the d-Mac Pencil Follower in the objective classification of pottery is described, followed by computer analysis of the resultant multivariate data. The use of the computer in the routine reduction of geophysical observations taken on archaeological sites is described. Complex filtering procedures for the removal of background effects and the enhancement of the archaeological anomalies are examined. Since other workers have concentrated on the applications of statistics in archaeology~ this thesis explores the relatively neglected fields of Graphics and Pottery Classification. Evidence is presented that significant advances have been made in the classification of pottery vessels and projectile points~ and in the graphical output of results. A number of new programs have been developed; these include software which may be operated from a remote terminal at an archaeological site. The P L U T A R C H System (Program Library Useful To ARCHaeologists) is described. This is a control program which uses interactive graphics and overlays to combine all the computer facilities available to the archaeologist. The individual graphics, statistics, instrument survey plotting and information retrieval techniques when combined in this way can communicate via global storage, and become even more powerful

Human-Centered Content-Based Image Retrieval

Retrieval of images that lack a (suitable) annotations cannot be achieved through (traditional) Information Retrieval (IR) techniques. Access through such collections can be achieved through the application of computer vision techniques on the IR problem, which is baptized Content-Based Image Retrieval (CBIR). In contrast with most purely technological approaches, the thesis Human-Centered Content-Based Image Retrieval approaches the problem from a human/user centered perspective. Psychophysical experiments were conducted in which people were asked to categorize colors. The data gathered from these experiments was fed to a Fast Exact Euclidean Distance (FEED) transform (Schouten & Van den Broek, 2004), which enabled the segmentation of color space based on human perception (Van den Broek et al., 2008). This unique color space segementation was exploited for texture analysis and image segmentation, and subsequently for full-featured CBIR. In addition, a unique CBIR-benchmark was developed (Van den Broek et al., 2004, 2005). This benchmark was used to explore what and how several parameters (e.g., color and distance measures) of the CBIR process influence retrieval results. In contrast with other research, users judgements were assigned as metric. The online IR and CBIR system Multimedia for Art Retrieval (M4ART) (URL: http://www.m4art.org) has been (partly) founded on the techniques discussed in this thesis. References: - Broek, E.L. van den, Kisters, P.M.F., and Vuurpijl, L.G. (2004). The utilization of human color categorization for content-based image retrieval. Proceedings of SPIE (Human Vision and Electronic Imaging), 5292, 351-362. [see also Chapter 7] - Broek, E.L. van den, Kisters, P.M.F., and Vuurpijl, L.G. (2005). Content-Based Image Retrieval Benchmarking: Utilizing Color Categories and Color Distributions. Journal of Imaging Science and Technology, 49(3), 293-301. [see also Chapter 8] - Broek, E.L. van den, Schouten, Th.E., and Kisters, P.M.F. (2008). Modeling Human Color Categorization. Pattern Recognition Letters, 29(8), 1136-1144. [see also Chapter 5] - Schouten, Th.E. and Broek, E.L. van den (2004). Fast Exact Euclidean Distance (FEED) transformation. In J. Kittler, M. Petrou, and M. Nixon (Eds.), Proceedings of the 17th IEEE International Conference on Pattern Recognition (ICPR 2004), Vol 3, p. 594-597. August 23-26, Cambridge - United Kingdom. [see also Appendix C