A Survey on Artificial Intelligence Techniques for Biomedical Image Analysis in Skeleton-Based Forensic Human Identification

This paper represents the first survey on the application of AI techniques for the analysis of biomedical images with forensic human identification purposes. Human identification is of great relevance in today’s society and, in particular, in medico-legal contexts. As consequence, all technological advances that are introduced in this field can contribute to the increasing necessity for accurate and robust tools that allow for establishing and verifying human identity. We first describe the importance and applicability of forensic anthropology in many identification scenarios. Later, we present the main trends related to the application of computer vision, machine learning and soft computing techniques to the estimation of the biological profile, the identification through comparative radiography and craniofacial superimposition, traumatism and pathology analysis, as well as facial reconstruction. The potentialities and limitations of the employed approaches are described, and we conclude with a discussion about methodological issues and future research.Spanish Ministry of Science, Innovation and UniversitiesEuropean Union (EU) PGC2018-101216-B-I00Regional Government of Andalusia under grant EXAISFI P18-FR-4262Instituto de Salud Carlos IIIEuropean Union (EU) DTS18/00136European Commission H2020-MSCA-IF-2016 through the Skeleton-ID Marie Curie Individual Fellowship 746592Spanish Ministry of Science, Innovation and Universities-CDTI, Neotec program 2019 EXP-00122609/SNEO-20191236European Union (EU)Xunta de Galicia ED431G 2019/01European Union (EU) RTI2018-095894-B-I0

Physical and statistical shape modelling in craniomaxillofacial surgery: a personalised approach for outcome prediction

Orthognathic surgery involves repositioning of the jaw bones to restore face function and shape for patients who require an operation as a result of a syndrome, due to growth disturbances in childhood or after trauma. As part of the preoperative assessment, three-dimensional medical imaging and computer-assisted surgical planning help to improve outcomes, and save time and cost. Computer-assisted surgical planning involves visualisation and manipulation of the patient anatomy and can be used to aid objective diagnosis, patient communication, outcome evaluation, and surgical simulation. Despite the benefits, the adoption of three-dimensional tools has remained limited beyond specialised hospitals and traditional two-dimensional cephalometric analysis is still the gold standard. This thesis presents a multidisciplinary approach to innovative surgical simulation involving clinical patient data, medical image analysis, engineering principles, and state-of-the-art machine learning and computer vision algorithms. Two novel three-dimensional computational models were developed to overcome the limitations of current computer-assisted surgical planning tools. First, a physical modelling approach – based on a probabilistic finite element model – provided patient-specific simulations and, through training and validation, population-specific parameters. The probabilistic model was equally accurate compared to two commercial programs whilst giving additional information regarding uncertainties relating to the material properties and the mismatch in bone position between planning and surgery. Second, a statistical modelling approach was developed that presents a paradigm shift in its modelling formulation and use. Specifically, a 3D morphable model was constructed from 5,000 non-patient and orthognathic patient faces for fully-automated diagnosis and surgical planning. Contrary to traditional physical models that are limited to a finite number of tests, the statistical model employs machine learning algorithms to provide the surgeon with a goal-driven patient-specific surgical plan. The findings in this thesis provide markers for future translational research and may accelerate the adoption of the next generation surgical planning tools to further supplement the clinical decision-making process and ultimately to improve patients’ quality of life

3D statistical shape analysis of the face in Apert syndrome

Timely diagnosis of craniofacial syndromes as well as adequate timing and choice of surgical technique are essential for proper care management. Statistical shape models and machine learning approaches are playing an increasing role in Medicine and have proven its usefulness. Frameworks that automate processes have become more popular. The use of 2D photographs for automated syndromic identification has shown its potential with the Face2Gene application. Yet, using 3D shape information without texture has not been studied in such depth. Moreover, the use of these models to understand shape change during growth and its applicability for surgical outcome measurements have not been analysed at length. This thesis presents a framework using state-of-the-art machine learning and computer vision algorithms to explore possibilities for automated syndrome identification based on shape information only. The purpose of this was to enhance understanding of the natural development of the Apert syndromic face and its abnormality as compared to a normative group. An additional method was used to objectify changes as result of facial bipartition distraction, a common surgical correction technique, providing information on the successfulness and on inadequacies in terms of facial normalisation. Growth curves were constructed to further quantify facial abnormalities in Apert syndrome over time along with 3D shape models for intuitive visualisation of the shape variations. Post-operative models were built and compared with age-matched normative data to understand where normalisation is coming short. The findings in this thesis provide markers for future translational research and may accelerate the adoption of the next generation diagnostics and surgical planning tools to further supplement the clinical decision-making process and ultimately to improve patients’ quality of life

Three-Dimensional Printing and Its Applications in Otorhinolaryngology–Head and Neck Surgery

Objective Three-dimensional (3D)-printing technology is being employed in a variety of medical and surgical specialties to improve patient care and advance resident physician training. As the costs of implementing 3D printing have declined, the use of this technology has expanded, especially within surgical specialties. This article explores the types of 3D printing available, highlights the benefits and drawbacks of each methodology, provides examples of how 3D printing has been applied within the field of otolaryngology–head and neck surgery, discusses future innovations, and explores the financial impact of these advances. Data Sources Articles were identified from PubMed and Ovid MEDLINE. Review Methods PubMed and Ovid Medline were queried for English articles published between 2011 and 2016, including a few articles prior to this time as relevant examples. Search terms included 3-dimensional printing, 3D printing, otolaryngology, additive manufacturing, craniofacial, reconstruction, temporal bone, airway, sinus, cost, and anatomic models. Conclusions Three-dimensional printing has been used in recent years in otolaryngology for preoperative planning, education, prostheses, grafting, and reconstruction. Emerging technologies include the printing of tissue scaffolds for the auricle and nose, more realistic training models, and personalized implantable medical devices. Implications for Practice After the up-front costs of 3D printing are accounted for, its utilization in surgical models, patient-specific implants, and custom instruments can reduce operating room time and thus decrease costs. Educational and training models provide an opportunity to better visualize anomalies, practice surgical technique, predict problems that might arise, and improve quality by reducing mistakes

Facial soft tissue segmentation

The importance of the face for socio-ecological interaction is the cause for a high demand on any surgical intervention on the facial musculo-skeletal system. Bones and soft-tissues are of major importance for any facial surgical treatment to guarantee an optimal, functional and aesthetical result. For this reason, surgeons want to pre-operatively plan, simulate and predict the outcome of the surgery allowing for shorter operation times and improved quality. Accurate simulation requires exact segmentation knowledge of the facial tissues. Thus semi-automatic segmentation techniques are required. This thesis proposes semi-automatic methods for segmentation of the facial soft-tissues, such as muscles, skin and fat, from CT and MRI datasets, using a Markov Random Fields (MRF) framework. Due to image noise, artifacts, weak edges and multiple objects of similar appearance in close proximity, it is difficult to segment the object of interest by using image information alone. Segmentations would leak at weak edges into neighboring structures that have a similar intensity profile. To overcome this problem, additional shape knowledge is incorporated in the energy function which can then be minimized using Graph-Cuts (GC). Incremental approaches by incorporating additional prior shape knowledge are presented. The proposed approaches are not object specific and can be applied to segment any class of objects be that anatomical or non-anatomical from medical or non-medical image datasets, whenever a statistical model is present. In the first approach a 3D mean shape template is used as shape prior, which is integrated into the MRF based energy function. Here, the shape knowledge is encoded into the data and the smoothness terms of the energy function that constrains the segmented parts to a reasonable shape. In the second approach, to improve handling of shape variations naturally found in the population, the fixed shape template is replaced by a more robust 3D statistical shape model based on Probabilistic Principal Component Analysis (PPCA). The advantages of using the Probabilistic PCA are that it allows reconstructing the optimal shape and computing the remaining variance of the statistical model from partial information. By using an iterative method, the statistical shape model is then refined using image based cues to get a better fitting of the statistical model to the patient's muscle anatomy. These image cues are based on the segmented muscle, edge information and intensity likelihood of the muscle. Here, a linear shape update mechanism is used to fit the statistical model to the image based cues. In the third approach, the shape refinement step is further improved by using a non-linear shape update mechanism where vertices of the 3D mesh of the statistical model incur the non-linear penalty depending on the remaining variability of the vertex. The non-linear shape update mechanism provides a more accurate shape update and helps in a finer shape fitting of the statistical model to the image based cues in areas where the shape variability is high. Finally, a unified approach is presented to segment the relevant facial muscles and the remaining facial soft-tissues (skin and fat). One soft-tissue layer is removed at a time such as the head and non-head regions followed by the skin. In the next step, bones are removed from the dataset, followed by the separation of the brain and non-brain regions as well as the removal of air cavities. Afterwards, facial fat is segmented using the standard Graph-Cuts approach. After separating the important anatomical structures, finally, a 3D fixed shape template mesh of the facial muscles is used to segment the relevant facial muscles. The proposed methods are tested on the challenging example of segmenting the masseter muscle. The datasets were noisy with almost all possessing mild to severe imaging artifacts such as high-density artifacts caused by e.g. dental fillings and dental implants. Qualitative and quantitative experimental results show that by incorporating prior shape knowledge leaking can be effectively constrained to obtain better segmentation results

Facial Reconstruction: A Systematic Review of Current Image Acquisition and Processing Techniques

Plastic and reconstructive surgery is based on a culmination of technological advances, diverse techniques, creative adaptations and strategic planning. 3D imaging is a modality that encompasses several of these criteria while encouraging the others. Imaging techniques used in facial imaging come in many different modalities and sub-modalities which is imperative for such a complex area of the body; there is a clear clinical need for hyper-specialized practice. However, with this complexity comes variability and thus there will always be an element of bias in the choices made for imaging techniques

Craniofacial Syndrome Identification Using Convolutional Mesh Autoencoders

Background: Clinical diagnosis of craniofacial anomalies requires expert knowledge. Recent studies have shown that artificial intelligence (AI) based facial analysis can match the diagnostic capabilities of expert clinicians in syndrome identification. In general, these systems use 2D images and analyse texture and colour. While these are powerful tools for photographic analysis, they are not suitable for use with medical imaging modalities such as ultrasound, MRI or CT, and are unable to take shape information into consideration when making a diagnostic prediction. 3D morphable models (3DMMs), and their recently proposed successors, mesh autoencoders, analyse surface topography rather than texture enabling analysis from photography and all common medical imaging modalities, and present an alternative to image-based analysis. // Methods: We present a craniofacial analysis framework for syndrome identification using Convolutional Mesh Autoencoders (CMAs). The models were trained using 3D photographs of the general population (LSFM and LYHM), computed tomography data (CT) scans from healthy infants and patients with 3 genetically distinct craniofacial syndromes (Muenke, Crouzon, Apert). // Findings: Machine diagnosis outperformed expert clinical diagnosis with an accuracy of 99.98%, sensitivity of 99.95% and specificity of 100%. The diagnostic precision of this technique supports its potential inclusion in clinical decision support systems. Its reliance on 3D topography characterisation makes it suitable for AI assisted diagnosis in medical imaging as well as photographic analysis in the clinical setting. // Interpretation: Our study demonstrates the use of 3D convolutional mesh autoencoders for the diagnosis of syndromic craniosynostosis. The topological nature of the tool presents opportunities for this method to be applied as a diagnostic tool across a number of 3D imaging modalities

Convolutional mesh autoencoders for the 3-dimensional identification of FGFR-related craniosynostosis

Clinical diagnosis of craniofacial anomalies requires expert knowledge. Recent studies have shown that artificial intelligence (AI) based facial analysis can match the diagnostic capabilities of expert clinicians in syndrome identification. In general, these systems use 2D images and analyse texture and colour. They are powerful tools for photographic analysis but are not suitable for use with medical imaging modalities such as ultrasound, MRI or CT, and are unable to take shape information into consideration when making a diagnostic prediction. 3D morphable models (3DMMs), and their recently proposed successors, mesh autoencoders, analyse surface topography rather than texture enabling analysis from photography and all common medical imaging modalities and present an alternative to image-based analysis. We present a craniofacial analysis framework for syndrome identification using Convolutional Mesh Autoencoders (CMAs). The models were trained using 3D photographs of the general population (LSFM and LYHM), computed tomography data (CT) scans from healthy infants and patients with 3 genetically distinct craniofacial syndromes (Muenke, Crouzon, Apert). Machine diagnosis outperformed expert clinical diagnosis with an accuracy of 99.98%, sensitivity of 99.95% and specificity of 100%. The diagnostic precision of this technique supports its potential inclusion in clinical decision support systems. Its reliance on 3D topography characterisation make it suitable for AI assisted diagnosis in medical imaging as well as photographic analysis in the clinical setting

SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator

Recent years have seen growing interest in 3D human faces modelling due to its wide applications in digital human, character generation and animation. Existing approaches overwhelmingly emphasized on modeling the exterior shapes, textures and skin properties of faces, ignoring the inherent correlation between inner skeletal structures and appearance. In this paper, we present SCULPTOR, 3D face creations with Skeleton Consistency Using a Learned Parametric facial generaTOR, aiming to facilitate easy creation of both anatomically correct and visually convincing face models via a hybrid parametric-physical representation. At the core of SCULPTOR is LUCY, the first large-scale shape-skeleton face dataset in collaboration with plastic surgeons. Named after the fossils of one of the oldest known human ancestors, our LUCY dataset contains high-quality Computed Tomography (CT) scans of the complete human head before and after orthognathic surgeries, critical for evaluating surgery results. LUCY consists of 144 scans of 72 subjects (31 male and 41 female) where each subject has two CT scans taken pre- and post-orthognathic operations. Based on our LUCY dataset, we learn a novel skeleton consistent parametric facial generator, SCULPTOR, which can create the unique and nuanced facial features that help define a character and at the same time maintain physiological soundness. Our SCULPTOR jointly models the skull, face geometry and face appearance under a unified data-driven framework, by separating the depiction of a 3D face into shape blend shape, pose blend shape and facial expression blend shape. SCULPTOR preserves both anatomic correctness and visual realism in facial generation tasks compared with existing methods. Finally, we showcase the robustness and effectiveness of SCULPTOR in various fancy applications unseen before.Comment: 16 page, 13 fig

A virtual world of paleontology

Computer-aided visualization and analysis of fossils has revolutionized the study of extinct organisms. Novel techniques allow fossils to be characterized in three dimensions and in unprecedented detail. This has enabled paleontologists to gain important insights into their anatomy, development, and preservation. New protocols allow more objective reconstructions of fossil organisms, including soft tissues, from incomplete remains. The resulting digital reconstructions can be used in functional analyses, rigorously testing long-standing hypotheses regarding the paleobiology of extinct organisms. These approaches are transforming our understanding of long-studied fossil groups, and of the narratives of organismal and ecological evolution that have been built upon them