A Deep Learning Approach to Teeth Segmentation and Orientation from Panoramic X-rays

Accurate teeth segmentation and orientation are fundamental in modern oral healthcare, enabling precise diagnosis, treatment planning, and dental implant design. In this study, we present a comprehensive approach to teeth segmentation and orientation from panoramic X-ray images, leveraging deep learning techniques. We build our model based on FUSegNet, a popular model originally developed for wound segmentation, and introduce modifications by incorporating grid-based attention gates into the skip connections. We introduce oriented bounding box (OBB) generation through principal component analysis (PCA) for precise tooth orientation estimation. Evaluating our approach on the publicly available DNS dataset, comprising 543 panoramic X-ray images, we achieve the highest Intersection-over-Union (IoU) score of 82.43% and Dice Similarity Coefficient (DSC) score of 90.37% among compared models in teeth instance segmentation. In OBB analysis, we obtain the Rotated IoU (RIoU) score of 82.82%. We also conduct detailed analyses of individual tooth labels and categorical performance, shedding light on strengths and weaknesses. The proposed model's accuracy and versatility offer promising prospects for improving dental diagnoses, treatment planning, and personalized healthcare in the oral domain. Our generated OBB coordinates and codes are available at https://github.com/mrinal054/Instance_teeth_segmentation

3D Teeth Reconstruction from Panoramic Radiographs using Neural Implicit Functions

Panoramic radiography is a widely used imaging modality in dental practice and research. However, it only provides flattened 2D images, which limits the detailed assessment of dental structures. In this paper, we propose Occudent, a framework for 3D teeth reconstruction from panoramic radiographs using neural implicit functions, which, to the best of our knowledge, is the first work to do so. For a given point in 3D space, the implicit function estimates whether the point is occupied by a tooth, and thus implicitly determines the boundaries of 3D tooth shapes. Firstly, Occudent applies multi-label segmentation to the input panoramic radiograph. Next, tooth shape embeddings as well as tooth class embeddings are generated from the segmentation outputs, which are fed to the reconstruction network. A novel module called Conditional eXcitation (CX) is proposed in order to effectively incorporate the combined shape and class embeddings into the implicit function. The performance of Occudent is evaluated using both quantitative and qualitative measures. Importantly, Occudent is trained and validated with actual panoramic radiographs as input, distinct from recent works which used synthesized images. Experiments demonstrate the superiority of Occudent over state-of-the-art methods.Comment: 12 pages, 2 figures, accepted to International Conference on Medical Image Computing and Computer-Assisted Intervention MICCAI 202

심층신경망을 이용한 자동화된 치과 의료영상 분석

학위논문(박사) -- 서울대학교대학원 : 치과대학 치의과학과, 2021.8. 한중석.목 적: 치과 영역에서도 심층신경망(Deep Neural Network) 모델을 이용한 방사선사진에서의 임플란트 분류, 병소 위치 탐지 등의 연구들이 진행되었으나, 최근 개발된 키포인트 탐지(keypoint detection) 모델 또는 전체적 구획화(panoptic segmentation) 모델을 의료분야에 적용한 연구는 아직 미비하다. 본 연구의 목적은 치근단 방사선사진에서 키포인트 탐지를 이용해 임플란트 골 소실 정도를 파악하는 모델과 panoptic segmentation을 파노라마영상에 적용하여 다양한 구조물들을 구획화하는 모델을 학습시켜 진료에 보조적으로 활용되도록 만들어보고, 이 모델들의 추론결과를 평가해보는 것이다. 방 법: 객체 탐지 및 구획화에 있어 널리 연구된 합성곱 신경망 모델인 Mask-RCNN을 키포인트 탐지가 가능한 형태로 준비하여 치근단 방사선사진에서 임플란트의 top, apex, 그리고 bone level 지점을 좌우로 총 6지점 탐지하게끔 학습시킨 뒤, 학습에 사용되지 않은 시험 데이터셋을 대상으로 탐지시킨다. 키포인트 탐지 평가용 지표인 object keypoint similarity (OKS) 및 이를 이용한 average precision (AP) 값을 계산하고, 평균 OKS값을 통해 모델 및 치과의사의 결과를 비교한다. 또한, 탐지된 키포인트를 바탕으로 방사선사진상에서의 골 소실 정도를 수치화한다. Panoptic segmentation을 위해서는 기존의 벤치마크에서 우수한 성적을 거둔 신경망 모델인 Panoptic DeepLab을 파노라마영상에서 주요 구조물(상악동, 상악골, 하악관, 하악골, 자연치, 치료된 치아, 임플란트)을 구획화하도록 학습시킨 뒤, 시험 데이터셋에서의 구획화 결과에 panoptic / semantic / instance segmentation 각각의 평가지표들을 적용하고, 픽셀들의 정답(ground truth) 클래스와 모델이 추론한 클래스에 대한 confusion matrix를 계산한다. 결 과: OKS값을 기반으로 계산한 키포인트 탐지 AP는, 모든 OKS threshold에 대한 평균의 경우, 상악 임플란트에서는 0.761, 하악 임플란트에서는 0.786이었다. 평균 OKS는 모델이 0.8885, 치과의사가 0.9012로, 통계적으로 유의미한 차이가 없었다 (p = 0.41). 모델의 평균 OKS 값은 사람의 키포인트 어노테이션 정규분포상에서 상위 66.92% 수준이었다. 파노라마영상 구조물 구획화에서는, panoptic segmentation 평가지표인 panoptic quality 값의 경우 모든 클래스의 평균은 80.47이었으며, 치료된 치아가 57.13으로 가장 낮았고 하악관이 65.97로 두번째로 낮은 값을 보였다. Semantic segmentation 평가지표인 global한 Intersection over Union (IoU) 값은 모든 클래스 평균 0.795였으며, 하악관이 0.639로 가장 낮았고 치료된 치아가 0.656으로 두번째로 낮은 값을 보였다. Confusion matrix 계산 결과, ground truth 픽셀들 중 올바르게 추론된 픽셀들의 비율은 하악관이 0.802로 가장 낮았다. 개별 객체에 대한 IoU를 기반으로 계산한 Instance segmentation 평가지표인 AP값은, 모든 IoU threshold에 대한 평균의 경우, 치료된 치아가 0.316, 임플란트가 0.414, 자연치가 0.520이었다. 결 론: 키포인트 탐지 신경망 모델을 이용하여, 치근단 방사선사진에서 임플란트의 주요 지점을 사람과 다소 유사한 수준으로 탐지할 수 있었다. 또한, 탐지된 지점들을 기반으로 방사선사진상에서의 임플란트 주위 골 소실 비율 계산을 자동화할 수 있고, 이 값은 임플란트 주위염의 심도 분류에 사용될 수 있다. 파노라마 영상에서는 panoptic segmentation이 가능한 신경망 모델을 이용하여 상악동과 하악관을 포함한 주요 구조물들을 구획화할 수 있었다. 따라서, 이와 같이 각 작업에 맞는 심층신경망을 적절한 데이터로 학습시킨다면 진료 보조 수단으로 활용될 수 있다.Purpose: In dentistry, deep neural network models have been applied in areas such as implant classification or lesion detection in radiographs. However, few studies have applied the recently developed keypoint detection model or panoptic segmentation model to medical or dental images. The purpose of this study is to train two neural network models to be used as aids in clinical practice and evaluate them: a model to determine the extent of implant bone loss using keypoint detection in periapical radiographs and a model that segments various structures on panoramic radiographs using panoptic segmentation. Methods: Mask-RCNN, a widely studied convolutional neural network for object detection and instance segmentation, was constructed in a form that is capable of keypoint detection, and trained to detect six points of an implant in a periapical radiograph: left and right of the top, apex, and bone level. Next, a test dataset was used to evaluate the inference results. Object keypoint similarity (OKS), a metric to evaluate the keypoint detection task, and average precision (AP), based on the OKS values, were calculated. Furthermore, the results of the model and those arrived at by a dentist were compared using the mean OKS. Based on the detected keypoint, the peri-implant bone loss ratio was obtained from the radiograph. For panoptic segmentation, Panoptic DeepLab, a neural network model ranked high in the previous benchmark, was trained to segment key structures in panoramic radiographs: maxillary sinus, maxilla, mandibular canal, mandible, natural tooth, treated tooth, and dental implant. Then, each evaluation metric of panoptic, semantic, and instance segmentation was applied to the inference results of the test dataset. Finally, the confusion matrix for the ground truth class of pixels and the class inferred by the model was obtained. Results: The AP of keypoint detection for the average of all OKS thresholds was 0.761 for the upper implants and 0.786 for the lower implants. The mean OKS was 0.8885 for the model and 0.9012 for the dentist; thus, the difference was not statistically significant (p = 0.41). The mean OKS of the model was in the top 66.92% of the normal distribution of human keypoint annotations. In panoramic radiograph segmentation, the average panoptic quality (PQ) of all classes was 80.47. The treated teeth showed the lowest PQ of 57.13, and the mandibular canal showed the second lowest PQ of 65.97. The Intersection over Union (IoU) was 0.795 on average for all classes, where the mandibular canal showed the lowest IoU of 0.639, and the treated tooth showed the second lowest IoU of 0.656. In the confusion matrix, the proportion of correctly inferred pixels among the ground truth pixels was the lowest in the mandibular canal at 0.802. The AP, averaged for all IoU thresholds, was 0.316 for the treated tooth, 0.414 for the dental implant, and 0.520 for the normal tooth. Conclusion: Using the keypoint detection neural network model, it was possible to detect major landmarks around dental implants in periapical radiographs to a degree similar to that of human experts. In addition, it was possible to automate the calculation of the peri-implant bone loss ratio on periapical radiographs based on the detected keypoints, and this value could be used to classify the degree of peri-implantitis. In panoramic radiographs, the major structures including the maxillary sinus and the mandibular canal could be segmented using a neural network model capable of panoptic segmentation. Thus, if deep neural networks suitable for each task are trained using suitable datasets, the proposed approach can be used to assist dental clinicians.Chapter 1. Introduction 1 Chapter 2. Materials and methods 5 Chapter 3. Results 23 Chapter 4. Discussion 32 Chapter 5. Conclusions 45 Published papers related to this study 46 References 47 Abbreviations 52 Abstract in Korean 53 Acknowledgements 56박

XAS: Automatic yet eXplainable Age and Sex determination by combining imprecise per-tooth predictions

Chronological age and biological sex estimation are two key tasks in a variety of procedures, including human identification and migration control. Issues such as these have led to the development of both semiautomatic and automatic prediction models, but the former are expensive in terms of time and human resources, while the latter lack the interpretability required to be applicable in real-life scenarios. This paper therefore proposes a new, fully automatic methodology for the estimation of age and sex. This first applies a tooth detection by means of a modified CNN with the objective of extracting the oriented bounding boxes of each tooth. Then, it feeds the image features inside the tooth boxes into a second CNN module designed to produce per-tooth age and sex probability distributions. The method then adopts an uncertainty-aware policy to aggregate these estimated distributions. Our approach yielded a lower mean absolute error than any other previously described, at 0.97 years. The accuracy of the sex classification was 91.82%, confirming the suitability of the teeth for this purpose. The proposed model also allows analyses of age and sex estimations on every tooth, enabling experts to identify the most relevant for each task or population cohort or to detect potential developmental problems. In conclusion, the performance of the method in both age and sex predictions is excellent and has a high degree of interpretability, making it suitable for use in a wide range of application scenariosS

Machine Learning in Dentistry: A Scoping Review

Machine learning (ML) is being increasingly employed in dental research and application. We aimed to systematically compile studies using ML in dentistry and assess their methodological quality, including the risk of bias and reporting standards. We evaluated studies employing ML in dentistry published from 1 January 2015 to 31 May 2021 on MEDLINE, IEEE Xplore, and arXiv. We assessed publication trends and the distribution of ML tasks (classification, object detection, semantic segmentation, instance segmentation, and generation) in different clinical fields. We appraised the risk of bias and adherence to reporting standards, using the QUADAS-2 and TRIPOD checklists, respectively. Out of 183 identified studies, 168 were included, focusing on various ML tasks and employing a broad range of ML models, input data, data sources, strategies to generate reference tests, and performance metrics. Classification tasks were most common. Forty-two different metrics were used to evaluate model performances, with accuracy, sensitivity, precision, and intersection-over-union being the most common. We observed considerable risk of bias and moderate adherence to reporting standards which hampers replication of results. A minimum (core) set of outcome and outcome metrics is necessary to facilitate comparisons across studies

Applications of artificial intelligence in dentistry: A comprehensive review

This work was funded by the Spanish Ministry of Sciences, Innovation and Universities under Projects RTI2018-101674-B-I00 and PGC2018-101904-A-100, University of Granada project A.TEP. 280.UGR18, I+D+I Junta de Andalucia 2020 project P20-00200, and Fapergs/Capes do Brasil grant 19/25510000928-3. Funding for open-access charge: Universidad de Granada/CBUAObjective: To perform a comprehensive review of the use of artificial intelligence (AI) and machine learning (ML) in dentistry, providing the community with a broad insight on the different advances that these technologies and tools have produced, paying special attention to the area of esthetic dentistry and color research. Materials and methods: The comprehensive review was conducted in MEDLINE/ PubMed, Web of Science, and Scopus databases, for papers published in English language in the last 20 years. Results: Out of 3871 eligible papers, 120 were included for final appraisal. Study methodologies included deep learning (DL; n = 76), fuzzy logic (FL; n = 12), and other ML techniques (n = 32), which were mainly applied to disease identification, image segmentation, image correction, and biomimetic color analysis and modeling. Conclusions: The insight provided by the present work has reported outstanding results in the design of high-performance decision support systems for the aforementioned areas. The future of digital dentistry goes through the design of integrated approaches providing personalized treatments to patients. In addition, esthetic dentistry can benefit from those advances by developing models allowing a complete characterization of tooth color, enhancing the accuracy of dental restorations. Clinical significance: The use of AI and ML has an increasing impact on the dental profession and is complementing the development of digital technologies and tools, with a wide application in treatment planning and esthetic dentistry procedures.Spanish Ministry of Sciences, Innovation and Universities RTI2018-101674-B-I00 PGC2018-101904-A-100University of Granada project A.TEP. 280.UGR18Junta de Andalucia P20-00200Fapergs/Capes do Brasil grant 19/25510000928-3Universidad de Granada/CBU

Künstliche Intelligenz in der Zahnheilkunde: Scoping-Review und Schließung beobachteter Wissenslücken durch eine methodische und eine klinische Studie

Objectives: The aims of this dissertation were to (1) conduct a scoping review of stud-ies on machine learning (ML) in dentistry and appraise their robustness, (2) perform a benchmarking study to systematically compare various ML algorithms for a specific dental task, and (3) evaluate the influence of a ML-based caries detection software on diagnostic accuracy and decision-making in a randomized controlled trial. Methods: The scoping review included studies using ML in dentistry published between 1st January 2015 and 31st May 2021 on MEDLINE, IEEE Xplore, and arXiv. The risk of bias and reporting quality were assessed with the QUADAS‐2 and TRIPOD checklists, respectively. In the benchmarking study, 216 ML models were built using permutations of six ML model architectures (U-Net, U-Net++, Feature Pyramid Networks, LinkNet, Pyramid Scene Parsing Network, and Mask Attention Network), 12 model backbones of varying complexities (ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, VGG13, VGG16, VGG19, DenseNet121, DenseNet161, DenseNet169, and Dense-Net201), and three initialization strategies (random, ImageNet, and CheXpert weights). 1,625 dental bitewing radiographs were used for training and testing. Five-fold cross-validation was carried out and model performance assessed using F1-score. In the clin-ical trial, each one of 22 dentists examined 20 randomly selected bitewing images for proximal caries; 10 images were evaluated with ML and 10 images without ML. Accura-cy in lesion detection and the suggested treatment were evaluated. Results: The scoping review included 168 studies, describing different ML tasks, mod-els, input data, methods to generate reference tests, and performance metrics, imped-ing comparison across studies. The studies showed considerable risk of bias and mod-erate adherence to reporting standards. In the benchmarking study, more complex models only minimally outperformed their simpler counterparts, if at all. Models initial-ized by ImageNet or CheXpert weights outperformed those using random weights (p<0.05). The clinical trial demonstrated that dentists using ML showed increased accu-racy (area under the receiver operating characteristic [mean (95% confidence interval): 0.89 (0.87–0.90)]) compared with those not using ML [0.85 (0.83–0.86); p<0.05], pri-marily due to their higher sensitivity [0.81 (0.74–0.87) compared to 0.72 (0.64–0.79); p<0.05]. Notably, dentists using ML also showed a higher frequency of invasive treat-ment decisions than those not using it (p<0.05). Conclusion: To facilitate comparisons across ML studies in dentistry, a minimum (core) set of outcomes and metrics should be developed, and researchers should strive to improve robustness and reporting quality of their studies. ML model choice should be performed on an informed basis, and simpler models may often be similarly capable as more complex ones. ML can increase dentists’ diagnostic accuracy but also lead to more invasive treatment.Ziele: Die Ziele dieser Dissertation waren, (1) ein Scoping-Review von Studien über maschinelles Lernen (ML) in der Zahnmedizin, (2) eine Benchmarking-Studie zum systematischen Vergleich verschiedener ML-Algorithmen für eine bestimmte zahnmedizinische Aufgabe, und (3) eine randomisierte kontrollierte Studie zur Bewertung einer ML-basierten Karies-Erkennungssoftware bezüglich diagnostischer Genauigkeit und Einfluss auf den Entscheidungsprozess durchzuführen. Methoden: Das Scoping-Review umfasste Studien über ML in der Zahnmedizin, veröffentlicht vom 1. Januar 2015 bis 31. Mai 2021 auf MEDLINE, IEEE Xplore und arXiv. Bias-Risiko und Berichtsqualität wurden mit den Checklisten QUADAS-2 beziehungsweise TRIPOD bewertet. In der Benchmarking-Studie wurden 216 ML-Modelle durch Permutationen von sechs Architekturen (U-Net, U-Net++, Feature Pyramid Networks, LinkNet, Pyramid Scene Parsing Network und Mask Attention Network), 12 Backbones (Res-Net18, ResNet34, ResNet50, ResNet101, ResNet152, VGG13, VGG16, VGG19, DenseNet121, DenseNet161, DenseNet169 und DenseNet201) und drei Initialisierungsstrategien (zufällige-, ImageNet- und CheXpert-Gewichtungen) erstellt. Zum Training und Testen wurden 1.625 Bissflügel-Röntgenaufnahmen genutzt. Es wurde eine fünffache Kreuzvalidierung durchgeführt und die Modellleistung anhand des F1-Scores bewertet. In der klinischen Studie untersuchten 22 Zahnärzte jeweils 20 zufällig ausgewählte Bissflügelbilder auf Approximalkaries; 10 Bilder wurden mit und 10 Bilder ohne ML ausgewertet. Die Genauigkeit in der Erkennung von Läsionen sowie die abgeleitete Therapieempfehlung wurden bewertet. Ergebnisse: Das Scoping-Review schloss 168 Studien ein, in denen verschiedene ML-Aufgaben, Modelle, Eingabedaten, Methoden zur Generierung von Referenztests und Leistungsmetriken beschrieben wurden. Die Studien zeigten ein erhebliches Bias-Risiko und eine mäßige Einhaltung der Berichtsstandards. In der Benchmarking-Studie hatten komplexere Modelle gegenüber einfachen Modellen allenfalls geringe Vorteile. Mit ImageNet- oder CheXpert-Gewichtungen initialisierte Modelle übertrafen solche mit Zufallsgewichtungen (p<0,05). In der klinischen Studie erreichten Zahnärzte mit ML eine höhere Genauigkeit bei der Kariesdetektion (Receiver-Operating-Charakteristik [Mittelwert (95 % Konfidenzintervall) 0,89 (0,87–0,90)]) als ohne ML [0,85 (0,83–0,86); p<0,05], hauptsächlich aufgrund höherer Sensitivität [0,81 (0,74–0,87) verglichen mit 0,72 (0,64–0,79); p<0,05]. Zahnärzte mit ML wählten auffallend häufiger invasive Behandlungen als ohne ML (p<0,05). Schlussfolgerung: Zur besseren Vergleichbarkeit von ML-Studien in der Zahnmedizin, sollten Core Outcomes und Metriken definiert sowie Robustheit und Berichtsqualität verbessert werden. Die Entwicklung von ML-Modellen sollte auf informierter Basis erfolgen, bei oft ähnlicher Leistung von einfacheren und komplexeren Modellen. ML kann die diagnostische Genauigkeit erhöhen, aber auch zu mehr invasiven Behandlungen führen