Convolutional neural network-based clinical predictors of oral dysplasia: class activation map analysis of deep learning results

Oral cancer/oral squamous cell carcinoma is among the top ten most common cancers globally, with over 500,000 new cases and 350,000 associated deaths every year worldwide. There is a critical need for objective, novel technologies that facilitate early, accurate diagnosis. For this purpose, we have developed a method to classify images as “suspicious” and “normal” by performing transfer learning on Inception-ResNet-V2 and generated automated heat maps to highlight the region of the images most likely to be involved in decision making. We have tested the developed method’s feasibility on two independent datasets of clinical photographic images of 30 and 24 patients from the UK and Brazil, respectively. Both 10-fold cross-validation and leave-one-patient-out validation methods were performed to test the system, achieving accuracies of 73.6% (±19%) and 90.9% (±12%), F1-scores of 97.9% and 87.2%, and precision values of 95.4% and 99.3% at recall values of 100.0% and 81.1% on these two respective cohorts. This study presents several novel findings and approaches, namely the development and validation of our methods on two datasets collected in different countries showing that using patches instead of the whole lesion image leads to better performance and analyzing which regions of the images are predictive of the classes using class activation map analysis

A deep-learning approach to aid in diagnosing Barrett’s oesophagus related dysplasia

Barrett's oesophagus is the only known precursor to oesophagus carcinoma. Histologically, it is defined as a condition of columnar cells replacing the standard squamous lining. Those altered cells are prone to cytological and architectural abnormalities, known as dysplasia. The dysplastic degree varies from low to high grade and can evolve into invasive carcinoma or adenocarcinoma. Thus, detecting high-grade and intramucosal carcinoma during the surveillance of Barrett's oesophagus patients is vital so they can be treated by surgical resection. Unfortunately, the achieved interobserver agreement for grading dysplasia among pathologists is only fair to moderate. Nowadays, grading Barrett's dysplasia is limited to visual examination by pathologists for glass or virtual slides. This work aims to diagnose different grades of dysplasia in Barrett’s oesophagus, particularly high-grade dysplasia, from virtual histopathological slides of oesophagus tissue. In the first approach, virtual slides were analysed at a low magnification to detect regions of interest and predict the grade of dysplasia based on the analysis of the virtual slides at 10X magnification. Transfer learning was employed to partially fine-tune two deep-learning networks using healthy and Barrett’s oesophagus tissue. Then, the two networks were connected. The proposed model achieved 0.57 sensitivity, 0.79 specificity and moderate agreement with a pathologist. On the contrary, the second approach processed the slides at a higher magnification (40X magnification). It adapted novelty detection and local outlier factor alongside transfer learning to solve the multiple instances learning problem. It increased the performance of the diagnosis to 0.84 sensitivity and 0.92 specificity, and the interobserver agreement reached a substantial level. Finally, the last approach mimics the pathologists’ procedure to diagnose dysplasia, relying on both magnifications. Thus, their behaviours during the assessment were analysed. As a result, it was found that employing a multi-scale approach to detect dysplastic tissue using a low magnification level (10X magnification) and grade dysplasia at a higher level (40X magnification). The proposed computer-aided diagnosis system was built using networks from the first two approaches. It scored 0.90 sensitivity, 0.94 specificity and a substantial agreement with the pathologist and a moderate agreement with the other expert

Confocal Microscopy and Nuclear Segmentation Algorithm for Quantitative Imaging of Epithelial Tissue

Carcinomas, cancers that originate in the epithelium, account for more than 80% of all cancers. When detected early, the 5-year survival rate is greatly increased. Biopsy and histopathology is the current gold standard for diagnosis of epithelial carcinomas which is an invasive, time-intensive, and stressful procedure. In vivo confocal microscopy has the potential to non-invasively image epithelial tissue in near-real time. This dissertation describes the development of a confocal microscope for imaging epithelial tissues and an image processing algorithm for segmentation of epithelial nuclei. A rapid beam and stage scanning combination was used to acquire fluorescence confocal images of cellular and tissue features along the length of excised mouse colon. A single 1 × 60 mm2 field of view is acquired in 10 seconds. Disruption of crypt structure such as size, shape, and distribution is visualized in images of inflamed colon tissue, while the normal mouse colon exhibited uniform crypt structure and distribution. An automated pulse coupled neural network segmentation algorithm was developed for epithelial nuclei segmentation. An increase in nuclear size and the nuclear-to-cytoplasmic ratio is a potential precursor to pre-cancer development. The spiking cortical model algorithm was evaluated using a developed confocal image model of epithelial tissues with varying contrast. It was further validated on reflectance confocal images of porcine and human oral tissue from two separate confocal imaging systems. Biopsies of human oral mucosa are used to determine the tissue and system effects on measurements of nuclear-to-cytoplasmic ratio

Artificial Intelligence-based methods in head and neck cancer diagnosis : an overview

Background This paper reviews recent literature employing Artificial Intelligence/Machine Learning (AI/ML) methods for diagnostic evaluation of head and neck cancers (HNC) using automated image analysis. Methods Electronic database searches using MEDLINE via OVID, EMBASE and Google Scholar were conducted to retrieve articles using AI/ML for diagnostic evaluation of HNC (2009–2020). No restrictions were placed on the AI/ML method or imaging modality used. Results In total, 32 articles were identified. HNC sites included oral cavity (n = 16), nasopharynx (n = 3), oropharynx (n = 3), larynx (n = 2), salivary glands (n = 2), sinonasal (n = 1) and in five studies multiple sites were studied. Imaging modalities included histological (n = 9), radiological (n = 8), hyperspectral (n = 6), endoscopic/clinical (n = 5), infrared thermal (n = 1) and optical (n = 1). Clinicopathologic/genomic data were used in two studies. Traditional ML methods were employed in 22 studies (69%), deep learning (DL) in eight studies (25%) and a combination of these methods in two studies (6%). Conclusions There is an increasing volume of studies exploring the role of AI/ML to aid HNC detection using a range of imaging modalities. These methods can achieve high degrees of accuracy that can exceed the abilities of human judgement in making data predictions. Large-scale multi-centric prospective studies are required to aid deployment into clinical practice

Computer Aided Dysplasia Grading for Barrett’s Oesophagus Virtual Slides

Dysplasia grading in Barrett’s Oesophagus has been an issue among pathologist worldwide. Despite of the increasing number of sufferers every year especially for westerners, dysplasia in Barrett’s Oesophagus can only be graded by a trained pathologist with visual examination. Therefore, we present our work on extracting textural and spatial features from the tissue regions. Our first approach is to extract only the epithelial layer of the tissue, based on the grading rules by pathologists. This is carried out by extracting sub images of a certain window size along the tissue epithelial layer. The textural features of these sub images were used to grade regions into dysplasia or not-dysplasia and we have achieved 82.5% AP with 0.82 precision and 0.86 recall value. Therefore, we have managed to overcame the ‘boundary-effect’ issues that have usually been avoided by selecting or cropping tissue image without the boundary. Secondly, the textural and spatial features of the whole tissue in the region were investigated. Experiments were carried out using Grey Level Co-occurrence Matrices at the pixel-level with a brute-force approach experiment, to cluster patches based on its texture similarities.Then, we have developed a texture-mapping technique that translates the spatial arrangement of tissue texture within a tissue region on the patch-level. As a result, three binary decision tree models were developed from the texture-mapping image, to grade each annotated regions into dysplasia Grade 1, Grade 3 and Grade 5 with 87.5%, 75.0% and 81.3% accuracy percentage with kappa score 0.75, 0.5 and 0.63 respectively. A binary decision tree was then used on the spatial arrangement of the tissue texture types with respect to the epithelial layer to help grade the regions. 75.0%, 68.8% and 68.8% accuracy percentage with kappa value of 0.5, 0.37 and 0.37 were achieved respectively for dysplasia Grade 1, Grade 3 and Grade 5. Based on the result achieved, we can conclude that the spatial information of tissue texture types with regards to the epithelial layer, is not as strong as is on the whole region. The binary decision tree grading models were applied on the broader tissue area; the whole virtual pathology slides itself. The consensus grading for each tissue is calculated with positivity table and scoring method. Finally, we present our own thresholded frequency method to grade virtual slides based on frequency of grading occurrence; and the result were compared to the pathologist’s grading. High agreement score with 0.80 KV was achieved and this is a massive improvement compared to a simple frequency scoring, which is only 0.47 KV

Unsupervised morphological segmentation of tissue compartments in histopathological images

Algorithmic segmentation of histologically relevant regions of tissues in digitized histopathological images is a critical step towards computer-assisted diagnosis and analysis. For example, automatic identification of epithelial and stromal tissues in images is important for spatial localisation and guidance in the analysis and characterisation of tumour micro-environment. Current segmentation approaches are based on supervised methods, which require extensive training data from high quality, manually annotated images. This is often difficult and costly to obtain. This paper presents an alternative data-independent framework based on unsupervised segmentation of oropharyngeal cancer tissue micro-arrays (TMAs). An automated segmentation algorithm based on mathematical morphology is first applied to light microscopy images stained with haematoxylin and eosin. This partitions the image into multiple binary ‘virtual-cells’, each enclosing a potential ‘nucleus’ (dark basins in the haematoxylin absorbance image). Colour and morphology measurements obtained from these virtual-cells as well as their enclosed nuclei are input into an advanced unsupervised learning model for the identification of epithelium and stromal tissues. Here we exploit two Consensus Clustering (CC) algorithms for the unsupervised recognition of tissue compartments, that consider the consensual opinion of a group of individual clustering algorithms. Unlike most unsupervised segmentation analyses, which depend on a single clustering method, the CC learning models allow for more robust and stable detection of tissue regions. The proposed framework performance has been evaluated on fifty-five hand-annotated tissue images of oropharyngeal tissues. Qualitative and quantitative results of the proposed segmentation algorithm compare favourably with eight popular tissue segmentation strategies. Furthermore, the unsupervised results obtained here outperform those obtained with individual clustering algorithms

Artificial Intelligence in Oral Health

This Special Issue is intended to lay the foundation of AI applications focusing on oral health, including general dentistry, periodontology, implantology, oral surgery, oral radiology, orthodontics, and prosthodontics, among others

Development of deep learning methods for head and neck cancer detection in hyperspectral imaging and digital pathology for surgical guidance

Surgeons performing routine cancer resections utilize palpation and visual inspection, along with time-consuming microscopic tissue analysis, to ensure removal of cancer. Despite this, inadequate surgical cancer margins are reported for up to 10-20% of head and neck squamous cell carcinoma (SCC) operations. There exists a need for surgical guidance with optical imaging to ensure complete cancer resection in the operating room. The objective of this dissertation is to evaluate hyperspectral imaging (HSI) as a non-contact, label-free optical imaging modality to provide intraoperative diagnostic information. For comparison of different optical methods, autofluorescence, RGB composite images synthesized from HSI, and two fluorescent dyes are also acquired and investigated for head and neck cancer detection. A novel and comprehensive dataset was obtained of 585 excised tissue specimens from 204 patients undergoing routine head and neck cancer surgeries. The first aim was to use SCC tissue specimens to determine the potential of HSI for surgical guidance in the challenging task of head and neck SCC detection. It is hypothesized that HSI could reduce time and provide quantitative cancer predictions. State-of-the-art deep learning algorithms were developed for SCC detection in 102 patients and compared to other optical methods. HSI detected SCC with a median AUC score of 85%, and several anatomical locations demonstrated good SCC detection, such as the larynx, oropharynx, hypopharynx, and nasal cavity. To understand the ability of HSI for SCC detection, the most important spectral features were calculated and correlated with known cancer physiology signals, notably oxygenated and deoxygenated hemoglobin. The second aim was to evaluate HSI for tumor detection in thyroid and salivary glands, and RGB images were synthesized using the spectral response curves of the human eye for comparison. Using deep learning, HSI detected thyroid tumors with 86% average AUC score, which outperformed fluorescent dyes and autofluorescence, but HSI-synthesized RGB imagery performed with 90% AUC score. The last aim was to develop deep learning algorithms for head and neck cancer detection in hundreds of digitized histology slides. Slides containing SCC or thyroid carcinoma can be distinguished from normal slides with 94% and 99% AUC scores, respectively, and SCC and thyroid carcinoma can be localized within whole-slide images with 92% and 95% AUC scores, respectively. In conclusion, the outcomes of this thesis work demonstrate that HSI and deep learning methods could aid surgeons and pathologists in detecting head and neck cancers.Ph.D

Quantitative chemical imaging: A top-down systems pathology approach to predict colon cancer patient survival

Colon cancer is the second deadliest cancer, affecting the quality of life in older patients. Prognosis is useful in developing an informed disease management strategy, which can improve mortality as well as patient comfort. Morphometric assessment provides diagnosis, grade, and stage information. However, it is subjective, requires multi-step sample processing, and annotations by pathologists. In addition, morphometric techniques offer minimal molecular information that can be crucial in determining prognosis. The interaction of the tumor with its surrounding stroma, comprised of several biomolecular factors and cells is a critical determinant of the behavior of the disease. To evaluate this interaction objectively, we need biomolecular profiling in spatially specific context. In this work, we achieved this by analyzing tissue microarrays using infrared spectroscopic imaging. We developed supervised classification algorithms that were used to reliably segment colon tissue into histological components, including differentiation of normal and desmoplastic stroma. Thus, infrared spectroscopic imaging enabled us to map the stromal changes around the tumor. This supervised classification achieved >0.90 area under the curve of the receiver operating characteristic curve for pixel level classification. Using these maps, we sought to define evaluation criteria to assess the segmented colon images to determine prognosis. We measured the interaction of tumor with the surrounding stroma containing activated fibroblast in the form of mathematical functions that took into account the structure of tumor and the prevalence of reactive stroma. Using these functions, we found that the interaction effect of large tumor size in the presence of a high density of activated fibroblasts provided patients with worse outcome. The overall 6-year probability of survival in patient groups that were classified as “low-risk” was 0.73 whereas in patients that were “high-risk” was 0.54 at p-value <0.0003. Remarkably, the risk score defined in this work was independent of patient risk assessed by stage and grade of the tumor. Thus, objective evaluation of prognosis, which adds to the current clinical regimen, was achieved by a completely automated analysis of unstained patient tissue to determine the risk of 6-year death. In this work, we demonstrate that quantitative chemical imaging using infrared spectroscopic imaging is an effective method to measure tumor-tumor microenvironment interactions. As a top-down systems pathology approach, our work integrated morphometry based spatial constraints and biochemistry based stromal changes to identify markers that gave us mechanistic insights into the tumor behavior. Our work shows that while the tumor microenvironment changes are prognostic, an interaction model that takes into account both the extent of microenvironment modifications, as well as the tumor morphology, is a better predictor of prognosis. Finally, we also developed automated tumor grade determination using deep learning based infrared image analysis. Thus, the computational models developed in this work provide an objective, processing-free and automated way to predict tumor behavior

Evaluation of the role of EGFR, SOX2 and PAX9 in oral carcinogenesis

PhD ThesisOral squamous cell carcinoma (OSCC) is a major global healthcare problem. OSCC has devastating consequences for many patients diagnosed with the disease. Outcomes may be improved if the disease is identified in its precursor stages, termed oral potentially malignant disorders (OPMD). Unfortunately, histological assessment of OPMD does not reliably predict which cases will progress to OSCC. Several candidate biomarkers have emerged in recent decades. To date, however, none have been validated for use in clinical practice. This study sought to address the continuing need for biomarkers that stratify OPMD according to their risk of malignant transformation. Our data show that EGFR gene copy number abnormalities correlate with malignant transformation in OPMD. EGFR genomic gain was also present in a quarter of early-stage OSCC. SOX2 had a heterogeneous expression profile in both OPMD and OSCC, limiting its clinical utility. Nevertheless, the pattern of SOX2 expression suggests it may be a marker of OSCC stem cells and consequently represent a potential chemotherapeutic target. PAX9 is down-regulated in OPMD and early-stage OSCC. Following a course of chemical induction, Pax9-deficient mice were more likely to develop OPMD and OSCC than controls. These findings support the hypothesis that PAX9 has a tumour-suppressor function. In addition to enhanced local sensitivity to chemical induction, Pax9-deficient mice were more susceptible to the toxic systemic effects of treatment. A modified protocol for chemical induction in Pax9-deficient mice is recommended. Paradoxically, our analysis of human tissues showed increased PAX9 expression in OPMD that underwent malignant transformation, suggesting that, in some circumstances, PAX9 may have a tumour-promoting effect. Finally, we summarise the generation of stably transfected cell lines in which PAX9 and SOX2 expression may be manipulated by tetracycline administration. These cell lines will facilitate future studies of the functional role of PAX9 and SOX2 in oral carcinogenesis