Airway Label Prediction in Video Bronchoscopy: Capturing Temporal Dependencies Utilizing Anatomical Knowledge

Purpose: Navigation guidance is a key requirement for a multitude of lung interventions using video bronchoscopy. State-of-the-art solutions focus on lung biopsies using electromagnetic tracking and intraoperative image registration w.r.t. preoperative CT scans for guidance. The requirement of patient-specific CT scans hampers the utilisation of navigation guidance for other applications such as intensive care units. Methods: This paper addresses navigation guidance solely incorporating bronchosopy video data. In contrast to state-of-the-art approaches we entirely omit the use of electromagnetic tracking and patient-specific CT scans. Guidance is enabled by means of topological bronchoscope localization w.r.t. an interpatient airway model. Particularly, we take maximally advantage of anatomical constraints of airway trees being sequentially traversed. This is realized by incorporating sequences of CNN-based airway likelihoods into a Hidden Markov Model. Results: Our approach is evaluated based on multiple experiments inside a lung phantom model. With the consideration of temporal context and use of anatomical knowledge for regularization, we are able to improve the accuracy up to to 0.98 compared to 0.81 (weighted F1: 0.98 compared to 0.81) for a classification based on individual frames. Conclusion: We combine CNN-based single image classification of airway segments with anatomical constraints and temporal HMM-based inference for the first time. Our approach renders vision-only guidance for bronchoscopy interventions in the absence of electromagnetic tracking and patient-specific CT scans possible.Comment: Submitted to International Journal of Computer Assisted Radiology and Surger

Robust electromagnetically guided endoscopic procedure using enhanced particle swarm optimization for multimodal information fusion

© 2015 American Association of Physicists in Medicine. Purpose: Electromagnetically guided endoscopic procedure, which aims at accurately and robustly localizing the endoscope, involves multimodal sensory information during interventions. However, it still remains challenging in how to integrate these information for precise and stable endoscopic guidance. To tackle such a challenge, this paper proposes a new framework on the basis of an enhanced particle swarm optimization method to effectively fuse these information for accurate and continuous endoscope localization. Methods: The authors use the particle swarm optimization method, which is one of stochastic evolutionary computation algorithms, to effectively fuse the multimodal information including preoperative information (i.e., computed tomography images) as a frame of reference, endoscopic camera videos, and positional sensor measurements (i.e., electromagnetic sensor outputs). Since the evolutionary computation method usually limits its possible premature convergence and evolutionary factors, the authors introduce the current (endoscopic camera and electromagnetic sensors) observation to boost the particle swarm optimization and also adaptively update evolutionary parameters in accordance with spatial constraints and the current observation, resulting in advantageous performance in the enhanced algorithm. Results: The experimental results demonstrate that the authors proposed method provides a more accurate and robust endoscopic guidance framework than state-of-the-art methods. The average guidance accuracy of the authors framework was about 3.0 mm and 5.6° while the previous methods show at least 3.9 mm and 7.0°. The average position and orientation smoothness of their method was 1.0 mm and 1.6°, which is significantly better than the other methods at least with (2.0 mm and 2.6°). Additionally, the average visual quality of the endoscopic guidance was improved to 0.29. Conclusions: A robust electromagnetically guided endoscopy framework was proposed on the basis of an enhanced particle swarm optimization method with using the current observation information and adaptive evolutionary factors. The authors proposed framework greatly reduced the guidance errors from (4.3, 7.8) to (3.0 mm, 5.6°), compared to state-of-the-art methods

BronchoTrack: Airway Lumen Tracking for Branch-Level Bronchoscopic Localization

Localizing the bronchoscope in real time is essential for ensuring intervention quality. However, most existing methods struggle to balance between speed and generalization. To address these challenges, we present BronchoTrack, an innovative real-time framework for accurate branch-level localization, encompassing lumen detection, tracking, and airway association.To achieve real-time performance, we employ a benchmark lightweight detector for efficient lumen detection. We are the first to introduce multi-object tracking to bronchoscopic localization, mitigating temporal confusion in lumen identification caused by rapid bronchoscope movement and complex airway structures. To ensure generalization across patient cases, we propose a training-free detection-airway association method based on a semantic airway graph that encodes the hierarchy of bronchial tree structures.Experiments on nine patient datasets demonstrate BronchoTrack's localization accuracy of 85.64 \%, while accessing up to the 4th generation of airways.Furthermore, we tested BronchoTrack in an in-vivo animal study using a porcine model, where it successfully localized the bronchoscope into the 8th generation airway.Experimental evaluation underscores BronchoTrack's real-time performance in both satisfying accuracy and generalization, demonstrating its potential for clinical applications

In vivo imaging of the airway wall in asthma: fibered confocal fluorescence microscopy in relation to histology and lung function

<p>Abstract</p> <p>Background</p> <p>Airway remodelling is a feature of asthma including fragmentation of elastic fibres observed in the superficial elastin network of the airway wall. Fibered confocal fluorescence microscopy (FCFM) is a new and non-invasive imaging technique performed during bronchoscopy that may visualize elastic fibres, as shown by <it>in vitro </it>spectral analysis of elastin powder. We hypothesized that FCFM images capture <it>in vivo </it>elastic fibre patterns within the airway wall and that such patterns correspond with airway histology. We aimed to establish the concordance between the bronchial elastic fibre pattern in histology and FCFM. Second, we examined whether elastic fibre patterns in histology and FCFM were different between asthmatic subjects and healthy controls. Finally, the association between these patterns and lung function parameters was investigated.</p> <p>Methods</p> <p>In a cross-sectional study comprising 16 subjects (8 atopic asthmatic patients with controlled disease and 8 healthy controls) spirometry and bronchoscopy were performed, with recording of FCFM images followed by endobronchial biopsy at the airway main carina. Elastic fibre patterns in histological sections and FCFM images were scored semi-quantitatively. Agreement between histology and FCFM was analysed using linearly weighted kappa κ_w.</p> <p>Results</p> <p>The patterns observed in histological sections and FCFM images could be divided into 3 distinct groups. There was good agreement between elastic fibre patterns in histology and FCFM patterns (κ_w0.744). The semi-quantitative pattern scores were not different between asthmatic patients and controls. Notably, there was a significant difference in post-bronchodilator FEV₁%predicted between the different patterns by histology (p = 0.001) and FCFM (p = 0.048), regardless of asthma or atopy.</p> <p>Conclusion</p> <p>FCFM captures the elastic fibre pattern within the airway wall in humans <it>in vivo</it>. The association between post-bronchodilator FEV₁%predicted and both histological and FCFM elastic fibre patterns points towards a structure-function relationship between extracellular matrix in the airway wall and lung function.</p> <p>Trial registration</p> <p>Netherlands Trial Register <a href="http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=NTR1306">NTR1306</a></p

Evolutionarily Optimized Electromagnetic Sensor Measurements for Robust Surgical Navigation

© 2001-2012 IEEE. Miniaturized electromagnetic sensors are increasingly introduced to navigate surgical instruments to anatomical targets during minimally invasive procedures, such as endoscopic surgery. These sensors are usually attached at the distal tips of surgical instruments to track their three-dimensional motion represented by the position and orientation in six degrees of freedom. Unfortunately, these sensors suffer from inaccurate measurements and jitter errors due to the patient movement (e.g., respiratory motion) and magnetic field distortion. This paper proposes an evolutionary computing strategy to optimize the sensor measurements and improve the tracking accuracy of surgical navigation. We modified two evolutionary computation algorithms and proposed adaptive particle swarm optimization (APSO) and observation-boosted differential evolution (OBDE) to enhance the navigation accuracy. The experimental results demonstrate that our modified algorithms to evolutionarily optimize electromagnetic sensor measurements can critically reduce the tracking error from 4.8 to 2.9 mm. In particular, OBDE outperforms APSO for electromagnetic endoscopic navigation

Intraoperative Extraction of Airways Anatomy in VideoBronchoscopy

A main bottleneck in bronchoscopic biopsy sampling is to efficiently reach the lesion navigating across bronchial levels. Any guidance system should be able to localize the scope position during the intervention with minimal costs and alteration of clinical protocols. With the final goal of an affordable image-based guidance, this work presents a novel strategy to extract and codify the anatomical structure of bronchi, as well as, the scope navigation path from videobronchoscopy. Experiments using interventional data show that our method accurately identifies the bronchial structure. Meanwhile, experiments using simulated data verify that the extracted navigation path matches the 3D route

Advanced Endoscopic Navigation:Surgical Big Data,Methodology,and Applications

随着科学技术的飞速发展，健康与环境问题日益成为人类面临的最重大问题之一。信息科学、计算机技术、电子工程与生物医学工程等学科的综合应用交叉前沿课题，研究现代工程技术方法，探索肿瘤癌症等疾病早期诊断、治疗和康复手段。本论文综述了计算机辅助微创外科手术导航、多模态医疗大数据、方法论及其临床应用：从引入微创外科手术导航概念出发，介绍了医疗大数据的术前与术中多模态医学成像方法、阐述了先进微创外科手术导航的核心流程包括计算解剖模型、术中实时导航方案、三维可视化方法及交互式软件技术，归纳了各类微创外科手术方法的临床应用。同时，重点讨论了全球各种手术导航技术在临床应用中的优缺点，分析了目前手术导航领域内的最新技术方法。在此基础上，提出了微创外科手术方法正向数字化、个性化、精准化、诊疗一体化、机器人化以及高度智能化的发展趋势。【Abstract】Interventional endoscopy (e.g., bronchoscopy, colonoscopy, laparoscopy, cystoscopy) is a widely performed procedure that involves either diagnosis of suspicious lesions or guidance for minimally invasive surgery in a variety of organs within the body cavity. Endoscopy may also be used to guide the introduction of certain items (e.g., stents) into the body. Endoscopic navigation systems seek to integrate big data with multimodal information (e.g., computed tomography, magnetic resonance images, endoscopic video sequences, ultrasound images, external trackers) relative to the patient's anatomy, control the movement of medical endoscopes and surgical tools, and guide the surgeon's actions during endoscopic interventions. Nevertheless, it remains challenging to realize the next generation of context-aware navigated endoscopy. This review presents a broad survey of various aspects of endoscopic navigation, particularly with respect to the development of endoscopic navigation techniques. First, we investigate big data with multimodal information involved in endoscopic navigation. Next, we focus on numerous methodologies used for endoscopic navigation. We then review different endoscopic procedures in clinical applications. Finally, we discuss novel techniques and promising directions for the development of endoscopic navigation.X.L. acknowledges funding from the Fundamental Research Funds for the Central Universities. T.M.P. acknowledges funding from the Canadian Foundation for Innovation, the Canadian Institutes for Health Research, the National Sciences and Engineering Research Council of Canada, and a grant from Intuitive Surgical Inc

Identification and quantification of the alveolar compartment by confocal laser endomicroscopy in patients with interstitial lung diseases

Tese de mestrado integrado, Engenharia Biomédica e Biofísica (Biofísica Médica e Fisiologia de Sistemas), Universidade de Lisboa, Faculdade de Ciências, 2018Doenças Intersticiais Pulmonares (DIP) é um termo que inclui mais de 200 doenças que afectam o parênquima pulmonar, partilhando manifestações clínicas, radiográficas e patológicas semelhantes. Este conjunto de doenças é bastante heterogéneo, apresentando cada tipo de DIP em diferente grau os elementos de inflamação e fibrose: enquanto a inflamação é reflectida pelo aumento de células inflamatórias e presença de nódulos ou edema, a fibrose reflecte-se pelas fibras adicionais de colagénio e elastina. Identificar o tipo de DIP de um doente é um processo difícil, sendo a Discussão Multidisciplinar o actual método de diagnóstico "gold standard": vários médicos especialistas compõem uma equipa multidisciplinar que vai ter em conta os dados clínicos, radiológicos e patológicos disponíveis para chegar a uma conclusão. Estes dados incluem imagens de tomografia computorizada de alta resolução (TCAR), a descrição da lavagem broncoalveolar e, quando possível, dados de biópsias. Apesar do esforço e competência da equipa multidisciplinar, 10% dos pacientes são categorizados como inclassificáveis devido a dados inadequados ou discrepância entre os dados existentes. A maior causa para DIP inclassificáveis é a ausência de dados histopatológicos associada aos riscos das biópsias cirúrgicas. É muito importante determinar a DIP específica de um doente, dadas as suas implicações no tratamento e gestão do mesmo. É particularmente crítica a distinção entre doentes com Fibrose Pulmonar Idiopática (FPI) e doentes sem FPI, dado que há terapias anti-fibróticas – como o Pirfenidone – indicadas para FPI que são extremamente dispendiosas, exigindo certeza no diagnóstico antes de serem prescritas. Além disso, o tratamento com agentes imunossupressores pode funcionar com o grupo dos não-FPI mas aumenta a morte e hospitalizações nos doentes com FPI. A discussão multidisciplinar pode beneficiar da informação adicional oferecida pelo Confocal Laser Endomicroscopy (CLE), uma técnica de imagiologia que torna possível visualizar os alvéolos pulmonares com resolução microscópica de forma minimamente invasiva, através de uma broncoscopia. O laser do CLE tem um comprimento de onda de 488 nm que permite observar a autofluorescência das fibras de elastina. Há evidências de que a quantidade de fibras de elastina é aumentada e a arquitectura destas fibras é alterada na presença de fibrose pulmonar, a qual está associada a algumas doenças intersticiais pulmonares incluindo a fibrose pulmonar idiopática. Até à data, os vídeos de Confocal Laser Endomicroscopy são, na maioria dos casos, analisados apenas visualmente, e pouca informação objectiva e consistente foi conseguida destes vídeos em doentes de DIP. No entanto, é possível obter informação mais relevante dos mesmos, convertendo-os em frames, pré-processando as imagens e extraindo atributos numéricos. Neste projecto, foram obtidas imagens dos alvéolos pulmonares de doentes de DIP através de CLE. O principal objectivo do projecto é melhorar a técnica de CLE e aumentar a sua usabilidade para que no futuro possa contribuir para facilitar a estratificação de doentes com DIP e eventualmente reduzir o número de biópsias pulmonares nestes doentes. Como mencionado, o instrumento de Confocal Laser Endomicroscopy emite uma luz laser azul de 488nm, a qual é reflectida no tecido e reorientada para o sistema de detecção pela mesma lente, passando por um pequeno orifício (pinhole). Isto permite que a luz focada seja recolhida e que feixes provenientes de planos fora de foco sejam excluídos, originando uma resolução microscópica que permite imagens ao nível celular. Quando o CLE é aplicado a imagem pulmonar, é possível observar as paredes alveolares pela autofluorescência natural presente nas fibras de elastina. No estudo clínico subjacente a este estudo, o protocolo de CLE foi aplicado a 20 pacientes, embora 8 tenham sido posteriormente excluídos da análise. Os vídeos de CLE obtidos sofreram duas selecções: uma com base na região onde uma biópsia (usada como referência) foi tirada e outra com base na qualidade técnica das imagens. Depois, os dados foram pré-processados: geraram-se imagens mosaico com um campo de visão alargado e, paralelamente converteram-se as sequências de vídeo em frames. A qualidade da imagem foi melhorada, filtrando o ruído electrónico para que posteriormente pudesse ser aplicada a análise de imagem. Esta análise extraiu valores numéricos que reflectem o estado do espaço alveolar, nomeadamente, variáveis de textura e medições relacionadas com as fibras de elastina. As imagens de CLE obtidas mostraram-se muito interessantes. A resolução é superior à tomografia computorizada de alta resolução e a tridimensionalidade acrescenta informação às biópsias. O facto de permitir feedback em tempo real e observar ao vivo os movimentos naturais da respiração contribui para a análise do estado do doente. A análise de textura feita às imagens serviu-se de um algoritmo de extracção de variáveis de Haralick a partir de uma Gray-Level Co-occurence Matrix (GLCM). Foram extraídas as variáveis de textura Momento Angular Secundário (Energia), Entropia, Momento de Diferença Inversa, Contraste, Variação e Correlação. O algoritmo de Ridge Detection (detecção de linhas) identificou a maior parte das fibras de elastina detectáveis por um observador humano e mediu o Número de Fibras, o seu Comprimento e Largura e o Número de Junções entre fibras, permitindo também calcular a Soma dos Comprimentos de todas as fibras. Estes algoritmos devolveram valores consistentes num processo mais eficiente comparado com um observador humano, conseguindo avaliar em poucos segundos múltiplas variáveis para todo o conjunto de dados. As medições relacionadas com as fibras de elastina pretendiam ajudar a identificar os doentes fibróticos. Era esperado que as fibras dos doentes fibróticos fossem mais largas, mas isso não se observou. Também se previa que este grupo de doentes apresentasse maior número de fibras e junções, mas não houve uma diferença significativa entre grupos. No entanto, quando o grupo fibrótico foi segregado, o número de fibras e junções parece separar a fibrose moderada da fibrose severa. Este resultado é interessante na medida em que sugere que a monitorização do número de fibras/junções com CLE pode potencialmente ser usado como medida de eficácia de medicação anti-fibrótica. Em relação às variáveis de textura, esperava-se que os doentes fibróticos apresentassem valores mais elevados de Entropia, Contraste e Variância e valores inferiores de Momento de Diferença Inversa, dado que o seu tecido pulmonar deveria corresponder a imagens mais complexas e heterogéneas com mais arestas presentes. No entanto, ainda não foi possível estabelecer diferenças significativas entre grupos. Apesar dos resultados com o conjunto de dados usado não ter demonstrado correlações fortes entre as conclusões do CLE e da TCAR/histopatologia, os valores das variáveis em si já contribuem para o estudo das DIP, nomeadamente da sua fisiologia. De facto, a amostra de doentes deste estudo era reduzida, mas com uma amostra maior, espera-se que algumas das varáveis se correlacionem com outras técnicas usadas no diagnóstico e permitam segregar os pacientes em grupos e eventualmente aplicar classificação de dados. Neste momento, é possível especular que algumas variáveis seriam melhores candidatas para um classificador, nomeadamente os Números de Fibras e Junções, a Soma dos Comprimentos das fibras e as variáveis de Haralick Entropia e Energia. O projecto apresentado nesta dissertação foi desenvolvido através de um estágio de 6 meses no departamento de Pneumologia no Academic Medical Center em Amsterdão, Países Baixos. No Academic Medical Center (AMC), fui acompanhada pelos estudantes de doutoramento Lizzy Wijmans - médica - e Paul Brinkman - engenheiro biomédico - e supervisionada pelo Dr. Jouke Annema, MD, PhD, Professor de endoscopia pulmonar. Este grupo de investigação do AMC está focado em técnicas inovadoras de imagiologia do sistema pulmonar e teve a oportunidade de reunir com a empresa MKT –que produz a tecnologia de Confocal Laser Endomicroscopy –, o que enriqueceu a discussão aqui apresentada. Do Departamento de Física da Faculdade de Ciências da Universidade de Lisboa, fui orientada pelo Prof. Nuno Matela.Interstitial Lung Diseases (ILD) is a heterogeneous group of more than 200 diseases which affect the lung parenchyma. To identify the type of ILD a patient suffers from is a difficult process, and 10% of the patients are categorized as unclassifiable, mostly due to the absence of histopathological data associated with the risks of lung biopsies. The patient specific diagnosis is important because of its implications to the patient treatment and management, being particularly relevant to identify lung fibrosis. The Confocal Laser Endomicroscopy (CLE) can add information to this process. CLE allows to image the lung tissue with a micrometer resolution in a minimally invasive way, through a bronchoscopy. The elastin fibers from the lung alveoli are visible with this technique due to their autofluorescence. Since there is evidence that the amount of elastin fibers increases, and their architecture is altered in lung fibrosis, CLE should be used to extract values reflecting this condition. Thus, the main goal of this project was to improve the CLE technique and increase its usability, by extracting numerical values from the images which would reflect the state of the alveolar space, particularly the elastin fibers. The ILD patients recruited for the study had their lung alveoli imaged with CLE. The CLE movies were selected, pre-processed – were converted into frames, had their image quality enhanced and some mosaics were obtained – and then analyzed. The ridge detection algorithm detected most fibers recognized by a human observer. It allowed the measurement of the Number of Detected Fibers, their Length and Width, the Number of Junctions between fibers and to calculate the Sum from all Fibers’ Lengths. The Gray-Level Co-occurrence Matrix allowed the extraction of the Haralick texture features: Angular Second Moment (Energy), Entropy, Inverse Difference Moment, Contrast, Variance and Correlation. These algorithms produced consistent and unbiased numerical features, in an efficient process which can analyze the entire data set in a few seconds. Regarding the fiber related measurements, it was expected for the fibrotic patients to have wider fibers and a higher number of fibers and junctions. In terms of texture variables, it was expected from the fibrotic patients to present higher values of Entropy, Contrast and Variance, and lower values of Inverse Difference Moment, given their lung tissue should correspond to more complex and heterogeneous images with more ridges present. Due to the small sample size, it was still not possible to stratify patients with this data set. Nevertheless, the measurements presented here already contribute to the study of ILD, helping to understand the disease physiology. It is hoped that in the future, these measurements will aid the diagnosis process specially in those cases when patients cannot undergo a surgical biopsy. Additionally, CLE could potentially be used as an anti-fibrotic medication efficiency measurement tool