Image operator learning coupled with CNN classification and its application to staff line removal

Many image transformations can be modeled by image operators that are characterized by pixel-wise local functions defined on a finite support window. In image operator learning, these functions are estimated from training data using machine learning techniques. Input size is usually a critical issue when using learning algorithms, and it limits the size of practicable windows. We propose the use of convolutional neural networks (CNNs) to overcome this limitation. The problem of removing staff-lines in music score images is chosen to evaluate the effects of window and convolutional mask sizes on the learned image operator performance. Results show that the CNN based solution outperforms previous ones obtained using conventional learning algorithms or heuristic algorithms, indicating the potential of CNNs as base classifiers in image operator learning. The implementations will be made available on the TRIOSlib project site.Comment: To appear in ICDAR 201

Crack Analyser: a novel image-based NDT approach for measuring crack severity ​

openIn Europa, le infrastrutture civili e di trasporto necessitano di una manutenzione efficace e proattiva per garantire il continuo funzionamento in sicurezza durante l'intero loro ciclo di vita. I paesi europei devono ogni anno stanziare enormi risorse per mantenere il loro livello di funzionalità. Ciò fa sorgere la necessità urgente di adottare approcci di ispezione di monitoraggio più rapidi e affidabili per aiutare ad affrontare questi problemi. Il deterioramento delle strutture è più spesso anticipato dalla formazione di fessure sulla superficie del calcestruzzo. La presenza di fessurazioni può essere sintomo di diverse problematiche quali dilatazioni e ritiri dovuti a sbalzi di temperatura, assestamenti della struttura, copertura impropria fornita in fase di getto, corrosione delle armature in acciaio, carichi pesanti applicati, vibrazioni insufficienti al momento della posa del calcestruzzo o perdite d'acqua per ritiro superficiale del calcestruzzo. Diventa quindi di primaria importanza l'identificazione, la misurazione e il monitoraggio delle fessurazioni sulla superficie del calcestruzzo. I principali metodi di ispezione attualmente adottati si basano su strumenti manuali e righelli: un’attività lunga e ingombrante, soggetta a errori e scarsamente oggettiva sull'analisi quantitativa perché fortemente dipendente dall'esperienza dell'operatore. Secondo la norma UNI EN 1992-1-1:2005, la larghezza massima delle fessure del calcestruzzo ammessa per una generica classe di rischio è di 0,3 mm. Per questo motivo, per misurare in modo accurato e affidabile la dimensione della fessura, è necessario l’impiego di strumenti di misura con caratteristiche metrologiche adeguate (es. precisione e accuratezza almeno un ordine inferiore al valore da misurare). In caso contrario, la severità della fessura potrebbe essere classificata erroneamente. Questo lavoro di tesi propone un nuovo approccio automatico, basato su immagini, in grado di localizzare e misurare fessure su superfici in calcestruzzo rispettando il vincolo metrologico imposto dalla norma UNI EN 1992-1-1:2005. Utilizzando una sola immagine, il metodo sviluppato è in grado di localizzare e misurare automaticamente e rapidamente la larghezza e la lunghezza di una fessura su una superficie. Il sistema di misura sviluppato sfrutta una singola telecamera operante nel campo del visibile per acquisire un'immagine digitalizzata della superficie da ispezionare. Il componente software del sistema riceve in input la singola immagine che inquadra la crepa e fornisce in output un'immagine aumentata dove viene evidenziata la crepa e la sua larghezza e lunghezza media/max. La misura della larghezza della fessura viene eseguita perpendicolarmente alla linea centrale della fessura con una precisione sub-pixel. Il sistema di misurazione è stato implementato su uno smartphone per eseguire ispezioni manuali da parte dell'operatore e su sistemi integrati per l'ispezione remota con robot o velivoli senza pilota (UAV)). Le strategie sviluppate possono essere facilmente estese a qualsiasi altro contesto in cui sia richiesto un controllo di qualità superficiale mirato all'identificazione e misura di eventuali danni o difettosità. ​Europe’s ageing transport infrastructure needs effective and proactive maintenance in order to continue its safe operation during the entire life cycle; European countries have to allocate huge resources for maintaining their service-ability level. This give rise to the necessity of an urgent need to adopt faster and more reliable monitoring inspection approaches to help tackling these issues. The deterioration of structures is most often foreseen by the formation of cracks on concrete surface. The presence of cracks can be a symptom of various problems like expansion and shrinks due to temperature differences, settlement of the structure, improper cover provided during concreting, corrosion of reinforcement steel, heavy load applied, insufficient vibration at the time of laying the concrete or loss of water from concrete surface shrinkage, therefore the identification, measurement and monitoring of cracks on the concrete surface becomes of primary importance. The main currently adopted inspection methods rely on visual marking and rulers, long and cumbersome activity, prone to errors and poorly objective on quantitative analysis because it strongly depends on operator experience. According to UNI EN 1992-1-1:2005 standard , the maximum admitted concrete crack width is 0.3 mm. For this reason, to accurately and reliably measure the target dimension, it is necessary to employ measurement instruments with suitable metrological characteristics (e.g. precision and accuracy at least one order lower than the value to be measured). Otherwise, the crack severity could be misclassified. This thesis work proposes a novel automatic image-based approach able to locate and measure cracks on concrete surfaces respecting the metrological constraint imposed by UNI EN 1992-1-1:2005 standard. Using only one image, the developed method is able to automatically and rapidly locate and measure the average width and length of a crack in an existing concrete structure. The measurement system developed exploits a single camera working in the visible range to acquire a digitized image of the structure being inspected. The software component of the system receives as input the single image framing the crack and gives as output an augmented image where the crack is highlighted as well as its average/max width and length. The measure of the crack width is performed perpendicularly to the crack central line with sub-pixel accuracy. The measurement system has been deployed on a smartphone for operator-based manual inspections as well on embedded systems for remote inspection with robots or Unmanned Aerial Vehicles (UAVs). The strategies developed can be easily extended from concrete inspection applications to any other context where a surface quality control targeted to the identification of eventual damages/defects is required. The activity was triggered by an explicit need within the EnDurCrete project. ​INGEGNERIA INDUSTRIALEembargoed_20220321Giulietti, Nicol

Advances in Image Processing, Analysis and Recognition Technology

For many decades, researchers have been trying to make computers’ analysis of images as effective as the system of human vision is. For this purpose, many algorithms and systems have previously been created. The whole process covers various stages, including image processing, representation and recognition. The results of this work can be applied to many computer-assisted areas of everyday life. They improve particular activities and provide handy tools, which are sometimes only for entertainment, but quite often, they significantly increase our safety. In fact, the practical implementation of image processing algorithms is particularly wide. Moreover, the rapid growth of computational complexity and computer efficiency has allowed for the development of more sophisticated and effective algorithms and tools. Although significant progress has been made so far, many issues still remain, resulting in the need for the development of novel approaches

Integrating Technology Into Wildlife Surveys

Technology is rapidly improving and being incorporated into field biology, with survey methods such as machine learning and uncrewed aircraft systems (UAS) headlining efforts. UAS paired with machine learning algorithms have been used to detect caribou, nesting waterfowl and seabirds, marine mammals, white-tailed deer, and more in over 19 studies within the last decade alone. Simultaneously, UAS and machine learning have also been implemented for infrastructure monitoring at wind energy facilities as wind energy construction and use has skyrocketed globally. As part of both pre-construction and regulatory compliance of newly constructed wind energy facilities, monitoring of impacts to wildlife is assessed through ground surveys following the USFWS Land-based Wind Energy Guidelines. To streamline efforts at wind energy facilities and improve efficiency, safety, and accuracy in data collection, UAS platforms may be leveraged to not only monitor infrastructure, but also impacts to wildlife in the form of both pre- and post-construction surveys. In this study, we train, validate, and test a machine learning approach, a convolutional neural network (CNN), in the detection and classification of bird and bat carcasses. Further, we compare the trained CNN to the currently accepted and widely used method of human ground surveyors in a simulated post-construction monitoring scenario. Last, we establish a baseline comparison of manual image review of waterfowl pair surveys with currently used ground surveyors that could inform both pre-construction efforts at energy facilities, along with long-standing federal and state breeding waterfowl surveys. For the initial training of the CNN, we collected 1,807 images of bird and bat carcasses that were split into 80.0% training and 20.0% validation image sets. Overall detection was extremely high at 98.7%. We further explored the dataset by evaluating the trained CNN’s ability to identify species and the variables that impacted identification. Classification of species was successful in 90.5% of images and was associated with sun angle and wind speed. Next, we performed a proof of concept to determine the utility of the trained CNN against ground surveyors in ground covers and with species that were both used in the initial training of the model and novel. Ground surveyors performed similar to those surveying at wind energy facilities with 63.2% detection, while the trained CNN fell short at 28.9%. Ground surveyor detection was weakly associated with carcass density within a plot and strongly with carcass size. Similarly, detection by the CNN was associated with carcass size, ground cover type, visual obstruction of vegetation, and weakly with carcass density within a plot. Finally, we examined differences in breeding waterfowl counts between ground surveyors and UAS image reviewers and found that manual review of UAS imagery yielded similar to slightly higher counts of waterfowl. Significant training, testing, and repeated validation of novel image data sets should be performed prior to implementing survey methods reliant upon machine learning algorithms. Additionally, further research is needed to determine potential biases of counting live waterfowl in aerial imagery, such as bird movement and double counting. While our initial results show that UAS imagery and machine learning can improve upon current techniques, extensive follow-up is strongly recommended in the form of proof-of-concept studies and additional validation to confirm the utility of the application in new environments with new species that allow models to be generalized. Remotely sensed imagery paired with machine learning algorithms have the potential to expedite and standardize monitoring of wildlife at wind energy facilities and beyond, improving data streams and potentially reducing costs for the benefit of both conservation agencies and the energy industry

Deep Learning-Based Robotic Perception for Adaptive Facility Disinfection

Hospitals, schools, airports, and other environments built for mass gatherings can become hot spots for microbial pathogen colonization, transmission, and exposure, greatly accelerating the spread of infectious diseases across communities, cities, nations, and the world. Outbreaks of infectious diseases impose huge burdens on our society. Mitigating the spread of infectious pathogens within mass-gathering facilities requires routine cleaning and disinfection, which are primarily performed by cleaning staff under current practice. However, manual disinfection is limited in terms of both effectiveness and efficiency, as it is labor-intensive, time-consuming, and health-undermining. While existing studies have developed a variety of robotic systems for disinfecting contaminated surfaces, those systems are not adequate for intelligent, precise, and environmentally adaptive disinfection. They are also difficult to deploy in mass-gathering infrastructure facilities, given the high volume of occupants. Therefore, there is a critical need to develop an adaptive robot system capable of complete and efficient indoor disinfection. The overarching goal of this research is to develop an artificial intelligence (AI)-enabled robotic system that adapts to ambient environments and social contexts for precise and efficient disinfection. This would maintain environmental hygiene and health, reduce unnecessary labor costs for cleaning, and mitigate opportunity costs incurred from infections. To these ends, this dissertation first develops a multi-classifier decision fusion method, which integrates scene graph and visual information, in order to recognize patterns in human activity in infrastructure facilities. Next, a deep-learning-based method is proposed for detecting and classifying indoor objects, and a new mechanism is developed to map detected objects in 3D maps. A novel framework is then developed to detect and segment object affordance and to project them into a 3D semantic map for precise disinfection. Subsequently, a novel deep-learning network, which integrates multi-scale features and multi-level features, and an encoder network are developed to recognize the materials of surfaces requiring disinfection. Finally, a novel computational method is developed to link the recognition of object surface information to robot disinfection actions with optimal disinfection parameters

Optimizing endoscopic strategies for colorectal cancer screening : improving colonoscopy effectiveness by optical, non-optical, and computer-based models

Introduction: Le cancer colorectal demeure un grave problème de santé publique au Canada. Les programmes de dépistage pourraient réduire l'incidence du cancer colorectal et la mortalité qui lui est associée. Une coloscopie de haute qualité est considérée comme un moyen rentable de prévenir le cancer en identifiant et en éliminant les lésions précurseurs du cancer. Bien que la coloscopie puisse servir de mesure préventive contre le cancer, la procédure peut imposer un fardeau supplémentaire à la santé publique par l'enlèvement et l'évaluation histologique de polypes colorectaux diminutifs et insignifiants, qui présentent un risque minime d'histologie avancée ou de cancer. La technologie de l'amélioration de l'image permettrait aux médecins de réséquer et de rejeter les polypes diminutifs ou de diagnostiquer et de laisser les polypes rectosigmoïdiens diminutifs sans examen histopathologique. Malgré la disponibilité de systèmes informatiques de caractérisation des polypes, la pratique du diagnostic optique reste limitée en raison de la crainte d'un mauvais diagnostic de cancer, d'une mauvaise surveillance des patients et des problèmes médico-légaux correspondants. Il est donc indispensable d'élaborer des stratégies alternatives de résection et d'élimination non optiques pour améliorer la précision et la sécurité du diagnostic optique et l'adapter à la pratique clinique. Ces stratégies doivent répondre à des critères cliniques simples et ne nécessitent pas de formation supplémentaire ni de dispositifs d'amélioration de l'image. De plus, la pratique sûre du diagnostic optique, la prise de décision appropriée concernant la technique de polypectomie ou l'intervalle de surveillance dépendent de l'estimation précise de la taille des polypes. La variabilité inter-endoscopistes dans la mesure de la taille des polypes exige le développement de méthodes fiables et validées pour augmenter la précision de la mesure de la taille. Une balance virtuelle intégrée à un endoscope haute définition est actuellement disponible pour le calcul automatique de la taille des polypes, mais sa faisabilité clinique n'a pas encore été établie. En dehors des points susmentionnés, une coloscopie de haute qualité nécessite l'examen complet de la muqueuse colique, ainsi que la visualisation de la valve iléocæcale et de l'orifice appendiculaire. À ce jour, aucune solution informatique n'a été capable d'assister les endoscopistes pendant les coloscopies en temps réel en détectant et en différenciant les points de repère cæcaux de façon automatique. Objectifs: Les objectifs de cette thèse sont : 1) d'étudier l'effet de la limitation du diagnostic optique aux polypes de 1 à 3 mm sur la sécurité du diagnostic optique pour le traitement des polypes diminutifs et l'acceptation par les endoscopistes de son utilisation dans les pratiques en temps réel tout en préservant ses potentiels de temps et de rentabilité ; 2) élaborer et examiner des stratégies non optiques de résection et d'élimination qui peuvent remplacer le diagnostic optique tout en offrant les mêmes possibilités d'économie de temps et d'argent ; 3) examiner la précision relative d'un endoscope à échelle virtuelle pour mesurer la taille des polypes ; 4) former, valider et tester un modèle d'intelligence artificielle qui peut prédire la complétude d'une procédure de coloscopie en identifiant les points de repère anatomiques du cæcum (c'est-à-dire la valve iléo-cæcale et l'orifice appendiculaire) et en les différenciant les uns des autres, des polypes et de la muqueuse normale. Méthodes: Pour atteindre le premier objectif de cette thèse, une analyse post-hoc de trois études prospectives a été réalisée pour évaluer la proportion de patients chez lesquels des adénomes avancés ont été découverts et le diagnostic optique a entraîné une surveillance retardée dans trois groupes de taille de polypes : 1–3, 1–5, et 1–10 mm. Pour atteindre le second objectif de cette thèse, deux stratégies non optiques ont été développées et testées dans deux études prospectives: une stratégie de résection et d'élimination basée sur la localisation qui utilise la localisation anatomique des polypes pour classer les polypes du côlon en non-néoplasiques ou néoplasiques à faible risque et une stratégie de résection et d'élimination basée sur les polypes qui attribue des intervalles de surveillance en fonction du nombre et de la taille des polypes. Dans les trois études, la concordance de l'attribution d'intervalles de surveillance basée sur un diagnostic optique à haute confiance ou sur des stratégies non optiques avec les recommandations basées sur la pathologie, ainsi que la proportion d'examens pathologiques évités et la proportion de communications immédiates d'intervalles de surveillance, ont été évaluées. Le troisième objectif de cette thèse a été abordé par le biais d'une étude de faisabilité pilote prospective qui a utilisé la mesure de spécimens de polypes immédiatement après leur prélèvement, suite à une polypectomie par un pied à coulisse Vernier comme référence pour comparer la précision relative des mesures de la taille des polypes entre les endoscopistes et un endoscope à échelle virtuelle. Enfin, le quatrième objectif de cette thèse a été évalué par l'enregistrement et l'annotation prospective de vidéos de coloscopie. Des images non modifiées de polype, de valve iléo-caecale, d'orifice appendiculaire et de muqueuse normale ont été extraites et utilisées pour développer et tester un modèle de réseau neuronal convolutionnel profond pour classer les images pour les points de repère qu'elles contiennent. Résultats: La réduction du seuil du diagnostic optique favoriserait la sécurité du diagnostic optique en diminuant de manière significative le risque d'écarter un polype avec une histologie avancée ou la mauvaise surveillance d'un patient avec de tels polypes. En outre, les stratégies non optiques de résection et d'élimination pourraient dépasser le critère de référence d'au moins 90% de concordance dans l'attribution des intervalles de surveillance post-polypectomie par rapport aux décisions basées sur l'évaluation pathologique. De plus, il a été démontré que l'endoscope à échelle virtuelle est plus précis que l'estimation visuelle de la taille des polypes en temps réel. Enfin, un modèle d'apprentissage profond s'est révélé très efficace pour détecter les repères cæcaux, les polypes et la muqueuse normale, à la fois individuellement et en combinaison. Discussion: La prédiction histologique optique des polypes de 1 à 3 mm est une approche efficace pour améliorer la sécurité et la faisabilité de la stratégie de résection et d'écartement dans la pratique. Les approches non optiques de résection et d'élimination offrent également des alternatives viables au diagnostic optique lorsque les endoscopistes ne sont pas en mesure de répondre aux conditions de mise en œuvre systématique du diagnostic optique, ou lorsque la technologie d'amélioration de l'image n'est pas accessible. Les stratégies de résection et de rejet, qu'elles soient optiques ou non, pourraient réduire les coûts supplémentaires liés aux examens histopathologiques et faciliter la communication du prochain intervalle de surveillance le même jour que la coloscopie de référence. Un endoscope virtuel à échelle réduite faciliterait l'utilisation du diagnostic optique pour la détection des polypes diminutifs et permet une prise de décision appropriée pendant et après la coloscopie. Enfin, le modèle d'apprentissage profond peut être utile pour promouvoir et contrôler la qualité des coloscopies par la prédiction d'une coloscopie complète. Cette technologie peut être intégrée dans le cadre d'une plateforme de vérification et de génération de rapports qui élimine le besoin d'intervention humaine. Conclusion: Les résultats présentés dans cette thèse contribueront à l'état actuel des connaissances dans la pratique de la coloscopie concernant les stratégies pour améliorer l'efficacité de la coloscopie dans la prévention du cancer colorectal. Cette étude fournira des indications précieuses pour les futurs chercheurs intéressés par le développement de méthodes efficaces de traitement des polypes colorectaux diminutifs. Le diagnostic optique nécessite une formation complémentaire et une mise en œuvre à l'aide de modules de caractérisation informatisés. En outre, malgré la lenteur de l'adoption des solutions informatiques dans la pratique clinique, la coloscopie assistée par l'IA ouvrira la voie à la détection automatique, à la caractérisation et à la rédaction semi-automatique des rapports de procédure.Introduction: Colorectal cancer remains a critical public health concern in Canada. Screening programs could reduce the incidence of colorectal cancer and its associated mortality. A high-quality colonoscopy is appraised to be a cost-effective means of cancer prevention through identifying and removing cancer precursor lesions. Although colonoscopy can serve as a preventative measure against cancer, the procedure can impose an additional burden on the public health by removing and histologically evaluating insignificant diminutive colorectal polyps, which pose a minimal risk of advanced histology or cancer. The image-enhance technology would enable physicians to resect and discard diminutive polyps or diagnose and leave diminutive rectosigmoid polyps without histopathology examination. Despite the availability of computer-based polyp characterization systems, the practice of optical diagnosis remains limited due to the fear of cancer misdiagnosis, patient mismanagement, and the related medicolegal issues. Thus, alternative non-optical resection and discard strategies are imperative for improving the accuracy and safety of optical diagnosis for adaptation to clinical practice. These strategies should follow simple clinical criteria and do not require additional education or image enhanced devices. Furthermore, the safe practice of optical diagnosis, adequate decision-making regarding polypectomy technique, or surveillance interval depends on accurate polyp size estimation. The inter-endoscopist variability in polyp sizing necessitates the development of reliable and validated methods to enhance the accuracy of size measurement. A virtual scale integrated into a high-definition endoscope is currently available for automated polyp sizing, but its clinical feasibility has not yet been demonstrated. In addition to the points mentioned above, a high-quality colonoscopy requires the complete examination of the entire colonic mucosa, as well as the visualization of the ileocecal valve and appendiceal orifice. To date, no computer-based solution has been able to support endoscopists during live colonoscopies by automatically detecting and differentiating cecal landmarks. Aims: The aims of this thesis are: 1) to investigate the effect of limiting optical diagnosis to polyps 1–3mm on the safety of optical diagnosis for the management of diminutive polyps and the acceptance of endoscopists for its use in real-time practices while preserving its time- and cost-effectiveness potentials; 2) to develop and examine non-optical resect and discard strategies that can replace optical diagnosis while offering the same time- and cost-saving potentials; 3) to examine the relative accuracy of a virtual scale endoscope for measuring polyp size; 4) to train, validate, and test an artificial intelligence-empower model that can predict the completeness of a colonoscopy procedure by identifying cecal anatomical landmarks (i.e., ileocecal valve and appendiceal orifice) and differentiating them from one another, polyps, and normal mucosa. Methods: To achieve the first aim of this thesis, a post-hoc analysis of three prospective studies was performed to evaluate the proportion of patients in which advanced adenomas were found and optical diagnosis resulted in delayed surveillance in three polyp size groups: 1‒3, 1‒5, and 1‒10 mm. To achieve the second aim of this thesis, two non-optical strategies were developed and tested in two prospective studies: a location-based resect and discard strategy that uses anatomical polyp location to classify colon polyps into non-neoplastic or low-risk neoplastic and a polyp-based resect and discard strategy that assigns surveillance intervals based on polyp number and size. In all three studies, the agreement of assigning surveillance intervals based on high-confidence optical diagnosis or non-optical strategies with pathology-based recommendations, as well as the proportion of avoided pathology examinations and the proportion of immediate surveillance interval communications, was evaluated. The third aim of this thesis was addressed through a prospective pilot feasibility study that used the measurement of polyp specimens immediately after retrieving, following a polypectomy by a Vernier caliper as a reference to compare the relative accuracy of polyp size measurements between endoscopists and a virtual scale endoscope. Finally, the fourth aim of this thesis was assessed through prospective recording and annotation of colonoscopy videos. Unaltered images of polyp, ileocecal valve, appendiceal orifice and normal mucosa were extracted and used to develop and test a deep convolutional neural network model for classifying images for the containing landmarks. Results: Reducing the threshold of optical diagnosis would promote the safety of optical diagnosis by significantly decreasing the risk of discarding a polyp with advanced histology or the mismanagement of a patient with such polyps. Additionally, the non-optical resect and discard strategies could surpass the benchmark of at least 90% agreement in the assignment of post-polypectomy surveillance intervals compared with decisions based on pathologic assessment. Moreover, the virtual scale endoscope was demonstrated to be more accurate than visual estimation of polyp size in real-time. Finally, a deep learning model proved to be highly effective in detecting cecal landmarks, polyps, and normal mucosa, both individually and in combination. Discussion: Optical histology prediction of polyps 1‒3 mm in size is an effective approach to enhance the safety and feasibility of resect and discard strategy in practice. Non-optical resect and discard approaches also offer feasible alternatives to optical diagnosis when endoscopists are unable to meet the conditions for routine implementation of optical diagnosis, or when image-enhanced technology is not accessible. Both optical and non-optical resect and discard strategies could reduce additional costs related to histopathology examinations and facilitate the communication of the next surveillance interval in the same day as the index colonoscopy. A virtual scale endoscope would facilitate the use of optical diagnosis for the detection of diminutive polyps and allows for appropriate decision-making during and after colonoscopy. Additionally, the deep learning model may be useful in promoting and monitoring the quality of colonoscopies through the prediction of a complete colonoscopy. This technology may be incorporated as part of a platform for auditing and report generation that eliminates the need for human intervention. Conclusion: The results presented in this thesis will contribute to the current state of knowledge in colonoscopy practice regarding strategies for improving the efficacy of colonoscopy in the prevention of colorectal cancer. This study will provide valuable insights for future researchers interested in developing effective methods for treating diminutive colorectal polyps. Optical diagnosis requires further training and implementation using computer-based characterization modules. Furthermore, despite the slow adoption of computer-based solutions in clinical practice, AI-empowered colonoscopy will eventually pave the way for automatic detection, characterization, and semi-automated completion of procedure reports in the future