BACKGROUND: Fluorescence angiography in colorectal surgery is a technique that may lead to lower anastomotic leak rates. However, the interpretation of the fluorescent signal is not standardised and there is a paucity of data regarding interobserver agreement. The aim of this study is to assess interobserver variability in selection of the transection point during fluorescence angiography before anastomosis. METHODS: An online survey with still images of fluorescence angiography was distributed through colorectal surgery channels containing images from 13 patients where several areas for transection were displayed to be chosen by raters. Agreement was assessed overall and between pre-planned rater cohorts (experts vs non-experts; trainees vs consultants; colorectal specialists vs non colorectal specialists), using Fleiss' kappa statistic. RESULTS: 101 raters had complete image ratings. No significant difference was found between raters when choosing a point of optimal bowel transection based on fluorescence angiography still images. There was no difference between pre-planned cohorts analysed (experts vs non-experts; trainees vs consultants; colorectal specialists vs non colorectal specialists). Agreement between these cohorts was poor (<.26). CONCLUSION: Whilst there is no learning curve for the technical adoption of FA, understanding the fluorescent signal characteristics is key to successful use. We found significant variation exists in interpretation of static fluorescence angiography data. Further efforts should be employed to standardise fluorescence angiography assessment

Bano, Sophia

Chand, Manish

Clancy, Neil T

Diana, Michelle

Lovat, Laurence B

Raza, Imran

Soares, Antonio S

Stoyanov, Danail

English

UCL Discovery

Interobserver variability in the assessment of fluorescence angiography in the colon  Antonio S Soares1,2 MD (ORCID ID 0000-0001-7773-2427), antonio.soares.17@ucl.ac.uk Neil T Clancy1,3, PhD (ORCID ID 0000-0002-7240-5790), n.clancy@ucl.ac.uk Sophia Bano1,4, PhD (ORCID ID 0000-0003-1329-4565), sophia.bano@ucl.ac.uk Imran Raza5, MD, PhD, I.raza@nhs.net Michelle Diana6,7 MD, PhD (ORCID ID 0000-0002-1390-8486), michele.diana@ircad.fr Laurence B Lovat1,2 BSc MBBS PhD FRCP (ORCID ID 0000-0003-4542-3915), l.lovat@ucl.ac.uk Danail Stoyanov1,4, PhD (ORCID ID 0000-0002-0980-3227), danail.stoyanov@ucl.ac.uk Manish Chand1,2 FRCS MBA FASCRS PhD (ORCID ID 0000-0001-9086-8724), m.chand@ucl.ac.uk  1 - Wellcome EPSRC Centre for Interventional and Surgical Sciences (WEISS) 2 – Division of Surgery and Interventional Sciences, University College London 3 - Department of Medical Physics and Biomedical Engineering, University College London 4 - Department of Computer Science, University College London 5 – University College London Hospital, University College London Hospitals NHS Trust, London, UK 6 – IRCAD, Research Institute against Digestive Cancer, Strasbourg, France  7 – ICube Lab, Photonics for Health, University of Strasbourg, France   Corresponding Author:  Manish Chand, Associate Professor of Colorectal Surgery  Email: m.chand@ucl.ac.uk Tel: 020.7679.9060 Charles Bell House, 43-45 Foley Street London W1W 7JN  Running head: Colorectal fluorescence assessment agreement  FUNDING This work is supported by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) (203145Z/16/Z); Engineering and Physical Sciences Research Council (EPSRC) [EP/P027938/1, EP/R004080/1, EP/P012841/1] and The Royal Academy of Engineering Chair in Emerging Technologies Scheme.  Abstract Background – Fluorescence angiography in colorectal surgery is a technique that may lead to lower anastomotic leak rates. However, the interpretation of the fluorescent signal is not standardised and there is a paucity of data regarding interobserver agreement. The aim of this study is to assess interobserver variability in selection of the transection point during fluorescence angiography before anastomosis.   Methods – An online survey with still images of fluorescence angiography was distributed through colorectal surgery channels containing images from 13 patients where several areas for transection were displayed to be chosen by raters. Agreement was assessed overall and between pre-planned rater cohorts (experts vs non-experts; trainees vs consultants; colorectal specialists vs non colorectal specialists), using Fleiss’ kappa statistic.  Results – 101 raters had complete image ratings. No significant difference was found between raters when choosing a point of optimal bowel transection based on fluorescence angiography still images. There was no difference between pre-planned cohorts analysed (experts vs non-experts; trainees vs consultants; colorectal specialists vs non colorectal specialists). Agreement between these cohorts was poor (< 0.26). Conclusion – Whilst there is no learning curve for the technical adoption of FA, understanding the fluorescent signal characteristics is key to successful use. We found significant variation exists in interpretation of static fluorescence angiography data. Further efforts should be employed to standardise fluorescence angiography assessment.    Key words  Colorectal surgery Image guided surgery Ergonomics and/or human factors study   Introduction Anastomotic leak (AL) in colorectal surgery is still common today1 and causes significant patient morbidity2 and costs to the health system3. Several strategies have been employed to avoid this complication and minimize its effects, ranging from better patient stratification4 to closer monitoring postoperatively (both biochemical5 and radiological6). Decision making with regard to the anastomotic level and thus perfusion of the bowel conduit is a main determinant in AL. Intraoperatively, the point of proximal transection is based on the extent of disease that is being resected and by assessment of the proximal colon viability. This decision is most commonly based on clinical judgement through the detection of marginal artery pulse, bleeding from the cut edges of the mesentery and tissue colour.  Recently, fluorescence angiography with indocyanine green (ICG) have been used to better assess bowel perfusion and subsequently, choice of transection point. After intravenous injection, ICG binds to albumin in the intravascular compartment and travels through the circulation. It will be present in all tissue that is perfused, and it can be visualised using a specialised camera system incorporating near infrared imaging. The use of fluorescence angiography has been shown to significantly reduce the odds ratio for anastomotic leak in observational studies7,8.  However, for FA to be useful it relies on meaningful interpretation of the displayed visual image. There is no documented learning curve for the technical aspects of FA, ie administering the injection, which is understandable but there is also no recommended learning curve for the interpretation of the images. And it is ultimately image interpretation that leads to the clinical decision making. This is often inaccurately referred to as the ‘number of cases used’. Previous work has compared expert assessment with non-experts using a dynamic video assessment tool9, however no consistent cut off was used to define expert. A better understanding of how to standardize the interpretation of ICG images and potentially, build a quantitative model to help in clinical decision making, is a current unmet need. The aim of this study was to identify user variability in the interpretation of ICG fluorescence angiography and to further determine whether level of seniority and/or experience influenced clinical decision-making when using ICG.     Methods Setting – an online survey, containing 15 pre-selected images showing the proximal colon before anastomosis, was created to assess user interpretation. Users were asked to select the optimal point of transection from a series of options.  Image selection – Intraoperative video from a random selection of patients that had undergone surgery and bowel resection using ICG to help in decision making was used. All videos were from a single surgical team at a tertiary referral centre for colorectal surgery. The technique for use of ICG has been extensively described elsewhere10. A still image was selected from the video footage that corresponded to maximal fluorescence intensity for that case as assessed by the operating surgeon. For each image, the length of colon was divided into equal segments representing the transection options as shown in figure 1 (full survey shown in supplementary figure 1). These were manually segmented along the longitudinal axis of the colon with an approximate distance of 1cm between segments. 1 cm cut-off has been shown to be consistent with the change in transection point when using fluorescence angiography11,12. Randomly ordered letters were attributed to these segments to allow unbiased selection.   In total, 15 images were included, corresponding to 13 different patients. 2 patients had the same view pre- and post-fluorescence angiography (cases #8 and #10; and cases #9 and #14 respectively, supplementary figure 1), allowing a direct comparison of choices made on the same specimen.  Participants – an online image bank was created. Participants were sent an invitation to join the study through the membership of the European Fluorescence Imaging-Guided Surgery (EURO-FIGS) registry. Each user was required to state their surgical and FA experience - previous experience with fluorescence (novice if under 5 uses of fluorescence angiography, expert otherwise); level of training (trainee or consultant); and area of specialisation (colorectal or not colorectal). The question “Would you use an automated system to standardise the decision of the transection point?” was completed by the study participants. Data analysis  Users’ responses were analysed and compared by case, experience, level of seniority, and area of surgical specialization. The areas selected for transection were analysed as categorical variables using a one-way ANOVA test. A p-value under 0.05 was considered significant. Interrater agreement was assessed between raters for the different images13. The Fleiss’ kappa statistic was calculated for every image14. For comparison, 95% confidence intervals were obtained by bootstrapping using 100 repetitions with replacement. Stata (StataCorp., Texas, USA) version 14.1 was used for statistical analysis. Raters were excluded for agreement analysis if not all selections were entered. Ethics – all patients provided written informed consent for video data collection during surgery according to institutional guidelines and no patient identifiable data was used for the present study. No identifiable data was collected about participants in the survey and therefore no ethical approval was necessary. .   Results The study included 108 participants. 7 participants were excluded from agreement analysis due to incomplete ratings on all images. The details of the participants are included in Table 1.  Overall, there was no significant difference in the point of transection between any of the groups – consultant versus trainees; novice versus expert; or colorectal versus general – Table 2.  The participant scores for each individual case and statistical analysis can be found in supplementary table 1. Within each cohort, there was a poor level of agreement in responses. This was consistent throughout the different cohort categories as can be seen in Table 3.  Of all participants, 68.5% replied yes to the question “Would you use an automated system to standardise the decision of the transection point?”.      Discussion and Conclusions The results of the present study show that there was no significant difference between the users when choosing a point of optimal bowel transection based on fluorescence angiography. There was no difference between consultants and trainees, fluorescence novices and experts, or colorectal subspecialists within colorectal surgery and generalists. Furthermore, there was a poor level of agreement within each cohort. These results reflect that the interpretation of fluorescence angiography images is not associated with expertise, seniority, or sub-specialism. There is increasing evidence to suggest fluorescence angiography has a beneficial effect in reducing the risk of anastomoses leak in colorectal surgery. There have been a number of review articles recently15,16 which have shown this to be the case despite the lack of a large scale, formal randomised controlled trial. However, as with any technology or new technique it is important to be able to safely deploy so that the benefits may be obtained without any concerns over safety or complications. In the case of fluorescence angiography, the technique may be simple however any associated complications would be associated with misinterpretation of the images and thus sub-optimal clinical decision making. This is a novel situation for surgical technologies and is perhaps more attuned with radiology training which is determined by image interpretation. An example would be examining an x-ray and being able to interpret a fracture from normal bone based on the image characteristics. Intuitively, one would have expected there to have been a difference between senior surgeons and trainees in determining the optimal transection point but that has not been the case in this study. That may be explained by the fact that optimal transection point could be argued to be a function of perfused bowel and maximal colonic length rather than just perfusion. It may be that the more junior surgeons have been more cautious whereas the more senior surgeons are attempting to maximise colonic length as well as perfusion. A similar study by Nial et al.9 has also shown the variability amongst users of fluorescence angiography. In their study, video footage was used rather than still images with the raters being asked to position the optimal transection point rather than choosing from pre-selected options. Despite the slightly different methodology the results of the study were similar. This suggests that there is a need to incorporate formal image interpretation training as part of the implementation of fluorescence angiography for new users. Developing such a program would further augment the user's ability to deploy fluorescence angiography in a more effective manner. This study has certain limitations. The first is that we used still images rather than video. This means that the rater is unable to appreciate the dynamic flow of ICG into the tissue which could perhaps influence their decision-making. Using still images is possible as we had chosen frames which showed maximal peak intensity. The second limitation is the quality of the images themselves. As is the case inflorescence angiography, the intensity of the fluorescence is not always reflective of perfusion, and this can be difficult to judge in such a survey. The rater may feel that the bowel is not adequately perfused when indeed this is the maximal peak intensity for that case. The third limitation is the rather rudimentary and arbitrary categorisation of experts, trainees, specialists, and generalists. There is currently no universal definition for this. Understanding what makes a user and expert is part of the challenge in fluorescence angiography. In conclusion, the interpretation of fluorescence and geography images remains variable. There is no validated oral consensus agreement on the number of cases required to reach competency or expertise in interpretation. We suggest that in addition to training in the actual technique of administering ICG, there should be image interpretation training in parallel.  Disclosures Michelle Diana has received travelling support from Diagnostic Green and consulting fees from Intuitive Surgical, unrelated to the present study. Manish Chand has received consulting fees from Stryker, unrelated to the present study. Antonio S Soares, Sophia Bano, Neil T Clancy, Imran Raza, Laurence B Lovat and Danail Stoyanov have no relevant conflicts of interest to be disclosed.  No industry funding was used for this study.  Author contribution Conceptualization: AS Soares, Neil T Clancy, S Bano, M Chand Methodology: AS Soares, M Chand, LB Lovat Writing - Review & Editing: AS Soares, Neil T Clancy, S Bano, M Diana, LB Lovat, D Stoyanov, M Chand  Figures Figure 1 - Case #1 image with categories to be selected Supplementary figure 1 – cases presented for assessment   Tables Table 1 - Details of study participants Table 2 - Differences in mean normalised score between different rater cohorts Table 3 - Agreement analysis Supplementary table 1 - Differences in areas selected for transection      References 1. Bhangu, A. et al. An international multicentre prospective audit of elective rectal cancer surgery; operative approach versus outcome, including transanal total mesorectal excision (TaTME). Color. Dis. 20, 33–46 (2018). 2. The National Emergency Laparotomy Project Team. Fourth Patient Report of the National Emergency Laparotomy Audit ( NELA ). 1–137 (2018). 3. Ashraf, S. Q. et al. The economic impact of anastomotic leakage after anterior resections in English NHS hospitals: Are we adequately remunerating them? Color. Dis. 15, 190–199 (2013). 4. Frasson, M. et al. Risk factors for anastomotic leak after colon resection for cancer: Multivariate analysis and nomogram from a multicentric, prospective, national study with 3193 patients. Ann. Surg. 262, 321–330 (2015). 5. Reynolds, I. S. et al. C-reactive protein as a predictor of anastomotic leak in the first week after anterior resection for rectal cancer. Color. Dis. 19, 812–818 (2017). 6. Kauv, P. et al. Anastomotic leakage after colorectal surgery: diagnostic accuracy of CT. Eur. Radiol. 25, 3543–3551 (2015). 7. Blanco-Colino, R. & Espin-Basany, E. Intraoperative use of ICG fluorescence imaging to reduce the risk of anastomotic leakage in colorectal surgery: a systematic review and meta-analysis. Tech. Coloproctol. 8–10 (2017). doi:10.1007/s10151-017-1731-8 8. Degett, T. H., Andersen, H. S. & Gögenur, I. Indocyanine green fluorescence angiography for intraoperative assessment of gastrointestinal anastomotic perfusion: a systematic review of clinical trials. Langenbeck’s Arch. Surg. 401, 767–775 (2016). 9. Hardy, N. P. et al. Inter-user variation in the interpretation of near infrared perfusion imaging using indocyanine green in colorectal surgery. Surg. Endosc. 1, 3 (2021). 10. Keller, D. S., Ishizawa, T., Cohen, R. & Chand, M. Indocyanine green fluorescence imaging in colorectal surgery: overview, applications, and future directions. lancet. Gastroenterol. Hepatol. 2, 757–766 (2017). 11. Ris, F. et al. Multicentre phase II trial of near-infrared imaging in elective colorectal surgery. British Journal of Surgery 105, 1359–1367 (2018). 12. Jafari, M. D. et al. Perfusion assessment in laparoscopic left-sided/anterior resection (PILLAR II): A multi-institutional study. J. Am. Coll. Surg. 220, 82-92.e1 (2015). 13. Mitani, A. A., Freer, P. E. & Nelson, K. P. Summary measures of agreement and association between many raters’ ordinal classifications. Ann. Epidemiol. 27, 677-685.e4 (2017). 14. Hernaez, R. Reliability and agreement studies: A guide for clinical investigators. Gut 64, 1018–1027 (2015). 15. Lin, J., Zheng, B., Lin, S., Chen, Z. & Chen, S. The efficacy of intraoperative ICG fluorescence angiography on anastomotic leak after resection for colorectal cancer: a meta-analysis. Int. J. Color. Dis. 2020 361 36, 27–39 (2020). 16. Liu, D., Liang, L., Liu, L. & Zhu, Z. Does intraoperative indocyanine green fluorescence angiography decrease the incidence of anastomotic leakage in colorectal surgery? A systematic review and meta-analysis. Int. J. Color. Dis. 2020 361 36, 57–66 (2020).   

Interobserver Variability in the Assessment of Fluorescence Angiography in the Colon

https://discovery.ucl.ac.uk/10162749/2/Lovat_Interobserver%20assessment%20of%20colonic%20fluorescence%20angiography_manuscript.pdf

Interobserver Variability in the Assessment of Fluorescence Angiography in the Colon

Abstract

Similar works

Full text

Available Versions

UCL Discovery