Boissonnat, G

Clancy, NT

Elson, DS

Lin, J

English

Spiral - Imperial College Digital Repository

Detection and Identification of Multispectral Structured Light Patterns for Minimally Invasive Surgery J. Lin1, N. T. Clancy1,2, G. Boissonnat3, D. S. Elson1,2  1Hamlyn Centre for Robotic Surgery, Imperial College London, UK 2Department of Surgery and Cancer, Imperial College London, UK 3Department of Bioengineering, Imperial College London, UK,  j.lin12@imperial.ac.uk  INTRODUCTION The detection of tissue surfaces during endoscopic surgery has the potential to allow patient-specific pre-operative images to be used for surgical guidance. The most commonly attempted methods include stereoscopy, SLAM, shape from shading/motion, time-of-flight, and structured lighting (SL) [1, 2, 3, 4, 5]. Of these, SL and time-of-flight have the advantage of not being dependent on tissue texture. Endoscopic implementations of SL are difficult due to requirements for high light levels and unambiguous feature detection, although there have been some recent reports in the literature [6, 7, 8].  In previous work, we have demonstrated a SL system that uses a supercontinuum fibre laser together with an optical fibre probe (diameter 1.9 mm) to create an array of spots that are each at a unique wavelength (~5 nm bandwidth). This approach has many advantages, including the high brightness of the pattern, detection by a standard endoscope camera, low sensitivity to tissue pigmentation, small diameter and flexibility. A beam chopper switches rapidly between SL and white light modes, providing video rate display for the surgeons. A central part of the process to reconstruct the tissue surface using this system is the identification of the coloured spots. Previously, extended H-maxima and RGB-wavelength conversion were used to estimate the spot centres and identify them. In this paper we present a faster, more accurate and robust method to detect centroids as well as boundaries of different projected multispectral laser spots and to achieve the unique identification of each. The other stages in the reconstruction procedure (calibration, triangulation and reconstruction) as well as a description of the hardware used are described in previous publications [6, 7]. MATERIALS AND METHODS A schematic of the algorithm is displayed in Fig. 1. The RGB images collected by the camera are firstly converted into images in HSV space to separate intensity and hue information for the following steps. Thresholding is used to suppress background noise and generate a convex hull that encloses the spot pattern, and then a binary image created by a lower threshold is combined with the convex hull, generating a mask representing the region of interest (ROI). The intensity and hue maps are filtered by Wiener and Gaussian filters, to suppress noise and enhance the intensity peaks of the spot centres respectively, followed by histogram equalization. Regional maxima of the intensity map are then calculated, using the extended H-maxima transform, which correspond to the bright central areas of each spot. These are treated as initial seeds for the region growing procedure. The criteria for this are established using ranges defined by the mean and variance of all pixel intensity and hue values in each seed. The grown result is evaluated by average spot size and similarity of hue value in order to merge small spots that are in contact. Spots identified as too small or large are improved by further region growing using ‘tighter’ or ‘looser’ growth criteria. Finally, morphological transforms such as image opening and closing are used to refine the shapes and boundaries of grown spots. 	Fig.	1: (a) General 3D reconstruction procedure. (b) Example of spot detection using region growing.	 After detection each spot is identified using an algorithm that incorporates both colour and spatial information in the pattern. A Delaunay triangulation is performed on a reference image where all spots have been detected. The normalised RGB values of the vertices of each triangle are then recorded. For a particular test image a spot of interest will be a node for several surrounding triangles. To identify each spot in other images the spot with the highest number of matching neighbours is chosen. This procedure is done by matching triangles based on the similarity of their interior angles (within a defined threshold) and the Euclidian distance between their nodes’ RGB values. RESULTS Seven SL images from a calibration object and ex vivo porcine tissue (three from a white plane, two from liver, one from kidney and one from heart) were tested using the spot detection method. The result is compared with manually-identified spots, evaluated by visual inspection and quantified in terms of sensitivity and precision. The results are shown in Fig. 2 and Table 1.    Fig. 2: Images of detected centroids and spot boundaries overlaid on RGB images of calibration and porcine tissue surfaces. (a) White plane. (b) Liver. (c) Heart. (d) Kidney. Table 1: Spot detection results for seven images. Sensitivity indicates the fraction of spots correctly detected; Precision is the fraction of correct detections in all detected spots.	Sensitivity Precision Surface 0.9417  0.9912  White Plane image 1 0.9667  1.0000  White Plane image 2 0.9508  0.9831  White Plane image 3 0.7672 0.9674 Liver image 1 0.7500  0.9355  Liver image 2 0.7297  0.9529  Heart 0.6102  0.8675  Kidney  The results shown in the above table indicate high sensitivity and precision for images of the white plane. However, sensitivity is lower for soft tissue. This is mainly caused by strong absorption at the blue end of the spectrum due to haemoglobin. Given that global parameters are adopted in the local maxima searching procedure, some blue spots are missing. For darker spots whose intensities are similar to those of the background, boundaries detected by region growing are also inaccurate. Furthermore, large distortions on some parts of tissue surface also result in severe spot shape deformation, introducing more detection error. Figure 3 shows the results of the spot identification algorithm between test and reference images in a calibration dataset. The performance of the algorithm was evaluated by analysing the number of correctly identified spots, or true positives (TP), and incorrectly detected spots, or false positives (FP). The probability that a spot was labelled was calculated using (PL=[TP+FP]/TP) along with the probability that each labelled spot was labelled correctly (PC/L=TP/[TP+FP]). High PL (89.8%) and PC/L (98.6%) values were obtained. Robustness was tested by removing spots at random and repeating the matching. As spots are removed PL decreases but PC/L remains stable close to 100%.   Fig. 3: Spot identification. (a) Reference showing numbered spots and triangulation. (b) Test image with matched triangles.	DISCUSSION Initial results from a multispectral spot detection algorithm have been presented, indicating ability to detect spot boundaries. Spot identification using colour and spatial information has also been demonstrated that is robust to occlusions or spots with low visibility. The results also show that our system can be potentially used in lesion detection and aid rigid and flexible endoscopy if adapted to ‘front-viewing’ systems. Future work will involve evaluation of the robustness of these algorithms in the presence of varying tissue type and morphology. Besides, this boundary detection method introduces a possibility for denser 3D reconstruction and adaption with near infrared cameras. Hardware modifications are also planned to alter the distribution of wavelengths of the spots and record high dynamic range images using different camera exposure times. Real-time computation would also allow a rapid reconstruction of the surface. ACKNOWLEDGEMENTS Funding for this research is provided by ERC grant 242991. We gratefully acknowledge the help of Dr Lena Maier-Hein and the German Cancer Research Centre (DKFZ) in the acquisition of these images. REFERENCES [1] Stoyanov D, Visentini-Scarzanella M, Pratt P, Yang GZ. Real-time stereo reconstruction in robotically assisted minimally invasive surgery. Med Image Comput Comput Assist Interv. 2010;13(Pt. 1):275-282. [2] Mountney P, Yang GZ. Motion compensated SLAM for image guided surgery. Med Image Comput Comput Assist Interv. 2010;13(Pt. 2):496-504. [3] Malti A, Bartoli A, Collins T. Template-based conformal shape-from-motion-and-shading for laparoscopy. International Conference on Information Processing in Computer-Assisted Interventions (IPCAI). 2012;7330:1-10. [4] Penne J, Holler K, Sturmer M, Schrauder T, Schneider A, Engelbrecht R, Feussner H, Schmauss B, Hornegger J. Time-of-flight 3-d endoscopy. Med Image Comput Comput Assist Interv. 2009;12(Pt. 1):467-474. [5] Wu TT, Qu JY. Optical imaging for medical diagnosis based on active stereo vision and motion tracking. Opt Express. 2007;15(16):10421-10426. [6] Clancy NT, Stoyanov D, Maier-Hein L, Groch A, Yang GZ, Elson DS. Spectrally encoded fibre-based structured lighting probe for intraoperative 3D imaging. Biomed Opt Express. 2011;2(11):3119-3128. [7] Clancy NT, Stoyanov D, Yang GZ, Elson DS. Stroboscopic illumination scheme for seamless 3D endoscopy. Proc SPIE. 2012;8214:82140M. (a) (b) (b) (c) (d) (a) [8] Schmalz C, Forster F, Schick A, Angelopoulou E. An endoscopic 3D scanner based on structured light. Med Image Anal. 2012;16(5):1063-1072. 

Detection and Identification of Multispectral Structured Light Patterns for Minimally Invasive Surgery

http://spiral.imperial.ac.uk/bitstream/10044/1/23512/2/Structured%20Lighting%20accepted.pdf

Detection and Identification of Multispectral Structured Light Patterns for Minimally Invasive Surgery

Abstract

Similar works

Full text

Available Versions

Spiral - Imperial College Digital Repository