Learning Deep Similarity Metric for 3D MR-TRUS Registration

Purpose: The fusion of transrectal ultrasound (TRUS) and magnetic resonance (MR) images for guiding targeted prostate biopsy has significantly improved the biopsy yield of aggressive cancers. A key component of MR-TRUS fusion is image registration. However, it is very challenging to obtain a robust automatic MR-TRUS registration due to the large appearance difference between the two imaging modalities. The work presented in this paper aims to tackle this problem by addressing two challenges: (i) the definition of a suitable similarity metric and (ii) the determination of a suitable optimization strategy. Methods: This work proposes the use of a deep convolutional neural network to learn a similarity metric for MR-TRUS registration. We also use a composite optimization strategy that explores the solution space in order to search for a suitable initialization for the second-order optimization of the learned metric. Further, a multi-pass approach is used in order to smooth the metric for optimization. Results: The learned similarity metric outperforms the classical mutual information and also the state-of-the-art MIND feature based methods. The results indicate that the overall registration framework has a large capture range. The proposed deep similarity metric based approach obtained a mean TRE of 3.86mm (with an initial TRE of 16mm) for this challenging problem. Conclusion: A similarity metric that is learned using a deep neural network can be used to assess the quality of any given image registration and can be used in conjunction with the aforementioned optimization framework to perform automatic registration that is robust to poor initialization.Comment: To appear on IJCAR

Anatomically Constrained Video-CT Registration via the V-IMLOP Algorithm

Functional endoscopic sinus surgery (FESS) is a surgical procedure used to treat acute cases of sinusitis and other sinus diseases. FESS is fast becoming the preferred choice of treatment due to its minimally invasive nature. However, due to the limited field of view of the endoscope, surgeons rely on navigation systems to guide them within the nasal cavity. State of the art navigation systems report registration accuracy of over 1mm, which is large compared to the size of the nasal airways. We present an anatomically constrained video-CT registration algorithm that incorporates multiple video features. Our algorithm is robust in the presence of outliers. We also test our algorithm on simulated and in-vivo data, and test its accuracy against degrading initializations.Comment: 8 pages, 4 figures, MICCA

A new mini-navigation tool allows accurate component placement during anterior total hip arthroplasty.

Introduction: Computer-assisted navigation systems have been explored in total hip arthroplasty (THA) to improve component positioning. While these systems traditionally rely on anterior pelvic plane registration, variances in soft tissue thickness overlying anatomical landmarks can lead to registration error, and the supine coronal plane has instead been proposed. The purpose of this study was to evaluate the accuracy of a novel navigation tool, using registration of the anterior pelvic plane or supine coronal plane during simulated anterior THA. Methods: Measurements regarding the acetabular component position, and changes in leg length and offset were recorded. Benchtop phantoms and target measurement values commonly seen in surgery were used for analysis. Measurements for anteversion and inclination, and changes in leg length and offset were recorded by the navigation tool and compared with the known target value of the simulation. Pearson\u27s Results: The device accurately measured cup position and leg length measurements to within 1° and 1 mm of the known target values, respectively. Across all simulations, there was a strong, positive relationship between values obtained by the device and the known target values ( Conclusion: The preliminary findings of this study suggest that the novel navigation tool tested is a potentially viable tool to improve the accuracy of component placement during THA using the anterior approach

An open environment CT-US fusion for tissue segmentation during interventional guidance.

Therapeutic ultrasound (US) can be noninvasively focused to activate drugs, ablate tumors and deliver drugs beyond the blood brain barrier. However, well-controlled guidance of US therapy requires fusion with a navigational modality, such as magnetic resonance imaging (MRI) or X-ray computed tomography (CT). Here, we developed and validated tissue characterization using a fusion between US and CT. The performance of the CT/US fusion was quantified by the calibration error, target registration error and fiducial registration error. Met-1 tumors in the fat pads of 12 female FVB mice provided a model of developing breast cancer with which to evaluate CT-based tissue segmentation. Hounsfield units (HU) within the tumor and surrounding fat pad were quantified, validated with histology and segmented for parametric analysis (fat: -300 to 0 HU, protein-rich: 1 to 300 HU, and bone: HU>300). Our open source CT/US fusion system differentiated soft tissue, bone and fat with a spatial accuracy of ∼1 mm. Region of interest (ROI) analysis of the tumor and surrounding fat pad using a 1 mm(2) ROI resulted in mean HU of 68±44 within the tumor and -97±52 within the fat pad adjacent to the tumor (p<0.005). The tumor area measured by CT and histology was correlated (r(2) = 0.92), while the area designated as fat decreased with increasing tumor size (r(2) = 0.51). Analysis of CT and histology images of the tumor and surrounding fat pad revealed an average percentage of fat of 65.3% vs. 75.2%, 36.5% vs. 48.4%, and 31.6% vs. 38.5% for tumors <75 mm(3), 75-150 mm(3) and >150 mm(3), respectively. Further, CT mapped bone-soft tissue interfaces near the acoustic beam during real-time imaging. Combined CT/US is a feasible method for guiding interventions by tracking the acoustic focus within a pre-acquired CT image volume and characterizing tissues proximal to and surrounding the acoustic focus

Relational Reasoning Network (RRN) for Anatomical Landmarking

Accurately identifying anatomical landmarks is a crucial step in deformation analysis and surgical planning for craniomaxillofacial (CMF) bones. Available methods require segmentation of the object of interest for precise landmarking. Unlike those, our purpose in this study is to perform anatomical landmarking using the inherent relation of CMF bones without explicitly segmenting them. We propose a new deep network architecture, called relational reasoning network (RRN), to accurately learn the local and the global relations of the landmarks. Specifically, we are interested in learning landmarks in CMF region: mandible, maxilla, and nasal bones. The proposed RRN works in an end-to-end manner, utilizing learned relations of the landmarks based on dense-block units and without the need for segmentation. For a given a few landmarks as input, the proposed system accurately and efficiently localizes the remaining landmarks on the aforementioned bones. For a comprehensive evaluation of RRN, we used cone-beam computed tomography (CBCT) scans of 250 patients. The proposed system identifies the landmark locations very accurately even when there are severe pathologies or deformations in the bones. The proposed RRN has also revealed unique relationships among the landmarks that help us infer several reasoning about informativeness of the landmark points. RRN is invariant to order of landmarks and it allowed us to discover the optimal configurations (number and location) for landmarks to be localized within the object of interest (mandible) or nearby objects (maxilla and nasal). To the best of our knowledge, this is the first of its kind algorithm finding anatomical relations of the objects using deep learning.Comment: 10 pages, 6 Figures, 3 Table

Image-guided port placement for minimally invasive cardiac surgery

Minimally invasive surgery is becoming popular for a number of interventions. Use of robotic surgical systems in coronary artery bypass intervention offers many benefits to patients, but is however limited by remaining challenges in port placement. Choosing the entry ports for the robotic tools has a large impact on the outcome of the surgery, and can be assisted by pre-operative planning and intra-operative guidance techniques. In this thesis, pre-operative 3D computed tomography (CT) imaging is used to plan minimally invasive robotic coronary artery bypass (MIRCAB) surgery. From a patient database, port placement optimization routines are implemented and validated. Computed port placement configurations approximated past expert chosen configurations with an error of 13.7 ±5.1 mm. Following optimization, statistical classification was used to assess patient candidacy for MIRCAB. Various pattern recognition techniques were used to predict MIRCAB success, and could be used in the future to reduce conversion rates to conventional open-chest surgery. Gaussian, Parzen window, and nearest neighbour classifiers all proved able to detect ‘candidate’ and ‘non-candidate’ MIRCAB patients. Intra-operative registration and laser projection of port placements was validated on a phantom and then evaluated in four patient cases. An image-guided laser projection system was developed to map port placement plans from pre-operative 3D images. Port placement mappings on the phantom setup were accurate with an error of 2.4 ± 0.4 mm. In the patient cases, projections remained within 1 cm of computed port positions. Misregistered port placement mappings in human trials were due mainly to the rigid-body registration assumption and can be improved by non-rigid techniques. Overall, this work presents an integrated approach for: 1) pre-operative port placement planning and classification of incoming MIRCAB patients; and 2) intra-operative guidance of port placement. Effective translation of these techniques to the clinic will enable MIRCAB as a more efficacious and accessible procedure

A Multi-Robot Cooperation Framework for Sewing Personalized Stent Grafts

This paper presents a multi-robot system for manufacturing personalized medical stent grafts. The proposed system adopts a modular design, which includes: a (personalized) mandrel module, a bimanual sewing module, and a vision module. The mandrel module incorporates the personalized geometry of patients, while the bimanual sewing module adopts a learning-by-demonstration approach to transfer human hand-sewing skills to the robots. The human demonstrations were firstly observed by the vision module and then encoded using a statistical model to generate the reference motion trajectories. During autonomous robot sewing, the vision module plays the role of coordinating multi-robot collaboration. Experiment results show that the robots can adapt to generalized stent designs. The proposed system can also be used for other manipulation tasks, especially for flexible production of customized products and where bimanual or multi-robot cooperation is required.Comment: 10 pages, 12 figures, accepted by IEEE Transactions on Industrial Informatics, Key words: modularity, medical device customization, multi-robot system, robot learning, visual servoing, robot sewin

로봇을 이용한 자율적 하악골채취 골절단술의 기초방법 개발과 그 정확도 평가

학위논문 (박사)-- 서울대학교 대학원 : 치의학대학원 치의과학과, 2019. 2. 김성민.Objectives: An autonomous robot osteotomy system using direct coordinate determination was developed in our study. The registration accuracy was evaluated by measuring the fiducial localization error (FLE) and target registration error (TRE) and the accuracy of the designed osteotomy method along a preprogrammed plan was evaluated. Furthermore, the accuracy of the robotic osteotomy and a manual osteotomy was compared in regard to cut position, length, angle and depth. Methods: A light-weight-robot was used in this study, with an electric gripper. A direct coordinate determination method, using three points on the teeth, was developed for registration and determination of FLE and TRE, as measured on a mandible model. Sixteen landmarks on the mandible were prepared with holes and zirconia beads and the TRE was computed in ten repeated measurements using the robot. A direct coordinate determination via three points was used for registering and a twenty stone model (7 cm x 7 cm x 3 cm). The osteotomy line was designed similar to the ramal bone graft (2 cm x 1 cm x 0.5 cm). To evaluate accuracy, we measured a position (how accurate the robot arm is located), length (how accurate the robot arm is moving while cutting), angle (the angle at which the robot arm is located), and depth (the depth of the disc cutting) error. Sixteen mandible phantoms were used to simulate the osteotomy for the ramus bone graft. An image of the phantom was obtained by three-dimensional camera scanning and a virtual ramal bone graft was designed with computer software. To evaluate an accuracy and precision, the mandible phantoms were scanned with cone beam computer tomography (CBCT). Cut position, length, angle and depth errors were measured and the results of the robotic surgery were compared with that of manual surgery. Results: The mean value of the FLE was 0.84 ± 0.38 mm and the third reference point which detected the lingual fossa of the right second molar had a larger error than the other reference points. The mean value of the TRE was 1.69 ± 0.82 mm and there were significant differences between the anterior body, posterior body, and coronoid/condyle groups. Landmarks at the anterior body had the lowest TRE (0.96 ± 0.47 mm) and landmarks on the coronoid and condyle had the highest TRE (2.12 ± 0.99 mm). An autonomous robot osteotomy with a direct coordinate determination using three points was successfully achieved. On the model RBG osteotomy, the posterior cut had 0.77±0.32 absolute mean value, the anterior cut had 0.82±0.43, the inferior cut had 0.76 ± 0.38 and the superior cut had 1.37 ± 0.83, respectively. The absolute mean values for osteotomy errors for position, length, angle, and depth were 0.93 ± 0.45 mm, 0.81 ± 0.34 mm, 1.26 ± 1.35°, and 1.19 ± 0.73 mm, respectively. The position and length errors were significantly lower than angle and depth errors. In the comparison between robotic surgery and manual surgery, there were significant differences of absolute mean value and variance in all categories. For the robotic surgery, the cut position, length, angle and depth errors were 0.70 ± 0.34 mm, 0.35 ± 0.19 mm, 1.32 ± 0.96° and 0.59 ± 0.46 mm, respectively. For the manual surgery, the cut position, length, angle and depth errors were 1.83 ± 0.65 mm, 0.62 ± 0.37 mm, 5.96 ± 3.47° and 0.40 ± 0.31 mm, respectively. The robotic surgery had significantly higher accuracy and lower variance for cut position, length and angle errors. On the other hand, the depth error had a significantly higher absolute mean value and variance than the robotic surgery. Conclusions: An autonomous robot osteotomy scheme was developed, using the direct coordinate determination by three points on the teeth, and proved an accurate method for registration. The incisal edge or buccal pit of the teeth were more proper reference points than the fossa of the teeth. The measured RMS of the TRE increased when the target moved away from the reference points. Robotic surgery showed high accuracy and precision in positioning and reduced accuracy in controlling the depth of disc sawing. The robotic surgery showed high accuracy and precision in positioning and somewhat low accuracy in controlling the depth of the disc sawing. Comparing robotic and manual surgeries, the robotic surgery was superior in accuracy and precision in position, length and angle. However, the manual surgery had higher accuracy and precision in depth.1. 목 적 본 연구에서는 세 점 접촉을 통한 좌표 결정 방식을 통해 실제 모델의 좌표와 로봇이 가지고 있는 좌표를 정합하는 방식을 이용하여 자율 로봇을 이용한 하악골채취 골절단술의 기초방법을 개발하고자 한다. 개발된 정합 방법의 위치 추적 오류 (fiducial localization error)와 목표 정합 오류 (target registration error)를 측정하여 정합의 정확성을 평가하고자 한다. 또한 사전에 프로그래밍된 골절단을 직육면체 모델에 시행하고 위치, 길이, 각도, 깊이의 오류를 측정하여 정확성을 알아보고자 한다. 추가적으로 3차원 가상수술을 통해 하악 상행지 골이식술(ramal bone graft)을 설계하고 하악 팬텀 모형에서 이에 맞게 자율 로봇이 골절단술을 수행하여 악골에서 있어서 로봇을 이용한 골절단술의 정확성을 평가해 보고 반대측은 대조군으로 외과의가 기존의 전통적인 방식으로 골절단술을 수행함으로써 양측을 비교하고자 한다. 2. 방 법 본 연구에서는 경량 로봇의 최종 작용체(end effector)에 전자 그리퍼(gripper)를 연결하고 이 그리퍼가 수술용 절삭기구나 디스크가 연결된 치과용 핸드피스를 잡고 골절단을 수행하도록 하였다. 실제 모델의 좌표와 로봇이 가지고 있는 좌표를 중첩하기 위해 세 점을 찍어 첫번째 점을 원점으로 하고, 두번째 점의 방향을 x축으로, 그리고 세 번째 점이 결정하는 평면을 xy 평면으로 인식하도록 하였다. 첫번째 실험에서는 위치 추적 오류와 목표 정합 오류의 평가를 위해 하악골 모델에 치아의 기준 세 점과 하악골의 총 16개의 목표 위치에 1mm 구멍을 뚫고 1mm 지름의 지르코니아 구를 적용하여 CBCT 상에서 잘 보일 수 있도록 하였다. 각 목표 위치에 10번씩 반복하여 위치를 인식하여 오류를 계산하고 목표 정합 오류의 위치별 차이를 분석하였다. 두번째 실험에서는 총 20 개의 직육면체 석고 모델 (7cm x 7cm x 3cm)을 제작하였고 석고의 절단 크기는 하악 상행지 골채취을 위한 골절단 크기 (2cm x 1cm x 0.5cm)와 동일하게 설계하였다. 로봇팔을 이용하여 3점 접촉을 하면 좌표값을 계산하여 미리 프로그래밍된 위치에서 골절단을 수행하였다. 로봇에 의해 수행된 석고 절단선은 위치, 길이 각도 및 깊이로 나누어 오류를 측정하였다. 세번째 실험에서는 하악 상행지 골채취를 위한 골절단 실험을 위해 총 16개의 하악 팬텀 모형을 사용하였다. 팬텀 모형을 삼차원 스캐닝으로 삼차원 영상을 얻고 가상 수술을 시행하여 골절단 크기와 형태 그리고 그 위치에 대한 계획을 세웠다. 이 가상 수술 계획에 따라 로봇이 팬텀 모델에 골절단 수술을 하였다. 반대 측은 대조군으로 기존의 전통적인 방식으로 외과의가 수행하여 양측의 오차를 비교하였다. 절단선의 위치, 길이, 각도 및 깊이를 측정하여 각각의 정확도를 비교하였다. 위치 오류는 x축으로는 로봇이 표면 접촉을 인식하고 골절단을 시행하기에 0의 값으로 측정되었고 y 축과 z 축으로 나누어 측정되었으며 평균값과 제곱평균제곱근를 계산하였다. 3. 결 과 위치 추적 오류와 목표 정합 오류는 각각 0.49±0.22 mm 와 0.98±0.47 mm로 측정되었으며 기준접에서 멀어질수록 목표 정합 오류는 더 큰 값을 보였다. 석고 모델 실험에서 절단선의 위치, 길이, 각도 및 깊이의 평균과 표준오차는 각각 0.93 ± 0.45 mm, 0.81 ± 0.34 mm, 1.26 ± 1.35°, 1.19 ± 0.73 mm 이었다. 위치가 가장 정확한 값을 보였으며 길이 그리고 깊이 순으로 오차가 증가하였으며, 각도와 절단 깊이 제어가 가장 오차가 많은 술식이었다. 하악 팬텀 수술에서 로봇을 이용한 골절단의 위치, 길이, 각도 및 깊이 오차 값은 각각 0.70 ± 0.34 mm, 0.35 ± 0.19 mm, 1.32 ± 0.96°, 0.59 ± 0.46 mm 였으며 외과의의 골절단에서는 값이 각각 1.83 ± 0.65 mm, 0.62 ± 0.37 mm, 5.96 ± 3.47°, 0.40 ± 0.31 mm 였다. 위치, 길이, 각도 오차는 로봇이 더 작은 값을 보였고 깊이 오차는 외과의의 수술에서 더 작은 값을 보였다. 4. 결 론 본 연구에서는 하악 상행지 골채취를 위한 자율 로봇을 이용한 골절단 시스템을 개발하였고 위치추적오류와 목표정합오류 모두 우수한 값을 보였다. 석고 모형과 하악 팬텀 모향을 이용한 두가지 실험 모두에서 유용성과 향상된 정확성을 확인할 수 있었다. 세점 접촉 좌표 결정 시스템은 실제 모델의 좌표를 로봇의 좌표로 등록하는 데 유용한 시스템이었으며, 하악 상행지 골절단술에 대한 자율로봇 시스템의 정확도는 기존의 외과의가 직접 수행하는 방식보다 우수하였다.Abstract (in English) 1. Introduction 1 2. Materials and Methods. 12 3. Results 26 4. Discussion 32 5. Conclusions 40 6. References 41 Tables and Figures 48 Abstract (in Korean) 74Docto