Surgical Navigation System for Transsphenoidal Pituitary Surgery Applying U-Net-Based Automatic Segmentation and Bendable Devices

Conventional navigation systems used in transsphenoidal pituitary surgery have limitations that may lead to organ damage, including long image registration time, absence of alarms when approaching vital organs and lack of 3-D model information. To resolve the problems of conventional navigation systems, this study proposes a U-Net-based, automatic segmentation algorithm for optical nerves and internal carotid arteries, by training patient computed tomography angiography images. The authors have also developed a bendable endoscope and surgical tool to eliminate blind regions that occur when using straight, rigid, conventional endoscopes and surgical tools during transsphenoidal pituitary surgery. In this study, the effectiveness of a U-Net-based navigation system integrated with bendable surgical tools and a bendable endoscope has been demonstrated through phantom-based experiments. In order to measure the U-net performance, the Jaccard similarity, recall and precision were calculated. In addition, the fiducial and target registration errors of the navigation system and the accuracy of the alarm warning functions were measured in the phantom-based environment. © 2019 by the authors.1

Navigation-assisted anchor insertion in shoulder arthroscopy: a validity study

Background: This study aimed to compare conventional and navigation-assisted arthroscopic rotator cuff repair in terms of anchor screw insertion. Methods: The surgical performance of five operators while using the conventional and proposed navigation-assisted systems in a phantom surgical model and cadaveric shoulders were compared. The participating operators were divided into two groups, the expert group (n = 3) and the novice group (n = 2). In the phantom model, the experimental tasks included anchor insertion in the rotator cuff footprint and sutures retrieval. A motion analysis camera system was used to track the surgeons’ hand movements. The surgical performance metric included the total path length, number of movements, and surgical duration. In cadaveric experiments, the repeatability and reproducibility of the anchor insertion angle were compared among the three experts, and the feasibility of the navigation-assisted anchor insertion was validated. Results: No significant differences in the total path length, number of movements, and time taken were found between the conventional and proposed systems in the phantom model. In cadaveric experiments, however, the clustering of the anchor insertion angle indicated that the proposed system enabled both novice and expert operators to reproducibly insert the anchor with an angle close to the predetermined target angle, resulting in an angle error of < 2° (P = 0.0002). Conclusion: The proposed navigation-assisted system improved the surgical performance from a novice level to an expert level. All the experts achieved high repeatability and reproducibility for anchor insertion. The navigation-assisted system may help surgeons, including those who are inexperienced, easily familiarize themselves to of suture anchors insertion in the right direction by providing better guidance for anchor orientation. Level of evidence: A retrospective study (level 2). © 2020, The Author(s).1

의료용 증강현실을 위한 좌표 정렬 방법에 관한 연구

Augmented reality, hand–eye calibration, patient–image registration, endoscope, optical tracking systemI proposed a one-step calibration method to simultaneously perform patient-to-image registration and hand-eye calibration that are necessary processes for implementing an augmented reality (AR) navigation using an optical tracking system in endoscopic surgeries. Until now, the registration and calibration processes have been separately performed, called the two-step calibration method here. The two-step method requires the different condition from the AR one for performing it. Although the optimal solutions from the two-step method are calculated under their conditions, they are not optimal for the AR condition due to multiplication errors as well as tracking errors. To alleviate the errors, a transformation from the coordinate system of the endoscope to that of the computed tomography images is accurately calculated and used in the one-step method. Because this transformation is free from the multiplication and tracking errors, it becomes an important factor to improve the AR accuracy. A series of experiments with synthetic and real data were performed. Considering the real characteristics of system errors, the synthetic data was generated. The effects of tracking noise, feature noise, the number of fiducial markers, and the number of poses were analyzed through the simulation. For experiments with real data, we used a phantom designed considering real sinus and skull-base surgeries, and the AR error of the one-step method was compared with that of the two-step method. The comparison results showed the significantly increased accuracy of the one-step method.NI. INTRODUCTION 1 1.1 Introduction to Augmented Reality 1 1.2 AR based Surgical Navigation 3 1.3 Challenges and Related Works 8 1.4 Thesis Contributions 9 Ⅱ. AR IMPLEMENTATIONS 10 2.1 Nomenclatures 11 2.2 AR Configuration 12 2.3 Camera Calibration 14 Ⅲ. TWO-STEP CALIBRATION METHOD 18 3.1 Patient–Image Registration 20 3.2 Hand–Eye Calibration 22 3.3 Accuracy Issue of Two-Step Calibration Method 25 Ⅳ. PROPOSED CALIBRATION METHOD 27 4.1 One-Step Calibration Method 28 4.1.1 Feature Extraction 30 4.1.2 Initial AR-Core Transformations 35 4.1.3 Fast Approach to find Correspondences 38 4.1.4 Refinement 41 4.1.5 Least-Square Method 42 4.2 Summary for Process of One-Step Method 43 4.3 Aspect of Convenience 43 V. EXPERIMENTS 46 5.1 Experiments with Synthetic Data 46 5.1.1 Initialization for Simulation 46 5.1.2 Noise Addition 49 5.2 Experiments with Real Data 50 5.2.1 Phantom Design 50 5.2.2 Experimental Setup 53 5.2.3 Evaluation 57 VI. RESULTS 61 6.1 Results with Synthetic Data 61 6.2 Results with Real Data 79 VII. DISCUSSION 95 VIII. CONCLUTION 99본 논문에서는 내시경 수술에서 위치 추적 센서를 이용한 증강현실 기반 내비게이션을 구현하는 데 필요한 과정인 환자–영상 정합과 핸드–아이 캘리브레이션을 동시에 수행하는 원-스텝 방법을 제안한다. 기존 방법에서는 이러한 정합과 캘리브레이션 과정을 별도로 진행하였다. 본 논문에서는 이를 투-스텝 방법이라고 한다. 투-스텝 방법은 정합과 캘리브레이션을 위해 필요한 데이터를 취득하기 위해 수술 중 증강현실 구성과 다른 환경 구성이 필요했다. 투-스텝 방법으로부터 구해진 해는 투-스텝을 수행하기 위한 환경에서 최적화되지만, 증강현실 환경에서는 증강현실 구현에서 발생되는 계산 오차와 위치 추적 센서의 오차로 인해 최적화된 해가 아니다. 제안된 방법에서는 언급된 오차들의 영향을 완화하기 위해 증강현실 환경에서 내시경–오브젝트간 변환행렬을 계산하고, 이를 이용하여 환자–영상 및 핸드–아이 관계를 동시에 계산한다. 이 변환행렬은 계산 오차와 센서 오차로부터 자유롭기 때문에 증강현실 정확도를 향상시키는 중요한 요소가 된다. 시뮬레이션 및 실제 데이터를 이용한 일련의 실험을 수행하였다. 실제 환경적 특성들을 고려하여 시뮬레이션 데이터가 생성되었고, 이를 이용하여 데이터의 수, 피부 마커의 수 및 노이즈 수준이 원-스텝과 투-스텝에 미치는 영향을 분석하였다. 실제 데이터를 이용한 실험에서는 인체 내 중요 장기 및 구조물의 크기를 고려하여 제작된 팬텀을 사용하였다. 원-스텝 방법의 증강현실 오차를 계산하였고 투-스텝 방법의 오차와 비교하였다. 결과적으로 원-스텝 방법을 통해 증강현실의 정확도 (최대 70%)와 편리성이 현저하게 향상될 수 있음을 확인하였다.DoctordCollectio

A Novel End-Effector Robot System Enabling to Monitor Upper-Extremity Posture during Robot-Aided Planar Reaching Movements

End-effector type robots have been popularly applied to robot-aided therapy for rehabilitation purpose. However, those robots have a key drawback for the purpose: lack of the user's posture (joint angle) information. This letter proposes a novel end-effector rehabilitation robot system that contains a contactless motion sensor to monitor upper- extremity posture during robot-aided reaching exercise. The sensor allows the posture estimation without complicated procedures but has an inaccuracy problem such as occlusion and an unreliable segment length. Therefore, we developed a posture monitoring method, which is an analytical method without training procedure, based on the combined use of the information obtained from the sensor and the robot. Eight healthy subjects participated in the experiment with planar reaching exercise for validation. The results of joint angle estimation, high correlation coefficient (0.95 ± 0.03) and small errors (3.55 ± 0.70 deg), show that the proposed system can provide affordable upper-extremity posture estimation. © 2020 IEEE.1

Simultaneous Optimization of Patient-Image Registration and Hand-Eye Calibration for Accurate Augmented Reality in Surgery

Objective: Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient-image registration and hand-eye calibration. Conventional methods collect the necessary data to compute two output transformation matrices separately. However, the AR display setting during surgery generally differs from that during preoperative processes. Although conventional methods can identify optimal solutions under initial conditions, AR display errors are unavoidable during surgery owing to the inherent computational complexity of AR processes, such as error accumulation over successive matrix multiplications, and tracking errors of position sensor. Methods: We propose the simultaneous optimization of patient-image registration and hand-eye calibration in an AR environment before surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during optimization. After including the tracking information from the position sensor, patient-image registration and hand-eye calibration are optimized in terms of least-squares. Results: Experiments with synthetic data verify that the proposed method is less sensitive to computation and tracking errors. A phantom experiment with a position sensor is also conducted. The accuracy of the proposed method is significantly higher than that of the conventional method. Conclusion: The AR accuracy of the proposed method is compared with those of the conventional ones, and the superiority of the proposed method is verified. Significance: This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy. © 1964-2012 IEEE.1

Effective calibration of an endoscope to an optical tracking system for medical augmented reality

Background: We investigated the methods of calibrating an endoscope to an optical tracking system (OTS) for high accuracy augmented reality (AR)-based surgical navigation. We compared the possible calibration methods, and suggested the best method in terms of accuracy and speed in a medical environment. Material and methods: A calibration board with an attached OTS marker was used to acquire the pose data of the endoscope for the calibration. The transformation matrix from the endoscope to the OTS marker was calculated using the data. The calibration was performed by moving either the board or the endoscope in various placements. The re-projection error was utilized for evaluating the matrix. Results: From the statistical analysis, the method of moving the board was significantly more accurate than the method of moving the endoscope (p < 0.05). This difference resulted mainly from the uneven error distribution in the OTS measurement range and also the hand tremor in holding the endoscope. Conclusions: To increase the accuracy of AR, camera-to-OTS calibration should be performed by moving the board, and the board and the endoscope should be as close as possible to the OTS. This finding can contribute to improving the visualization accuracy in AR-based surgical navigation

Heterogeneous Stitching of X-ray Images According to Homographic Evaluation

The C-arm X-ray system is a common intraoperative imaging modality used to observe the state of a fractured bone in orthopedic surgery. Using C-arm, the bone fragments are aligned during surgery, and their lengths and angles with respect to the entire bone are measured to verify the fracture reduction. Since the field-of-view of the C-arm is too narrow to visualize the entire bone, a panoramic X-ray image is utilized to enlarge it by stitching multiple images. To achieve X-ray image stitching with feature detection, the extraction of accurate and densely matched features within the overlap region between images is imperative. However, since the features are highly affected by the properties and sizes of the overlap regions in consecutive X-ray images, the accuracy and density of matched features cannot be guaranteed. To solve this problem, a heterogeneous stitching of X-ray images was proposed. This heterogeneous stitching was completed according to the overlap region based on homographic evaluation. To acquire sufficiently matched features within the limited overlap region, integrated feature detection was used to estimate a homography. The homography was then evaluated to confirm its accuracy. When the estimated homography was incorrect, local regions around the matched feature were derived from integrated feature detection and substituted to re-estimate the homography. Successful X-ray image stitching of the C-arm was achieved by estimating the optimal homography for each image. Based on phantom and ex-vivo experiments using the proposed method, we confirmed a panoramic X-ray image construction that was robust compared to the conventional methods. © 2021, Society for Imaging Informatics in Medicine.FALS

A shape-partitioned statistical shape model for highly deformed femurs using X-ray images

To develop a patient-specific 3 D reconstruction of a femur modeled using the statistical shape model (SSM) and X-ray images, it is assumed that the target shape is not outside the range of variations allowed by the SSM built from a training dataset. We propose the shape-partitioned statistical shape model (SPSSM) to cover significant variations in the target shape. This model can divide a shape into several segments of anatomical interest. We break up the eigenvector matrix into the corresponding representative matrices for the SPSSM by preserving the relevant rows of the original matrix without segmenting the shape and building an independent SSM for each segment. To quantify the reconstruction error of the proposed method, we generated two groups of deformation models of the femur which cannot be easily represented by the conventional SSM. One group of femurs had an anteversion angle deformation, and the other group of femurs had two different scales of the femoral head. Each experiment was performed using the leave-one-out method for twelve femurs. When the femoral head was rotated by 30°, the average reconstruction error of the conventional SSM was 5.34 mm, which was reduced to 3.82 mm for the proposed SPSSM. When the femoral head size was decreased by 20%, the average reconstruction error of the SSM was 4.70 mm, which was reduced to 3.56 mm for the SPSSM. When the femoral head size was increased by 20%, the average reconstruction error of the SSM was 4.28 mm, which was reduced to 3.10 mm for the SPSSM. The experimental results for the two groups of deformation models showed that the proposed SPSSM outperformed the conventional SSM. © 2022 The Author(s). Published by Informa UK Limited, trading as Taylor &amp; Francis Group.TRU

Compact Bone Surgery Robot With a High-Resolution and High-Rigidity Remote Center of Motion Mechanism

Objective: Two important and difficult tasks during a bone drilling procedure are guiding the orientation of the drilling axis toward the target and maintaining the orientation against the drilling force. To accomplish these tasks, a remote center of motion (RCM) mechanism is adopted to align the orientation of the drilling axis without changing the entry point. However, existing RCM mechanisms do not provide sufficient resolution and rigidity to address hard tissue cases. Methods: We propose a new type of RCM mechanism that uses two sets of linear actuators and a gearless-arc guide to have a high resolution and rigidity. In addition, we designed a single motor-based drilling mechanism based on rolling friction. To achieve automatic control of the guiding and drilling process, we incorporated a computer-tomography-based navigation system that was equipped with an optical tracking system. Results: The effectiveness of the integrated robotic system was demonstrated through a series of experiments and ex vivo drilling tests on swine femurs. The proposed robotic system withstood a maximum external force of 51 N to maintain the joint angle, and the average drilling error was less than 1.2 mm. Conclusion: This study confirms the feasibility of the proposed bone drilling robotic system with a high-resolution and high-rigidity RCM mechanism. Significance: This drilling system is the first successful trial based on an RCM mechanism and a single motor-based drilling mechanism, reducing the footprint and required motors with respect to previous bone surgical robots. © 1964-2012 IEEE.1