An efficient hybrid method for 3D to 2D medical image registration

PURPOSE: The purpose of this paper is to present a method for registration of 3D computed tomography to 2D single-plane fluoroscopy knee images to provide 3D motion information for knee joints. This 3D kinematic information has unique utility for examining joint kinematics in conditions such as ligament injury, osteoarthritis and after joint replacement. METHODS: We proposed a non-invasive rigid body image registration method which is based on two different multimodal similarity measures. This hybrid registration method helps to achieve a trade-off among different challenges including, time complexity and accuracy. RESULTS: We performed a number of experiments to evaluate the performance of the proposed method. The experimental results show that the proposed method is as accurate as one of the most recent registration methods while it is several times faster than that method. CONCLUSION: The proposed method is a non-invasive, fast and accurate registration method, which can provide 3D information for knee joint kinematic measurements. This information can be very helpful in improving the accuracy of diagnosis and providing targeted treatment

Augmented Image-Guidance for Transcatheter Aortic Valve Implantation

The introduction of transcatheter aortic valve implantation (TAVI), an innovative stent-based technique for delivery of a bioprosthetic valve, has resulted in a paradigm shift in treatment options for elderly patients with aortic stenosis. While there have been major advancements in valve design and access routes, TAVI still relies largely on single-plane fluoroscopy for intraoperative navigation and guidance, which provides only gross imaging of anatomical structures. Inadequate imaging leading to suboptimal valve positioning contributes to many of the early complications experienced by TAVI patients, including valve embolism, coronary ostia obstruction, paravalvular leak, heart block, and secondary nephrotoxicity from contrast use. A potential method of providing improved image-guidance for TAVI is to combine the information derived from intra-operative fluoroscopy and TEE with pre-operative CT data. This would allow the 3D anatomy of the aortic root to be visualized along with real-time information about valve and prosthesis motion. The combined information can be visualized as a `merged\u27 image where the different imaging modalities are overlaid upon each other, or as an `augmented\u27 image, where the location of key target features identified on one image are displayed on a different imaging modality. This research develops image registration techniques to bring fluoroscopy, TEE, and CT models into a common coordinate frame with an image processing workflow that is compatible with the TAVI procedure. The techniques are designed to be fast enough to allow for real-time image fusion and visualization during the procedure, with an intra-procedural set-up requiring only a few minutes. TEE to fluoroscopy registration was achieved using a single-perspective TEE probe pose estimation technique. The alignment of CT and TEE images was achieved using custom-designed algorithms to extract aortic root contours from XPlane TEE images, and matching the shape of these contours to a CT-derived surface model. Registration accuracy was assessed on porcine and human images by identifying targets (such as guidewires or coronary ostia) on the different imaging modalities and measuring the correspondence of these targets after registration. The merged images demonstrated good visual alignment of aortic root structures, and quantitative assessment measured an accuracy of less than 1.5mm error for TEE-fluoroscopy registration and less than 6mm error for CT-TEE registration. These results suggest that the image processing techniques presented have potential for development into a clinical tool to guide TAVI. Such a tool could potentially reduce TAVI complications, reducing morbidity and mortality and allowing for a safer procedure

Fusion of interventional ultrasound & X-ray

In einer immer älter werdenden Bevölkerung wird die Behandlung von strukturellen Herzkrankheiten zunehmend wichtiger. Verbesserte medizinische Bildgebung und die Einführung neuer Kathetertechnologien führten dazu, dass immer mehr herkömmliche chirurgische Eingriffe am offenen Herzen durch minimal invasive Methoden abgelöst werden. Diese modernen Interventionen müssen durch verschiedenste Bildgebungsverfahren navigiert werden. Hierzu werden hauptsächlich Röntgenfluoroskopie und transösophageale Echokardiografie (TEE) eingesetzt. Röntgen bietet eine gute Visualisierung der eingeführten Katheter, was essentiell für eine gute Navigation ist. TEE hingegen bietet die Möglichkeit der Weichteilgewebedarstellung und kann damit vor allem zur Darstellung von anatomischen Strukturen, wie z.B. Herzklappen, genutzt werden. Beide Modalitäten erzeugen Bilder in Echtzeit und werden für die erfolgreiche Durchführung minimal invasiver Herzchirurgie zwingend benötigt. Üblicherweise sind beide Systeme eigenständig und nicht miteinander verbunden. Es ist anzunehmen, dass eine Bildfusion beider Welten einen großen Vorteil für die behandelnden Operateure erzeugen kann, vor allem eine verbesserte Kommunikation im Behandlungsteam. Ebenso können sich aus der Anwendung heraus neue chirurgische Worfklows ergeben. Eine direkte Fusion beider Systeme scheint nicht möglich, da die Bilddaten eine zu unterschiedliche Charakteristik aufweisen. Daher kommt in dieser Arbeit eine indirekte Registriermethode zum Einsatz. Die TEE-Sonde ist während der Intervention ständig im Fluoroskopiebild sichtbar. Dadurch wird es möglich, die Sonde im Röntgenbild zu registrieren und daraus die 3D Position abzuleiten. Der Zusammenhang zwischen Ultraschallbild und Ultraschallsonde wird durch eine Kalibrierung bestimmt. In dieser Arbeit wurde die Methode der 2D-3D Registrierung gewählt, um die TEE Sonde auf 2D Röntgenbildern zu erkennen. Es werden verschiedene Beiträge präsentiert, welche einen herkömmlichen 2D-3D Registrieralgorithmus verbessern. Nicht nur im Bereich der Ultraschall-Röntgen-Fusion, sondern auch im Hinblick auf allgemeine Registrierprobleme. Eine eingeführte Methode ist die der planaren Parameter. Diese verbessert die Robustheit und die Registriergeschwindigkeit, vor allem während der Registrierung eines Objekts aus zwei nicht-orthogonalen Richtungen. Ein weiterer Ansatz ist der Austausch der herkömmlichen Erzeugung von sogenannten digital reconstructed radiographs. Diese sind zwar ein integraler Bestandteil einer 2D-3D Registrierung aber gleichzeitig sehr zeitaufwendig zu berechnen. Es führt zu einem erheblichen Geschwindigkeitsgewinn die herkömmliche Methode durch schnelles Rendering von Dreiecksnetzen zu ersetzen. Ebenso wird gezeigt, dass eine Kombination von schnellen lernbasierten Detektionsalgorithmen und 2D-3D Registrierung die Genauigkeit und die Registrierreichweite verbessert. Zum Abschluss werden die ersten Ergebnisse eines klinischen Prototypen präsentiert, welcher die zuvor genannten Methoden verwendet.Today, in an elderly community, the treatment of structural heart disease will become more and more important. Constant improvements of medical imaging technologies and the introduction of new catheter devices caused the trend to replace conventional open heart surgery by minimal invasive interventions. These advanced interventions need to be guided by different medical imaging modalities. The two main imaging systems here are X-ray fluoroscopy and Transesophageal  Echocardiography (TEE). While X-ray provides a good visualization of inserted catheters, which is essential for catheter navigation, TEE can display soft tissues, especially anatomical structures like heart valves. Both modalities provide real-time imaging and are necessary to lead minimal invasive heart surgery to success. Usually, the two systems are detached and not connected. It is conceivable that a fusion of both worlds can create a strong benefit for the physicians. It can lead to a better communication within the clinical team and can probably enable new surgical workflows. Because of the completely different characteristics of the image data, a direct fusion seems to be impossible. Therefore, an indirect registration of Ultrasound and X-ray images is used. The TEE probe is usually visible in the X-ray image during the described minimal-invasive interventions. Thereby, it becomes possible to register the TEE probe in the fluoroscopic images and to establish its 3D position. The relationship of the Ultrasound image to the Ultrasound probe is known by calibration. To register the TEE probe on 2D X-ray images, a 2D-3D registration approach is chosen in this thesis. Several contributions are presented, which are improving the common 2D-3D registration algorithm for the task of Ultrasound and X-ray fusion, but also for general 2D-3D registration problems. One presented approach is the introduction of planar parameters that increase robustness and speed during the registration of an object on two non-orthogonal views. Another approach is to replace the conventional generation of digital reconstructedradiographs, which is an integral part of 2D-3D registration but also a performance bottleneck, with fast triangular mesh rendering. This will result in a significant performance speed-up. It is also shown that a combination of fast learning-based detection algorithms with 2D-3D registration will increase the accuracy and the capture range, instead of employing them solely to the  registration/detection of a TEE probe. Finally, a first clinical prototype is presented which employs the presented approaches and first clinical results are shown

AN AUTOMATED, DEEP LEARNING APPROACH TO SYSTEMATICALLY & SEQUENTIALLY DERIVE THREE-DIMENSIONAL KNEE KINEMATICS DIRECTLY FROM TWO-DIMENSIONAL FLUOROSCOPIC VIDEO

Total knee arthroplasty (TKA), also known as total knee replacement, is a surgical procedure to replace damaged parts of the knee joint with artificial components. It aims to relieve pain and improve knee function. TKA can improve knee kinematics and reduce pain, but it may also cause altered joint mechanics and complications. Proper patient selection, implant design, and surgical technique are important for successful outcomes. Kinematics analysis plays a vital role in TKA by evaluating knee joint movement and mechanics. It helps assess surgery success, guides implant and technique selection, informs implant design improvements, detects problems early, and improves patient outcomes. However, evaluating the kinematics of patients using conventional approaches presents significant challenges. The reliance on 3D CAD models limits applicability, as not all patients have access to such models. Moreover, the manual and time-consuming nature of the process makes it impractical for timely evaluations. Furthermore, the evaluation is confined to laboratory settings, limiting its feasibility in various locations. This study aims to address these limitations by introducing a new methodology for analyzing in vivo 3D kinematics using an automated deep learning approach. The proposed methodology involves several steps, starting with image segmentation of the femur and tibia using a robust deep learning approach. Subsequently, 3D reconstruction of the implants is performed, followed by automated registration. Finally, efficient knee kinematics modeling is conducted. The final kinematics results showed potential for reducing workload and increasing efficiency. The algorithms demonstrated high speed and accuracy, which could enable real-time TKA kinematics analysis in the operating room or clinical settings. Unlike previous studies that relied on sponsorships and limited patient samples, this algorithm allows the analysis of any patient, anywhere, and at any time, accommodating larger subject populations and complete fluoroscopic sequences. Although further improvements can be made, the study showcases the potential of machine learning to expand access to TKA analysis tools and advance biomedical engineering applications

Fast catheter segmentation and tracking based on x-ray fluoroscopic and echocardiographic modalities for catheter-based cardiac minimally invasive interventions

X-ray fluoroscopy and echocardiography imaging (ultrasound, US) are two imaging modalities that are widely used in cardiac catheterization. For these modalities, a fast, accurate and stable algorithm for the detection and tracking of catheters is required to allow clinicians to observe the catheter location in real-time. Currently X-ray fluoroscopy is routinely used as the standard modality in catheter ablation interventions. However, it lacks the ability to visualize soft tissue and uses harmful radiation. US does not have these limitations but often contains acoustic artifacts and has a small field of view. These make the detection and tracking of the catheter in US very challenging. The first contribution in this thesis is a framework which combines Kalman filter and discrete optimization for multiple catheter segmentation and tracking in X-ray images. Kalman filter is used to identify the whole catheter from a single point detected on the catheter in the first frame of a sequence of x-ray images. An energy-based formulation is developed that can be used to track the catheters in the following frames. We also propose a discrete optimization for minimizing the energy function in each frame of the X-ray image sequence. Our approach is robust to tangential motion of the catheter and combines the tubular and salient feature measurements into a single robust and efficient framework. The second contribution is an algorithm for catheter extraction in 3D ultrasound images based on (a) the registration between the X-ray and ultrasound images and (b) the segmentation of the catheter in X-ray images. The search space for the catheter extraction in the ultrasound images is constrained to lie on or close to a curved surface in the ultrasound volume. The curved surface corresponds to the back-projection of the extracted catheter from the X-ray image to the ultrasound volume. Blob-like features are detected in the US images and organized in a graphical model. The extracted catheter is modelled as the optimal path in this graphical model. Both contributions allow the use of ultrasound imaging for the improved visualization of soft tissue. However, X-ray imaging is still required for each ultrasound frame and the amount of X-ray exposure has not been reduced. The final contribution in this thesis is a system that can track the catheter in ultrasound volumes automatically without the need for X-ray imaging during the tracking. Instead X-ray imaging is only required for the system initialization and for recovery from tracking failures. This allows a significant reduction in the amount of X-ray exposure for patient and clinicians.Open Acces

Multi-modal Image Registration

In different areas, particularly medical image analysis, there is a vital need to access and analyse dynamic three dimensional (3D) images of the anatomical structures of the human body. This can enable specialists to track events as well as clinically conduct and evaluate surgical and radio therapeutical procedures. For example, measuring the 3D kinematics of knee joints in a dynamic manner is essential for understanding their normal functions and diagnosing any pathology, such as ligament injury and osteoarthritis. For evaluations of subsequent treatments, such as surgery and rehabilitation, and designs of joint replacements, having knowledge of the movements of knee joints is necessary. Image registration is increasingly being applied to medical image analysis. Whereas in mono-modal registration, the images to be registered are acquired by the same sensor, in multi-modal image registration, they can be taken from different devices or imaging protocols which makes this registration process much more challenging. The invasive or non-invasive nature of the registration method used, the computational time it requires as well as its accuracy and robustness against a large range of initial displacements are the most important features used for its evaluation. As currently available approaches have limited capabilities to register images with large initial displacements and are either not sufficiently accurate or very computationally expensive, the objective of this research is to propose new registration methods, that provide dynamic 3D images, to address these issues. In the first part of this study, I conducted research on registering an individuals’ natural knee bones that can provide 3D information of knee joint kinematics which can be very helpful for improving the accuracy of diagnosis and enabling targeted treatments. A fast, accurate and robust hybrid rigid body registration method based on two different multi-modal similarity measures, the edge position difference (EPD) and sum-of-conditional variance (SCV), is proposed. It uses a gradient descent optimisation technique to register multi-modal images and determine the best transformation parameters. It helps to achieve a trade-off among different challenges, including time complexity, accuracy and robustness against a large range of initial displacements. To evaluate it, several experiments were performed on two different databases: one collected from the knee bones of four patients and the other from three knee cadavers installed on a mechanical positioning system, with the results showing that this method is accurate, fast and robust against large initial displacement. Then, I conducted research on registering implanted human knee joints and proposed a non-invasive, robust 3D-to-2D registration method which can be used for 3D evaluations of the status of knee implants after joint replacement surgeries. In this method, 3D models of the implants for an individual with the relevant post-operative fluoroscopy frames are able to be used in the registration process. As a result, it is possible to perform 3D analysis at any time after a surgery by simply taking single-plane radiographs. This approach uses the EPD multi-modal similarity measure together with a steepest descent optimisation method. It applies coarse-to-fine registration steps to determine the transformation parameters that lead to the best alignment between the model used and X-ray images to be registered. The experimental results showed that not only does the proposed registration method have a high success rate but that it is also much faster than the most relevant competitive approach. Although the experiments were designed for a 3D analysis of total knee arthroplasty (TKA) components, this proposed method can be applied to other joints such as the ankle or hip. In the final part of my research, I developed a multi-frame 2D fluoroscopy to 3D model registration method for measuring the kinematics of post-operative knee joints. It uses a coarse-to-fine approach and applies the normalised EPD (NEPD) and SCV similarity measures together with a gradient descent optimisation method and an interpolation estimation one. In order to measure the kinematics of post- operative knee joints, after a TKA surgery, a 3D knee implant model can be registered with a number of single-plane fluoroscopy frames of the patient’s knee. Generally, when this number is quite high, the computational cost for registering the frames and a 3D model is expensive. Therefore, in order to speed up the registration process, a cubic spline interpolation prediction method is applied to initialise and estimate the 3D positions of the 3D model in each fluoroscopy frame instead of applying a registration algorithm on all the frames, one after the other. The estimated 3D positions are then tuned using a registration improvement step. The experimental results demonstrated that the proposed registration method is much faster than the best existing one and achieves almost the same accuracy. It also provides smooth registration results which can lead to more natural 3D modelling of joint movements