Mixed Reality and Artificial Intelligence: a Holistic Approach to Multimodal Visualization and Extended Interaction in Knee Osteotomy

Objective: Recent advancements in augmented reality led to planning and navigation systems for orthopedic surgery. However little is known about mixed reality (MR) in orthopedics. Furthermore, artificial intelligence (AI) has the potential to boost the capabilities of MR by enabling automation and personalization. The purpose of this work is to assess Holoknee prototype, based on AI and MR for multimodal data visualization and surgical planning in knee osteotomy, developed to run on the HoloLens 2 headset. Methods: Two preclinical test sessions were performed with 11 participants (eight surgeons, two residents, and one medical student) executing three times six tasks, corresponding to a number of holographic data interactions and preoperative planning steps. At the end of each session, participants answered a questionnaire on user perception and usability. Results: During the second trial, the participants were faster in all tasks than in the first one, while in the third one, the time of execution decreased only for two tasks (&#x201C;Patient selection&#x201D; and &#x201C;Scrolling through radiograph&#x201D;) with respect to the second attempt, but without statistically significant difference (respectively $p$ &#x003D; 0.14 and $p$ &#x003D; 0.13, p < 0.05 ). All subjects strongly agreed that MR can be used effectively for surgical training, whereas 10 (90.9&#x0025;) strongly agreed that it can be used effectively for preoperative planning. Six (54.5&#x0025;) agreed and two of them (18.2&#x0025;) strongly agreed that it can be used effectively for intraoperative guidance. Discussion/Conclusion: In this work, we presented Holoknee, the first holistic application of AI and MR for surgical planning for knee osteotomy. It reported promising results on its potential translation to surgical training, preoperative planning, and surgical guidance. Clinical and Translational Impact Statement - Holoknee can be helpful to support surgeons in the preoperative planning of knee osteotomy. It has the potential to impact positively the training of the future generation of residents and aid surgeons in the intraoperative stage

Bi-unicompartmental versus total knee arthroplasty: long term results

Objectives. The hypothesis of the current study is that Bi-Unicompartmental knee arthroplasty (Bi-Uni) could have long term outcomes equivalent to totl knee arthroplasty (TKA).Methods. A total of 19 patients treated from January 1999 to March 2003 with the simultaneous implantation of 2 Unicompartmental knee arthroplasties (UKA) were matched with 18 patients who had undergone a computer-assisted TKA for bicompartmental tibio-femoral osteoarthritis of the knee between August 1999 and September 2002.Results. At the last follow-up no statistical significant differences were seen for KSS, Function score and WOMAC Arthritis Index (pain score), while statistical differences were reported for the function (p&lt;0.05) and stiffness (p&lt;0.01) WOMAC indexes, respectively, in favor of the Bi-Uni group.Conclusion. The results of this study indicate that Bi-Uni is a valid alternative to address medial and lateral tibio-femoral osteoarthritis of the knee in selected cases at least as well as TKA

Patient-specific modeling of the trochlear morphologic anomalies by means of hyperbolic paraboloids

Diagnostic and therapeutic purposes are issuing pressing demands to improve the evaluation of the dysplasia condition of the femoral trochlea. The traditional clinical assessment of the dysplasia, based on Dejour classification, recognized 4 increasing (A, B, C, D) levels of severity. It has been extensively questioned in the literature that this classification methodology can be defective suggesting that quantitative measures can ensure more reliable criteria for the dysplasia severity assessment. This study reports on a novel technique to model the trochlear surface (TS), digitally reconstructed by 3D volumetric imaging, using three hyperbolic paraboloids (HP), one to describe the global trochlear aspect, two to represent the local aspects of the medial and lateral compartments, respectively. Results on a cohort of 43 patients, affected by aspecific anterior knee pain, demonstrate the consistency of the estimated model parameters with the morphologic aspect of the TS. The obtained small fitting error (on average lower than 0.80 mm) demonstrated that the ventral aspect of the trochlear morphology can be modeled with high accuracy by HPs. We also showed that HP modeling provides a continuous representation of morphologic variations in shape parameter space while we found that similar morphologic anomalies of the trochlear aspect are actually attributed to different severity grades in the Dejour classification. This finding is in agreement with recent works in the literature reporting that morphometric parameters can only optimistically be used to discriminate between the Grade A and the remaining three grades. In conclusion, we can assert that the proposed methodology is a further step toward modeling of anatomical surfaces that can be used to quantify deviations to normality on a patient-specific basis

Stacked sparse autoencoder networks and statistical shape models for automatic staging of distal femur trochlear dysplasia

Background: The quantitative morphological analysis of the trochlear region in the distal femur and the precise staging of the potential dysplastic condition constitute a key point for the use of personalized treatment options for the patella-femoral joint. In this paper, we integrated statistical shape models (SSM), able to represent the individual morphology of the trochlea by means of a set of parameters and stacked sparse autoencoder (SSPA) networks, which exploit the parameters to discriminate among different levels of abnormalities. Methods: Two datasets of distal femur reconstructions were obtained from CT scans, including pathologic and physiologic shapes. Both of them were processed to compute SSM of healthy and dysplastic trochlear regions. The parameters obtained by the 3D-3D reconstruction of a femur shape were fed into a trained SSPA classifier to automatically establish the membership to one of three clinical conditions, namely, healthy, mild dysplasia, and severe dysplasia of the trochlea. The validation was performed on a subset of the shapes not used in the construction of the SSM, by verifying the occurrence of a correct classification. Results: A major finding of the work is that SSM are able to represent anomalies of the trochlear geometry by means of specific eigenmodes of variation and to model the interplay between morphologic features related to dysplasia. Exploiting the patient-specific morphing parameters of SSM, computed by means of a 3D-3D reconstruction, SSPA is demonstrated to outperform traditional discriminant analysis in classifying healthy, mild, and severe trochlear dysplasia providing 99%, 97%, and 98% accuracy for each of the three classes, respectively (discriminant analysis accuracy: 85%, 89%, and 77%). Conclusions: From a clinical point of view, this paper contributes to support the increasing role of SSM, integrated with deep learning techniques, in diagnostics and therapy definition as quantitative and advanced visualization tools

CEL-Unet: Distance Weighted Maps and Multi-Scale Pyramidal Edge Extraction for Accurate Osteoarthritic Bone Segmentation in CT Scans

Unet architectures are being investigated for automatic image segmentation of bones in CT scans because of their ability to address size-varying anatomies and pathological deformations. Nonetheless, changes in mineral density, narrowing of joint spaces and formation of largely irregular osteophytes may easily disrupt automatism requiring extensive manual refinement. A novel Unet variant, called CEL-Unet, is presented to boost the segmentation quality of the femur and tibia in the osteoarthritic knee joint. The neural network embeds region-aware and two contour-aware branches in the decoding path. The paper features three main technical novelties: 1) directed connections between contour and region branches progressively at different decoding scales; 2) pyramidal edge extraction in the contour branch to perform multi-resolution edge processing; 3) distance-weighted cross-entropy loss function to increase delineation quality at the sharp edges of the shapes. A set of 700 knee CT scans was used to train the model and test segmentation performance. Qualitatively CEL-Unet correctly segmented cases where the state-of-the-art architectures failed. Quantitatively, the Jaccard indexes of femur and tibia segmentation were 0.98 and 0.97, with median 3D reconstruction errors less than 0.80 and 0.60 mm, overcoming competitive Unet models. The results were evaluated against knee arthroplasty planning based on personalized surgical instruments (PSI). Excellent agreement with reference data was found for femoral (0.11°) and tibial (0.05°) alignments of the distal and proximal cuts computed on the reconstructed surfaces. The bone segmentation was effective for large pathological deformations and osteophytes, making the techniques potentially usable in PSI-based surgical planning, where the reconstruction accuracy of the bony shapes is one of the main critical factors for the success of the operation

2D/3D reconstruction of the distal femur using statistical shape models addressing personalized surgical instruments in knee arthroplasty: A feasibility analysis

Background: Personalized surgical instruments (PSI) have gained success in the domain of total knee replacement, demonstrating clinical outcomes similar or even superior to both traditional and navigated surgeries. The key requirement for prototyping PSI is the availability of the digital bony surface. In this paper, we aim at verifying whether the 2D/3D reconstruction of the distal femur, based on statistical shape models (SSM), grants sufficient accuracy, especially in the condylar regions, to support a PSI technique. Methods: Computed tomographic knee datasets acquired on 100 patients with severe cartilage damage were retrospectively considered in this work. All the patients underwent total knee replacement using the PSI-based surgical technique. Eighty out of 100 reconstructed distal femur surfaces were used to build the statistical model. The remaining 20 surfaces were used for testing. The 2D/3D reconstruction process was based on digital reconstructed radiographies (DRRs) obtained with a simulated X-ray projection process. An iterative optimization procedure, based on an evolutionary algorithm, systematically morphed the statistical model to decrease the difference between the DRR, obtained by the original CT dataset, and the DRR obtained from the morphed surface. Results: Over the 80 variations, the first ten modes were found sufficient to reconstruct the distal femur surface with accuracy. Using three DRR, the maximum Hausdorff and RMS distance errors were lower than 1.50 and 0.75 mm, respectively. As expected, the reconstruction quality improved by increasing the number of DRRs. Statistical difference (P < 0.001) was found in the 2 vs 3, 2 vs 4 and 2 vs 5 DRR, thus proving that adding just a single displaced projection to the two traditional sagittal and coronal X-ray images improved significantly the reconstruction quality. The effect of the PSI contact area errors on the distal cut direction featured a maximum median error lower than 2° and 0.5° on the sagittal and frontal plane, respectively. Statistical difference was found (P < 0.0001) in the reconstruction accuracy when comparing SSM built using pathologic with respect to non-pathologic shapes (cadavers), meaning that, to improve the patient-specific reconstruction, the morphologic anomalies, specific to the pathology, must be embedded into the SSM. Conclusions: We showed that the X-ray based reconstruction of the distal femur is reasonable also in presence of pathologic bony conditions, featuring accuracy results similar to earlier reports in the literature that reconstructed normal femurs. This finding discloses the chance of applying the proposed methodology to the reconstruction of bony surfaces used in the PSI surgical approach

Deep 3D Convolutional Networks to Segment Bones Affected by Severe Osteoarthritis in CT Scans for PSI-Based Knee Surgical Planning

Segmentation of bony structures in CT scans is a crucial step in knee arthroplasty based on personalized surgical instruments (PSI). As a matter of fact, the success of the surgery depends on the quality of the matching between the patient-specific resection jigs, manufactured exploiting the patient bony surfaces attained by segmentation, and true patient surfaces. Severe pathological conditions as chronic osteoarthritis, deteriorating the cartilages, narrowing the intra-articular spaces and leading to bone impingement, complicate the segmentation making the recognition of bony boundaries sub-optimal for traditional semi-automated methods and often extremely difficult even for expert radiologists. Deep convolutional neural networks (CNNs) have been investigated in the last years towards automatic labeling of diagnostic images, especially harnessing the encoding-decoding U-Net architecture. In this article, we implemented deep CNNs to encompass the concurrent segmentation of the distal femur and the proximal tibia in CT images and evaluate how segmentation uncertainty may impact on the surgical planning. A retrospective set of 200 knee CT scans of patients was used to train the network and test the segmentation performances. Tests on a subset of 20 scans provided median dice, sensitivity and positive predictive value indices greater than 96% for both shapes, with median 3D reconstruction error in the range of 0.5mm. Median 3D errors on both PSI femoral and tibial contact areas and surgical cut alignments were less than 2mm and 2°, respectively, which can be considered clinically acceptable. These results substantiate that deep CNN architectures can disclose the opportunity of segmenting bone shapes in CT scans for PSI-based surgical planning with promising accuracy. However, we observed that segmentation scores alone cannot be taken as representative of the 3D errors at the contact areas of the PSI. Therefore when comparing segmentation algorithms of PSI-based surgical planning the 3D errors should be explicitly analyzed