Farshad, M

Jokeit, M

Kim, K H

Snedeker, J G

Widmer, J

English

ZORA

Zurich Open Repository andArchiveUniversity of ZurichUniversity LibraryStrickhofstrasse 39CH-8057 Zurichwww.zora.uzh.chYear: 2022Mesh-based 3D Reconstruction from Bi-planar RadiographsJokeit, M ; Kim, K H ; Snedeker, J G ; Farshad, M ; Widmer, JPosted at the Zurich Open Repository and Archive, University of ZurichZORA URL: https://doi.org/10.5167/uzh-227854Scientific Publication in Electronic FormPublished Version  The following work is licensed under a Publisher License.Originally published at:Jokeit, M; Kim, K H; Snedeker, J G; Farshad, M; Widmer, J (2022). Mesh-based 3D Reconstruction from Bi-planarRadiographs. University of Massachusetts Amherst: OpenReview.Medical Imaging with Deep Learning – Under Review 2022 Short Paper – MIDL 2022 submissionMesh-based 3D Reconstruction from Bi-planar RadiographsMoritz Jokeit1,2 moritz.jokeit@hest.ethz.ch1 Institute for Biomechanics, ETH Zurich, Zurich, Switzerland2 Department of Orthopaedics, Balgrist University Hospital, Zurich, SwitzerlandJi Hyun Kim1 jihyun.kim@hest.ethz.chJess G. Snedeker1,2 jess.snedeker@hest.ethz.chMazda Farshad2 mazda.farshad@balgrist.chJonas Widmer1,2 jonas.widmer@balgrist.chEditors: Under Review for MIDL 2022AbstractReconstruction of 3D surfaces from sparse 2D data is a challenging problem that attractedincreasing attention also in the medical field where image acquisition is expensive and thepatients often bear high radiation doses (CT, fluoroscopy). Further, advances in computer-guided surgical assistant systems and preoperative planning necessitate fast 3D reconstruc-tion from sparse image data. Recent learning-based approaches showed notable success inreconstructing primitive objects leveraging abundant artificial data sets. However, quality3D data in the clinical context is often scarce. This motivates the exploitation of domainknowledge in form of anatomical shape priors to simplify the reconstruction problem. Fur-ther, mesh-sensitive applications (e.g., finite element analysis of implant design) greatlybenefit from pre-defined mesh topologies. Thus, we propose a concept for implementingand training a learning-based patient-specific 3D reconstruction from bi-planar radiographsbased on altering anatomical template meshes.Keywords: 3D reconstruction, deep learning, mesh, medical imaging, X-ray, digitallyreconstructed radiograph (DRR), style transfer, computed tomography (CT).1. IntroductionReconstruction and registration of the patient’s anatomy become more important withadvances in computer-assisted surgery and planning (Fürnstahl et al., 2012). Current ap-proaches often rely on segmented CT scans, which are costly, time-consuming to acquireand expose the patient to high radiation doses. However, 3D reconstruction from scarce 2Ddata leads to an ill-posed problem that can not be solved with conventional methods. Here,learning-based solutions for sparse-view 3D reconstruction provide a promising alternative(Kasten et al., 2020). Wherever data are sparse, domain knowledge helps to simplify thesolution process. Inspired by early medical segmentation algorithms using shape priors, wewant to exploit anatomical model templates to enhance the learning-based 3D reconstruc-tion from bi-planar radiographs. The end-to-end mesh prediction facilitates patient-specificmusculoskeletal modeling, finite element analysis, or preoperative planning.© 2022 M. Jokeit, J.H. Kim, J.G. Snedeker, M. Farshad & J. Widmer.Jokeit Kim Snedeker Farshad Widmer2. MethodsThe proposed pipeline aims to reconstruct patient-specific geometries provided with onlybi-planar radiographs and a template mesh of the desired anatomical structure. Figure 1describes the workflow on the example of the pelvic bone. The core of our reconstructionpipeline is a deformation network that predicts a translation vector for each vertex ofthe mesh template which best fits the radiographic input. Following the idea of Wenet al. (2019), convolutional encoder networks (VGG-16) extract information from the tworadiographs with bi-planar perceptual feature pooling. The resulting feature vector is thenconcatenated with the first layer of the reconstruction network. As a primary variant,we suggest employing a fully-connected architecture as in Pan et al. (2019) with ReLUactivations in the first three layers and tanh activations in the output layer. Finally, applyingthe predicted translation vectors to the template nodes yields the patient-specific mesh.To train the network we propose a loss term that simply considers the sum of the eu-clidean distances between every vertex xi and yi of the predicted mesh and the ground truth,respectively: Ld =Pi ||xi − yi||2. Extending the loss function with geometry regularizerspotentially benefits the mesh’s smoothness (Wen et al., 2019).To obtain training data, we derive digitally reconstructed radiographs (DRRs) from1,184 CT volumes (Liu et al., 2021). Then, an iterative closest point algorithm fits thetemplate mesh to the corresponding 3D segmentations generating the ground truth meshthat complements the DRR (e.g., Scalismo). A frequent problem of DRRs is their mismatchin style regarding the emulated image modality. However, our preliminary results showedthat generative networks like CycleGAN can be successfully trained to imitate the desiredimage style in an unsupervised manner.Figure 1: Concept for the implementation of the proposed reconstruction pipeline. Cross-view perceptual feature pooling adapted from Wen et al. (2019).2Mesh-based 3D Reconstruction from Bi-planar Radiographs3. ConclusionWe believe combining shape priors and learning-based reconstruction has potential in aplethora of medical applications, e.g., a predefined mesh facilitates further use in patient-specific models and implant design. However, a mesh-based reconstruction comes not with-out challenges regarding the accuracy and implementational complexity. The presentedconcept will likely adapt to new challenges and ideas that occur during the implementationprocess. For instance, Wen et al. (2019) already demonstrated that graph convolutions arewell suited for mesh-based reconstruction. The known geometry and physical properties ofthe X-ray device offer even more potential for domain knowledge exploitation. Further, theimage encoder could be replaced by lighter and more efficient network architectures.AcknowledgmentsWe thank Prof. Dr. Philipp Fürnstahl and Dr. Hooman Esfandiari from ROCS, BalgristUniversity Hospital, University of Zurich, for their valuable inputs, and the PromedicaStiftung, Chur, for financial support. This research was supported by grants from NVIDIAand utilized NVIDIA RTX A6000 GPUs.ReferencesPhilipp Fürnstahl, Gábor Székely, Christian Gerber, Jürg Hodler, Jess Gerrit Snedeker,and Matthias Harders. Computer assisted reconstruction of complex proximal humerusfractures for preoperative planning. Medical Image Analysis, 16(3):704–720, April 2012.ISSN 1361-8415. doi: 10.1016/j.media.2010.07.012. URL https://www.sciencedirect.com/science/article/pii/S1361841510001040.Yoni Kasten, Daniel Doktofsky, and Ilya Kovler. End-To-End Convolutional NeuralNetwork for 3D Reconstruction of Knee Bones from Bi-planar X-Ray Images. InFarah Deeba, Patricia Johnson, Tobias Würfl, and Jong Chul Ye, editors, MachineLearning for Medical Image Reconstruction, Lecture Notes in Computer Science, pages123–133, Cham, 2020. Springer International Publishing. ISBN 9783030615987. doi:10.1007/978-3-030-61598-7 12.Pengbo Liu, Hu Han, Yuanqi Du, Heqin Zhu, Yinhao Li, Feng Gu, Honghu Xiao, Jun Li,Chunpeng Zhao, Li Xiao, Xinbao Wu, and S. Kevin Zhou. Deep Learning to SegmentPelvic Bones: Large-scale CT Datasets and Baseline Models. arXiv:2012.08721 [cs],March 2021. URL http://arxiv.org/abs/2012.08721. arXiv: 2012.08721.Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep MeshReconstruction From Single RGB Images via Topology Modification Networks. In2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9963–9972, Seoul, Korea (South), October 2019. IEEE. ISBN 978-1-72814-803-8. doi:10.1109/ICCV.2019.01006. URL https://ieeexplore.ieee.org/document/9009447/.Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. Pixel2Mesh++: Multi-View 3DMesh Generation via Deformation. arXiv:1908.01491 [cs], August 2019. URL http://arxiv.org/abs/1908.01491. arXiv: 1908.01491.3

Mesh-based 3D Reconstruction from Bi-planar Radiographs

https://www.zora.uzh.ch/id/eprint/227854/1/2022_Jokeit_Mesh_based_3D_Reconstruction_from_Bi_planar_Radiographs_MIDL.pdf

Mesh-based 3D Reconstruction from Bi-planar Radiographs

Abstract

Similar works

Full text

Available Versions

ZORA