Point set registration (PSR) is an essential problem in the field of surgical navigation and augmented reality (AR). In surgical navigation, the aim of registration is mapping the pre-operative space to the intra-operative space. This article introduces a reliable hybrid mixture model, in which the reliability of the normal vectors in the generalized point set (GPS) is examined and exploited. The motivation of considering the reliability of orientation information is that normal vectors cannot be estimated or measured accurately in the clinic. The point set (PS) is divided into two subsets according to the reliability of normal vectors. PSR is cast into the maximum likelihood estimation (MLE) problem. The expectation maximization (EM) framework is used to solve the MLE problem. In the E-step, the posterior probabilities between points in two PSs are computed. In the M-step, the transformation matrix and model components are updated by optimizing the objective function. We have demonstrated through extensive experiments on the human femur bone PS that the proposed algorithm outperforms the state-of-the-art ones in terms of accuracy, robustness, and convergence speed
