Keypoints-based surface representation for 3D modeling and 3D object recognition

The three-dimensional (3D) modeling and recognition of 3D objects have been traditionally performed using local features to represent the underlying 3D surface. Extraction of features requires cropping of several local surface patches around detected keypoints. Although an important step, the extraction and representation of such local patches adds to the computational complexity of the algorithms. This paper proposes a novel Keypoints-based Surface Representation (KSR) technique. The proposed technique has the following two characteristics: (1) It does not rely on the computation of features on a small surface patch cropped around a detected keypoint. Rather, it exploits the geometrical relationship between the detected 3D keypoints for local surface representation. (2) KSR is computationally efficient, requiring only seconds to process 3D models with over 50,000 points with a MATLAB implementation. Experimental results on the UWA and Stanford 3D models dataset suggest that it can accurately perform pairwise and multiview range image registration (3D modeling). KSR was also tested for 3D object recognition with occluded scenes. Recognition results on the UWA dataset show that the proposed technique outperforms existing methods including 3D-Tensor, VD-LSD, keypoint-depth based feature, spherical harmonics and spin image with a recognition rate of 95.9%. The proposed approach also achieves a recognition rate of 93.5% on the challenging Ca'Fascori dataset compared to 92.5% achieved by game-theoretic. The proposed method is computationally efficient compared to state-of-the-art local feature methods

Real-time Topology-Aware Augmented Reality

Augmented Reality (AR) technology fuses virtual information with the real-world en- vironment to enhance the way people interact with digital information in their physical world. This thesis is concerned with topology-aware AR systems designed to be aware of the topology changes in the surroundings and explore the topological features of scenes. Topological structures, such as graphs, can provide information on the relationship between point clouds to improve the quality of point cloud-based real-world 3D map reconstruc- tions for topology-aware AR systems. The reconstructed 3D maps provide information to improve the registration accuracy between virtual objects and the physical environment. Furthermore, 3D maps also help to reduce registration failures caused by complex and dynamic scenes, such as object occlusions, object motion, and object deformation. This thesis explores algorithms, computational methods, and frameworks for dense 3D surface reconstructions based on monocular videos and images for augmented reality applications. The main contributions of this PhD work are: 1) Proposed a graph deep learning-based framework for monocular depth estimation, which learns non-Euclidean features and improves the accuracy of depth estimations. Mathematical background on group equivariance, including translation equivariance and permutation equivariance, is also introduced to provide theoretical support for the proposed network; 2) Conducted two use cases to demonstrate the capabilities of the proposed methods in improving fine details of depth estimation for complex and unstructured environments with free camera motions; 3) A further improved the framework to address low-illumination endoscopy videos; 4) Proposed a statistical method to handle the non-rigid point cloud registration with special topology changes. Within which, a clustering and refinement scheme is proposed to deal with distribution irregularities of point sets; 5) Developed a framework to demonstrate the functionality of the proposed method in AR. Under challenging scenes such as endoscopy and unmanned aerial vehicle videos, the proposed methods outperform the state-of-the-art algorithms with robustness and accuracy. For example, the proposed depth estimation method improves the 3D data acquisition, the Break and Splice framework improves the 3D dynamic reconstruction, and the proposed AR framework provides a solution in dynamic scenes for medical applications