Learning SO(3) Equivariant Representations with Spherical CNNs

We address the problem of 3D rotation equivariance in convolutional neural networks. 3D rotations have been a challenging nuisance in 3D classification tasks requiring higher capacity and extended data augmentation in order to tackle it. We model 3D data with multi-valued spherical functions and we propose a novel spherical convolutional network that implements exact convolutions on the sphere by realizing them in the spherical harmonic domain. Resulting filters have local symmetry and are localized by enforcing smooth spectra. We apply a novel pooling on the spectral domain and our operations are independent of the underlying spherical resolution throughout the network. We show that networks with much lower capacity and without requiring data augmentation can exhibit performance comparable to the state of the art in standard retrieval and classification benchmarks.Comment: Camera-ready. Accepted to ECCV'18 as oral presentatio

3D Pottery Shape Similarity Matching Based on Digital Signatures

Budgeting is an element of management controll system that works as planning device. Budgeting is accepted as key element in the system of planning and controlling. The purpose of the study is know the influence of budgeting arrangement participation, the difficulty degree of budgeting objectives and budgeting evaluation to the realization degree of budgeting revenue from tax of land and construction/Pajak Bumi dan Bangunan. Samples of the study are KP PBB Kanwil XI DJP East Java with 8 (eight) number of KP PBB that consist of KP PBB Surabaya I, Surabaya, II, Surabaya III, Gresik, Bojonegoro, Pamekasan, Madiun, Ngawi. Data analysis for this study using miltiple regression. The results of this study shows that variation pattern the changes variables of budgeting arrangement participation, the difficulty degree of budgeting objectives and budgeting evaluation explains contribution the influence variable of realization degree of budgeting revenue from land and construction/Pajak Bumi dan Bangunan in the amount of 38,9% (R-square).

Classification of Protein-Binding Sites Using a Spherical Convolutional Neural Network

The analysis and comparison of protein-binding sites aid various applications in the drug discovery process, e.g., hit finding, drug repurposing, and polypharmacology. Classification of binding sites has been a hot topic for the past 30 years, and many different methods have been published. The rapid development of machine learning computational algorithms, coupled with the large volume of publicly available protein–ligand 3D structures, makes it possible to apply deep learning techniques in binding site comparison. Our method uses a cutting-edge spherical convolutional neural network based on the DeepSphere architecture to learn global representations of protein-binding sites. The model was trained on TOUGH-C1 and TOUGH-M1 data and validated with the ProSPECCTs datasets. Our results show that our model can (1) perform well in protein-binding site similarity and classification tasks and (2) learn and separate the physicochemical properties of binding sites. Lastly, we tested the model on a set of kinases, where the results show that it is able to cluster the different kinase subfamilies effectively. This example demonstrates the method’s promise for lead hopping within or outside a protein target, directly based on binding site information

Shape Analysis Using Spectral Geometry

Shape analysis is a fundamental research topic in computer graphics and computer vision. To date, more and more 3D data is produced by those advanced acquisition capture devices, e.g., laser scanners, depth cameras, and CT/MRI scanners. The increasing data demands advanced analysis tools including shape matching, retrieval, deformation, etc. Nevertheless, 3D Shapes are represented with Euclidean transformations such as translation, scaling, and rotation and digital mesh representations are irregularly sampled. The shape can also deform non-linearly and the sampling may vary. In order to address these challenging problems, we investigate Laplace-Beltrami shape spectra from the differential geometry perspective, focusing more on the intrinsic properties. In this dissertation, the shapes are represented with 2 manifolds, which are differentiable. First, we discuss in detail about the salient geometric feature points in the Laplace-Beltrami spectral domain instead of traditional spatial domains. Simultaneously, the local shape descriptor of a feature point is the Laplace-Beltrami spectrum of the spatial region associated to the point, which are stable and distinctive. The salient spectral geometric features are invariant to spatial Euclidean transforms, isometric deformations and mesh triangulations. Both global and partial matching can be achieved with these salient feature points. Next, we introduce a novel method to analyze a set of poses, i.e., near-isometric deformations, of 3D models that are unregistered. Different shapes of poses are transformed from the 3D spatial domain to a geometry spectral one where all near isometric deformations, mesh triangulations and Euclidean transformations are filtered away. Semantic parts of that model are then determined based on the computed geometric properties of all the mapped vertices in the geometry spectral domain while semantic skeleton can be automatically built with joints detected. Finally we prove the shape spectrum is a continuous function to a scale function on the conformal factor of the manifold. The derivatives of the eigenvalues are analytically expressed with those of the scale function. The property applies to both continuous domain and discrete triangle meshes. On the triangle meshes, a spectrum alignment algorithm is developed. Given two closed triangle meshes, the eigenvalues can be aligned from one to the other and the eigenfunction distributions are aligned as well. This extends the shape spectra across non-isometric deformations, supporting a registration-free analysis of general motion data

Spherical correlation as a similarity measure for 3-D radiation patterns of musical instruments

We investigate the use of spherical cross-correlation as a similarity measure of sound radiation patterns, with potential applications for their study, organization, and manipulation. This work is motivated by the application of corpus-based synthesis techniques to spatial projection based on the radiation patterns of orchestral instruments. To this end, we wish to derive spatial descriptors to complement other audio features available for the organization of the sample corpus. Considering two directivity functions on the sphere, their spherical correlation can be computed from their spherical harmonic coefficients. In addition, one can search for the 3-D rotation matrix which maximizes the cross-correlation, i.e. which offers the optimal spherical shape matching. The mathematical foundations of these tools are well established in the literature; however, their practical use in the field of acoustics remains relatively limited and challenging. As a proof of concept, we apply these techniques both to simulated radiation data and to measurements derived from an existing database of 3-D directivity patterns of orchestral instruments. Using these examples we present several test cases to compare the results of spherical correlation to mathematical and acoustical expectations. A range of visualization methods are applied to analyze the test cases, including multi-dimensional scaling, employed as an efficient technique for data reduction and navigation. This article is an extended version of a study previously published in [Carpentier and Einbond. 16th Congrès Français d’Acoustique (CFA), Marseille, France, April 2022, pp. 1–6. https://openaccess.city.ac.uk/id/eprint/28202/]

3D shape matching and registration : a probabilistic perspective

Dense correspondence is a key area in computer vision and medical image analysis. It has applications in registration and shape analysis. In this thesis, we develop a technique to recover dense correspondences between the surfaces of neuroanatomical objects over heterogeneous populations of individuals. We recover dense correspondences based on 3D shape matching. In this thesis, the 3D shape matching problem is formulated under the framework of Markov Random Fields (MRFs). We represent the surfaces of neuroanatomical objects as genus zero voxel-based meshes. The surface meshes are projected into a Markov random field space. The projection carries both geometric and topological information in terms of Gaussian curvature and mesh neighbourhood from the original space to the random field space. Gaussian curvature is projected to the nodes of the MRF, and the mesh neighbourhood structure is projected to the edges. 3D shape matching between two surface meshes is then performed by solving an energy function minimisation problem formulated with MRFs. The outcome of the 3D shape matching is dense point-to-point correspondences. However, the minimisation of the energy function is NP hard. In this thesis, we use belief propagation to perform the probabilistic inference for 3D shape matching. A sparse update loopy belief propagation algorithm adapted to the 3D shape matching is proposed to obtain an approximate global solution for the 3D shape matching problem. The sparse update loopy belief propagation algorithm demonstrates significant efficiency gain compared to standard belief propagation. The computational complexity and convergence property analysis for the sparse update loopy belief propagation algorithm are also conducted in the thesis. We also investigate randomised algorithms to minimise the energy function. In order to enhance the shape matching rate and increase the inlier support set, we propose a novel clamping technique. The clamping technique is realized by combining the loopy belief propagation message updating rule with the feedback from 3D rigid body registration. By using this clamping technique, the correct shape matching rate is increased significantly. Finally, we investigate 3D shape registration techniques based on the 3D shape matching result. Based on the point-to-point dense correspondences obtained from the 3D shape matching, a three-point based transformation estimation technique is combined with the RANdom SAmple Consensus (RANSAC) algorithm to obtain the inlier support set. The global registration approach is purely dependent on point-wise correspondences between two meshed surfaces. It has the advantage that the need for orientation initialisation is eliminated and that all shapes of spherical topology. The comparison of our MRF based 3D registration approach with a state-of-the-art registration algorithm, the first order ellipsoid template, is conducted in the experiments. These show dense correspondence for pairs of hippocampi from two different data sets, each of around 20 60+ year old healthy individuals

Comparing Features of Three-Dimensional Object Models Using Registration Based on Surface Curvature Signatures

This dissertation presents a technique for comparing local shape properties for similar three-dimensional objects represented by meshes. Our novel shape representation, the curvature map, describes shape as a function of surface curvature in the region around a point. A multi-pass approach is applied to the curvature map to detect features at different scales. The feature detection step does not require user input or parameter tuning. We use features ordered by strength, the similarity of pairs of features, and pruning based on geometric consistency to efficiently determine key corresponding locations on the objects. For genus zero objects, the corresponding locations are used to generate a consistent spherical parameterization that defines the point-to-point correspondence used for the final shape comparison

Reconhecimento das configurações de mão da LIBRAS a partir de malhas 3D

Orientador: Prof. Dr. Daniel WeingaertnerDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Curso de Pós-Graduaçao em Informática. Defesa: Curitiba, 13/03/2013Bibliografia: fls. 68-73Resumo: O reconhecimento automático de sinais e um processo importante para uma boa utilização dos meios de comunicacão digitais por deficientes auditivos e, alem disso, favorece a comunicacao entre surdos e ouvintes que nao compreendem a língua de sinais. A abordagem de reconhecimento de sinais utilizada neste trabalho baseia-se nos parâmetros globais da LIBRAS - língua brasileira de sinais: configuracão de mão, locacao ou ponto de articulaçao, movimento, orientacao da palma da mao e expressão facial. A uniao de parâmetros globais forma sinais assim como fonemas formam palavras na língua falada. Este trabalho apresenta uma forma de reconhecer um dos parâmetros globais da LIBRAS, a configuracão de mao, a partir de malhas tridimensionais. A língua brasileira de sinais conta com 61 configuracoes de mao[16], este trabalho fez uso de uma base de dados contendo 610 vídeos de 5 usuarios distintos em duas tomadas, totalizando 10 capturas para cada configuracao de mao. De cada vídeo foram extraídos manualmente dois quadros retratando as visoes frontal e lateral da mao que, após segmentados e pré-processados, foram utilizados como entrada para o processamento de reconstrucao 3D. A geracao da malha 3D a partir das visães frontal e lateral da mão foi feita com o uso da tecnica de reconstruçao por silhueta[7]. O reconhecimento das configuracoes de mao a partir das malhas 3D foi feito com o uso do classificador SVM - Support Vector Machine. As características utilizadas para distinguir as malhas foram obtidas com o metodo Spherical Harmonics[25], um descritor de malhas 3D invariante à rotacao, translacao e escala. Os resultados atingiram uma taxa de acerto media de 98.52% com Ranking 5 demonstrando a eficiencia do metodo.Abstract: Automatic recognition of Sign Language signs is an important process that enhances the quality of use of digital media by hearing impaired people. Additionally, sign recognition enables a way of communication between deaf and hearing people who do not understand Sign Language. The approach of sign recognition used in this work is based on the global parameters of LIBRAS (Brazilian Sign Language): hand configuration, location or point of articulation, movement, palm orientation and facial expression. These parameters are combined to comprise signs, in a similar manner that phonemes are used to form words in spoken (oral) language. This paper presents a way to recognize one of the LIBRAS global parameters, the hand configuration, from 3D meshes. The Brazilian Sign Language has 61 hand configurations [16]. This work made use of a database containing 610 videos of 5 different users signing each hand configuration twice at distinct times, totaling 10 captures for each hand configuration. Two pictures depicting the front and the side views of the hand were manually extracted from each video. These pictures were segmented and pre-processed, after which they were used as input to the 3D reconstruction processing. The generation of the 3D meshes from the front and side images of the hand configuration was done using the Shape from Silhouette technique[7]. The recognition of the hand configurations from the 3D meshes was done with the use of SVM classifier - Support Vector Machine. The characteristics used to distinguish the mesh were obtained using the Spherical Harmonics [25] method: a 3D mesh descriptor that is rotation, translation and scale invariant. Results achieved a hit rate average of 98.52% with Rank 5, demonstrating the efficiency of the method

Learning Equivariant Representations

State-of-the-art deep learning systems often require large amounts of data and computation. For this reason, leveraging known or unknown structure of the data is paramount. Convolutional neural networks (CNNs) are successful examples of this principle, their defining characteristic being the shift-equivariance. By sliding a filter over the input, when the input shifts, the response shifts by the same amount, exploiting the structure of natural images where semantic content is independent of absolute pixel positions. This property is essential to the success of CNNs in audio, image and video recognition tasks. In this thesis, we extend equivariance to other kinds of transformations, such as rotation and scaling. We propose equivariant models for different transformations defined by groups of symmetries. The main contributions are (i) polar transformer networks, achieving equivariance to the group of similarities on the plane, (ii) equivariant multi-view networks, achieving equivariance to the group of symmetries of the icosahedron, (iii) spherical CNNs, achieving equivariance to the continuous 3D rotation group, (iv) cross-domain image embeddings, achieving equivariance to 3D rotations for 2D inputs, and (v) spin-weighted spherical CNNs, generalizing the spherical CNNs and achieving equivariance to 3D rotations for spherical vector fields. Applications include image classification, 3D shape classification and retrieval, panoramic image classification and segmentation, shape alignment and pose estimation. What these models have in common is that they leverage symmetries in the data to reduce sample and model complexity and improve generalization performance. The advantages are more significant on (but not limited to) challenging tasks where data is limited or input perturbations such as arbitrary rotations are present