A Hierarchical Approach to Multimodal Classification

Abstract. Data models that are induced in classifier construction often consists of multiple parts, each of which explains part of the data. Classi-fication methods for such models are called the multimodal classification methods. The model parts may overlap or have insufficient coverage. How to deal best with the problems of overlapping and insufficient cov-erage? In this paper we propose hierarchical or layered approach to this problem. Rather than seeking a single model, we consider a series of models under gradually relaxing conditions, which form a hierarchical structure. To demonstrate the effectiveness of this approach we imple-mented it in two classifiers that construct multi-part models: one based on the so-called lattice machine and the other one based on rough set rule induction. This leads to hierarchical versions of the classifiers. The classification performance of these two hierarchical classifiers is compared with C4.5, Support Vector Machine (SVM), rule based classifiers (with the optimisation of rule shortening) implemented in Rough Set Explo-ration System (RSES), and a method combining k-nn with rough set rule induction (RIONA in RSES). The results of the experiments show that this hierarchical approach leads to improved multimodal classifiers

Effective identification of terrain positions from gridded DEM data using multimodal classification integration

Terrain positions are widely used to describe the Earth’s topographic features and play an important role in the studies of landform evolution, soil erosion and hydrological modeling. This work develops a new multimodal classification system with enhanced classification performance by integrating different approaches for terrain position identification. The adopted classification approaches include local terrain attribute (LA)-based and regional terrain attribute (RA)-based, rule-based and supervised, and pixel-based and object-oriented methods. Firstly, a double-level definition scheme is presented for terrain positions. Then, utilizing a hierarchical framework, a multimodal approach is developed by integrating different classification techniques. Finally, an assessment method is established to evaluate the new classification system from different aspects. The experimental results, obtained at a Loess Plateau region in northern China on a 5 m digital elevation model (DEM), show reasonably positional relationship, and larger inter-class and smaller intra-class variances. This indicates that identified terrain positions are consistent with the actual topography from both overall and local perspectives, and have relatively good integrity and rationality. This study demonstrates that the current multimodal classification system, developed by taking advantage of various classification methods, can reflect the geographic meanings and topographic features of terrain positions from different levels

Toward a New Approach in Fruit Recognition using Hybrid RGBD Features and Fruit Hierarchy Property

We present hierarchical multi-feature classification (HMC) system for multiclass fruit recognition problem. Our approach to HMC exploits the advantages of combining multimodal features  and  the  fruit  hierarchy  property.  In  the construction of hybrid features, we take the advantage of using color feature in the fruit recognition problem and combine it with 3D shape feature of depth channel of RGBD (Red, Green, Blue, Depth) images. Meanwhile, given a set of fruit species and variety, with a preexisting hierarchy among them, we consider the problem of assigning images to one of these fruit variety from the point of view of a hierarchy. We report on computational experiment using this approach. We show that the use of hierarchy structure along with hybrid RGBD features can improve the classification performance

Multimodal Multipart Learning for Action Recognition in Depth Videos

The articulated and complex nature of human actions makes the task of action recognition difficult. One approach to handle this complexity is dividing it to the kinetics of body parts and analyzing the actions based on these partial descriptors. We propose a joint sparse regression based learning method which utilizes the structured sparsity to model each action as a combination of multimodal features from a sparse set of body parts. To represent dynamics and appearance of parts, we employ a heterogeneous set of depth and skeleton based features. The proper structure of multimodal multipart features are formulated into the learning framework via the proposed hierarchical mixed norm, to regularize the structured features of each part and to apply sparsity between them, in favor of a group feature selection. Our experimental results expose the effectiveness of the proposed learning method in which it outperforms other methods in all three tested datasets while saturating one of them by achieving perfect accuracy

SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model

To realize human-like robot intelligence, a large-scale cognitive architecture is required for robots to understand the environment through a variety of sensors with which they are equipped. In this paper, we propose a novel framework named Serket that enables the construction of a large-scale generative model and its inference easily by connecting sub-modules to allow the robots to acquire various capabilities through interaction with their environments and others. We consider that large-scale cognitive models can be constructed by connecting smaller fundamental models hierarchically while maintaining their programmatic independence. Moreover, connected modules are dependent on each other, and parameters are required to be optimized as a whole. Conventionally, the equations for parameter estimation have to be derived and implemented depending on the models. However, it becomes harder to derive and implement those of a larger scale model. To solve these problems, in this paper, we propose a method for parameter estimation by communicating the minimal parameters between various modules while maintaining their programmatic independence. Therefore, Serket makes it easy to construct large-scale models and estimate their parameters via the connection of modules. Experimental results demonstrated that the model can be constructed by connecting modules, the parameters can be optimized as a whole, and they are comparable with the original models that we have proposed