Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion

Abstract. The paper introduces a new framework for feature learning in classification motivated by information theory. We first systematically study the information structure and present a novel perspective revealing the two key factors in information utilization: class-relevance and redun-dancy. We derive a new information decomposition model where a novel concept called class-relevant redundancy is introduced. Subsequently a new algorithm called Conditional Informative Feature Extraction is for-mulated, which maximizes the joint class-relevant information by explic-itly reducing the class-relevant redundancies among features. To address the computational difficulties in information-based optimization, we in-corporate Parzen window estimation into the discrete approximation of the objective function and propose a Local Active Region method which substantially increases the optimization efficiency. To effectively utilize the extracted feature set, we propose a Bayesian MAP formulation for feature fusion, which unifies Laplacian Sparse Prior and Multivariate Logistic Regression to learn a fusion rule with good generalization ca-pability. Realizing the inefficiency caused by separate treatment of the extraction stage and the fusion stage, we further develop an improved design of the framework to coordinate the two stages by introducing a feedback from the fusion stage to the extraction stage, which signifi-cantly enhances the learning efficiency. The results of the comparative experiments show remarkable improvements achieved by our framework.

On the accuracy of statistical pattern recognizers

Applied Science

Pattern Recognition as a Human Centered non-Euclidean Problem

Regularities in the world are human defined. Patterns in the observed phenomena are there because we define and recognize them as such. Automatic pattern recognition tries to bridge the gap between human judgment and measurements made by artificial sensors. This is done in two steps: representation and generalization. Traditional representations of real world objects to be recognized, like features and pixels, either neglect possibly significant aspects of the objects, or neglect their dependencies. We therefor reconsider human recognition and observe that it is based on our direct experience of similarity or dissimilarity of objects. Using these concepts, a pattern recognition system can be defined in a natural way by a pairwise comparison of objects. This results in the dissimilarity representation for pattern recognition. An analysis of dissimilarity measures optimized for performance shows that they tend to be non-Euclidean. The Euclidean vector spaces, traditionally used in pattern recognition and machine learning may thereby be suboptimal. The causes and consequences of the use of non-Euclidean representations will be discussed. It is conjectured that human judgment of object differences result in these non-Euclidean representations as object structure is taken into account.MediamaticsElectrical Engineering, Mathematics and Computer Scienc

The Origin of Patterns

The question is discussed from where the patterns arise that are recognized in the world. Are they elements of the outside world, or do they originate from the concepts that live in the mind of the observer? It is argued that they are created during observation, due to the knowledge on which the observation ability is based. For an experienced observer this may result in a direct recognition of an object or phenomenon without any reasoning. Afterwards and using conscious effort he may be able to supply features or arguments that he might have used for his recognition. The discussion is phrased in the philosophical debate between monism, in which the observer is an element of the observed world, and dualism, in which these two are fully separated. Direct recognition can be understood from a monistic point of view. After the definition of features and the formulation of a reasoning, dualism may arise. An artificial pattern recognition system based on these specifications thereby creates a clear dualistic situation. It fully separates the two worlds by physical sensors and mechanical reasoning. This dualistic position can be solved by a responsible integration of artificially intelligent systems in human controlled applications. A set of simple experiments based on the classification of histopathological slides is presented to illustrate the discussion.Pattern Recognition and Bioinformatic