Towards On-line Domain-Independent Big Data Learning: Novel Theories and Applications

Feature extraction is an extremely important pre-processing step to pattern recognition, and machine learning problems. This thesis highlights how one can best extract features from the data in an exhaustively online and purely adaptive manner. The solution to this problem is given for both labeled and unlabeled datasets, by presenting a number of novel on-line learning approaches. Specifically, the differential equation method for solving the generalized eigenvalue problem is used to derive a number of novel machine learning and feature extraction algorithms. The incremental eigen-solution method is used to derive a novel incremental extension of linear discriminant analysis (LDA). Further the proposed incremental version is combined with extreme learning machine (ELM) in which the ELM is used as a preprocessor before learning. In this first key contribution, the dynamic random expansion characteristic of ELM is combined with the proposed incremental LDA technique, and shown to offer a significant improvement in maximizing the discrimination between points in two different classes, while minimizing the distance within each class, in comparison with other standard state-of-the-art incremental and batch techniques. In the second contribution, the differential equation method for solving the generalized eigenvalue problem is used to derive a novel state-of-the-art purely incremental version of slow feature analysis (SLA) algorithm, termed the generalized eigenvalue based slow feature analysis (GENEIGSFA) technique. Further the time series expansion of echo state network (ESN) and radial basis functions (EBF) are used as a pre-processor before learning. In addition, the higher order derivatives are used as a smoothing constraint in the output signal. Finally, an online extension of the generalized eigenvalue problem, derived from James Stone’s criterion, is tested, evaluated and compared with the standard batch version of the slow feature analysis technique, to demonstrate its comparative effectiveness. In the third contribution, light-weight extensions of the statistical technique known as canonical correlation analysis (CCA) for both twinned and multiple data streams, are derived by using the same existing method of solving the generalized eigenvalue problem. Further the proposed method is enhanced by maximizing the covariance between data streams while simultaneously maximizing the rate of change of variances within each data stream. A recurrent set of connections used by ESN are used as a pre-processor between the inputs and the canonical projections in order to capture shared temporal information in two or more data streams. A solution to the problem of identifying a low dimensional manifold on a high dimensional dataspace is then presented in an incremental and adaptive manner. Finally, an online locally optimized extension of Laplacian Eigenmaps is derived termed the generalized incremental laplacian eigenmaps technique (GENILE). Apart from exploiting the benefit of the incremental nature of the proposed manifold based dimensionality reduction technique, most of the time the projections produced by this method are shown to produce a better classification accuracy in comparison with standard batch versions of these techniques - on both artificial and real datasets

Discriminant feature extraction by generalized difference subspace

This paper reveals the discriminant ability of the orthogonal projection of data onto a generalized difference subspace (GDS) both theoretically and experimentally. In our previous work, we have demonstrated that GDS projection works as the quasi-orthogonalization of class subspaces. Interestingly, GDS projection also works as a discriminant feature extraction through a similar mechanism to the Fisher discriminant analysis (FDA). A direct proof of the connection between GDS projection and FDA is difficult due to the significant difference in their formulations. To avoid the difficulty, we first introduce geometrical Fisher discriminant analysis (gFDA) based on a simplified Fisher criterion. gFDA can work stably even under few samples, bypassing the small sample size (SSS) problem of FDA. Next, we prove that gFDA is equivalent to GDS projection with a small correction term. This equivalence ensures GDS projection to inherit the discriminant ability from FDA via gFDA. Furthermore, we discuss two useful extensions of these methods, 1) nonlinear extension by kernel trick, 2) the combination of convolutional neural network (CNN) features. The equivalence and the effectiveness of the extensions have been verified through extensive experiments on the extended Yale B+, CMU face database, ALOI, ETH80, MNIST and CIFAR10, focusing on the SSS problem

Gender discrimination and prediction on the basis of facial metric information

AbstractHorizontal and vertical facial measurements are statistically independent. Discriminant analysis shows that five of such normalized distances explain over 95% of the gender differences of “training” samples and predict the gender of 90% novel test faces exhibiting various facial expressions. The robustness of the method and its results are assessed. It is argued that these distances (termed fiducial) are compatible with those found experimentally by psychophysical and neurophysiological studies. In consequence, partial explanations for the effects observed in these experiments can be found in the intrinsic statistical nature of the facial stimuli used

Reproductive Isolating Mechanisms and Communication in Greater Prairie Chickens (Tympanuchus Cupido) and Sharp-Tailed Grouse (Pedioecetes Phasianellus)

Sympatric populations of greater prairie chickens (Tympanuchus cupido) and sharp-tailed grouse (Pedioecetes phasianel lus) were studied between 1975 and 1978 in northwestern Minnesota for the pur poses of identifying and determining the strength of reproductive isolating mechanisms between them. Major emphasis was placed on ethological mechanisms but other factors were also examined. Approximately a fifth of the sharptail’s distribution overlaps half of the prairie chicken’s within North America. Within the zone of sympatry, prairie chicken populations are in small patches and may be liable to hybrid swarming, particularly with changes in land-use practices. The rate cf hybridization is around 1% but may be much greater in specific areas. Within the Minnesota study area, hybridization increased from 1-3.7% as the ratio between numbers of prairie chickens and sharptails increased. Habitat preferences and seasonal and daily patterns of activity were sufficiently similar between species to allow complete intermixing. Breeding experiments conducted in captivity showed that hybrids and backcrosses were interfertile. Thus, non-communica- tory mechanisms were weak or non-existent. Agonistic displays, including forward displays, face offs and stand offs were similar between SDecies and probably facilitated spacing as all males held interspecifically exclusive territories on mixed displa.- grounds. Most epigamic behaviors such as booming displays, whoops, dancing and chi Iks were polyvalent and had many species-specific characteristics. Discriminant analysis and canonical correlation analysis were used to show that whoops and chi Iks, which were mostly epigamic, were most different between species; whines, which were polyvalent, were more similar; and cackles, the most aggressive of the 3 sets of vocalizations, were most similar between species. Interrelationships of homologous displays, as determined by cluster analyses, and by temporal occurrence of common displays, were not sufficiently different to be effective isolating mechanisms. Displays of hybrids were intermediate in form between both parental species and may have repulsed females. Intraspecific playback experiments of vocalizations showed that prairie chicken booms, whoops and composite calls (consisting of a segment of a recording made while a prairie chicken hen visited a display ground) had agonistic functions. Prairie chicken whines also elicited significant responses but this call appeared to serve as an alarm. Sharptail males responded aggressively to gobbles, coos, cackles and alert. The functions of dancing in intermale communication were unclear. Analyses of activity rates showed that males of both species responded .tore vigorously to live and taxidermist mounts of conspe- cific hens than to heterospecifics. Male prairie chickens, unlike sharptails, frequently courted live heterospecific females, even if doing so led to fights. Prairie chicken males reacted aggressively to sharptail coos, gobbles, cork notes and composite sounds while sharp-tails only responded to prairie chicken cackles. Increased selectivity of sharptails for nonspecific stimuli may be due to greater historical contact with confamilials and a resulting channelization of reproduc tive and aggressive energies to meet intraspecific competition. During experiments conducted in captivity, females of both species strongly preferred conspecific territories and males despite being raised in mixed-species groups from hatching. FI hybrid and backcross females were more ambiguous but may have preferred sharp- tail males. Precise factors determining mate choice by females were unidentified but appeared to relate to possession of a territory and to behaviors of males. Of the mechanisms studied, behavior, particularly communication, seemed to be most important in maintaining species integrity between greater prairie chickens and sharp-tailed grouse. Based on similari ties in the displays of the two species and cn apparent fertility of hybrids, both grouse should be considered congeneric under Tympanuchus