An Extended Result on the Optimal Estimation under Minimum Error Entropy Criterion

The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion is the Renyi entropy or equivalently, the \alpha-order information potential (IP).Comment: 15 pages, no figures, submitted to Entrop

An immune-inspired, dependence-based approach to blind inversion of wiener systems

Dissertação (mestrado)—Universidade de Brasília, Faculdade de Tecnologia, Departamento de Engenharia Elérica, 2016.Nas últimas décadas, o estudo de métodos para a inversão cega de sistemas de Wiener tem recebido uma atenção signi cativa, especialmente em áreas como a biologia, química, sociologia e na indústria. Um grande número de métodos tem sido desenvolvidos com diferentes abordagens e análises teóricas do problema, que incluem algoritmos de gradiente para minimizar a taxa de informação mútua do sinal extraído, algoritmos genéticos para executar a tarefa de procurar os parâmetros ótimos assim como algoritmos imuno-inspirados. Estes métodos têm como requisito fundamental que o sinal de entrada seja originalmente i.i.d., além de algumas outras condições de suavidade. Cenários de aplicação que cumprem com este requisito podem ser difíceis de ocorrer, na prática; por isso, considerar fontes não-independentes tem se tornado uma importante abordagem. Neste trabalho, propõem-se dois métodos baseados nas funções de autocorrelação e autocorrentropia para explorar a estrutura do tempo de um determinado sinal, com a nalidade de promover a inversão cega dos sistemas de Wiener usando sistemas Hammerstein. Filtros lineares com e sem realimentação são considerados e um algoritmo imuno-inspirado é usado para permitir a otimização de parâmetros sem a necessidade de manipular analiticamente a função custo, ao mesmo tempo que se aumenta a probabilidade de convergência global. Os resultados experimentais indicam que ambas as funções proporcionam meios e cazes para a inversão do sistema e também ilustram o efeito de realimentação linear sobre o desempenho global do sistema.In the last decades, the study of blind inversion of Wiener systems has received signi cant attention, in a special manner in areas such as biology, chemistry, sociology, psychology and industry. A large number of methods have been developed with di erent approaches and theoretical analysis of the problem, which include a gradient algorithm to minimize the mutual information rate of the extracted signal, genetic algorithms to perform the task of searching for the optimal parameters as well as immune-inspired algorithms. These methods have the particular requirement that the input signal must be i.i.d. and, besides some smoothness conditions. This requirement may be hard to be present in real-world problems, hence, considering non-independent sources have become an interesting approach. In this work, we propose two methods based on the autocorrelation and autocorrentropy functions for representing the time structure of a given signal, in order to cope with the unsupervised inversion of Wiener systems by Hammerstein systems. Linear lters with and without feedback are considered and an immune-inspired algorithm is used to allow parameter optimization without the need for explicitly manipulating the cost function, with the additional bene t of increasing the probability of global convergence. The experimental results indicate that both functions provide e ective means for system inversion and also illustrate the e ect of linear feedback on the overall system performance

A Novel Nonparametric Distance Estimator for Densities with Error Bounds

The use of a metric to assess distance between probability densities is an important practical problem. In this work, a particular metric induced by an a-divergence is studied. The Hellinger metric can be interpreted as a particular case within the framework of generalized Tsallis divergences and entropies. The nonparametric Parzen's density estimator emerges as a natural candidate to estimate the underlying probability density function, since it may account for data from different groups, or experiments with distinct instrumental precisions, i.e., non-independent and identically distributed (non-i.i.d.) data. However, the information theoretic derived metric of the nonparametric Parzen's density estimator displays infinite variance, limiting the direct use of resampling estimators. Based on measure theory, we present a change of measure to build a finite variance density allowing the use of resampling estimators. In order to counteract the poor scaling with dimension, we propose a new nonparametric two-stage robust resampling estimator of Hellinger's metric error bounds for heterocedastic data. The approach presents very promising results allowing the use of different covariances for different clusters with impact on the distance evaluation

Adaptive neural network cascade control system with entropy-based design

A neural network (NN) based cascade control system is developed, in which the primary PID controller is constructed by NN. A new entropy-based measure, named the centred error entropy (CEE) index, which is a weighted combination of the error cross correntropy (ECC) criterion and the error entropy criterion (EEC), is proposed to tune the NN-PID controller. The purpose of introducing CEE in controller design is to ensure that the uncertainty in the tracking error is minimised and also the peak value of the error probability density function (PDF) being controlled towards zero. The NN-controller design based on this new performance function is developed and the convergent conditions are. During the control process, the CEE index is estimated by a Gaussian kernel function. Adaptive rules are developed to update the kernel size in order to achieve more accurate estimation of the CEE index. This NN cascade control approach is applied to superheated steam temperature control of a simulated power plant system, from which the effectiveness and strength of the proposed strategy are discussed by comparison with NN-PID controllers tuned with EEC and ECC criterions

Sensory fusion applied to power system state estimation considering information theory concepts

With the increasing integration of synchronized phasor measurement units (PMU) in power grids appears the necessity to create methods capable to merge the information obtained from different classes of sensors, namely PMU and the conventional sensors already integrated in the SCADA system. Thus, this dissertation proposes a sensory fusion that guarantees the previous requirement. Beyond that, this thesis proposes an application of concepts related with Information theory to the state estimation problem. This application aims to propose a robust state estimator without the necessity of previous treatment of the acquired data

Empirical data analytics

In this paper, we propose an approach to data analysis, which is based entirely on the empirical observations of discrete data samples and the relative proximity of these points in the data space. At the core of the proposed new approach is the typicality—an empirically derived quantity that resembles probability. This nonparametric measure is a normalized form of the square centrality (centrality is a measure of closeness used in graph theory). It is also closely linked to the cumulative proximity and eccentricity (a measure of the tail of the distributions that is very useful for anomaly detection and analysis of extreme values). In this paper, we introduce and study two types of typicality, namely its local and global versions. The local typicality resembles the well-known probability density function (pdf), probability mass function, and fuzzy set membership but differs from all of them. The global typicality, on the other hand, resembles well-known histograms but also differs from them. A distinctive feature of the proposed new approach, empirical data analysis (EDA), is that it is not limited by restrictive impractical prior assumptions about the data generation model as the traditional probability theory and statistical learning approaches are. Moreover, it does not require an explicit and binary assumption of either randomness or determinism of the empirically observed data, their independence, or even their number (it can be as low as a couple of data samples). The typicality is considered as a fundamental quantity in the pattern analysis, which is derived directly from data and is stated in a discrete form in contrast to the traditional approach where a continuous pdf is assumed a priori and estimated from data afterward. The typicality introduced in this paper is free from the paradoxes of the pdf. Typicality is objectivist while the fuzzy sets and the belief-based branch of the probability theory are subjectivist. The local typicality is expressed in a closed analytical form and can be calculated recursively, thus, computationally very efficiently. The other nonparametric ensemble properties of the data introduced and studied in this paper, namely, the square centrality, cumulative proximity, and eccentricity, can also be updated recursively for various types of distance metrics. Finally, a new type of classifier called naïve typicality-based EDA class is introduced, which is based on the newly introduced global typicality. This is only one of the wide range of possible applications of EDA including but not limited for anomaly detection, clustering, classification, control, prediction, control, rare events analysis, etc., which will be the subject of further research