Inductive reasoning and Kolmogorov complexity

AbstractReasoning to obtain the “truth” about reality, from external data, is an important, controversial, and complicated issue in man's effort to understand nature. (Yet, today, we try to make machines do this.) There have been old useful principles, new exciting models, and intricate theories scattered in vastly different areas including philosophy of science, statistics, computer science, and psychology. We focus on inductive reasoning in correspondence with ideas of R. J. Solomonoff. While his proposals result in perfect procedures, they involve the noncomputable notion of Kolmogorov complexity. In this paper we develop the thesis that Solomonoff's method is fundamental in the sense that many other induction principles can be viewed as particular ways to obtain computable approximations to it. We demonstrate this explicitly in the cases of Gold's paradigm for inductive inference, Rissanen's minimum description length (MDL) principle, Fisher's maximum likelihood principle, and Jaynes' maximum entropy principle. We present several new theorems and derivations to this effect. We also delimit what can be learned and what cannot be learned in terms of Kolmogorov complexity, and we describe an experiment in machine learning of handwritten characters. We also give an application of Kolmogorov complexity in Valiant style learning, where we want to learn a concept probably approximately correct in feasible time and examples

Conjoint probabilistic subband modeling

Thesis (Ph. D.)--Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1997.Includes bibliographical references (leaves 125-133).by Ashok Chhabedia Popat.Ph.D

Probabilistic Independence Networks for Hidden Markov Probability Models

Graphical techniques for modeling the dependencies of randomvariables have been explored in a variety of different areas includingstatistics, statistical physics, artificial intelligence, speech recognition, image processing, and genetics.Formalisms for manipulating these models have been developedrelatively independently in these research communities. In this paper weexplore hidden Markov models (HMMs) and related structures within the general framework of probabilistic independencenetworks (PINs). The paper contains a self-contained review of the basic principles of PINs.It is shown that the well-known forward-backward (F-B) and Viterbialgorithms for HMMs are special cases of more general inference algorithms forarbitrary PINs. Furthermore, the existence of inference and estimationalgorithms for more general graphical models provides a set of analysistools for HMM practitioners who wish to explore a richer class of HMMstructures.Examples of relatively complex models to handle sensorfusion and coarticulationin speech recognitionare introduced and treated within the graphical model framework toillustrate the advantages of the general approach

Minimum description complexity

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2002.Includes bibliographical references (p. 136-140).The classical problem of model selection among parametric model sets is considered. The goal is to choose a model set which best represents observed data. The critical task is the choice of a criterion for model set comparison. Pioneer information theoretic based approaches to this problem are Akaike information criterion (AIC) and different forms of minimum description length (MDL). The prior assumption in these methods is that the unknown true model is a member of all the competing sets. We introduce a new method of model selection: minimum description complexity (MDC). The approach is motivated by the Kullback-Leibler information distance. The method suggests choosing the model set for which the model set relative entropy is minimum. We provide a probabilistic method of MDC estimation for a class of parametric model sets. In this calculation the key factor is our prior assumption: unlike the existing methods, no assumption of the true model being a member of the competing model sets is needed. The main strength of the MDC calculation is in its method of extracting information from the observed data.(cont.) Interesting results exhibit the advantages of MDC over MDL and AIC both theoretically and practically. It is illustrated that, under particular conditions, AIC is a special case of MDC. Application of MDC in system identification and signal denoising is investigated. The proposed method answers the challenging question of quality evaluation in identification of stable LTI systems under a fair prior assumption on the unmodeled dynamics. MDC also provides a new solution to a class of denoising problems. We elaborate the theoretical superiority of MDC over the existing thresholding denoising methods.by Soosan Beheshti.Ph.D

Empirical state space modelling with application in online diagnosis of multivariate non-linear dynamic systems

Dissertation (Ph.D)--University of Stellenbosch, 1999.ENGLISH ABSTRACT: System identification has been sufficiently formalized for linear systems, but not for empirical identification of non-linear, multivariate dynamic systems. Therefore this dissertation formalizes and extends non-linear empirical system identification for the broad class of nonlinear multivariate systems that can be parameterized as state space systems. The established, but rather ad hoc methods of time series embedding and nonlinear modeling, using multilayer perceptron network and radial basis function network model structures, are interpreted in context with the established linear system identification framework. First, the methodological framework was formulated for the identification of non-linear state space systems from one-dimensional time series using a surrogate data method. It was clearly demonstrated on an autocatalytic process in a continuously stirred tank reactor, that validation of dynamic models by one-step predictions is insufficient proof of model quality. In addition, the classification of data as either dynamic or random was performed, using the same surrogate data technique. The classification technique proved to be robust in the presence of up to at least 10% measurement and dynamic noise. Next, the formulation of a nearly real-time algorithm for detection and removal of radial outliers in multidimensional data was pursued. A convex hull technique was proposed and demonstrated on random data, as well as real test data recorded from an internal combustion engine. The results showed the convex hull technique to be effective at a computational cost two orders of magnitude lower than the more proficient Rocke and Woodruff technique, used as a benchmark, and incurred low cost (0.9%) in terms of falsely identifying outliers. Following the identification of systems from one-dimensional time series, the methodological framework was expanded to accommodate the identification of nonlinear state space systems from multivariate time series. System parameterization was accomplished by combining individual embeddings of each variable in the multivariate time series, and then separating this combined space into independent components, using independent component analysis. This method of parameterization was successfully applied in the simulation of the abovementioned autocatalytic process. In addition, the parameterization method was implemented in the one-step prediction of atmospheric N02 concentrations, which could become part of an environmental control system for Cape Town. Furthermore, the combination of the embedding strategy and separation by independent component analysis was able to isolate some of the noise components from the embedded data. Finally the foregoing system identification methodology was applied to the online diagnosis of temporal trends in critical system states. The methodology was supplemented by the formulation of a statistical likelihood criterion for simultaneous interpretation of multivariate system states. This technology was successfully applied to the diagnosis of the temporal deterioration of the piston rings in a compression ignition engine under test conditions. The diagnostic results indicated the beginning of significant piston ring wear, which was confirmed by physical inspection of the engine after conclusion of the test. The technology will be further developed and commercialized.AFRIKAANSE OPSOMMING: Stelselidentifikasie is weI genoegsaam ten opsigte van lineere stelsels geformaliseer, maar nie ten opsigte van die identifikasie van nie-lineere, multiveranderlike stelsels nie. In hierdie tesis word nie-lineere, empiriese stelselidentifikasie gevolglik ten opsigte van die wye klas van nielineere, multiveranderlike stelsels, wat geparameteriseer kan word as toestandveranderlike stelsels, geformaliseer en uitgebrei. Die gevestigde, maar betreklik ad hoc metodes vir tydreeksontvouing en nie-lineere modellering (met behulp van multilaag-perseptron- en radiaalbasisfunksie-modelstrukture) word in konteks met die gevestigde line ere stelselidentifikasieraamwerk vertolk. Eerstens is die metodologiese raamwerk vir die identifikasie van nie-lineere, toestandsveranderlike stelsels uit eendimensionele tydreekse met behulp van In surrogaatdatametode geformuleer. Daar is duidelik by wyse van 'n outokatalitiese proses in 'n deurlopend geroerde tenkreaktor getoon dat die bevestiging van dinamiese modelle deur middel van enkelstapvoorspellings onvoldoende bewys van die kwaliteit van die modelle is. Bykomend is die klassifikasie van tydreekse as 6f dinamies Of willekeurig, met behulp van dieselfde surrogaattegniek gedoen. Die klassifikasietegniek het in die teenwoordigheid van tot minstens 10% meetgeraas en dinamiese geraas robuust vertoon. / Vervolgens is die formulering van In bykans intydse algoritme vir die opspoor en verwydering van radiale uitskieters in multiveranderlike data aangepak. 'n Konvekse hulstegniek is V:oorgestel en op ewekansige data, sowel as op werklike toetsdata wat van 'n binnebrandenjin opgeneem is, gedemonstreer. Volgens die resultate was die konvekse hulstegniek effektief teen 'n rekenkoste twee grootte-ordes kleiner as die meer vermoende Rocke en Woodrufftegniek, wat as meetstandaard beskou is. Die konvekse hulstegniek het ook 'n lae loopkoste (0.9%) betreffende die valse identifisering van uitskieters behaal. Na aanleiding van die identifisering van stelsels uit eendimensionele tydreekse, is die metodologiese raamwerk uitgebiei om die identifikasie van nie-lineere, toestandsveranderlike stelsels uit multiveranderlike data te omvat. Stelselparameterisering is bereik deur individuele ontvouings van elke veranderlike in die multidimensionele tydreeks met die skeiding van die gesamenlike ontvouingsruimte tot onafhanklike komponente saam te span. Sodanige skeiding is deur middel van onafhanklike komponentanalise behaal. Hierdie metode van parameterisering is suksesvc1 op die simulering van bogenoemde outokatalitiese proses toegepas. Die parameteriseringsmetode is bykomend in die enkelstapvoorspelling van atmosferiese N02-konsentrasies ingespan en sal moontlik deel van 'n voorgestelde omgewingsbestuurstelsel vir Kaapstad uitmaak. Die kombinasie van die ontvouingstrategie en skeiding deur onafhanklike komponentanalise was verder ook in staat om van die geraaskomponente in die data uit te lig. Ten slotte is die voorafgaande tegnologie vir stelselidentifikasie op die lopende diagnose van tydsgebonde neigings in kritiese stelseltoestande toegepas. Die metodologie is met die formulering van 'n statistiese waarskynlikheidsmaatstaf vir die gelyktydige vertolking van multiveranderlike stelseltoestande aangevul. Hierdie tegnologie is suksesvol op die diagnose van die tydsgebonde verswakking van die suierringe in 'n kompressieontstekingenj in tydens toetstoestande toegepas. Die diagnostiese resultate het die aanvang van beduidende slytasie in die suierringe aangedui, wat later tydens fisiese inspeksie van die enjin met afloop van die toets, bevestig is. Die tegnologie sal verder ontwikkel en markgereed gemaak word

Minimum description length revisited

This is an up-to-date introduction to and overview of the Minimum Description Length (MDL) Principle, a theory of inductive inference that can be applied to general problems in statistics, machine learning and pattern recognition. While MDL was originally based on data compression ideas, this introduction can be read without any knowledge thereof. It takes into account all major developments since 2007, the last time an extensive overview was written. These include new methods for model selection and averaging and hypothesis testing, as well as the first completely general definition of MDL estimators. Incorporating these developments, MDL can be seen as a powerful extension of both penalized likelihood and Bayesian approaches, in which penalization functions and prior distributions are replaced by more general luckiness functions, average-case methodology is replaced by a more robust worst-case approach, and in which methods classically viewed as highly distinct, such as AIC versus BIC and cross-validation versus Bayes can, to a large extent, be viewed from a unified perspective.Peer reviewe