Accurate Computation of the Relative Entropy Between Stochastic Regular Grammars

Works dealing with grammatical inference of stochastic grammars often evaluate the relative entropy between the model and the true grammar by means of large test sets generated with the true distribution. In this paper, an iterative procedure to compute the relative entropy between two stochastic deterministic regular grammars is proposed. Resum'e Les travails sur l'inf'erence de grammaires stochastiques 'evaluent l'entropie relative entre le mod`ele et la vraie grammaire en utilisant grands ensembles de test g'en'er'es avec la distribution correcte. Dans cet article, on propose une proc'edure it'erative pour calculer l'entropie relative entre deux grammaires. 1 Introduction Stochastic models have been widely used in computer science, especially in those tasks dealing with noisy data or random sources such as pattern recognition, natural language modeling, etc. A stochastic model predicts a probability distribution for the events in the class under consideration and one of the mo..

Accurate Computation of the Relative Entropy Between Stochastic Regular Grammars

Works dealing with grammatical inference of stochastic grammars often evaluate the relative entropy between the model and the true grammar by means of large test sets generated with the true distribution. In this paper, an iterative procedure to compute the relative entropy between two stochastic deterministic regular grammars is proposed. Resum'e Les travails sur l'inf'erence de grammaires stochastiques 'evaluent l'entropie relative entre le mod`ele et la vraie grammaire en utilisant grands ensembles de test g'en'er'es avec la distribution correcte. Dans cet article, on propose une proc'edure it'erative pour calculer l'entropie relative entre deux grammaires. 1 Introduction Stochastic models have been widely used in computer science, especially in those tasks dealing with noisy data or random sources such as pattern recognition, natural language modeling, etc. A stochastic model predicts a probability distribution for the events in the class under consideration and one of the mo..

In Language and Information Technologies

With the rising amount of available multilingual text data, computational linguistics faces an opportunity and a challenge. This text can enrich the domains of NLP applications and improve their performance. Traditional supervised learning for this kind of data would require annotation of part of this text for induction of natural language structure. For these large amounts of rich text, such an annotation task can be daunting and expensive. Unsupervised learning of natural language structure can compensate for the need for such annotation. Natural language structure can be modeled through the use of probabilistic grammars, generative statistical models which are useful for compositional and sequential structures. Probabilistic grammars are widely used in natural language processing, but they are also used in other fields as well, such as computer vision, computational biology and cognitive science. This dissertation focuses on presenting a theoretical and an empirical analysis for the learning of these widely used grammars in the unsupervised setting. We analyze computational properties involved in estimation of probabilisti