Contains fulltext :
                  227633.pdf (publisher's version ) (Open Access

Hiemstra, Djoerd

English

Hiemstra, D.

Radboud Repository

Probability Smoothing

Djoerd Hiemstra

Crossref

University of Twente Research Information

PROBABILITY SMOOTHINGDjoerd HiemstraUniversity of Twentehttp://www.cs.utwente.nl/∼hiemstraDEFINITIONProbability smoothing is a language modeling technique that assigns some non-zero probability to events thatwere unseen in the training data. This has the effect that the probability mass is divided over more events, hencethe probability distribution becomes more smooth.MAIN TEXTSmoothing overcomes the so-called sparse data problem, that is, many events that are plausible in reality are notfound in the data used to estimate probabilities. When using maximum likelihood estimates, unseen events areassigned zero probability. In case of information retrieval, most events are unseen in the data, even if simpleunigram language models are used (see N-GRAM MODELS): Documents are relatively short (say on averageseveral hundreds of words), whereas the vocabulary is typically big (maybe millions of words), so the vast majorityof words does not occur in the document. A small document about “information retrieval” might not mention theword “search”, but that does not mean it is not relevant to the query “text search”. The sparse data problem isthe reason that it is hard for information retrieval systems to obtain high recall values without degrading valuesfor precision, and smoothing is a means to increase recall (possibly degrading precision in the process). Manyapproaches to smoothing are proposed in the field of automatic speech recognition [1]. A smoothing methodmay be as simple so-called Laplace smoothing, which adds an extra count to every possible word. The followingequations show respectively (8) the unsmoothed, or maximum likelihood estimate, (9) Laplace smoothing, (10)Linear interpolation smoothing, and (11) Dirichlet smoothing [3]:PML(T = t|D=d) = tf (t, d) /�t′tf (t′, d)(8)PLP (T = t|D=d) = (tf (t, d) + 1) /�t′(tf (t′, d) + 1)(9)PLI (T = t|D=d) = λPML(T = t|D=d) + (1−λ)PML(T = t|C)(10)PDi(T = t|D=d) = (tf (t, d) + µPML(T = t|C)) / ((�t′tf (t′, d)) + µ)(11)Here, tf (t, d) is the frequency of occurrence of the term t in the document d, and PML(T |C) is the probabilityof a term occurring in the entire collection C. Both linear interpolation smoothing (see also the entryLANGUAGE MODELS) and Dirichlet smoothing assign a probability proportional to the term occurrence inthe collection to unseen terms. Here, λ (0 < λ < 1) and µ (µ > 0) are unknown parameters that should betuned to optimize retrieval effectiveness. Linear interpolation smoothing has the same effect on all documents,whereas Dirichlet smoothing has a relatively big effect on small documents, but a relatively small effect on biggerdocuments. Many smoothed estimators used for language models in information retrieval (including Laplace andDirichlet smoothing) are approximations to the Bayesian predictive distribution [2].CROSS REFERENCELANGUAGE MODELS, N-GRAM MODELSRECOMMENDED READING[1] Stanley F. Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. TechnicalReport TR-10-98, Center for Research in Computing Technology. Harvard University, August 1998.[2] Hugo Zaragoza, Djoerd Hiemstra, Michael Tipping, and Stephen Robertson. Bayesian Extension to the LanguageModel for Ad Hoc Information Retrieval. In Proceedings of the 26th Annual International ACM SIGIR Conferenceon Research and Development in Information Retrieval, pages 4-9, 2003.[3] ChengXiang Zhai and John Lafferty. A study of smoothing methods for language models applied to informationretrieval. ACM Transactions on Information Systems (TOIS) 22(2), pages 179-214, 2004.5

A study of smoothing methods for language models applied to information retrieval.

An empirical study of smoothing techniques for language modeling.

Bayesian Extension to the Language Model for Ad Hoc Information Retrieval.

https://doi.org/10.1007/978-0-387-39940-9_936

Probability smoothing

Probability smoothing

Abstract

Similar works

Full text

Available Versions

Radboud Repository

Crossref

University of Twente Research Information