A Tri-Valued Belief Network Model for Information Retrieval

Das-Neves, Fernando

Computer Science Technical Reports @Virginia Tech

A Tri-Valued Belief Network Model for Information Retrieval December 2001  Fernando Das-Neves Computer Science Dept. Virgina Polytechnic Institute and State University Blacksburg, VA 24060   1. IR models at Combining Evidence Graphical Models [Pearl88] [Buntine94] [Jensen2001] allow for greater flexibility at modeling relations among different sources, when compared with other methods like LSI, or standard vector models, because each source can be modeled independently. Graphical models rely on a subjective instead of frequentist understanding of probabilities. Such probabilities do not necessarily represent relative frequencies (although they are sometimes calculated like they were), but instead they represent the degree of certainty, belief or support about a certain feature (a term, a link, etc.), and about one feature given another feature (conditional probabilities). The most common type of graphical model applied to IR is Bayesian Networks. The first successful application of Bayesian networks to IR is the Inference Network Model in [Fung95], [Turtle90]. Belief Networks [Ribeiro-Neto96] [Silva2000] are an alternative approach to Inference Networks when explicitly modeling an information retrieval system. While similar in expressive power to inference networks, belief networks can express any inference network used to retrieve documents by content similarity, while the opposite is not necessarily true. The key difference is in the modeling of p(dj|t) (probability of a document given a set of terms or concepts) in belief networks, as opposed to p(t|dj) used in Bayesian networks. Since in a Bayesian network (both of the inference and belief type), instantiating di makes p(t1|dj) and p(t2|dj) defined and mutually independent, then p(t|dj) can be calculated from the independent probabilities p(dj|ti) in a belief network, and so belief networks can reproduce the ranking generated by an inference network. Generating the ranking produced by belief network from an inference network is not always possible, as a belief network can reproduce the cosine similarity ranking, while this is not possible for an inference network (see [Ribeiro96] for details). Figure 4 illustrates the differences in the probability structure between the two networks (note the direction of the arrows between the terms and the documents):  Inference Network  d1 d2 dm t1 t2 tn q  Belief Network  d1 d2 dm t1 t2 tn q  Figure 4.  Examples of Inference and Belief Networks to calculate relevance of a document given a query.In these graphs, an arrow between two nodes represent the conditional probability of the variable pointed by the arrowhead, given the value of variable at the tail of the arrow. Variables not connected by arrows are assumed to be independent. While these networks look almost identical, the conditional probability structure is different, and it has two important consequences: 1) The “hammock" structure between a document and a query can be made to represent a vector dot-product, with p(di|t) and p(q|t) the two vectors of dimension n to multiply, with p(di|tj) and p(q|tj) being the individual components of the vectors. From this point of view, belief networks constitute a tool to combine the best of vector and probabilistic operations. 2) Belief networks can calculate any ordering calculated by an inference network, including the traditional cosine similarity used in SMART, while the opposite is not true. An inference network, on the other hand, ranks the documents by calculating p(q|d) using the chain rule: for the set U of variables in the model ∏= (var)).|(var)( parentspUP  Inference networks can model a structure that calculates a ranking similar to the cosine similarity, but the calculation has an extra term that is document-dependent, and so for Bayesian networks cannot replicate the ranking given by cosine similarity.  It is worth noting that we are showing here only the basic networks. It is possible to include relevance feedback as part of the network as new nodes, or relations between terms like synonyms as new nodes connecting related terms (as long as they do not introduce cycles). New nodes also arise from other sources of information, as clusters, or hubs and authorities like in CLEVER [Kleinberg98]. [Turtle90], [Haines93] and [Silva2000] provide more information on these extensions of the basic network model. Our retrieval scheme, explained in detail below, allows not only for the specification of document features that are important or irrelevant to the user need, but also allows for a neutral interpretation, letting the user say “I don’t know”. Document features that do not explicitly appear in the query formulation are taken as “undetermined” by the system. Document content is treated as usual.  2. How we perform Retrieval Our model for retrieval is a belief network combining multiple sources of evidence. A belief network let us combine probabilities and vector operations, allowing us a fast calculation of similarity values even for big collections. Our belief network is formally described as follows:  2.1 Notation  • An uppercase letter like A letter represents a set.  • One or more lowercase letters without a subindex, (like da, for example), are subsets of other sets (context given where is used).  • A lowercase letter with a subindex, like ai , represents the ith element of the set named with the same letter but in uppercase.  • |A| is the number of elements of set A.  • A  is the vector representation of the set A, given some ordering of the set elements, and ia is the ith component of A . Note that vectors and sets are equivalent. • For a set of variables x1…xn, ∑==ni ixnx11  2.2 Variables and their Relationships:  2.2.1 Structure.  The definition of the belief network is based on the basic belief network in [Ribeiro96]. In our case, there is one belief network per document d with the following structure:   ...... ...dC 1 C 2 C L...K G A G R Gk 2 k u rmr 1 r 2ana 2a 1Q k Q a Q rk 1 Figure 2. Belief Network to calculate and combine evidence from terms, authors  and references.  where  • d is a document,  • D is the set of all documents in the collection,  • Qk; Qa; Qr represent the query information about document content, authors and references, respectively,  • KG; AG;RG represent the set of Keywords (Keyword Group), Authors and  • References, respectively,  • ikq, kaq, nrq are the vector representations of kik Qq ∈ , aa Qq k ∈, rr Qq n ∈, for ||1 |,|1 |,|1 RGnAGjKGi ≤≤≤≤≤≤ , • ki, ai; ri represent a particular author, a keyword or a reference,  • Cl represents a document cluster, for ||1 Dl ≤≤ .  By the structure of the belief network, we treat content-based and structural information the same way. Variables ki , ai and ri represent a link (although not explicit) between all documents that contain the information represented by the variable, which must appear in at least one document for the variable to belong to the bayesian network.   2.2.2 Variables.  The following table describes the domain (possible set of values) of the variables described above:  Variable Domain Qk ; Qa ; Qr {1, 0} ikq, kaq, nrq {yes, no, unknown} ki, ai, ri {yes, no} KG, AG, RG {relevant, irrelevant} d {relevant, irrelevant} Cl {1, 0} Table 2. Variables of the Belief Network in Figure 2, and their possible values.  2.3 Interpretation of variable values.  For a certain document d; ki = yes iff the keyword or concept represented by ki is relevant for d, otherwise ki is considered non-relevant for d and ki = no. In the same way and for the same document d, ai = yes or ri = yes if the corresponding author or reference appears in d. We call the sets KG; AG and RG sources of evidence of d.  A query Q from the user is represented as three independent and disjoint query subsets Qk; Qa ; Qr, so Q = rak QQQQ, and kik Qq ∈ , aa Qq i ∈, rr Qq i ∈.If the query formulation included the keyword ki then yesq ik = , otherwise unknownq ik =  unless the user has explicitly said that ikq  is not relevant for the query subset Qk, case in which then . noq ik = . For example, if Qk = {k1; k4 ; not(k10)}, then yesqq kk == 41 , noqk =10 q k10 = no, and all other unknownq jk = . Under this interpretation, absence of evidence about relevance of a keyword, author or reference in the query formulation does not constitute proof of its irrelevance.  The same definitions also apply for all the iaq  and irq .  Any of query subsets Qk ; Qa and Qr is relevant if the probability of at least one of the elements of the query subset is non-zero. That is, if we call G one of the query subsets of Q and there is some vi ∈ G such that p(G|vi ) ≠0, then G is relevant. A document d is relevant given a query Q if at least one of Qk; Qa; Qr , is relevant.  A cluster Cl = 1 if at least one document belongs to Cl, and p(Cl = 1|d) is the probability of the information in cluster Cl being related to the content of document d.   2.4 Definition of Probabilities:  The advantage of this type of belief network is that every hammock structure can be interpreted as a vector dot-product, and so it let us combine vector operations with probabilistic operations.  When combining evidence for all the sources, we want that  • Absence of evidence from one source does not affect other sources  • Evidence from more than one source is more effective that evidence from only one source  • If a source of evidence uniquely identifies a document, all other sources are irrelevant.  A noisy-or [Good61] of the sources satisfies the above conditions, under the added assumption that all sources are independent:  RG)))]wr(p(d  (AG)))  wa(p(d (KG)))wk(p(d  [(   relevant) RG  relevant; AG  relevant;KG  relevantp(d |1|1|11|−×−×−−=====     [1]  Here wk, wa, and wr weight-adjusting functions, that transform each of the probabilities according to some measure of the importance of the sources. The functions wk, wa, and wr must be monotonically increasing and between 0 and 1.   For the sake of deriving some of the probabilities, and to simplify the explanation, we are going to use AG as the source of evidence, an ai and as the particular instantiation of that source in a particular document. Because of the similar structure, the same rationale can be applied also to KG an RG, unless otherwise noticed.  Let us define the sets A = { set of all authors in all documents }, and the set a to be a subset of authors such that a ⊂ A, and vectors ad  and and  such that        0≠iad  if ai is an author of d, and 0 otherwise, and        0≠iand  if ai is not an author of d, and 0 otherwise.  By the fundamental rule of probabilities we have )()(aaQpQAGp p(AG|Qa) 	= , and  )()|()|()( iAaiaia apaQpaAGpQAGpi∑∈=	   [2]  (see the original Belief Network paper [Ribeiro96] for the rationale of this equation).  If we define p(d|AG) = p(AG|Qa ), then given p(AG|ai); p(Qa|ai), and p(ai) we can calculate equation [1].  Let  xi = p(AG|ai = yes),  1 ≤ i ≤|a| yj = p(AG|aj = no) 1 ≤ j ≤|A|-|a| zk = p(Qa|ak) 1 ≤ k ≤|A|  xi is the probability that the author ai is one of the authors of document d. yi is the probability that author i is not one of the authors of document d. zi is the probability that ka aq k = . Being 0 otherwise. We ignore any author that is not an author of any document in the collection. For the case were for all aa Qq i ∈  each iaq  either yes or no (the set of authors of a document is completely specified in the query, both the ones that are authors and the ones that are not), we desire a complete exact match between Qa and A to be the idf(a), the inverse of the number of documents that have the set a and only a as authors. The reason for this is that for an exact specification we want the probability to be proportional to the number of those cases in the document collection.  To say this formally, for the case aaa nddQ  +=  (that is, aQ  is all 1s), we want, given that p(ai) is constant for all documents and all authors (a reasonable and common assumption for a-priori author probabilities),   )()(||1||1aidfapzyzx iAiAjjjii =+∑ ∑= = This we can approximate by  )(||1||1aidfzyzxAiAjjjii =+∑ ∑= =        [3]  Since p(ai) is a constant that affects all document in the same way.  Please note that for the same index i., xi and yi are never 1 simultaneously.  Note also that ∑==||1||Ai idax  and ∑=−=|| ||||A ji j dnaAy   2.5 Boolean Approach.  One way to give values to xi, yi and zi is  xi = 1/|A| iff ai =yes, yj = 1/|A| iff aj =no, and zk = 1 iff kka aQ = . That is, the probability of an author being relevant or non-relevant is constant.  Then, ( ) ∑∑∑====+||1||1||1Ai iiAj jjAi iizgzyzx , where gi = 1/|A|. For the case 1)|()(( =∀ ia aqpi i  (that is, all iaq  matched the corresponding ai),  we have that [3] then becomes )()(1||1||1aidfaidfAgzg AAi iAi ii=⋅⋅== ∑∑== as we wanted. Also, for the case if only positive information (that is, only authors who may be authors of d are specified in Qa), equation [3] becomes ||),( A aQadaa dQsim= , which is the intersection of the boolean sets Qa and da.(Qa is Boolean in this case because for positive information only, each component has either the value yes or unknown).  2.6 Another Approach.  However, this is not the only possible definition of probabilities. If we define xi and zi then it is also possible to deduce the formulation for yi , subject to the constraint that the resulting formulation is a probability. Again, for the case aaa nddQ  +=  (another way of saying the query included all authors, and matched exactly the set of all authors of d), we have that all zi ≠ 0. If we define )nono| a p(qyes) yes| ap(q iaia ii =====  = some constant zc for all i , then we want          ( ) ( ) )(|1||1||1aidfyxzyxz dnay jdax icAj jAi ic ji=+=+ ∑∑∑∑ ∈∈==  and therefore   czaidfydaAxda 1)(|)||(||| =−+           [4] Now we need to define xi. A possible definition is |da|)ersetidf(x daisup= , so that ∑==||1)(supdai dai ersetidfx   where set daauthors ine all the t least thhat have adocument tnumber of  ) ersetidf( da1sup =    this definition is useful because it relates the probability of a document having a set of authors to the size of the set of document authored by all those authors, with possibly other authors. Note also that 0 ≤ idf(supersetda) ≤ idf(da).  With these definitions of x and zk we can calculate ||||||)(daAxdaczdaidfy−−=  therefore ( )2||||||)(daAxdaczdaidfiy−−= . since 1,0 ≤≤ yx , for these definitions to be valid they need to satisfy these two conditions: 1) 0||)( ≥− xdazdaidfc, and 2) daAxdazdaidfc−≤− ||)(   Condition 1 implies ∑ ≤=cidazdaidfxersetidf )()(sup . Since 0 ≤ zc ≤ 1 it must be that cdazdaidfersetdf )()(sup ≤ , which is only possible to guarantee when zc = 1. Therefore we define 1) ======= nono| a p(qyes) yes| ap(qz iaiak ii .  With these values for zk, to probe that the definition of xi satisfies condition 2 is to prove that ∑ ++−≥ )(daidfdaAxi . We have four possible cases:  Case 1) |da| = 1 and idf(da) = 1/|D| Here ||11)( DAdaidfdaAxi ++−=++−≥∑ . Since idf(da) = 1/|D| and because 1/|D| ≤  idf(supersetda) ≤ idf(da), it must be that idf(supersetda) = idf(da) = 1/|D|.  Since -|A|+1 ≤ 0, it follows that ||111 DADxi ++−≥=∑ . Case 2) |da| = |A| and idf(da) = 1/|D|  Following the reasoning in case 1, it must be that idf(supersetda) = idf(da)=1/|D|. Furthermore in this case -|A| + |da| = 0, then ||11 DDxi∑ ≥= .  Case 3) |da| = |A| and idf(da) = 1 Now we need to prove∑ ++−≥ )(daidfdaAxi = idf(da). Since |da| = |A| means all the authors are included in da, and idf(da) = 1 means there is only one document matching, it follows that there must be only one document in the collection. Therefore  |D| = 1, and since da cannot be empty (we have to be matching at least one author), the only possible answer is idf(supersetda)=idf(a) as needed.  Case 4) |da| = 1 and idf(da) = 1 ∑ ++−≥ )(daidfdaAxi then ∑ +−=++−≥ 2)(1 |A|daidfAxi . We need to look at two subcases, namely |A| = 1, and |A| > 1.   If |A|=1, then idf(a)=idf(supersetda), because then the same author is the only author of all documents, and condition 2 becomes ∑ =++−≥= )()(11 daidfdaidfAxi .   If |A|>1 then condition 2, for case 4 becomes∑ =++−≥ 0)(1 daidfAxi , which is always true.  Therefore we have proven case 4.  This is all we need to calculate p(d|AG). For all structural information, as References, Journal where the document was published, or Authors, all of the above applies. For content information (set KG), we follow the same approach as in [Baeza99] and we define  dd ) p(KG|k jii , = , where )()(, jjji ktfkidfd ×=if term kj=yes in document di, or 0 otherwise, and  kkik Qq ) |kp(Q j = , where jkq= weight of term kj in query Qk, iff jk kq j = in d, or 0 otherwise.  These definitions of p(KG|kj) and p(Qk|kj), when applied to equation [2], make the result of equation [2] to be the cosine similarity between the query and the document, times a constant p(ki).   2.7 Cost of calculating p(d|Q).  Under the above definition of the probabilities in the belief network, it is easy to see that if values like idf(da) and idf(supersetda) are precomputed while the collection is indexed, the cost of calculating p(d|Q) is O( total number of authors + number of Keywords + number of references) for each document d and query Q, since the cost of calculating each probability becomes constant, and calculating the importance of a source is calculating a set of dot-products.   2.8 Document/Cluster probability.  For two documents d1 and d2, we define            p(d1|d2 ) = p(d1|Qk = KG of d2 ;Qa = AG of d2 ;Qr = RG of d2 )  If for a moment we assume we know p(di|Cn ); then the probability of cluster n given document i, is  dp)p(C|Cp(d ) |dp(Cinn iin )() =  by Bayes Rule.  If we assume the existence of |D| clusters (as many clusters as documents), and a-priori probabilities p(Ci) = p(di) constant for all 1 ≤ i ≤ |D|, then p(Cn|di) = p(di|Cn), which we can define in many ways, for example ∑∈=nj Cdjinni ddpCCdp )|(||1)|(  (average similarity between the document and all other documents for which p(dj|Cn) ≠ 0: To avoid the problem of all documents depending on all the other documents' probabilities, at the beginning we can initialize p(di|Cn)  = 1 iff i = n; and 0 otherwise (at the beginning, each document belongs to its own cluster, which is consistent with the initial state of many clustering algorithms).    3. REFERENCES [Pearl88] Pearl, J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo, CA, 1988. [Buntine94] Buntine, W. Operations for learning with graphical models. Journal of Artificial Intelligence Research, 1994. [Jensen2001] Jensen, F. Bayesian Networks and Decision Graphs. Springer-Verlag, 2001. [Fung95] Fung, R., and Favero, B. Applying Bayesian Networks to Information Retrieval. Communications of the ACM, Vol 38, No 3, 1995. [Turtle90] Turtle, H., Croft, W. Inference networks for document retrieval. In Proceedings of SIGIR’90, pp.1-24, 1990. [Silva2000] Silva, I., Ribeiro-Neto, B., Calado, P., Moura, E., Ziviani, N. Link-based and content-based evidential information in a belief network model. In Proceedings of SIGIR 2000, pp. 96-103, 2000. [Ribeiro-Neto96] Ribeiro-Neto, B., Muntz, R. A belief network for IR. In Proceedings of SIGIR’96. pp. 253-260, 1996. [Haines93] Haines, D., Croft, B. Relevance feedback and inference networks. In Proceedings of SIGIR’93, pp. 2-11, 1993. [Kleinberg98] Kleinberg, J. Authoritative Sources in a Hyperlinked Environment. In Proceedings of ACM-SIAM Symposium on Discrete Algorithms, pp 668-677, 1998.  [Baeza99] Baeza-Yates, R, Ribeiro-Neto, B.  Modern Information Retrieval. ACM Press, 1999.  

http://eprints.cs.vt.edu/archive/00000549/01/FDN01.pdf

A Tri-Valued Belief Network Model for Information Retrieval

Abstract

Similar works

Full text

Available Versions

Computer Science Technical Reports @Virginia Tech