Summary To realize more efficient information retrieval it is critical to improve the user&apos;s original query, because novice users can not be expected to formulate precise and effective queries. Queries can often be improved by adding extra terms that appear in relevant documents but which were not included in the original query. This is called query expansion. Query refinement, a variant of query expansion, interactively recommends new terms related to the original query. Because previous research did not offer any criterion to guarantee optimality, this paper proposes an optimal algorithm for query refinement with reference to the Bayes criterion

Fumitaro Goto

Fumito Masui

Hiroshi Masui

Masakiyo Suzuki

Yasunari Maeda

CiteSeerX

IJCSNS International Journal of Computer Science and Network Security, VOL.11 No.10, October 2011 
 
 
 
91
Manuscript received October 5, 2011 
Manuscript revised October 20, 2011 
The Bayesian Optimal Algorithm for Query Refinement in 
Information Retrieval 
Yasunari Maeda,  Fumitaro Goto,  Hiroshi Masui,  Fumito Masui  and  Masakiyo Suzuki 
   
Kitami Institute of Technology,  165 Koen-cho, Kitami-shi, Hokkaido 090-8507 Japan 
 
Summary 
To realize more efficient information retrieval it is critical to 
improve the user’s original query, because novice users can not 
be expected to formulate precise and effective queries. Queries 
can often be improved by adding extra terms that appear in 
relevant documents but which were not included in the original 
query. This is called query expansion. Query refinement, a 
variant of query expansion, interactively recommends new terms 
related to the original query. Because previous research did not 
offer any criterion to guarantee optimality, this paper proposes 
an optimal algorithm for query refinement with reference to the 
Bayes criterion. 
Key words: 
Information retrieval, query refinement, Markov decision 
processes, Bayes criterion 
1. Introduction 
These days many people use IR(Information Retrieval) 
systems. It is very important to improve the user’s original 
query, because it is difficult to formulate precise and 
effective queries. Queries can often be improved by 
adding extra terms that appear in relevant documents but 
which were not included in the original query. This is 
called query expansion. There are two forms of query 
expansion: relevance feedback and query refinement. 
  In relevance feedback[6] the system interactively 
recommends documents related to the original query, and 
the user chooses the documents preferred and performs IR 
using the new keys. Obviously there is a trade-off between 
IR result accuracy and the burden placed on the user. This 
trade-off can be resolved using statistical decision 
theory[1]. 
  In query refinement[7] the system interactively 
recommends new terms related to the original query, and 
the user chooses the terms preferred and performs IR 
using the new keys. The same trade-off exists between IR 
result accuracy and the user’s burden, but no previous 
paper has resolved this problem theoretically. This paper 
applies statistical decision theory to provide the first 
solution to this trade-off. 
  Query refinement can be considered as the task of 
recommending related terms and search documents 
wanted by the user under the condition that the user’s 
reaction in selecting related terms is unknown. More 
generally, it can be considered as the task of control given 
some unknown information. This kind of task is also being 
studied in the field of RL(Reinforcement Learning). Most 
models in the RL field employ MDP(Markov Decision 
Processes)[5] with some unknown information. This 
research also adopts MDP modeling with some unknown 
information for query refinement. Two RL previous 
algorithms that adopt MDP are described below. 
  QL(Q-Learning)[8] is the most famous RL algorithm and 
guarantees convergence to the optimal solution when the 
number of samples for learning is infinite. Obviously this 
assumption is not practical in the IR field because in query 
refinement the number of samples is the number of times 
that the system recommends related terms. 
  Martin considered this task based upon statistical 
decision theory[4]. His algorithm guarantees optimality 
with reference to the Bayes criterion when the number of 
samples is finite. However, it fails to represent the trade-
off between IR result accuracy and the user’s burden. 
Accordingly, we propose a new algorithm that can 
represent this trade-off and guarantee optimality with 
reference to the Bayes criterion when the number of 
samples is finite. The proposed algorithm is regarded as a 
generalization of Martin’s algorithm. Hereafter, optimality 
means optimality with reference to the Bayes criterion. 
  We describe MDP in section 2, and Martin’s algorithm in 
section 3. The model of our research is described in 
section 4. The proposed algorithm is introduced in section 
5. Section 6 concludes the paper. 
2. Markov Decision Processes 
Query refinement can be regarded as a RL problem, and 
most RL models are based on MDP. An MDP combines 
finite states, finite actions, a transition probability matrix 
and a reward function. When an action is chosen, a state 
transition occurs depending on the transition probability 
matrix, and when a state transition occurs a reward occurs. 
The purpose of MDP is to maximize the total reward 
received by repeatedly choosing actions. 
  Generally, there are two kinds of tasks in MDP, the 
discounted problem and the undiscounted problem. The 
IJCSNS International Journal of Computer Science and Network Security, VOL.11 No.10, October 2011 
 
 
92
goal of the discounted problem is to maximize the 
expected total discounted reward which is described as 
follows: 
⎟
⎠
⎞⎜
⎝
⎛ ∑=
∞
=0t
t
t zEv β ,                                        (1) 
where β , 0 1≤ ≤β  is a discount factor and zt  is the 
reward received at time t . In the discounted problem, 
rewards are discounted. The goal of the undiscounted 
problem is to maximize the average reward which is 
described as follows: 
h E z T
T
t
t
T
= +⎛
⎝⎜
⎞
⎠⎟
⎛
⎝
⎜
⎞
⎠
⎟
→∞ =
∑lim 1
0
.                                   (2) 
3. Previous Research 
In this section we explain Martin’s algorithm which is 
closely related to our research. In Martin’s algorithm the 
total discounted reward is maximized with reference to the 
Bayes criterion when the number of samples is finite. At 
first we define some terms. 
  We say θ  is a continuous parameter that dominates the 
transition probability matrix, Θ  is a set of parameters and 
θ* , θ* ∈Θ  is the true parameter. S , s Si ∈  is a finite set 
of states. A , a Ak ∈  is a finite set of actions. ( )r s a si k j, ,  is 
a reward function representing the reward for transition 
from state si  to state s j  when action ak  is chosen. 
( )p s s aj i k, ,θ  is an element of the transition probability 
matrix dominated by θ , which means the probability of 
transition from state si  to state s j  when action ak  is 
chosen. T  is the number of samples, and is regarded as 
the length of the task. xt , x St ∈  is a state at time t . x0  is 
the initial state. yt  is the action chosen at time t . 
x y x y x y xt t t0 0 1 1 1 1L − −  is the transition history from time 0  
to time t . And x y x y x y xt t t0 0 1 1 1 1L − −  is also described as 
( )x yx t . ( )p θ  is the prior probability density function. 
( )p x yx tθ⎛⎝⎜
⎞
⎠⎟  is the posterior probability density function 
when the transition history is ( )x yx t . ( )π x yx t⎛⎝⎜ ⎞⎠⎟  is the 
policy determining the action at state xt  when transition 
history is ( )x yx t . Π , ( )π ⋅ ∈Π  is a finite set of policies. 
In this task, the true parameter θ*  is unknown; all other 
things are known. 
  In Martin’s algorithm the optimal action at state xt , 
when the transition history is ( )x yx t , is chosen as follows: 
( )( )
( )( ) ( ) ( )( )( ),,,,,maxarg itit
s
t
tia
t
Martin
asyxxrsaxryxxaxsp
yxx
i
β
π
+∑=
   (3) 
where 
( )( ) ( )( ) ( )( )
( )( )( ) ( ) ( )( )( )( ),,,
,,
ji
t
Martini
t
ji
t
Martini
s
i
t
i
t
Martiniji
t
sasyxxasyxxrsasyxxsr
asyxxasyxxsspasyxxr
j
πβπ
π
+
∑=  (4) 
( )( ) ( )( ) ( ) .,,,, ∫=
Θ
θθθ daxspyxxpyxxaxsp ti
tt
ti        (5) 
Formula (3) is calculated by applying DP(Dynamic 
Programming). 
  Martin’s algorithm, however, can not represent the trade-
off between IR result accuracy and the user’s burden. 
Accordingly, we expand Martin’s model by applying 
query refinement in the next section. 
4. Model of Query Refinement 
4.1 Outline of Query Refinement 
We outline query refinement by dividing it into several 
steps. ( M0 M1 , M2  and M3  are constants.) 
Step1: The user determines M1  keywords. 
Step2: The system returns M2  related terms or M3  
documents. 
Step3: If the user wants to continue, he answers 
acceptance or rejection on each related term or document 
returned by the system in Step2. If the user wants to finish, 
he answers that he is satisfied or not satisfied and finishes. 
Step4: Until the user finishes, Step2 and Step3 are 
repeated M0  times at most. After repeating Step2 and 
Step3 M0  times, the user has to answer that he is satisfied 
or not satisfied, and this process is finished. 
4.2 Definitions 
We give some definitions which are different from 
Martin’s definitions or new here. Other definitions are the 
same as in Martin’s paper. 
  WORD , w WORDi ∈  is a finite set of words. M1  
keywords are chosen from WORD . DB , DB WORD⊆  is a 
finite set of words which exist in documents in a document 
database. DOC  is a finite set of documents in the 
document database. S , s Si ∈  is a finite set of states. si  is 
represented by a vector as follows except for s1  and s S : 
( ) ( ) ( ) ( )(
( )),,,
,,,,,,,2,,1,
DOCWORDis
WORDisDBisisissi
+
=
L
LL                 (6) 
where 
IJCSNS International Journal of Computer Science and Network Security, VOL.11 No.10, October 2011 
 
 
93
( )
⎪
⎪
⎩
⎪
⎪
⎨
⎧
=
,,3
;,2
;,1
;,0
,
userthebyrejected
userthebyaccepted
keyword
valuedefault
jis                   (7) 
⋅  is the cardinality of a set. s1  is an absorbing state 
when the user wants to finish and is satisfied. s S  is an 
absorbing state when the user wants to finish and is not 
satisfied. When s1  or s S  is reached, the query refinement 
process is finished. A , a Ak ∈  is a finite set of actions. ak  
indicates which M2  related terms or M3  documents are 
returned, and is represented by a vector as follows: 
( ) ( ) ( )( )a a i a i a i WORD DOCi = +, , , , , , ,1 2 L                (8) 
where 
( )a i j this word or document is not returned
this word or document is returned
,
, ;
, .
=
⎧
⎨
⎩
0
1
     (9) 
( )S s ai k, , ( )S s a Si k, ⊆  is a finite set of states that could be 
reached from state si  when action ak  is chosen. ( )A si , 
( )A s Ai ⊆  is a finite set of actions that could be chosen at 
state si . It becomes possible to represent the trade-off 
between IR result accuracy and the user’s burden by 
defining a reward function, discount factor and increase 
factor as follows. ( )r s a si k j, ,  is the reward function 
representing the reward for transition from state si  to state 
s j  when action ak  is chosen. It is described as follows: 
( )r s a s
R s s s s s s
R s s s s s s
else
i k j
i i S j
i i S j S, ,
, , , ;
, , , ;
, ,
=
≠ ≠ =
≠ ≠ =
⎧
⎨
⎪⎪
⎩
⎪
⎪
1 1 1
2 1
0
         (10) 
where R1 , 0 1≤ < ∞R  is the positive reward gained when 
the user is satisfied, R2 , −∞ < ≤R2 0  is the negative 
reward gained when the user is not satisfied. α1 , 
0 11≤ ≤α  is a discount factor. When R1  is gained at time 
t , R1  is valued at α1 1t R . α2 , 1 2≤ ≤ ∞α  is an increase 
factor. When R2  is gained at time t , R2  is valued at 
α2 2
t R . Initial state x0  is determined by M1  keywords. 
( )XY x0 , is a finite set of transition histories all of which 
are from x0  to s1  or s S . ( )XY x0 ,π , ( ) ( )XY x XY x0 0,π ⊆  
is a finite set of transition histories all of which are from 
x0  to s1  or s S  when policy π  is used. In this task, the 
true parameter θ*  is unknown; all other things are known. 
4.3 Utility Function 
( )u x yx n , ,π θ⎛⎝⎜ ⎞⎠⎟  is the utility function that indicates the 
value of the reward given when the true parameter is θ , 
policy π  is used, and the transition history is ( )x yx n . 
( )u x yx n , ,π θ⎛⎝⎜ ⎞⎠⎟  is described as follows: 
( )( )
( ) ( )
( ) ( )
( ) ( )⎪
⎪
⎩
⎪⎪
⎨
⎧
∉
=∈
=∈
= −
−
.,,0
;,,,
;,,,
,,
0
02
1
2
101
1
1
π
πα
πα
θπ
xXYyxx
sxxXYyxxR
sxxXYyxxR
yxxu
n
Sn
nn
n
nn
n (11) 
In fact, the value of the utility function doesn’t depend on 
π , θ  but on ( )x yx n . Accordingly, it can also be described 
as follows: 
( )( ) ( )( )
( ) ( )
( ) ( )
( ) ( )⎪⎪⎩
⎪⎪
⎨
⎧
∉
=∈
=∈
=
′=
−
−
.,0
;,,
;,,
,,
0
02
1
2
101
1
1
xXYyxx
sxxXYyxxR
sxxXYyxxR
yxxuyxxu
n
Sn
nn
n
nn
nn
α
α
θπ
            (12) 
4.4 Bayes Expected Utility Function 
The Bayes expected utility function yields the expected 
value of the utility function when the prior probability 
density function is ( )p θ , the policy π  is used, and the 
initial state is x0 . The Bayes expected utility function is 
described as follows: 
( )( )
( ) ( )( )
( ) ( )
( )( ) .,,,,
,,
0
01 ,
0
0
θθπθπθ
θπ
πθ
dyxxuxyxxpp
pxuB
nM
n xXYyxx
n
n
∑ ∑∫=
= ∈Θ∈
  (13) 
4.5 Bayes Decision 
Bayes Decision is the optimal decision with reference to 
the Bayes criterion. In this task the Bayes decision is the 
policy that maximizes the Bayes expected utility function 
when the prior probability density function is ( )p θ , and is 
described as follows: 
( )( ) ( )( )BD p Bu x pθ π θ
π
= arg max , , .0                  (14) 
5. Proposed Algorithm 
The algorithm proposed herein calculates the Bayes 
decision described by formula (14). It represents the trade-
off between IR result accuracy and the user’s burden by 
adopting the reward function of formula (10), together 
with a discount factor and an increase factor. 
IJCSNS International Journal of Computer Science and Network Security, VOL.11 No.10, October 2011 
 
 
94
  The Bayes decision of formula (14) can also be described 
as follows: 
( )( )
( )
( )
( )
( )(
( )
( )
( )
( )(
( )
( )( )
( )
( )( )(
( ) ( )
( )( ) ( )( )) ).,,
max
,,max
,,max
,,maxarg
00
000
101001010
22322
11211
00100
1
11
,
3
,
2
223
21100
,
100112
100
,
0001
L
L
MM
MMM
yxSxxAy
yxSxxAy
yxSxxAy
yxSxxAy
yxxuyxxyxxp
yxxuyxxyxxp
xyxyxuxyxyxxp
xyxuxyxxp
pBD
MMMMM
′
∑+
′∑+
′∑+
′∑
=
−
−−
∈∈
∈∈
∈∈
∈∈
−−−−
θ
   (15) 
The Bayes decision of formula (15) can be calculated by 
applying DP from time M0 1−  to time 0 . When 
t M= −0 1 the action is decided as follows: 
( )( )
( )
( )( )( ( )( )).,,,,maxarg 22111 ttSttttxAa
t
pro
yxxaxspRyxxaxspR
yxx
t
αα
π
+=
∈
  (16) 
When t M= −0 2  the action is decided as follows: 
( )( )
( )
( )( )(
( )( ) ( )( )
( )
( )( )) ,,,,,
,,maxarg
22
,
,
111
1
t
tS
t
sss
axSs
j
tt
tj
t
t
t
xAa
t
pro
yxxaxspRasyxxryxxaxsp
yxxaxspRyxx
Sj
tj
t
α
απ
+∑+
=
≠
∈
∈ (17) 
where 
( )( ) ( )( ) ( )( )
( )( ) ( )( ).,,
,,
22
111
j
t
j
t
projS
t
j
t
j
t
proj
t
j
t
asyxxasyxxsspR
asyxxasyxxsspRasyxxr
πα
πα
+
=      (18) 
When 0 30≤ ≤ −t M  the action is decided as follows: 
( )( )
( )
( )( )(
( )( ) ( )( )
( )
( )( )) ,,,,,
,,maxarg
22
,
,
111
1
t
tS
t
sss
axSs
j
tt
tj
t
t
t
xAa
t
pro
yxxaxspRasyxxryxxaxsp
yxxaxspRyxx
Sj
tj
t
α
απ
+∑+
=
≠
∈
∈ (19) 
where 
( )( ) ( )( ) ( )( )
( )( ) ( )( )
( )( )( )
( ) ( )( )( )
( )( ) ( )( ).,,
,,
,,
22
,
,
111
1
j
t
j
t
projS
t
kj
t
proj
t
sss
asyxxsSs
j
t
j
t
projk
j
t
j
t
proj
t
j
t
asyxxasyxxsspR
sasyxxasyxxr
asyxxasyxxssp
asyxxasyxxsspRasyxxr
Sk
j
t
jk
πα
π
π
πα
π
+
∑+
=
≠
∈     (20) 
  Thus we can maximize the expected reward, which 
represents the trade-off between IR result accuracy and the 
user’s burden, with reference to the Bayes criterion. 
6. Conclusion 
This paper has described how to optimize query 
refinement. Previous research[7] on query refinement did 
not theoretically consider the trade-off between IR result 
accuracy and the user’s burden. We proposed a new 
algorithm that maximizes the expected reward, which 
represents the trade-off between IR result accuracy and the 
user’s burden, with reference to the Bayes criterion; we 
apply MDP with an unknown parameter. The proposed 
algorithm is also regarded as a generalization of Martin’s 
algorithm[4] which solves the discounted problem of 
MDP with reference to the Bayes criterion. 
  This paper models the task of query refinement as the 
MDP problem of length Mo , and the proposed algorithm 
is the optimal algorithm for solving that MDP problem. 
Roughly speaking , the algorithm in the previous 
research[7] on query refinement can be regarded as an 
approximate algorithm that solves the MDP problem 
whose length is 1 , note that the MDP problem that should 
be solved has length of Mo . Simulations showed that with 
the discounted problem, the reward increases with task 
length[3]. It can be easily imagined that the task of query 
refinement shows the same property. 
  The order of the computational complexity of the 
proposed algorithm is exponential with respect to M0 . 
The order of Martin’s algorithm is also exponential. An 
improved algorithm has been proposed for discounted 
problem[2]. The order of the improved algorithm is 
polynomial under the condition that the Bayes decision is 
calculated. We expect that our algorithm can be improved 
in the same way. Its computational complexity is still very 
big, even though it has polynomial order. A future task is 
to further reduce its computational complexity. 
  In the proposed algorithm, the transition probability 
matrix dominated by an unknown parameter is learned 
while performing query refinement. The learned transition 
probability matrix can be seen as a kind of thesauri or 
ontology for query refinement or IR. 
 
References 
[1] Kaji, M., Ukita, Y., Matsushima, T., “A Note on 
Information Retrieval by Relevance Feedback”(in Japanese), 
The 21st Symposium on Information Theory and Its 
Applications, 1998. 
[2] Maeda, Y., Ukita, Y., Matsushima, T., Hirasawa, S., “The 
Optimal Algorithms for the Reinforcement Learning 
Problem Separated into a Learning Period and a Control 
Period”(in Japanese), Trans.IPS.Japan, Vol.39, No.4, 
pp.1116-1126, 1998. 
[3] Maeda, Y., Ukita, Y., Matsushima, T., Hirasawa, S., 
“Proposition of Learning Algorithms on Markov Decision 
Processes with Unknown Parameter”(in Japanese), The 19th 
Symposium on Information Theory and Its Applications, 
pp.597-600, 1996. 
[4] Martin, J.J., “Bayesian Decision Problems and Markov 
Chains”, John Wiley & Sons, 1967. 
[5] Martin,L.P., ”Markov Decision Processes”, John Wiley & 
Sons, 1994. 
IJCSNS International Journal of Computer Science and Network Security, VOL.11 No.10, October 2011 
 
 
95
[6] Salton, G., Buckley, C., “Improving Retrieval Performance 
Relevance Feedback”, Journal of the American Society for 
Information Society, Vol.41, No.4, pp.288-297, 1990. 
[7] Velez, B., Weiss, R., Sheldon, M.A., Gifford, D.K., “Fast 
and Efficient Query Refinement”, SIGIR97, pp.6-15, 1997. 
[8] Watkins, C.J.C.H., Dayan, P., “Q-Learning”, Machine 
Learning, Vol.8, pp.279-292, 1992. 
 


The Bayesian Optimal Algorithm for Query Refinement in Information Retrieval

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=E896F3967249D0473FBDE5BF3A135387?doi=10.1.1.1069.820&rep=rep1&type=pdf

The Bayesian Optimal Algorithm for Query Refinement in Information Retrieval

Abstract

Similar works

Full text

Available Versions

CiteSeerX