CAI, Peng

GAO, Wei

WONG, Kam-Fai

ZHOU, Aoying

English

T. Hastie

Crossref

Weight-Based Boosting Model for Cross-Domain Relevance Ranking Adaptation

Institutional Knowledge at Singapore Management University

Singapore Management University Institutional Knowledge at Singapore Management University Research Collection School Of Information Systems School of Information Systems 4-2011 Weight-based boosting model for cross-domain relevance ranking adaptation Peng CAI Wei GAO Singapore Management University, weigao@smu.edu.sg Kam-Fai WONG Aoying ZHOU Follow this and additional works at: https://ink.library.smu.edu.sg/sis_research  Part of the Databases and Information Systems Commons Citation CAI, Peng; GAO, Wei; WONG, Kam-Fai; and ZHOU, Aoying. Weight-based boosting model for cross-domain relevance ranking adaptation. (2011). Proceedings of the 33rd European Conference on Information Retrieval (ECIR 2011). 562-567. Research Collection School Of Information Systems. Available at: https://ink.library.smu.edu.sg/sis_research/4596 This Conference Proceeding Article is brought to you for free and open access by the School of Information Systems at Institutional Knowledge at Singapore Management University. It has been accepted for inclusion in Research Collection School Of Information Systems by an authorized administrator of Institutional Knowledge at Singapore Management University. For more information, please email libIR@smu.edu.sg. Weight-Based Boosting Model for Cross-DomainRelevance Ranking AdaptationPeng Cai1, Wei Gao2, Kam-Fai Wong2,3, and Aoying Zhou11 East China Normal University, Shanghai, Chinapengcai2010@gmail.com, ayzhou@sei.ecnu.edu.cn2 The Chinese University of Hong Kong, Shatin, N.T., Hong Kong{wgao,kfwong}@se.cuhk.edu.hk3 Key Laboratory of High Conﬁdence Software Technologies, Ministry of Education,Beijing, ChinaAbstract. Adaptation techniques based on importance weighting wereshown eﬀective for RankSVM and RankNet, viz., each training instanceis assigned a target weight denoting its importance to the target domainand incorporated into loss functions. In this work, we extend RankBoostusing importance weighting framework for ranking adaptation. We ﬁndit non-trivial to incorporate the target weight into the boosting-basedranking algorithms because it plays a contradictory role against the in-nate weight of boosting, namely source weight that focuses on adjust-ing source-domain ranking accuracy. Our experiments show that amongthree variants, the additive weight-based RankBoost, which dynamicallybalances the two types of weights, signiﬁcantly and consistently outper-forms the baseline trained directly on the source domain.1 IntroductionLearning to rank [4] is to derive eﬀective relevance ranking functions based ona large set of human-labeled data. Boosting has been extensively studied forlearning to rank [1,2,8,9]. However, existing ranking algorithms, including theboosting-based ones, are only proven eﬀective for data from the same domain.In real applications, it is prohibitive to annotate training data for every searchdomain. Ranking performance may suﬀer when the training and test have totake place on diﬀerent domains.A promising direction is to learn a cross-domain adaptation model for ranking.Two key problems should be resolved: (1) how to measure the relatedness of twodomains appropriately; (2) how to utilize this information in ranking algorithmsfor adaption. [3] adopted a classiﬁcation hyperplane to derive the importanceweight of documents in the source domain that reﬂects their similarity to thetarget domain and is then incorporated into the rank loss function.When applying this method, we ﬁnd it non-trivial to integrate importanceweight into boosting-based algorithms such as RankBoost [2]. The reason is This work is partially supported by NSFC grant (No. 60925008), 973 program (No.2010CB731402) and 863 program (No. 2009AA01Z150) of China.P. Clough et al. (Eds.): ECIR 2011, LNCS 6611, pp. 562–567, 2011.c© Springer-Verlag Berlin Heidelberg 2011Weight-Based Boosting Model 563that these algorithms bear an inherently weight-based exponential loss and thisinnate weight in the loss function (source weight) plays a contradictory role withthe importance weight introduced for adaptation purpose (target weight). Anappropriate balance must be made between these two types of weight; otherwiseadaptation may fail since the model can easily overﬁt source data due to thegreat impact of source weight on weak rankers selection.In this work, we develop three Weight-based RankBoost (WRB) algorithmsto balance the source and target weights, namely expWRB, linWRB and ad-ditive WRB (addWRB). The ﬁrst two methods incorporate the target weightin straightforward and static ways, and the third combines the weights from aglobal perspective based on a forward stage-wise additive approach [6] to achievea dynamic tradeoﬀ. Our results demonstrate that addWRB consistently and sig-niﬁcantly outperforms the baseline trained directly on the source domain.2 Target Weight and Source Weight2.1 Source WeightRankBoost [2] aims to ﬁnd a ranking function F to minimize the number ofmisordered document pairs. Given document pairs {〈xi, xj〉}, the ranking lossis deﬁned as rLoss(F ) =∑i,j W (xi, xj)I(F (xi) ≥ F (xj)), where W (xi, xj) isthe source weight distribution, I(.) is an binary indicator function, and F (xi) ≥F (xj) suggests that the ranking function assigns a higher score to xi than toxj while the ground truth is xi has a lower rating than xj . At each round oftraining, W (.) is updated for the next round to focus on those misordered pairs.The update formula in round t is given as follows:Wt+1 =1ZtWt(xi, xj) exp(αt(ft(xi)− ft(xj))) (1)where ft(x) is the 0-1 valued weak ranker derived from a ranking feature x, αtis the coeﬃcient of ft so that F =∑t αtft(x), and Zt is normalization factor.Inherently, source weight is designed to control the selection of weak rankers tominimize ranking errors in source domain.2.2 Target WeightIn ranking adaptation, the knowledge of relevance judgement should be strength-ened on those documents that are similar to the target domain so that the learn-ing can be focused on correctly ranking these important documents. [3] used thecross-domain similarity to transfer ranking knowledge. The distance of a source-domain document to the classiﬁcation hyperplane was calculated as target weightto measure the importance of the document. Then the pointwise weight was con-verted to pairwise for compatible with the popular pairwise approach (see [3] fordetails). The general loss term was extended as follows:∑i,jw(xi, xj) ∗ rLossij(.) (2)where rLossij(.) is the pairwise loss and w(xi, xj) is the target weight.564 P. Cai et al.Algorithm 1. expWRB–Weighted RankBoost with target weight inside the exponentInput: Query-document set of source domain; Target weights of M document pairs {w(xi, xj)}M1based on the ground truth.Output: Ranking function F (x).1. Initialize W1(xi, xj) = 1M for all i, j;2. for t = 1; t ≤ T ; t + + do3. Select weak ranker ft(x) using Wt and w;4. Set coeﬃcient αt for ft(x);5. For each (xi, xj), update source weight usingWt+1 = 1Zt Wt(xi, xj) exp(αtw(xi, xj)(ft(xi)− ft(xj)));6. end for7. return F (x) = ∑Tt=1 αtft(x)3 Weight-Based Boosting Models for Ranking AdaptationIn standard RankBoost, the source weight is updated iteratively so that theweak rankers can focus on those misordered pairs with large source weight thatcommonly reside near the decision boundary deemed more diﬃcult to order.However, these pairs may be unnecessarily important to the target domain.Meanwhile, for those misordered pairs with low source weight, even though theycontain some important cross-domain ranking knowledge (i.e., having high targetweight), the algorithm does not prioritize to correct their ranking. The two typesof weight play contradictory roles and must be appropriately balanced.The objective of our adaptive RankBoost is to minimize the weighted rank-ing loss wLoss(F ) =∑i,j W (xi, xj)I(F (xi) ≥ F (xj))w(xi, xj) following Eq. 2.There are two straightforward ways to incorporate the target weight into thesource weight’s update formula (Eq. 1):Wt+1 =1ZtWt(xi, xj) exp(αt w(xi, xj) (ft(xi)− ft(xj))), (3)Wt+1 =1ZtWt(xi, xj) w(xi, xj) exp(αt(ft(xi)− ft(xj))). (4)Thus, we obtain two versions of Weight-based RankBoost (WRB), namely ex-pWRB corresponding to the target weight inside the exponent (Eq. 3) and lin-WRB corresponding to the linearly combined target weight (Eq. 4).3.1 expWRBThe procedure of expWRB is shown as Algorithm 1. Other than the updatingof source weight in step 5, expWRB also diﬀers from standard RankBoost instep 3 where both source and target weights are used to search for the objectivefunction F to minimize the weighted rank loss.In each round t, we can choose an appropriate αt and ft(x) to minimize Ztin step 3. Zt is minimized by maximizing rt =∑i,j Wt(xi, xj)w(xi, xj)(ft(xj)−ft(xi)) and set αt = 12 ln1+rt1−rt in step 4 [2].Weight-Based Boosting Model 5653.2 linWRBReplacing the updating rule in step 5 in Algorithm 1 with Eq. 4, we can obtainlinWRB with linearly combined target weight. Similarly, we minimize Zt for min-imizing the weighted loss in each round following [2]. Given a binary weak rankerft(x) ∈ {0, 1} and a ∈ {−1, 0,+1}, we set Ra =∑i,j W (xi, xj)w(xi, xj)I(ft(xi)−ft(xj) = a). Then Zt = R+1 exp(αt) + R0 + R−1 exp(−αt). Zt is minimized bysetting αt = 12 lnR−1R+1. The weak ranker ft is selected when Zt is minimal.3.3 Additive Weight-Based RankBoostStandard RankBoost updates source weight in the round t+1 based on the cur-rent weak ranker ft locally (see Eq. 1). A better way is to calculate Wt+1 globallyusing the ensemble function Ft that combines all the weak rankers learned upto the current round like an additive approach [6]. The update rule is givenby Wt+1 = 1Zt exp(Ft(xi) − Ft(xj)) where Ft(x) = Ft−1(x) + αtft(x), whicheliminates the previous round source weight. Then the target weight can beincorporated straightforwardly as Wt+1 = 1Zt w(xi, xj) exp(Ft(xi)− Ft(xj)).However, the model easily overﬁts the source domain due to the great impacton the updating of source weight from the exponential term. We introduce ascaling factor λ to adjust the source weight dynamically. The idea is that weupdate λ considering ranking diﬃculty measured by the proportion of correctlyordered pairs in each round:λt = λt−1 ∗ # of correctly ordered pairs by FtTotal # of pairs to rank (5)In a diﬃcult task where wrong pairs dominate, λ decreases quickly and cancelout the exponential growth of source weight so that target weight can aﬀectweak ranker selection properly.Based on this intuition, we propose the Additive Weight-based RankBoost(addWRB) given as Algorithm 2. A forward stagewise additive approach [6] isused to search for the strong ranking function F . That is, in each round, a weakranker ft is selected and combined with Ft−1 using coeﬃcient αt. The sourceweight is then updated in step 8, where the ensemble function is scaled by λtinside the exponent that is further combined with the target weight linearly.4 Experiments and ResultsEvaluation is done on LETOR3.0 benchmark dataset [5] with the Web trackdocuments of TREC 2003 and 2004. We treat each ranking task, namely HomePage Finding (HP), Named Page Finding (NP) and Topic Distillation (TD) [7] asan individual domain. Generally, it is relatively easier to determine a homepageor named page than an entry point of good websites.We use the same method as [3] to estimate target weights. Note that ranklabels are not used for weighting. The baseline is a RankBoost directly trained566 P. Cai et al.Algorithm 2. Additive Weight-based RankBoost (addWRB)1. Initialize W1 =w(xi,xj)∑i,jw(xi,xj)for all i, j;2. Set λ0 = 1, F0 = 0;3. for t = 1; t ≤ T ; t + + do4. Select weak ranker ft(x) using distribution Wt;5. Set coeﬃcient αt for ft(x);6. Ft(x) = Ft−1(x) + αtft(x);7. Compute λt using Eq. 5;8. For each (xi, xj), update source weight usingWt+1 = 1Zt w(xi, xj) exp(λt(Ft(xi)− Ft(xj)));9. end for10. return F (x) = ∑Tk=1 αkfk(x)Table 1. MAP results of three adaptation tasks. †, ‡ and  indicate signiﬁcantly betterthan baseline, expWRB and linWRB, respectively (95% conﬁdence level).model HP→NP NP→TD TD→NPY2003 Y2004 Y2003 Y2004 Y03-Y04 Y2003 Y2004baseline 0.5834 0.5455 0.1734 0.1657 0.1062 0.4101 0.3061expWRB 0.5481 0.5206 0.1352 0.1437 0.1485† 0.5493† 0.5159†linWRB 0.6245†‡ 0.5824†‡ 0.1755‡ 0.1444 0.1433† 0.3344 0.2239addWRB 0.6280†‡ 0.6025†‡ 0.2139†‡ 0.1505 0.1541† 0.5537† 0.5774†on the source domain without target weight. Always we leveraged on decisionstumps to implement binary weak rankers.We examined HP to NP, NP to TD and TD to NP adaptations to study ifour algorithms can adapt across similar tasks, from easier to more diﬃcult taskand in the reverse case. The MAP results are reported in Table 1.HP to NP Adaptation. We observe that addWRB outperforms all otheralgorithms. T-tests indicate that both addWRB and linWRB are signiﬁcantlybetter than the baseline and expWRB (p < 0.02). This indicates both algorithmscan eﬀectively balance the two types of weights.Note that expWRB failed here. We found that lots of pairs were orderedcorrectly in HP training, resulting in small source weights. So the target weightinside the exponent quickly dominated the updating of source weight and thesame weak ranker was chosen repeatedly. The model becomes not generalizable.NP to TD Adaptation. NP is rather diﬀerent from TD. On 2003 data, ad-dWRB works better than other variants, and t-test indicates that the improve-ments are statistically signiﬁcant (p < 0.001). On 2004 data, all three variantsunderperform the baseline. This is consistent to [3] using other algorithms dueto the shortage of training data from the source domain. Actually, only a halfnumber queries are available in NP04 than NP03. To avoid under-training, weturned to examine NP03 to TD04 adaptation where our algorithms signiﬁcantlyoutperformed the baseline (p < 0.001).TD to NP Adaptation. Here we study how our models can adapt from adiﬃcult task to a simple one. We observe that expWRB and addWRB are sig-niﬁcantly better than the baseline (p < 0.0001) whereas linWRB fails. The targetWeight-Based Boosting Model 567weight aﬀects linWRB little when source pairs are diﬃcult to rank because of theexponential source weights. So the performance is mainly determined by sourceweight leading to the failing adaptation. In contrast, expWRB’s target weightinside the exponent can eﬀectively balance the growth of source weights.The Scaling Factor λ. We also examine the inﬂuence of λ on 2003 data tounveil how λ reacts to diﬀerent problem diﬃculty. We observe that λ in TDto NP is much lower and decreases much faster than NP to TD. Since TD ismore complex where lots of pairs are wrong, λ decreases quickly to balance theexponential growth of source weights.5 ConclusionsWe proposed three variants of weight-based boosting models for ranking adapta-tion based on RankBoost algorithm, namely expWRB, linWRB and addWRB.The challenge is to balance the innate weight distribution of RankBoost andthe target weight introduced for adaptation. expWRB and linWRB incorporatetarget weight in straightforward yet static ways. addWRB uses an additive ap-proach, where the inﬂuence of source weight can be scaled dynamically accordingto the problem diﬃculty. Experiments demonstrate that the performance of ex-pWRB and linWRB varies with the problem diﬃculty of source domain, andaddWRB consistently and signiﬁcantly outperforms the baseline.References1. Amini, M.R., Truong, T.V., Goutte, C.: A boosting algorithm for learning bipartiteranking functions with partially labeled data. In: Proc. of SIGIR (2008)2. Freund, Y., Iyer, R., Schapire, R., Singer, Y.: An eﬃcient boosting algorithm forcombining preferences. Journal of Machine Learning Research (2004)3. Gao, W., Cai, P., Wong, K.-F., Zhou, A.: Learning to rank only using training datafrom related domain. In: Proc. of SIGIR (2010)4. Liu, T.-Y.: Learning to rank for information retrieval. In: Foundations and Trendsin Information Retrieval (2009)5. Qin, T., Liu, T.-Y., Xu, J., Li, H.: LETOR: A benchmark collection for research onlearning to rank for information retrieval. Information Retrieval (2010)6. Hastie, T., Tibshirani, R., Friedman, J.H.: The elements of statistical learning.Springer, Heidelberg (2001)7. Voorhees, E.M.: Overview of TREC 2004. In: Proc. of TREC (2004)8. Xu, J., Li, H.: Adarank: A boosting algorithm for information retrieval. In: Proc. ofSIGIR (2007)9. Zheng, Z., Zha, H., Zhang, T., Chapelle, O., Chen, K., Sun, G.: A general boostingmethod and its application to learning ranking functions for web search. In: Proc.of NIPS (2007)

Weight-based boosting model for cross-domain relevance ranking adaptation

Singapore Management University 
Institutional Knowledge at Singapore Management University 
Research Collection School Of Information 
Systems School of Information Systems 
4-2011 
Weight-based boosting model for cross-domain relevance ranking 
adaptation 
Peng CAI 
Wei GAO 
Singapore Management University, weigao@smu.edu.sg 
Kam-Fai WONG 
Aoying ZHOU 
Follow this and additional works at: https://ink.library.smu.edu.sg/sis_research 
 Part of the Databases and Information Systems Commons 
Citation 
CAI, Peng; GAO, Wei; WONG, Kam-Fai; and ZHOU, Aoying. Weight-based boosting model for cross-domain 
relevance ranking adaptation. (2011). Proceedings of the 33rd European Conference on Information 
Retrieval (ECIR 2011). 562-567. Research Collection School Of Information Systems. 
Available at: https://ink.library.smu.edu.sg/sis_research/4596 
This Conference Proceeding Article is brought to you for free and open access by the School of Information 
Systems at Institutional Knowledge at Singapore Management University. It has been accepted for inclusion in 
Research Collection School Of Information Systems by an authorized administrator of Institutional Knowledge at 
Singapore Management University. For more information, please email library@smu.edu.sg. 
Weight-Based Boosting Model for Cross-Domain
Relevance Ranking Adaptation
Peng Cai1, Wei Gao2, Kam-Fai Wong2,3, and Aoying Zhou1
1 East China Normal University, Shanghai, China
pengcai2010@gmail.com, ayzhou@sei.ecnu.edu.cn
2 The Chinese University of Hong Kong, Shatin, N.T., Hong Kong
{wgao,kfwong}@se.cuhk.edu.hk
3 Key Laboratory of High Confidence Software Technologies, Ministry of Education,
Beijing, China
Abstract. Adaptation techniques based on importance weighting were
shown effective for RankSVM and RankNet, viz., each training instance
is assigned a target weight denoting its importance to the target domain
and incorporated into loss functions. In this work, we extend RankBoost
using importance weighting framework for ranking adaptation. We find
it non-trivial to incorporate the target weight into the boosting-based
ranking algorithms because it plays a contradictory role against the in-
nate weight of boosting, namely source weight that focuses on adjust-
ing source-domain ranking accuracy. Our experiments show that among
three variants, the additive weight-based RankBoost, which dynamically
balances the two types of weights, significantly and consistently outper-
forms the baseline trained directly on the source domain.
1 Introduction
Learning to rank [4] is to derive effective relevance ranking functions based on
a large set of human-labeled data. Boosting has been extensively studied for
learning to rank [1,2,8,9]. However, existing ranking algorithms, including the
boosting-based ones, are only proven effective for data from the same domain.
In real applications, it is prohibitive to annotate training data for every search
domain. Ranking performance may suffer when the training and test have to
take place on different domains.
A promising direction is to learn a cross-domain adaptation model for ranking.
Two key problems should be resolved: (1) how to measure the relatedness of two
domains appropriately; (2) how to utilize this information in ranking algorithms
for adaption. [3] adopted a classification hyperplane to derive the importance
weight of documents in the source domain that reflects their similarity to the
target domain and is then incorporated into the rank loss function.
When applying this method, we find it non-trivial to integrate importance
weight into boosting-based algorithms such as RankBoost [2]. The reason is
 This work is partially supported by NSFC grant (No. 60925008), 973 program (No.
2010CB731402) and 863 program (No. 2009AA01Z150) of China.
P. Clough et al. (Eds.): ECIR 2011, LNCS 6611, pp. 562–567, 2011.
c© Springer-Verlag Berlin Heidelberg 2011
Weight-Based Boosting Model 563
that these algorithms bear an inherently weight-based exponential loss and this
innate weight in the loss function (source weight) plays a contradictory role with
the importance weight introduced for adaptation purpose (target weight). An
appropriate balance must be made between these two types of weight; otherwise
adaptation may fail since the model can easily overfit source data due to the
great impact of source weight on weak rankers selection.
In this work, we develop three Weight-based RankBoost (WRB) algorithms
to balance the source and target weights, namely expWRB, linWRB and ad-
ditive WRB (addWRB). The first two methods incorporate the target weight
in straightforward and static ways, and the third combines the weights from a
global perspective based on a forward stage-wise additive approach [6] to achieve
a dynamic tradeoff. Our results demonstrate that addWRB consistently and sig-
nificantly outperforms the baseline trained directly on the source domain.
2 Target Weight and Source Weight
2.1 Source Weight
RankBoost [2] aims to find a ranking function F to minimize the number of
misordered document pairs. Given document pairs {〈xi, xj〉}, the ranking loss
is defined as rLoss(F ) =
∑
i,j W (xi, xj)I(F (xi) ≥ F (xj)), where W (xi, xj) is
the source weight distribution, I(.) is an binary indicator function, and F (xi) ≥
F (xj) suggests that the ranking function assigns a higher score to xi than to
xj while the ground truth is xi has a lower rating than xj . At each round of
training, W (.) is updated for the next round to focus on those misordered pairs.
The update formula in round t is given as follows:
Wt+1 =
1
Zt
Wt(xi, xj) exp(αt(ft(xi) − ft(xj))) (1)
where ft(x) is the 0-1 valued weak ranker derived from a ranking feature x, αt
is the coefficient of ft so that F =
∑
t αtft(x), and Zt is normalization factor.
Inherently, source weight is designed to control the selection of weak rankers to
minimize ranking errors in source domain.
2.2 Target Weight
In ranking adaptation, the knowledge of relevance judgement should be strength-
ened on those documents that are similar to the target domain so that the learn-
ing can be focused on correctly ranking these important documents. [3] used the
cross-domain similarity to transfer ranking knowledge. The distance of a source-
domain document to the classification hyperplane was calculated as target weight
to measure the importance of the document. Then the pointwise weight was con-
verted to pairwise for compatible with the popular pairwise approach (see [3] for
details). The general loss term was extended as follows:∑
i,j
w(xi, xj) ∗ rLossij(.) (2)
where rLossij(.) is the pairwise loss and w(xi, xj) is the target weight.
564 P. Cai et al.
Algorithm 1. expWRB–Weighted RankBoost with target weight inside the exponent
Input: Query-document set of source domain; Target weights of M document pairs {w(xi, xj)}M1
based on the ground truth.
Output: Ranking function F (x).
1. Initialize W1(xi, xj) = 1M for all i, j;
2. for t = 1; t ≤ T ; t + + do
3. Select weak ranker ft(x) using Wt and w;
4. Set coefficient αt for ft(x);
5. For each (xi, xj), update source weight using
Wt+1 = 1Zt Wt(xi, xj) exp(αtw(xi, xj)(ft(xi) − ft(xj)));
6. end for
7. return F (x) = ∑Tt=1 αtft(x)
3 Weight-Based Boosting Models for Ranking Adaptation
In standard RankBoost, the source weight is updated iteratively so that the
weak rankers can focus on those misordered pairs with large source weight that
commonly reside near the decision boundary deemed more difficult to order.
However, these pairs may be unnecessarily important to the target domain.
Meanwhile, for those misordered pairs with low source weight, even though they
contain some important cross-domain ranking knowledge (i.e., having high target
weight), the algorithm does not prioritize to correct their ranking. The two types
of weight play contradictory roles and must be appropriately balanced.
The objective of our adaptive RankBoost is to minimize the weighted rank-
ing loss wLoss(F ) =
∑
i,j W (xi, xj)I(F (xi) ≥ F (xj))w(xi, xj) following Eq. 2.
There are two straightforward ways to incorporate the target weight into the
source weight’s update formula (Eq. 1):
Wt+1 =
1
Zt
Wt(xi, xj) exp(αt w(xi, xj) (ft(xi) − ft(xj))), (3)
Wt+1 =
1
Zt
Wt(xi, xj) w(xi, xj) exp(αt(ft(xi) − ft(xj))). (4)
Thus, we obtain two versions of Weight-based RankBoost (WRB), namely ex-
pWRB corresponding to the target weight inside the exponent (Eq. 3) and lin-
WRB corresponding to the linearly combined target weight (Eq. 4).
3.1 expWRB
The procedure of expWRB is shown as Algorithm 1. Other than the updating
of source weight in step 5, expWRB also differs from standard RankBoost in
step 3 where both source and target weights are used to search for the objective
function F to minimize the weighted rank loss.
In each round t, we can choose an appropriate αt and ft(x) to minimize Zt
in step 3. Zt is minimized by maximizing rt =
∑
i,j Wt(xi, xj)w(xi, xj)(ft(xj)−
ft(xi)) and set αt = 12 ln
1+rt
1−rt in step 4 [2].
Weight-Based Boosting Model 565
3.2 linWRB
Replacing the updating rule in step 5 in Algorithm 1 with Eq. 4, we can obtain
linWRB with linearly combined target weight. Similarly, we minimize Zt for min-
imizing the weighted loss in each round following [2]. Given a binary weak ranker
ft(x) ∈ {0, 1} and a ∈ {−1, 0, +1}, we set Ra =
∑
i,j W (xi, xj)w(xi, xj)I(ft(xi)−
ft(xj) = a). Then Zt = R+1 exp(αt) + R0 + R−1 exp(−αt). Zt is minimized by
setting αt = 12 ln
R−1
R+1
. The weak ranker ft is selected when Zt is minimal.
3.3 Additive Weight-Based RankBoost
Standard RankBoost updates source weight in the round t+1 based on the cur-
rent weak ranker ft locally (see Eq. 1). A better way is to calculate Wt+1 globally
using the ensemble function Ft that combines all the weak rankers learned up
to the current round like an additive approach [6]. The update rule is given
by Wt+1 = 1Zt exp(Ft(xi) − Ft(xj)) where Ft(x) = Ft−1(x) + αtft(x), which
eliminates the previous round source weight. Then the target weight can be
incorporated straightforwardly as Wt+1 = 1Zt w(xi, xj) exp(Ft(xi) − Ft(xj)).
However, the model easily overfits the source domain due to the great impact
on the updating of source weight from the exponential term. We introduce a
scaling factor λ to adjust the source weight dynamically. The idea is that we
update λ considering ranking difficulty measured by the proportion of correctly
ordered pairs in each round:
λt = λt−1 ∗
# of correctly ordered pairs by Ft
Total # of pairs to rank
(5)
In a difficult task where wrong pairs dominate, λ decreases quickly and cancel
out the exponential growth of source weight so that target weight can affect
weak ranker selection properly.
Based on this intuition, we propose the Additive Weight-based RankBoost
(addWRB) given as Algorithm 2. A forward stagewise additive approach [6] is
used to search for the strong ranking function F . That is, in each round, a weak
ranker ft is selected and combined with Ft−1 using coefficient αt. The source
weight is then updated in step 8, where the ensemble function is scaled by λt
inside the exponent that is further combined with the target weight linearly.
4 Experiments and Results
Evaluation is done on LETOR3.0 benchmark dataset [5] with the Web track
documents of TREC 2003 and 2004. We treat each ranking task, namely Home
Page Finding (HP), Named Page Finding (NP) and Topic Distillation (TD) [7] as
an individual domain. Generally, it is relatively easier to determine a homepage
or named page than an entry point of good websites.
We use the same method as [3] to estimate target weights. Note that rank
labels are not used for weighting. The baseline is a RankBoost directly trained
566 P. Cai et al.
Algorithm 2. Additive Weight-based RankBoost (addWRB)
1. Initialize W1 =
w(xi,xj)∑
i,j
w(xi,xj)
for all i, j;
2. Set λ0 = 1, F0 = 0;
3. for t = 1; t ≤ T ; t + + do
4. Select weak ranker ft(x) using distribution Wt;
5. Set coefficient αt for ft(x);
6. Ft(x) = Ft−1(x) + αtft(x);
7. Compute λt using Eq. 5;
8. For each (xi, xj), update source weight using
Wt+1 = 1Zt w(xi, xj) exp(λt(Ft(xi) − Ft(xj)));
9. end for
10. return F (x) = ∑Tk=1 αkfk(x)
Table 1. MAP results of three adaptation tasks. †, ‡ and 
 indicate significantly better
than baseline, expWRB and linWRB, respectively (95% confidence level).
model HP→NP NP→TD TD→NPY2003 Y2004 Y2003 Y2004 Y03-Y04 Y2003 Y2004
baseline 0.5834 0.5455 0.1734 0.1657 0.1062 0.4101 0.3061
expWRB 0.5481 0.5206 0.1352 0.1437 0.1485† 0.5493† 0.5159†
linWRB 0.6245†‡ 0.5824†‡ 0.1755‡ 0.1444 0.1433† 0.3344 0.2239
addWRB 0.6280†‡ 0.6025†‡ 0.2139†‡ 0.1505 0.1541† 0.5537† 0.5774†
on the source domain without target weight. Always we leveraged on decision
stumps to implement binary weak rankers.
We examined HP to NP, NP to TD and TD to NP adaptations to study if
our algorithms can adapt across similar tasks, from easier to more difficult task
and in the reverse case. The MAP results are reported in Table 1.
HP to NP Adaptation. We observe that addWRB outperforms all other
algorithms. T-tests indicate that both addWRB and linWRB are significantly
better than the baseline and expWRB (p < 0.02). This indicates both algorithms
can effectively balance the two types of weights.
Note that expWRB failed here. We found that lots of pairs were ordered
correctly in HP training, resulting in small source weights. So the target weight
inside the exponent quickly dominated the updating of source weight and the
same weak ranker was chosen repeatedly. The model becomes not generalizable.
NP to TD Adaptation. NP is rather different from TD. On 2003 data, ad-
dWRB works better than other variants, and t-test indicates that the improve-
ments are statistically significant (p < 0.001). On 2004 data, all three variants
underperform the baseline. This is consistent to [3] using other algorithms due
to the shortage of training data from the source domain. Actually, only a half
number queries are available in NP04 than NP03. To avoid under-training, we
turned to examine NP03 to TD04 adaptation where our algorithms significantly
outperformed the baseline (p < 0.001).
TD to NP Adaptation. Here we study how our models can adapt from a
difficult task to a simple one. We observe that expWRB and addWRB are sig-
nificantly better than the baseline (p < 0.0001) whereas linWRB fails. The target
Weight-Based Boosting Model 567
weight affects linWRB little when source pairs are difficult to rank because of the
exponential source weights. So the performance is mainly determined by source
weight leading to the failing adaptation. In contrast, expWRB’s target weight
inside the exponent can effectively balance the growth of source weights.
The Scaling Factor λ. We also examine the influence of λ on 2003 data to
unveil how λ reacts to different problem difficulty. We observe that λ in TD
to NP is much lower and decreases much faster than NP to TD. Since TD is
more complex where lots of pairs are wrong, λ decreases quickly to balance the
exponential growth of source weights.
5 Conclusions
We proposed three variants of weight-based boosting models for ranking adapta-
tion based on RankBoost algorithm, namely expWRB, linWRB and addWRB.
The challenge is to balance the innate weight distribution of RankBoost and
the target weight introduced for adaptation. expWRB and linWRB incorporate
target weight in straightforward yet static ways. addWRB uses an additive ap-
proach, where the influence of source weight can be scaled dynamically according
to the problem difficulty. Experiments demonstrate that the performance of ex-
pWRB and linWRB varies with the problem difficulty of source domain, and
addWRB consistently and significantly outperforms the baseline.
References
1. Amini, M.R., Truong, T.V., Goutte, C.: A boosting algorithm for learning bipartite
ranking functions with partially labeled data. In: Proc. of SIGIR (2008)
2. Freund, Y., Iyer, R., Schapire, R., Singer, Y.: An efficient boosting algorithm for
combining preferences. Journal of Machine Learning Research (2004)
3. Gao, W., Cai, P., Wong, K.-F., Zhou, A.: Learning to rank only using training data
from related domain. In: Proc. of SIGIR (2010)
4. Liu, T.-Y.: Learning to rank for information retrieval. In: Foundations and Trends
in Information Retrieval (2009)
5. Qin, T., Liu, T.-Y., Xu, J., Li, H.: LETOR: A benchmark collection for research on
learning to rank for information retrieval. Information Retrieval (2010)
6. Hastie, T., Tibshirani, R., Friedman, J.H.: The elements of statistical learning.
Springer, Heidelberg (2001)
7. Voorhees, E.M.: Overview of TREC 2004. In: Proc. of TREC (2004)
8. Xu, J., Li, H.: Adarank: A boosting algorithm for information retrieval. In: Proc. of
SIGIR (2007)
9. Zheng, Z., Zha, H., Zhang, T., Chapelle, O., Chen, K., Sun, G.: A general boosting
method and its application to learning ranking functions for web search. In: Proc.
of NIPS (2007)


https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=5599&amp;context=sis_research

Weight-based boosting model for cross-domain relevance ranking adaptation

Abstract

Similar works

Full text

Available Versions

Crossref

Institutional Knowledge at Singapore Management University

Institutional Knowledge at Singapore Management University