Machine Lea ning Me hods o Ranking in In o ma ion Re ie al: When
LambdaRank G adien Incohe ency Leads o Un ai T aining
Fede ico Ma cuzzi, Claudio Lucchese and Sal a o e O lando
iNEST Spoke 6 “Tou ism, Cul u e and C ea i e Indus ies” RT1: Rice ca Indus iale (RI).
Sub. RT1.3: In eg a ed da a pla o ms o suppo s a egic decision-making and
empowe a ele s and isi o s.
Abs ac
In o ma ion Re ie al sys ems a e widely used in a as numbe o scena ios, such as
ou ism, educa ion, esea ch, e c. In pa icula , o ou ism applica ions, in o ma ion sys-
ems such as web sea ch engines and ecommenda ion sys ems a e used o allow a use
(i.e., he ou is ) o look o ou is des ina ions, hospi ali y acili ies, and gene al e i o ial
in o ma ion. Nowadays, he co e o in o ma ion sys ems is machine lea ning, in pa icula ,
lea ning o ank in he case o web sea ch engines. Lea ning o ank lea ns a anke ha ,
gi en a use ’s needs (i.e., he que y), e u ns a lis o esul s o de ed by ele ance o he
que y, wi h he mos ele an esul s a he op o he e u ned lis . One o he mos e ec i e
and e icien app oaches o lea ning o ank is he well-known LambdaMART algo i hm.
LambdaMART de ines a g adien o each documen w. . . he in o ma ion e ie al me -
ic o be op imised, he so-called lambdas. In ui i ely, each lambda desc ibes, wi h some
deg ee o app oxima ion, how much a documen sco e should be pushed up o down o
imp o e he anking. In his wo k, we show ha lambdas can be incohe en w. . . he
me ic op imised: a documen wi h high ele ance o he use que y can ecei e a down-
wa d push la ge han a documen wi h lowe ele ance. In addi ion, op imising unca ed
me ics in o de o speed up he aining ime can exace ba e his disc epancy and lead o
wo se model lea ning. This beha iou goes a beyond he expec ed deg ee o app oxima-
ion. Mo eo e , we show how he p ac ice o unca ed me ic op imisa ion in oduces an
unwan ed un ai compa ison be ween i ems ( acili ies, accommoda ion, e c.) equally el-
e an o he use que y. We analyse he idiosync asies o LambdaMART g adien s, and
we in oduce some s a egies o emo e he exace ba ion o g adien incohe ency and he
un ai i em compa ison. We empi ically demons a e on publicly a ailable da ase s ha
he p oposed app oach leads o a ai e aining p ocess and o models ha can achie e
s a is ically signi ican imp o emen s in e ms o NDCG.
1 In oduc ion
In o ma ion Re ie al (IR) is he ield o esea ch ocused on e ie ing use ul in o ma ion
om a huge collec ion o da a. Se e al a eas ela ed o compu e science ha e de eloped
IR s a egies.
Two o he mos amous and widely used applica ions o IR s a egies a e Web Sea ch
Engines and Recommenda ion Sys ems. Nowadays, web sea ch engines and ecom-
menda ion sys ems a e used in a la ge a ie y o scena ios, such as esea ch, educa ion,
en e ainmen , ou ism, e c. In he con ex o ou ism, web sea ch engines and ecommen-
da ion sys ems a e used o book accommoda ion, look o ac i i ies and cul u al e en s in
a speci ic ou is a ea, o ind gene al e i o ial in o ma ion o ou is s.
Nowadays, hese in o ma ion e ie al sys ems make ex ensi e use o machine lea n-
ing and deep lea ning app oaches o pe o m he a o emen ioned asks. Among hese
app oaches, he e is lea ning o Rank (LTR), one o he mos applied app oaches in IR.
LTR is a machine lea ning app oach ha includes supe ised lea ning algo i hms o con-
s uc anke s ha sa is y use needs by so ing by ele ance a huge se o i ems ha
mos ly con ain i ele an i ems and only a small ac ion o ele an ones. In o he wo ds,
a anke akes as inpu a use que y (e.g., “accommoda ion in Venice”) and e u ns a sho
so ed lis o i ems ele an o he que y (e.g., a lis o accommoda ion in Venice). These
i ems a e ex ac ed om a huge se o i ems (e.g., he Wo ld Wide Web) ha a e mos ly
non- ele an (e.g., on he en i e Wo ld Wide Web he e a e many mo e accommoda ions
ou side Venice han in Venice).
In o de o p o ide ele an esul s, LTR algo i hms op imise in o ma ion e ie al me -
ics. Howe e , mos o he in o ma ion e ie al me ics a e anked-based (i.e., hey ely
on he o de o he i ems), so hey a e discon inuous o la e e ywhe e. Consequen ly,
g adien descen s a egies o op imise such me ics can no be applied di ec ly.
Despi e he non-di e en iabili y o IR me ics, many LTR algo i hms a e g adien -based,
which ei he op imise an app oxima e e sion o he anking me ic o build g adien s based
on heu is ics as in he case o LambdaMART algo i hm de ined by Bu ges (2010). Lamb-
daMART op imises a non-di e en iable loss unc ion ha i doesn’ know by c ea ing an
ad-hoc g adien o each documen based on heu is ics ha ake in o accoun he con i-
bu ion o he documen in he ank and he ela ionship i has wi h he o he documen s.
Since LambdaMART is based on heu is ics, i s g adien s a e no exac . In his wo k, we
wan o show how LambdaMART and de i a i es such as LambdaRank by Bu ges e al.
(2006) and he me ic-d i en loss unc ions de ined by Wang e al. (2018) ha e an inhe en
p oblem due o he heu is ics used and can gene a e incohe encies in g adien s.
Mo eo e , machine lea ning algo i hms (and consequen ly lea ning o ank algo i hms,
oo) can p o ide un ai ness o ce ain g oups o use s, i ems, e c. The un ai ness co -
e s di e en scena ios, such as disc imina ion by gende and ace, un ai exposu e o
he same i em om di e en endo s in e-comme ce, e c. Un ai ness in machine lea ning
can be caused by di e en aspec s, such as bias in he aining se ha disad an ages
one g oup o e ano he (e.g., black people a e disc imina ed mo e han non-back people),
limi ed a ailabili y o esou ces (e.g., one posi ion a ailable bu en equally ele an candi-
da es o he posi ion), o he in insic design o he machine lea ning algo i hm (e.g., no all
he aining ins ances a e equally conside ed du ing he lea ning phase o he algo i hm).
We ocus on he las aspec ha gene a es un ai ness in in o ma ion e ie al sys ems.
In his wo k, we disco e ed h ee unexpec ed beha iou s in he well-known Lamb-
daRank algo i hm and i s de i a i es, such as LambdaMART. i) we disco e ed ha Lamb-
daMART a e a ec ed by g adien incohe encies ha comp omise he model lea ning. A
g adien incohe ency is when an i em wi h high ele ance o he que y ecei es a down-
wa d push la ge han an i em wi h lowe ele ance. ii) we disco e ed ha he p ac ice
o op imise unca ed in o ma ion e ie al me ics (i.e., me ics op imised only in he op-k
posi ion (i.e., hose o in e es o he use s) o educe he aining ime (i.e., o imp o e
he algo i hm e iciency) exace ba e he phenomena o g adien incohe ency ha leads o
wo se model lea ning. iii) we disco e ed ha unca ed me ic op imisa ion in oduces an
undesi ed un ai compa ison o i ems/documen s in he aining se . In pa icula , we dis-
co e ed ha documen s ha a e equally ele an o a que y a e ea ed di e en ly, o he
de imen o equally ele an documen s posi ioned lowe down in he anking and which
would mos need o be pushed upwa d.
The main con ibu ion o his wo k is Lambda-eX an imp o emen o LambdaMART
objec i e unc ion, ha ex ends he se o documen pai s p ocessed du ing aining, o op-
imise unca ed anking me ics (i.e., a mo e e icien aining) while a oiding un ai docu-
men compa ison and he exace ba ion in he numbe o g adien s incohe encies. We em-
pi ically demons a ed h ough i e publicly a ailable da ase s ha Lambda-eX achie es
s a is ically signi ican imp o emen s in e ms o NDCG han he baselines. Mo eo e ,
Lambda-eX emo es he exace ba ion o he g adien incohe encies in oduced by un-
ca ed me ic op imisa ion ha comp omise he exposu e o documen s wi h espec o
equally ele an documen s wi hou g adien incohe encies.
2 G adien Incohe ency and Un ai Documen Compa ison
G adien -based lea ning algo i hms, such as a i icial neu al ne wo ks o g adien -boos ed
decision ees, un i e a i e upda es o build a anke ha minimises a gi en cos unc ion C.
Fo ins ance, g adien -boos ed decision ees i e a i ely lea n a new ee ha app oxima es
∂C/∂si o each documen diin he aining se Dand i s sco e si. Un o una ely, mos IR
me ics a e ank-based: hey depend on anking π a he han on si. This makes he cos
unc ion ei he la o non-di e en iable. No e ha πis he anking o e he documen s
di∈Dso ed in dec easing o de o sco es sip edic ed by he anke , and π[i]deno es
he posi ion o documen diin he anking.
LambdaRank’s cos unc ion de ined by Bu ges e al. (2006) is one o he mos ele an
app oaches used o ackle his p oblem, and i s ems om he RankNe cos p oposed by
Bu ges e al. (2005), which is enhanced by conside ing he impac on he IR me ic. The
g adien is compu ed on he basis o pai -wise lambdas λij as ollows:
λi=∑
j:(i,j)∈I
λij −∑
k:(k,i)∈I
λki (1)
whe e Iis he se o o de ed documen s pai s (i, j)such ha yi> yj, i.e., I={(i, j)|
di, dj∈D∧yi> yj}. The alue o λij es ima es he change on he cos unc ion Cwhen
he dis ance be ween he wo sco es siand sjis modi ied.
To manage use beha iou and inc ease aining e iciency, eal-wo ld applica ions o
in o ma ion e ie al sys ems mos ly y o op imise he e ec i eness only o he i s k
esul s. IR me ics na u ally p o ide a unca ed e sion, i.e. NDCG@kis compu ed by
conside ing only he con ibu ion o he op-k anked documen s.
By aining he model o op imise a unca ed me ic Z o a ce ain unca ion le el τ,
pai s o documen s anked beyond τa e no conside ed since he co esponding con i-
bu ion o he me ic is equal o 0. Thus, in o de o educe he aining ime, he numbe o
documen pai s in Iis limi ed while compu ing he g adien s λiin Equa ion 1 by eplacing
he se Iwi h Iτ={(i, j)|di, dj∈D∧yi> yj∧min(π[i], π[j]) ≤τ}.
Table 1: De ailed compu a ion o LambdaMART g adien s.
diπ[i]yisiλi
d11 4 0.02 λ1=λ12 +λ13 ≈0.176 + 0.221 ≈0.397
d22 0 0.01 λ2=−λ12 −λ32 ≈ −0.176 −0.004 ≈ −0.180
d33 1 0.00 λ3=−λ13 +λ32 ≈ −0.221 + 0.004 ≈ −0.217
I is impo an o no e ha , al hough closely ela ed, he unca ion le el τis di e en
om he me ic cu o k. The o me a ec s he numbe o documen pai s o p ocess, and
he la e a ec s he e alua ion o he me ic. Mo eo e , hey may no be equal, i.e. τmay
be sligh ly la ge han k o p ocess mo e pai s du ing he aining phase.
Table 1 shows an example o LambdaMART g adien s when maximising NDCG. The
que y has only h ee documen s wi h hei anks π[i]and sco es sip edic ed by he model,
and ele ance label yi. The op- anked documen wi h ele ance equal o 4 and is co -
ec ly pushed up by he g adien λ1. In e es ingly enough, he second and hi d documen s
a e mis anked wi h labels 0 and 1 espec i ely. The LambdaMART g adien is nega i e
o bo h documen s, bu he documen wi h he la ge label is pushed down wi h g ea e
s eng h. We may conclude ha such g adien s a e no going o imp o e he anking bu
a he inc ease he gap be ween he wo mis anked documen s. We call his phenomenon
g adien s incohe ency.
To explain in de ail he eason o such beha iou , in Table 1 we epo he compu a ion
o he documen g adien s λias a unc ion o he pai -wise λij acco ding o Equa ion 1 in
case o he NDCG me ic. Documen d1has a posi i e g adien λ1as i is anked highe han
documen s wi h smalle ele ance labels. Documen d2is he leas ele an and ecei es
a nega i e g adien con ibu ion om bo h he o he documen s. Unexpec edly, documen
d3 ecei es he s onges downwa d push e en i i has a highe label han d2. The eason
is ha swapping documen d1wi h d3has a la ge impac on he NDCG han swapping d1
wi h d2, esul ing in λ13 > λ12. LambdaMART p e e s a oiding he isk o mo ing d1 o he
hi d posi ion a he han pushing d3up o he second place. Indeed, his comes om he
discoun ac o o NDCG me ic ha demo es documen s’ con ibu ions in he lowe anks.
These g adien s clea ly push he anking away om he ideal con igu a ion as we would
p e e ha ing λ3la ge han λ2.
The phenomenon o g adien incohe encies is signi ican ly u he exace ba ed when
op imising unca ed me ics. This is because by emo ing (i, j)pai s om se I, he λi
g adien s o he ele an documen s unde τwill be incomple e since hey will lose some
λij. As a esul , ele an documen s below τwill ecei e e en less upwa d push.
Mo eo e , he subs i u ion o he se Iwi h Iτin oduces an un ai op imisa ion p ocess
among equally ele an documen s. In ac , while op imising unca ed me ic, wo equally
ele an documen s can ecei e a di e en numbe o λij con ibu ions, no expe ienced
du ing un- unca ed me ic op imisa ion.
In Figu e 1, i is possible o see a clea example o un ai documen compa ison in o-
duced by unca ed me ic op imisa ion. In he example, a unca ed me ic op imisa ion
is o ced by implying τ= 1. When unca ed me ic op imisa ion is used he documen
in posi ion 1wi h ele ance 2 ecei es mo e con ibu ion om he o he documen s han
he documen in posi ion 2while being equally ele an . In ac , he documen in posi ion
2
0
0
0
2
1
2
3
4
5
LambdaRank
wi h
0.04
0.03
0.02
0.01
0.00
2
0
0
0
0.03
0.02
0.01
0.00
2
1
2
3
4
5
LambdaRank
wi h
0.04
Figu e 1: Examples o un ai documen compa ison. When unca ed me ic op imisa ion
is used (e.g., τ= 1), he documen in posi ion 1wi h ele ance 2 ecei es mo e con ibu ion
om he o he documen s han he documen in posi ion 2while being equally ele an .
2 ecei es con ibu ion λij= 0 om all documen s below he unca ion le el, while he
documen in posi ion 1 ecei es con ibu ion λij≥0.
3 Lambda-eX
The main con ibu ion o his wo k is Lambda-eX (Ma cuzzi e al. 2023), a lea ning algo-
i hm able o cancel he g adien incohe ency exace ba ion and a oid he in oduc ion o
un ai documen compa isons. We claim ha he exace ba ion o he incohe encies and
he un ai compa isons a e due o missing compu a ions o he λij g adien s. Mo e speci i-
cally, ele an documen s ha a e no anked abo e he unca ion le el a e no e alua ed
agains all he candida e documen s o he que y bu only agains he op-k, and his ig-
no es some o he λij and unde -es ima es hei g adien .
To keep he aining ocused on use beha iou and ackle he p oblem o g adien
incohe encies and un ai documen compa ison, we designed Lambda-eX, which ex ends
he se o documen pai s conside ed by LambdaMART when compu ing g adien s. To
achie e his goal, we de ine a se o documen s X⊆Dso as o include in Xall he
documen s o which we wan a comple e g adien es ima ion, i.e., hose ha dese e
o no o be anked a he op posi ions. Mo eo e , he se Xcon ains he documen s
o which we wan a ai compa ison. As men ioned, Xis a subse o D, so no all he
documen s in Da e p o ided wi h a ai compa ison. Howe e , due o a limi ed numbe o
posi ions e ec i ely iewed by he use s, i.e., he op-kposi ions, we do no equi e ha
all documen s ha e a ai compa ison, bu jus hose ha should be anked in he op-k
posi ion o maximise he use need, i.e., he ele ance o he use que y.
Lambda-eX hus compu es each λig adien as in Equa ion 1 bu basing on he se :
IX={(i, j)|di, dj∈D∧yi> yj∧(di∈X∨dj∈X)}
How does Lambda-eX popula e he se X? Le kbe he cu o o he IR me ic being
op imised. Lambda-eX includes in Xall he documen s anked in he op-kposi ions by he
cu en model and he con ende documen s below he cu o k. The con ende documen s
a e hose no placed in he op-kposi ions bu equi ed o maximise he anking me ic.
Since he numbe o con ende documen s could be la ge, o limi he size o X o abou
k, Lambda-eX uses some c i e ia o selec he con ende documen s o include in X. We
p opose h ee di e en ways o selec he con ende documen s.
•s a ic: le hbe he numbe o documen s ha a e equi ed o maximise he me ic
Z, bu he model did no ank among he op-kposi ions. This s a egy adds in X
he hmos ele an documen s no ye in he op-k. This s a egy p o ides a pa -
ial ai documen compassion since i compa es he minimal numbe o documen s
ha dese e o be in he op-kin o de minimise he aining ime, i.e., he aining
e iciency.
• andom: analogous o s a ic, bu ies a e b oken andomly ins ead o ank-based.
A andom selec ion allows he model o be ai e since i compa es all con ende s
du ing aining and imp o es model gene alisa ion.
•all: analogous o s a ic, bu ies a e no b oken. I he e a e mo e han hdocu-
men s wi h he desi ed ele an labels, hey a e all included in X. This enhances
gene alisa ion and ai ness a he expense o e iciency. Wi h he all s a egy, all
documen s ha should be anked in he op-ka e compa ed wi h he o he s, so p o-
iding he ai es s a egy among he h ee.
We also p opose wo hyb id a ian s: all-s a ic and all- andom. The goal is o limi
he size o X. Depending on he que y, he all may po en ially include all he ele an
documen s. To a oid such blow-up o X, he hyb id s a egies oll back o ei he s a ic o
andom in his degene a e case; o he wise, hey implemen he all s a egy.
To e alua e he e ec i eness o he p oposed app oach, we de ined wo baselines:
LambdaMART ha op imise he unca ed e sion o NDCG me ic wi h unca ion le el τ
equal o he me ic cu o kand he o he wi h τ=k+3 as sugges ed in Co po a ion (2023).
Table 2: S a is ically signi ican imp o emen w. . . LambdaMARTτ=k+3 acco ding o
Fishe ’s andomisa ion es de ined by Fishe (1935) (wi h a wo-sided p- alue) a e ma ked
wi h bold (p= 0.05) and bold-i alic (p= 0.01).
da ase NDCG@k LambdaMART Lambda-eX
kτ=k τ =k+3 s a ic andom all all-s a ic all- andom
Is ella-X 5 73.32 75.35** 75.19** 75.17** 75.15** 75.19** 75.17**
Is ella-S 5 70.19 70.64* 70.67** 70.71** 70.55 70.65* 70.64*
Is ella-F 5 67.02 67.62** 67.55** 67.67** 67.50** 67.68** 67.71**
Yahoo! Se 1 5 75.35 75.85** 75.67 75.59 75.63 75.73 75.67
MSLR-30K 5 50.66 51.22 50.95 50.96 51.24 51.42* 51.38
Is ella-X 10 77.53 78.61 78.61 78.61 78.61 78.61 78.61
Is ella-S 10 76.35 76.71** 76.66** 76.70** 76.69** 76.72** 76.70**
Is ella-F 10 71.85 72.39** 72.42** 72.46** 72.42** 72.35** 72.46**
Yahoo! Se 1 10 79.62 79.84 79.66 79.75 79.78 79.81 79.80
MSLR-30K 10 52.66 52.98 52.96 53.08 53.23* 53.19 53.14
Is ella-X 15 79.00 79.45 79.44 79.48 79.44 79.44 79.48
Is ella-S 15 80.63 80.73* 80.69 80.71* 80.75** 80.80** 80.73*
Is ella-F 15 75.46 75.87** 75.94** 75.90** 75.92** 76.00** 76.00**
Yahoo! Se 1 15 82.01 82.03 81.94 82.07 82.04 82.07 82.04
MSLR-30K 15 54.60 54.67 54.82 54.93** 54.84 54.75 54.83
We compa ed he baselines wi h each e sion o Lambda-eX de ined in Sec ion 3. In Table
2, we summa ise he pe o mance o each model. The esul s show how Lambda-eX
ob ains s a is ical signi icance imp o emen s by o e coming he exace ba ion in g adien s
incohe encies in oduced by LambdaMART when op imising a unca ed me ic.
4 Conclusion
We ha e shown ha LambdaMART is a ec ed by incohe encies in he compu a ion o
he g adien s, exace ba ed by he unca ion le el. We showed ha he aining e iciency
in oduced by unca ed me ic op imisa ion comes a he expanses o ai ness be ween
equally ele an documen s. We designed a new app oach called Lambda-eX o coun e
hese issues while keeping he ocus on he me ic o be op imised and he aining e i-
ciency. Th ough ex ensi e expe imen s, we ha e shown ha Lambda-eX is able o achie e
a s a is ically signi ican imp o emen wi h espec o he baselines.
Re e ences
Bu ges, C. J. C. (2010), ‘F om ankne o lambda ank o lambdama : An o e iew’.
Bu ges, C. J. C., Ragno, R. & Le, Q. V. (2006), Lea ning o ank wi h nonsmoo h cos
unc ions, in B. Schölkop , J. C. Pla & T. Ho mann, eds, ‘Ad ances in Neu al In o ma ion
P ocessing Sys ems 19, P oceedings o he Twen ie h Annual Con e ence on Neu al
In o ma ion P ocessing Sys ems, Vancou e , B i ish Columbia, Canada, Decembe 4-7,
2006’, MIT P ess, pp. 193–200.
Bu ges, C. J. C., Shaked, T., Renshaw, E., Lazie , A., Deeds, M., Hamil on, N. & Hullende ,
G. N. (2005), Lea ning o ank using g adien descen , in L. D. Raed & S. W obel, eds,
‘Machine Lea ning, P oceedings o he Twen y-Second In e na ional Con e ence (ICML
2005), Bonn, Ge many, Augus 7-11, 2005’, Vol. 119 o ACM In e na ional Con e ence
P oceeding Se ies, ACM, pp. 89–96.
Co po a ion, M. (2023), Ligh GBM Release 3.3.3.99.
Fishe , R. (1935), The design o expe imen s. 1935, Oli e and Boyd, Edinbu gh.
Ma cuzzi, F., Lucchese, C. & O lando, S. (2023), Lambda ank g adien s a e incohe en ,
in I. F ommholz, F. Hop ga ne , M. Lee, M. Oakes, M. Lalmas, M. Zhang & R. L. T.
San os, eds, ‘P oceedings o he 32nd ACM In e na ional Con e ence on In o ma ion
and Knowledge Managemen , CIKM 2023, Bi mingham, Uni ed Kingdom, Oc obe 21-
25, 2023’, ACM, pp. 1777–1786.
URL: h ps://doi.o g/10.1145/3583780.3614948
Wang, X., Li, C., Golbandi, N., Bende sky, M. & Najo k, M. (2018), The lambdaloss ame-
wo k o anking me ic op imiza ion, in A. Cuzzoc ea, J. Allan, N. W. Pa on, D. S i as-
a a, R. Ag awal, A. Z. B ode , M. J. Zaki, K. S. Candan, A. Lab inidis, A. Schus e &
H. Wang, eds, ‘P oceedings o he 27 h ACM In e na ional Con e ence on In o ma ion
and Knowledge Managemen , CIKM 2018, To ino, I aly, Oc obe 22-26, 2018’, ACM,
pp. 1313–1322.
