Compu e s & G aphics 124 (2024) 104052
Con en s lis s a ailable a ScienceDi ec
Compu e s & G aphics
jou nal homepage: www.else ie .com/loca e/cag
Special Sec ion on 3DOR2024
ShapeBench: A new app oach o benchma king local 3D shape
desc ip o s
Ba I e an Blokland
Sem Sælands ei 9, Gløshaugen, 7034 T ondheim, No way
ARTICLE INFO
Keywo ds:
3D local shape desc ip o s
ShapeBench
Benchma k
ABSTRACT
The ShapeBench e alua ion me hodology is p oposed as an ex ension o he popula A ea Unde P ecision-
Recall Cu e (PRC/AUC) o measu ing he ma ching pe o mance o local 3D shape desc ip o s. I is obse ed
ha he PRC inadequa ely accoun s o o he simila su aces in he same o di e en objec s when de e mining
whe he a candida e ma ch is a ue posi i e. The no el Desc ip o Dis ance Index (DDI) me ic is in oduced
o add ess his limi a ion. In con as o p e ious e alua ion me hodologies, which iden i y en i e objec s in
a gi en scene, he DDI me ic measu es desc ip o pe o mance by analysing poin - o-poin dis ances. The
ShapeBench me hodology is also mo e scalable han p e ious app oaches, by using p ocedu al gene a ion.
The benchma k is used o e alua e bo h old and new desc ip o s. The esul s p oduced by he implemen a ion
o he benchma k a e ully eplicable, and a e made publicly a ailable.
1. Mo i a ion
The abili y o compa e he simila i y o 3D su aces is c ucial in a
numbe o applica ions, such as 3D egis a ion [1], bin picking [2],
Simul aneous Localisa ion and Mapping (SLAM) [3] and 3D objec
e ie al [4]. A wide a ie y o me hods ha e been p oposed, bo h in
he o m o adi ional algo i hms [5,6] and, mo e ecen ly, lea ned
ea u es [7,8]. E alua ing he pe o mance o 3D su ace ma ching
me hods p o ides unde s anding o hei s eng hs and weaknesses, and
is hus c ucial o de e mining hei p ac ical applicabili y.
This pape ocuses on imp o ing he P ecision Recall Cu e (PRC)
— he mos popula me hodology o e alua ing local 3D shape de-
sc ip o me hods — along wi h i s associa ed A ea Unde Cu e (AUC𝑝𝑟)
me ic [9–11]. Howe e , i s applica ion domain ex ends o any su ace
poin ma ching algo i hm. The AUC𝑝𝑟 me ic measu es he ex en o
which a pa icula me hod can co ec ly iden i y models in a se o
scenes. The se o models 𝑀con ains known objec s. Subse s o models
a e placed in di e en a angemen s, and exposed o a ious ad e se
condi ions, o cons uc he se o scenes 𝑆. A de ailed desc ip ion o
how he PRC and AUC𝑝𝑟 a e calcula ed is gi en in Sec ion 2.1.
We obse e se e al issues wi h he PRC me hodology, and how i
is used o e alua e local desc ip o me hods in p e ious wo k. Mos
pe inen ly, he PRC me hodology assumes ha each su ace poin
in a scene has a mos one ma ching su ace poin on exac ly one
speci ic model. This does no adequa ely accoun o he possibili y
o mul iple ma ches o exis , which can be caused by he p esence o
sel -simila i y wi hin a model, o di e en models con aining pa ially
This a icle has been ce i ied as Replicable by he G aphics Replicabili y S amp Ini ia i e: h p://www. eplicabili ys amp.o g.
E-mail add ess: [email p o ec ed].
simila geome y. The me hodology can he e o e coun ue posi i es
as alse posi i es. An example o an objec commonly used in p e ious
e alua ions exhibi ing sel -simila i y is shown in Fig. 1. Compu ing a
g ound u h o all su ace ma ches is compu a ionally in ac able.
The ShapeBench me hodology is p oposed o add ess his, which —
in con as o p e ious wo k — measu es he ma ching pe o mance o
a local 3D shape desc ip o by compa ing dis ances be ween indi idual
desc ip o pai s. The desc ip o s a e compu ed o co esponding poin s
on a model, and a modi ied e sion o he same model ha cons i u es
he scene. Using his app oach a oids he need o compu e all g ound
u h ma ches, as he same poin on wo a ian s o he same su ace
is always a known ma ch. The no el Desc ip o Dis ance Index (DDI)
me ic is also p oposed as a means o con ex ualise he compu ed
dis ances be ween desc ip o pai s, by quan i ying he deg ee o which
one desc ip o can dis inguish i s coun e pa om noise. The DDI is
in ended o be used in conjunc ion wi h he PRC, and can assis in
explaining obse ed pe o mance by he PRC, and isualise how pe -
o mance deg ades when a me hod misiden i ies a nea es neighbou
poin .
Ano he mo i a ion o compa ing a model agains a modi ied
coun e pa is ha i imp o es he scalabili y o he e alua ion me hod-
ology. P e ious wo k has commonly elied on da ase s o cap u ed
3D da a. The esul s p esen ed in his a icle show ha he quan i y
and a ie y o objec s in hese da ase s is likely insu icien .Table 1
con ains an o e iew o e a ious ecen ly p oposed me hods, along
h ps://doi.o g/10.1016/j.cag.2024.104052
Recei ed 12 Ap il 2024; Recei ed in e ised o m 16 Augus 2024; Accep ed 19 Augus 2024
A ailable online 22 Augus 2024
0097-8493/©2024 The Au ho (s). Published by Else ie L d. This is an open access a icle unde he CC BY license
(h p://c ea i ecommons.o g/licenses/by/4.0/).
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Table 1
An o e iew o e da ase s used o he e alua ions in a numbe o ecen pape s, as well as some examples o la ge da ase s. Da ase s ha we e no used in a pa icula e alua ion
a e ma ked wi h a hyphen (–) o isual cla i y. All da ase s wi h equi alen model se names use he same (sub)se o models.
Da ase in o ma ion Model se Models Scenes Used in e alua ion
Da ase [10] [12] [9] [13] [14] [11]
Bologna 3D Re ie al (B3R) [15] S an o d 6 18 Yes – Yes Yes Yes Yes
Random Views [15] S an o d 6 36 – – Yes – – –
Bologna Da ase 1&2 - S an o d [16] S an o d 6 45 – Yes Yes – – –
UWA 3D Modelling UWA 4 75 Yes – – YesaYes –
UWA Objec Re ie al [17,18] UWA 5 50 Yes Yes – Yes Yes Yes
Bologna Da ase 3 - SpaceTime S e eo [16] Kinec (+clu e ) 8 15 – – Yes – – Yes
Bologna Da ase 5 - Kinec [19] Kinec (+clu e ) 6 16 Yes – Yesa–––
Bologna Objec Recogni ion Kinec (+clu e ) 6 17 – Yes – – – –
Bologna Mesh Regis a ion Kinec 6 95 Yes – – Yes – –
Queens LiDAR [20] Queens 5 63 – Yes – – – –
7-scenes [21] 7-scenes 7 n/a – – Yes – – –
DTU [22] DTU 45 3204 – – – – – –
ShapeNe Co e [23] ShapeNe Co e 51,300 n/a – – – – – –
ABC [24] ABC 1,000,000 n/a – – – – – –
Obja e se [25] Obja e se 798,759 n/a – – – – – –
aThe in o ma ion p o ided in he pape was insu icien o accu a ely deduce which exac da ase was used. A bes guess has been used ins ead.
Fig. 1. All su aces ha a e simila o he indica ed poin in he A madillo model om
he S an o d 3D scanning eposi o y acco ding o he RICI desc ip o . A da ke ed
colou indica es a be e ma ch.
wi h all da ase s used o e alua e hem. These da ase s con ain a mos
8 di e en models, and se e al also sha e he same model se .
The issue o quan i y can in heo y be ec i ied by using la ge , mo e
a ied, da ase s. Howe e , da ase s consis ing o eal wo ld 3D scans
scale poo ly. Each scene mus be cons uc ed, cap u ed, and s o ed
sepa a ely. Mul iple da ase s a ailable oday con aining only single
objec s equi e mo e han a e aby e o s o e, such as he ABC and
Obja e se da ase s lis ed in Table 1. An associa ed se o scenes would
equi e a mul iple o ha . The p oposed ShapeBench me hodology
he e o e cons uc s scenes in a p ocedu al and eplicable manne ,
only equi ing a se o models as inpu . Scenes a e gene a ed using
a sequence o one o mo e il e s, each simula ing eal wo ld ad e se
condi ions.
The use o a i icial da a has he added bene i ha he e ec o
di e en ad e se ma ching condi ions can be s udied in isola ion. Real
da a o en inhe en ly con ains combina ions o hese. One downside o
a i icial da a is ha mul iple e ec s ha na u ally occu in eal scans
mus now be app oxima ed o simula ed h ough one o mo e il e s
ins ead.
2. Rela ed wo k
An o e iew is p o ided o e di e en me ics and e alua ion
me hodologies ha ha e been used in p e ious wo k, wi h a special
ocus on he PRC me hodology. A b ie desc ip ion is also gi en o he
me hods ha we e used o es he p oposed ShapeBench benchma k
in Sec ion 5.
2.1. The PRC me hodology
An o e iew o e he p ocedu e o compu ing he PRC and he
associa ed AUC me ic is gi en he e. Because implemen a ion de ails
o he PRC a y, he e sion desc ibed by Guo e al. [5] is used as a
e e ence.
Fo compu ing he PRC, a se o poin s 𝑃𝑆⊆ 𝑆is andomly sampled
om he su ace o he scene 𝑆, which may be done using a keypoin
de ec o . Using known g ound u h ans o ma ions, ano he se 𝑃𝑀=
{𝑇(𝑞) ∶𝑞∈𝑃𝑆}is cons uc ed o model su ace poin s ha co espond
o hose in 𝑃𝑆, whe e 𝑇(𝑞)is he g ound u h ans o ma ion ha
ans o ms he poin q in o he coo dina e space o he model i belongs
o. A e compu ing a ea u e ec o o each poin in 𝑃𝑆and 𝑃𝑀using
he me hod being es ed, he closes wo poin s in ea u e space 𝑝𝑚1and
𝑝𝑚2a e ound in 𝑃𝑀 o each poin 𝑝𝑠in 𝑃𝑆. Using hese, he nea es
neighbou dis ance a io, 𝜎, is de ined in Eq. (1), whe e 𝑓(𝑝)deno es a
ea u e ec o o a gi en poin 𝑝, and 𝑑(𝑓1, 𝑓2)a unc ion compu ing
he dis ance be ween wo ea u e ec o s.
𝜎=𝑑(𝑓(𝑝𝑠), 𝑓(𝑝𝑚1))
𝑑(𝑓(𝑝𝑠), 𝑓(𝑝𝑚2)) (1)
I he alue o 𝜎is below a h eshold 𝜏, he poin pai 𝑝𝑠and 𝑝𝑚1is
conside ed a ma ch. Fo he poin o be coun ed as a ue ma ch, wo
condi ions mus also be sa is ied. Condi ion 1 equi es ha bo h poin s
co espond o he same objec , and condi ion 2 ha he Euclidean
dis ance be ween 𝑇(𝑝𝑠)and 𝑝𝑚1is less han hal o he suppo adius.
The suppo adius o a local shape desc ip o is a pa ame e ha
de e mines he size o he suppo olume, usually a sphe e o cylinde .
All su aces wi hin his olume a e ep esen ed by he desc ip o . I
ei he o hese condi ions is no sa is ied, he pai is ins ead conside ed
a alse posi i e.
The PRC is compu ed by i s compu ing he alues o 𝜎, and he
wo c i e ia, o each co esponding poin pai in 𝑃𝑆and 𝑃𝑀. Va ying
he alue o 𝜏be ween 0 and 1, and compu ing he P ecision and Recall
o each poin 𝑝𝑠, yields he PRC cu e. P ecision and Recall a e de ined
in Eqs. (2) and (3), espec i ely. The a ea below his cu e cons i u es
he de i ed A ea Unde Cu e (𝐴𝑈 𝐶𝑝𝑟) me ic.
𝑃 𝑟𝑒𝑐 𝑖𝑠𝑖𝑜𝑛 =| ue ma ches|
| ue ma ches|+| alse posi i es|(2)
𝑅𝑒𝑐 𝑎𝑙 𝑙=| ue ma ches|
|co esponding g ound u h poin s|(3)
The dis ance h eshold 𝜏was ini ially p oposed by Lowe [26] o
de e mining whe he a da abase o keypoin s con ained a good ma ch
2
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
o a gi en que y. Dis inc i e desc ip o s end o only ha e a single
good nea es neighbou , causing he 𝜎 a io o be low. The h eshold
was la e adop ed o e alua ing 3D desc ip o s [27,28].
2.2. E alua ion me hodologies
While only he PRC me hodology has been discussed in de ail, i is
no he only one which has been used o e alua ing desc ip o s o da e.
We he e o e highligh some o he no able me ics he e.
A classic me ic is he Recei e Ope a ing Cha ac e is ics (ROC),
de eloped du ing he second wo ld wa o e alua e he pe o mance
o ada ope a o s. This me ic plo s he ue posi i e a e agains he
alse posi i e a e, whe e he ue posi i e a e is equi alen o he
ecall me ic in Eq. (3). While no a common occu ence, he me ic
has seen use in he o m o a con usion ma ix [17,29,30]. The a ea
unde he ROC cu e can be used as an agg ega e me ic o he o e all
pe o mance o a es ed me hod [31], in a simila ashion o he PRC.
Ano he me ic ha has been used is he Cumula i e Ma ch Cha ac-
e is ic (CMC) [7,32], which uses a ixed numbe o que y desc ip o s
and hei co esponding lis s o nea es neighbou s in ea u e space
o compu e he ac ion whe e he g ound u h nea es neighbou is
in he op 𝑛nea es neighbou s. The ac ion is subsequen ly plo ed
o a ying alues o 𝑛. Van Blokland e al. use a a ia ion o his
me ic [33,34], compu ing he CMC solely o 𝑛= 0, and plo ing he
a ia ion o i s alue ac oss a numbe o scenes.
Whe eas he PRC app oach uses he a ea unde nea h he p ecision–
ecall cu e o compu e an o e all pe o mance me ic, Buch e al.
ins ead used he maximum F1 sco e [35], de ined as he maximum
ha monic mean ac oss all he compu ed p ecision– ecall alues.
2.3. Local 3D shape desc ip o s
3D desc ip o s a e commonly classi ied in o global and local de-
sc ip o s. Global desc ip o s aim o ep esen an en i e model in a single
desc ip o . This has a clea space ad an age o e local desc ip o s,
which use many desc ip o s o ep esen smalle po ions o an objec .
Howe e , local desc ip o s end o be less sensi i e o challenging
ma ching condi ions such as occlusion [36]. They also ely on an
objec being segmen ed om he en i onmen . Examples include he
SSCD [37] and PANORAMA [38] desc ip o s. Local desc ip o s a e
o en combined wi h keypoin de ec o s o i s loca e dis inc poin s
o in e es in a scene in o de o educe he olume o desc ip o s ha
need o be compu ed and compa ed.
One o he ea lie local 3D shape desc ip o s is he Spin Im-
age [39], p oposed by Johnson and Hebe . The desc ip o is a his-
og am ha compu es he dis ibu ion o poin s in he cylind ical
coo dina e space desc ibed by a gi en keypoin and i s associa ed
no mal ec o . Tomba i e al. p oposed he Unique Shape Con ex
(USC) [16], which uses a sphe ical suppo olume subdi ided in o pa -
i ions along he azimu h, ele a ion, and adial di ec ions. A his og am
is subsequen ly compu ed o e he su ace poin s in he suppo olume
ha all in o each bin, scaled by he local densi y o each poin . The
me hod is an ex ension o he 3D Shape Con ex [40], and add esses
i s p ima y limi a ion by using a local e e ence ame o o ien he
suppo olume o he desc ip o in a epea able manne .
The Signa u e o His og ams o O ienTa ions (SHOT) [19] p oposed
by Sal i e al. uses he same local e e ence ame and spa ial subdi i-
sion o i s suppo olume as he USC desc ip o . In con as o USC,
SHOT accumula es his og ams o cosines o each spa ial bin. These
cosines a e compu ed be ween he no mal ec o s o su ace poin s and
he o ien a ion o he desc ip o .
The Ro a ional P ojec ion S a is ics (RoPS) desc ip o p oposed by
Guo e al. [41] also uses a local e e ence ame o o ien he poin s
p esen in he suppo olume. These poin s a e subsequen ly o a ed in
se e al inc emen s along each majo axis. Fo each o a ed poin cloud,
all poin s a e p ojec ed on he 𝑥𝑦,𝑦𝑧, and 𝑥𝑧 planes, and a his og am is
compu ed o e hei dis ibu ion. Va ious s a is ics a e compu ed and
conca ena ed o o m he RoPS desc ip o .
The Radial In e sec ion Coun Image (RICI) desc ip o p oposed by
an Blokland and Theoha is [33] is a his og am u ilising a ia ions in
he numbe o in e sec ions be ween ci cles and he objec su ace. The
same au ho s no iced ha in e sec ion coun s do no a y be ween mos
adjacen ci cles, and he e o e p oposed a mo e compac bina y e sion
o he desc ip o , called he Quick In e sec ion Coun Change Image
(QUICCI) [34].
O he ele an examples o local 3D shape desc ip o s include he
Fas Poin Fea u e His og am (FPFH) [42], and CoSPAIR [43].
3. The ShapeBench benchma k
The p oposed ShapeBench e alua ion me hodology is now p e-
sen ed. I s objec i e is o e alua e whe he a desc ip o is capable o
co ec ly de e mining simila i y in su ace poin pai s, and o wha
ex en his capabili y is main ained when p esen ed wi h a ious ad-
e se condi ions ha a e common in p ac ical applica ions. In o de
o achie e his, i is necessa y o es ablish g ound u h ma ches be-
ween su ace poin s and hei su ounding su ace pa ches. While me -
ics exis o de e mining he simila i y o such pa ches, exhaus i ely
de ec ing all ma ching poin pai s in a la ge da ase is in ac able.
The benchma k is he e o e buil a ound compa ing poin pai s on
su aces ha a e gua an eed o be a co ec ma ch: wo copies o he
exac same su ace. One o hese wo copies is le unmodi ied and
ep esen s he model, while one o mo e al e a ions a e applied o he
o he copy o c ea e a scene objec . These al e a ions a e applied as a
sequence o one o mo e il e s, whe e he ou pu o one il e is used as
he inpu o he nex . Fil e s ha e ixed pa ame e s, and a e agnos ic
o any o he il e s applied on he sample objec . A e he scene mesh
is compu ed, he e ec on he ma ching capabili y o he desc ip o can
be measu ed by he dis ance be ween he desc ip o pai s compu ed o
co esponding poin s on he model and scene objec s.
An o e iew o e he benchma king p ocedu e is shown in Fig. 2.
A se o model objec s is i s d awn a andom om a la ge da ase .
Fo each o hese models, 100 e ices a e andomly selec ed om he
objec . The co esponding poin s on he scene mesh a e loca ed a e
he il e sequence has comple ed. No e ha his may cause some poin s
o be los i he po ion o he su ace hey we e loca ed on is emo ed
by a il e . Each il e aims o simula e a eal wo ld phenomenon such
as clu e (su aces in he suppo olume ha a e no pa o he model)
and occlusion (po ions o he objec su ace a e missing due o hese
no being isible om he poin o iew o a 3D cap u ing de ice).
A desc ip o pai is compu ed o each o he emaining poin pai s,
each espec i ely cap u ing co esponding poin s on he su ace o he
model and scene. This esul s in a model desc ip o 𝐷𝑚, and a scene
desc ip o 𝐷𝑠 ha has unde gone some modi ica ion. This desc ip o
pai is inally used o compu e he Desc ip o Dis ance Index and
PRC/AUC me ics.
Each il e epo s he alue o he independen a iable i simula es.
This alue may ei he be selec ed a andom, o mus be compu ed a e
he il e has comple ed. Fo example, a il e al e ing he o ien a ion
o no mal ec o s will epo he o a ion angle i andomly selec ed.
A il e emo ing occluded su aces can only compu e he amoun o
a ea ha was emo ed a e i has been applied. The abili y o a y
such an independen a iable comes om unning he expe imen many
imes on many di e en objec pai s, inc easing he likelihood ha he
a iable happens o ha e a gi en alue o in e es . I is also wo h
no ing ha all independen a iables mus be compu ed on a poin by
poin basis, a he han o he en i e objec . Fig. 3demons a es why
hese alues a e loca ion dependen .
3
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Fig. 2. An o e iew o e he p oposed benchma k.
Fig. 3. A demons a ion o a si ua ion whe e a desc ip o nuisance is localised o a
po ion on he objec . Two desc ip o s a e compu ed o wo poin s and hei suppo
egions a e shown. The suppo egion o poin A only con ains he objec o in e es ,
while poin B also con ains clu e .
3.1. The desc ip o dis ance index
The PRC was shown o be a ec ed by he exis ence o mul iple
alid ma ches o he same su ace. Accu a ely de e mining all hese is
compu a ionally in ac able. The nea es o second neighbou dis ance
a io 𝜎is also a ec ed by his issue, as wo alid ma ches a e likely
o esul in a high dis ance a io. A seconda y me ic ha is used in
conjunc ion wi h he PRC, and a oids elying on measu es ha a e
suscep ible o mul iple simila i y is he e o e desi able. The Desc ip o
Dis ance Index (DDI) me ic is he e o e p oposed.
In con as o he PRC, he DDI aims o compa e dis ances be ween
co esponding model and scene poin pai s di ec ly. Un o una ely,
his is no possible di ec ly, as he compu ed dis ance alues a y
ac oss desc ip o s and dis ance unc ions. An addi ional unc ion is
he e o e needed o ansla e desc ip o dis ances in o a space ha
allows compa ison.
No malising all dis ances is no possible, as dis ance unc ions do
no necessa ily scale linea ly. Using he same dis ance unc ion o
all me hods isks disad an aging some me hods i ano he dis ance
unc ion would yield be e pe o mance. I is, howe e , possible o
compa e dis ances be ween a desc ip o and o he desc ip o s when
using he same desc ip o me hod and dis ance unc ion.
The Desc ip o Dis ance Index (DDI) hus elies on a la ge se
o desc ip o s compu ed o andom e ices sampled om andomly
chosen objec s om he da ase , called he e e ence se 𝑅. The DDI o
a gi en pai o desc ip o s 𝑓1, 𝑓2is de ined as he ca dinali y o he se
o desc ip o s om 𝑅 ha a e close in ea u e space o 𝑓1 han 𝑓2. The
me ic he e o e e ec i ely measu es he ex en o which 𝑓2, om he
pe spec i e o 𝑓1, is indis inguishable om noise. In ou expe imen s,
he size o he e e ence se was se o 1 000 000 desc ip o s.
The p ocess o c ea ing he e e ence se uses he same andom
seed o all o he es ed me hods. Each me hod is he e o e asked o
compu e a desc ip o o he exac same poin s om he exac same
da ase objec s, which ensu es ha all me hods a e es ed on equal
g ound, and allows compa ison o DDI alues ac oss me hods. The
me ic is also no sensi i e o he exis ence o mul iple simila su aces,
as equi alen local su aces should p oduce equi alen desc ip o s, and
only desc ip o s whose dis ance alue is lowe a e coun ed. I 𝑓2is
compu ed o e he same su ace used o compu e 𝑓1bu has been
al e ed in some way, he pu pose o he me ic s ill holds because 𝑓2is
now objec i ely less dis inguishable om noise om he pe spec i e o
𝑓1.
3.2. Da ase
An app op ia e da ase mus be selec ed o se e as a model se and
inpu o he ma ching condi ions being es ed by he il e sequence.
The da ase should con ain a wide a ie y o 3D da a ha is ep esen-
a i e o he a ious use cases in which he es ed me hods may be
applied.
The Obja e se da ase [25] was selec ed, which co e s many do-
mains such as household objec s, u ni u e and ehicles. The da ase
con ains a o al o 798,759 iles om which 8124 we e excluded due
o con aining a poin cloud o , in a ew cases, o ailing o pa se. Poin
clouds we e excluded because sampling iangle meshes in o poin
clouds yields mo e simila su aces ac oss he wo modali ies compa ed
o sampling poin clouds in o iangle meshes. The ABC da ase and
ShapeNe Co e da ase s we e also conside ed, bu bo h mos ly consis
o CAD d awings wi h limi ed a ie y o applica ion domains.
A de i ed e sion o he da ase was c ea ed o simpli y dis i-
bu ion. This e sion only con ains e ex posi ions, no mals and, o
3D meshes, he polygon de ini ions o each objec . The comp es-
sion o ma is lossless and educed he o al size o he da ase om
app oxima ely 8.1 TB o app oxima ely 1.5 TB.
3.3. Pa ame e s
The e a e a ious pa ame e s ha mus be selec ed in o de o be
able o compa e ma ching pe o mance in a manne ha does no
bene i speci ic me hods. This includes he suppo adius, scale, and
sample coun used o sampling poin clouds.
One downside o using a i icial da a is ha he e is no in o ma ion
a ailable ega ding he physical dimensions o each model. One op ion
is o use he mesh esolu ion as an indica ion o scale, howe e his
app oach does no yield sa is ac o y scales o objec s wi h a high
a iance in edge leng hs. Objec s a e ins ead i ed in o a uni sphe e,
which also aids he in e p e a ion o any ele an dis ances in any
p oduced esul s. This was achie ed using he seb algo i hm [44], and
i s publicly a ailable implemen a ion [45].
All local shape desc ip o s use a suppo olume su ounding he
e e ence poin o de e mine which su aces o ep esen . A la ge
4
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Fig. 4. An o e iew o e he lowes , mean, and highes dis ances obse ed ac oss all 1010 desc ip o pai s (all possible pai s om wo se s o 100 000 desc ip o s each) o each
suppo adius ha was es ed. The chosen suppo adius is indica ed wi h a e ical line on each cha .
olume cap u es mo e su ace in o ma ion, bu also has a g ea e isk
o including clu e . Smalle olumes isk a educ ion in desc ip i e
capabili y. To da e he e is no commonly accep ed o es ablished
app oach o de e mining he suppo adius, and i is usually le up
o he use o selec .
F om he pe spec i e o ai ness, a suppo adius de e mines how
much su ace in o ma ion is gi en o he desc ip o . I is inhe en ly
impossible o con ol he amoun o in o ma ion p o ided o each
desc ip o , as he e exis s a ia ion in he shapes o he suppo olumes
ac oss me hods. The s eng hs and weaknesses inhe en o each me hod
can a ec how well a pa icula me hod pe o ms in a benchma k. We
hus conclude ha a adius mus be chosen on a pe -me hod basis,
and aim o choose he adius ha maximises he me hod’s capabili y
o disc imina e o he non-ma ching desc ip o s.
The means by which his is achie ed is o gene a e wo se s o
100 000 desc ip o s o each suppo adius be ween 0.01 and 1.5
( ecall ha all objec s a e i ed in o a uni sphe e), wi h inc emen s
o 0.015. Fo each pai o se s, he a e age dis ance be ween all
possible desc ip o pai s in each se is compu ed. I is conjec u ed ha
he suppo adius ha maximises his a e age dis ance would imply
ha he desc ip o is on a e age op imally capable o disc imina e
i s desc ip o s. This ixed suppo adius is subsequen ly used o all
desc ip o s compu ed o ha me hod. The minimum, maximum, and
a e age dis ance o each es ed suppo adius is shown in Fig. 4. The
se o models used o compu ing he suppo adius is di e en om
he one used o selec ing he se o models and e e ence desc ip o s.
A sligh ly modi ied e sion o he suppo adius selec ion p ocedu e
was used o he USC desc ip o . The mechanism used by his desc ip o
o no malise bin con ibu ions is no e ec i e, and caused he a e age
dis ances be ween desc ip o pai s o dec ease o highe suppo adii.
The addi ion o a no malisa ion s ep co ec ed his p oblem. This s ep
is only used du ing he suppo adius selec ion p ocess.
Ano he ele an pa ame e is he numbe o poin samples used
o uni o mly sample he iangle meshes om he da ase in o poin
clouds. This s ep is needed when es ing me hods using hese as inpu .
The dispa i y be ween inpu modali ies ep esen s o some ex en a
sou ce o un ai ness be ween me hods ha use one o he o he . A low
esolu ion poin cloud con ains less in o ma ion han he iangle mesh
i was sampled om, while a high esolu ion g ea ly inc eases execu ion
ime.
Using he numbe o e ices o iangles, o mesh esolu ion o
a mesh o se he sample coun is no a good solu ion because he
sizes o iangles can a y signi ican ly, e en wi hin ce ain meshes.
We he e o e use a sample coun o 1 000 000 pe uni a ea. The a ea is
calcula ed a e i ing he objec in o a uni sphe e. This ensu es ha all
su aces a e sampled wi h a oughly equi alen esolu ion. To alle ia e
some o he e ec s o sampling noise and excessi e compu a ion ime,
a lowe and uppe bound o 1 000 000 and 5 000 000 poin s a e used,
espec i ely.
4. Fil e s
Each o he il e s used o simula ing a ious ad e se ma ching
condi ions a e now mo i a ed and desc ibed in de ail.
4.1. Clu e
This il e simula es he e ec s o clu e being p esen in he
icini y o he model, adding su aces o he suppo olumes o es ed
desc ip o s ha do no belong o he model i sel . The in ensi y o
clu e is measu ed using Eq. (4).
𝐶 𝑙 𝑢𝑡𝑡𝑒𝑟 =Non model a ea in suppo olume
Model a ea in suppo olume (4)
The il e has been implemen ed by i s sampling 10 clu e objec s
a andom om he da ase . These added objec s a e subsequen ly
simula ed using he Jol Physics lib a y [46], which ensu es objec s
adhe e o physical cons ain s such as colliding wi h o he objec s and
g a i y. Objec s a e ini ially placed in a e ical s ack in he ai , a e
which g a i y is applied and he objec s all on o a g ound plane. The
simula ion ends when no mo e mo emen is de ec ed. Clu e objec s
a e a ac ed o he sample objec o inc ease he likelihood ha he
objec s o m a pile.
One limi a ion o he Jol Physics lib a y is ha i does no suppo
he simula ion o g oups o a bi a y mesh su aces. The V-HACD
algo i hm by Mammou e al. [47] was he e o e used, h ough i s
publicly a ailable implemen a ion [48], o i s subdi ide each mesh
in o a se o con ex hulls ha app oxima e he o iginal su ace. These
a e used as a p oxy du ing he simula ion. This, in a e cases, yields
degene a e hulls, which a e emo ed.
4.2. Occlusion
Occlusion is he esul o su aces no being isible om he poin o
iew o a cap u e de ice, some imes also e e ed o as pa iali y. This
il e ende s a high esolu ion image o he inpu scene om a andom
iewing di ec ion, and emo es any iangles ha a e no isible in he
image. The in ensi y o he occlusion il e is gi en in Eq. (5).
𝑂 𝑐 𝑐 𝑙 𝑢𝑠𝑖𝑜𝑛 = 1 −Model a ea in scene in suppo olume
Model a ea in suppo olume (5)
4.3. Al e na e mesh esolu ion
When an objec is acqui ed using di e en acquisi ion me hods,
such as di e en 3D cap u e de ices, he esolu ion o he p oduced
mesh can a y due o a ia ions in se ings and ha dwa e limi a ions.
Tes ing a ia ions in mesh esolu ion is a common occu ence in de-
sc ip o e alua ions done o da e. The p e alen way in which his
is implemen ed in p e ious wo k is h ough he use o a decima ion
algo i hm. The speci ic algo i hms ha a e usually used o his pu pose
ely p ima ily on edge and hal -edge collapse, along wi h a g eedy
scheme o deciding which edge o collapse nex .
While his achie es he desi ed e ec o educing mesh esolu ion,
simila decima ion a es can ha e an inconsis en e ec on he mesh
i sel . A mesh consis ing mos ly o la su aces will see li le physical
change a e educing i o a low e ex coun , while he same educ ion
applied o mesh wi h mo e o ganic shapes will be a mo e p onounced.
5
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Fig. 5. Two pic u es o he same objec su ace whe e he o iginal mesh in ahas been
cap u ed by a simula ed 3D cap u e de ice in b.
Ano he d awback o using decima ion is ha such algo i hms o en
do no p oduce any o he sampling a e ac s commonly ound when
using low quali y cap u ing equipmen . An example o his is shown
in Fig. 5, whe e he edge is ep esen ed une enly due o sampling
a e ac s. A decima ion algo i hm would no be inclined o p oduce
such meshes, as i a emp s o main ain he shape o he mesh, and
is mo e p one o simpli y such idges o sha p edges. Addi ionally, a
low esolu ion scanne may be able o pick up smalle de ails, which a
decima ion algo i hm is no gua an eed o keep. We he e o e do no
conside decima ion o be a good esolu ion educ ion s a egy ha is
g ounded in eal wo ld phenomena.
One po en ial solu ion ha could be used ins ead is a emeshing
algo i hm, which a emp s o ec ea e a mesh using app oxima ely
equila e al iangles wi h a gi en edge leng h. By using a a ge edge
leng h ha is la ge han he a e age edge leng h o he o iginal mesh,
he esul ing mesh should ha e ewe iangles han he o iginal while
app oxima ing he o iginal su ace. Howe e , his app oach p o ed
in easible because a p ope a ge edge leng h is di icul o es ablish.
Speci ying he a ge edge leng h ei he as a cons an o using he a -
e age edge leng h o he inpu mesh isks c ea ing an excessi e numbe
o iangles. This in u n causes high compu a ion imes and memo y
equi emen s in di e en implemen a ions o emeshing algo i hms ha
a e cu en ly a ailable.
The adop ed solu ion o his il e ins ead a emp s o simula e an
ideal low esolu ion scanne , by ende ing he scene om a andom
poin o iew a a esolu ion o 640 ×480 pixels. The dep h bu e
is subsequen ly used o econs uc he mesh. Va ia ion in he mesh
esolu ion is achie ed by a ying he dis ance o he objec o he
i ual dep h came a. Due o he pe spec i e p ojec ion, he objec
will on a e age co e ewe pixels in he dep h bu e , hus being
econs uc ed using ewe iangles.
4.4. Al e na e iangula ion
When he same objec is cap u ed epea edly, he p oduced mesh
su ace should be simila in shape when assuming he cap u e quali y
was easonable. Howe e , he manne in which he su ace is iangu-
la ed is unlikely o be simila due o a ious sou ces o noise du ing
he econs uc ion p ocess. An example o his is shown in Fig. 6. I a
me hod should hus be able o ecognise an equi alen su ace, i mus
be capable o doing so i espec i e o how ha su ace is ep esen ed.
Fu he mo e, e en i a keypoin de ec o is able o loca e he same
keypoin in bo h mesh a ian s, he exac loca ion o each keypoin
ela i e o he o iginal su ace may ha e shi ed.
Remeshing is a good candida e o implemen ing a simila e ec o
he al e na e mesh esolu ion il e . Howe e , as s a ed in Sec ion 4.3,
cu en emeshing algo i hms we e no ound o be iable. We in-
s ead used a mesh smoo hing algo i hm p oposed by Su azhsky and
Go sman [49], and i s implemen a ion om he CGAL lib a y [50].
The algo i hm adjus s e ex posi ions o o m highe quali y iangles
(e.g. mo e equila e al in shape, and simila in a ea), while main aining
he o e all shape o he mesh. The esul is a simila mesh wi h
Fig. 6. Two di e en 3D cap u es o he same objec . The su aces being ep esen ed
a e he same, bu he posi ions o e ices and iangles is di e en be ween hem.
displaced e ices, which is in line wi h he objec i e o his il e .
The in ensi y o he e ec o his il e is measu ed by compu ing he
dis ance o nea es e ex on he il e ed mesh o each poin on he
model su ace.
4.5. De ia ed no mal ec o
Many me hods o es ima ing no mal ec o s ha e been p oposed
o da e. Howe e , ac o s such as noisy inpu da a and es ima ion
e o s can p opaga e o de ia ions in he compu ed no mal ec o s.
Unde s anding how hese de ia ions a ec he ma ching pe o mance
o a me hod is he e o e ele an . This il e adjus s all no mals o he
inpu model by compu ing a new no mal ec o ha de ia es om he
o iginal by a uni o mly sampled andom angle. The selec ed angle is
chosen o be be ween 0 and 30 deg ees. The azimu h di ec ion in which
he no mal is o a ed is also chosen andomly.
4.6. De ia ed suppo adius
Calib a ion o es ima ion e o s in 3D cap u ing equipmen can
cause he scale o a p oduced mesh o a y sligh ly ac oss epea ed
cap u es. Al e na i ely, i he suppo adius o a poin is selec ed using
an algo i hm on a pe poin basis, e o s in he adius es ima ion may
cause a simila e ec .
The il e scales he suppo adius by a andomly chosen ac o 𝑠
be ween 0.75 and 1.25. Fo he ease o implemen a ion, his is done by
scaling he mesh by a ac o o 2 −𝑠, which achie es he same e ec .
4.7. Gaussian noise
Noise is a common occu ence in cap u ed 3D da a due o a ious
sou ces o inaccu acies du ing he cap u ing p ocess. Each e ex wi h a
unique e ex posi ion is displaced by a dis ance sampled om a no mal
dis ibu ion. The displacemen di ec ion is chosen by compu ing he
a e age di ec ion o he no mal ec o s o all e ices sha ing he same
e ex posi ion. The same s anda d de ia ion is used o all e ices
in he objec , whose alue is selec ed andomly be ween 0.0001 and
0.01. These we e chosen o be easonable pe u ba ions o wha can
be expec ed o scans o a ying quali y.
5. Resul s
The p oposed e alua ion me hodology is used o e alua e he
QUICCI, RICI, USC, Spin Image, SHOT, and RoPS desc ip o s. These
we e chosen o be a ep esen a i e se o bo h popula classic desc ip-
o s, while also including some ha ha e been p oposed mo e ecen ly.
The used suppo adius, dis ance unc ions, and o he me hod speci ic
pa ame e s a e lis ed in Table 2. The AUC𝑝𝑟 was compu ed using he
se o all model desc ip o s, in acco dance wi h i s implemen a ion in
p e ious wo k.
The USC me hod should be no ed speci ically he e. The inpu poin
clouds o his pa icula me hod we e downsampled o 1% o he
numbe o poin s used o o he poin cloud based me hods. The
6
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Table 2
An o e iew o e he es ed me hods and he ele an pa ame e s used.
Me hod Suppo adius Me hod speci ic pa ame e s Dis ance unc ion
QUICCI 0.39 Resolu ion: 31 ×32 Weigh ed Hamming [51]
RICI 0.255 Resolu ion: 32 ×32 Clu e esis an squa ed sum o di e ences [33]
RoPS 0.675 Poin samples pe uni a ea: 100 000 Euclidean dis ance
Poin sample limi : 5 000 000
Spin image 0.81 Resolu ion: 32 ×32, suppo angle 𝐴𝑠: 180°Pea son co ela ion
USC 0.135 Resolu ion: J=10, K=14, L=14 Euclidean dis ance
𝑟𝑚𝑖𝑛:0.014,𝛿:0.01, sampling densi y: 1%
SHOT 0.15 Resolu ion: 𝑠= 11,𝜆= 8,𝜇= 2,𝑅= 2Euclidean dis ance
desc ip o equi es he compu a ion o apoin densi y alue o each
poin in he poin cloud, which is an O(𝑛2) ope a ion. This was done
because compu ing desc ip o s a he ull poin cloud esolu ion p o ed
in ac able. The esul s in his sec ion o his me hod show ha his has
likely in luenced he ma ching pe o mance o his me hod, as i is no
consis en ly able o iden i y iden ical geome y. We conside he esul s
o his me hod alid despi e his p oblem, because using his me hod
in a p ac ical con ex would likely be done using a downsampled poin
cloud anyway. Howe e , i is likely ha he ma ching pe o mance o
he ull esolu ion poin clouds would be highe .
The benchma k is un o a o al o 10 il e con igu a ions, one o
each o he 7 p esen ed il e s whe e ha il e is un by i sel , and h ee
combina ions o wo il e s. The same oo andom seed is used o all
o hese con igu a ions, which means ha he e e ence se 𝑅, he se
o sample objec s, and he su ace poin s sampled om hose objec s
a e all iden ical ac oss all i e a ions o he benchma k p ocess.
Each expe imen p oduces 1 000 000 da a poin s, al hough depend-
ing on he il e (s) ha a e applied, a po ion o hese may be los .
Fo example, he occlusion il e emo es any e ices ha a e no
isible om he pe spec i e o he came a. A plo wi h he dis ibu ion
o sample coun s is he e o e gi en alongside he obse ed ma ching
pe o mance o each il e con igu a ion.
The benchma k i sel was implemen ed in C++, using he desc ip-
o implemen a ions om he libShapeDesc ip o lib a y [52]. While
he lib a y con ains GPU implemen a ions o a numbe o he used
desc ip o s, we ound ha using he CPU a ian s was mo e e ec i e
o he pu poses o his benchma k, as he quan i y o desc ip o s
being compu ed a a ime was no su icien o sa u a e he s eam
p ocesso s o he GPU wi h wo k, making ha pa h slowe han using
he CPU. Howe e , ou implemen a ion o he benchma k does suppo
desc ip o s implemen ed as GPU ke nels.
5.1. Single il e expe imen s
The cha s in his sec ion measu e he e ec o a single il e (and as
such a single independen a iable). Each cha con ains a isualisa ion
o he dis ibu ion o DDI alues, as well as a cu e showing he
compu ed AUC𝑝𝑟 o he same se o esul s. Values o he DDI can
a y be ween 0 and he size o he e e ence desc ip o se , which has
been se o 1 000 000. The cha s a e cons uc ed by i s di iding he
ange o he 𝑥-axis in o 75 subdi isions. Fo all sample poin s whose
x-coo dina es all in each o hese subdi isions, he sha e o DDI alues
ha alls in o each o de o magni ude is compu ed. The numbe o
samples in each o de o magni ude is subsequen ly no malised o he
o al numbe o poin samples in ha subdi ision, which yields he
p opo ional DDI alue. The AUC𝑝𝑟 alue compu ed o each subdi ision
is plo ed alongside hese.
A highe p opo ion o low DDI alues is desi able. Ideally, all DDI
alues a e ze o, which would esul in hei ela i e p opo ion being 1
o each subdi ision. An example whe e his is almos he case can be
seen in Fig. 10(d). A DDI alue o 0 indica es ha he me hod uniquely
iden i ied he co ec model desc ip o ou o all desc ip o s in he
e e ence se .
The highe anges o DDI alues isualise how quickly ma ching
pe o mance de e io a es, which p o ides mo e con ex han a single
cu e. An example o his can be seen in Figs. 11(a) and 11(b), whe e
only plo ing he p opo ion o samples ha ha e a DDI o 0 would ha e
shown bo h me hods o be app oxima ely equi alen in pe o mance,
while he p opo ional DDI shows a mo e apid decline in pe o mance
in he case o he RICI desc ip o . This is shown as a la ge p opo ion
o highe DDI alues.
The plo s in his sec ion indica e ha alues o AUC𝑝𝑟 and he
ac ion o esul s whose DDI alue was measu ed o be 0 a e o en
simila . The e a e some excep ions, such as Figs. 8(a),8(b), and 12(c).
Limi ed es ing indica es ha hese disc epancies a e p ima ily caused
by a poo 𝜎 a io. Howe e , excluding he 𝜎 a io s ill le a simila
gap o hose seen on o he plo s. I may be possible o explain his
disc epancy by he exis ence o mul iple iable ma ches. I s impac hus
appea s o be measu able, bu limi ed.
5.1.1. Clu e
Fig. 7shows he esul s o he expe imen whe e solely he clu e
il e is applied on he model objec . The RICI desc ip o is shown o
be highly esis an o he e ec s o clu e he e. While USC, SHOT, and
RoPS show poo esis ance. We conjec u e ha RoPS is a he sensi i e
o clu e due o he his og am s ep using a bounding box ha co e s
all poin samples p esen in he suppo olume. When clu e is added,
he dimensions o his bounding box change, causing he his og am o
lose co espondence wi h i s clu e ee coun e pa . RoPS and SHOT
bo h also include a no malisa ion s ep, which may be sensi i e o he
p esence o clu e .
In he case o USC, despi e he con ibu ions o indi idual poin s
being no malised by he local poin densi y when he his og am is con-
s uc ed, he e ec o clu e is ha he alues o indi idual desc ip o
bins a e inc eased. This in u n esul s in added dis ance o i s nea es
neighbou due o he use o he Euclidean dis ance unc ion. Clu e
has a simila e ec on he Spin Image, bu he use o he Pea son
Co ela ion dis ance unc ion likely educes some o he impac .
The sample coun s in Fig. 7(g) show how he dis ibu ion o clu e
a ies ac oss di e en suppo adii. The me hods wi h la ge suppo
adii expe ience la ge amoun s o clu e mo e o en, as would be
expec ed.
5.1.2. Occlusion
The esul s o he expe imen whe e only he occlusion il e was
applied a e shown in Fig. 8. He e he Spin Image pe o ms bes . The
QUICCI and RICI desc ip o s demons a e a capabili y o co ec ly
iden i ying he model desc ip o when po ions o he objec su ace
a e missing. Howe e , as discussed p e iously, his comes a he cos
o lowe 𝜎 a ios. These may pa ially be explained in he case o he
QUICCI desc ip o by a educ ion in he numbe o se bi s ( o 1, speci -
ically) by he occlusion il e . The emaining se bi s a e mo e likely o
be e o e lap wi h mo e dis an neighbou s, which is emphasised by
he used weigh ed Hamming dis ance unc ion.
The sample coun dis ibu ion shows ha he occu ence o a pa -
iali y o 50% is common, despi e he a ia ion in suppo adii amongs
he es ed me hods. Wi h espec o eplicabili y, he e a e small a ia-
ions induced in o he esul s, depending on which OpenGL implemen-
a ion is used. We ha e used he one p o ided by Mesa 23.1.4. The
same applies o he al e na e mesh esolu ion il e .
7
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Fig. 7. The e ec o a ying le els o clu e on he ma ching pe o mance o a ious desc ip o s. Fig. 7(g) shows he numbe o sample poin s pe his og am bin.
Fig. 8. The e ec o a ying le els o occlusion on he ma ching pe o mance o a ious desc ip o s. Fig. 8(g) shows he numbe o sample poin s pe his og am bin.
Fig. 9. Resul s o he al e na e iangula ion il e . Fig. 9(g) shows he numbe o sample poin s pe his og am bin.
We compa ed ou esul s o he clu e and occlusion il e s o hose
p esen ed by Guo e al. [5] in Fig. 7g and h. Bo h Figu es appea o
exhibi high le els o noise, whe e mos cu es luc ua e o a ying
deg ees. In con as , ou quan i a i e esul s, compu ed o e app ox-
ima ely wo o h ee o de s o magni ude mo e sample poin s, indica e
ha hese cu es should in mos cases be mono onically dec easing
wi h inc easing le els o clu e and occlusion. While he au ho s we e
no able o de e mine one hemsel es, i appea s ha a andom e o
is likely p esen in he da a.
Fo he USC desc ip o , nei he o he epo ed cu es ma ches wi h
he conclusions o ou e alua ion. The Spin image esul s o occlusion
show some simila i y, whe e mos obse a ions a e wi hin an es ima ed
e o ma gin o 0.2 AUC𝑝𝑟. The epo ed occlusion esul s o RoPS a e
in line wi h ou own esul s.
5.1.3. Al e na e iangula ion
Fo he al e na e iangula ion il e , whose esul s a e shown in
Fig. 9, only weak co ela ion was obse ed be ween he a e age edge
leng h (mesh esolu ion), and he ma ching pe o mance o he di -
e en desc ip o s. O he es ed desc ip o s, RoPS and SHOT exhibi
simila ma ching pe o mance, wi h USC pe o ming bes . Ou es ing
did no show a ela ionship be ween he ma ching pe o mance o a
desc ip o , and he e ex coun o he inpu mesh.
5.1.4. De ia ed no mal ec o
The de ia ed no mal ec o il e andomly chooses he angle by
which he no mal ec o o he scene mesh poin is pe u bed. This
yields an e en dis ibu ion o poin samples ac oss he di e en o a ion
angles. The QUICCI, RICI, SHOT, and Spin Image desc ip o s use he
no mal ec o o a keypoin o o ien hei his og ams, and a e a ec ed
by pe u ba ions o his ec o , as can be seen in Fig. 10.
The QUICCI and RICI desc ip o s ely on he simila i y o as e ised
local con ou s (e.g. he ci cula shape o a bicycle wheel). A o a ion o
he no mal ec o would cause he posi ion o hese as e ised con ou s
o shi wi hin he desc ip o image. We conjec u e ha his is he cause
o he d op in ma ching pe o mance when he no mal ec o de ia ion
angle is inc eased.
A simila e ec occu s in he case o he Spin Image, which, ins ead
o in e sec ion coun s, es ima es he mesh su ace a ea in e sec ing a
his og am bin, when ha bin is o a ed a ound a common axis o
one o a ion [53]. I s imp o ed pe o mance o e he QUICCI and RICI
desc ip o s may be explained by ha changes in he a ea in e sec ing
wi h each bin wi h inc easing no mal ec o de ia ion angles a e mo e
g adual han in e sec ion coun s.
The SHOT desc ip o pe o ms be e a highe no mal ec o de-
ia ion angles han he QUICCI, RICI, and Spin Image desc ip o s.
We conjec u e ha his is caused by he compa a i ely la ge olume
desc ibed by each his og am bin. The SHOT desc ip o does no achie e
pe ec ma ching pe o mance when he no mal ec o is le in ac .
This can be explained by ha he il e also modi ies he no mals o all
e ices in he scene. The SHOT desc ip o uses hese o compu e i s
his og ams.
The USC and RoPS desc ip o s do no u ilise he in o ma ion o
no mal ec o s, bu hei esul s ha e been included because i shows
ha RoPS achie es a nea pe ec ma ching sco e when p o ided wi h
e ec i ely equi alen geome y. The same is ue o USC, al hough
as men ioned p e iously, he low esolu ion used o compu ing hese
desc ip o s appea s o diminish i s ma ching capabili ies.
5.1.5. De ia ed suppo adius
The esul s o he suppo adius de ia ion il e in Fig. 11 show
ha he di e en desc ip o s ha e a ying sensi i i y le els o scale
and/o suppo adius misma ches. The pe o mance o QUICCI and
RICI can be explained wi h easons ha a e simila o hose ou lined o
he de ia ed no mal ec o . The as e isa ion done by hese desc ip o s
elies on in e sec ion coun s occu ing a speci ic dis ances, and when
hese a e displaced by a change in scale, he obse ed ma ching pe o -
mance d ops. QUICCI appea s o be sligh ly mo e esis an han RICI.
The SHOT desc ip o demons a es excellen pe o mance in his il e .
The il e chooses he applied scale ac o om a uni o m dis ibu ion,
which hus esul s in an app oxima ely cons an sample dis ibu ion.
8
B.I. an Blokland Compu e s & G aphics 124 (2024) 104052
Fig. 10. Resul s o he de ia ed no mal ec o il e . The ho izon al axis ep esen s he angle by which he no mal ec o o he scene poin was o a ed. Fig. 10(g) shows he
numbe o sample poin s pe his og am bin.
Fig. 11. Resul s o he suppo adius de ia ion il e . Fig. 11(g) shows he numbe o sample poin s pe his og am bin.
Fig. 12. Resul s he Gaussian noise il e . Fig. 12(g) shows he numbe o sample poin s pe his og am bin.
Fig. 13. Resul s he al e na e mesh esolu ion il e . Fig. 13(g) shows he numbe o sample poin s pe his og am bin.
5.1.6. Gaussian noise
When applying a ying le els o Gaussian noise, he esul s in
Fig. 12 show ha he Spin Image and SHOT desc ip o a e highly
esis an . This may o he Spin Image be explained by ha he a ea
(by p oxy he numbe o sample poin s) does no change much wi h
highe le els o noise.
In he case o he QUICCI and RICI desc ip o , he oughness o he
su ace induces addi ional a ia ions in numbe o in e sec ion coun s
obse ed by he desc ip o , educing i s abili y o disc imina e. Fo
he USC desc ip o , only 900 000 esul s we e compu ed due o he
desc ip o expe iencing excessi e execu ion imes.
5.1.7. Al e na e mesh esolu ion
The inal single il e expe imen is applying he al e na e mesh
esolu ion il e , whose esul s a e shown in Fig. 13. Fo his il e , he
cen e o he sample objec is placed a a andomly selec ed dis ance
om he came a. While he ma ching pe o mance o all me hods
is poo , he Spin Image appea s o be mos esis an o he educed
econs uc ed mesh esolu ion.
5.1.8. Summa y
In o de o gain an o e iew o e how well each me hod pe o ms
ac oss he di e en il e s, we compu ed a summa y cha , shown in
Fig. 14. We used a simila app oach o he PRC e alua ion me hod-
ology, by compu ing he a ea unde nea h he cu e whe e DDI is 0.
I should be no ed ha while pe o mance can be compa ed ac oss
me hods wi hin he same il e , i does no di ec ly ansla e be ween
di e en il e s, due o each il e imposing di e en ma ching con-
di ions on he scene. The ange o each independen a iable was
also selec ed a bi a ily o each cha , and he a ea unde he DDI
cu e ep esen s he ex en o which a me hod has achie ed good
pe o mance in he en i e y o ha ange. The in en o he cha is o
highligh cases whe e a me hod migh pe o m be e o wo se ela i e
o he o he es ed me hods.
The cha shows ha he QUICCI and RICI desc ip o s a e clea ly
supe io o clu e ed en i onmen s. The pe o mance o he RoPS,
SHOT, and USC desc ip o s in clu e ed scenes a e no missing in his
plo . The a ea desc ibed by hei DDI cu es is small. QUICCI and RICI
also pe o m well in occluded scenes – along wi h he Spin Image – bu
exhibi compa a i ely weak pe o mance when con on ed wi h a ious
ypes o noise, whe e he Spin Image, SHOT, and USC desc ip o s excel.
The only excep ion is de ia ions in he no mal ec o , o which he Spin
Image is sensi i e.
5.2. Dual il e expe imen s
The dual il e expe imen s use a pipeline wi h wo il e s each,
and a e hus cap u ing he e ec s o wo independen a iables. To
isualise hese, a 2D hea map is used, which coun s he ac ion o
samples whose DDI is 0. I a bin has less han 5 samples, i is emo ed.
Remo ed bins show a backg ound g id ins ead o a hea map pixel.
The i s o he es ed dual il e pipelines is he clu e il e ol-
lowed by he occlusion il e , which is a common occu ence in physical
en i onmen s. The esul s o his expe imen a e shown in Fig. 15. We
9
