Recei ed 8 Decembe 2022, accep ed 25 Decembe 2022, da e o publica ion 30 Decembe 2022,
da e o cu en e sion 12 Janua y 2023.
Digi al Objec Iden i ie 10.1109/ACCESS.2022.3233411
E alua ion o Class Dis ibu ion and Class
Combina ions on Seman ic Segmen a ion
o 3D Poin Clouds Wi h Poin Ne
EIKE BARNEFSKE AND HARALD STERNBERG
Ha enCi y Uni e si y Hambu g, Hyd og aphy and Geodesy, 22335 Hambu g, Ge many
Co esponding au ho : Eike Ba ne ske (eike.ba ne ske@hcu-hambu g.de)
ABSTRACT Poin clouds a e gene a ed by ligh imaging, de ec ion and anging (LIDAR) scanne s o dep h
imaging came as, which cap u e he geome y om he scanned objec s wi h high accu acy. Un o una ely,
hese sys ems a e unable o iden i y he seman ics o he objec s. Seman ic 3D poin clouds a e an impo an
basis o modeling he eal wo ld in digi al applica ions. Manual seman ic segmen a ion is a labo and
cos in ensi e ask. Au oma ion o seman ic segmen a ion using machine lea ning and deep lea ning (DL)
app oaches is he e o e an in e es ing subjec o esea ch. In pa icula , poin -based ne wo k a chi ec u es,
such as Poin Ne , lead o a bene icial seman ic segmen a ion in indi idual applica ions. Fo he applica ion
o DL me hods, a la ge numbe o hype pa ame e s (HPs) ha e o be de e mined and hese HPs in luence
he aining success. In ou wo k, he in es iga ed HPs a e he class dis ibu ion and he class combina ion.
By means o se en combina ions o classes ollowing a hie a chical scheme and ou me hods o adap he
class sizes, hese HPs a e in es iga ed in a de ailed and s uc u ed manne . The in es iga ed se ings show
an inc eased seman ic segmen a ion pe o mance, by an inc ease o 31% in ecall o he class E oneous
poin s o ha all classes ha e a ecall o highe han 50%. Howe e , based on ou esul s he co ec se ing
o only hese HPs does no lead o a simple, uni e sal and p ac ical seman ic segmen a ion p ocedu e.
INDEX TERMS 3D poin clouds, da a hype pa ame e , hie a chical class combina ion, hype pa ame e ,
Poin Ne , seman ic classes, seman ic segmen a ion, unbalanced da a.
I. INTRODUCTION
Scenes o he eal wo ld a e scanned wi h dep h imaging
came as and ligh imaging, de ec ion and anging (LIDAR)
scanne s in a sho ime wi h high geome ic esolu ion
and accu acy [1]. The digi ized scenes a e mos ly unso ed,
uns uc u ed and incomple e poin clouds [2], [3], which
o m he basis o a geome ic model. These kind o models
a e use ul in a wide a ie y o applica ions such as, u ban
planning, ou ism ma ke ing, indoo na iga ion, obo ic
con ol, au onomous d i ing, building cons uc ion plan-
ning, building ope a ion, he i age p ese a ion, a chaeologi-
cal in es iga ions, o es y and ag icul u e, o in as uc u e
main enance [4], [5], [6], [7], [8]. The c ea ion o hese
models is o en done by hand, because humans a e excellen
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Vlad Diaconi a .
a in e p e ing isualized 3D poin clouds and iden i ying
seman ic objec s wi hin hem. Au oma ed modeling by an
algo i hm equi es ha each poin ca ies seman ic ea u es
ha can be used o o m disc e e seman ic objec s in a scene.
Ex ending he poin cloud wi h seman ic ea u es is seman ic
segmen a ion. The au oma ed seman ic segmen a ion is o en
pe o med by Machine Lea ning (ML) and Deep Lea ning
(DL) app oaches, which a e a cu en esea ch opics [4], [7],
[9], [10].
DL-me hods o seman ic segmen a ion o 2D images
achie e e y high accu acies, bu canno be simply applied
o poin clouds due o he abo e men ioned p ope ies. Many
app oaches exis whe e he poin cloud is i s ans o med
in o an o de and s uc u e [11], [12]. Howe e , poin -based
me hods such as Poin Ne [13] o RandLA-Ne [14] omi his
s ep and can pe o m a seman ic segmen a ion di ec ly om
he o iginal poin cloud. In o de o use hese seman ically
3826 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ VOLUME 11, 2023
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
segmen ed poin clouds o c ea e a building in o ma ion
model (BIM), he seman ic poin segmen s mus mee ce ain
accu acy equi emen s ha a ise om he model speci ica-
ions [15]. These accu acy equi emen s a e de ined in he
Le el o Accu acy (LoA) [16], Le el o De ail (LoD) [17]
o he Le el o De elopmen (LoDe ) [18]. Fo a BIM o
he le el LoA2 (15 mm o 500 mm) o LoDe 200 (design
planning) and highe , he poin cloud segmen s o en canno
ul ill he geome ic o seman ic equi emen s, so ha an
imp o emen o he seman ic segmen a ion s ep is necessa y.
Conside ing he complexi y o he poin cloud da ase s, he
au oma ic seman ic segmen a ion is a key p ocessing s ep o
an e icien modeling.
Inc eased accu acy o hese seman ic segmen a ion me h-
ods is possible wi h aining da a [19] and Hype pa ame e s
(HPs) [20], in addi ion o he adap a ion o he ne wo k
a chi ec u e. HPs a e selec ed be o e aining and commonly
p io knowledge is used o he selec ion. They con ol and
in luence he aining p og ess [20]. In his wo k he in luence
o he HPs Unbalanced class dis ibu ions and di e en Class
combina ions a e in es iga ed using he es ablished ne wo k
a chi ec u e Poin Ne (Sec ion IV). Fo his pu pose, ou
da a- o algo i hm-based me hods o ha monizing class sizes
a e applied and adap ed. In addi ion, a hie a chical class
de ini ion o equen classes in a BIM is de eloped and
applied. Fu he cen al con ibu ions o his wo k a e:
•A e iew o HP de e mina ion me hods, da a augmen-
a ion me hods, and hie a chical seman ic segmen a ion
me hods (Sec ion II).
•The c ea ion o a new medium-sized da ase ha is
sui able o BIM applica ions (Sec ion III).
•The sys ema ic e alua ion o da a augmen a ion
me hods and o hie a chical class combina ions
(Sec ion Vand VI).
All indings a e summa ized in Sec ion VII and an ou look
on u he in es iga ions is gi en.
II. STATE OF THE ART
A ML model is in luenced by a la ge numbe o HPs. One
key challenge when wo king wi h complex ML me hods is o
ully cap u e hese HPs and o de ine he op imal alues o
hem, as in es iga ed by [21], [22], [23], and [24]. In Fig. 1,
he ele an HPs o seman ic segmen a ions a e g ouped in o
six clus e s. The op ow ep esen s gene al HPs ha ela e o
ne wo k a chi ec u e, egula ion, op imiza ion, and ini ializa-
ion [25]. In he bo om ow (a ea wi h he blue backg ound),
Da a Hype pa ame e s (DHPs) a e shown. The DHPs depend
on he da a cha ac e is ics and no on he chosen model.
DHPs can be dis inguished acco ding o seman ic, s uc-
u al, geome ical and spec al cha ac e is ics o he da ase .
The seman ic cha ac e is ics o poin clouds a e desc ibed
by [26], [27], and [28]. In e ms o s uc u al cha ac e is ics,
he de ini ion o poin neighbo hood [29], da a augmen a-
ion [30], and he unbalanced class dis ibu ion o aining
da a in gene al (e.g., images) [31] a e opics ha ha e al eady
been in es iga ed in o he s udies. Gene ally, geome ical and
spec al ea u es o poin clouds a e o en used as aining
da a by manual augmen a ion o poin ea u e spaces [32].
These hand d awn ea u es a e poin no mals, eigen alues,
densi y alues o mixed ea u es [33], [34], [35]. So a , only
ew s udies on he unbalanced class dis ibu ion and hie a -
chic seman ic segmen a ion in poin clouds a e published.
An o e iew o hem is p esen ed in Sec ions II-B and II-C.
A. Poin Ne
DL-models o seman ic segmen a ions o poin clouds
a e usually dis inguished by he inpu o ma s in o which
he poin cloud is ans o med. A ca ego iza ion is p e-
sen ed in [36]. In hei wo k, a ca ego iza ion is made
in o disc e iza ion-based / s uc u e-based (e.g., as oxel),
p ojec ion-based (e.g., 2D-image), and poin -based (e.g., aw
poin s o g aph) me hods, which can u he e ined (Fig. 2).
While ini ially disc e iza ion-based [37], [38], [39] and
p ojec ion-based me hods [40] we e p edominan ly used,
nowadays mos o he (non- eal- ime) models a e poin -
based [41]. Poin -based me hods use he uno de ed poin s
hemsel es o pe o m seman ic segmen a ion.
One o he mos widely used poin -based me hod is
Poin Ne [13]. Poin Ne add esses he s uc u al disad an-
age o he poin cloud o ma when p ocessing hem wi h
DL-me hods. This means ha poin s do no ha e o be placed
in a ixed o de p io o p ocessing. They can be a anged ee
in o ien a ion and posi ion in space.
The ull unc ionali y o Poin Ne is explained in he
i s published a icle om he de elope s [13] and in many
e iews such as [43] and [44]. In he ollowing, he cen al
p ocessing s eps o Poin Ne a e p esen ed o a be e unde -
s anding o ou in es iga ions. Fu he mo e, he limi a ions o
Poin Ne will be ou lined.
1) PROCESSING STEPS OF Poin Ne
P ocessing wi h Poin Ne can be di ided in o h ee main
s eps. In he i s p ocessing s ep, he ea u es a e ans-
o med in o a uni o m n-dimensional space using an a ine
ans o ma ion (wi h he T-Ne module o Poin Ne ). The
ans o ma ion pa ame e s a e lea ned by he ne wo k. This
ans o ma ion ensu es ha all inpu blocks a e nea ly a
he same posi ion and almos ha e he same o ien a ion.
An example wi h a poin cloud o a chai is gi en in Fig. 3.
This ans o ma ion is epea ed a e he i s ex ac ion o
dep h ea u es, so ha he dep h ea u es a e also aligned in
he complex ea u e space (e.g., 64 dimensions) [13].
The second p ocessing s ep is he ex ac ion o dep h
ea u es-based on he inpu ea u es (e.g., 3D coo dina es,
poin no mals o colo alues) o p e ious dep h ea u es.
This is done using di e en ans o ma ion laye s o a mul i-
laye -pe cep on [13]. In mos implemen a ions o Poin Ne ,
a 1D o a 2D con olu ional laye is used. As shown in
Fig. 4a, he ows o he enso a e equal o he numbe o
block poin s and only one column is occupied. The ea u es
o he poin s a e a anged in he dep h laye o he enso .
VOLUME 11, 2023 3827
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
FIGURE 1. In luencing a iables and pa ame e s o he de elopmen o a DL-based seman ic segmen a ion me hod. The pa ame e s and in luencing
a iables shown a e a selec ion and migh be adap ed o o he applica ions.
FIGURE 2. P epa a ion o poin clouds o seman ic segmen a ion wi h
DL, by p ojec ion in o image space, o ganiza ion in o a 3D s uc u e, and
usage o he aw poin cloud. (Figu e aken om [42] and adap ed.)
Each con olu ional il e con ains only one alue (which is
ixed wi hin he con olu ion), so he dep h ea u es a e based
only on he p e ious ea u es o a poin (Fig. 4a). Depending
on he implemen a ion, di e en numbe s o con olu ional
laye s and il e s a e used.
The hi d p ocessing s ep is he agg ega ion o he ea u es
o he indi idual poin s in o a global ea u e ec o o he
espec i e inpu block. This is done using he max-pooling
unc ion, in which only he la ges alue is kep o each
ea u e (Fig. 4b). I esul s in a ea u e ec o ha can be
used o classi ica ion o he poin cloud block. Fo seg-
men a ion, his global ea u e ec o is aken and appended
o all indi idual poin ea u e ec o s. The e is now a com-
bina ion o in e -poin and global ea u es o each poin ,
om which u he dep h ea u es a e gene a ed. The dep h
ea u es a e used o classi y each poin (e.g., wi h a so max
unc ion) [13].
FIGURE 3. In en ion o he T-Ne module is ha a poin cloud is always
aligned in a simila way by means o an a ine ans o ma ion.
2) LIMITATIONS AND ADVANCEMENTS OF Poin Ne
The cen al p oblem wi h Poin Ne is he selec ion o poin s
o a block. This o ins ance is he case, i he a ea o
be segmen ed seman ically is e y la ge, he poin densi-
ies a e in-homogeneous o di e en equen classes a e
included. Rega ding his challenge, di e en ex ensions, such
as Poin Ne ++ [45] o a sys ema ic neighbo hood sea ches,
3828 VOLUME 11, 2023
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
FIGURE 4. Con olu ion (a) and max-pooling (b) unc ions o ea u e
enhancemen and agg ega ion wi h Poin Ne .
such as by [46] ha e been de eloped. Howe e , hese de el-
opmen s also encoun e limi a ions wi h ex a-la ge and
highly de ailed da ase s.
B. UNBALANCED CLASS DISTRIBUTION
One majo conce n o he seman ic segmen a ion ask sol ed
wi h ML me hods is, ha di e en seman ic classes in
eal-wo ld da a consis o di e en numbe s o indi idual da a
objec s [47]. Fo example, he backg ound o an image is
desc ibed by he majo i y o indi idual pixels and he e o e i
is lea ned mo e equen by mos ML algo i hms. O en he
algo i hm lea ns only he backg ound, because his way he
highes accu acy is achie ed o he whole da ase [31].
Basically, his p oblem exis s o all ML me hods, such
as Suppo Vec o Machine, k-Nea es -Neighbo (kNN),
K-Mean Clus e ing, Con olu ional Neu al Ne wo k (CNN)
and all kind o da a ypes, such as da a se ies, images,
image da abases o poin clouds [48]. Va ious me hods a e
de eloped o sol e he class imbalance p oblem o ce ain
da a ypes. These me hods can be clus e ed in o ou me hod
g oups (Fig. 5).
FIGURE 5. Fou me hod g oups o add ess he p oblem o unbalanced
class dis ibu ion. A anged acco ding o simila i ies o me hods.
The i s me hod g oup, he da a-based me hods, encloses
all me hods, which ac i ely change he numbe o he indi-
idual da a objec s (e.g., poin s o images). The da ase is
il e ed o augmen ed in such a way ha he numbe o objec s
be ween di e en classes becomes equal o mo e simila .
Me hods ha educe he numbe o da a objec s a e
e e ed as unde -sampling (US) me hods. These me hods
andomly [49] o sys ema ically [50], [51], [52] selec da a
objec s pe class o es ablish equali y and ensu es ha only
o iginal (measu ed) da a is used. The US me hods ha e he
cen al disad an age ha pa s o he knowledge a e no used
and he e o lea ning is only pe o med on a subse o he
in o ma ion. The use o only a subse could lead o changes
in he local neighbo hood [31].
Unlike US, andom [49] o sys ema ic [48], [53], [54], [55]
o e -sampling (OS) me hods enla ge he da ase and augmen
i wi h a i icial o duplica ed da a. One popula me hod is
he Syn he ic Mino i y O e sampling Technique (SMOTE)
by [56], whe e he neighbo hood is conside ed o con ol
he OS me hod. The da ase s become la ge wi hou gain-
ing any new knowledge. In addi ion, ypical da a objec s
in he in equen classes a e emphasized s ongly, he e o e
he ained model may no ans e well o new unknown
da ase s. Me hods ha minimize he ad e se in luence o
bo h app oaches use he o iginal and OS da ase s in di e en
aining phases [57] o combine he OS and US me hods, each
based on he epoch esul [48].
The second g oup o me hods a e he algo i hm-based
me hods. They modi y he lea ning algo i hm and aim o
a s onge impac o classes wi h ewe da a objec s. Tha
can be ei he done by adap ing he loss unc ion [58], [59],
[60], modi ying he ne wo k a chi ec u e [61], [62], [63],
[64] o weigh ing he p edic ions [65], [66]. Lea ning can
ad an ageously be done di ec ly wi h aw da a. Bu , i he
class di e ences a e e y la ge, weigh ing can lead o a w ong
ela ionship and a mino class may becomes oo dominan .
The hi d g oup o me hods a e named as hyb id me hods.
They apply da a- and algo i hm-based me hods oge he . The
da a a e combined in a i s phase a he le el o o dinal
ea u es [47] o a he le el o de i ed ea u es o ob ain highly
di e en iable ea u es in he aining da a. The ea u es can
be c ea ed by g ouping he ini ial ea u es and de i ing new
ea u es [67]. O he app oaches c ea e embedded ea u es
and adjus i in a o o he mino class [68] o aking in o
accoun possible high and low classi ica ion p obabili ies
based on he ea u e dis ibu ion wi hin he classes and i s
bounda ies [69]. Hyb id me hods a e applied o CNN such
ha emaining di e ences due o he equaliza ion o class
sizes o op imiza ion o he da a a e made by adjus ing a loss
unc ion o using mul iple loss unc ions.
The las me hod g oup is ensemble lea ning. These me h-
ods a e applied o adi ionally weak lea ning me hods, such
as Decision T ees o K-mean clus e ing. In ensemble lea n-
ing, di e en classi ie s o he same classi ie a e ained
wi h di e en combina ions o da a o pa ame e s. The esul s
o all classi ie s a e e alua ed o a combined esul using
ha d o so o ing [70] (s acking and bagging). Boos ing,
as in SMOTEBoos [71], can be used as an al e na i e. He e,
a e each un, he classi ica ion pa ame e s (e.g., selec ion o
aining da a) a e adjus ed so ha mo e a en ion is payed o
ha d- o-lea n ea u es.
C. HIERARCHICAL CLASS COMBINATION FOR
SEMANTIC SEGMENTATION
The e m hie a chical seman ic segmen a ion is used in wo
de ini ions. The i s de ini ion is abou he geome ic size
change o he segmen s. The segmen s can g ow (segmen s
VOLUME 11, 2023 3829
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
a e me ged) o sh ink (segmen s a e spli ) in he in eg a i e
and hie a chical segmen a ion p ocess. The names and num-
be s o he classes always emain he same. In his app oach,
he seman ics is added o he segmen s in a subsequen clas-
si ica ion [72], [73]. O en, he poin clouds a e ans o med
in o g aphs, which a e g adually e ined o gene a e local
ea u es [74], [75], [76].
The second de ini ion ocuses on di e en classes a di -
e en s ages o he segmen a ion. He e, he class de ini ion
is hie a chical and he seman ic in o ma ion changes by each
le el. This de ini ion o hie a chical seman ic segmen a ion
is less desc ibed in li e a u e, because o a seman ic seg-
men a ion usually a ix se o seman ic classes is de ined
in ad anced and he p ocess is done in one s ep. F equen
and in equen classes a e de e mined and segmen ed in he
same s ep. In con as , i he se o seman ic classes is com-
plex and/o o ien ed o a p ede ined hie a chical seman ic
schema, such as he Ci yGML [77], he Indus y Founda ion
Classes (IFC) [18] o a non-ins i u ional schema [78], [79],
[80] ano he s a egy can be applied. This pe o ms seman ic
segmen a ion in se e al sub-s eps. Seman ic schemes usually
ha e mul iple aspec s, such as geome y and seman ic, and
a e o ganized in o LoD [17]. The seman ic LoD de e mines
which class is de e mined in which le el. The eby, o each
poin only one class should be de ined in one LoD. As shown
in [81], his app oach can help o be e dis inguish seman ic
classes wi h simila geome ic ea u es ha appea in di e en
LoDs. In addi ion, a combina ion o ea u es om di e en
LoD can help o inc ease he seman ic accu acy o a seman ic
segmen a ion [81].
III. DATASET
Ou da ase consis s o mo e han 76 million indi idual poin s
ep esen ing 27 ooms o he Ha enCi y Uni e si y Hambu g
main building (Figs. 6,7and 8). A subse o he poin cloud
was c ea ed o his wo k and con ains he class E oneous
Poin s (subse A). This subse was ex ended by an exis ing
da ase wi hou he class E oneous Poin s (subse B). Subse
B was o iginally c ea ed o he Le el 5 Indoo Na iga-
ion p ojec [82] and was eo ganized and imp o ed o ou
expe imen s. The da ase is o ganized by ooms, which can
be selec ed indi idually. The ooms a e di e en in e ms
o u nishings, usage and shapes. Semina ooms, lec u e
halls, o ices, co ee ki chens, co ido s and en ance halls a e
p esen in he da ase .
All ooms we e su eyed using e es ial lase scanne s
Z+F Image 5010 o 5016. The su ey was pe o med wi h
a esolu ion o 6 mm a a dis ance o 10 m and he quali y
le el ‘‘no mal’’ [83]. Small and good obse able ooms up o
abou 75 m2we e su eyed om a single iewpoin . La ge
o winding ooms we e su eyed wi h mul iple iewpoin s
so ha all u nishings and building pa s we e cap u ed com-
ple ely. Small co e age gaps (e.g., on walls o on he loo due
o obscu ing u ni u e) a e p esen in he da a and accep ed i
he o e all geome y o he seman ic classes pe oom can be
de i ed om he poin cloud (Fig. 9).
FIGURE 6. Poin cloud da ase om he main building o Ha enCi y
Uni e si y Hambu g (en ance le el).
FIGURE 7. Poin cloud da ase om he main building o Ha enCi y
Uni e si y Hambu g (o ice le el).
FIGURE 8. Poin cloud da ase om he main building o Ha enCi y
Uni e si y Hambu g (lec u e hall le el).
The egis a ion o he indi idual poin clouds we e ca ied
ou ia disc e e a ge s, which we e measu ed au oma ically
and manually in he scanned poin clouds. Using he coo -
dina es o a geode ic ne measu emen ( ia o al s a ion) he
scanned poin clouds a e ans e ed in o a global and uni o m
coo dina e sys em (geo- e e encing). The di ision by ooms
was done in a manual segmen a ion p ocedu e. Fo his pu -
pose, he spaces a e oughly selec ed in he en i e poin cloud
and a pa ial poin cloud is copied. The pa ial poin clouds
a e p ocessed so ha only poin s o he espec i e oom a e
included. This p ocedu e leads o a mo e comple e poin
cloud, because poin s o iewpoin s in neighbo ing ooms a e
conside ed.
The second segmen a ion s ep is based on he seman ic
classes and was pe o med wi h CloudCompa e [84] and
Au ocad Recap [85]. To achie e a high quali y o he man-
ually classi ied poin s, each poin cloud was seman ically
segmen ed a leas h ee imes by di e en anno a o s. The
anno a o s we e p e iously ained in he ask and ecei ed
eedback on in e media e esul s. The indi idual segmen-
a ions o he same ooms we e combined so ha coa se
indi idual e o s a e emo ed.
3830 VOLUME 11, 2023

E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
FIGURE 9. Gaps (whi e a eas) in he poin cloud caused by occlusions
(e.g., u ni u e), and which a e ole a ed in he da ase .
IV. METHODOLOGY
The in luence o ce ain DHPs, especially o poin clouds
a e s ill li le sys ema ically s udied. The seman ic classes a e
usually de ined acco ding o he applica ion, such as p ocess-
ing a eal LIDAR poin clouds o indus ial use [2] o ha ing a
Scan2BIM applica ion [4]. The seman ic segmen a ion o all
de ined classes is usually achie ed in one s ep. Re e ence [86]
obse ed ha he class de ini ion and he class con en ha e an
in luence on he seman ic segmen a ion esul . As an al e na-
i e o he applica ion-o ien ed class de ini ion, an algo i hm-
o ien ed class de ini ion is also possible. This insigh leads o
a p ocess in which he DHPs a e se in a o o he algo i hm
by conside ing: The numbe o classes, he numbe o poin s
pe class, he p esence o absence o e oneous poin s and
he geome ic di e ence o he objec s in di e en classes.
To in es iga e he DHP, an applica ion and expe imen a ion
en i onmen (AEE) was de eloped in which he common HPs
and DHPs can be easily cus omized. The AEE o e s di e en
op ions o he poin cloud augmen a ion using balancing
(imp o emen ) echniques.
FIGURE 10. P ocessing s eps o seman ic segmen a ion o poin clouds.
The illed boxes a e examined in de ail.
The in es iga ions ollow he wo k low shown in Fig.10.
The da a eco ding is ollowed by he egis a ion, he o ga-
niza ion o he sub-scans, and he manual seman ic seg-
men a ion o he poin clouds, so ha hese can be used as
aining and e alua ion da a. These s eps a e ollowed by
a da ase op imiza ion, which is he cen al ocus o his
wo k (Sec ions IV-B and IV-C). Nex , he da a is p epa ed
o he p ocessing s ep wi h he chosen au oma ic seman ic
segmen a ion me hod (Sec ion IV-A) and he algo i hm is
ained. The e iciency o he aining is e alua ed wi h
‘‘unknown da a’’ o he same da ase (e.g., o he ooms).
A. APPLICATION AND EXPERIMENTATION ENVIRONMENT
Ou wo k is based on he DL-a chi ec u e Poin Ne [13],
o which op imal HPs we e de e mined based on com-
p ehensi e p elimina y in es iga ions and li e a u e esea ch
[41], [87]. Poin Ne is one o he es ablished and ounda-
ional DL-a chi ec u es which makes ou in es iga ion esul s
compa able wi h o he s udies. All pa ame e s o he ne wo k
a chi ec u e (excep o he numbe o classes) emain as in
he implemen a ion o [88]. The AEE is de eloped ha Poin -
Ne can be eplaced by o he poin -based DL-a chi ec u es.
The main d awback o Poin Ne is ha only a small numbe
o poin s and only local ea u es a e used o assign a poin
o a class. Ou app oach o con ol he inpu o poin s is
simple and is based on a andom and uni o m spli ing o
he poin cloud in o h ee equally sized sub-poin clouds and
he de e mina ion o a Local Neighbo hood Box (LNB). The
o igin o coo dina es o he en i e poin cloud is de ined by
he smalles alues o he x- and y-coo dina es. This o igin
o coo dina es is used o he i s sub-poin cloud. Fo he
ollowing sub-poin clouds, i is shi ed in he x-y-plane by
a ac ion o he LNB edge leng h and addi ionally o a ed
by a ix angle (Fig. 11). Fo each o he shi ed and o a ed
sub poin clouds, he LNB a e de e mined using he s uc u e
algo i hms o pyn cloud lib a y.
The local neighbo hood is de ined by a 1 x 1 m LNB whose
heigh is he maximum possible oom heigh o he da ase .
By shi ing and o a ing, six di e en local neighbo hoods
a e c ea ed o each o iginal LNB. The o a ed poin cloud
is an ex ension o he o iginal poin cloud. F om each LNB
a ce ain numbe o n andomly selec ed poin s is aken as
ne wo k-inpu un il all poin s ha e been ed in o he ne wo k.
I he e a e no npoin s le , he inpu is illed by andom
copied poin s om he LNB. In addi ion o he global no -
malized oom coo dina es (xglo,yglo, and zglo) and he poin
no mals (xn,yn,zn), he local no malized coo dina es o he
LNBs (xloc,yloc,zloc) a e calcula ed. These nine geome ic
ea u es a e used as inpu ea u es o all expe imen s.
B. METHODS FOR HARMONIZING THE UNBALANCED
CLASS DISTRIBUTION
The seman ic classes o poin clouds om eal objec s di e
by he numbe o poin s. Objec s, such as walls and loo s,
ake up mo e a ea (as well as poin s), compa ed o objec s,
such as doo s and e oneous poin s. This is due o he ac
ha mos su eying sys ems egula ly scan su aces wi h a
ixed angula inc emen ela ed o he senso , which changes
wi h dis ance. In addi ion, he measu ing sys ems cap u e
a eas and no edges. Two backwa ds a ise om he cap u e
condi ions o he aining o seman ic segmen a ion me h-
ods. Fi s , a lo o in o ma ion is collec ed which p o ides
no o li le new in o ma ion o he sepa a ion o seman ic
objec s. Second, he e is o en a lack o in o ma ion abou
VOLUME 11, 2023 3831
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
FIGURE 11. Desc ibing he neighbo hood o Poin Ne inpu s using o e lapping LNBs.
geome ically complex and a iable objec s, as well as o he
class edge a eas.
ML-based seman ic segmen a ion me hods lea n a ela ion-
ship be ween inpu ea u es and seman ic class o e la ge
amoun s o da a and y o de e mine an op imal sepa a ion
o e he majo i y o poin ea u es. I one class is dominan
in he numbe o poin s, i can be obse ed ha he bes
esul s a e ob ained by assigning almos all o all poin s o
his class. In p ocessing o medical images, his p oblem
is well known by he ac ha only a ew pixels show he
anomaly and mos pixels show no mal o gans [89]. Fo ou
poin clouds, he p oblem is ans e able, because mos poin s
belong o equen classes, such as wall, loo o ceiling.
The unde lying idea o sol e his opic is o ocus on he
in o ma ion ha is impo an o he sepa a ion o he classes
and o inc ease i s impo ance. These is usually done by
augmen ing he poin s o he in equen classes. The emphasis
on he in equen class(es) is in es iga ed in he expe imen al
s udies o Sec ion V-D by means o ou echniques. These
echniques a e he SMOTE [56], he s ack augmen a ion (SA)
and wo adap ions o he loss unc ion.
1) SMOTE
In he applied implemen a ion o SMOTE, he amoun o all
classes a e expanded o he numbe o poin s o he la ges
class. The eby, all classes consis ed o he same numbe
o poin s and a e gi en homogeneously dis ibu ed in o he
model. O he a ian s o he SMOTE implemen a ion, e.g.
up o 50% o he size class o a combina ion wi h a US
me hod could be examined al e na i ely. By using SMOTE,
he expansion is con olled by he local neighbo hood, so he
la e lea ning ocus is placed on he a eas o he poin cloud
ha desc ibe in equen and usually mo e complex objec
classes. SMOTE uses he kNN algo i hm o de e mine he k
nea es neighbo s o each poin . The numbe o neighbo s kis
he ac o by which he poin cloud is augmen ed. I k=1,
hen he poin cloud is doubled. I he poin cloud should be
augmen ed o a ce ain numbe , hen he mul iplica ion num-
be is k+1. The unnecessa y poin s ha e o be ( andomly)
dele ed a e wo ds. Fo he calcula ion o he coo dina es o
he augmen ed poin s, he ec o be ween he s a ing poin
and he nea es poin is de e mined. The ec o be ween his
poin s is mul iplied wi h a andom alue om 0 o 1 and
added o he s a ing poin . The coo dina es o a new poin
a e loca ed in be ween bo h o iginal poin s (Fig. 12). Wi h
SMOTE he densi y o he poin cloud is a i icially inc eased
in he a eas o he mino i y classes [56].
FIGURE 12. Calcula ions o da a augmen a ion wi h he SMOTE me hod
by [56]. In his example he poin cloud is mul iplied by k=5 imes.
2) STACK AUGMENTATION
SA is also a da a-based augmen a ion me hod. Howe e , he
da a is no augmen ed in a p ocess ahead o DL-me hod and
no o a ix amoun o poin s. Ins ead he da ase is expanded
du ing he c ea ion o he aining da ase . The ad an age o
he SA is a smalle inc ease o da a, hus he augmen a ion
is mainly applied o poin s o in equen classes. The basic
idea o he app oach has been de eloped by [55]. They spli
he poin cloud in o chunks as ne wo k-inpu (simila o a
oxel). Wi hin a chunk he numbe o poin s was educed o
3832 VOLUME 11, 2023
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
a ixed amoun o 4096 poin s. The con en o he chunks is
analyzed in ega d o he numbe o poin s pe class. Chunks
wi h many poin s o in equen classes a e augmen ed mo e
equen ly han chunks wi h many poin s o equen classes.
The equency o each chunk is de e mined by a nonlinea
unc ion. Using his da a augmen a ion s a egy, [55] a e able
o achie e an inc ease o abou 10% o ecall and p ecision
o he ou doo lase scanne da ase Seman ic3D [37].
FIGURE 13. P ocess o s ack augmen a ion o an op imiza ion o he
class dis ibu ion.
We adap he me hod o [55] o ou da a p ocessing and
simpli y he calcula ion o he augmen a ion ac o . The
augmen a ion and he analysis was pe o med on he basis o
a s ack wi h 1024 poin s, which is he inpu o he Poin Ne .
S acks a e simila o chunks, howe e hey do no ha e a
ixed spa ial dimension, since a s ack consis s o andomly
selec ed poin s o an LNB (Fig. 13). The augmen a ion deg ee
is de e mined by calcula ing he a ge p opo ion o each
class (i all classes would be equal) and compa ing i wi h
he ac ual dis ibu ion. I he ac ual p opo ion o a class is
smalle han he a ge p opo ion, hen s acks in which his
class is dominan a e copied in o an augmen a ion da ase
(Fig. 13, s ep 1). The augmen ed da ase is duplica ed a e all
s acks ha e been analyzed. The numbe o augmen a ions (n)
is de e mined by he ac ha he smalles class mus ha e i s
a ge p opo ion (Fig. 13, s ep 2). Fo ins ance, he poin s o
a poin cloud should be classi ied in o h ee seman ic classes,
so he a ge p opo ion is 33.3% o which he smalles class
is augmen ed.
Wi h his augmen a ion me hod, he ocus should be
di ec ed o he in equen objec s in he poin cloud bu wi h-
ou losing in o ma ion o la ge objec s. Especially poin s in
he edge zones, whe e small and la ge seman ic objec s mee ,
should be used mo e in he aining. S acks ha con ain a
majo i y o in equen classes should be augmen ed. This can
be a s ack, ha consis s only o poin s o he in equen classes
(Fig. 14b), bu also s acks which con ain ew poin s o he
equen classes (Fig. 14a). S acks wi h a majo i y o equen
classes a e no augmen ed (Fig. 14c).
3) WEIGHTED LOSS FUNCTION
The hi d and o h me hods o minimizing he unbalance
class dis ibu ion a e algo i hm-based and add esses he loss
FIGURE 14. Geome ic isualiza ion o he s acks o inpu o a ne wo k.
The black box ep esen s he bounda ies o a s ack. (a) Majo i y o poin s
is om he in equen class. (b) Only poin s om he in equen class a e
p esen . (c) Majo i y o poin s a e om he equen class. This s ack will
no be used o augmen a ion.
unc ion ha is used o calcula e he classi ica ion e o a e
each aining pass. The loss unc ion ype used in his wo k
is he Ca ego ical-C oss-En opy (CCE) loss unc ion which
is ex ended by wo weigh ing op ions. The concep o loss
calcula ion is shown in Fig. 15 and can be b ie ly desc ibed
as ollows.
FIGURE 15. P ocess o ea u e ex ac ion, classi ica ion and loss
calcula ion a Poin Ne . P edic ion class sco e (s) and g ound u h label
a ge ( ) ec o as one-ho enc yp ed ma ix.
The aw aining da a gi en o he ne wo k is unbalanced,
and he dep h ea u es a e compu ed based on he o iginal
da a. Applying a classi ica ion unc ion (e.g., so max), a one-
ho -encode class ec o o each poin is de e mined based
on he ea u es. The class ec o consis s o he same numbe
o elemen s as possible a ge classes (C) exis s. In he case
o he so max unc ion, he ec o is no malized such ha
he ec o sum is always one and he alues o he ec o
exp ess he p obabili y o each class. In he p edic ions o
he DL-me hod, commonly he maximum alue o each poin
ec o is de e mined and he one-ho enc yp ion is dec yp ed.
Also, he class ec o is used o de e mine he loss du ing
he aining. Fo Poin Ne he CCE unc ion om (1) is
VOLUME 11, 2023 3833
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
commonly used.
CCE = −
C
X
c=1
c∗ln(sc) (1)
The CCE unc ion is used o calcula e he loss o a clas-
si ica ion by summa ion o all mul iplica ions be ween he
loga i hmized elemen s o he class ec o (sc) and he co -
esponding elemen s o he a ge ec o ( c) om g ound
u h (GT) label. The eby, mean loss o each inpu s ack and
o he en i e poin cloud is de e mined. The mean loss does
no dis inguish whe he he classes o he poin s a e di icul
o easy o lea n o i he poin s a e equen o in equen .
Classes ha occu in equen ly and ha e a high loss a e
included in he mean alue o a less ex en han classes ha
occu equen ly. The algo i hm lea ns equen ly occu ing
classes be e . To minimize his disad an age o he in equen
classes, he CCE can be imp o ed by a weigh ec o (w),
as desc ibed in (2).
WCCE = −
C
X
c=1
c∗wc∗ln(sc) (2)
The a ge ec o ( c) is mul iplied wi h he weigh ec-
o (wc), allowing he loss o he in equen classes being
emphasized in he mean loss. This loss unc ion is called
weigh ed CCE (WCCE) loss unc ion and is shown in (2).
wc=1−Pc
P(3)
The calcula ion o hese weigh s is usually done by he
class dis ibu ion [90]. The weigh s in ou expe imen s a e
calcula ed and es ed using wo independen expe imen s.
In he i s expe imen , he p opo ion o a class is de e -
mined by calcula ing he a io o he amoun o poin s o
one class (Pc) and calcula ing he amoun o all poin s (P).
The a io o Pcand Pboos he equen classes, so i mus
be sub ac ed om 1 o emphasize he in equen classes (3).
This me hod educes he loss, which can lead o a oo ea ly
e mina ion o he aining phase. To minimize his educ ion
o he loss, he weigh s can be calcula ed acco ding o (4).
wc=1
C−Pc
P+1 (4)
The mino o supe io p opo ion o each class is calcula ed
by (4). Mino o supe io p opo ion esul om he di e ence
o a class dis ibu ion ha ing classes o he iden ical size. This
leads o he ac ha equen classes ge weak weigh s and
in equen classes ge s ong weigh s wi hou changing he
o al amoun o he loss.
The wo WCCE unc ions a e de eloped on he basis
o [91] code and ha e been in eg a ed as an op ion in o he
AEE. A majo ad an age o his me hod is ha he weigh ing
is only e ec i e du ing aining and ( heo e ically) he algo-
i hm does no ha e o be ained again on he o iginal da a.
The ea u e ex ac ion and he classi ica ion i sel a e only
indi ec ly in luenced by lea nable weigh s.
FIGURE 16. Seman ic model o he examina ions. A dis inc ion is made
be ween he main classes o Building pa s, In e io and E oneous
Poin s. Sub-classes a e conside ed sepa a ely s a ing om le el 2. Each
le el can and canno include e oneous poin s.
C. HIERARCHICAL SEMANTIC CLASS COMBINATION
The class de ini ion speci ies he seman ic classes in which
a poin should be subdi ided. The size o he indi idual
seman ic classes is indi ec ly gi en by his class de ini ion.
In applica ions whe e weak ML me hods, such as Random
Fo es , a e used, hie a chical class de ini ions a e used o
inc ease he e iciency [92], [93]. A hie a chical class de i-
ni ion consis s o se e al le els. Gene al classes a e de ined
in he op laye , which a e subdi ided u he and u he
un il he a ge classes o an applica ion a e eached. Fo
ins ance in he op laye , building pa s and in e io can
be dis inguished, which can be u he dis inguished in o
classes, such as Wall, Floo , Ceiling o Window. Using a
hie a chical class de ini ion can be bene icial o he seman ic
segmen a ion because ewe dis inc ions in one s ep need o
be made and he imbalance o he classes a e minimized by a
op imal de ini ion. The s udy o [81] on Poin Ne ++ shows
ha combining ea u e ec o s om di e en hie a chical
laye s o he class de ini ion esul s in a be e disc imina ion
o some classes. In hei esea ch unmanned ae ial ehicle
LIDAR da a is analyzed and [81] s a e ha many di e en
seman ic classes a e geome ically simila . I hese geome ic
classes a e al eady sepa a ed by p e ious le els, con usion
be ween hese classes is elimina ed.
Based on he wo k o [81] and ou heo e ical conside a-
ions, we de eloped a class de ini ion o indoo applica ions,
which is summa ized in Fig. 16. The ull hie a chical class
de ini ion is shown in Tables 13 and 14 in he appendix.
By de eloping his, ade-o s we e made be ween seman ic
easonableness, he di e en geome ic shapes o objec s in
a class, and class sizes. The goal is o o m classes ha a e
usable o a possible seman ic applica ion, ha a e geome i-
cally di e en , and a e as simila in dis ibu ion as possible.
In pa icula , he equal class dis ibu ion is o en in con adic-
ion o o he goals. These goals can possibly be achie ed by
combining he in equen and geome ically simila classes
Doo and Windows in o Opening.
In addi ion o he classes o eal objec s, he class E o-
neous Poin s is o med as an ex a seman ic class o a subse
o he da ase . This seman ic class includes he poin s ha a e
3834 VOLUME 11, 2023
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
TABLE 9. Seman ic accu acy o he class combina ion 3-1 (subse A). The
symbols ↑and ↓indica e a change o mo e and less han 10%, esp.,
compa ed o he base me hod.
TABLE 10. Seman ic accu acy o he class combina ion 3-4 (subse A).
The geome ic accu acy is exp essed by he SDFP poin s.The symbols ↑
and ↓indica e a change o mo e and less han 10%, esp., compa ed o
he base me hod.
accu acy o his class. The SDFP poin s is la ge han
6000 mm, so ha his seman ic class occupies nea ly he
whole oom.
Since he classes Window and Doo a e no lea ned by
he me hods, hey a e combined in an in e media e s ep o
he class Opening. The idea behind he summa y is, ha his
class could be subdi ided in he case o a good seman ic
segmen a ion in a ollowing s ep, wi hou nega i ely a ec ing
he class Wall.
Fo combina ion 3-4 he Base, WCCEa and WCCEb
me hods mee he equi emen s wi h es ic ions. The me hod
SA does no mee he equi emen s and he SOMTE class
mee s he equi emen s. The CE a e o he Base, WCCEa
and WCCEb me hods is high wi h a alue o 0.72. The
SMOTE me hod shows he op imal class dis ibu ion and he
SA me hod imp o es he CE a e o a su icien sco e o
0.26. Due o he ough desc ip ion o he dis ibu ion o hese
combina ion, i can be seen ha an inc ease o he seman ic
accu acy is achie ed (Table 10). The ecognizabili y o he
TABLE 11. Seman ic accu acy o he class combina ion 3-2 (subse B).
The symbols ↑and ↓indica e a change o mo e and less han 10%, esp.,
compa ed o he base me hod.
class Opening is low compa ed o he o he classes. The PP
o his class is o almos any me hod below 50%, and he
SDFP poin s is highe han 3600 mm. Ne e heless, a good
seman ic segmen a ion can be pe o med wi h he SMOTE
me hod. Bu i does no wo k o he combina ion 3-1.
Fo combina ions 3-1 and 3-4, he TL phase leads o esul s
compa able o he Base me hod.
Fo combina ion 3-2, no me hod mee he equi emen s.
The CE a e o he Base, WCCEa and WCCEb me hods
is wi h 0.72 high. The SMOTE me hod shows he op imal
class dis ibu ion and he SA me hod imp o es he a e o a
mode a e sco e o 0.38 (Table 11).
Fo combina ion 3-2, he majo i y o he poin s o he
classes Doo and Window a e no assigned o he co ec
classes. In addi ion, he geome ically simila class Wall is
less ecognized compa ed o combina ions 3-1 and 3-4. The
PP o Doo and Window is low wi h a maximum o 33%
o e all me hods (Table 11). The geome ic accu acy o he
wo classes has a high SDFP poin s. Fo he class Window,
he SDFP poin s is la ge han 4300 mm and o he class
Doo i is la ge han 3600 mm. Based on hese e alua ion
pa ame e s, i can be s a ed ha he class dis ibu ion has
no in luence in his case. A seman ic segmen a ion wi h he
class combina ion 2-2 leads o a high seman ic and geome ic
accu acy only o he classes Floo and Ceiling. Also, he
combina ion o he classes Doo and Window o Opening in
an in e media e s ep is es ed in combina ion 3-3, oo.
Fo combina ion 3-3, he me hods Base, SA and WCCEb
mee he equi emen s. The SMOTE and he WCCEa me h-
ods mee he equi emen s wi h es ic ions. The CE a e o
he Base, WCCEa and WCCEb me hods is wi h alue o
0.40 mode a e. The SMOTE me hod shows he op imal class
dis ibu ion and he SA me hod imp o es he a e o an sco e
o 0.08. The class dis ibu ion becomes a o able a e he
consolida ion (Table 12).
The combina ion 3-3 leads o an inc ease in he seman ic
segmen a ion accu acy o all classes. The RP o he class
Opening is highe han 50% o all me hods. The class Wall,
VOLUME 11, 2023 3841

E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
TABLE 12. Seman ic accu acy o he class combina ion 3-3 (subse B).
The symbols ↑and ↓indica e a change o mo e and less han 10%, esp.,
compa ed o he base me hod.
wi h which he class Opening is o en con used, is co ec ly
ecognized only by SMOTE and WCCEa o he poin majo -
i y. Based on he low PP o 29% o 32%, he con usion
wi h he class Opening is con i med (Table 12). E en wi h
his combina ion, he neighbo ing classes Wall and Opening
canno be accu a ely sepa a ed. La ge a ia ions o di e en
ooms a e obse ed he e, bu he e a e no ooms ha can be
segmen ed seman ically e y accu a ely. An in luence o he
class E oneous Poin s is no obse ed.
A TL wi h he Base me hod esul s in an inc ease o 1% o
3% o RP and PP o all me hods and classes.
D. SUMMERY AND OVERALL FINDINGS
The esul s show o he in es iga ed se ings, class de ini ion
and class combina ion, ha wo examined DHPs ha e only a
mino in luence on he seman ic and geome ic accu acy o
seman ic segmen a ion. The applied augmen a ion me hods
lead o an imp o ed ecogni ion o he in equen classes.
In he classi ica ion s ep, poin s a e mo e o en assigned
o an in equen class. This leads o a educ ion in PP o
in equen classes. The SDFP poin s emains unchanged
o e en dec eases due o he used augmen a ion me hods.
The PP o he segmen a ion imp o es s onge o equen
classes.
The numbe o classes i sel has no in luence on he
seman ic segmen a ion pe o mance. Ins ead, he geome ic
simila i y and he dis ance o he objec s a e impo an o
dis inguishing classes. The classes Floo and Ceiling can be
well dis inguished because o he la ge geome ic dis ance
(no sha ed bounda y), whe eas he classes Window and
Wall a e di icul o dis inguish. When de ining a class, he
geome ic dis inguishabili y o he objec s mus be aken
in o accoun . This mus be alid o he en i e da ase ,
since ooms, o example, a y s ong in size, shape and
u nishing.
Using a class combina ion wi hou e oneous poin s leads
o an inc ease in PP and RP o he classes ha al eady ha e
a highe PP in a seman ic segmen a ion wi h he class E o-
neous Poin s. Classes ha ha e a lowe seman ic PP in he
seman ic segmen a ion wi h e oneous poin s a e ecognized
wo se wi hou his class and ha e a lowe PP.
Applying an addi ional TL phase, whe e he p e ious esul
se es as a s a ing poin o aining wi h he Base me hod,
does no lead o an inc ease in accu acy. Fo he SMOTE
and SA me hods, i esul in less equen de ec ion o he
in equen classes and a simila pe o mance as wi h he Base
me hod.
VII. CONCLUSION AND OUTLOOK
The pe o mance o DL-me hods in seman ic segmen a ion
is in luenced among o he ac o s by HPs. In his wo k,
he DHP, class combina ions and me hods o minimize he
unbalanced classes ha e been s udied. Fo he in es iga ion,
an AEE has been de eloped in which he es ablished Poin Ne
a chi ec u e has been implemen ed.
The class combina ions we e o ganized in a hie a chic
o de , so ha a seman ic segmen a ion is pe o med only
o a pa icula pa o he poin cloud, o combina ions in
le el 3 and 4. In equen classes we e combined and seman i-
cally segmen ed a e wa ds. This esul ed in highe seman ic
and geome ic accu acy o he class Building Pa s and i s
equen sub classes. The class E oneous Poin s leads o a
sligh ly highe seman ic accu acy o in equen classes.
The use o wo da a-based augmen a ion me hods and wo
algo i hm-based me hods only achie ed a small inc ease in
seman ic ecogni ion. The applied me hods usually inc ease
he RP, so ha he in equen classes a e ecognized mo e
o en and he mo e equen classes become mo e p ecise.
This is ad an ageous o he combina ions in le el 1 and 2,
because only he mo e equen classes a e needed o a
building modeling.
The p ima y goal o his wo k is o inc ease RP and PP
o o e 50% o all classes using he augmen a ion me hods.
This goal was only achie ed o he combina ion 3-4 wi h he
SMOTE me hod. An inc ease in RP o a alue highe han
50% is achie ed wi h he SMOTE me hod addi ional ou
imes, whe eas he WCCEa me hod ul ills i o i e o he
se en combina ions. This inc ease o he RP is achie ed ou
imes wi h he WCCEb me hod. The SA me hod esul s in an
inc ease in RP and PP, bu less han 50% in mos cases. Wi h
he Base me hod, a RP o all classes highe han 50% was
achie ed wice. The p ima y goal was pa ly achie ed.
In he cou se o he in es iga ion, i was disco e ed ha
he geome ic simila i y o classes mus be conside ed when
o ming he class combina ions. Also, he choice o LNB has
a la ge impac on he segmen a ion pe o mance. Based on
ou obse a ions, he choice o he local neighbo hood and he
di e ences be ween he indi idual ooms in he da ase a e
highly in luen ial. The ocus o u he in es iga ions should
be on hese DHPs. The in luence o da a augmen a ion me h-
ods is measu able, bu cu en ly o li le ele ance acco ding
o ou sample BIM applica ion. In e ms o augmen a ion
me hods, we plan o examine he impac o US me hods
as well as a combina ion o US me hods, OS me hods and
weigh ed loss unc ions.
3842 VOLUME 11, 2023
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
TABLE 13. Seman ic class de ini ions o he classes o wo op le els.
TABLE 14. Seman ic class de ini ion o classes o he supe -class Building Pa s.
APPENDIX
CLASS DEFINITION
The class de ini ion o he wo uppe le els (Fig. 14) a e
shown in Table 13. The class de ini ions o he supe -class
Building Pa s a e summa ized in Table 14. This class de ini-
ion is de eloped o a seman ic segmen a ion as a basis o
c ea ing a BIM model o a public building.
ACKNOWLEDGMENT
Many hanks o he anno a o s: Clemens Semmel o h, S e-
anie S and, and Olga Konko a. Special hanks o Anne e
Scheide , Lena Ba ne ske, and Ch is ophe Klocke o p oo -
eading.
REFERENCES
[1] J. Zhao, X. Zhang, and Y. Wang, ‘‘Indoo 3D poin clouds seman ic seg-
men a ion bases on modi ied poin ne ne wo k,’’ In . A ch. Pho og amm.,
Remo e Sens. Spa ial In . Sci., ol. 43, pp. 369–373, Aug. 2020.
[2] M. Soilán, R. Lindenbe gh, B. Ri ei o, and A. Sánchez-Rod íguez,
‘‘Poin ne o he au oma ic classi ica ion o ae ial poin clouds,’’ ISPRS
Ann. Pho og amm., Remo e Sens. Spa ial In . Sci., ol. 4, pp. 445–452,
May 2019.
[3] S. De Gey e , J. Ve mande e, H. De Win e , M. Bassie , and M. Ve gauwen,
‘‘Poin cloud alida ion: On he impac o lase scanning echnologies on
he seman ic segmen a ion o BIM modeling and e alua ion,’’ Remo e
Sens., ol. 14, no. 3, p. 582, Jan. 2022.
[4] F. Noichl, A. B aun, and A. Bo mann, ‘‘‘BIM- o-scan’ o scan- o-BIM:
Gene a ing ealis ic syn he ic g ound u h poin clouds based on indus-
ial 3D models,’’ in P oc. Eu . Con . Compu . Cons uc ., Jul. 2021,
pp. 164–172.
[5] S. Chen, J. Fang, Q. Zhang, W. Liu, and X. Wang, ‘‘Hie a chical agg ega-
ion o 3D ins ance segmen a ion,’’ in P oc. IEEE/CVF In . Con . Compu .
Vis. (ICCV), Oc . 2021, pp. 15447–15456.
[6] F. Poux and R. Billen, ‘‘Voxel-based 3D poin cloud seman ic segmen a-
ion: Unsupe ised geome ic and ela ionship ea u ing s deep lea ning
me hods,’’ ISPRS In . J. Geo-In ., ol. 8, no. 5, p. 213, May 2019.
[7] Y. Xie, J. Tian, and X. X. Zhu, ‘‘Linking poin s wi h labels in 3D: A e iew
o poin cloud seman ic segmen a ion,’’ IEEE Geosci. Remo e Sens. Mag.,
ol. 8, no. 4, pp. 38–59, Dec. 2020.
[8] C. Mo bidoni, R. Pie dicca, R. Qua ini, and E. F on oni, ‘‘G aph CNN
wi h adius dis ance o seman ic segmen a ion o his o ical buildings TLS
poin clouds,’’ In . A ch. Pho og amm., Remo e Sens. Spa ial In . Sci.,
ols. XLIV-4/W1-2020, pp. 95–102, Sep. 2020.
[9] L. Winiwa e and G. Mandlbu ge , ‘‘Classi ica ion o 3D poin clouds
using deep neu al ne wo ks,’’ in P oc. D eilände agung de DGPF, de
OVG und de SGPF in Vienna, ol. 28. Ös e eich-Publika ionen de
DGPF, 2019, pp. 663–674.
[10] B. Gao, Y. Pan, C. Li, S. Geng, and H. Zhao, ‘‘A e we hung y o 3D LiDAR
da a o seman ic segmen a ion? A su ey o da ase s and me hods,’’ IEEE
T ans. In ell. T ansp. Sys ., ol. 23, no. 7, pp. 6063–6081, Jul. 2022.
[11] H. Riemenschneide , A. Bodis-Szomo u, J. Weissenbe g, and L. V. Gool,
‘‘Lea ning whe e o classi y in mul i- iew seman ic segmen a ion,’’ in
P oc. Eu . Con . Compu . Vis. (ECCV). Sp inge , 2014, pp. 516–532.
[12] D. Ma u ana and S. Sche e , ‘‘VoxNe : A 3D con olu ional neu al ne wo k
o eal- ime objec ecogni ion,’’ in P oc. IEEE/RSJ In . Con . In ell.
Robo s Sys . (IROS), Sep. 2015, pp. 922–928.
[13] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, ‘‘Poin Ne : Deep lea ning on
poin se s o 3D classi ica ion and segmen a ion,’’ in P oc. IEEE Con .
Compu . Vis. Pa e n Recogni . (CVPR), Jul. 2017, pp. 77–85.
[14] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. T igoni, and
A. Ma kham, ‘‘RandLA-Ne : E icien seman ic segmen a ion o la ge-
scale poin clouds,’’ in P oc. IEEE/CVF Con . Compu . Vis. Pa e n Recog-
ni . (CVPR), Jun. 2020, pp. 11105–11114.
VOLUME 11, 2023 3843
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
[15] T. Bende , M. Hä ig, E. Jaspe s, M. K äme , M. May, M. Schlund , and
N. Tu ianskyj, ‘‘Building in o ma ion modeling,’’ in CAFM-Handbuch.
Wiesbaden, Ge many: Sp inge , 2018, pp. 295–324
[16] Enginee ing Su ey, S anda d DIN18710, Sep. 2010.
[17] M.-O. Löwne and G. G öge , ‘‘Das neue lod konzep ü ci ygml 3.0,’’ in
GeoFo um MV, ol. 13. Ros ock-Wa nemünde, Ge many, 2017, pp. 23–30.
[18] (Jun. 24, 2021). BuildingSMART. Indus y Founda ion Classes 4.0.2.1.
[Online]. A ailable: h ps://s anda ds.buildingsma .o g
[19] E. S. Malin e ni, R. Pie dicca, M. Paolan i, M. Ma ini, C. Mo bidoni,
F. Ma one, and A. Lingua, ‘‘Deep lea ning o seman ic segmen a ion o
3D poin cloud,’’ In . A ch. Pho og amm., Remo e Sens. Spa ial In . Sci.,
ol. 43, pp. 735–742, Aug. 2019.
[20] D. Passos and P. Mish a, ‘‘A u o ial on au oma ic hype pa ame-
e uning o deep spec al modelling o eg ession and classi i-
ca ion asks,’’ Chemome ic In ell. Lab. Sys ., ol. 223, Ap . 2022,
A . no. 104520. [Online]. A ailable: h ps://www.sciencedi ec .com/sci
ence/a icle/pii/S0169743922000314
[21] M. Claesen and B. De Moo , ‘‘Hype pa ame e sea ch in machine lea n-
ing,’’ 2015, a Xi :1502.02127.
[22] L. N. Smi h, ‘‘A disciplined app oach o neu al ne wo k hype -pa ame e s:
Pa 1—Lea ning a e, ba ch size, momen um, and weigh decay,’’ 2018,
a Xi :1803.09820.
[23] C. Cooney, A. Ko ik, R. Folli, and D. Coyle, ‘‘E alua ion o hype -
pa ame e op imiza ion in machine and deep lea ning me hods o
decoding imagined speech EEG,’’ Senso s, ol. 20, no. 16, p. 4629,
Aug. 2020.
[24] T. Yu and H. Zhu, ‘‘Hype -pa ame e op imiza ion: A e iew o algo i hms
and applica ions,’’ Ma . 2020, a Xi :2003.05689.
[25] M. Feu e and F. Hu e , ‘‘Hype pa ame e op imiza ion,’’ in Au oma ed
Machine Lea ning. Cham, Swi ze land: Sp inge , 2019, pp. 3–33.
[26] J. Behley, M. Ga bade, A. Milio o, J. Quenzel, S. Behnke, C. S achniss,
and J. Gall, ‘‘Seman icKITTI: A da ase o seman ic scene unde s anding
o LiDAR sequences,’’ in P oc. IEEE/CVF In . Con . Compu . Vis. (ICCV),
Oc . 2019, pp. 9296–9306.
[27] X. Wang, B. Zhou, Y. Shi, X. Chen, Q. Zhao, and K. Xu, ‘‘Shape2Mo ion:
Join analysis o mo ion pa s and a ibu es om 3D shapes,’’ in P oc.
IEEE/CVF Con . Compu . Vis. Pa e n Recogni . (CVPR), Jun. 2019,
pp. 8868–8876.
[28] W. Zimme , A. Rangesh, and M. T i edi, ‘‘3D BAT: A semi-au oma ic,
web-based 3D anno a ion oolbox o ull-su ound, mul i-modal da a
s eams,’’ in P oc. IEEE In ell. Vehicles Symp. (IV), Jun. 2019,
pp. 1816–1821.
[29] M. Weinmann, B. Ju zi, C. Malle , and M. Weinmann, ‘‘Geome ic
ea u es and hei ele ance o 3D poin cloud classi ica ion,’’ ISPRS
Ann. Pho og amm., Remo e Sens. Spa ial In . Sci., ol. 4, pp. 157–164,
May 2017.
[30] M. Negassi, D. Wagne , and A. Rei e e , ‘‘Sma (sampling)augmen : Op i-
mal and e icien da a augmen a ion o seman ic segmen a ion,’’ Algo-
i hms, ol. 15, no. 5, p. 165, May 2022.
[31] J. M. Johnson and T. M. Khoshgo aa ,‘‘Su ey on deep lea ning wi h class
imbalance,’’ J. Big Da a, ol. 6, no. 1, Ma . 2019.
[32] M. Weinmann, B. Ju zi, S. Hinz, and C. Malle , ‘‘Seman ic poin cloud
in e p e a ion based on op imal neighbo hoods, ele an ea u es and
e icien classi ie s,’’ ISPRS J. Pho og amm. Remo e Sens., ol. 105,
pp. 286–304, Jul. 2015.
[33] M. Weinmann, B. Ju zi, and C. Malle , ‘‘Seman ic 3D scene in e p e a ion:
A amewo k combining op imal neighbo hood size selec ion wi h ele an
ea u es,’’ ISPRS Ann. Pho og amm., Remo e Sens. Spa ial In . Sci., ol. 2,
pp. 181–188, Aug. 2014.
[34] T. Hackel, ‘‘La ge-scale machine lea ning o poin cloud p ocessing,’’
Ph.D. disse a ion, Ins . Geodesy Pho og amm., ETH Zü ich, Zü ich,
Swi ze land, 2018.
[35] N. Shukla, Machine Lea ning Wi h Tenso Flow. Shel e Island, NY, USA:
Manning, Publ., 2018.
[36] E. Camu o, D. Ma i, and S. Milani, ‘‘Recen ad ancemen s in lea ning
algo i hms o poin clouds: An upda ed o e iew,’’ Senso s, ol. 22, no. 4,
p. 1357, Feb. 2022.
[37] T. Hackel, N. Sa ino , L. Ladicky, J. D. Wegne , K. Schindle , and
M. Polle eys, ‘‘Seman ic3D.Ne : A new la ge-scale poin cloud classi i-
ca ion benchma k,’’ ISPRS Ann. Pho og amm. Remo e Sens. Spa ial In .
Sci., ol. 4, pp. 91–98, May 2017.
[38] S. Wi ges, T. Fische , C. S ille , and J. B. F ias, ‘‘Objec de ec ion
and classi ica ion in occupancy g id maps using deep con olu ional ne -
wo ks,’’ in P oc. 21s In . Con . In ell. T ansp. Sys . (ITSC), No . 2018,
pp. 3530–3535.
[39] A. Dai, D. Ri chie, M. Bokeloh, S. Reed, J. S u m, and M. Nießne ,
‘‘ScanComple e: La ge-scale scene comple ion and seman ic segmen a ion
o 3D scans,’’ in P oc. CVPR, ol. 1, Jun. 2018, p. 2.
[40] B. Yang, W. Luo, and R. U asun, ‘‘PIXOR: Real- ime 3D objec de ec-
ion om poin clouds,’’ in P oc. IEEE/CVF Con . Compu . Vis. Pa e n
Recogni ., Jun. 2018, pp. 7652–7660.
[41] W. Liu, J. Sun, W. Li, T. Hu, and P. Wang, ‘‘Deep lea ning on poin clouds
and i s applica ion: A su ey,’’ Senso s, ol. 19, no. 19, p. 4188, Sep. 2019.
[42] E. Ba ne ske and S. Ha ald, ‘‘Au oma isch seman isch-segmen ie e punk-
wolken - möglichkei en und he aus o de ungen,’’ in DVW-Semina MST
on (A)nwendungen bis (Z)ukun s echnologien, ol. 103. Augsbu g,
Ge many: Wißne -Ve lag, 2022, pp. 173–184.
[43] D. Koguciuk, Ł. Chechliński, and T. El-Gaaly, ‘‘3D objec ecogni ion wi h
ensemble lea ning—A s udy o poin cloud-based deep lea ning models,’’
in Ad ances in Visual Compu ing. Cham, Swi ze land: Sp inge , 2019,
pp. 100–114.
[44] S. A. Bello, S. Yu, C. Wang, J. M. Adam, and J. Li, ‘‘Re iew: Deep lea ning
on 3D poin clouds,’’ Remo e Sens., ol. 12, no. 11, p. 1729, May 2020.
[45] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, ‘‘Poin Ne ++: Deep hie a chical
ea u e lea ning on poin se s in a me ic space,’’ in P oc. Ad . Neu al In .
P ocess. Sys ., 2017, pp. 5099–5108.
[46] F. Engelmann, T. Kon ogianni, A. He mans, and B. Leibe, ‘‘Explo ing
spa ial con ex o 3D seman ic segmen a ion o poin clouds,’’ in P oc.
IEEE Con . Compu . Vis. Pa e n Recogni ., Oc . 2017, pp. 716–724.
[47] H.-I. Lin and M. C. Nguyen, ‘‘Boos ing mino i y class p edic ion on
imbalanced poin cloud da a,’’ Appl. Sci., ol. 10, no. 3, p. 973, Feb. 2020.
[48] Z. Jiang, T. Pan, C. Zhang, and J. Yang, ‘‘A new o e sampling me hod
based on he classi ica ion con ibu ion deg ee,’’ Symme y, ol. 13, no. 2,
p. 194, Jan. 2021.
[49] J. V. Hulse, T. M. Khoshgo aa , and A. Napoli ano, ‘‘Expe imen al pe -
spec i es on lea ning om imbalanced da a,’’ in P oc. 24 h In . Con . Mach.
Lea n. (ICML), 2007, pp. 935–942.
[50] I. Mani and I. Zhang, ‘‘KNN app oach o unbalanced da a dis ibu ions:
A case s udy in ol ing in o ma ion ex ac ion,’’ in P oc. Wo kshop Lea n.
Imbalanced Da ase s (ICML), ol. 126, 2003, pp. 1–7.
[51] M. Kuba and S. Ma win, ‘‘Add essing he cu se o imbalanced aining
se s: One-sided selec ion,’’ in P oc. 14 h In . Con . Mach. Lea n., 1997,
pp. 179–186.
[52] R. Ba andela, R. M. Valdo inos, J. S. Sánchez, and F. J. Fe i, ‘‘The imbal-
anced aining sample p oblem: Unde o o e sampling?’’ in S uc u al,
Syn ac ic, and S a is ical Pa e n Recogni ion (Lec u e No es in Compu e
Science). Be lin, Ge many: Sp inge , 2004, pp. 806–814.
[53] T. Jo and N. Japkowicz, ‘‘Class imbalances e sus small disjunc s,’’ ACM
SIGKDD Explo . Newsle ., ol. 6, no. 1, pp. 40–49, Jun. 2004.
[54] P. Hensman and D. Masko, ‘‘The impac o imbalanced aining da a o
con olu ional neu al ne wo ks,’’ KTH, School Compu . Sci. Commun.,
S ockholm, Sweden, May 2015. [Online]. A ailable: h ps://www.
k h.se/social/ iles/588617eb 2765401c cc478c/PHensmanDMasko_
dkand15.pd
[55] D. G i i hs and J. Boehm, ‘‘Weigh ed poin cloud augmen a ion o neu al
ne wo k aining da a class-imbalance,’’ In . A ch. Pho og amm., Remo e
Sens. Spa ial In . Sci., ol. 43, pp. 981–987, Jun. 2019.
[56] N. V. Chawla, K. W. Bowye , L. O. Hall, and W. P. Kegelmeye , ‘‘SMOTE:
Syn he ic mino i y o e -sampling echnique,’’ J. A i . In ell. Res., ol. 16,
pp. 321–357, Jun. 2002.
[57] H. Lee, M. Pa k, and J. Kim, ‘‘Plank on classi ica ion on imbalanced
la ge scale da abase ia con olu ional neu al ne wo ks wi h ans e
lea ning,’’ in P oc. IEEE In . Con . Image P ocess. (ICIP), Sep. 2016,
pp. 3713–3717.
[58] S. Wang, W. Liu, J. Wu, L. Cao, Q. Meng, and P. J. Kennedy, ‘‘T aining
deep neu al ne wo ks on imbalanced da a se s,’’ in P oc. In . Join Con .
Neu al Ne w. (IJCNN), Jul. 2016, pp. 4368–4374.
[59] T.-Y. Lin, P. Goyal, R. Gi shick, K. He, and P. Dollá , ‘‘Focal loss o
dense objec de ec ion,’’ in P oc. In . Con . Compu . Vis. (ICCV), Oc . 2017,
pp. 2999–3007.
[60] H. Wang, Z. Cui, Y. Chen, M. A idan, A. B. Abdallah, and
A. K onze , ‘‘P edic ing hospi al eadmission ia cos -sensi i e deep
lea ning,’’ IEEE/ACM T ans. Compu . Biol. Bioin ., ol. 15, no. 6,
pp. 1968–1978, Dec. 2018.
[61] C. Zhang, K. C. Tan, and R. Ren, ‘‘T aining cos -sensi i e deep belie
ne wo ks on imbalance da a p oblems,’’ in P oc. In . Join Con . Neu al
Ne w. (IJCNN), Jul. 2016, pp. 4362–4367.
[62] Y. Zhang, L. Shuai, Y. Ren, and H. Chen, ‘‘Image classi ica ion wi h ca -
ego y cen e s in class imbalance si ua ion,’’ in P oc. 33 d You h Academic
Annu. Con . Chin. Assoc. Au om. (YAC), May 2018, pp. 359–363.
3844 VOLUME 11, 2023
E. Ba ne ske, H. S e nbe g: E alua ion o Class Dis ibu ion and Class Combina ions on Seman ic Segmen a ion
[63] S. H. Khan, M. Haya , M. Bennamoun, F. A. Sohel, and R. Togne i, ‘‘Cos -
sensi i e lea ning o deep ea u e ep esen a ions om imbalanced da a,’’
IEEE T ans. Neu al Ne w. Lea n. Sys ., ol. 29, no. 8, pp. 3573–3587,
Aug. 2018.
[64] W. Ding, D.-Y. Huang, Z. Chen, X. Yu, and W. Lin, ‘‘Facial ac ion ecog-
ni ion using e y deep ne wo ks o highly imbalanced class dis ibu ion,’’
in P oc. Asia–Paci ic Signal In . P ocess. Assoc. Annu. Summi Con .
(APSIPA ASC), Dec. 2017, pp. 1368–1372.
[65] M. Buda, A. Maki, and M. A. Mazu owski, ‘‘A sys ema ic s udy o he
class imbalance p oblem in con olu ional neu al ne wo ks,’’ Neu al Ne w.,
ol. 106, pp. 249–259, Oc . 2018.
[66] J. Mo el, A. Bac, and T. Kanai, ‘‘Segmen a ion o unbalanced and in-
homogeneous poin clouds and i s applica ion o 3D scanned ees,’’ Vis.
Compu ., ol. 36, nos. 10–12, pp. 2419–2431, Sep. 2020.
[67] C. Huang, Y. Li, C. C. Loy, and X. Tang, ‘‘Lea ning deep ep esen a ion
o imbalanced classi ica ion,’’ in P oc. IEEE Con . Compu . Vis. Pa e n
Recogni ., Jun. 2016, pp. 5375–5384.
[68] S. Ando and C. Y. Huang, ‘‘Deep o e -sampling amewo k o classi ying
imbalanced da a,’’ in Machine Lea ning and Knowledge Disco e y in
Da abases. Cham, Swi ze land: Sp inge , 2017, pp. 770–785.
[69] Q. Dong, S. Gong, and X. Zhu, ‘‘Imbalanced deep lea ning by mino i y
class inc emen al ec i ica ion,’’ IEEE T ans. Pa e n Anal. Mach. In ell.,
ol. 41, no. 6, pp. 1367–1381, Jun. 2019.
[70] T.-Y. Liu, ‘‘EasyEnsemble and ea u e selec ion o imbalance da a
se s,’’ in P oc. In . Join Con . Bioin ., Sys . Biol. In ell. Compu ., 2009,
pp. 517–520.
[71] N. V. Chawla, A. Laza e ic, L. O. Hall, and K. W. Bowye , ‘‘SMOTE-
Boos : Imp o ing p edic ion o he mino i y class in boos ing,’’ in
Knowledge Disco e y in Da abases. Be lin, Ge many: Sp inge , 2003,
pp. 107–119.
[72] O. Hassaan, A. Shamail, Z. Bu , and M. Taj, ‘‘Poin cloud segmen a ion
using hie a chical ee o a chi ec u al models,’’ in P oc. IEEE In . Con .
Acous ., Speech Signal P ocess. (ICASSP), May 2019, pp. 1582–1586.
[73] B.-S. Hua, Q.-H. Pham, D. T. Nguyen, M.-K. T an, L.-F. Yu, and
S.-K. Yeung, ‘‘SceneNN: A scene meshes da ase wi h aNNo a ions,’’ in
P oc. 4 h In . Con . 3D Vis. (3DV), Oc . 2016, pp. 92–101.
[74] L. Jiang, H. Zhao, S. Liu, X. Shen, C.-W. Fu, and J. Jia, ‘‘Hie a chi-
cal poin -edge in e ac ion ne wo k o poin cloud seman ic segmen a-
ion,’’ in P oc. IEEE/CVF In . Con . Compu . Vis. (ICCV), Oc . 2019,
pp. 10432–10440.
[75] Y. Li and G. Baciu, ‘‘HSGAN: Hie a chical g aph lea ning o poin cloud
gene a ion,’’ IEEE T ans. Image P ocess., ol. 30, pp. 4540–4554, 2021.
[76] T. Jiang, J. Sun, S. Liu, X. Zhang, Q. Wu, and Y. Wang, ‘‘Hie a chical
seman ic segmen a ion o u ban scene poin clouds ia g oup p oposal
and g aph a en ion ne wo k,’’ In . J. Appl. Ea h Obse . Geoin o ma ion,
ol. 105, Dec. 2021, A . no. 102626.
[77] T. H. Kolbe, T. Ku zne , C. S. Smy h, C. Nagel, C. Roensdo , and
C. Heazel, ‘‘OGC ci y geog aphyma kup language (Ci yGML) pa 1:
Concep ual models anda d,’’ Pen Geospa ial Conso ium, A ling on, VA,
USA, Tech. Rep. 20-010, 2021.
[78] Y. Ve die, F. La a ge, and P. Alliez, ‘‘LOD gene a ion o u ban scenes,’’
ACM T ans. G aph., ol. 34, no. 3, pp. 1–14, May 2015.
[79] L. Tang, L. Li, S. Ying, and Y. Lei, ‘‘A ull le el-o -de ail speci ica ion
o 3D building models combining indoo and ou doo scenes,’’ ISPRS In .
J. Geo-In ., ol. 7, no. 11, p. 419, Oc . 2018.
[80] H. Ledoux, K. A. Oho i, K. Kuma , B. Dukai, A. Labe ski, and S. Vi alis,
‘‘Ci yJSON: A compac and easy- o-use encoding o he Ci yGML
da a model,’’ Open Geospa ial Da a, So w. S anda ds, ol. 4, no. 1,
pp. 127–140, Jun. 2019.
[81] X. Li, C. Li, Z. Tong, A. Lim, J. Yuan, Y. Wu, J. Tang, and R. Huang,
‘‘Campus3D: A pho og amme y poin cloud benchma k o hie a chical
unde s anding o ou doo scene,’’ in P oc. 28 h ACM In . Con . Mul imedia,
Oc . 2020, pp. 238–246.
[82] V. S ojano ic e al., ‘‘A concep ual digi al win o 5G indoo na iga ion,’’
in P oc. 11 h In . Con . Mobile Se ices, Resou ., Use s (MOBILITY),
Ap . 2021, pp. 5–14.
[83] Reaching New Le els, z+ Image 5016, Use Manual, 2.1,
Zolle +F öhlich-GmbH, Wangen im Allgäu, Ge many, 2019.
[84] CloudCompa e. 3D Poin Cloud and Mesh P ocessing So wa e
Open-Sou ce P ojec . Accessed: Jun. 24, 2021. [Online]. A ailable:
h p://www.cloudcompa e.o g/
[85] Au odesk-Recap. You ube Channel. Accessed: Jun. 24, 2022. [Online].
A ailable: h ps://www.you ube.com/use /au odesk ecap/
[86] E. Ba ne ske and H. S e nbe g, ‘‘E alua ing he quali y o seman ic seg-
men ed 3D poin clouds,’’ Remo e Sens., ol. 14, no. 3, p. 446, Jan. 2022.
[87] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, ‘‘Deep
lea ning o 3D poin clouds: A su ey,’’ IEEE T ans. Pa e n Anal. Mach.
In ell., ol. 43, no. 12, pp. 4338–4364, Dec. 2021.
[88] S. Rakshi and S. Paul. (Oc . 2020). Poin Cloud Segmen a ion wi h Poin -
Ne . [Online]. A ailable: h ps://gi hub.com/soumik12345/poin -cloud-
segmen a ion
[89] Z. Li, K. Kamni sas, and B. Glocke , ‘‘Analyzing o e i ing unde class
imbalance in neu al ne wo ks o image segmen a ion,’’ IEEE T ans. Med.
Imag., ol. 40, no. 3, pp. 1065–1077, Ma . 2021.
[90] M. Abdou, M. Elkha eeb, I. Sobh, and A. Elsallab, ‘‘End- o-end
3D-Poin Cloud seman ic segmen a ion o au onomous d i ing,’’ 2019,
a Xi :1906.10964.
[91] M. Ba el. (Dec. 2019). Mul i-Class Weigh ed Loss o Seman ic Image
Segmen a ion in Ke as/Tenso low. Accessed: Sep. 23, 2022. [Online].
A ailable: h ps://s acko e low.com/ques ions/59520807/mul i-class-
weigh ed-loss- o -seman ic-image-segmen a ion-in-ke as- enso low
[92] E. G illi, D. Dininno, G. Pe ucci, and F. Remondino, ‘‘FROM 2D TO 3D
supe ised segmen a ion and classi ica ion o cul u al he i age applica-
ions,’’ In . A ch. Pho og amm., Remo e Sens. Spa ial In . Sci., ol. 43,
pp. 399–406, May 2018.
[93] S. Te uggi, E. G illi, M. Russo, F. Fassi, and F. Remondino, ‘‘A hie a chical
machine lea ning app oach o mul i-le el and mul i- esolu ion 3D poin
cloud classi ica ion,’’ Remo e Sens., ol. 12, no. 16, p. 2598, Aug. 2020.
[94] E. Ba ne ske and H. S e nbe g, ‘‘Klassi izie ung on ehle ha gemesse-
nen punk en in 3D-punk wolken mi con ne ,’’ in Ingenieu e messung 20.
Bei äge zum 19. T. Wunde lich, Ed. Be lin, Ge many: He be Wichmann
Ve lag, Ma . 2020, pp. 127–139.
[95] S. J. Reddi, S. Kale, and S. Kuma , ‘‘On he con e gence o Adam and
beyond,’’ in P oc. In . Con . Lea n. Rep esen ., 2018, pp. 1–23.
[96] C. Schuld , H. Shoush a i, N. Hellweg, and H. S e nbe g, ‘‘L5IN: O e iew
o an indoo na iga ion pilo p ojec ,’’ Remo e Sens., ol. 13, no. 4, p. 624,
Feb. 2021.
[97] J. Blankenbach, Ingenieu geodäsie. Be lin, Ge many: Sp inge , 2017,
pp. 23–53.
[98] BIM-Fo um. Le el o De elopmen Speci ica ion Pa 1 & Commen a y.
Accessed: Dec. 8, 2020. [Online]. A ailable: h ps://bim o um.o g/lod/
EIKE BARNEFSKE ecei ed he bachelo ’s and
mas e ’s deg ees in geoma ics om Ha enCi y
Uni e si y Hambu g, and he M.Sc. deg ee,
in 2016. He is cu en ly pu suing he Ph.D. deg ee
in hyd og aphy and geodesy. F om 2016 o 2017,
he wo ked as a Resea ch Assis an o enginee -
ing geodesy and geode ic me ology. Since 2017,
he has been a Resea ch Assis an . His esea ch
in e es s include analysis o mass da a, such
as lase scanning da a, and he de elopmen o
mul i-senso -sys ems.
HARALD STERNBERG ecei ed he deg ee in
su eying om he Uni e si y o he Fede al
A med Fo ces Ge many, Munich, in 1986, and
he D .-Ing. deg ee om he Uni e si y o he
Bundesweh Ge many, in 1999. He wo ked in
he adminis a i e chain as an A ille y O i-
ce and as a Resea ch Assis an wi h he Uni-
e si y o he Bundesweh Ge many. In 2001,
he became a P o esso o enginee ing geodesy
wi h he Hambu g Uni e si y o Applied Sciences,
and om 2009 o 2017, he was a P o esso o enginee ing geodesy and
geode ic me ology wi h Ha enCi y Uni e si y Hambu g, whe e he was also
he Vice-P esiden o s udies and eaching, om 2009 o 2022. In 2017,
he ook o e he p o esso ship o hyd og aphy and geodesy. His esea ch
in e es s include mobile mapping sys ems on di e en ca ie s (ca s, ships,
and indoo ca s), he use o low-cos senso s o posi ioning, indoo posi-
ioning, including wi h 5G, moni o ing o s uc u es, au onomous unde -
wa e ehicles, au oma ic analysis o unde wa e images, in e p e a ion o
backsca e da a, and analysis o mass da a using a i icial in elligence.
VOLUME 11, 2023 3845