Au oma ed segmen a ion and classi ica ion wi h a i icial neu al
ne wo ks o objec s in 3D poin clouds
App o ed DISSERTATION
o ob ain he academic deg ee Dok o -Ingenieu (D .-Ing.)
submi ed o he
Ha enCi y Uni e si ä Hambu g
in he ield o
Geodesy and Geoin o ma ics
by
Eike Ruben Ba ne ske
Hambu g, 2023
This disse a ion is simul aneously published in:
Ausschuss Geodäsie de Baye ischen Akademie de Wissenscha en (DGK), Reihe C,
Disse a ionen, He N . 915, München 2023, ISBN 978-3-7696-5327-4, ISSN 0065-5325,
www.dgk.badw.de.
Submi ed on: May 4, 2023
Dispu a ion day: July 20, 2023
License No ice
This wo k is licensed unde he C ea i e Commons BY 4.0 License. Fo he scien i ic a i-
cles, which a e pa o his hesis (see appendix), licenses a e alid acco ding o he o iginal
publica ion.
Supe iso s
1. Supe iso : P o . D .-Ing. Ha ald S e nbe g Ha enCi y Uni e si ä Hambu g
2. Supe iso : P o . D .-Ing. Alexande Rei e e Albe -Ludwigs-Uni e si ä F eibu g
Addi ional Supe iso : P o . D .-Ing. Jochen Schiewe Ha enCi y Uni e si ä Hambu g
Danksagung
Zu alle e s möch e mich bei meinem Dok o a e Ha ald S e nbe g ü den s e s o enen
und e auens ollen Aus ausch und die Un e s ü zung, insbesonde e, wenn es schwie ig
wu de, bedanken. Danke ü dein Ve auen in mich.
Fü die ielen gu en Anme kungen zu Fo schung, die Feedbacks, die E mu igungen, die
Impulse und die Diskussionen möch e ich mich wei e hin bei Alexande Rei e e bedanken.
Ein g oße Dank geh an Anne e Scheide , Meike Ah end und Clemens Semmel o h, die
mich nich nu bei achlichen F agen, bei den du chge üh en Un e suchungen und den Ko-
ek u en, sonde n auch du ch iele bes ä kende und e mu igende Gesp äche un e s ü z
haben.
Bedanken möch e ich mich bei meinen Kolleginnen und Kollegen de A bei sg uppe und
des geodä ischen Labo s ü das gu e A bei sum eld, die ielen kleinen Hil es ellung, euch
O enhei und das schöne Gemeinscha sge ühl.
Ein besonde e Dank geh an meine El e n Die e und Gab ielle, meine G oßel e n Oska und
Hella, meine Schwes e Lena und meine F eundin Sa ah ü die unzähligen Momen e, in de-
nen ih mich un e s ü z hab , die ielen au bauenden Gesp äche und das nie nachlassende
Ve auen in mich.
Bei Almu , Ka ha ina, La s und Jochen möch e ich mich da ü bedanken, dass ih mich auch
mal om Sch eib isch weglocken konn e . Mi und bei euch konn e ich bei Kuchennachmi a-
gen, au unse en Reisen und bei den ielen ollen E en s die K a anken, die ich b auch e.
Zule z möch e ich mich noch bei den Kame aden de F eiwilligen Feue weh Fuhlsbü -
el da ü bedanken, dass ich on euch den nö igen Rückhal , das Ve s ändnis und die
Bes ä kung ü das Beenden de A bei e ah en du e.
i
Abs ac
The eco ding o objec s su aces wi h Ligh Imaging, De ec ion and Ranging (LIDAR) scan-
ne s is a well-es ablished su eying me hod o he highly accu a e and de ailed geome ic
c ea ion o models. The esul o LIDAR eco dings is a h ee-dimensional (3D) poin cloud
wi h geome ic and spec al (in ensi y and colo alues) ea u es ha ep esen a geome ic
model o eali y. This model is usually au oma ically ex ended by he human imagina ion
wi h seman ic in o ma ion by looking a i , so ha objec classes, indi idual objec s o mea-
su emen e o s in he poin cloud can be eliably iden i ied. The easy in e p e a ion o poin
cloud scenes and i s e ec i e eco ding wi h LIDAR scanne s has led o he ac ha poin
clouds become a quasi- o ma s anda d o 3D models, besides o mesh, oxel and pa ame -
ic models. Seman ic ea u es a e necessa y o au oma ic p ocessing o poin clouds, o
example, in a building in o ma ion model. Cu en ly, seman ic enhancemen o poin cloud
in o ma ion is mos ly done manually, and au oma ion (e.g., ia deep lea ning me hods) is
s ill a subjec o esea ch. In pa icula , A i icial Neu al Ne wo ks (ANN) ha e p o en o be
e ec i e o his ask when he da a and hype pa ame e s (HPs) a e op imized.
In his hesis, he Poin Ne ANN was used as an example o esea ch which a e op imal poin
cloud da a and HPs. The c ea ion o aining da a wi h manual anno a ion ools, he imple-
men a ion and esea ch o p ocesses o au oma ic seman ic segmen a ion, and he de el-
opmen o a heu is ic quali y model o he e alua ion o poin cloud da ase s and o seman ic
segmen a ion p ocesses a e he cen al esea ch issues. The anno a ion ool, Poin Cloud
Classi ica ion Tools (PCCT), was de eloped o in es iga e au oma ion, aining p ocesses
o anno a o s, and ea u es in luence. Fo au oma ic poin cloud p ocessing, in luences a e
poin s om e oneous measu emen s, he class inequali y and he seman ic class de ini-
ions. Di e en class de ini ions and me hods o minimizing he di e ences in class sizes
ha e been de eloped, adap a ions in poin cloud p e-p ocessing ha e been applied and he
weigh ing o in equen classes ha e been op imized.
The esea ch esul s show ha op imal (da a-based) HPs o seman ic segmen a ion o a
building da ase can be de ined. This HP se and he app oach can be used as guidelines
o simila p ojec s. An inc ease in ecall o mo e han 50% o in equen ly occu ing classes
can be achie ed by algo i hm-based class de ini ion and class size conside a ion. Using he
heu is ic quali y model, a ailable aining da a and seman ic segmen a ions can be e alua ed
and compa ed.
iii

Zusammen assung
Die lächenha e E assung on Objek obe lächen mi Ligh imaging, de ec ion and anging
(LIDAR) Scanne n is ein e ablie es Ve messungs e ah en zu hoch-genauen und de ail-
eichen geome ischen E s ellung on Modellen. Das E gebnis de LIDAR E assung is
eine d eidimensionale (3D) Punk wolke mi geome ischen und spek alen Me kmalen, die
ein geome isches Modell de Reali ä da s ellen. Dieses Modell kann du ch Menschen beim
Be ach en meis au oma isch um seman ische In o ma ionen e wei e we den, so dass Ob-
jek klassen, einzelne Objek e ode Mess ehle in de Punk wolke siche e kann we den. Die
ein ache In e p e a ion du ch den Menschen on Punk wolkenszenen und de en e ek i en
E assung mi LIDAR Scanne n ha dazu ge üh , dass Punk wolken neben den Mesh-, den
Voxel- und den pa ame ischen Modellen quasi zu einem Fo ma s anda d gewo den sind.
Seman ische Me kmale sind ü die au oma ische Ve a bei ung de Punk wolken, z. B. in ei-
nem Bauwe ksin o ma ionsmodell, no wendig. Die seman ische E wei e ung de Punk wol-
kenin o ma ionen wi d ak uell meis händisch du chge üh und eine Au oma isie ung (z. B.
mi els Deep Lea ning Ve ah en) is Gegens and de Fo schung. Insbesonde e haben sich
ü diese Au gabe Küns liche Neu onale Ne ze (KNN) als e ek i e wiesen, wenn die Da en
und Hype pa ame e op imie sind.
In diese A bei wu de am Beispiel des KNN Poin Ne e o sch , welche Punk wolkenda-
en und Hype pa am e op imal sind. Die E s ellung on T ainingsda en mi händischen
Anno a ionswe kzeugen, die Implemen ie ung und E o schung on P ozessen zu au o-
ma ischen seman ischen Segmen ie ung, sowie die En wicklung eines heu is ischen Qua-
li ä smodells zu E alua ion on Punk wolkenda ensä zen und on seman ischen Segmen-
ie ungsp ozessen s anden im Fokus. Das Anno a ionswe kzeug Poin Cloud Classi ica ion
Tools (PCCT) wu de en wickel , mi dem die Au oma isie ung, die T ainingsp ozesse on An-
no a o en und die Funk ionen in Anno a ionswe kzeugen un e such we den. Bei de au o-
ma ischen Punk wolken e a bei ung sind die Ein lüsse Punk e aus ehle ha en Messungen,
Klassenungleichhei und die seman ische Klassende ini ion zu be ücksich igen. Ve schiede-
ne Klassende ini ionen und Me hoden ü die Minimie ung de un e schiedlichen Klasseng ö-
ßen wu den en wickel , Adap ionen bei de Punk wolken o e a bei ung wu den angewen-
de und die Gewich ung on sel enen Klassen wu de op imie .
Die Fo schungse gebnisse zeigen, dass op imale (da enbasie e) Hype pa ame e ü die
seman ische Segmen ie ung eines Bauwe ksda ensa zes de inie we den können. Diese
Hype pa am e und das Vo gehen können als Rich linien ü ähnliche P ojek e e wende
we den. Eine S eige ung de seman ischen Genauigkei um bis 50% (Recall) is bei sel-
en o kommenden Klassen kann du ch eine algo i hmusbezogene Klassende ini ion und
die Be ücksich igung de Klasseng ößen e ziel we den. Mi els des heu is ischen Quali-
ä smodells können e ügba e T ainingsda en und seman ische Segmen ie ungen e aluie
we den.
Table o Con en s
Danksagung i
Abs ac iii
Zusammen assung
Lis o Abb e ia ions ix
Lis o Figu es xii
Lis o Tables x i
1 In oduc ion 1
1.1 Mo i a ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Resea ch gaps in seman ic segmen a ion o poin clouds . . . . . . . . . . . 3
1.3 Resea ch objec i es and ques ions . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Ou line o he hesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 S a e o he a 7
2.1 Reco ding sys ems o poin clouds . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Big da a and machine lea ning . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Da a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.2 Da a p e-p ocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.3 Clus e ing and machine lea ning . . . . . . . . . . . . . . . . . . . . . 13
2.2.4 Deep lea ning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.5 E alua ion scheme and me ics . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Manual seman ic segmen a ion o poin clouds . . . . . . . . . . . . . . . . . 20
2.4 T aining da a o poin cloud applica ions . . . . . . . . . . . . . . . . . . . . . 24
2.5 Machine lea ning me hods o poin clouds . . . . . . . . . . . . . . . . . . . 25
2.6 Deep lea ning me hods o poin clouds . . . . . . . . . . . . . . . . . . . . . 28
2.6.1 Seman ic segmen a ion wi h 2D p ojec ion-based deep lea ning me hods 29
2.6.2 Seman ic segmen a ion wi h 3D g id-based deep lea ning me hods . . 32
2.6.3 Seman ic segmen a ion wi h 3D poin -based deep lea ning me hods . 34
3 Connec ions o esea ch publica ions 39
3.1 PAPER 0: PCCT: A poin cloud classi ica ion ool o c ea e 3D aining da a o
adjus and de elop 3D Con Ne . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 PAPER 1: Classi ica ion o e oneously measu ed poin s in 3D poin clouds
wi h Con Ne . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.3 PAPER 2: E alua ing he quali y o seman ic segmen ed 3D poin clouds . . 40
Lis o Figu es
27 O e iew o he connec ions o he ou esea ch publica ions. PAPER 0: De-
elopmen o a b owse -based classi ica ion ool. PAPER 1: De elopmen
o a wo k low o seman ic segmen a ion wi h Poin Ne and in es iga ions on
he in luence o class E oneous poin s. PAPER 2: De elopmen o a quali y
model o he c ea ion and e alua ion o seman ic poin clouds. This quali y
model is used o he e alua ion o he ool in PAPER 0 and he wo k low in
PAPER 3. PAPER 3: Ex ension o he wo k low om PAPER 1 and de elop-
men o me hods o op imize he da ase o DL applica ions. In es iga ions o
he de elopmen on he Poin Ne algo i hm. . . . . . . . . . . . . . . . . . . . 43
28 Cen al issues o imp o emen in a ailable poin cloud anno a ion ools: Da a
secu i y, mul i-use -capabili y, segmen a ion and classi ica ion unc ions, and
au oma ion o sub-ope a ion s eps. . . . . . . . . . . . . . . . . . . . . . . . . 49
29 P ocess o seman ic segmen a ion o poin clouds se ing as an abs ac
model o he eali y. Taken om PAPER 2 and adap ed. . . . . . . . . . . . . 54
30 Quali y model o seman ic enhanced poin clouds. Se en ele an cha ac e -
is ics wi h desc ip i e quali y pa ame e s a e shown. Classi ica ion o neces-
sa y pa ame e s o : Manual segmen a ions ( illed blue ci cles), manual ain-
ing da a gene a ion (un illed blue ci cles) and au oma ic seman ic segmen a-
ion ( illed g een ci cles). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
31 Con e ing he quali y model in o an e alua ion ma ix o use on da ase s, an-
no a ion ools, au oma ic seman ic segmen a ion, and de elopmen moni o ing. 57
32 Concep o a wo k low o apply DL me hods o seman ic poin cloud segmen-
a ion. Th ee modes o he p ocedu e: T aining, e alua ion and applica ion. . 59
33 DHPs o seman ic poin clouds. The DHPs can be dis inguished acco ding
o s uc u al, seman ic, geome ic and spec al cha ac e is ics. A selec ion o
he mos common DHPs o each p ope y is summa ized. . . . . . . . . . . . 62
34 Seman ic segmen a ion accu acy (IoU) o ou common ne wo k a chi ec u es
o he da ase : Seman ic3d.ne [40]. Selec ion o ou om eigh classes o
his da ase . The class Scanning A i ac s, which is equal o he class E o-
neous poin , can be de ec ed poo ly compa ed o he la ge classes. Values
a e aken om he leade boa d o [40]. . . . . . . . . . . . . . . . . . . . . . 63
35 Compa ison o seman ic accu acy ( ecall and p ecision) on he poin cloud o
he Ha enCi y (ou doo ) da ase : a) Wi hou he class E oneous poin s and b)
Wi h he class E oneous poin s. Selec ion o h ee classes ha ha e di e en
equencies in he da ase . Da a om PAPER 1. . . . . . . . . . . . . . . . . 63
36 S ep-wise seman ic segmen a ion o imp o ed di e en ia ion o classes wi h
simila ea u es. Wi h ne wo k A, a segmen a ion is pe o med o gene al
classes, which is e ined in ne wo k B. . . . . . . . . . . . . . . . . . . . . . . 64
37 Da ase op imiza ion me hods o seman ic poin cloud segmen a ion: a)
Da ase expansion by andomly copying poin s, b) weigh ing he loss unc-
ion, and c) da ase expansion by copying inpu s wi h in equen poin s. . . . 65
xi

Lis o Figu es
38 P ocess o c ea ing an adjacency ma ix and applying i as a ne wo k inpu . . 68
39 Eigen alue based ea u es calcula ed om geome ic ea u es (x, y, z): a) GT
seman ic segmen a ion. b) Sum o eigen alues as ea u e. c) Plana i y as a
ea u e. d) Linea i y as a ea u e. A his og am is shown nex o he legend. . 69
40 Seman ic poin cloud o he classes Objec s and E oneous poin s. The se-
man ic segmen a ion is pe o med using he Poin Ne -based wo k low wi h he
ea u es: x-, y-,z-coo dina es, sum o eigen alues,plana i y and linea i y. . . 70
41 Seman ic poin cloud o he classes Building pa s and In e io . The seman ic
segmen a ion is pe o med using he Poin Ne -based wo k low wi h he ea-
u es: x-, y-, z-coo dina es, sum o eigen alues,plana i y and linea i y. . . . . 70
42 G aphical Abs ac : E alua ion o seman ic segmen a ion me hods using he
quali y model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XLI
43 G aphical Abs ac : The poin clouds a e sepa a ed in o di e en seman ic
combina ions o he aining ( i s ow). Di e en me hods a e used o ex-
end he class dis ibu ion (second line). A DL algo i hm is used o ain he
combina ions and ex ensions, which a e hen e alua ed acco ding o ixed
e alua ion c i e ia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LXXXIII
x
Lis o Tables
1 Comme cial ools o seman ic segmen a ion o 3D poin clouds, which a e no
ela ed o a speci ic scien i ic wo k. Abb e ia ions: Bounding box (BB), o line
ool (OT), web se ice (WS). . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2 Open-sou ce so wa e o seman ic segmen a ion o 3D poin clouds, which
a e no ela ed o a speci ic scien i ic wo k. Abb e ia ions: Bounding box (BB),
o line ool (OT), web se ice (WS) and Robo Ope a ing Sys em (ROS). . . . 21
3 Pa ame e s o he ha dwa e and so wa e used o de elopmen and es ing
(single wo ks a ion). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4 Gene al HPs o CNN a chi ec u es. Op imized se o HPs and ypical alues
anges o hese HPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5Poin Ne -speci ic HPs. Op imized se o HPs and ypical alues ange o
hese HPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6 Con ibu ion o Pape No. 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . XXVII
7 Con ibu ion o Pape No. 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . XLI
8 Con ibu ion o Pape No. 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . LXXXIII
9 Con ibu ion o Pape No. 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . CV
x i
1 In oduc ion
The digi iza ion o e e yday li e is a end ha has accele a ed in ecen yea s, pa icula ly
as a esul o he global Co ona pandemic. New ideas on how e e yday li e and he wo k-
ing li e can be digi ally designed ha e been de eloped and b ough o ma ke ma u i y in a
e y sho ime [1]. These applica ions equen ly use da a ha ep esen s he eal wo ld,
as an abs ac and geome ic copy. C ea ing a geome ic model o eal-wo ld objec s (e.g.,
building componen s, s uc u es, coun ies, con inen s) is a co e compe ency o su eyo s,
and has been he basis o maps, h ee-dimensional (3D) isualiza ions (e.g., globes), and
knowledge [2, 3]. Wi h e y as , p ecise and easy o use measu emen sys ems o su ace
eco dings, digi iza ion can be p e o med much as e , bu usually only he geome y and no
he seman ics is eco ded. In his hesis me hods a e in es iga ed, which allow o gene a e
seman ic in o ma ion om geome ic (and some imes om spec al) measu ed alues. The
basis a e poin clouds, which a e eco ded wi h Ligh Imaging, De ec ion and Ranging (LI-
DAR) scanne s and dep h imaging came as. The poin clouds a e seman ically enhanced
by machine lea ning (ML) and deep lea ning (DL) me hods. In pa icula , he cons uc ed
en i onmen , i.e., buildings, ci ies, and long-s e ched in as uc u e s uc u es, a e objec s
o which seman ic segmen a ion is necessa y [4, 5]. The mo i a ion o a eliable seman ic
segmen a ion is explained in sec ion 1.1. The wo ollowing sec ions 1.2 and 1.3 explain he
Resea ch Gaps (RGPs) as well as he esea ch objec i es (ROs) and, he esea ch ques-
ions (RQs). In he las sec ion o he in oduc ion (sec ion 1.4), he s uc u e o he hesis,
he ela ionships be ween he sec ions and he o m o p esen a ion a e desc ibed.
1.1 Mo i a ion
Cadas e, Geog aphic In o ma ion Sys ems (GISs) and Building In o ma ion Models (BIMs)
a e he mos used applica ions o ep esen he eal wo ld in an abs ac (digi al) model and
o use hem o answe ing speci ic issues o m opic such as land use, building condi ion, o
mass de e mina ions. These da a collec ions a e he basis o public ac ion o adminis a ion
and economy, s a egic planning o social de elopmen s and poli ical decisions, so ha hey
a e o impo ance [2, 6].
In he da a collec ions seman ic, opological, hema ic, spec al and objec -inhe en cha ac-
e is ics a e combined wi h geome ic and geog aphic objec cha ac e is ics. T adi ionally,
hese da a collec ions a e o ganized in wo-dimensional (2D) ep esen a ions (e.g., maps o
images) in combina ion wi h egis e s (e.g., p ope y egis e s, land cha ge egis e s o land
egis e s). Wi h he ad an age o GIS and digi al use ools, a pa adigm shi has occu ed
owa ds he di ec s o age o objec - ela ed in o ma ion in he o m o a ibu es o a model
[6]. A BIM is a da a model ha ep esen s, among o he cha ac e is ics, in pa icula he ge-
ome ic cha ac e is ics o objec s as a olume ic 3D model. The g ea ad an age o a BIM
1
1. In oduc ion
is ha he de ails o geome ic cha ac e is ics and objec in o ma ion can be ep esen ed in
a scalable and hie a chical manne . In a BIM, which o iginally comes om planning, he
dimensions o he building objec s become mo e de ailed and seman ic in o ma ion become
mo e accu a e as he planning p oceeds. A he beginning o he planning i is only known
ha a oom needs a doo and app oxima ely on which wall i has o be, he posi ion o he
doo , i s shape and ma e ials become mo e conc e e as he planning ad ances. This is cu -
en ly ep esen ed by he i e Le el o De elopmen (LoDe ), [6, 7]. These LoDe , can be
u he e ined o espec i e cha ac e is ics, such as Le el o accu acy (LoA), in o ma ion
con en o Deg ee o Modeling [8, 9]. BIM p ope ies can also be used a e comple ing he
cons uc ion o a building, such as o compa ing he as-buil planning wi h he as-is execu ion
( inal su ey) [10]. In he ope a ion o a cons uc ion, BIM is a da a o ma ha can be in e-
g a ed in cons uc ion main enance p og ams, con ibu ing o he e ec i e and e icien use
o a building o in as uc u e [11, 12]. Applica ions include indoo na iga ion and imp o ed
space u iliza ion [13, 14], as well as building con ol, epai -planning [15], and eme gency
exi simula ion [16, 17].
I da a o a BIM is no a ailable om he planning o does no ma ch he as-is s a us, he da a
is usually su eyed by o al s a ions, pho og amme ic o LIDAR sys ems [6, 18]. Since mos
me hods scan a su ace, his p ocess is commonly called Scan2BIM.Scan2BIM o mo e
gene ally Scan2Model desc ibes he p ocedu e om he eco ding o a comple e model o
he eal wo ld in an accu acy and le el o de ail a ising om he applica ion [19]. In he
Scan2Model p ocess, he poin clouds a e usually combined wi h o he eco dings and, i
necessa y, he calcula ion o he poin cloud is ca ied ou o pho og amme ic sys ems.
In e es ial lase scanning (TLS), which is cu en ly he s anda d me hod o mos high-
accu acy models, he egis a ion is done wi h common poin s in he o e lapping a eas o
he scans [18]. Mobile Mul i Senso Sys ems (MSSs), such as scanne backpacks [20,
21] o ehicle-based sys ems [22, 23] usually use ajec o y o connec indi idual scans, bu
may also be suppo ed by common poin s. In mos applica ion, il e ing is used o pa ially
emo e he mismeasu emen s. The nex s ep is seman ic segmen a ion. Seman ic segmen-
a ion can be combined wi h modeling, i a di ec au oma ic o manual c ea ion o pa ame ic
geome ies is done [24, 25, 26]. These me hods a e used in applica ions o c ea e loo plans
[27] o su ace models (e.g., meshes o oxels) [28] om he poin clouds. These models a e
usually no ans e ed back in o he da a o ma poin cloud, bu hey o m pa ame e izable
geome ies, such as cylinde s, cubes, planes, lines o ci cles, which a e used o build com-
plex objec -o ien ed models, such as GML [29] Ci yGML [30], Indoo GML [31] o Indus y
Founda ion Classes (IFC) [32]. The me hods a e usually e y speci ic o an applica ion and
equi e de ailed p io knowledge abou he da a and he ask. The mos common applica ions
o hese me hods a e he modeling o building s uc u es.
Howe e , in mos applica ions, seman ic segmen a ion and modeling a e pe o med inde-
penden ly. The seman ic segmen a ion o poin clouds is he mos complica ed s ep o his
p ocess chain o au oma e, as he objec s a y in geome ic size, shape and he eco ded
scenes di e signi ican ly [33]. Pa ame e s and h esholds o sepa a ion by seman ic ob-
2
1. In oduc ion
jec s can be insu icien ly de ined, which s ill makes ML and DL mos sui ed me hods [34].
The pe o mance o ML and DL a ies, depending on he da a and he complexi y o he
class de ini ion acco ding o which he poin cloud should be segmen ed [35, 33]. Howe e ,
s ic ly poin clouds a e impe ec da a o seman ic segmen a ions, since he aining da a
a e usually only a ailable o a small ex en , do no ha e a homogeneous s uc u e, and a e
e oneous [35]. These disad an ages o poin clouds lead o a ying seman ic accu acies o
di e en classes, o sys ema ic con usions be ween classes and o un a o able ounda ions
o modeling [36].
O e coming he impe ec ion and unde s anding i s causes o he case o seman ic seg-
men a ion wi h DL o building econs uc ion is he mo i a ion o his hesis. The aspec o
p ocessing poin clouds wi h DL, he quali y o poin clouds and he gene a ion o aining
da a om poin clouds mus be examined in a s uc u ed manne as i is explained in [37, 38,
39, 40, 41, 42, 43]. Like [36], his wo k ocuses on he poin clouds and i s weaknesses, as
well as me hods o o e come hem.
1.2 Resea ch gaps in seman ic segmen a ion o poin clouds
DL is he mos sui able me hod o he seman ic segmen a ion o poin clouds bu i has se -
e al downsides and aspec s ha a e less esea ched. The main esea ches on DL me hods
deal wi h he ollowing aspec s:
• Op imiza ion o algo i hms and ne wo k a chi ec u es [44, 45].
• Enhancemen o he benchma k da ase s collec ions [40, 46, 47, 48].
• Neighbo hood ep esen a ion o algo i hms inpu [49, 50, 51].
• Au oma ic ans o ma ion o poin cloud in o ma ion in o pa ame e models [35, 52].
• Op imiza ion o manual anno a ions [53, 54, 55].
• In es iga ion o he impac o poin clouds and i s p e-p ocessing o op imal seman ic
segmen a ion [36, 56].
• De elopmen o quali y models and cha ac e is ics o seman ic poin clouds [18].
• Conca ena ion o DL wi h ML [57, 58].
The indings o one esea ch aspec some imes p o ide he ounda ion o he o he s. This
can be seen in he example o he de elopmen o he ne wo k a chi ec u e o Poin Ne [45].
This ne wo k a chi ec u e enables an e icien and di ec p ocessing o la ge poin cloud
scenes (>1 million poin s). Now, he da a p e-p ocessing o he poin cloud o ma is no
longe a p ima y issue, bu he da a con en is a new issue. The main esea ch in he ield
o DL me hods on poin clouds does no add ess eal and p ac ical applica ions, he e o e
many ele an and in luencing pa ame e s a e neglec ed. This leads o he RGPs add essed
in his hesis:
3

1. In oduc ion
RGP 1: In luence o da ase cha ac e is ics, poin s and poin clouds o seman ic seg-
men a ion.
RGP 2: De elopmen o a heu is ic desc ip ion and e alua ion o seman ically seg-
men ed poin clouds.
RGP 3: The de elopmen o a wo k low o seman ic segmen a ion o poin clouds in
building modeling p ocesses.
The esea ch objec i es (ROs) a e de i ed om he RGPs, bu do no necessa ily add ess he
en i e esea ch gap. How he RGPs can be closed is explained in mo e de ail in sec ion 1.3
based on he ROs and he RQs.
1.3 Resea ch objec i es and ques ions
The iden i ied RGPs a e in he o e lapping ield o he disciplines o compu e science, ma h-
ema ics, da a science, compu e e sion, ci il enginee ing, acili y managemen , as well as
geodesy and geoin o ma ics. In o de o close hese gaps, inno a i e da a models and p o-
cessing algo i hms mus be implemen ed by means o mode n high-pe o mance compu e
sys ems o add ess opics a ising in he digi iza ion o buildings. Reco ded digi al da ase s o
buildings ha e measu emen e o s, a y in e ms o seman ic class sizes, include ine and
coa se objec s in uns uc u ed and he e ogeneous poin clouds (Figu e 1).
Figu e 1: Measu ed TLS poin cloud wi h segmen a ion and anno a ion e o s. Class E o-
neous poin s in ed and class Objec in blue.
These da ase s a e no op imal o p ocessing wi h ML o DL me hods, due o he da a con en
and da a o ma . Ne e heless, ML and DL me hods a e he mos e icien and accu a e
me hods o seman ic segmen a ion i he da a is homogeneous, s uc u ed, and a anged
in a as e . In o de o ha monize cha ac e is ics o poin cloud da ase s and DL algo i hms,
he ollowing h ee ROs a e ackled:
4
1. In oduc ion
RO 1: E alua ion o me hods o he manual anno a ion o poin clouds ega ding e i-
ciency, usabili y, accu acy, and he de elopmen o an expe imen al anno a ion ool.
RO 2: De elopmen o a quali y model ha heu is ically desc ibes seman ic poin
clouds.
RO 3: De elopmen o a wo k low o seman ic segmen a ion in o de o in es iga e
he in luence o poin clouds con en and o ma in DL me hods.
RO 1 can be achie ed by explaining he de eloping s eps o he Poin Cloud Classi ica ion
Tool (PCCT) (PAPER 0) and he in es iga ions o i s usabili y. Fo manually anno a ed poin
clouds wi h a compu e , he use s mus ha e segmen a ion ools, a isualiza ion o he poin
cloud (on a sc een), guidelines o he classi ica ion p ocess, and ools o he classi ica ion.
Based on hese s a emen s, he ollowing RQs should be answe ed:
RQ 1.1 Which anno a ion ools (manual segmen a ion) o poin clouds exis ? Wha
unc ions can be ound in hese ools? How e icien , eliable and e ec i e a e hese
ools and how can hese cha ac e is ics be de e mined?
RQ 1.2 Which anno a ion ools can be used o he seman ic segmen a ion o challeng-
ing eal-wo ld indoo TLS poin clouds?
RQ 1.3 How can seman ic segmen a ion ools o poin clouds be enhanced and im-
p o ed?
RQ 1.4 How o become a good anno a o o seman ic poin clouds? How can he
pe o mance o anno a o s be measu ed? Wha do anno a o s need and how can he
ool suppo hem?
In o de o answe he ques ions o RO 1, a heu is ic quali y model mus be used. The
de elopmen o a quali y model is he RO 2. The quali y model e alua es he seman ic
segmen a ion p ocess and he seman ic poin cloud. The de elopmen o he model is guided
by ollowing RQs:
RQ 2.1 Wha a e sui able seman ic poin clouds? Wha a e he cha ac e is ics o poin
clouds? How can he cha ac e is ics o he poin cloud be de e mined, measu ed and
compa ed?
RQ 2.2 How is a quali y model o seman ic poin clouds designed? Which pa ame e s
a e necessa y o he desc ip ion o he cha ac e is ics? Does he quali y pa ame e s
di e o anno a ion and au oma ic seman ic segmen a ion?
RQ 2.3 How can he quali y model be applied o he seman ic segmen a ions o build-
ing poin clouds
RO 3 is based on RO 1 and RO 2 and is he ealiza ion wi h he wo k low o seman ic
segmen a ions. The aining da a c ea ed by he PCCT o o he ools and he pe o mance
e alua ion o he wo k low by he quali y model a e necessa y o answe he RQ 3.1 o RQ
3.3. The wo k low is de eloped o he poin clouds c ea ed by TLS wi h impe ec ions as
5
1. In oduc ion
shown in Figu e 1. In he wo k low, es ablished DL me hods a e in eg a ed. The o mal and
con en in luencing pa ame e s o poin clouds a e e alua ed in expe imen s. The RQs which
guide he de elopmen a e:
RQ 3.1 How can DL me hods be in eg a ed in a wo k low o seman ic segmen a ion
o poin clouds? Which DL me hods a e sui able?
RQ 3.2 Which hype pa am e s need o be de ined o applying Poin Ne in a seman ic
segmen a ion wo k low? How a e he alues o hese hype pa am e s de e mined?
RQ 3.3 How can he in luence o he da ase be con olled by da a-based hype pa am-
e e s in he seman ic segmen a ion o poin clouds? Wha a e he main in luences?
The h ee cen al ROs a e co e ed by h ee pee - e iewed and one ex end-abs ac -pee -
e iewed publica ions. The e is no one- o-one assignmen o one RO o one publica ion.
Ins ead, single o mul iple RQs a e co e ed in each publica ion. The connec ions be ween
he publica ions and he ROs a e explained in sec ion 3.5 and Figu e 27.
1.4 Ou line o he hesis
This cumula i e disse a ion consis s o a amewo k hesis (sec ions 1 o 5) and he ou
publica ions in he appendices A (pee - e iewed publica ions) and B (non-pee - e iewed
publica ion). The amewo k hesis p esen s he s a e o he a , he e minologies (sec ion 2),
he connec ions be ween he indi idual publica ions (Sec ion 3.5), and he ROs, RQs, and
esul s (sec ion 4). Sec ion 5 summa izes he key conclusions and ou lines u he esea ch
app oaches.
PAPER 0 is in sec ion B.1 and PAPER 1 o PAPER 3 a e in sec ions A.1 o A.3. A e e -
ence o he publica ions is made by he indica ion PAPER #. Re e ences a e used o a oid
epe i ions o esul s, p oo s, and de ailed desc ip ions ha ha e al eady been discussed
in he publica ions. Fo be e comp ehension, conclusions and gene al obse a ions a e
discussed in sec ion 3 and in he indi idual RQs (sec ion 4).
6
2 S a e o he a
Seman ic poin clouds a e he ounda ion o modeling complex en i onmen s. Thei deploy-
men co e s he en i e p ocess wi h he acquisi ion, he pa ame e -based il e ing and he
seman ic segmen a ion o he poin cloud (s eps 1 o 3 in Figu e 2). Based on he seman ic
poin clouds, pa ame ic, solid models, Compu e Aided Design (CAD) and BIM models a e
c ea ed. These models a e used in GIS [59], cons uc ion managemen applica ions [12, 60],
and in p i a e and public egis e s [61] (s eps 4 and 5 in Figu e 2).
Figu e 2: P ocess o c ea ing seman ic poin clouds. Reco ding o poin clouds wi h an op ical
eco ding sys em. Regis a ion o he indi idual eco dings, esul ing in a comple e
poin cloud. Seman ic segmen a ion acco ding o gi en classes se . Modeling o
objec s in he seman ic poin cloud. Implemen a ion o he models in o an applica-
ion. Taken om [62] and adap ed.
In con ex o buildings, LIDAR scanne s and measu emen came as a e used o eco d en i e
su aces in a as way. The wo king p inciple, he di e ences o he measu emen sys em as
well as i s in luence on he poin clouds a e explained in sec ion 2.1. The eco ding and he
seman ic segmen a ion a e signi ican o he quali y o he seman ic poin cloud. Seman ic
segmen a ions a e pe o med manually as well as au oma ically and due o he size and
complexi y o he da a his opic comes unde big da a.big da a applica ions equi e special
da a handling, which is ca ied ou wi h ML me hods. The basics o big da a and ML a e
in oduced in sec ion 2.2. The p ocess o manual seman ic segmen a ion is in sec ion 2.3.
ML models lea n he ela ionship be ween inpu and a ge da a om he da a i sel . The
cha ac e is ics o aining da a a e desc ibed in sec ion 2.4. The s a e o he a o au oma ic
seman ic segmen a ion o poin clouds is desc ibed by ML and DL me hods in sec ions 2.5
and 2.6.
2.1 Reco ding sys ems o poin clouds
Poin clouds ha e become a quasi-s anda d o s o ing eco dings and isualizing he su -
aces o eal objec s in he digi al domain. This quasi-s anda d can be explained by he
ac ha many eco ding sys ems ha e o s o e he measu ed alues e y as (da a-s eam)
and which does no allow an o de (so ing) acco ding o he con ained objec classes [63].
7
2. S a e o he a
based me hods. In addi ion, a dis inc ion is made be ween knowledge-based (black) supe -
ised (o ange) and unsupe ised (g een) me hods.
Figu e 8: Summa y clus e ing and seman ic simila i y segmen a ion me hods. Knowledge-
based clus e ing me hods (black), da a-based clus e ing me hods (g een), and
da a-based seman ic segmen a ion me hods (o ange).
In p og amming ypically, knowledge is used o p ocess and analyze da a (knowledge-
based). This equi es ha he da a con en is known and can be selec ed ia pa ame e s
such as h esholds o numbe o objec s sea ched. I condi ions a e me , clus e s can be
o med using h eshold selec ion [95] (e.g. h eshold > a spec al alue) o i ing a geom-
e y in he poin cloud. Me hods such as Random Sample Consensus (RANSAC) [96, 97],
equi e some pa ame e s, such as numbe o objec s and i s shape, and andomly sea ch
he da a o hese pa e ns o o m clus e s.
In addi ion o pa ame e -based me hods, which a e highly dependen on a-p io i knowledge,
da a-based me hods a e an al e na i e ha lea n he ela ionship be ween da a and a ge
class om he da a i sel . Th ee ca ego ies o da a-d i en app oaches a e desc ibed in he
li e a u e. These a e supe ised lea ning,unsupe ised lea ning, and ein o cemen lea n-
ing. In supe ised lea ning, he a ge a iables a e known and he algo i hm lea ns o de e -
mines he ela ionship be ween ea u es and he a ge a iables. In unsupe ised lea ning,
no a ge a iables a e gi en and a ix o an unspeci ied numbe o clus e s wi h high simila -
i y in he ea u es is o med. Rein o cemen lea ning is based on he idea o ial-and-e o .
The algo i hm pe o ms he classi ica ion ask many imes and ge s eedback a he end o
each pass indica ing whe he he classi ica ion is co ec o inco ec [98].The las me hod is
usually no used o seman ic segmen a ion.
14

2. S a e o he a
Unsupe ised lea ning is used p ima ily o clus e ing da a objec s. The eby, di e ences
and simila i ies in he ea u es a e de e mined e.g. ia s a ic me hod, ea u e o de s o ans-
o ma ion in ano he ea u e space [99]. Disc imina ing me hods, such as edge de ec ion
[100] o P incipal Componen Analysis (PCA) [101, 102], de ine di e ences by bounda ies
in he ea u e space. Based on hese bounda ies (o h esholds), he unlabeled clus e s a e
o med. Gene a i e unsupe ised me hods, such as g aph-based me hods [103, 104], Re-
gional G owing (RG) [105], o k-Means [106], s a a one o mo e s a ing poin s and g ow
a ound he objec poin s ha ha e he g ea es simila i y. The esul ing a eas and s uc u es
a e he unnamed clus e s. Using use knowledge o da a-based me hods, he unnamed
clus e s become classes.
Supe ised lea ning me hods, such as Nai e Bayes [89, 107, 108], Logis ic Reg ession [89,
109], k-Nea es -Neighbo s (kNN) [86, 110, 111], Suppo -Vec o -Machines (SVM) [86, 112,
113], and Decision T ees (DTs) [108, 114] use a ge a iables o op imize he lea nable model
pa ame e s. In addi ion o he lea nable pa ame e s, each ML me hod has addi ional pa am-
e e s ha mus be speci ied p io o aining. These a e called hype pa ame e s (HPs) and
include he algo i hm i sel , he p opo ion o aining and es da a, lea ning a es (LRs), and
s opping c i e ia. The HPs a e discussed in PAPER 3. The p e iously men ioned me hods
a e mos ly classi ied as weak ML. They allow di ec seman ic segmen a ion o a p ede ined
de ined se o classes. The labeled da a is needed o his pu pose (sec ion 2.3). The adjec-
i e weak e e o he ac ha he ea u es a e used di ec ly o seman ic segmen a ion and
no dep h ea u es a e o med om he aw ea u es. This equi es ha he necessa y inde-
penden a iables ha e been op imally chosen and ha he ea u es a e ee o g oss e o s.
Da a p e-p ocessing has an e en la ge impac han in DL [86]. To make he me hods mo e
obus o a ying da a, Ensemble Lea ning (EL) me hods such as Random Fo es (RF) [115]
we e de eloped. RF use mul iple DTs and all a e ained unde di e en condi ions. The e-
sul s a e combined using me hods such as o ing, bagging, s acking o boos ing. EL also
uses di e en combina ions o independen a iables o minimize he in luence o co ela ed
o i ele an independen a iables. The EL leads o a measu able inc ease accu acy o
mos applica ions [86, 116].
2.2.4 Deep lea ning
DL me hods a e usually mo e obus o e o s and majo changes in ea u es han ML me h-
ods, such as SVM o RF. They ha e become e y impo an wi h he ise o big da a, as
hey ind hidden pa e ns in la ge and complex da ase s [117]. Commonly, DL is used as a
synonym o A i icial Neu al Ne wo ks (ANNs). The unc ionali ies, he di e en ypes, as
well as he ad an ages and disad an ages o ANN a e b ie ly explained in his sec ion. The
use o ANN o seman ic segmen a ion o poin clouds will be discussed in mo e de ail in
sec ion 2.6.
An ANN is a ma hema ical- echnical model o a na u al neu al ne wo k such as hose ound
in b ains [118]. The e a e s a ic and dynamic componen s in he ANN, which a e con olled
15
2. S a e o he a
by he ini ial HPs. The s a ic componen s a e he p ocessing uni (neu on), he connec ions
(weigh s) and he ne wo k opology (ne wo k a chi ec u e). The lea ning phase and he p o-
cessing phase a e he dynamic componen s [117].
Figu e 9: Neu on and ne wo k a chi ec u e: a) Neu on a chi ec u e and unc ion. a) Simple
ANN wi h inpu and ou pu laye s. b) ANN wi h a hidden laye . Inspi ed by [119,
120]
The neu on is an independen uni ha pe o ms a pa ial ope a ion o he ne wo k. The
neu on p ocesses he nume ical in o ma ion by agg ega ing i s inpu and calcula ing a new
ac ua ion alue wi h a ( ypically nonlinea ) unc ion (Figu e 9a). S ep-,Sigmoid- o ReLu-
unc ions a e used. Neu ons a e o ganized in laye s and o wa d pa s o he in o ma ion o
o he neu ons. The simples ANN consis s o only wo laye s and can only be used o linea
p oblems (Figu e 9b) [120, 121]. The inpu laye has as many neu ons as he e a e inde-
penden a iables in he da ase and o wa ds hem o he ou pu laye o , in mo e complex
ne wo ks (as in Figu e 9c), o he hidden laye . In he ou pu laye , he e is one neu on o
each a ge a iable. The laye s a e connec ed by weigh ed and di ec ed g aphs. I all neu-
ons o one laye a e connec ed o all neu ons o he nex laye , his is called ully connec ed
(FC) laye . Also, spa sely connec ed (SC) laye s whe e some neu ons ha e connec ions
a e equen ly used. In o ma ion can low in all di ec ions. In p ac ice, o s a ic classi ica ion
asks, he eed- o wa d (FF) a chi ec u es ha e become mos popula . The FF a chi ec u es
eed in o ma ion om he inpu laye h ough all hidden laye s o he ou pu laye . The ou pu
laye p o ides a quasi-p obabili y o each class. A classi ica ion unc ion (e.g., So max) is
used o pe o m he in e p e a ion o he ou pu laye esul s. Du ing he lea ning phase, a
la ge numbe o ea u es along wi h he labeled class ( aining da a) a e ed in o he ANN and
a e each pass, he loss is de e mined ac oss all lea ning samples. By compa ing ne wo k
p edic ions and a ge da a, he ne wo k loss is de e mined. This loss needs o be minimized
by op imizing he weigh s on he g aphs using back-p opaga ion [119, 120]. A e he ne wo k
has been ained se e al imes and he loss alue is minimized, he ANN can be es ed wi h
independen da a (sec ion 2.2.5). Once he es pa ame e s a e inalized, he ANN can be
used in he p ocessing o in e ence phase [98, 117, 122, 123].
A special ype o ANN uses equal weigh s o all inpu s (sha e weigh s). These inpu s a e
images o 3D da a and ca y in o ma ion in he a angemen o ea u es (neighbo hood de-
penden da a). Commonly, ANNs o his kind o da a a e called Con olu ional Neu al Ne -
16
2. S a e o he a
wo ks (CNNs) [124]. Using CNN, each ea u e o each a iable is loaded as a 1D, 2D o
3D enso . Fo each ne wo k eed, he e a e as many enso s as he e a e a iables in he
i s laye . To ex ac dep h ea u es, each enso is mul iplied by weigh s o a ea u e map
(F-map) and hese p oduc s a e summed up, so ha a new dep h ea u e is c ea ed om all
inpu a iables. The F-map is a enso , wi h a ixed wid h and leng h (usually a ew en ies
la ge), ha is shi ed o e he inpu enso such ha he new ea u es emain local. The e
a e mul iple F-maps o each con olu ional laye (Con laye ), so se e al ea u e a iables
a e gi en o he nex laye . The F-maps co espond o he weigh s in he ANN. When he
F-map is longe and wide han one en y, he wid h and leng h o he ne ea u e enso will
be educed (Figu e 10). S onge ea u es a e o med by con olu ion and pooling laye s.
Commonly, he classi ica ion s ep is done wi h a FC laye [71, 118, 120, 125].
Figu e 10: Func ion o he one Con laye a a CNN. Inspi ed by [120].
Figu e 11: Common CNN-A chi ec u es o seman ic segmen a ion. a) Encode -Ne wo k
and b) Encode -Decode -Ne wo k
Special CNN a chi ec u es a e he sha ed Mul i Laye Pe cep on (MLP), Encode -Ne wo ks
(EN), and Encode -Decode -Ne wo k (EDN). These a e o en used in seman ic segmen-
a ions. The MLP is s ic ly an ANN, such as in Figu e 9c, and uni o ex ac ea u es om
da a inpu s. By implemen ing his uni wi h a 1D CNN, mo e ope a ions wi h iden ical weigh s
can be pe o med in pa allel [126]. EN encode he inpu da a o dep h ea u es by chained
17
2. S a e o he a
Con laye and used a he end a FC laye o classi y each poin (Figu e 11a). This me hod
is used o spa se poin clouds o classi ica ion ques ions [127]. In he EDN, he ea u es
a e encoded and summa ized in he encode phase. In he decode phase, he ea u es a e
expanded o he numbe o inpu poin s and decoded (hie a chical app oach) [128]. Fea u es
can be sha ed be ween encode and decode laye s o he same size h ough connec ions
(Figu e 11b).
An al e na i e way o dis ibu e in o ma ion be ween inpu s is o use Recu en Neu al Ne -
wo ks (RNNs). RNNs inhe i in o ma ion om p e ious inpu s o he cu en inpu and subse-
quen inpu s. The alue o he p e ious in o ma ion become lowe o e he ime (Figu e 12).
RNNs a e mos ly implemen ed in he o m o Long Te m Sho Memo y (LTSM) ne wo ks,
which a e explained in [129].
Figu e 12: ANN wi h a ecu en laye . The ou pu s o he ecu en laye is used as addi ional
inpu in he nex pass. Wi h ime he inpu s become less meaning ul, so ha i s
in luence is lowe ed ia weigh s.
Compa ed o mos o he ML me hods, ANN and CNN ha e a high lea ning capaci y, when
la ge aining da ase s a e a ailable. They can be e icien ly adap ed o new asks, once
he in as uc u e o aining is se . They usually gene alize be e han ML me hods and
a e mo e obus o e o s. Disad an ages o DL a e long aining imes, lack o o small
aceabili y o he lea nable pa ame e s and he e is a need o la ge amoun s o aining
da a [117, 120].
2.2.5 E alua ion scheme and me ics
Mos algo i hms use in e media e classi ica ion esul s o op imize he lea nable pa ame e s,
he eby alida ion is al eady pa o he lea ning. This alida ion is done using only e y
ew me ics, which mos ly desc ibe he seman ic accu acy. Typically, in supe ised lea ning,
O e all Accu acy (OA) and loss a e used. In ein o cemen lea ning, bina y answe s ( alse
o ue) a e gi en. In non-supe ised lea ning, no alida ion occu s du ing lea ning in his
18
2. S a e o he a
sense [89]. The alida ion du ing lea ning gi es insu icien in o ma ion o e alua e he pe -
o mance o he ained model on new simila da a and o each indi idual seman ic class.
Be o e a model can be p oduc i ely applied, a ull e alua ion o he model wi h unknown da a
mus be pe o med. This mus p o ide in o ma ion on seman ic sensi i i y ( ecall) and speci-
ici y (p ecision), e alua e he choice o HPs, and p o ide o he me ics such as geome ic
accu acy [108].
The basis o he e alua ion is a g ound u h (GT) da ase ha is used o alida e whe he he
classi ica ion o each da a poin is co ec . I his is he case, he poin is conside ed o be ue
posi i e (TP), i no , he poin is conside ed as alse posi i e (FP) in he p edic ed class and
as alse nega i e (FN) in he ue class. This classi ica ion o poin s is usually p esen ed in a
con usion ma ix [90] (Figu e 13) and is he basis o compu ing o he seman ic me ics, which
[108] desc ibes in gene al e ms. A e iew o me ics in poin clouds is done in PAPER 2.
Figu e 13: Con usion ma ix o he example o h ee classes. TP = ue posi i e, FP = alse
posi i e and FN = alse nega i e. TP o he one classes is equal o ue nega i e
(TN) o all o he classes.
The au oma ic classi ica ion me hods ha e a la ge numbe o HP ha ha e o be cus omized.
The co ec choice o HP is he p e equisi e o op imizing he lea nable pa ame e s and suc-
ceeding in classi ica ion. In a b oade sense, he aining o he model is no comple e a e
aining o he lea nable pa ame e s. Ra he , his is only one pass o he in eg a i e op imiza-
ion o he HPs. This op imiza ion wi h a ious manual and au oma ic me hods is p esen ed
o gene al models in [90], o ML me hods in [86, 108], and o DL me hods in [130]. In
PAPER 3, DL me hods a e e iewed in de ail. E alua ion using non-seman ic me ics is
necessa y o special (e.g., geode ic) issues, bu is a ely p esen ed in he li e a u e.
19

2. S a e o he a
2.3 Manual seman ic segmen a ion o poin clouds
Seman ic poin clouds a e he basis o c ea ing su ace models [4], de eloping BIM appli-
ca ions [131], building he na iga ion basis o au onomous ehicles [47], and de eloping
algo i hms o au oma ic seman ic analysis o 3D poin clouds [39, 132]. Manually enhanc-
ing poin clouds by segmen ing he indi idual objec s in he poin cloud and assigning a label
is named as poin cloud anno a ion. Poin cloud anno a ion is a e y complex ask ha is ime
consuming and mos o en pe o med by expe s [133]. To speed up his ask and allow less
expe ienced anno a o s (e.g., c owd wo ke s) o do his, a ious so wa e ools ha e been
de eloped o make anno a ions mo e eliable and simple. A b ie summe y o hese ools
and p o ide s o hese se ices (Da a as Se ice) is gi en in he ollowing. These ools and
hei desc ibed unc ionali ies o m he basis o he PCCT. The mo i a ion o he PCCT is o
p oduce independen , eliable, as and wi hou addi ional cos s es da a o examina ions,
as he e we e only ew simila ools a ailable a he beginning o his hesis (sec ion 3.1).
The li e a u e e iew on a ious manual (open-sou ce and comme cial) ools o seman ic
segmen a ion o 3D poin clouds shows ha eigh p ope ies o he ools a e ele an . These
p ope ies a e isualiza ion o he poin cloud, big-da a-capabili y, ools o segmen a ion,
mul i-use capabili y, adap abili y o new ci cums ances, eedback capabili y o anno a o s,
semi-au oma ion, and anno a ion e alua ion. An o e iew is gi en in Figu e 14. A selec ion
o he e iewed ools o seman ic segmen a ion o poin clouds, showing me hods di e si y,
is p esen ed in Tables 1 and 2.
Figu e 14: Requi emen s o a poin classi ica ion ool.
20
2. S a e o he a
Table 1: Comme cial ools o seman ic segmen a ion o 3D poin clouds, which a e no e-
la ed o a speci ic scien i ic wo k. Abb e ia ions: Bounding box (BB), o line ool
(OT), web se ice (WS).
Tool name Selec ion Applica ion OT / WS
(Au oCAD) Recap [134] F eehand, il e , i , polygon TLS OT
Poin Cab [135] F eehand, il e TLS OT
AWS SageMake [136] By own design All WS
basic.ia [137] BB, semi-au oma ic Au on. d i ing WS,
scale [138] BB, semi-au oma ic Au on. d i ing WS
Poin Cloud Technology [139] Da a as Se ice All WS
Table 2: Open-sou ce so wa e o seman ic segmen a ion o 3D poin clouds, which a e no
ela ed o a speci ic scien i ic wo k. Abb e ia ions: Bounding box (BB), o line ool
(OT), web se ice (WS) and Robo Ope a ing Sys em (ROS).
Tool name Selec ion Applica ion OT / WS
Cloud Compa e [140] F eehand, il e , i , polygon All OT
MeshLab [141] F eehand, polygon All appl. OT
Mul i-Label PC [140] RG All appl. OT / ROS
Go Then Tag [142] Solid i , pencil All appl. OT
PC Anno a e [55] Solid i TLS, Au on. d i ing OT
Seman icKITTI [53] F eehand, b ush Au on. d i ing OT
3D Anno a ion [143] BB, semi-au oma ic Au on. d i ing OT
LATTE [144] BB, semi-au oma ic Au on. d i ing OT
3D BAT [133] BB, semi-au oma ic Au on. d i ing WS
SAnE[145] BB, semi-au oma ic Au on. d i ing OT
Visualizing 3D da a and na iga ing h ough i on a wo-dimensional sc een is desc ibed by
[146] as a cen al p oblem, because he da a can only be seen om one pe spec i e, which
leads o mis akes in in e p e a ion [147]. [146, 147] add ess his p oblem by isualizing he
da a on a 3D display wall and use a ouch sc een able o na iga ion. The idea o p ocessing
3D da a in a 3D space is also add essed by he Poin A Me applica ion [148], which uses
i ual eali y (VR) glasses o isualiza ion. The anno a o s wea VR-glasses and can mo e
eely h ough he poin cloud. The anno a o s segmen and classi y indi idual objec s ia
he con olle s by placing a bounding box (BB) a ound he poin s belonging o an objec . All
o he ools om Tables 1 and 2 use a s anda d 2D sc eens o he seman ic segmen a ion
on which he poin cloud is displayed in a p ede ined pe spec i e [143] o as a ee na igable
model. The ee-pe spec i e choice is de aul .
The ee pe spec i e choice is ad an ageous o manual segmen a ion o objec s o di e en
sizes. This op ion allows o look om any angle and a any zoom le el a he a eas o be p o-
cessed. Howe e , using his op ion equi es ha he poin cloud can be loaded in a e y high
esolu ion, ideally wi hou delays. Fas loading is an aspec ha conce ns big da a capabili y
and is usually implemen ed by spli ing he poin cloud in o 3D iles. The 3D iles a e usually
ealized by kd- ee o oc ee me hods. These me hods o ganize he poin cloud hie a chi-
cally, so ha only he necessa y da a sec ion is comple ely loaded a any gi en ime. The
21
2. S a e o he a
me hods kd- ee [149] and oc ee [150] a e s a e o he a in mass da a p ocessing [142].
The choice o a hie a chical s uc u e has a g ea ad an age o isualiza ion, because he
poin cloud can be used in ull de ail. Howe e , o segmen a ion, his pa i ioning can be dis-
ad an ageous, because du ing segmen a ion he s o age s uc u e is changed and has o be
ecalcula ed again. In p ac ical applica ions (e.g., Recap [134]), i is obse ed ha hese cal-
cula ions can be educed i only all poin s o o he classes a e dele ed om an exis ing da a
s uc u e. By dele ing he poin s, he exis ing da a s uc u e emains unchanged and does
no need o be ecalcula ed du ing segmen a ion. Loading he poin clouds wi h all ea u es
in o he wo king memo y (di ec use access) is e y ime-consuming, so in many applica-
ions only pa s o he da ase can be loaded and p ocessed a any gi en ime. The coa se
subdi ision is usually done acco ding o seman ic aspec s, such as oads [53], measu emen
d i es [47], eco ding s a ions o ooms [27]. Seldom, pe manen da abase sys ems (e.g.,
Ma iaDB and Pos g eSQL) a e used o benchma ks, because he da a is mean o be ex-
changed. In addi ion, olde -based da a s o age, po able da abases such as 5h o SQLi e
a e some imes used. These o ma s ha e he ad an age ha he da a can be loaded ia
S uc u ed Que y Language (SQL) commands e icien ly by se e al use s a he same ime.
Besides solu ions o empo a y and pe manen s o age o poin cloud da a, he il e ing o
he poin cloud acco ding o poin cloud densi y o geome ical aspec s is an impo an as-
pec . Many manu ac u e s o eco ding sys ems o e op imized pa ame e -based il e s in
hei own so wa e o poin cloud p e-p ocessing o gene al s a ic il e s, such as S a is i-
cal Ou lie Remo al (SOR) [151] o Voxel-Subsampling and Fas Clus e S a is ical Ou lie
Remo al (FCSOR) [152]. I is impo an ha he geome y o he objec is no changed be-
yond wha has been done by he eco ding sys em and ha , known measu emen e o s a e
minimized.
The anno a ion o he poin cloud consis s o segmen a ion and classi ica ion. T adi ionally,
o segmen a ion, a pe spec i e is selec ed in which he objec o be classi ied can be ec-
ognized well. The objec is sepa a ed om he en i onmen wi h a polygon o lasso and as-
signed o a seman ic class [147]. Besides he ee- o m polygons o lasso using he mouse,
he selec ion o poin s is o en done by b ush echnique (sweeping o e an a ea wi h he
mouse) [53], placing BBs o e he objec [145] o selec ing by pa ame ic 3D solids [55]. The
selec ion o poin s by pa ame ic 3D scenes is done pu ely by humans, who in e p e he 3D
scenes di e en ly, a ia e in he deg ee o ca e ul wo k, and a e di e en good ained o he
ask. This is concluded by [143] unde he ac o o human e o . To minimize his occu ing
ac o , segmen a ion is o en conside ed as a con ol sc ew o au oma ion. The app oaches
o he au oma ic geome ic segmen a ion can be summa ized in i e main me hods and
one mixed me hod (hyb id me hods). These basic segmen a ion me hods a e e isi ed in
sec ion 2.5 and used in a modi ica ion o he da ase poin cloud. The basic segmen a ion
me hods a e acco ding o [147]:
• Edge-based segmen a ion.
• Reginal g ow.
• Model i ing.
• T adi ional Machine Lea ning.
• Deep Lea ning. 22
2. S a e o he a
• Hyb id me hods.
Fo a de ailed desc ip ion o he main me hods and examples, e e ences a e made in Table 1
by [147]. The lis can be ex ended by he objec acking espec i ely ins ance da ase s, such
as applied in au onomous d i ing [53, 55, 145].
Mul i-use capabili y plays a mino ole in many scien i ic manual seman ic segmen a ion
ools as he da ase s a e mos ly sha ed by he esea che s as in [41, 53, 55]. The anno-
a o s mos ly p ocess one assigned sub-da ase locally wi h a speci ically de eloped ool o
acco ding o a p ocess desc ip ion [41, 153] o a gene al poin cloud p ocessing p og am,
such as Cloud Compa e [154]. Especially when special ha dwa e, as in [148], o ex a pow-
e ul ha dwa e [134] is used, he scalabili y by he numbe o wo ks a ions is usually no longe
e icien and economical. Mul i-use capabili y is mos ly implemen ed in he scien i ic con ex
by c owd-wo king-se ices (CWS), such as Amazon-Web-Se ices, also known as Amazon
Mechanical Tu ks [136]. Fo example, his is p opaga ed in [133] and conside ed du ing
so wa e de elopmen . Few applica ions [155] a e iden i ied ha use he AWS o simila
se ices. Comme cial se ice p o ide s, such as basic.ia o scale, p o ide mul i-use web
applica ions o a ious da a classi ica ion asks o deli e eady-labeled da a. [55] explain
in hei discussion o he ca ego y o anno a ion ools, ha ew in o ma ion is known abou
he p ocess, he da a accu acy and he da a p i acy.
The comme cial anno a ion se ices o poin clouds ocus on he ma ke o au onomous d i -
ing. This is done by he BB selec ion o he da a, he ini ial class se s ha p ima ily include
a ic pa icipan s, su ace ypes, and s ee u ni u e, and he ajec o y-op imized isualiza-
ion and p ocessing. Also, many scien i ic wo ks, such as [53, 55, 143, 144, 145], a e op i-
mized o au onomous d i ing. Howe e , mos o hese anno a ion ools a e ans e able o
mobile mapping applica ions, because he class selec ion in hese is adap able o al eady in-
cludes mos classes o ou doo applica ions. T adi ionally, indoo poin clouds a e cap u ed
wi h RGB-D came as, so seman ic segmen a ion is done wi h 2D anno a ion ools, such as
LabelMe [156] o he ools desc ibed in he e iew by [157]. Poin clouds ha a e sou ced by
TLS a e p edominan ly anno a ed using Cloud Compa e o comme cial applica ions, such
as Recap [134] and Poin Cap [135]. These ools a e op imized o iewpoin -based eco d-
ing. Each anno a ion me hod is usually de eloped o a speci ic da ase (da a o ma ) and a
speci ic ask, and usually equi es majo e o o adap o a sligh ly di e en applica ion.
The e alua ion o manual seman ic segmen a ions and he ela ed eedback and aining
o anno a o s a e e iewed in de ail in PAPER 2. In addi ion o he s a emen s he e, he
expe iences o [54] can be ollowed o he aining o he anno a o s. They emphasize he
selec ion o he anno a o s, he p e ious expe ience, an in ensi e aining phase be o e he
p ope ask and an anno a o -bias (indi idual e o s).
23
2. S a e o he a
Figu e 20: BEV p ojec ion. The poin cloud is o ien ed along he z-axis and ans o med in o
a as e plan wi h ix a ix as e s uc u e.
wi h a FC ne wo k. In he in e e ence phase, he passable a ea o each scan is seman i-
cally segmen ed in a ew mic oseconds wi h app oxima ely 90% ecall and 90% p ecision5
[202]. Following his ask and app oach, [203] ha e de eloped a simila me hod using he
LoDNN Ne wo k [202] o seman ic segmen a ion. The e, in addi ion o he geome ic ea-
u es, ea u es such as pixel densi ies a e used in o de o ake gene aliza ion in o accoun .
Fu he mo e, WV poin clouds a e seman ically segmen ed wi h he U-Ne [128], a e hey
ha e been ans o med in o a pano amic image. Di e en esolu ions o he pano amic im-
ages a e e alua ed wi h he U-Ne and hen he sub seman ic segmen a ions a e combined
o a join one using h esholds. I s pe o mance is alida ed on he Seman icKITTI bench-
ma ks [53, 204] and is close o 90% o ecall and p ecision.Su Con [205] and PIXOR
[206] aim o de ec indi idual objec s, such as o he ca s, pedes ians o cyclis s, which a e
o a pa icula in e es in he poin cloud. This is done in he i s s eps as desc ibed abo e.
The seman ic segmen a ion is pe o med using a CNN chained by FC laye , whe e he FC
laye is used o exp ess he loca ion, o ien a ion, and eliabili y o a BB ha en elops he ob-
jec . The accu acy o seman ic de ec ion a ies be ween an a e age p ecision o 55% and
75% [206]. A comple e seman ic segmen a ion o WV scenes a e in ended wi h he me hods
SqueezeSeg [58], RangeNe ++ [207], LU-Ne [208] and SalsaNex [209]. These me hods do
no di e undamen ally in he scheme o da a p ocessing. In all wo ks, he poin clouds a e
p ojec ed on o a sphe e, which is hen un olled as a pano ama (Figu e 21a). Fu he mo e,
di e en imp o emen s o he geome ic esolu ion a e de eloped and applied. The back
p ojec ion om he image o he poin cloud is add essed and op imized by a kNN s ep [207]
and Condi ional Random Fields (CRF) [58]. Besides U-Ne ,Da kne 53 [199], SquezzeNe
[210], and ResNe -18 [211] a e used and adop ed. The pe o mance o hese me hods a ies
om 52% o 60% 6In e sec ion o e Union (IoU).
TLS poin clouds and poin clouds gene a ed wi h mobile MSS ha e a much highe densi y
and canno be mapped om a single pe spec i e. Poin s would be missed by occlusions
5Published by he de elope s.
6Valida ed on he Seman icKITTI da ase by [46].
30

2. S a e o he a
Figu e 21: P ojec ion o a 3D poin cloud in o (2D) image. a) Sphe ical o cylind ical p ojec-
ion. b) Mul i- iew-image p ojec ion and ans o ma ion.
du ing seman ic segmen a ion o he geome ic con ex would no be iden i ied. Mul i- iew-
image app oaches (Figu e 21b) a e de eloped by [212, 213]. The poin cloud is conside ed
as a su ace and a mesh is compu ed om i . [212] use andomly gene a ed images ha
ep esen comple ely he mesh a di e en dis ances and o a ions. These images a e se-
man ically segmen ed wi h EDNs, such as U-Ne and SegeezeNe , and he seman ics a e
p ojec ed back on o he mesh. F om his, he seman ic in o ma ion is ans e ed o he poin
cloud. The app oach o [213] use planes ha angen ially in e sec he poin cloud in one
poin . S a ing om his poin , all neighbo ing poin s in small a ea ound ha he angen ial
poin a e p ojec ed in o he plane, and he a eas wi hou in o ma ion a e comple ed by in e -
pola ion be ween poin s (Figu e 22). The plane is o e laid wi h a pixel g id and all images
a e seman ically segmen ed by U-Ne . The IoU o hese me hods a ies be ween 51% and
67%7.
Figu e 22: P ojec ion o he poin s on o a angen plane. C ea ion o a mul i- iew image
(Simpli ied 2D illus a ion).
A ecen wo k use 2D CNN o seman ic segmen a ion o 3D poin clouds by using he eco d-
ing p o iles [214], uses he poin clouds wi h o he da a such as images [215], o uses 3D
CNN in ai -bo n lase scanning (ALS) analysis [216]. [214] use a p o ile lase scanne ha
7Valida ed by de elope s on he Seman ic3D.ne da ase [40].
31
2. S a e o he a
gene a es a 2D poin cloud. This poin cloud is ans o med in o a as e image and e alua ed
wi h a 2D CNN. In he da a usion me hod, [215] use images o he seman ic segmen a ion. A
mobile mapping sys em cap u es he images synch onously o he lase scans. The seman-
ic in o ma ion is gene a ed in he images and ans o med o he poin cloud. This me hod
equi es a e y accu a e synch oniza ion and calib a ion o he scanne and he came as.
3D CNN ha e a wide geome ic dimension and con olu e he da ase in h ee di ec ions.
This is compu a ionally in ensi e, so [216] addi ionally ans o m he ALS poin cloud in o
o hopho os.
In addi ion o as e -based 2D CNN, la ice ep esen a ions o he poin clouds a e used.
The la ice app oaches use a bila e al Con laye [217] o ans o m he poin clouds in o he
2D s uc u e. They could be p ocessed wi h 2D CNN. Ad an age o hese me hods is ha
3D poin s and geo e e enced images can be used, as in SPATNe [218], and be used o
seman ic segmen a ion.
2.6.2 Seman ic segmen a ion wi h 3D g id-based deep lea ning me hods
In he con ex o 3D g id-based DL me hods, poin clouds a e con e ed in o 3D as e s uc-
u es o e alua ion wi h CNNs. These can be oxel-based, ee-based o la ice-based. This
in e media e o ma allows o apply Con laye s con olu in h ee dimensions (3D CNN). Mos
o he 3D CNN a chi ec u es a e based on he 2D CNN a chi ec u es [219] applied o as e
images. Fo seman ic segmen a ion, EDN and EN wi h a FC laye a e used, which assign a
class o each oxel. By in e pola ion in he 3D space, he labels a e ans e ed o he poin s
o u he e inemen s o he seman ic segmen a ion a e pe o med. A gene al pipeline o
hese me hods is shown in Figu e 23. The ad an age o using 3D CNN is ha he in o ma ion
is p ese ed in all dimensions and he segmen s a e no mixed in he educed dimension.
A majo i y o 3D CNN ne wo ks commonly me ge mul iple poin s in o one oxel, so ha he
da a is gene alized as well (Figu e 24a). The main disad an age o 3D CNN compa ed o 2D
CNN is ha he applica ion and aining imes a e signi ican ly inc eased, since he numbe
o ope a ions is po en ia ed. This has led o many ea ly a chi ec u es consis ing o ew laye s
[124] and he oxel s uc u e being ans o med in o an occupancy g id (Figu e 24b) [220].
The EN wi h FC laye pe o ms a classi ica ion o each ne wo k inpu . In VoxNe , he de ec-
ion o objec s in he poin cloud is pe o med wi h such an a chi ec u e [124]. [219] p oceed
iden ically, bu only label he oxel ep esen a ion. Fo he ans e om oxel o poin , an
in e media e s ep is in oduced ha akes in o accoun he dis ance o he oxel cen e poin .
[40, 221] use a sub- oxel g id compu ed o each poin as ne wo k inpu . Small-dimensional
Con laye s wi h 16 x 16 x 16 oxels a e used as a basis. In he a chi ec u e o [40], he local
neighbo hood is addi ionally conside ed by using oxel g ids wi h i e di e en oxel edge
leng hs (2.5 cm o 40.0cm). Fo each o hese i e oxel g ids, a VGG16-like [222] ne wo k
a chi ec u e ha is ex ended as 3D CNN is used. All he sub-CNN esul s a e combined and
he classi ica ion is pe o med in he FC laye .
32
2. S a e o he a
Figu e 23: Wo k low seman ic segmen a ion u ilizing 3D g id s uc u es. Top: The en i e
poin cloud is ans e ed o one g id. I e a i ely, se e al oxels a e ed in o a 3D
CNN. The oxel g id is seman ically segmen ed. The in o ma ion is passed by
in e pola ions o he poin cloud. Bo om: A sub- oxel g id is c ea ed o each
poin . Each sub- oxel g id is classi ied by he 3D CNN. Each poin is di ec ly
assigned o one class.
Figu e 24: Voxel s uc u es (in 2D pe spec i e). a) Regula oxel g id. b) Occupancy g id.
c) Oc ee wi h e inemen due o poin cloud densi y.
O he 3D CNN a e based on a EDN, whe e a pa o a poin cloud is ans o med in o a
oxel s uc u e. The FC ne wo k can ac as an encode o de ec a speci ic objec in he
poin cloud. Vo e3D [223] and Vo e3Deep [224] use such a CNN a chi ec u e, embedded
wi h a o ing algo i hm, o de ec he BB o objec s ele an o au onomous d i ing. EDN o
spa ial classi ica ion o RGB-D images ind applica ions in he a chi ec u es: SSCNe [225],
ScanNe [226], and ScanComple e [227]. The oxel- esol ed RGB-D image is gi en as an
occupancy g id wi h a esolu ion o se e al cen ime e s as inpu o he CNN. In his case, he
key ea u e is occupancy o non-occupancy s a e o he oxels (Figu e 24b). Th ough he
EDN, he occupancy g id is di ec ly classi ied and a ansmission o he seman ic labels is
made o a poin cloud o a mesh. [227] use a hie a chical-chained a chi ec u e o e icien ly
inc ease segmen a ion esolu ions. Addi ionally, his ype o a chi ec u e can ill gaps c ea ed
by eco ding pe spec i es [225, 227]. The SEGCloud a chi ec u e has a CRF laye a e he
3D CNN laye , which is used o ine (sub- oxel) segmen a ion [228]. Thus, combining
adi ional ML and DL me hods con ibu es o an e icien and easily inc ease in seman ic
segmen a ion accu acy.
33
2. S a e o he a
Oc Ne [229] uses an oc ee as oxel s uc u e (Figu e 24c). Wi h he oc ee, he sizes o he
oxel cells a e adjus ed based on he occupancy o he oxels [230]. Oc Ne is used o se-
man ic segmen a ion o poin clouds ep esen ing la ge acades. In his con ex , he heigh
o he acade speci ies he maximum size o he oc ee. The ea u es o he poin s alling in
a oxel a e combined by compu ing an a e age and calling his alue oxel ea u e. The GT
label is de e mined using he dominan class o he poin s in his oxel. Seman ic segmen a-
ion is pe o med using a 3D EDN, so ha di ec ly he selec ed po ion can be seman ically
segmen ed. Besides oc ees, kd- ees [231] a e also used o small poin clouds applying
FC laye o 1D-con olu ion o ea u e ex ac ion and seman ic segmen a ion [232].
2.6.3 Seman ic segmen a ion wi h 3D poin -based deep lea ning me hods
The DL me hods om sec ions 2.6.1 and 2.6.2 ha e he disad an age ha poin s always
ha e o be con e ed in o a as e geome y and in o ma ion is gene alized. Di ec poin -
based seman ic segmen a ion became popula wi h he de elopmen o Poin Ne [45]. Poin -
Ne uses MLPs o ex ac dep h ea u es, which ex ac ed indi idual o each poin . The
indi idual poin ea u es a e combined ia a max-pooling unc ion, esul ing in global ea-
u es desc ibing he dominan ea u es o all poin s cu en ly ed in o he ne wo k. A de ailed
desc ip ion o o Poin Ne can be ound in PAPER 1 and PAPER 3. Due o he poin -wise ex-
ac ion, he o de o he poin s is no impo an . Howe e , his also has he disad an age ha
neighbo ly ela ions, which a e desc ibed by se e al poin s, a e no conside ed in seman ic
segmen a ion. Mo eo e , he global ea u es e e only o he cu en inpu , consequen ly
in mos cases he ea u es desc ibe only a e y small pa o he poin clouds. In p inciple,
seman ic segmen a ion can be pe o med using poin -based ea u es, hus esea che s use
Poin Ne o pa s o Poin Ne equen ly. Besides Poin Ne -based enhancemen s, which will
be discussed in mo e de ail below, RandLANe [44] is one mo e ecen de eloped ne wo k
o seman ic segmen a ion o la ge poin clouds. In RandLANe , he poin clouds a e seman-
ically segmen ed in one s ep by andomly educing hem.
The key weaknesses o Poin Ne conce ning neighbo hoods a e di ec ly add essed in [233,
234, 235, 236, 237, 238] bu non o hese wo ks o e come all he weaknesses. In Poin Ne ++
[233], Poin Ne laye s a e in eg a ed in an EDN. This ne wo k conside s he hie a chical local
neighbo hoods o he poin s. The cen al modules o Poin Ne ++ a e sampling, g ouping
and ea u e ex ac ion wi h Poin Ne . The Fa hes Poin Sampling (FPS) algo i hm is used
o de ec p incipal poin s. The ea u es o he nea su ounding poin s a e g ouped in e e y
p incipal poin and ed as one uni o he Poin Ne laye (Figu e 25). Wi h hese ex ensions,
la ge poin clouds can be segmen ed seman ically. The same objec i e bu wi h wo di e en
app oaches a e pu sued in [234]. Thei i s app oach uses di e en sized inpu le els (a ea
sizes as shown in Figu e 26b) and combines ea u es o hem. The ea u es gene a ed
om he inpu le els a e gi en in a consolida ion uni . The second app oach uses a ixed
inpu block size (Figu e 26a). The ea u es om di e en inpu blocks a e ed in o a RNN
consolida ion uni . The in o ma ion om ou inpu blocks a e conside ed as in o ma ion
34
2. S a e o he a
sequence and h ough he RNN, sha ed ea u es a e c ea ed ha a e used o he seman ic
segmen a ion [234].
Figu e 25: S uc u e and unc ions Poin Ne ++. Encoding o poin ea u es in an i e a i e p o-
cess conside ing he local neighbo hood: (1) Selec ing npoin s ha a e maximal
wide away om each o he . (2) G ouping o he ea u es in he neighbo hoods.
(3) Applying a Poin Ne laye o ea u e ex ac ion. (4) Repea ing his p ocess.
Decoding by join and s ep-wise in e pola ion o he ea u es. (5) Classi ica ion
laye a he end. Figu e om [233].
Figu e 26: Two me hods o c ea ing neighbo hood inpu blocks. a) Fixed block size wi h
he blocks sha ing ea u es ia RNN. b) Va iable block size wi h di e en ixed o
dynamic adii. Used in CNN o MLP a chi ec u es. Inspi ed by [233, 234].
Simila enhancemen is desc ibed in [50]. Local mul i-scale neighbo hoods a e implemen ed
by a py amid pooling unc ion and in o ma ion is dis ibu ed ac oss he ne wo k ia a RNN
laye . This enables lea ning om he cohe ence o objec s. The enhancemen s o inpu
ea u es and in oducing Con laye ins ead o MLP a e desc ibed in [236]. Sel -o ganizing
35

2. S a e o he a
(SO) ne wo ks ha use SO maps as inpu s and Poin Ne as encode s a e de eloped in [235].
A ea u e ne wo k consis ing o conca ena ed ea u e modules and a Poin Ne ne wo k was
de eloped by [237]. In he ea u e module, local ea u es a e o med by he kNN and global
ea u e by he K-means. Fu he mo e, a cen oied loss ea u e is in oduced. [238] use he
Poin Ne ne wo k a chi ec u e bu sub-sample he poin cloud by g id in ad ance and pos -
connec a CRF ope a o o ine segmen a ion.
The Poin Ne ++ is also he basis o many enhancemen s. Poin Ne ++ is based on a s an-
da d EDN, which addi ionally p ocesses ea u es o di e en abs ac ion le els (hie a chi-
cally). A no able cha ac e is ic o Poin Ne ++ is he use o he MLP as a cen al module.
The enhancemen s o Poin Ne ++ aim on using he local neighbo hood o he poin s as a
ea u e. In o de o desc ibe he local o ien a ion o he poin s, [239] add a scale-in a ian -
ea u e- ans o ma ion (SIFT) in o he Poin Ne ++ as an in e media e laye . They e e his
laye as Poin SIFT, which desc ibes he o ien a ion o geome ic ea u es on di e en le els
o abs ac ion. A Local Spa ial Awa e (LSA) laye has been de eloped by [240]. This can
be used o model he ea u e dis ibu ion wi hin an inpu se as a unc ion. They cons uc
he LSANe o his laye , implemen ing Poin Ne ++ o local geome ic ea u e ex ac ion.
The Shu lePoin Ne is de eloped by [241]. Once again, a new laye is implemen ed in o an
exis ing Poin Ne ++. This laye consis s o a kNN based g ouping o he inpu poin se . Fo
each subg oup, independen ea u es a e compu ed using an MLP. The new ea u es a e
shu led, conca ena ed, and ed in o he nex laye . Some o he MLP laye s a e eplaced by
his new laye in Poin Ne ++.
In addi ion o de elopmen s di ec ly ela ed o Poin Ne , o he poin -based me hods ha e
been de eloped and a selec ion o some ne wo ks is b ie ly desc ibed below. The di ision
o Figu e 19, which subdi ides RNN-based, poin -based CNN and g aph-based me hods, is
ollowed.
RNNs a e o en used in conjunc ion wi h he MLP ne wo k o Poin Ne , as shown p e iously.
Al e na i ely, RSNe is a ne wo k whe e he en i e poin cloud is spli in o mul iple iews in
x, y, and zdi ec ions. Each slice di ec ion is p ocessed independen ly. Slices a e used o
c ea e an o de in he poin cloud. The ea u es o he di e en slices a e gi en in o a RNN
laye , sha ing some o he in o ma ion. The ex ac ed ea u es o he slices a e agg ega ed
and hen he ea u es o all slice di ec ions a e used o he poin -wise seman ic segmen a ion
[242].
Con laye s con ol e an o de ed geome ic neighbo hood so ha less da a objec s, such
as pixels, ca y mo e and mo e deep ea u es. A ans e o his app oach o indi idual
independen poin s is a ge ed o poin -based CNNs. Unlike he MLP, a eal con olu ion
is pe o med o e a egional ea u e dis ibu ion. In Poin CNN [243] his idea is desc ibed
in de ail and a χ-Con laye is in oduced. Poin CNN uses an EDN. The coo dina es a e
ans o med in o he ea u e space wi hin a small egion. This can be done in simple e ms
by compu ing he de ia ions o each poin o a p incipal poin . The p incipal poin ca ies he
a ea ea u es and is combined wi h o he p incipal poin s in he nex s age.
A simila app oach is applied o he KPCon laye by [49]. As an ini ial s ep, he poin cloud
is homogenized using a g id-sampling il e . The poin spacing is equal and he objec s
36
2. S a e o he a
become dis inc by he ea u es o he poin s. Fo example, poin s ha a e no occupied
a e ma ked wi h ee o 0and poin s ha a e pa o an objec a e ma ked wi h he ea u es
o he measu emen and occupied o 1. The KPCon laye in he basic e sion is buil on
sphe ical neighbo hoods in which he co ela ion coe icien s a e calcula ed o all ea u es
and all poin s o he cen e o he sphe e. The co ela ion coe icien s can be mul iplied by
any weigh ma ix o he Con laye such ha hey can be chained as EDN. ShellNe [244]
also uses sphe ical inpu egions in which he ea u es a e compu ed. Ci cles wi h di e en
adii a e used and he ea u es a e p ocessed oge he . Poin Con [245] con e s he local
neighbo hood o poin s in o a con inuous densi y unc ion and a weigh unc ion ha can
be p ocessed wi h Con laye . The ne wo ks A-CNN [246] and Dila ed Poin Con olu ions
[247] a e de eloped on poin -based CNN in which he selec ion o poin s a e op imized. A-
CNN a anges he poin cloud by a local p ojec ion o he sub-poin cloud on o a disk and
p ocesses he ea u es o he poin s wi h an encoding MLP ne wo k [246]. [247] pe o m
poin -wise classi ica ion wi h a CNN added by one FC laye and conside di e en ecep i e
ields in he enc yp ion phase. As an al e na i e o ea u e di e ences, which ep esen he
ela ionship be ween poin s in a poin cloud, g aphs a e widely used.
G aphs ep esen he ela ionship be ween da a objec s (e.g., poin s) h ough edges. Edges
desc ibe on one hand which da a is in a ela ionship and on he o he hand by he edge
weigh s how his ela ionship looks. Thus edge weigh s a e usually mul idimensional ec o s.
The e o e, g aphs ake on an o de ing ole o a wide a ie y o da a (e.g., social ne wo ks
and poin clouds) ha a e p ocessed and analyzed wi h ANNs. A gene al o e iew o G aph
Neu al Ne wo ks (GNNs) is gi en in [248], which p o ides a clus e ing o he di e en ypes o
GNNs. Simple g aphs wi h ew nodes a e used o he join p ocessing o RGB- and D-images
in o de o use he local dep h in o ma ion, such as in he case o adjacen objec s wi h simila
RGB alues [233]. GNNs in combina ion wi h MLPs ha e he pu pose ha poin ea u es and
local neighbo hood ea u es a e used oge he o dep h ea u e ex ac ion. In he Fea u e-
based G aph Con olu ional Ne wo k, ini ial poin ea u es a e ex ac ed and combined by
he g aph ep esen a ion. Subsequen ly, a sub-g aph con olu ional ne wo k is build, which
ex ac s neighbo hood-based ea u es. These dep h- ea u es a e used o a poin -wise clas-
si ica ion by a FC laye [249]. A pa allel ea u e decoding wi h g aphs and indi idual poin s
a e desc ibed in [250]. Fo his, he indi idual ea u es be ween he b anches a e sha ed
a di e en hie a chical le els. The Dynamic Capsule G aph ne wo k a chi ec u e is based
on an EN ha p ocesses a he inpu laye independen ly di e en ea u e ypes, such as
eigen alues, spec al alues and coo dina es. The ea u es a e combined and sha ed dep h
ea u es a e c ea ed by a chain o encapsula ed g aph Con laye s, which a e summa ized
by pooling laye s. The e mina ion laye s a e MLPs, whe e a se o desc ibed ea u es is
gene a ed o each poin [51]. O he GNNs in oduce an ini ial weigh ing o edge weigh s o
ea u es, du ing he encoding phase, o ein o ce o he di e en ia ion powe o he ele an
ea u es [155].
The EdgeCon laye , ep esen ing he local neighbo hood, is o en implemen ed in MLP ne -
wo ks o minimize he disad an ages o Poin Ne . This EdgeCon laye is used o building
37
2. S a e o he a
modeling [251] and in he analysis o ALS poin clouds [252]. Howe e , g aphs can also
be used o p e-segmen a ion o he ull poin cloud. In his case, he g aphs a e used o
build sub-segmen s o hose poin s ha ha e a la ge ea u e simila i y. These sub-segmen s
can be p ocessed sepa a ely in sub-ne wo ks [253]. Addi ionally, he esul s o hese sub-
ne wo ks can be used o con ex -based segmen a ion [254].
38
3 Connec ions o esea ch publica ions
This sec ion explains he connec ions be ween he ou key publica ions. The connec ion wi h
he ROs, he applied app oaches and he gene al esul s a e summa ized in sec ions 3.1 o
3.4. The connec ions o he ROs and he publica ions a e ou lined in sec ion 3.5.
3.1 PAPER 0: PCCT: A poin cloud classi ica ion ool o c ea e 3D
aining da a o adjus and de elop 3D Con Ne
The key opic o PAPER 0 is he de elopmen o a mul i-use , b owse -based ool o seman ic
segmen a ions o 3D poin clouds. This ool is named PCCT1and consis s o h ee indepen-
den modules. The da a is exchanged ia a Ma iaDB da abase hos ed on a web-se e . All
h ee modules can be accessed om any compu e wi hin he Ha enCi y Uni e si y (HCU)
campus ne wo k wi hou any local ins alla ions. The i s module uploads new poin clouds
in o he da abase. Du ing he upload, he poin clouds a e con e ed in o a 2D image ep-
esen a ion. This 2D image ep esen a ion is used o an au oma ic seman ic segmen a ion
and isualiza ion wi hin he b owse -based ool. A e he con e sion, indi idual noise pix-
els a e elimina ed by il e algo i hms and an edge op imiza ion is pe o med. RG me hods
a e used o c ea e segmen s based on ea u es, such as RGB and in ensi y alues. Each
image shows only one segmen . The image is linked o he 3D ca esian coo dina es o he
co esponding poin s ia a connec ion in he da abase. In he second module, hese seg-
men images a e andomly loaded by he b owse ool and he anno a o s use a d op-down
lis o selec he app op ia e class o he displayed segmen . Thus, a class is assigned o
each image espec i ely segmen . The hi d module es ablishes he ela ionship be ween
he ca esian coo dina es and he classi ied images. Since each image is classi ied mul iple
imes and by di e en anno a o s, a o ing p ocedu e is in oduced o assign he mos likely
classes o he poin s. The poin ea u es a e enhanced by he seman ic classes. The en-
hanced poin clouds a e expo ed in di e en ASCII o ma s. Di e en p ojec ion me hods
and se s o ea u es o segmen a ion a e es ed and applied in s udies o PAPER 1 o PA-
PER 3. In addi ion, indi idual p ocessing s a egies a e used o indoo and ou doo da ase s,
as hey ha e di e en cha ac e is ics. A b ie s udy is ca ied ou o in es iga e he seman ic
accu acy o he PCCT. I is shown ha he de eloped ool is sui able o he ask, bu an
op imiza ion o he ool pa ame e s is necessa y in o de o achie e highe seman ic accu-
acies. The op imiza ion has o be done on he basis o cha ac e is ics which a e de ined in
PAPER 2. The use ulness o manual seman ically segmen ed poin clouds is demons a ed
on he example o an applica ion wi h Poin Ne .
1Gi hub eposi o y: h ps://gi hub.com/eb17/PCCT
39
4. E alua ion o he esea ch esul s
Howe e , he applica ion addi ionally speci ies which seman ic objec s o he poin cloud a e
de e mined and how hey a e geome ically selec ed. Typically, BB o i egula 3D solids
a e used in mos da ase s and ools. A summa y o he anno a ion ools mos ly applied
o BIM and DL applica ions a e p esen ed in Tables 1 and 2 (sec ion 2.3), as well as in
Table 5 o PAPER 2. These ools a e applica ion d i en wi h excep ion o Cloud Compa e and
Recap. Tools ha e been de eloped o one speci ic p oblem in connec ion wi h a seman ic
segmen a ion. Fu he mo e, hese ools a e mos ly linked o one speci ic da ase . Da ase s
wi hou a ela ion o a speci ic ool a e a e. Some o hese da ase s a e Seman ic3d.ne [40]
and TUM-MLS-2016 [153], which we e c ea ed by Cloud Compa e.
All ools di ide he anno a ion in o wo s eps. These s eps a e segmen a ion and labeling.
Fo he mos ools, labeling is a simple assignmen o a class o an encoded class alue o a
p e iously c ea ed segmen . This s ep is usually no au oma ed. The mo e complica ed s ep
is segmen a ion. In he manual segmen a ion acco ding o seman ic aspec s, he exac a ea
mus be shown in which an objec can be sepa a ed unambiguously om he en i onmen .
This is pa icula ly challenging i e y la ge and e y small objec s occu . In addi ion, he
objec space mus no be oo la ge, so ha a luen na iga ion and isualiza ion h ough a
de ailed poin cloud is possible. In o de o make his possible, he poin cloud is usually
di ided in o smalle sec ions in ad anced, based on eco ding loca ions, ooms o dis ance
in e als. Due o he complexi y o he segmen a ion acco ding o seman ic equi emen s,
e o s o en occu , so his s ep is opic o au oma ion. The di e en me hods o au oma ic
segmen a ion and isualiza ion a e explained in sec ion 2.5, as well as in PAPER 0 and
PAPER 2.
The au oma ion o he segmen a ion does no necessa ily lead o mo e accu a e seman ic
segmen s, bu only o he ac ha hese a e always de e mined he same. The de e mina ion
and selec ion o he HPs o hese segmen a ion me hods is he cen al adjus men sc ew
in he me hod de elopmen . The selec ion o he HPs is a e y ime-consuming ask and
mus be ca ied ou and checked o each indi idual da ase . DL app oaches ha only use
geome ic ea u es a e no published, e en i e.g. eigen alues and geome ic pa ame e s a e
sui able o his pu pose, as shown by [184] o ML applica ions. Mo e common seman ic
segmen a ion app oaches a e based on ea u es, such as colo and in ensi y alues, since
mos da ase s a e c ea ed by RGB-D came as and simple LIDAR scanne s (Tables 2 and 3 o
PAPER 2). These me hods a e also desc ibed in mo e de ail in sec ion 2.5 and PAPER 0.
The accu acy, eliabili y and e iciency a e cha ac e is ics ha can be used o compa e di e -
en me hods ega ding a ce ain applica ion. These cha ac e is ics can only be de e mined
wi h e o and always o a speci ic da ase . Usually, hi d-pa y ools do no epo hese
cha ac e is ics. In o de o de e mine he accu acy, a e e ence poin cloud showing he
same scene and wi h a highe accu acy mus be a ailable. The simples way o c ea e such
a da ase is o use syn he ic poin clouds de i ed om models [35, 166, 260]. Al e na i ely, a
da ase can be c ea ed wi h a highe accu a e and handheld measu ing sys em as desc ibed
in PAPER 2. In he conside ed case, a handheld scanne is used o c ea e objec -by-objec
segmen s. The combina ion o a e y accu a e eco ding sys em and seman ic segmen-
46

4. E alua ion o he esea ch esul s
a ion in he ield esul s in a eliable and accu a e seman ic poin cloud. The c ea ion is
labo -in ensi e in he ield and can usually only be applied o small poin clouds, due o he
usage o special measu ing sys ems. I such a poin cloud is a ailable, a geome y compa -
ison can be done o de e mine inco ec ly segmen ed poin s.
In o de o de e mine he e iciency o anno a ion ools, he equi ed ime mus be pu in
ela ion o he achie able accu acy o he seman ic segmen a ion. The cos s o ha dwa e,
so wa e and ene gy a e mos ly negligible, since he wo k o humans labo ime causes
he highes cos s. Ve y ew da ase s o ools [41, 53, 54] indica e how long he seman ic
segmen a ion akes. Un o una ely, his in o ma ion is gi en usually only o one anno a o o
a small g oup o anno a o s as discussed in RQ 1.4 (sec ion 4.1.4). Caused by his missing
in o ma ion he e iciency can no be de e mined o mos da ase s.
Reliabili y is de e mined by mul iple independen anno a ions and compassion wi h e e ence
da a. This does no equi e a GT da ase , bu o compa abili y o ools, he same da ase
should always be used. In addi ion, eliabili y in his de ini ion also includes usabili y and
desc ibes how a ce ain g oup o humans sol es he seman ic segmen a ion ask. Thus,
aspec s such as ask comp ehension and mo i a ion can be included in his cha ac e is ic.
Fo mo e de ails o his aspec see RQ 1.4 in sec ion 4.1.4.
Conclusion and ou look: The a ailable ools a e highly specialized and no sui able o
mul i-disciplina y applica ions. Many inno a i e echnical solu ions a e p esen ed in he li e -
a u e, bu hese equi e di e en inpu o ma s and lead o di e en seman ic ep esen a ions.
Two basic unc ions, segmen a ion and classi ica ion, a e a ailable in mos ools. These unc-
ions a ec he quali y o he seman ic poin cloud, hese a e discussed in RQs 1.2, 1.3 and
1.4. Me a da a is incomple ely ob ainable o many da ase s and anno a ion ools, and a
compa ison o hem is o en no possible. The e alua ion o poin cloud da ase s and anno-
a ion ools is add essed in RO 2. This analysis is e y ime-consuming, bu necessa y o a
be e unde s anding o poin cloud da a and algo i hms, as seen om he i s in es iga ions
o de e mine he h ee mos discussed cha ac e is ics.
4.1.2 Anno a ion ools o indoo e es ial lase scanning poin clouds
RQ 1.2: Which anno a ion ools can be used o he seman ic segmen a ion o challenging
eal-wo ld indoo TLS poin clouds?
Me hodology: Anno a ion ools and p ocesses we e selec ed based on he li e a u e e-
iew explained in he answe o RQ 1.1 in sec ion 4.1.1. The ools Seman icKITTI [53],
PC-Anno a e [55], Recap [134], and Cloud Compa e p ocesses om TUM-MLS-2016 [153]
and Seman ic3d.ne [40] a e es ed wi h he e e ence da ase published in PAPER 2. In
addi ion, he PCCT de eloped in PAPER 0 is e alua ed o in es iga e i s quali y. In a pilo
s udy, all ools a e e alua ed by wo olun ee s in e ms o usabili y (da a o ma , a ailabili y
o he so wa e, echnical equi emen s, app oxima e p ocessing ime). In he main s udy,
en olun ee s a e asked o pe o m seman ic segmen a ions wi h Recap and PCCT (pub-
47
4. E alua ion o he esea ch esul s
lished in PAPER 2). In a su ey, p io knowledge, me ics (e.g., p ocessing ime), imp ession
o usabili y, expec ed accu acy, and desi ed changes a e asked. The su ey is e alua ed
oge he wi h he esul s o he seman ic segmen a ion.
Findings: The esul s o he pilo s udy shows ha Seman icKITTI and PC-Anno a e a e no
sui able o seman ic segmen a ion o challenging eal-wo ld indoo TLS poin clouds. The
anno a ion ool o Seman icKITTI is op imized o mobile eco ded inpu da a and o dynamic-
changing en i onmen s. Fo a seman ic segmen a ion o TLS poin clouds, a pseudo na -
iga ion ile would be needed in o de o use his ool. The PC-Anno a e ool has a limi ed
selec ion o classes and a seman ic segmen a ion is only e icien ly possible ia he i o eg-
ula geome ies, which leads o inaccu a e segmen a ions o de ailed indoo scenes. The
p ocesses o [40, 153] a e complex and equi e a solid knowledge o Cloud Compa e, which
canno be assumed o all po en ial anno a o s. The e o e, hese ools we e excluded om
he main s udy.
The main s udy is p esen ed in sec ion 4 o PAPER 2. All quali y pa ame e s o he model om
sec ion 4.2 and in sec ion 3 o PAPER 2 a e de e mined, so ha among o he pa ame e s
he accu acy and he e iciency a e e alua ed. The s udy esul s demons a e ha he mos
accu a e seman ic segmen a ion is p e o med by anno a ion ools wi h a ee choice o he
pe spec i e and a lasso unc ion o segmen a ion. The anno a o s wo k e y de ail-o ien ed,
which leads o an ex ended p ocessing ime and a dec ease in e iciency. The e ec i eness
is highe wi h a ools such as Recap. In gene al, he PCCT, which only allows he anno a o
o classi y, is e y e icien , bu o e y small objec s i is no e ec i e. Measu emen e o s
in poin clouds make he segmen a ion di icul o any ool, because bounda ies be ween an
objec and poin s ep esen ing mix-pixel e o s, de use e lec ion and come ails canno be
clea ly iden i ied. Geome y-based il e ing can make manual and au oma ic seman ic seg-
men a ion e ec i e, eliable, and accu a e. The segmen a ion o a coa se p e-segmen a ion
o ooms allows smoo he na iga ion h ough he poin clouds. Cu en ha dwa e each i s
limi s o p ocessing e y la ge poin clouds wi h such ools, because he wo king memo ies
a e no la ge enough.
Conclusion and ou look: The esea ch o he a ailable anno a ion ools shows ha only
ew ools a e sui able o he applica ion o modeling indoo ooms. To he bes o he au ho ’s
knowledge, a sys ema ic e alua ion o hese ools has been ca ied ou in PAPER 2 o he
i s ime. An anno a ion ool ha can be used ac oss a ious disciplines is u gen ly needed.
A basis o his de elopmen can be he PCCT. Poin cloud anno a ion ools om comme cial
se ice p o ide s and CWS a e no in es iga ed in de ail due o he lack o anspa ency
ega ding cos s, da a secu i y, da a igh s, and wo king condi ions o c owd-wo ke s. A e
all, he comme cial ools a e he d i e s o many applica ions in which seman ic poin clouds
a e needed.
48
4. E alua ion o he esea ch esul s
4.1.3 De elopmen o an anno a ion ool
RQ 1.3: How can seman ic segmen a ion ools o poin clouds be enhanced and im-
p o ed?
Me hodology: A he beginning o his esea ch1, ew scien i ic ools o seman ic poin
cloud segmen a ion we e a ailable. Comme cial se ice p o ide s p edominan ly o e ed
seman ic segmen a ion o image da a. Comme cial and open-sou ce o line so wa e o
poin cloud p ocessing, such as Geomagic W ap [261], Cloud Compa e, and ex ensions o
CAD p og ams, a e he s a e o he a . An adap a ion o exis ing sys ems o imp o e hem
is no echnically pu pose ul, he e o e he comple e de elopmen o he PCCT is necessa y.
A concep o da a managemen , implemen a ion o segmen a ion me hods, classi ica ion
and isualiza ion is de eloped on he basis o he analyzed anno a ion ools om RQ 1.1 in
sec ion 4.1.1. The PCCT is i e a i ely de eloped and e alua ed in s udies ha a e explained
in he RQs 1.4, 2.1, 2.2 and 2.3.
Findings: The anno a ion ools a ailable on he ma ke show po en ial o u he de el-
opmen s ega ding issues such as da a secu i y, capabili y o a da ase o mul iple-use s,
segmen a ion and classi ica ion unc ions, and au oma ion o ime-consuming sub-ope a ion
s eps (Figu e 28). These issues a e conside ed du ing he de elopmen o he PCCT. The
PCCT is an expe imen al ool, which can be used o op imize he issues and e alua e he
expe imen .
Figu e 28: Cen al issues o imp o emen in a ailable poin cloud anno a ion ools: Da a
secu i y, mul i-use -capabili y, segmen a ion and classi ica ion unc ions, and au-
oma ion o sub-ope a ion s eps.
Poin clouds a e de ailed ep esen a ions o eal buildings and make hidden in o ma ion isi-
ble. This in o ma ion mus be kep sa e o hi d-pa y access o c i ical in as uc u es, such
as o po acili ies, u ili y lines, ai po s, p isons, ail oad acili ies, o esea ch acili ies.
1In 2017
49
4. E alua ion o he esea ch esul s
The usage o CWSs is usually no possible o hese in as uc u es [55]. P ocessing o a
la ge da ase by only one anno a o is also in mos cases no possible and can also lead o
classi ica ion bias in he seman ic poin cloud. The e o e, i is necessa y o s o e he da a
in such a way ha i can be accessed in pa allel. In he bes case, only pa s o he poin
cloud a e made a ailable o he use , allowing o edi bu p e en ing unde s anding he en-
i e in as uc u e. The classi ica ion bias can be minimized by ha ing di e en anno a o s o
pe o m he seman ic segmen a ion. Based on hese conside a ions, a da abase-based con-
cep is de eloped o he PCCT, as published in PAPER 0. The e, a copy o indi idual poin
cloud sec ions a e p o ided o p ocessing ia a b owse -based ool. The esul s o di e en
anno a o s and classi ica ion passes a e connec ed o he o iginal poin cloud, bu a inal
assignmen o he seman ics o a poin is made a e max- o ing o e all classi ica ions.
The class de ini ion o he seman ic segmen a ion mus be ans e able in o he anno a ion
ool. In o de o use he anno a ion ool in almos any applica ion he lis o possible classes
• mus be e-de ined in each case.
• include e y gene al classes, co e a la ge amoun o classes.
• has a hie a chical o ganiza ion.
A cus omizable lis o classes in he ool can lead o he ac ha he class de ini ion is no
longe unique. A e y gene al lis limi s he applica ion scope and a lis wi h oo many classes
can no longe be o e looked, which can lead o di e en unde s andings by he indi idual
anno a o s. Fo example, i he class Wall and Facade a e a ailable, i is no always clea
how hey di e . Fo he PCCT, i is expe imen ed wi h a class de ini ion ha is as gene al
as possible and specialized o building pa s and u ni u es. E en wi h his lis , a con usion
can be seen in he s udy esul s due di e en unde s anding o classes. The hie a chical
o ganiza ion o he classes using e.g. he Wo dNe scheme [262] is a echnique, which allows
a maximum o a ie y and uniqueness. The echnical implemen a ion and he usabili y o his
a ian is complex, since an e ec i e na iga ion mus be applied o a lis o se e al housand
wo ds. Cu en ly, lis o classes ha can be c ea ed by an adminis a o a e mos e ec i e
o p ac ical applica ions.
Some app oaches o au oma ion o segmen a ion o poin clouds a e desc ibed in sec-
ion 2.3. The special cha ac e is ic o TLS poin clouds is ha hey a e eco ded om a
ixed poin o iew and usually ep esen a 360° iew o he scene. A ans o ma ion o he
ca esian coo dina es in o pola coo dina es is di ec ly possible. The ix angula inc emen s
o he pola coo dina es allow a ans o ma ion o he 3D poin cloud o a s uc u ed 2D im-
age. The dis ance measu emen s can become a ea u e a iable. G aph-based me hods in
2D applica ions, such as [95, 263], ha e a high deg ee o de elopmen and p o ide unique
segmen s. Mo eo e , 2D segmen s can usually be isualized and in e p e ed be e by hu-
mans han 3D segmen s. O he de elopmen s use BEV app oaches o [4, 27] o a seman ic
segmen a ion o buildings a e he emo al o ceilings and loo s. These app oaches use a
loo by loo ep esen a ion o he building. A disad an age o he 2D p ojec ion is ha he
geome ical dep h is usually no conside ed in he segmen a ion s ep and a dis o ion oc-
50
4. E alua ion o he esea ch esul s
cu s. Di e en adii and p ojec ions ha e been es ed in he PCCT o pe o m segmen a ion
au oma ically and accu a ely ([264] as well as in PAPER 0 and PAPER 2).
Conclusion and ou look: The de elopmen o an expe imen al p o o ype anno a ion ool
o seman ic segmen a ions is implemen ed wi h he PCCT. The PCCT is based on a ans-
o ma ion o he 3D poin s o 2D pixel, which is pa ially disad an ageous o some o he ap-
plica ions. Di e en in luencing a iables can be es ed wi h he modula -s uc u ed PCCT.
An anno a ion ool o any kind o applica ions and ha i s o all equi emen s om abo e is
no a ailable on he ma ke ye . The op imiza ion o anno a ion ools is s ill impo an bu an
unde - esea ched opic o mo e accu a e and eliable seman ic poin clouds.
4.1.4 Human ac o in seman ic segmen a ions o poin clouds
RQ 1.4: How o become a good anno a o o seman ic poin clouds? How can he pe o -
mance o anno a o s be measu ed? Wha do anno a o s need and how can he ool suppo
hem?
Me hodology: The in luence o he human anno a o is a pa o he s udy in PAPER 2
and is de e mined o he PCCT and Recap. Quan i a i e pa ame e s such as p ocessing
ime, p ecision, and accu acies a e measu ed o calcula ed by means o he high-quali y
e e ence poin cloud da ase . Quali a i e cha ac e is ics a e assessed by a ques ionnai e
ha is answe ed be o e, du ing, and a e he ask by he olun ee anno a o s. Addi ionally,i
is asked o a sel -assessmen , p e ious expe ience and a desc ip ions o how he ools we e
used.
Findings: The ac ha he anno a o has a key ole o he quali y and usabili y o a seman ic
poin cloud has been no ed by [40, 55]. [54] de eloped aluable ules o humb o he selec-
ion and aining o anno a o s. Feedback du ing he anno a ion is gi en by eedback unc ion
in he ool o [265]. Humans a e e y good a ecognizing a ying shapes o objec s [266].
Following hese ideas, a s udy is conduc ed, whose esul s desc ibe he human in luence.
Guidelines o p ocesses and ool de elopmen s should esul om his. The en olun ee s
o he s udy had no, li le, medium o e y much expe ience in he handling o poin clouds
and he seman ic enhancemen o poin clouds in ad ance. This p e ious knowledge allows
an e alua ion o anno a o s aining, de eloped execu ion p ocess and ool unc ions.
T aining documen s a e p epa ed o each in es iga ed ool and a e gi en o he olun ee s
in ad ance. These documen s a e assessed and he olun ee s ha e o pa aph ase he ask
in hei own wo ds. By his i s ask, i could be de e mined ha illus a ions con ibu e o
a be e unde s anding o he ask. The ideas o he applica ion and o poin clouds as well
as i s in e p e a ion a ia e s ongly be ween he anno a o s. The gi en eedback is used
o imp o e he aining documen s wi h example images and de ailed class desc ip ions.
Help ul in he class desc ip ion is o clea ly include o exclude objec s ha a e geome ically
o seman ically simila o o he s. As an example: A doo consis s o he ame, he lea and
he handle, bu no o he window nex o he ame. Di e en olun ee s ha e examined he
documen s a di e en s ages o he de elopmen be o e he inale expe imen .
51

4. E alua ion o he esea ch esul s
The aining and he eedback p ocess a e planned in de ail based on e iew o he li e a u e.
A ew days be o e he expe imen s, all documen s ( ask desc ip ion, class de ini ion and
illus a ed ins uc ions) a e gi en o he olun ee s. In o de o ge amilia ized wi h he ask
and o answe he i s pa o he ques ionnai e. Be o e he anno a ion expe imen , he ools
a e explained and ques ions could be asked. The anno a ion is done wi hou any supe ision.
All olun ee s a e able o sol e he asks. The a e age class accu acy o mos anno a o s
is abo e 90% ( ecall and p ecision). La ge di e ences can be obse ed among he di e en
classes, such as Floo and Ceiling a e abo e 95% o he pa ame e s ecall and p ecision.
The in equen classes Chai and Table a e less accu a e and usually a y be ween 85% and
95% o ecall and p ecision. The pe cen age o TP poin s in he class E oneous poin s is
usually lowe han 50% (p ecision), because in case o doub objec poin s usually become
e oneous poin s. La ge a ia ion is ound in he ime equi ed. Some olun ee s inish
wi hin less han 9% o he maximum ime. La ge di e ences in ime a e also ound be ween
he ools. The PCCT is mo e e icien han Recap, bu he esul s a e no as co ec and he
simplici y o use is pe cei ed as edious. The usage o Recap is complica ed a he beginning
o some olun ee s, so e o s occu ed mo e equen ly due o he segmen a ion unc ions
and he p ocessing akes a long ime. Poin densi y and a ie y in he ac i i y a e seen as
pa icula ly impo an , in addi ion o a clea ask desc ip ion. The na iga ion h ough he poin
clouds and sel -dependen segmen a ion, such as using a lasso, a e unc ions ha make his
possible.
A ela ion be ween high e iciency and co ec ness canno be ound. The olun ee s ind
i help ul o be able o ask ques ions du ing he ask, since he e a e occasional misunde -
s andings o ambigui ies. These esul s a e aken om he s udy in PAPER 2.
Conclusion and ou look: A good anno a o does no need o be an expe in poin clouds.
Unique ask desc ip ions, class de ini ions, and con inual eedback a e mos c i ical o a
success ul seman ic segmen a ion. Exclusi e classi ica ion ools, such as PCCT, le el he
en y h eshold, bu leads o i ing wi h e y la ge da ase s.
4.2 De elopmen o a quali y model o heu is ically desc ibing
seman ic poin clouds
The indings o he second RO makes he quali y o a seman ic poin cloud measu able.
Fo his pu pose, he cha ac e is ics o he poin cloud a e in es iga ed (sec ion 4.2.1) and a
quali y model is de eloped (sec ion 4.2.2). In o de o use he quali y model, i is ans o med
in o an e alua ion ma ix wi h which seman ic poin cloud da ase s, anno a ion ools and
au oma ic wo k lows can be e alua ed (sec ion 4.2.3).
52
4. E alua ion o he esea ch esul s
4.2.1 Poin cloud quali y
RQ 2.1: Wha a e sui able seman ic poin clouds? Wha a e he cha ac e is ics o poin
clouds? How can he cha ac e is ics o he poin cloud be de e mined, measu ed and com-
pa ed?
Me hodology: A li e a u e esea ch o he cha ac e is ics o poin clouds has been pe o med
and he c ea ion p ocess o a manual and au oma ic seman ic segmen ed poin clouds has
been analyzed on he HCU main building da ase .
Findings: The de ini ion o a sui able seman ic poin cloud, which is de eloped as a esul
o his wo k, is gi en by he ollowing s a emen .
De ini ion 1: A good seman ic poin cloud has a homogeneous densi y, is ee om
da a gaps, measu emen and egis a ion e o s, he geome y o he seman ic seg-
men s co esponds o he objec s in eali y and he labels accu a ely desc ibe he objec
seman ics.
A seman ic poin cloud ha comple ely ul ills de ini ion 1 usually does no exis , so ha he
deg ee o indi idual cha ac e is ics a e de e mined. This is necessa y because poin clouds
wi h low quali y le els a e no sui able o some applica ions as discussed in RQ 2.2 (sec-
ion 4.2.2). Be o e he quali y can be de e mined, he c ea ion p ocess and he cha ac e is ics
o he seman ic poin cloud, as well as a de ini ion o e o s mus be s a ed:
De ini ion 2: E o s a e he in luences ha lead o no ul illing he de ini ion o a
sui able seman ic poin cloud.
The c ea ion p ocess o seman ic poin clouds usually consis o he eco ding, he egis a-
ion and a subsequen segmen a ion acco ding o seman ic p ope ies o ep esen ed ob-
jec s. The geome ic co ec ness wi h espec o he eco ded su ace is he mos s udied
cha ac e is ic o he eco ding and egis a ion [81, 82, 83]. The pa ame e s s anda d de i-
a ion and de ia ion om a a ge geome y a e ypically used o desc ibe he cha ac e is ic
geome ic co ec ness. Seman ic segmen a ion is he c ea ion o an abs ac model om he
da a model (measu ed alues) and he knowledge abou he eal wo ld. Such a p ocess is
de ined in [267] and shown in Figu e 29. As addi ional in o ma ion he knowledge o a human
anno a o o he DL algo i hms is used o seman ically enhance he poin cloud. The accu-
acy o he seman ic segmen a ion is desc ibed by he pe o mance o he selec ed manual
o au oma ic algo i hm. This is usually exp essed in e ms o a a io o inco ec and co ec
da a objec s (pixels and poin s). The pa ame e s p ecision, ecall o IoU a e commonly used.
The geome ic shape is no exp essed by hese pa ame e s.
Va ying e o desc ip ions a e ound in publica ions abou seman ic poin clouds. Measu e-
men e o s occu in he o m o in e e ence poin s and noise a ound a su ace, which a e
analyzed o he eco ding sys em. In he p ocess o seman ic segmen a ion, hese poin s
become an addi ional seman ic class. They a e no longe e o s o he seman ic segmen a-
ion. E o s in seman ic segmen a ion a e poin s ha a e assigned o he w ong class. The
de ini ion o e o s change in he wo-s ep p ocess. Howe e , i is necessa y o conside he
53
4. E alua ion o he esea ch esul s
e o s om he p e ious s ep in he ollowing one. In he inal poin cloud, he geome y and
seman ics o he poin cloud should be co ec ly ep esen ed. The eco ding and egis a ion
accu acy a e usually no used in seman ic segmen a ion. They a e ele an o de i ing ge-
ome ies and models om he seman ic poin cloud, so ha he objec s a e no dis o ed and
show he geome y.
Figu e 29: P ocess o seman ic segmen a ion o poin clouds se ing as an abs ac model
o he eali y. Taken om PAPER 2 and adap ed.
Fo an e alua ion and usage o he poin clouds, no only jus discussed cha ac e is ics ge-
ome ic and seman ic p ecision, as well as co ec ness a e ele an . O he cha ac e is ics
a e:
• he a ailabili y o da a and me ada a,
• he p ocess eliabili y,
• he comple eness o da a and p ocessing,
• and he consis ency o da a con en (e.g., ype o a iables),
as explained and de eloped in PAPER 2. In o de o de e mine and compa e he cha ac-
e is ics, he de eloped quali y model is an e ec i e ool. Fo each cha ac e is ic, quali a i e
and quan i a i e pa ame e s a e de ined. The de elopmen o he se o quali y pa ame e s
is desc ibed in mo e de ail in PAPER 2 and is discussed in RQ 2.2 (sec ion 4.2.2). Thus he
compa ison o di e en seman ic poin clouds is possible. In o de o de e mine he deg ee
o quali y o he poin cloud o an applica ion, h eshold alues mus be de ined o each
quali y pa ame e .
Conclusion and ou look: The quali y o a seman ic poin cloud is de e mined by di e en
s eps, which a e usually conca ena ed. The de ini ion o wha is an e o changes om s ep
o s ep. In o de o conside all e o s in he inal seman ic poin cloud, hese o a desc ip ion
54
4. E alua ion o he esea ch esul s
o hese mus be passed on a each s ep. The quali y o a seman ic poin cloud consis s
o many di e en cha ac e is ics, which a e de e mined in he indi idual s ages. S uc u ing
hem in o a quali y model de eloped speci ically o seman ic poin clouds is e ec i e and
implemen ed.
4.2.2 Quali y model o poin clouds
RQ 2.2: How is a quali y model o seman ic poin clouds designed? Which pa ame e s a e
necessa y o he desc ip ion o he cha ac e is ics? Does he quali y pa ame e s di e o
anno a ion and au oma ic seman ic segmen a ion?
Me hodology: The quali y model o [92] is used as he basis o he quali y model o seman ic
poin clouds. The cha ac e is ics om RQ 2.1 (sec ion 4.2.1) ep esen he s uc u e o he
quali y model. In he cou se o he anno a ion p ocess he desc ip i e quali y pa ame e s a e
de e mined and e alua ed.
Findings: The de eloped quali y model desc ibes se en cha ac e is ics o he poin cloud
which a e di ec ly o indi ec ly ela ed o he seman ic segmen a ion. Di ec ela ed cha ac-
e is ics a e accu acy and p ecision. Fo example, indi ec cha ac e is ics a e usage con-
s ain s, such as o he da ase s o [53, 64, 226, 268] which a e only allowed o be used as
a benchma k.
Figu e 30: Quali y model o seman ic enhanced poin clouds. Se en ele an cha ac e is ics
wi h desc ip i e quali y pa ame e s a e shown. Classi ica ion o necessa y pa am-
e e s o : Manual segmen a ions ( illed blue ci cles), manual aining da a gen-
e a ion (un illed blue ci cles) and au oma ic seman ic segmen a ion ( illed g een
ci cles).
The cha ac e is ics become measu able and compa able by quali y pa ame e s. The quali y
pa ame e s o he de eloped model a e summa ized in Figu e 30 wi h he espec i e cha -
ac e is ics. Some quali y pa ame e s a e in a ela ion o o he s, so ha e.g. he p ecision
55
4. E alua ion o he esea ch esul s
4.3.3 Da a p e-p ocessing and da a in luence
RQ 3.3: How can he in luence o he da ase be con olled by da a-based hype pa ame e s
in he seman ic segmen a ion o poin clouds? Wha a e he main in luences?
Me hodology: The da ase , whose cha ac e is ics a e desc ibed and con olled by he
DHPs, is an impo an in luencing a iable o seman ic segmen a ion o poin clouds. The
cha ac e is ics, he class dis ibu ion, he class de ini ion and he inco ec ly measu ed poin s
a e analyzed and e alua ed by empi ical in es iga ions wi h he de eloped wo k low o PA-
PER 1 and PAPER 3.
Findings: The DHPs a e se by he da ase . Examples a e he a ailable ea u e a iables,
he class dis ibu ion o he size o he da ase . A collec ion o he mos common DHPs is
summa ized in Fig 33. An o e iew o he s uc u e o he da ase , he seman ic con en and
he ea u es allow a sys ema iza ion o he in luences and i s in es iga ion. The seman ic
DHPs ha e been in es iga ed in RO 2 (sec ions 4.2.2 and 4.2.3). The selec ion and local
compu a ion o geome ic and spec al ea u es a e s udied o gene al ML me hods in [39,
273]. In addi ion, he da ase size, he no maliza ion o ea u es, and he densi y o he poin
clouds a e conside ed in many ne wo k a chi ec u e de elopmen s [37, 200, 274]. Rules o
humb can be de i ed om his, bu hey a e no suppo ed by any sys ema ic p oo . The
s uc u e o poin cloud da ase s is usually only in es iga ed wi h espec o he inpu o ma s.
Wi h ew excep ions [172, 250, 271, 275, 276] class de ini ions, cha ac e is ics o e oneous
poin s, and class size di e ences a e no conside ed in poin cloud da ase s, e en hough
hese a e conside ed o be a well-known in luencing ac o in seman ic segmen a ion o im-
ages [258, 277]. These h ee DHPs a e explo ed in PAPER 1 and PAPER 3.
Figu e 33: DHPs o seman ic poin clouds. The DHPs can be dis inguished acco ding o
s uc u al, seman ic, geome ic and spec al cha ac e is ics. A selec ion o he
mos common DHPs o each p ope y is summa ized.
The class E oneous poin s is usually de e mined less p ecisely by mos algo i hms o se-
man ic poin cloud segmen a ions han he objec classes. This is shown by he analysis o
poin -based CNN a he leade boa d o he TLS da ase o Seman ic3D.ne [40] (Figu e 34).
In addi ion, his analysis shows ha mo e equen classes, such as Building and Road, a e
de e mined mo e accu a ely han he smalle class T ee.
62

4. E alua ion o he esea ch esul s
Figu e 34: Seman ic segmen a ion accu acy (IoU) o ou common ne wo k a chi ec u es o
he da ase : Seman ic3d.ne [40]. Selec ion o ou om eigh classes o his
da ase . The class Scanning A i ac s, which is equal o he class E oneous poin ,
can be de ec ed poo ly compa ed o he la ge classes. Values a e aken om he
leade boa d o [40].
The obse a ions indica e ha such in luences exis (Figu e 34). In PAPER 1, he in luence o
he p esence o absence o he class E oneous poin s a e in es iga ed. Figu e 35a shows
he esul s o he seman ic segmen a ion wi hou E oneous poin s. The equen classes
T ee and Building a e de e mined wi h mo e han 80% ecall and p ecision. The in equen
class S ee Fu ni u e is de e mined wi h less han 10% ecall and p ecision. I he class
E oneous poin s is added, ecall and p ecision o all classes a e lowe han 54%, as shown
in Figu e 35b. F om his example, i can be seen ha he e is an in luence o he e oneous
poin s in he seman ic segmen a ion o poin cloud da ase s. E oneous poin s a e a anged
simila ly as objec poin s, as e oneous poin s a e caused by mul iple and di use e lec ions.
In he la ge s udy o PAPER 2, he in luence could be con i med. Howe e , wi h a la ge
indoo da ase s he in luence is less. Fo in equen class a posi i e e ec o he p esence
o he class E oneous poin s can also be obse ed by a highe p ecision alue.
Figu e 35: Compa ison o seman ic accu acy ( ecall and p ecision) on he poin cloud o he
Ha enCi y (ou doo ) da ase : a) Wi hou he class E oneous poin s and b) Wi h
he class E oneous poin s. Selec ion o h ee classes ha ha e di e en equen-
cies in he da ase . Da a om PAPER 1.
63
4. E alua ion o he esea ch esul s
The in luence o he class E oneous poin s is he e o e no he only c ucial ac o o he
seman ic accu acy, bu he combina ion o he seman ic classes and i s poin dis ibu ion. The
di ision acco ding o classes akes place on he basis o he class de ini ion, ha ules which
classes a e de e mined wi h he seman ic segmen a ion. The bes possible di e en ia ion is
always possible i he ea u es o he poin clouds can be clea ly sepa a ed om each o he .
The ceiling and he loo can be well sepa a ed by di e en alues o he ea u e a iable
heigh . Such conside a ions can be aken in o accoun when de eloping he class de ini ion.
I , due o he ask, a sepa a ion by classes wi h e y simila ea u es is no possible, a s ep-
wise seman ic segmen a ion can be pe o med as ou lined in Figu e 36. Simila classes
a e combined in a supe class in he i s s age (Ne wo k A) and hen Ne wo k B is used
o he sepa a ion. The in luence o a class de ini ion and he hie a chical p ocess could be
demons a ed in he s udy o PAPER 2. This shows a sligh inc ease in seman ic accu acy
o he Window and Doo classes. Howe e , his de eloped p ocess is s ongly linked o he
indi idual ooms.
Figu e 36: S ep-wise seman ic segmen a ion o imp o ed di e en ia ion o classes wi h sim-
ila ea u es. Wi h ne wo k A, a segmen a ion is pe o med o gene al classes,
which is e ined in ne wo k B.
An adjus men o he class de ini ion does no necessa ily lead o he classes ha ing he same
numbe o poin s. Classes such as Wall,Floo and Ceiling a e mo e equen ly ep esen ed
classes in he poin cloud, han Doo s,Fu ni u e and Windows due o hei la ge su aces.
The lea ning algo i hm will lea n hese classes mo e o en han he in equen ones due o he
mo e equen eeding wi h poin s whose class is wall, loo o ceiling. To enhance lea ning
in a o o he in equen classes, hei p opo ion can be a i icially inc eased (Figu e 37a),
he inpu s can be emphasized wi h a highe p opo ion o in equen poin s (Figu e 37c), o
in a loss calcula ion, he poin s o he in equen classes can be ewa ded by a highe weigh
(Figu e 37b). Ex ensions o poin s can be done andomly o by conside ing local condi ions,
as wi h he SMOTE me hod. These h ee app oaches a e in es iga ed in PAPER 3 in se e al
a ian s using he gene al HP se om RQ 3.2 (sec ion 4.3.2). Again, a modes and scene-
dependen inc ease in seman ic accu acy is obse ed due o a highe ecall o he in equen
classes.
64
4. E alua ion o he esea ch esul s
Figu e 37: Da ase op imiza ion me hods o seman ic poin cloud segmen a ion: a) Da ase
expansion by andomly copying poin s, b) weigh ing he loss unc ion, and c)
da ase expansion by copying inpu s wi h in equen poin s.
Conclusion and ou look: DHPs ep esen a measu able in luence o he seman ic seg-
men a ion o poin clouds. The DHPs: Class de ini ions, p opo ion o e oneous poin s and
class size di e ences in luence he seman ic segmen a ion esul s and a e op imized in he
con ex o his wo k. Howe e , his op imiza ion can only be alid o a p opo ion o he
scenes. Fu he in es iga ions on DHPs a e necessa y o es ablish ules o he op imal
choice o hem. An analysis o he seman ics in he scenes is necessa y.
65
5 Conclusion and ou look
This sec ion summa izes he key indings, he conclusions and he esponses o he RQs
(sec ion 5.1). In e media e conclusions a e summa ized in sec ion 4 a he end o each RQ.
Nex s eps o u he de elopmen and op imiza ion based on he esul s o his wo k a e
desc ibed in sec ion 5.2.
5.1 Conclusion
This hesis shows ha he de elopmen o a wo k low wi h which any ype o poin clouds
can be seman ically augmen ed, in any kind o applica ion is no possible a he cu en s a e
o he a . The key easons a e lack o knowledge abou da ase s and unde ined ules o
HPs. Ne e heless, DL me hods a e mos sui able o seman ic segmen a ion.
The seman ic enhancemen o 3D poin clouds is a necessa y s ep in o de o p oduce highly
accu a e digi al models o he eal wo ld. The seman ics o poin clouds is cen al o he
usabili y and he in e p e abili y, i au oma ic digi al p ocessing should o mus used. DL
algo i hms, such as poin -based CNN, p oduce accu a e seman ic poin clouds i op imal HPs
and su icien aining poin clouds a e used. Op imal HPs, algo i hms, and aining da a we e
explo ed on he HCU main building da ase and p edominan ly wi h he Poin Ne me hod in
his hesis. In o de o in es iga e he in luences h ee de elopmen s we e necessa y:
• The de elopmen o a wo k low o au oma ic seman ic segmen a ion.
• The de elopmen o a ool o manual seman ic segmen a ion.
• The de elopmen o a quali y model o he p ocess and he seman ic poin cloud.
The i s challenge in wo k low de elopmen was, ha mos DL me hods use hei own da a
p e-p ocessing me hods. This is dic a ed by he da a o ma o he eco ding senso and is
no adap ed o he op imal pe o mance o he algo i hm o o he da a con en .
The second challenge is he ad ancemen o he ha dwa e, APIs, and DL me hods. To e -
ec i ely conside new ha dwa e and API de elopmen s as well as di e en DL algo i hms,
a modula wo k low which is independen o he da ase o ma s has been de eloped. This
wo k low consis s o he modules o ea u e ex ension, ea u e alue no maliza ion, inpu
o ma ing, aining, e alua ion and applica ion. Each module can be modi ied by a ew pa-
ame e s. The desc ibed wo k low is a b idge be ween high-end de elopmen s and p ac ical
measu emen s.
Tools o poin cloud anno a ions shall accele a e, simpli y, s anda dize and op imize he
e y labo -in ensi e, indi idual p ocess o poin cloud anno a ion. The seman ic poin clouds
a e c ucial, because hey a e he knowledge ca ie o DL algo i hms. The de elopmen o
he PCCT is based on he abo e equi emen s and educes indi idual human in luences by
66
5. Conclusion and ou look
means o au oma ic segmen a ion. A educ ion o he p ocessing ime compa ed o Recap
was possible by 42% on a e age. Howe e , his esul es in a dec ease in seman ic ac-
cu acy o up o 16% o ecall and up o 12% o p ecision ac oss all classes. The PCCT
mul i-use capabili y makes i ideal o s udies in which di e en segmen a ion pe o mances
a e in es iga ed as an in luence o allow e icien p ocessing o la ge da ase s by di e en
use s.
Which me ics a e used o an e alua ion is no s anda dized in li e a u e and he exac ex-
p esses o each me ic is some imes no clea . In addi ion, hese me ics usually only ep e-
sen he seman ic accu acy in ela ion o a GT da ase . All hese ambigui ies in he de ini ions
limi he meaning ulness o he me ics. A comple e e alua ion o a seman ic poin cloud in-
cludes se e al cha ac e is ics, such as geome ic accu acy, eliabili y, comple eness, a ail-
abili y, and in eg i y. These cha ac e is ics o he seman ic poin cloud a e ep esen ed by he
quali y model de eloped in his wo k. A comple e e alua ion and compa ison o he da ase
cha ac e is ics and pe o mance o all p ocessing s eps is hus possible. The me ics a e
in eg a ed in o he quali y model as quali y pa ame e s and de ine oge he wi h addi ional
quali y pa ame e s a highe signi icance model o sys ema ic in es iga ions, compa isons
and he examina ion o he sui abili y o da a and algo i hms in a speci ic applica ion.
The h ee de elopmen s o he hesis a e used o s udy he in luence o da ase s and poin
clouds in manual and au oma ic seman ic segmen a ions. Di e ences in accu acy, e ec-
i eness, and e iciency in manual seman ic segmen a ion we e iden i ied, caused by he
unc ions in he ools, he use aining, and he poin clouds. I was ound ha u he de-
elopmen s o anno a ion ools a e manda o y in o de o p oduce su icien aining da a o
p oduc i e applica ions o DL algo i hms. T aining da a poin clouds a e key ma e ials, bu
ha e been a ely s udied.
This wo k ocuses on he cha ac e is ics o he da ase s and he poin clouds. The p esence
o e oneous poin s a ec s he seman ic segmen a ion by dec easing he seman ic accu acy
o equen classes. In con as , o in equen classes an inc ease in seman ic accu acy is
obse ed o up o 22% (in e io ) o he Poin Ne baseline me hod, i he class E oneous
poin s is pa o he class de ini ion. Unequal class pa i ioning leads o he ac ha in e-
quen classes ha e a lowe accu acy. In many examples o his hesis i can be obse ed
ha equen classes a e lea ned e y well (>90% ecall) and in equen classes a e no
lea ned (<50% ecall). Sys ema ic and a i icial modi ica ion o he poin cloud da ase can
imp o e he ecognizabili y o in equen classes ( ecall >50%). Classes ha show e y
simila ea u es a e mo e di icul o sepa a e han classes ha ha e di e en ea u es (e.g.,
heigh s). A hie a chical app oach o he class de ini ion could in some cases (e.g., windows
and doo s as openings) imp o e he seman ic segmen a ion. This can be obse ed om he
ecall, which is up o 43% highe o he class Openings in he baseline me hod.
Finally, no only he cha ac e is ics o he indi idual poin s a e c ucial, bu also he cha ac e -
is ics o a neighbo hood. How he local and global neighbo hood can be aken in o accoun
is discussed in many pape s, bu gene al ules ha allow o apply hem a e no a ailable ye .
67

5. Conclusion and ou look
Some a emp s ha e been made o ake in o accoun di e en densi ies and di e en la ge
a eas in he inpu o he algo i hm. Howe e , i s in luence emains o be in es iga ed in de ail.
Some possible app oaches ha e been examined in his wo k and some new app oaches will
be explained in he ou look (sec ion 5.2).
5.2 Ou look
The esea ch o his disse a ion e eals h ee addi ional esea ch a eas ha equi e u he
in es iga ions and de elopmen s. These esea ch a eas a e he op imiza ion o he inpu o
DL algo i hms (sec ion 5.2.1), he manual p e-selec ion o ea u e a iables (sec ion 5.2.2),
and he applica ion o he quali y model (sec ion 5.2.3). Fu he mo e, esea ch on algo i hms,
on s a egies o op imiza ion o HPs and combina ions o ML and DL a e he cu en issues.
5.2.1 Inpu o ma
In he seman ic segmen a ion o poin clouds wi h DL me hods, he seman ics a e lea ned
om he a angemen o he poin s and i s addi ional ea u e a iables, such as in ensi y, poin
no mals o colo alues. Poin clouds con aining hund eds o housands o poin s canno be
ed in o a ne wo k a chi ec u e a once, so only a subse can be p ocessed a each ime. As a
esul , a subse o he in o ma ion can be used o global ea u e ex ac ion and classi ica ion.
In o ma ion om he en i e poin cloud is no known a all (in he case o Poin Ne ) o only
insu icien ly known (in he case o RandLaNe o Poin Ne ++). Fo Poin Ne and 2D CNNs
a possible app oach o add ess his issue is he use o g aphs as inpu o ma , such as
desc ibed in sec ion 2.6.2. Supplemen ing hese me hods o he li e a u e, local and global
adjacency ma ices exp essing he adjacency o he poin s can be compu ed om a kNN
g aph o each poin . Adjacency ma ices ha e he ad an age ha hey o de he opological
ela ionships and can be ep esen ed in a 2D o ma . By mul iplying he adjacency ma ices
wi h he ea u es o he poin s, a enso can be compu ed o inpu o a 2D CNN, such as U-
Ne (Figu e 38). Also, he adjacency ma ix o a local a ea may be used as a di ec inpu o
Poin Ne and is a ca ie o addi ional in o ma ion abou he local ela ionships o he poin s.
Figu e 38: P ocess o c ea ing an adjacency ma ix and applying i as a ne wo k inpu .
68
5. Conclusion and ou look
Ini ial es s o his me hod show ha an inc ease in seman ic accu acy is possible wi h a
Poin Ne a chi ec u e. Cu en ly, da a p epa a ion is he bo leneck o his me hod. La ge
s udies wi h di e en da ase s need o be conduc ed o alida e his obse a ion. The inpu
o ma o he poin cloud o he algo i hm is seen as an impo an in luence ha mus be
in es iga ed sys ema ically in u u e esea ch.
5.2.2 Hand-c a ed ea u e selec ion
The manual selec ion o ea u es o seman ic segmen a ion wi h ML me hods is necessa y
o p e-p ocessing o poin cloud da a, which was in es iga ed and op imized in [185]. This
idea is applied o CNN in [40, 278] by compu ing momen s and eigen alues as addi ional ea-
u es. Subsequen ly, an op imiza ion o he se o inpu ea u es is pe o med. Eigen alues
and momen s ca y in o ma ion no only abou he poin i sel , bu also abou he neighbo -
hood, so hey b ing in mo e global in o ma ion in o he algo i hm. To compu e his ype o
ea u es, i is necessa y o de ine a local neighbo hood o e which he eigen alues a e de e -
mined. In a es , he sum o eigen alue, he plana i y and he linea i y a e calcula ed, using
a adius o 3.5 cm o including he neighbo hood. These di e ences o alues a e shown in
Figu es 39b o 39d. Figu e 39a shows he GT classi ica ion o he poin cloud. I can be seen
ha objec bounda ies can be dis inguished mo e accu a e han in he case o mos spec al
ea u es.
Figu e 39: Eigen alue based ea u es calcula ed om geome ic ea u es (x, y, z): a) GT
seman ic segmen a ion. b) Sum o eigen alues as ea u e. c) Plana i y as a
ea u e. d) Linea i y as a ea u e. A his og am is shown nex o he legend.
In he expe imen o his app oach wo es s we e pe o med wi h he Poin Ne wo k low.
These expe imen s show ha o he class combina ion E oneous poin s and Objec s com-
69
5. Conclusion and ou look
pa able accu acies a e achie ed wi h baseline me hod (Figu e 40). Fo he class combina ion
o In e io and Building pa s, i can be demons a e ha wi h he new eigen alue ea u e se
ecall and p ecision dec ease o mo e han 30% (Figu e 41). In u he s udies he ea u e
calcula ion and he a iable selec ion ha e o be op imized o imp o e au oma ic seman ic
segmen a ion.
Figu e 40: Seman ic poin cloud o he classes Objec s and E oneous poin s. The seman ic
segmen a ion is pe o med using he Poin Ne -based wo k low wi h he ea u es:
x-, y-,z-coo dina es, sum o eigen alues,plana i y and linea i y.
Figu e 41: Seman ic poin cloud o he classes Building pa s and In e io . The seman ic
segmen a ion is pe o med using he Poin Ne -based wo k low wi h he ea u es:
x-, y-, z-coo dina es, sum o eigen alues,plana i y and linea i y.
5.2.3 Poin cloud quali y assessmen
The e alua ion o seman ic segmen a ion in eal-wo ld applica ions has been ea ed only
ma ginally in he de elopmen s so a . Since he in luence o he da a and i s enhancemen
is c ucial o he pe o mance o he algo i hms, hese should be in es iga ed in mo e de ail.
The de eloped quali y model o his wo k p o ides a basis o his, o which h esholds pe
quali y pa ame e s ill ha e o be de ined. These should be p ima y o equen applica ions
o seman ic segmen a ion, as i is he case o indoo scenes, acades, o s ee scenes. In
addi ion, hese h esholds mus be algo i hm-dependen ly de e mined.
70
Bibliog aphy
[1] P. Welche ing. Nach de Co ona-Pandemie - Wie die Digi alisie ung de A bei swel
je z wei e geh . Ed. by Deu schland unk. 2021. URL: h ps://www.deu schland unk.
de/nach-de - co ona-pandemie- wie-die- digi alisie ung-de - 100.h ml ( isi ed on
04/15/2023).
[2] H. Kahmen. Angewand e Geodäsie: Ve messungskunde. De G uy e Ve lag, 2005.
DOI: 10.1515/9783110911145.
[3] A. Wiese , H. Kuhlmann, V. Schwiege , and W. Niemeie . “Ingenieu geodäsie – eine
Ein üh ung”. In: Ingenieu geodäsie. Ed. by W. Schwa z. Sp inge Spek um, Be lin,
Heidelbe g, 2017, pp. 1–22. DOI: 10.1007/978-3-662-47188-3_19.
[4] V. S ojano ic, H. Shoush a i, C. Aska , A. Scheide , C. Schuld , N. Hellweg, and H.
S e nbe g. “A Concep ual Digi al Twin o 5G Indoo Na iga ion”. In: The Ele en h In-
e na ional Con e ence on Mobile Se ices, Resou ces and Use s - MOBILITY 2021.
2021, pp. 5–14.
[5] H. Hosamo, M. H. Hosamo, H. K. Nielsen, P. R. S enne ig, and K. S id . “Digi al Twin
o HVAC sys em (HVACDT) o mul iobjec i e op imiza ion o ene gy consump ion and
he mal com o based on BIM amewo k wi h ANN-MOGA”. In: Ad ances in Building
Ene gy Resea ch 17.2 (2022), pp. 125–171. DOI: 10.1080/17512549.2022.2136240.
[6] R. Kaden, C. Clemen, R. Seuß, J. Blankenbach, R. Becke , A. Eichho n, A.
Donaubaue , and U. G ube . Lei aden Geodäsie und BIM. Tech. ep. 2.1. DVW e.V.
und Runde Tisch GIS e.V., 2020. URL: h ps : / / d w . de / images / anhang / 2757 /
lei aden-geodaesie-und-bim2020online e sion.pd ( isi ed on 07/01/2022).
[7] T. Bende , M. Hä ig, E. Jaspe s, M. K äme , M. May, M. Schlund , and N. Tu ianskyj.
“Building In o ma ion Modeling”. In: CAFM-Handbuch. Sp inge Fachmedien Wies-
baden, 2018. Chap. 11, pp. 295–324. DOI: 10.1007/978-3-658-21357-2_11.
[8] BIM.Hambu g. BIM-Lei aden ü die FHH Hambu g. Tech. ep. BIM.Hambu g, 2019.
[9] DIN 18710. Enginee ing su ey. Deu sche No m, 2010.
[10] R. Becke , E. Lublasse , J. Ma ens, R. Wollenbe g, H. Zhang, S. B ell-Cokcan, and
J. Blankenbach. “Enabling BIM o P ope y Managemen o Exis ing Buildings Based
on Au oma ed As-is Cap u ing”. In: P oceedings o he In e na ional Symposium on
Au oma ion and Robo ics in Cons uc ion. In e na ional Associa ion o Au oma ion
and Robo ics in Cons uc ion, 2019. DOI: 10.22260/isa c2019/0028.
[11] K. Soliman, K. Naji, M. Gunduz, O. B. Tokdemi , F. Faqih, and T. Zayed. “BIM-based
Facili y Managemen Models o Exis ing Buildings”. In: Jou nal o Enginee ing Re-
sea ch (2021). DOI: 10.36909/je .11433.
I
Bibliog aphy
[75] O. Wasenmülle and D. S icke . “Compa ison o Kinec V1 and V2 Dep h Images in
Te ms o Accu acy and P ecision”. In: Compu e Vision – ACCV 2016 Wo kshops.
Sp inge In e na ional Publishing, 2017, pp. 34–45. DOI: 10.1007/978-3-319-54427-
4_3.
[76] K. Saue mann. “Modellie ung on Bes andsbauwe ken am Beispiel de Messe Do -
mund, Halle 4”. In: 22. In e na ionale Geodä ische Woche Obe gu gl. Ed. by T.
Weinold. 2023, pp. 67–77. ISBN: 978-3-87907-738-0.
[77] C. Wang, S. Hou, C. Wen, Z. Gong, Q. Li, X. Sun, and J. Li. “Seman ic Line
F amewo k-Based Indoo Building Modeling Using Backpacked Lase Scanning
Poin Cloud”. In: ISPRS Jou nal o Pho og amme y and Remo e Sensing 143 (2018),
pp. 150–166. DOI: 10.1016/j.isp sjp s.2018.03.025.
[78] B. S ahl and A. Rei e e . “Mobile Mapping Pla o m wi h In eg a ed End-To-End Da a
P ocessing Chain o Sma Ci y Applica ions”. In: Remo e Sensing Technologies and
Applica ions in U ban En i onmen s VII. Ed. by N. Ch ysoulakis, T. E be sede , and
Y. Zhang. SPIE, 2022. DOI: 10.1117/12.2633965.
[79] T. Luhmann, S. Robson, S. Kyle, and J. Boehm. Close-Range Pho og amme y and
3D Imaging. De G uy e , 2013. DOI: 10.1515/9783110302783.
[80] F. Nei zel, B. Go don, and D. Wujanz. DVW-Me kbla 7-2014, Ve ah en zu s an-
da disie en Übe p ü ung on e es ischen Lase scanne n (TLS). Tech. ep. DVW,
2014. URL: h ps://d w.de/ e oe en lichungen/s andpunk e/1149- e ah en-zu -
s anda disie en- uebe p ue ung- on- e es ischen- lase scanne n- ls ( isi ed on
10/28/2021).
[81] D. Wujanz, M. Bu ge , F. Tschi schwi z, T. Nie zschmann, F. Nei zel, and T. Ke -
s en. “De e mina ion o In ensi y-Based S ochas ic Models o Te es ial Lase Scan-
ne s U ilising 3D-Poin Clouds”. In: Senso s 18.7 (2018), p. 2187. DOI: 10.3390/
s18072187.
[82] H. Neue . “Quali ä sbe ach ungen zu TLS-Da en”. In: DVW-Semina Qual-
i ä ssiche ung geodä ische Mess- und Auswe e e ah en. Vol. 95. DVW-
A bei sk eis 3 »Messme hoden und Sys eme«, 2019, pp. 69–89.
[83] T. P. Ke s en, M. Linds aed , and M. S ange. “Geome ische Genauigkei sun e -
suchungen ak uelle e es ische Lase scanne im Labo und im Feld”. In: AVN
2.128 (2021), pp. 59–67.
[84] B. O ne , S. Papp, and M. Mei -Hube . “Einlei ung”. In: Handbuch Da a Science. Ca l
Hanse Ve lag GmbH & Co. KG, 2019, pp. 1–14. DOI: 10.3139/9783446459755.001.
[85] T. A. Runkle . “Ein üh ung”. In: Da a Mining. Sp inge Fachmedien Wiesbaden, 2015,
pp. 1–3. DOI: 10.1007/978-3-8348-2171-3_1.
[86] G. Langs and R. Wazi . “Machine Lea ning”. In: Handbuch Da a Science. Ca l Hanse
Ve lag GmbH & Co. KG, 2019, pp. 177–198. DOI: 10.3139/9783446459755.006.
VIII

Bibliog aphy
[87] M. Mei -Hube , S. Papp, and B. O ne . “Da ena chi ek u en”. In: Handbuch Da a
Science. Ca l Hanse Ve lag GmbH & Co. KG, 2019, pp. 103–123. DOI: 10.3139/
9783446459755.003.
[88] T. Dullien. “Maschinelles Le nen und küns liche In elligenz in de In o ma ionssiche -
hei ”. In: Da enschu z und Da ensiche hei 42.10 (2018), pp. 618–622. DOI: 10.1007/
s11623-018-1012-3.
[89] R. Wazi and G. Langs. “S a is ik-G undlagen”. In: Handbuch Da a Science. Ca l
Hanse Ve lag GmbH & Co. KG, 2019, pp. 141–176. DOI: 10.3139/9783446459755.
005.
[90] A. Ng and K. Soo. “Das Wich igs e in Kü ze …” In: Da a Science – Was is das
eigen lich?! Sp inge Be lin Heidelbe g, 2018, pp. 1–18. DOI: 10.1007/978-3-662-
56776-0_1.
[91] B. O ne . “Da a Pipelines”. In: Handbuch Da a Science. Ca l Hanse Ve lag GmbH
& Co. KG, 2019, pp. 125–140. DOI: 10.3139/9783446459755.004.
[92] T. Wil scho. “Siche e In o ma ion du ch in as uk u ges ü z e Fah e assis enzsys-
eme zu S eige ung de Ve keh ssiche hei an S aßenkno enpunk en”. PhD hesis.
Uni e si y S u ga , 2004.
[93] T. A. Runkle . “Da en o e a bei ung”. In: Da a Mining. Sp inge Fachmedien Wies-
baden, 2015, pp. 23–36. DOI: 10.1007/978-3-8348-2171-3_3.
[94] M. Weinmann, B. Ju zi, S. Hinz, and C. Malle . “Seman ic Poin Cloud In e p e a ion
Based on Op imal Neighbo hoods, Rele an Fea u es and E icien Classi ie s”. In:
ISPRS Jou nal o Pho og amme y and Remo e Sensing 105 (2015), pp. 286–304.
DOI: 10.1016/j.isp sjp s.2015.01.016.
[95] P. F. Felzenszwalb and D. P. Hu enloche . “E icien G aph-Based Image Segmen-
a ion”. In: In e na ional Jou nal o Compu e Vision 59.2 (2004), pp. 167–181. DOI:
10.1023/b: isi.0000022288.19776.77.
[96] R. Schnabel, R. Wahl, and R. Klein. “E icien RANSAC and o Poin -Cloud and
Shape De ec ion”. In: Compu e G aphics Fo um 26.2 (2007), pp. 214–226. ISSN:
0167-7055. DOI: 10.1111/j.1467-8659.2007.01016.x.
[97] M. A. Fischle and R. C. Bolles. “Random Sample Consensus”. In: Communica ions
o he ACM 24.6 (1981), pp. 381–395. DOI: 10.1145/358669.358692.
[98] N. Shukla. Machine Lea ning wi h Tenso low. Ed. by K. F icklas. manning pub, 2018.
272 pp. ISBN: 1617293873.
[99] A. Y. Ng and M. I. Jo dan. “On Disc imina i e s. Gene a i e Classi ie s: A Compa -
ison o Logis ic Reg ession and Nai e Bayes.” In: Ad ances in Neu al In o ma ion
P ocessing Sys ems (2002), pp. 841–848.
[100] E. Che, J. Jung, and M. Olsen. “Objec Recogni ion, Segmen a ion, and Classi ica ion
o Mobile Lase Scanning Poin Clouds: A S a e o he A Re iew”. In: Senso s 19.4
(2019), p. 810. DOI: 10.3390/s19040810.
IX
Bibliog aphy
[101] L. I. Smi h. A u o ial on P incipal Componen s Analysis. Tech. ep. Repo No. OUCS-
2002-12. Uni e si y o O ago, NZ, 2002. URL: h p://hdl.handle.ne /10523/7534
( isi ed on 02/21/2023).
[102] T. A. Runkle . “Da en isualisie ung”. In: Da a Mining. Sp inge Fachmedien Wies-
baden, 2015, pp. 37–58. DOI: 10.1007/978-3-8348-2171-3_4.
[103] Y. Xu, L. Hoegne , S. Tu as, and U. S illa. “Voxel- and G aph-Based Poin Cloud
Segmen a ion o 3D Scenes Using Pe cep ual G ouping Laws”. In: ISPRS Annals
o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences IV-1/W1
(2017), pp. 43–50. DOI: 10.5194/isp s-annals-i -1-w1-43-2017.
[104] R. T iebel, J. Shin, and R. Y. Siegwa . “Segmen a ion and Unsupe ised Pa -based
Disco e y o Repe i i e Objec s”. In: Robo ics: Science and Sys ems VI. Robo ics:
Science and Sys ems Founda ion, 2010. DOI: 10.15607/RSS.2010.VI.009.
[105] A.-V. Vo, L. T uong-Hong, D. F. Lae e , and M. Be olo o. “Oc ee-Based Region
G owing o Poin Cloud Segmen a ion”. In: ISPRS Jou nal o Pho og amme y and
Remo e Sensing 104 (2015), pp. 88–100. DOI: 10.1016/j.isp sjp s.2015.01.011.
[106] M. Ahmed, R. Se aj, and S. M. S. Islam. “The k-means Algo i hm: A Comp ehensi e
Su ey and Pe o mance E alua ion”. In: Elec onics 9.8 (2020), p. 1295. DOI: 10.
3390/elec onics9081295.
[107] H. Zhang. “The Op imali y o Nai e Bayes”. In: The Flo ida AI Resea ch Socie y. 2004.
[108] T. A. Runkle . “Klassi ika ion”. In: Da a Mining. Sp inge Fachmedien Wiesbaden,
2015, pp. 89–107. DOI: 10.1007/978-3-8348-2171-3_8.
[109] J. C. S ol z us. “Logis ic Reg ession: A B ie P ime ”. In: Academic Eme gency
Medicine 18.10 (2011), pp. 1099–1104. DOI: 10.1111/j.1553-2712.2011.01185.x.
[110] L. Jiang, Z. Cai, D. Wang, and S. Jiang. “Su ey o Imp o ing K-Nea es -Neighbo
o Classi ica ion”. In: IEEE Fou h In e na ional Con e ence on Fuzzy Sys ems and
Knowledge Disco e y. IEEE, 2007. DOI: 10.1109/ skd.2007.552.
[111] G. Shakhna o ich and T. Da ell. Nea es -Neighbo Me hods in Lea ning and Vision.
Theo y and P ac ice (Neu al In o ma ion P ocessing). The MIT P ess, 2006, p. 262.
ISBN: 9780262195478.
[112] K. Kunze. Haup semina Machine Lea ning:Suppo Vec o Machines, Ke nels. 2004.
URL: h ps://campa .in. um.de/ wiki/pub/Fa /MachineLea ningWiSe2003/kunze_
ausa bei ung.pd ( isi ed on 01/12/2023).
[113] S. Le ine. Week 6: Suppo Vec o Machines. 2016. URL: h ps : / / cou ses . cs .
washing on.edu/cou ses/cse446/16sp/s m_2.pd ( isi ed on 01/13/2023).
[114] B. DeVille. “Decision T ees”. In: Wiley In e disciplina y Re iews: Compu a ional
S a is ics 5.6 (2013), pp. 448–455. DOI: 10.1002/wics.1278.
[115] L. B eiman. “Random Fo es s”. In: Machine Lea ning 45.1 (2001), pp. 5–32. DOI:
10.1023/a:1010933404324.
X
Bibliog aphy
[116] D. Koguciuk, Ł. Chechliński, and T. El-Gaaly. “3D Objec Recogni ion wi h Ensemble
Lea ning - A S udy o Poin Cloud-Based Deep Lea ning Models”. In: Ad ances in
Visual Compu ing (2019), pp. 100–114. DOI: 10.1007/978-3-030-33723-0_9.
[117] S. S ecke . “Küns liche Neu onale Ne ze - Au bau und Funk ionsweise”. In: A -
bei spapie e WI. Leh s uhl ü Allg. BWL und Wi scha sin o ma ik, Johannes
Gu enbe g-Uni e si ä Mainz, 1997.
[118] I. Good ellow, Y. Bengio, and A. Cou ille. Deep Lea ning. The MIT P ess, 2016.
800 pp. ISBN: 0262035618. URL: h ps://www.ebook.de/de/p oduc /26337726/
ian_good ellow_yoshua_bengio_aa on_cou ille_deep_lea ning.h ml ( isi ed on
01/13/2023).
[119] T. Rashid. Neu onale Ne ze selbs p og ammie en. D Punk Ve lag GmbH, 2017.
232 pp. ISBN: 3960090439.
[120] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Eme . “E icien P ocessing o Deep Neu-
al Ne wo ks: A Tu o ial and Su ey”. In: P oceedings o he IEEE 105.12 (2017),
pp. 2295–2329. DOI: 10.1109/jp oc.2017.2761740.
[121] M. Heine . “A i icial Neu al Ne wo ks – How o Open he Black Boxes?” In: Fi s
Wo kshop on Applica ion o A i icial In elligence and Inno a ions in Enginee ing
Geodesy. 2008, pp. 42–62.
[122] I. Good ellow. Deep Feed o wa d Ne wo ks. 2016. URL: h p : / / www .
deeplea ningbook.o g/lec u e_slides.h ml ( isi ed on 01/13/2023).
[123] L. Fei-Fei, J. Wu, and R. Gao. CS231n: Con olu ional Neu al Ne wo ks o Visual
Recogni ion. Ed. by S and o d ision and Lea ning Lab. 2020. URL: h ps://cs231n.
gi hub.io/neu al-ne wo ks-case-s udy/ ( isi ed on 02/02/2023).
[124] D. Ma u ana and S. Sche e . “VoxNe : A 3D Con olu ional Neu al Ne wo k o
Real-Time Objec Recogni ion”. In: IEEE/RSJ In e na ional Con e ence on In elligen
Robo s and Sys ems. IEEE, 2015, pp. 922–928. DOI: 10.1109/i os.2015.7353481.
[125] N. Hö . “Bildsegmen a ion in Objek -Klassen mi Kon olu ionalen Neu onalen Ne -
zen”. Bachelo hesis. Uni e si y Bonn, 2014.
[126] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. “Deep Lea ning o 3D
Poin Clouds: A Su ey”. In: IEEE T ansac ions on Pa e n Analysis and Machine
In elligence 43.12 (2019), pp. 4338–4364. DOI: 10.1109/ pami.2020.3005434.
[127] A. K izhe sky, I. Su ske e , and G. E. Hin on. “ImageNe Classi ica ion wi h Deep
Con olu ional Neu al Ne wo ks”. In: Communica ions o he ACM 60.6 (2017), pp. 84–
90. DOI: 10.1145/3065386.
[128] O. Ronnebe ge , P. Fische , and T. B ox. “U-Ne : Con olu ional Ne wo ks o Biomed-
ical Image Segmen a ion”. In: In e na ional Con e ence on Medical Image Compu -
ing and Compu e -Assis ed In e en ion. Sp inge In e na ional Publishing, 2015,
pp. 234–241. DOI: 10.1007/978-3-319-24574-4_28.
XI
Bibliog aphy
[129] C. Olah. Blog: Unde s anding LSTM Ne wo ks. 2015. URL: h ps://colah.gi hub.io/
pos s/2015-08-Unde s anding-LSTMs/ ( isi ed on 03/29/2023).
[130] D. Passos and P. Mish a. “A Tu o ial on Au oma ic Hype pa ame e Tuning o Deep
Spec al Modelling o Reg ession and Classi ica ion Tasks”. In: Chemome ics and
In elligen Labo a o y Sys ems 223 (2022), p. 104520. ISSN: 0169-7439. DOI: 10.
1016/j.chemolab.2022.104520.
[131] F. Kau mann, C. Glock, and T. Tschicka d . “ScaleBIM: In oducing a Scalable Mod-
ula F amewo k o T ans e Poin Clouds in o Seman ically Rich Building In o ma ion
Models”. In: P oceedings o he 2022 Eu opean Con e ence on Compu ing in Con-
s uc ion. Uni e si y o Tu in, 2022. DOI: 10.35490/ec3.2022.194.
[132] T. Hackel. “La ge-scale Machine Lea ning o Poin Cloud P ocessing”. PhD hesis.
ETH Zü ich, Ins i u e o Geodesy and Pho og amme y, 2018. DOI: 10.3929/e hz-b-
000264691.
[133] W. Zimme , A. Rangesh, and M. T i edi. “3D BAT: A Semi-Au oma ic, Web-based 3D
Anno a ion Toolbox o Full-Su ound, Mul i-Modal Da a S eams”. In: IEEE In elligen
Vehicles Symposium. IEEE, 2019, pp. 1816–1821. DOI: 10.1109/i s.2019.8814071.
[134] Au odesk-Recap. You ube Channel. 2021. URL: h p://h ps://www.you ube.com/
use /au odesk ecap/ ( isi ed on 06/24/2021).
[135] Poin Cab. Punk wolken So wa e ü alle Bedü nisse. 2023. URL: h ps://poin cab-
so wa e.com ( isi ed on 01/23/2023).
[136] J. Simon. Blog: New - Label 3D Poin Clouds wi h Amazon SageMake G ound T u h.
Ed. by A. SageMake . 2020. URL: h ps://aws.amazon.com/de/blogs/aws/new-label-
3d-poin -clouds-wi h-amazon-sagemake -g ound- u h/ ( isi ed on 01/23/2023).
[137] basicAI. AI T aining Da a Solu ions o All Indus ies. 2023. URL: h ps://www.basic.
ai/se ices ( isi ed on 01/23/2023).
[138] scale. Be e Da a, 2023. URL: h ps://scale.com/ ( isi ed on 01/23/2023).
[139] R. Rich e . Wie unse e Pla om unk ionie . Ed. by P. C. Technology. 2023. URL:
h ps://www.poin cloud echnology.com/de/#loesung ( isi ed on 01/23/2023).
[140] R. Monica, J. Aleo i, M. Zillich, and M. Vincze. “Mul i-label Poin Cloud Anno a ion by
Selec ion o Spa se Con ol Poin s”. In: In e na ional Con e ence on 3D Vision. IEEE,
2017. DOI: 10.1109/3d .2017.00042.
[141] P. Cignoni, M. Callie i, M. Co sini, M. Dellepiane, F. Gano elli, and G. Ranzuglia.
“MeshLab: An Open-Sou ce Mesh P ocessing Tool”. In: Eu og aphics I alian Chap e
Con e ence (2008).
[142] M. Vei and A. Capobianco. “Go Then Tag: A 3-D Poin Cloud Anno a ion Technique”.
In: IEEE Symposium on 3D Use In e aces. IEEE, 2014. DOI: 10.1109/3dui.2014.
6798886.
XII
Bibliog aphy
[143] S. Kulka ni, M. Chand asheka aiah, and S. Raghunandan. “3D Anno a ion Tool Using
LiDAR”. In: IEEE Global Con e ence o Ad ancemen in Technology. Bangalo e, In-
dia: IEEE, 2019, pp. 1–4. ISBN: 978-1-7281-3695-0. DOI: 10.1109/gca 47503.2019.
8978301.
[144] B. Wang, V. Wu, B. Wu, and K. Keu ze . “LATTE: Accele a ing LiDAR Poin Cloud An-
no a ion ia Senso Fusion, One-Click Anno a ion, and T acking”. In: IEEE In elligen
T anspo a ion Sys ems Con e ence. IEEE, 2019. DOI: 10.1109/i sc.2019.8916980.
[145] H. A. A ie , M. A ie , G. Zhang, Z. Liu, M. Bha , U. G. Indahl, H. T ei e, and D. Zhao.
“SAnE: Sma Anno a ion and E alua ion Tools o Poin Cloud Da a”. In: IEEE Access
8 (2020), pp. 131848–131858. DOI: 10.1109/access.2020.3009914.
[146] D. Co ey, N. Malb aa en, T. B. Le, I. Bo azjani, F. So i opoulos, A. G. E dman, and
D. F. Kee e. “In e ac i e Slice WIM: Na iga ing and In e oga ing Volume Da a Se s
Using a Mul isu ace, Mul i ouch VR In e ace”. In: IEEE T ansac ions on Visualiza ion
and Compu e G aphics 18.10 (2012), pp. 1614–1626. DOI: 10.1109/ cg.2011.283.
[147] N. O’Mahony, S. Campbell, A. Ca alho, L. K palko a, D. Rio dan, and J. Walsh.
“Poin Cloud Anno a ion Me hods o 3D Deep Lea ning”. In: 13 h In e na ional Con-
e ence on Sensing Technology. IEEE, 2019. DOI: 10.1109/ics 46873.2019.9047730.
[148] F. Wi h, J. Quehl, J. O a, and C. S ille . “Poin A Me: E icien 3D Poin Cloud Labeling
in Vi ual Reali y”. In: IEEE In elligen Vehicles Symposium. IEEE, 2019. DOI: 10.
1109/i s.2019.8814115.
[149] S. Maneewong a ana and D. M. Moun . “Analysis o App oxima e Nea es Neighbo
Sea ching wi h Clus e ed Poin Se s”. In: A Xi (1999). DOI: 10.48550/ARXIV.CS/
9901013.
[150] J. Wilhelms and A. V. Gelde . “Oc ees Fo Fas e Isosu ace Gene a ion”. In: P o-
ceedings o he 1990 Wo kshop on Volume Visualiza ion - VVS ’90. ACM P ess, 1990.
DOI: 10.1145/99307.99321.
[151] R. B. Rusu, Z. C. Ma on, N. Blodow, A. Holzbach, and M. Bee z. “Model-Based
and Lea ned Seman ic Objec Labeling in 3D Poin Cloud Maps o Ki chen En i on-
men s”. In: IEEE/RSJ In e na ional Con e ence on In elligen Robo s and Sys ems.
IEEE, 2009. DOI: 10.1109/i os.2009.5354759.
[152] H. Bal a, J. Velagic, W. Bosschae s, G. D. Cubbe , and B. Siciliano. “Fas S a is ical
Ou lie Remo al Based Me hod o La ge 3D Poin Clouds o Ou doo En i onmen s”.
In: IFAC-Pape sOnLine 51.22 (2018), pp. 348–353. DOI: 10.1016/j.i acol.2018.11.
566.
[153] J. Zhu, J. Geh ung, R. Huang, B. Bo gmann, Z. Sun, L. Hoegne , M. Hebel, Y. Xu,
and U. S illa. “TUM-MLS-2016: An Anno a ed Mobile LiDAR Da ase o he TUM Ci y
Campus o Seman ic Poin Cloud In e p e a ion in U ban A eas”. In: Remo e Sensing
12.11 (2020), p. 1875. DOI: 10.3390/ s12111875.
XIII

Bibliog aphy
[154] CloudCompa e. 3D Poin Cloud and Mesh P ocessing So wa e Open-Sou ce
P ojec . Ve sion 2.12. 2021. URL: h p : / / www . cloudcompa e . o g/ ( isi ed on
06/24/2021).
[155] Y. Wang, Y. Sun, Z. Liu, S. E. Sa ma, M. M. B ons ein, and J. M. Solomon. “Dynamic
G aph CNN o Lea ning on Poin Clouds”. In: ACM T ansac ions on G aphics 38.5
(2019), pp. 1–12. DOI: 10.1145/3326362.
[156] K. Wada. Labelme: Image Polygonal Anno a ion wi h Py hon. 2016. DOI: 10.5281/
zenodo.5711226. URL: h ps://gi hub.com/wken a o/labelme ( isi ed on 07/15/2021).
[157] E. Gau , V. Saxena, and S. K. Singh. “Video Anno a ion Tools: A Re iew”. In: IEEE
In e na ional Con e ence on Ad ances in Compu ing, Communica ion Con ol and
Ne wo king. IEEE, 2018. DOI: 10.1109/icacccn.2018.8748669.
[158] X. Royna d, J.-E. Deschaud, and F. Goule e. “Pa is-Lille-3D: A La ge and High-
Quali y G ound-T u h U ban Poin Cloud Da ase o Au oma ic Segmen a ion and
Classi ica ion”. In: The In e na ional Jou nal o Robo ics Resea ch 37.6 (2018),
pp. 545–557. DOI: 10.1177/0278364918767506.
[159] W. Tan, N. Qin, L. Ma, Y. Li, J. Du, G. Cai, K. Yang, and J. Li. “To on o-3D: A La ge-
scale Mobile LiDAR Da ase o Seman ic Segmen a ion o U ban Roadways”. In:
Con e ence on Compu e Vision and Pa e n Recogni ion Wo kshops. IEEE, 2020.
DOI: 10.1109/c p w50498.2020.00109.
[160] C. Wang, Y. Dai, N. Elsheimy, C. Wen, G. Re sche , Z. Kang, and A. Lingua. “IS-
PRS Benchma k on mul isenso y indoo mapping and posi ioning”. In: ISPRS An-
nals o Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences V-5-2020
(2020), pp. 117–123. DOI: 10.5194/isp s-annals- -5-2020-117-2020.
[161] G. Poie , M. Seidl, M. Zeppelzaue , C. Reinbache , M. Schaich, G. Bellandi, A. Ma -
e a, and H. Bischo . “Pe oSu 3D – A Da ase o high- esolu ion 3D Su ace Seg-
men a ion”. In: P oceedings o he 15 h In e na ional Wo kshop on Con en -Based
Mul imedia Indexing. ACM, 2017. DOI: 10.1145/3095713.3095719.
[162] W. Liu, J. Sun, W. Li, T. Hu, and P. Wang. “Deep Lea ning on Poin Clouds and I s
Applica ion: A Su ey”. In: Senso s 19.19 (2019), p. 4188. DOI: 10.3390/s19194188.
[163] B. Gao, Y. Pan, C. Li, S. Geng, and H. Zhao. “A e We Hung y o 3D LiDAR Da a o
Seman ic Segmen a ion? A Su ey o Da ase s and Me hods”. In: IEEE T ansac ions
on In elligen T anspo a ion Sys ems 23.7 (2020), pp. 6063–6081. ISSN: 1524-9050.
DOI: 10.1109/ i s.2021.3076844.
[164] A. X. Chang, T. Funkhouse , L. Guibas, P. Han ahan, Q. Huang, Z. Li, S. Sa a ese,
M. Sa a, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNe : An In o ma ion-Rich
3D Model Reposi o y. Tech. ep. S an o d Uni e si y, P ince on Uni e si y, Toyo a
Technological Ins i u e a Chicago, 2015. a Xi : 1512.03012.
XIV
Bibliog aphy
[165] J. Iqbal, R. Xu, S. Sun, and C. Li. “Simula ion o an Au onomous Mobile Robo o
LiDAR-Based In-Field Pheno yping and Na iga ion”. In: Robo ics 9.2 (2020), p. 46.
DOI: 10.3390/ obo ics9020046.
[166] L. Winiwa e , A. M. E. Pena, H. Weise , K. Ande s, J. M. Sánchez, M. Sea le, and B.
Hö le. “Vi ual Lase Scanning wi h HELIOS++: A No el Take on Ray T acing-Based
Simula ion o Topog aphic Full-Wa e o m 3D Lase Scanning”. In: Remo e Sensing
o En i onmen 269 (2021), p. 112772. DOI: 10.1016/j. se.2021.112772.
[167] V. S ojano ic, M. T app, J. Döllne , and R. Rich e . “Classi ica ion o Indoo Poin
Clouds Using Mul i iews”. In: The 24 h In e na ional Con e ence on 3D Web Tech-
nology. ACM, 2019. DOI: 10.1145/3329714.3338129.
[168] J. Geh ung, M. Hebel, M. A ens, and U. S illa. “An App oach o Ex ac Mo ing Ob-
jec s om MLS Da a Using a Volume ic Backg ound Rep esen a ion”. In: ISPRS
Annals o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences
IV-1/W1 (2017), pp. 107–114. DOI: 10.5194/isp s-annals-i -1-w1-107-2017.
[169] E. G illi, F. Menna, and F. Remondino. “A Re iew o Poin Clouds Segmen a ion and
Classi ica ion Algo i hms”. In: The In e na ional A chi es o he Pho og amme y, Re-
mo e Sensing and Spa ial In o ma ion Sciences XLII-2/W3 (2017), pp. 339–344. DOI:
10.5194/isp s-a chi es-xlii-2-w3-339-2017.
[170] P. Klinge . “Übe sich und Bewe ung on ak uellen Segmen ie ungs- und Klassi-
izie ungsme hoden on Punk wolken”. Bachlo hesis. Ha enCi y Uni e si y Ham-
bu g, 2019.
[171] P. Babahajiani, L. Fan, J.-K. Kämä äinen, and M. Gabbouj. “Comp ehensi e Au-
oma ed 3D U ban En i onmen Modelling Using Te es ial Lase Scanning Poin
Cloud”. In: IEEE Con e ence on Compu e Vision and Pa e n Recogni ion Wo k-
shops. 2016. DOI: 10.1109/c p w.2016.87.
[172] O. Hassaan, A. Shamail, Z. Bu , and M. Taj. “Poin Cloud Segmen a ion using Hie a -
chical T ee o A chi ec u al Models”. In: IEEE In e na ional Con e ence on Acous ics,
Speech and Signal P ocessing. IEEE, 2019, pp. 1582–1586. DOI: 10.1109/icassp.
2019.8682708.
[173] E. Cas illo, J. Liang, and H. Zhao. “Poin Cloud Segmen a ion and Denoising ia Con-
s ained Nonlinea Leas Squa es No mal Es ima es”. In: Ma hema ics and Visualiza-
ion. Sp inge Be lin Heidelbe g, 2012, pp. 283–299. DOI: 10.1007/978-3-642-34141-
0_13.
[174] P. J. Besl and R. C. Jain. “Segmen a ion h ough Va iable-O de Su ace Fi ing”.
In: IEEE T ansac ions on Pa e n Analysis and Machine In elligence 10.2 (1988),
pp. 167–192. DOI: 10.1109/34.3881.
[175] F. Pauling, M. Bosse, and R. Zlo . “Au oma ic Segmen a ion o 3D Lase Poin Clouds
by Ellipsoidal Region G owing”. In: Aus alasian Con e ence on Robo ics and Au-
oma ion. Sydney, Aus alia, 2009.
XV
Bibliog aphy
[176] J. S om, A. Richa dson, and E. Olson. “G aph-Based Segmen a ion o Colo ed 3D
Lase Poin Clouds”. In: IEEE/RSJ In e na ional Con e ence on In elligen Robo s and
Sys ems. 2131-2136. IEEE, 2010. DOI: 10.1109/i os.2010.5650459.
[177] A. Golo inskiy and T. Funkhouse . “Min-Cu Based Segmen a ion o Poin Clouds”. In:
12 h In e na ional Con e ence on Compu e Vision Wo kshops. IEEE, 2009, pp. 39–
46. DOI: 10.1109/icc w.2009.5457721.
[178] F. Moosmann, O. Pink, and C. S ille . “Segmen a ion o 3D Lida Da a in non- la
U ban En i onmen s using a Local Con exi y C i e ion”. In: IEEE In elligen Vehicles
Symposium. IEEE, 2009, pp. 215–220. DOI: 10.1109/i s.2009.5164280.
[179] A. Aijazi, P. Checchin, and L. T assoudaine. “Segmen a ion Based Classi ica ion o
3D U ban Poin Clouds: A Supe -Voxel Based App oach wi h E alua ion”. In: Remo e
Sensing 5.4 (2013), pp. 1624–1650. DOI: 10.3390/ s5041624.
[180] P. V. C. Hough. “Me hod and Means o Recognizing Complex Pa e ns”. US Pa en
3069654. 1962.
[181] D. Bo mann, J. Elsebe g, K. Lingemann, and A. Nüch e . “The 3d Hough T ans o m
o Plane De ec ion in Poin Clouds: A Re iew and a New Accumula o Design”. In:
3D Resea ch 2.2 (2011). ISSN: 2092-6731. DOI: 10.1007/3d es.02(2011)3.
[182] W. Shi, W. Ahmed, N. Li, W. Fan, H. Xiang, and M. Wang. “Seman ic Geome ic
Modelling o Uns uc u ed Indoo Poin Cloud”. In: ISPRS In e na ional Jou nal o
Geo-In o ma ion 8.1 (2018), p. 9. DOI: 10.3390/ijgi8010009.
[183] M. Weinmann, B. Ju zi, and C. Malle . “Fea u e Rele ance Assessmen o he Se-
man ic In e p e a ion o 3D Poin Cloud Da a”. In: ISPRS Annals o he Pho og am-
me y, Remo e Sensing and Spa ial In o ma ion Sciences II-5/W2 (2013), pp. 313–
318. DOI: 10.5194/isp sannals-ii-5-w2-313-2013.
[184] M. Weinmann, B. Ju zi, and C. Malle . “Seman ic 3D Scene In e p e a ion: A F ame-
wo k Combining Op imal Neighbo hood Size Selec ion wi h Rele an Fea u es”. In:
ISPRS Annals o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sci-
ences II-3 (2014), pp. 181–188. DOI: 10.5194/isp sannals-ii-3-181-2014.
[185] M. Weinmann, A. Schmid , C. Malle , S. Hinz, F. Ro ens eine , and B. Ju zi. “Con-
ex ual Classi ica ion o Poin Cloud Da a by Exploi ing Indi idual 3d Neigbou hoods”.
In: ISPRS Annals o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion
Sciences II-3/W4 (2015), pp. 271–278. DOI: 10.5194/isp sannals-ii-3-w4-271-2015.
[186] H. Thomas, F. Goule e, J.-E. Deschaud, B. Ma co egui, and Y. LeGall. “Seman ic
Classi ica ion o 3D Poin Clouds wi h Mul iscale Sphe ical Neighbo hoods”. In: IEEE
In e na ional Con e ence on 3D Vision. IEEE, 2018. DOI: 10.1109/3d .2018.00052.
[187] Z. La i and A. Habib. “Al e na i e Me hodologies o he Es ima ion o Local Poin
Densi y Index: Mo ing Towa ds Adap i e Lida Da a P ocessing”. In: The In e na ional
A chi es o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences
XXXIX-B3 (2012), pp. 127–132. DOI: 10.5194/isp sa chi es-xxxix-b3-127-2012.
XVI
Bibliog aphy
[188] F. Poux and R. Billen. “Voxel-based 3D Poin Cloud Seman ic Segmen a ion: Un-
supe ised Geome ic and Rela ionship Fea u ing s. Deep Lea ning Me hods”. In:
ISPRS In e na ional Jou nal o Geo-In o ma ion 8.5 (2019), p. 213. DOI: 10.3390/
ijgi8050213.
[189] F. Poux, C. Ma es, and L. Kobbel . “Unsupe ised Segmen a ion o Indoo 3D Poin
Cloud: Applica ion o Objec -Based Classi ica ion”. In: The In e na ional A chi es o
he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences XLIV-4/W1-
2020 (2020), pp. 111–118. DOI: 10.5194/isp s-a chi es-xli -4-w1-2020-111-2020.
[190] E. G illi, D. Dininno, G. Pe ucci, and F. Remondino. “F om 2D o 3D Supe ised Seg-
men a ion and Classi ica ion o Cul u al He i age Applica ions”. In: The In e na ional
A chi es o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences
XLII-2 (2018), pp. 399–406. DOI: 10.5194/isp s-a chi es-xlii-2-399-2018.
[191] S. Te uggi, E. G illi, M. Russo, F. Fassi, and F. Remondino. “A Hie a chical Machine
Lea ning App oach o Mul i-Le el and Mul i-Resolu ion 3D Poin Cloud Classi ica-
ion”. In: Remo e Sensing 12.16 (2020), p. 2598. DOI: 10.3390/ s12162598.
[192] E. G illi, F. Poux, and F. Remondino. “Unsupe ised Objec -Based Clus e ing in Sup-
po o Supe ised Poin -Based 3d Poin Cloud Classi ica ion”. In: The In e na ional
A chi es o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences
XLIII-B2-2021 (2021), pp. 471–478. DOI: 10.5194/isp s-a chi es-xliii-b2-2021-471-
2021.
[193] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin. “Cylind ical and
Asymme ical 3D Con olu ion Ne wo ks o LiDAR Segmen a ion”. In: Con e ence on
Compu e Vision and Pa e n Recogni ion. IEEE, 2020. DOI: 10.1109/c p 46437.
2021.00981.
[194] H. Radi and W. Ali. “VolMap: A Real- ime Model o Seman ic Segmen a ion o a
LiDAR su ounding iew”. In: ICML (Thi y-six h In e na ional Con e ence on Machine
Lea ning) Wo kshop on AI o Au onomous D i ing. a Xi , 2019.
[195] R. A. Rosu, P. Schü , J. Quenzel, and S. Behnke. “La iceNe : Fas Poin Cloud Seg-
men a ion Using Pe mu ohed al La ices”. In: P oceedings o Robo ics: Science and
Sys ems. Robo ics: Science and Sys ems Founda ion, 2020. DOI: 10.15607/ ss.
2020.x i.006.
[196] H.-Y. Meng, L. Gao, Y.-K. Lai, and D. Manocha. “VV-Ne : Voxel VAE Ne Wi h G oup
Con olu ions o Poin Cloud Segmen a ion”. In: In e na ional Con e ence on Com-
pu e Vision. IEEE, 2019. DOI: 10.1109/icc .2019.00859.
[197] A. K izhe sky. “One wei d ick o pa allelizing con olu ional neu al ne wo ks”. In:
a Xi (2014). DOI: 10.48550/ARXIV.1404.5997. a Xi : 1404.5997.
[198] J. Redmon, S. Di ala, R. Gi shick, and A. Fa hadi. “You Only Look Once: Uni ied,
Real-Time Objec De ec ion”. In: IEEE Con e ence on Compu e Vision and Pa e n
Recogni ion. 2015, pp. 779–788. DOI: 10.1109/c p .2016.91.
XVII
Bibliog aphy
[265] X. Wang, B. Zhou, Y. Shi, X. Chen, Q. Zhao, and K. Xu. “Shape2Mo ion: Join Analysis
o Mo ion Pa s and A ibu es om 3D Shapes”. In: Con e ence on Compu e Vision
and Pa e n Recogni ion. IEEE, 2019, pp. 8868–8876. DOI: 10.1109/c p .2019.
00908.
[266] J. S allkamp, M. Schlipsing, J. Salmen, and C. Igel. “Man s. Compu e : Benchma k-
ing Machine Lea ning Algo i hms o T a ic Sign Recogni ion”. In: Neu al ne wo ks.
Vol. 32. 32. Else ie BV, 2012, pp. 323–332. DOI: 10.1016/j.neune .2012.02.016.
[267] G. Joos. “Zu Quali ä on objek s uk u ie en Geoda en”. PhD hesis. Uni e si ä de
Bundesweh München, 2000.
[268] I. A meni, O. Sene , A. R. Zami , H. Jiang, I. B ilakis, M. Fische , and S. Sa a ese. “3D
Seman ic Pa sing o La ge-Scale Indoo Spaces”. In: Con e ence on Compu e Vision
and Pa e n Recogni ion. IEEE, 2016, pp. 1534–1543. DOI: 10.1109/c p .2016.170.
[269] M. A. Uy, Q.-H. Pham, B.-S. Hua, D. T. Nguyen, and S.-K. Yeung. “Re isi ing Poin
Cloud Classi ica ion: A New Benchma k Da ase and Classi ica ion Model on Real-
Wo ld Da a”. In: In e na ional Con e ence on Compu e Vision. IEEE, 2019. DOI: 10.
1109/icc .2019.00167.
[270] T. Yu and H. Zhu. “Hype -Pa ame e Op imiza ion: A Re iew o Algo i hms and Ap-
plica ions”. In: a Xi a Xi :2003.05689 (2020). a Xi : 2003.05689 1 [cs.LG].
[271] D. G i i hs and J. Boehm. “Weigh ed Poin Cloud Augmen a ion o Neu al Ne wo k
T aining Da a Class-Imbalance”. In: ISPRS The In e na ional A chi es o he Pho-
og amme y, Remo e Sensing and Spa ial In o ma ion Sciences XLII-2/W13 (2019),
pp. 981–987. DOI: 10.5194/isp s-a chi es-XLII-2-W13-981-2019.
[272] J. Mo el, A. Bac, and T. Kanai. “Segmen a ion o Unbalanced and In-Homogeneous
Poin Clouds and I s Applica ion o 3D Scanned T ees”. In: The Visual Compu e
36.10-12 (2020), pp. 2419–2431. DOI: 10.1007/s00371-020-01966-7.
[273] V. Kasi eddy and B. Akinci. “Assessing he Impac o 3D Poin Neighbo hood Size
Selec ion on Unsupe ised Spall Classi ica ion wi h 3D B idge Poin Clouds”. In: Ad-
anced Enginee ing In o ma ics 52 (2022), p. 101624. DOI: 10.1016/j.aei.2022.
101624.
[274] Y. Liu, B. Fan, G. Meng, J. Lu, S. Xiang, and C. Pan. “DensePoin : Lea ning Densely
Con ex ual Rep esen a ion o E icien Poin Cloud P ocessing”. In: In e na ional
Con e ence on Compu e Vision. IEEE, 2019. DOI: 10.1109/icc .2019.00534.
[275] B.-S. Hua, Q.-H. Pham, D. T. Nguyen, M.-K. T an, L.-F. Yu, and S.-K. Yeung. “Sce-
neNN: A Scene Meshes Da ase wi h aNNo a ions”. In: Fou h In e na ional Con e -
ence on 3D Vision. IEEE, 2016, pp. 92–101. DOI: 10.1109/3d .2016.18.
[276] H. Zhao, L. Jiang, C.-W. Fu, and J. Jia. “Poin Web: Enhancing Local Neighbo hood
Fea u es o Poin Cloud P ocessing”. In: Con e ence on Compu e Vision and Pa e n
Recogni ion. IEEE, 2019. DOI: 10.1109/c p .2019.00571.
XXIV

Bibliog aphy
[277] S. Ando and C. Y. Huang. “Deep O e -sampling F amewo k o Classi ying Imbal-
anced Da a”. In: Eu opean Con e ence, ECML PKDD. Sp inge In e na ional Pub-
lishing, Cham, 2017, pp. 770–785. DOI: 10.1007/978-3-319-71249-9_46.
[278] M. Joseph-Ri lin, A. Z i in, and R. Kimmel. “Momene : Fla o he Momen s in Lea n-
ing o Classi y Shapes”. In: In e na ional Con e ence on Compu e Vision Wo kshop.
IEEE, 2019. DOI: 10.1109/icc w.2019.00503.
XXV
A Pee - e iewed publica ions
A.1 Klassi izie ung on ehle ha gemessenen Punk en in
3D-Punk wolken mi Con Ne
Re e ence:
Ba ne ske, E. & S e nbe g, H. (2020): Klassi izie ung on ehle ha gemessenen Punk en
in 3D-Punk wolken mi Con Ne . In Wunde lich, T. (Ed.), Ingenieu e messung 20. Bei äge
zum 19. In e na ionalen Ingenieu e messungsku s München, 2020, He be Wichmann
Ve lag, 2020, 127-139.
Con ibu ion o Co-Au ho s:
Table 6: Con ibu ion o Pape No. 1
In ol ed in Es ima ed con ibu ion
Ideas and concep ual design 90%
Compu a ion and esul s 100%
Analysis and in e p e a ion 95%
Manusc ip , igu es and ables 100%
To al: 96%
I he eby con i m he co ec ness o he decla a ion o he con ibu ion o Eike Ba ne ske o
Pape No. 1 in Table 6:
P o . D .-Ing. Ha ald S e nbe g, Ha enCi y Uni e si ä Hambu g
XXVII
A Pee - e iewed
publica ions
A.1 Klassi izie ung on ehle ha gemessenen Punk en in
3D-Punk wolken mi Con Ne
Re e ence:
Ba ne ske, E. & S e nbe g, H. (2020):
Klassi izie ung on ehle ha gemessenen Punk en
in 3D-Punk wolken mi Con Ne . In Wunde lich, T. (Ed.), Ingenieu e messung 20. Bei äge
zum 19. In e na ionalen Ingenieu e messungsku s München, 2020, He be Wichmann
Ve lag, 2020, 127-139.
Con ibu ion o Co-Au ho s:
Table 6: Con ibu ion o Pape No. 1
In ol ed in
Es ima ed con ibu ion
Ideas
and concep ual design
90%
Compu a ion and
esul s
100%
Analysis and
in e p e a ion
95%
Manusc ip , igu es
and ables
100%
To al:
96%
I he eby
con i m he co ec ness o he decla a ion o he con ibu ion o Eike Ba ne ske o
Pape No. 1 in Table 6:
P o . D
.-Ing. Ha ald S e nbe g, Ha enCi y Uni e si ä Hambu g
XXVII
1
T. Wunde lich (H sg.), Ingenieu e messung 2020 © He be Wichmann Ve lag
Klassi izie ung on ehle ha gemessenen Punk en in
3D-Punk wolken mi Con Ne
Eike BARNEFSKE und Ha ald STERNBERG
1 Einlei ung
Punk wolken dienen als ein aches Modell ode als G undlage on Planungen, geome ischen
Analysen und komplexen Modellie ungen. Häu ig sind im e s en Sch i Teilpunk wolken
ü diese Au gaben zu e s ellen. Eine Fil e ung nach ehle ha gemessenen Punk en ode ein
modellbasie es Reduzie en de Punk wolkendich e sind häu ig die e s en Klassi izie ungen,
die au eine Punk wolke angewende we den. Hie ü we den o angig händische ode mo-
dellbasie e Ve ah en genu z . Al e na i zu modell- bzw. wissensbasie en Klassi izie-
ungs e ah en we den da enbasie e Klassi izie ungen en wickel , um die Auswe ungszei
zu eduzie en, die Klassi izie ungsquali ä bei un e schiedlichen Au nahmesys emen und
Szenen zu s eige n, sowie um die Klassi izie ung zu au oma isie en. Hie ü inden u. a. Con-
olu ional Neu ale Ne zwe ke (Con Ne ) Anwendung, die das Wissen aus den klassi izie en
Punk wolken (T ainingsda en) le nen und dieses in de Anwendungsphase nu zen, um unbe-
kann e Punk e zu klassi izie en. Die Klassi izie ungsleis ung de Con Ne wi d olglich
du ch die Ne zwe ka chi ek u und die T ainingsda en (Wissen des Algo i hmus) beein luss .
A bei en on QI ET AL. (2017A), HACKEL ET AL. (2017) u. a. zu seman ischen Klassi izie-
ung mi Con Ne ich en sich an olgendes Ziel: Es sollen die Punk e iden i izie we den,
welche bes imm e Objek e in de Au nahmeszene (z. B. Bäume ode S aßen) besch eiben.
Hie bei we den abe ehle ha gemessene Punk e, die i. d. R. nu wenige P ozen de gesam-
en Punk wolke einnehmen, nich be ach e und ein ach eine Objek klasse zugeo dne .
Diese ehle ha en Punk e e en z. B. in de Fo m on Mixed Pixel, Meh wegee ek en, ei-
nem g oßen Obe lächen auschen ode Phan ompunk en au .
De Ein luss de ehle ha gemessenen Punk e au die seman ische Klassi izie ung und eine
Klassi izie ung nach ehle ha en gemessenen Punk en is Gegens and de Un e suchungen
diese A bei . Hie ü we den händisch klassi izie e Punk wolken in un e schiedlichen Klas-
senkombina ionen ü das T aining und die E aluie ung de Klassi izie ungsleis ung e wen-
de .
Es we den die Un e schiede und Ähnlichkei en on modell- und da enbasie en Klassi ika i-
ons e ah en o ges ell (Abschni 2.1). Die Funk ionsweise on Con Ne (Abschni 2.2)
und e schiedene Con Ne -Modelle ü s uk u ie e und uns uk u ie e Punk wolken we -
den e läu e (Abschni e 2.3 und 2.4). De Ein luss bei ie un e schiedlichen Klassenkom-
bina ionen mi und ohne ehle ha en Punk en au die Klassi izie ungsleis ung wi d am Bei-
spiel de Ne zwe ka chi ek u on Poin Ne un e such (Abschni e 3.1 bis 3.3). Au bauend
au den Beobach ungen de Un e suchungen we den S a egien ü das T aining on
Con Ne zu Klassi izie ung on ehle ha gemessenen Punk en o ges ell . Zudem we den
Ideen o ges ell , um den Ein luss on ehle ha gemessenen Punk en bei Klassi izie ungen
zu minie en (Abschni 3.4).
XXVIII
2 Eike Ba ne ske und Ha ald S e nbe g
2 Klassi izie ung on Punk wolken
Punk wolken s ellen geome isch einen e ass en Raum da , lassen abe ohne wei e es Wis-
sen eine seman ische T ennung on einzelnen Objek en in de e ass en Szene nich zu. Diese
seman ische Ze legung de Punk wolke in Un e punk wolken is ein wich ige Sch i des
Auswe ungsp ozesses on Punk wolken, dami zum einen ehle ha gemessene Punk e aus
de Punk wolke en e n we den und zum ande en Objek e in eine Szene seman isch un e -
schieden we den können. Eine seman ische Un e scheidung is wich ig, da nich alle e ass-
en Objek e ü eine F ages ellung on In e esse sind. Fü die E s ellung on S ad modellen
sind z. B. Bauwe ke on In e esse, Fah zeuge hingegen we den hie als s ö ende Objek e
dekla ie . Soll die Punk wolke ü die Analyse des Ve keh s aums, z. B. bei de Ve keh s-
planung ode de Na iga ion, e wende we den, sind Punk e, die Objek e au den Ve keh s-
wegen (Au os, Fußgänge und Rad ah e ) e assen, on o nehmlichem In e esse.
Die T ennung on Punk en, die Objek e besch eiben und Punk en, die au g und on Mess-
ehle n en s anden sind, wi d i. d. R mi Fil e n du chge üh . Diese Fil e nu zen allgemeine
ode senso spezi ische Modelle zu Un e scheidung, ob ein Punk zu einem Objek gehö
ode au g und eines Mess ehle s en s anden is . Zudem we den die Fil e zum Homogenisie-
en de Punk wolkendich e und zu Auswahl on Punk wolkenabschni en eingese z . Fil e
haben den Nach eil, dass iel Wissen übe die Punk wolke o handen sein muss und dieses
ü jeden möglichen Fall angewende we den muss.
In einem nach olgenden Sch i kommen i. d. R. ande e Modelle zu au oma ischen seman i-
schen T ennung de Punk wolken nach Objek klassen zum Einsa z. Beispielha we den
diese modellbasie en Ve ah en o ges ell . Im Gegensa z zu den modellbasie en Klassi i-
zie ungs e ah en we den e meh da enbasie e Klassie ungs e ah en en wickel , die die
T ennung de Punk wolken nich au g und on o gegebenem Wissen, sonde n on e le n-
em Wissen ü die seman ische T ennung de Punk wolken du ch üh en. Den populä s en
Ansa z s ellen zu zei Con Ne da , dessen Einsa z an einigen Beispielen o ges ell wi d.
2.1 Modellbasie e Segmen ie ung und Klassi izie ung
Die Klassi izie ung on Punk wolken basie au den ie zen alen A bei ssch i en, und
zwa (1) de De ek ion on Me kmalen, (2) dem So ie en de Punk e nach diesen Me kma-
len, (3) dem Fes legen on G enzen, die die G uppen (Segmen e) mi ähnlichen Me kmalen
oneinande un e scheiden, und (4) dem Zuweisen eines Klassennamens an alle Segmen e
mi gleichen bzw. ähnlichen Me kmalen. Die A bei ssch i e 1 bis 3 we den als Segmen ie-
ung bezeichne , au die eine Klassi ika ion olgen kann. Bei modellbasie en Ve ah en kön-
nen diese A bei ssch i e i. d. R. eindeu ig on dem de Klassi ika ion un e schieden we den.
Dies is bei da enbasie en Ve ah en zunehmend nich meh möglich, da die Gene ie ung
de Segmen e au g und on Klassenme kmalen in einem Sch i e olg . Eine modellbasie e
Auswe ung on Punk wolken hingegen is in e schiedenen Auswe es u en gu zu un e ei-
len. In jede S u e wi d nach einem bes imm en und besch eibba en Me kmal in de Punk -
wolke gesuch . Punk e, die das gesuch e Me kmal mi ähnliche Ausp ägung agen, we den
als Segmen e ode Klassen zusammenge ass . Ein Beispiel hie ü is die Un e eilung on
Punk wolken in zwei ode meh e e Segmen e in Abhängigkei on de Dis anz zum Au nah-
mes ando (Me kmal is hie die Dis anz). Die e s e Auswe ungss u e is häu ig das „Fil-
e n“ on ehle ha en Punk en, de en Au e en in de Punk wolke zum Teil besch ieben
we den kann. Diese S u e olgen e schiedene wei e e S u en, in denen Segmen ie ung- und
XXIX

Klassi izie ung on ehle ha gemessenen Punk en in 3D-Punk wolken mi Con Ne 3
Klassi ika ions e ah en mi dem Ziel de Gene ie ung on Objek klassen angewende we -
den.
Die g aphbasie e Segmen ie ung on Punk wolken is ein wei e b ei e es Ve ah en, an-
hand dessen die A bei ssch i e 1 bis 3 de Segmen ie ung gu nachzu ollziehen sind. Die
einzelnen Punk e de Punk wolke s ellen die Kno en des G aphen da , die du ch Kan en mi -
einande e bunden sind. Jede Kan e e häl , au g und de Me kmalsun e schiede zwischen
den Punk en, ein ode meh e e Gewich e, die gemessen ode be echne we den (1).
STORM ET AL., (2010) nu zen z. B. Fa bin o ma ionen (RGB-We e), euklidische Dis anzen
und die Rich ung de Punk no malen, die übe ein lokales Ne z be echne we den. Wei e e
Me kmale, die Lase scanne messen und ü die Un e scheidung on Objek en einen Meh -
we da s ellen, sind die In ensi ä ode die Rückkeh eihen olge des emp angenen Signals.
Bei de g aphbasie en Segmen ie ung we den die Kan en mi den dazugehö igen Punk en
anhand de Gewich e eines Me kmales, i. d. R. abs eigend, so ie (2) und ein S a g enzwe
ü jedes Gewich wi d es geleg (3). In einem i e a i en P ozess we den Punk e einem Seg-
men zugeo dne , Segmen e zusammenge ass ode neue Segmen e e s ell . Die En schei-
dung, ob Segmen e zusammenge ass ode neue Segmen e gebilde we den, wi d du ch einen
G enzwe ode du ch alle G enzwe e bes imm (FELZENSZWALB & HUTTENLOCHER, 2004).
E wei e ungen des Algo i hmus sehen ein dynamisches Anpassen de G enzwe e o , um
op imale und de aillie e Segmen e zu be echnen. Dieses Ve ah en wi d häu ig um Voxel-
Gi e e wei e (wie bei AIJAZI ET AL., 2013), da bei uns uk u ie en Punk wolken du ch ein
es es ode ein dynamisches Gi e die Auswe ung e ein ach und beschleunig we den
kann. Die g aphbasie e Segmen ie ung kann au e schiedenen Obe lächen, wie e masch-
en Punk wolken, Voxel-Gi e n ode Obe lächenmodellen, e olgen. Bei de E s ellung die-
se Modelle e olg imme eine Gene alisie ung de Messwe e, so dass eine punk scha e
Segmen ie ung, wie sie ü die Klassi ika ion on Mess ehle n no wendig wä e, nich meh
möglich is .
2.2 Klassi izie ung mi Con Ne
Con Ne s we den ü die de aillie e und au oma ische Klassi ika ion on Bilde n eingese z ,
um die Inhal e de Bilde au oma isch zu en schlüsseln und diese nach seman ischen Aspek-
en zu clus e n (GIRSHICK ET AL., 2014, GIRSHICK, 2015, REN ET AL., 2016). In digi alen Bil-
de n sind die Me kmale, die ü die Klassi ika ion on Objek en e wende we den, in gleich-
mäßigen und gleich g oßen Ras e n (Pixel) angeo dne . Diese Ano dnung de Me kmale und
die scha e Abg enzung de Me kmale bei gleichzei ige lückenlose Ve ügba kei e mögli-
chen ein so o iges und e izien es Ve a bei en de Bilde mi Ve ah en de Ma izen ech-
nungen. Mi els de Me kmale in den Eingabebilde n und dessen Nachba scha , we den neue
mul idimensionale Me kmale in eine Con olu ional-Schich ex ahie . Neue Me kmale
we den du ch das Mul iplizie en de In o ma ion mi es en Gewich en, die in einem ein-
ode meh dimensionalen Fil e angeo dne sind, bes imm . Die G öße des Fil e s und die Ge-
wich e, we den ü die Klassi izie ungsau gabe so ausgewähl , dass eindeu ige Me kmale
bes imm we den können (Abb. 1).
XXX
4 Eike Ba ne ske und Ha ald S e nbe g
Abb. 1: Funk ion eine Con olu ional-Schich am Beispiel eine 5 x 5 Eingabe und eines 3 x 3 Fil e s
ohne Aus üllen des Fil e s. Im Anschluss an diese Schich können wei e e Con olu ional- ode Pooling-
Schich en olgen.
Lieg ü jeden Me kmals äge meh als ein Me kmal o (dieses is z. B. de Fall, wenn ein
Bild aus d ei Fa bkanälen bes eh ), dann wi d de Fil e au jeden In o ma ionskanal ange-
wende und die Summe de neuen Me kmale je T äge bes imm . Die Me kmale aus eine
ode meh e en Con olu ion-Schich en we den du ch das Pooling agg egie . Hie bei wi d ein
wei e es Ras e übe eine es e Anzahl an Me kmals äge n geleg und die Me kmale we den
zu einem We in eine Ras e zelle zusammenge ass . Hie ü wi d meis de g öß e, de
kleins e ode de mi le e Me kmalswe e wende . Die eigen liche Klassi izie ung wi d
du ch ein „no males“, küns liches neu ales Ne z (KNN) du chge üh , in dem alle Me kmale
de le z en Con olu ion-Pooling-Schich mi den möglichen Ne zausgaben (Klassen) e -
knüp we den, so dass ein Vek o au ges ell wi d, de ü jede Klasse eine Ne zausgabe
ausgib . Funk ionen wie So max, die au dem Vek o angewende we den, e möglichen das
Bes immen de wah scheinlichs en Klasse ü jeden Me kmals äge bzw. jedes Pixel
(SZE ET AL., 2017).
2.3 Con Ne ü Punk wolkenklassi ika ionen mi Gi e s uk u en
Punk wolken sind i. d. R. unso ie , weisen egional un e schiedliche Punk dich en und eine
un egelmäßige Ve eilung de Me kmale au , so dass iele Punk wolkenklassi izie ungs e -
ah en einen Zwischensch i benö igen. Diese Zwischensch i ha das Ziel, die Punk wol-
ken in eine S uk u zu übe üh en, die Pixel ode Voxel nu z . Hie ü we den die Punk wol-
ken in ande e Räume p ojizie und In o ma ionen zusammenge ass . De ails gehen du ch
diese Vo e a bei ung e lo en und ehle ha e Punk e, die nu e einzel au e en, we den
bei de spä e en Klassi izie ung älschliche weise eine ande en Objek g uppe zugeo dne .
Anwendungen, bei denen g öße e einzelne Objek e in de Punk wolke wäh end de Au -
nahme zu klassi izie en bzw. du ch eine Bounding Box zu ma kie en sind, sind ak uell nu
du ch eine Gene alisie ung de Punk wolke o de Klassi izie ung möglich. PIXOR
(YANG ET AL., 2018) is ein Con Ne ü die Klassi izie ung on Fah zeugen und de en Be-
wegungs ich ung in d eidimensionalen Punk wolken. Hie ü wi d eine Gene alisie ung
du ch das E zeugen eine Vogelpe spek i enansich du chge üh . Diese 2D-Ansich wi d in
Voxel / Pixel un e eil . Au diese o e a bei e en Punk wolke können die 2D-Con Ne
angewende we den.
XXXI
Klassi izie ung on ehle ha gemessenen Punk en in 3D-Punk wolken mi Con Ne 5
Die Voxel-S uk u wi d in eine Vielzahl on A bei en als G undlage ü ein occupancy g id
e wende . Ein occupancy g id is eine Ras e s uk u , in de die Zellen dem Zus and „ o -
handen sein“ ode „nich o handenen sein“ on Punk en zugeo dne we den. Du ch dieses
Ras e we den die Punk wolken abs ahie . Ziel de Ve wendung des occupancy g ids is es,
eine Punk wolke e izien in die 26 Klassen des Sydney U ban Objec s Da ase (DEUGE ET.
AL., 2013) zu un e eilen. Bei VoxNe (MATURANA & SCHERER, 2015) wi d die Punk wolke
in quad a ische Voxel-Segmen e un e eil . Jedes Voxel-Segmen wi d wiede um in 32³ Sub-
oxel un e eil , ü die ein We ü den Bese zungszus and (z. B. binä e ode als punk -
dich e We ) be echne wi d. Jedes Voxel-Segmen is ein Tenso aus 32 x 32 x 32 Ein ägen.
Diese Tenso wi d an ein 3D-Con Ne übe geben und ü das Voxel-Segmen wi d eine
Klasse bes imm , de alle Punk e, die in dieses Segmen allen, zugeo dne we den.
HACKEL ET AL. (2017) olgen diesem Ve ah en, be echnen abe um jeden Punk de Punk -
wolke ein 16 x 16 x 16 g oßes Voxel-Gi e bei ün un e schiedlich g oßen Kan enlängen
( on 2,5 bis 40 cm). Fü jeden de Voxel wi d ein Bese zungszus and be echne , so dass ein
5 x 16 x 16 x 16 Tenso en s eh , de die geome ische Nachba scha des Punk es besch eib .
Die Me kmale dieses Tenso s ü jeden Punk we den mi einem Con Ne in Anlehnung an
das Con Ne VGG on SIMONYAN & ZISSERMAN (2014) e a bei e , so dass ü jeden Punk
die Klassi izie ung du ch den So max-Laye (Klassi izie ungs unk ion) e olg . Diese An-
sa z, de punk o ien ie en Klassi izie ung on komplexen d eidimensionalen Punk wolken,
wi d bei Poin Ne und dessen E wei e ungen wei e e olg .
2.4 Poin Ne und E wei e ungen
Das Poin Ne (QI ET AL. 2017A) in seine G und o m bes eh aus eine Eingabeschich , in de
eine olls ändige, kleine Punk wolke (2000 bis 4000 Punk e) ode ein Punk wolkensegmen
(Ausschni eine g oßen Punk wolke) als Tenso e a bei e wi d. Die Punk wolke bzw. de
Tenso bes eh mindes ens aus den Punk en (n) mi Koo dina en ipel (obliga o isch) und den
op ionalen Me kmalen (m), wie No malen-Vek o en de Punk e, RGB- ode In ensi ä swe -
en. Die We e des eingelesenen Tenso s we den du ch ein T-Ne , eine Con Ne ü eine
S a kö pe ans o ma ion, in den Schwe punk des Punk wolkensegmen s ans o mie .
Diese T ans o ma ion kann sowohl au Me kmale als auch au Koo dina en angewende we -
den. Nach de T ans o ma ion sind die Ve a bei ungssch i e des Poin Ne , die hochdimen-
sionale Me kmalsex ak ion, die So ie ung und das Zusammen assen on Me kmalen, so
dass Punk e au g und de Me kmale eine Klasse zugeo dne we den können. Die Funk ions-
weise on Poin Ne en sp ich dabei zweie e ke e e Funk ionen. Die inne e Funk ion ex-
ahie die Me kmale au G undlage de Me kmale de o he igen Schich en. Dieses wi d
du ch Mul ilaye Pe cep on (MLP) e eich . MLP sind meh Schich en on e ke en No en
eines KNN. Die äuße e Funk ion is die so ie ende bzw. agg egie ende Funk ion, die Me k-
male zusammen ass . Diese wi d du ch eine Max-Pooling–Schich umgese z . Fü die Seg-
men ie ung bzw. punk weise Klassi ika ion we den lokale und globale Me kmale mi einan-
de kombinie . D. h. ein neue Tenso mi den Dimensionen (n x mlokal + mgobal), de aus den
agg egie en Me kmalen und den lokalen Me kmalen jedes Punk es bes eh , wi d e s ell .
Aus diesem Tenso we den wiede neue Me kmale je Punk ex ahie und agg egie . Fü
jeden Punk we den Me kmale du ch die MLP zusammenge ass , so dass eine Klassi izie-
ung, in k o gegebenen Klassen, e olgen kann. Diese Klassi izie ung e olg au g und des
höchs en We es des Klassen ek o s jedes Punk es (Abb. 2).
XXXII
6 Eike Ba ne ske und Ha ald S e nbe g
Abb. 2: Ve ein ach e Da s ellung des Poin Ne Ve ah ens zu Segmen ie ung und Klassi izie ung on
Punk wolken in Anlehnung an QI ET AL., (2017A). Mul ilaye Pe cep on (MLP) we den ü die Ex-
ak ion on Me kmalen e wende , die du ch eine Max Pooling Funk ion zusammenge ass we den.
Die Conca -Funk ion kombinie Tenso en. Fü jeden Punk wi d de höchs e Ausgabewe aus den
o gegebenen Klassen bes imm und so klassi izie .
Poin Ne in diese G und o m kann nu Me kmale nu zen, die im Punk wolkensegmen o -
handen sind. Bei g oßen und un e schiedlich dich en Punk wolken üh dies zu ehle ha en
Klassi ika ionse gebnissen. QI ET AL., (2017B) nu zen Poin Ne als ein Baus ein, üh en abe
eine S uk u on un e schiedlichen Schich en ein, in denen eine g oße Punk wolke sch i -
weise e kleine wi d (Poin Ne ++). Aus de Punk wolke we den Punk e mi dem a hes -
poin -sample ( ps) -Algo i hmus ausgewähl , die das Zen um eine Region bilden. Die
Punk e, die zu diese Region g uppie we den, we den übe einen es en Radius ausgewähl ,
wodu ch die G öße de Region imme kons an is , abe die Anzahl de Punk e a iie . Fü
jede Region wi d au G undlage de Me kmale ein egionale Me kmals ek o du ch Poin -
Ne be echne , so dass ü jede Region ein neue Me kmals ek o en s eh . Aus allen neuen
Me kmals ek o en we den in gleiche Weise in de olgenden Schich neue Me kmals ek o-
en be echne . Wenn ein bes imm es Abs ak ionsle el ü die Punk wolke, bzw. nun die
Me kmale e eich is , dann we den diese Me kmale in de Segmen ie ungsphase wiede en -
schlüssel . Schich weise we den die Me kmale an die Zen alpunk e übe agen. Die Me k-
male we den du ch eine In e pola ion an die benachba en Punk e in de jeweiligen Schich
übe agen, so dass alle Punk e einen Me kmals ek o mi ih en Me kmalen de zwei o he-
igen Schich en haben. Mi els eines Poin Ne Baus eins we den ü jeden Punk aus diesen
Me kmalen neue Me kmale agg egie . In de le z en Segmen ie ungsschich ha jede Punk
ein Se an Me kmalen, welches ü die Klassi izie ung jedes Punk es e wende wi d. Hie bei
lieg die Annahme o , dass sich Me kmale gleichmäßig ausb ei en, was abe bei Punk wol-
ken mi he e ogenen Objek o kommen nich zwangsläu ig is . Poin Ne ++ kann um Be-
echnungssch i e, die die un e schiedliche Punk dich e be ücksich igen, e wei e we den.
ENGELMANN ET AL., (2017) ad essie en eben alls das Poin Ne P oblem, dass keine Me k-
male auße halb eines Punk segmen es ge eil we den und op imie en die Klassi ika ionsleis-
ung du ch das Teilen on Me kmalen zwischen benachba en Punk segmen en. Dieses en -
sp ich de na ü lichen Me kmalsausb ei ung in Punk wolken mi eine Vielzahl on un e -
schiedlichen Objek en. In diese E wei e ung we den zwei P ozesske en o ges ell , die zum
einen die Eingabeschich und zum ande en die Agg ega ion on Me kmalen e schiedene
XXXIII
Klassi izie ung on ehle ha gemessenen Punk en in 3D-Punk wolken mi Con Ne 13
BARNEFSKE, E. & STERNBERG, H. (2019): PCCT: A Poin Cloud Classi ica ion Tool To
C ea e 3D T aining Da a To Adjus And De elop 3D Con Ne . ISPRS Annals o he
Pho og amme y, Remo e Sensing and Spa ial In o ma ion Sciences, 4216, 35-40.
DEUGE, M. D., QUADROS, A., HUNG, C., DOUILLARD, B. (2013): Unsupe ised Fea u e
Lea ning o Classi ica ion o Ou doo 3D Scans. Aus alasian Con e ence on Robo ics
and Au oma ion.
ENGELMANN, F., KONTOGIANNIA, T., HERMANS, A. & LEIBE, B. (2017): Explo ing Spa ial
Con ex o 3D Seman ic Segmen a ion o Poin Clouds. P oceedings o he IEEE Con-
e ence on Compu e Vision and Pa e n Recogni ion, 716-124.
FELZENSZWALB, P. F. & HUTTENLOCHER, D. P. (2004): E icien G aph-based Image Seg-
men a ion. In e na ional Jou nal o Compu e Vision, 59, 2, 167–181.
GIRSHICK, R. (2015): Fas R-CNN. P oceedings o he IEEE Con e ence on Compu e Vi-
sion and Pa e n Recogni ion, 1440–1448.
GIRSHICK, R., DONAHUE J., DARRELL, T. & MALIK, J. (2014): Rich Fea u e Hie a chies o
Accu a e Objec De ec ion and Seman ic Segmen a ion. P oceedings o he IEEE Con-
e ence on Compu e Vision and Pa e n Recogni ion, 580–587.
GOODFELLOW, I., BENGIO, Y. & COURVILLE, A. (2016): Deep Lea ning. The MIT P ess.
HACKEL, T., SAVINOV, N., LADICKY, L., WEGNER, J.D., SCHINDLER, K. & POLLEFEYS, M.
(2017): SEMANTIC3D.NET: A New La ge-scale Poin Cloud Classi ica ion Bench-
ma k. ISPRS Annals o he Pho og amme y, Remo e Sensing and Spa ial In o ma ion
Sciences, 91–98.
MATURANA, D. & SCHERER, S. (2015): VoxNe : A 3D Con olu ional Neu al Ne wo k o
Real-Time Objec Recogni ion. P oceedings o IEEE In e na ional Con e ence on In el-
ligen Robo s and Sys ems, 922–928.
QI, C. R., SU, H., MO, K. & GUIBAS, L. J. (2017A): Poin Ne : Deep Lea ning on Poin Se s
o 3D Classi ica ion and Segmen a ion. P oceedings o he IEEE Con e ence on Com-
pu e Vision and Pa e n Recogni ion, 77-85.
QI, C. R.; YI, L.; SU, H. & GUIBAS, L. J. (2017B): Poin Ne ++: Deep Hie a chical Fea u e
Lea ning on Poin Se s in a Me ic Space. Ad ances in Neu al In o ma ion P ocessing
Sys ems, 5099-5108
REN, S., HE, K., GIRSHICK, R. & SUN, J (2016): Fas e R-CNN: Towa ds Real- ime Objec
De ec ion wi h Region P oposal Ne wo ks. Ad ances in Neu al In o ma ion P ocessing
Sys ems, 91–99.
SIMONYAN, K. & ZISSERMAN, A. (2014): Ve y Deep Con olu ional Ne wo ks o La ge-
scale Image Recogni ion, P oceedings o he ICLR, 1409-1556.
STROM, J., RICHARDSON, A. & OLSON, E. (2010): G aph-based Segmen a ion o Colo ed
3D Lase Poin Clouds. P oceedings o IEEE In e na ional Con e ence on In elligen
Robo s and Sys ems, 2131-2136.
SZE, V., CHEN, Y.-H., YANG, T.-J. & EMER, J.S. (2017): E icien P ocessing o Deep Neu-
al Ne wo ks: A u o ial and su ey. P oceedings o he IEEE, 105, 12, 2295-2329.
YANG, B., LUO, W. & URTASUN, R. (2018): PIXOR: Real-Time 3D Objec De ec ion om
Poin Clouds. P oceedings o he IEEE Con e ence on Compu e Vision and Pa e n
Recogni ion, 7652–7660.
XL

A. Pee - e iewed publica ions
A.2 E alua ing he Quali y o Seman ic Segmen ed 3D Poin Clouds
Re e ence:
Ba ne ske, E.& S e nbe g, H. (2022): E alua ing he Quali y o Seman ic Segmen ed 3D
Poin Clouds. Remo e Sensing, 14, 446. DOI: 10.3390/ s14030446
G aphical Abs ac :
Figu e 42: G aphical Abs ac : E alua ion o seman ic segmen a ion me hods using he qual-
i y model.
Con ibu ion o Co-Au ho s:
Table 7: Con ibu ion o Pape No. 2
In ol ed in Es ima ed con ibu ion
Ideas and concep ual design 90%
Compu a ion and esul s 100%
Analysis and in e p e a ion 95%
Manusc ip , igu es and ables 100%
To al: 96%
I he eby con i m he co ec ness o he decla a ion o he con ibu ion o Eike Ba ne ske o
Pape 2 in Table 7:
P o . D .-Ing. Ha ald S e nbe g, Ha enCi y Uni e si ä Hambu g
XLI
A. Pee
- e iewed publica ions
A. Pee - e iewed publica ionsA. Pee
A.2 E alua ing
he Quali y o Seman ic Segmen ed 3D Poin Clouds
Re e ence:
Ba ne ske, E.& S e nbe g, H. (2022):
E alua ing he Quali y o Seman ic Segmen ed 3D
Poin Clouds. Remo e Sensing, 14, 446. DOI: 10.3390/ s14030446
G aphical Abs ac :
Figu e 42:
G aphical Abs ac : E alua ion o seman ic segmen a ion me hods using he qual-
i y model.
Con ibu ion o Co-Au ho s:
Table 7: Con ibu ion o Pape No. 2
In ol ed in
Es ima ed con ibu ion
Ideas
and concep ual design
90%
Compu a ion and
esul s
100%
Analysis and
in e p e a ion
95%
Manusc ip , igu es
and ables
100%
To al:
96%
I he eby
con i m he co ec ness o he decla a ion o he con ibu ion o Eike Ba ne ske o
Pape 2 in Table 7:
P o . D
.-Ing. Ha ald S e nbe g, Ha enCi y Uni e si ä Hambu g
XLI


Ci a ion: Ba ne ske, E.; S e nbe g, H.
E alua ing he Quali y o Seman ic
Segmen ed 3D Poin Clouds. Remo e
Sens. 2022,14, 446. h ps://doi.o g/
10.3390/ s14030446
Academic Edi o : Sande Oude
Elbe ink
Recei ed: 20 Decembe 2021
Accep ed: 13 Janua y 2022
Published: 18 Janua y 2022
Publishe ’s No e: MDPI s ays neu al
wi h ega d o ju isdic ional claims in
published maps and ins i u ional a il-
ia ions.
Copy igh : © 2022 by he au ho s.
Licensee MDPI, Basel, Swi ze land.
This a icle is an open access a icle
dis ibu ed unde he e ms and
condi ions o he C ea i e Commons
A ibu ion (CC BY) license (h ps://
c ea i ecommons.o g/licenses/by/
4.0/).
emo e sensing
A icle
E alua ing he Quali y o Seman ic Segmen ed 3D Poin Clouds
Eike Ba ne ske * and Ha ald S e nbe g
Depa men o Hyd og aphy and Geodesy, Ha enCi y Uni e si y Hambu g, Henning-Vosche au-Pla z 1,
20457 Hambu g, Ge many; ha ald.s e nbe [email p o ec ed]
*Co espondence: eike.ba ne ske@hcu-hambu g.de
Abs ac :
Recen ly, 3D poin clouds ha e become a quasi-s anda d o digi iza ion. Poin cloud
p ocessing emains a challenge due o he complex and uns uc u ed na u e o poin clouds. Cu en ly,
mos au oma ic poin cloud segmen a ion me hods a e da a-based and gain knowledge om manually
segmen ed g ound u h (GT) poin clouds. The c ea ion o GT poin clouds by cap u ing da a wi h
an op ical senso and hen pe o ming a manual o semi-au oma ic segmen a ion is a less s udied
esea ch ield. Usually, GT poin clouds a e seman ically segmen ed only once and conside ed o be
ee o seman ic e o s. In his wo k, i is shown ha his assump ion has no o e all alidi y i he
eali y is o be ep esen ed by a seman ic poin cloud. Ou quali y model has been de eloped
o desc ibe and e alua e seman ic GT poin clouds and hei manual c ea ion p ocesses. I is
applied on ou da ase and publicly a ailable poin cloud da ase s. Fu he mo e, we belie e
ha his quali y model con ibu es o he objec i e e alua ion and compa abili y o da a-based
segmen a ion algo i hms.
Keywo ds:
3D poin cloud; quali y model; anno a ion ools; da ase s; e alua ion me ic;
e alua ion pa ame e
1. In oduc ion
A majo esea ch opic in geodesy is o digi ize ac i i ies in cons uc ion [
1
–
3
], in
building main enance [
4
,
5
] and in na iga ion [
6
,
7
]. Fo he digi iza ion o hese asks,
digi al building pa s and u nishing objec s mus be o med and p ocessed. Digi al models
o eal-wo ld buildings (digi al wins) a e needed o make complex and la ge seman ic
da a in e p e able o humans and machines [
8
]. The c ea ion o digi al wins is o en based
on 3D poin clouds, which a e e icien ly cap u ed wi h dep h imaging came as o ligh
imaging, de ec ion and anging (LIDAR) sys ems. The 3D poin cloud wi hou any seman ic
ea u es can al eady be conside ed a model, since humans can use hei knowledge o
in e p e seman ic poin g oups as single objec s. These seman ic poin g oups a e, e.g., he
objec s and scanning a i ac s, as shown in Figu e 1.
Figu e 1.
Examples o objec s (chai and able) and scanning a i ac s in a poin cloud. Common
scanning a i ac s a e: come ails, mixed pixels on edges (jump edges), mul i-pa h e ec s and
de used e lec ions.
Remo e Sens. 2022,14, 446. h ps://doi.o g/10.3390/ s14030446 h ps://www.mdpi.com/jou nal/ emo esensing
XLII
Remo e Sens. 2022,14, 446 2 o 41
Fo he digi al p ocessing o poin clouds, seman ic in o ma ion has o be gi en o
he poin cloud o o m seman ic segmen s. The ini ial seman ic segmen a ion is always
pe o med by humans. Fo his pu pose, di e en ools can be used o o m segmen s as
e icien ly, eliably, p ecisely and co ec ly as possible and o assign he co ec seman ic
label. The e iciency, eliabili y, p ecision and co ec ness o seman ic segmen a ion a e
cha ac e is ics ha desc ibe he quali y o a seman ic poin cloud. These cha ac e is ics build
he quali y model, which desc ibes how well he c ea ion o he seman ic poin cloud wo ks.
E alua ion me ics now become pa ame e s o he quali y model, which desc ibe he poin
cloud cha ac e is ics. A compa ison o di e en segmen a ions is possible wi h he quali y
pa ame e . Me hod compa isons a e common in au oma ic seman ic segmen a ion [
9
–
12
],
which ypically uses machine lea ning (ML) and a i icial in elligence (AI). Fo me hod
compa isons, poin cloud benchma ks a e used [
13
,
14
]. Seman ic poin cloud benchma ks
a e poin clouds o which a seman ic g ound u h (GT) is gi en. I is assumed ha he
GT poin clouds a e ee o seman ic and geome ic e o s. Howe e , un o una ely, in
mos cases, a comple e e alua ion o he manually o semi-au oma ically c ea ed seman ic
poin cloud benchma ks is no pe o med. The cha ac e is ics o a seman ic poin cloud
ha can be e alua ed a y s ongly among he published poin clouds. In some wo ks, he
seman ic accu acy o a poin cloud is e alua ed comple ely [
13
] o by spo checks [
14
,
15
].
O he wo ks e alua e only he comple eness and co ec ness o a building model [
16
].
E en i some cha ac e is ics o he poin cloud can be e alua ed, hen a compa ison o he
e alua ion me ic is o en no possible, since no uni o m me ics a e de ined. Fo example,
in e sec ion o e union (IoU), F1-sco e, o e all accu acy, ecall, p ecision and many o he s
a e used o alida e he accu acy. The a ie y p oblem o he e alua ion me ic o he case
o objec de ec ion in images is well known and a ool o ansla e he e alua ion me ics
o comp ession was de eloped [17].
To he bes o ou knowledge, a holis ic quali y model in which a ailabili y, in eg i y
and accu acy a e ep esen ed does no exis o seman ic poin clouds. Such a quali y
model has he po en ial o make he in es iga ion o exis ing and upcoming GT poin
cloud da ase s compa able. De ia ion om eali y, he a ailabili y o in o ma ion and
applicabili y o a ce ain pu pose can be de e mined wi h ou quali y model o indoo
poin clouds.
Fundamen al o he de elopmen o he quali y model is he de ini ion o he seman ic
segmen a ion, as well as i s sepa a ion in o de ec ion and classi ica ion (Sec ion 2.1). The
cap u e me hods o 3D poin clouds o indoo applica ions (Sec ion 2.2), he exis ing poin
cloud da ase s (Sec ion 2.3), as well as he ools o manual and semi-au oma ic seman ic
segmen a ions (Sec ion 2.4) de e mine he cha ac e is ics needed in he quali y model. The
de elopmen o he quali y model is de i ed om a p ocess desc ip ion (Sec ion 3.1), a
class de ini ion (Sec ion 3.2) and a da a model (Sec ion 3.3). The quali y cha ac e is ics
and pa ame e s a e de ined and discussed in Sec ion 3.4. The desc ip i e and e alua i e
use o he quali y model is p esen ed and discussed based on di e en poin clouds in
Sec ions 4.1 and 4.2. Finally, Sec ion 5summa izes he main conclusions and gi es an
ou look o u he de elopmen and possible use o he quali y model.
2. S a e o he A
The su aces o eal objec s a e o en ep esen ed as 3D poin clouds a e digi iza ion.
These 3D poin clouds a e an unso ed lis o coo dina es wi h addi ional (spec al)
in o ma ion. This ep esen a ion is pa icula ly well sui ed o measu ing sys ems ha
use high- equency scanning o objec su aces. Ve y e icien s o age o single poin s o
poin g oups (lines o a ays) is hus possible. This has caused he poin cloud o become a
quasi-s anda d o 3D objec ep esen a ions. The poin cloud ep esen s e y e icien ly,
accu a ely and wi h a high esolu ion he geome y o scenes and objec s. Un o una ely,
wi h poin clouds, he sepa a ion o indi idual objec s is no possible igh away. Thus, i is
a necessa y nex p ocessing s ep o de i e in o ma ion o models om poin clouds.
XLIII
Remo e Sens. 2022,14, 446 3 o 41
Cu en esea ch on he sepa a ion o poin clouds is mainly applied o au onomous
ope a ing sys ems, building modeling and compu e ision (CV) asks. Au onomous ope a ing
sys ems include au onomously d i ing ca s, whe e in o ma ion o obs acle a oidance,
ou e planning and sign ecogni ion has o be gene a ed om he 3D poin clouds [
18
,
19
].
CV and building modeling aim o en ich he poin cloud wi h seman ic in o ma ion. The
en iched poin clouds a e he basis o decision making and he c ea ion o seman ic models.
I he poin clouds ep esen complex scenes in which indi idual objec s appea se e al
imes, hen ins ancing is o en he goal. Applica ions include he modeling o digi al wins
o he c ea ion o ci y models, as well as he di ec c ea ion o simple building models based
on poin clouds and p io knowledge [20–22].
Di e en ypes o acquisi ion sys ems, segmen a ion ools and seman ic poin cloud
da ase s a e a ailable, o ming he basis o he de elopmen o au oma ic poin cloud
sepa a ion me hods. The applica ion o hese se s he quali y o a seman ic poin cloud. A
la ge amoun o seman ic aining and benchma k poin clouds a e a ailable.
2.1. Classi ica ion, Objec De ec ion and Segmen a ion
The de ini ion o classi ica ion, objec de ec ion and segmen a ion is no clea in he
li e a u e, and hese e ms a y by esea ch and applica ion ield. Di e en e ms a e
used o he same sepa a ion ask, o he meaning o he e ms may be ambiguous. Some
e iews [
23
,
24
] dis inguish be ween classi ica ion, objec de ec ion and segmen a ion. O he
esea che s [
25
] use segmen a ion as an all-encompassing e m o a ious ca ego iza ion
me hods. To a oid misunde s andings, classi ica ion, objec de ec ion as well as seman ic
and ins ance segmen a ion a e b ie ly de ined below o his wo k.
Classi ica ion:
Classi ica ion is he assignmen o a class ea u e (label) o one objec .
This can be a single poin , a poin cloud, a segmen o a poin cloud o ano he geome y
ype. Usually, seman ic labels o IDs a e assigned. The classi ica ion in he ollowing is
unde s ood as he assignmen o one seman ic label o one poin cloud segmen .
Objec de ec ion:
In objec de ec ion, speci ic objec s a e de ined based on geome ic
o spec al ea u es in he poin clouds. The indi idual objec and no he en i e poin cloud
is o in e es , so ha la ge pa s o he poin cloud a e no e alua ed in de ail. Se e al
objec s in a poin cloud can be de ec ed and a unique iden i ie is ob ained. Objec de ec ion
is o en used in conjunc ion wi h acking objec s in applica ions wi h mul iple sub-poin
clouds. The objec s a e usually oughly desc ibed in e ms o geome ic size, posi ion and
o ien a ion using bounding boxes. In o he cases, i is no he objec s as a whole ha a e o
in e es , bu only ce ain su aces o shapes [
26
]. These a e sea ched o in he poin clouds
(shape de ec ion).
Seman ic segmen a ion:
The seman ic segmen a ion has he goal o ex ending he
ea u es o he poin s by seman ic labels. Seman ic labels a e seman ic classes ha usually
desc ibe eal-wo ld objec s. The di e ence o he classi ica ion is ha he segmen s a e
o med in his p ocess s ep and a label is se o all poin s o he segmen . A seman ic
segmen can consis o se e al geome ically independen segmen s. Fo example, a
poin can belong o he class able; complemen a ily, i can belong o he subclass able leg.
Mo eo e , he esul s o he classi ica ion o each poin can o m a new segmen .
Ins ance segmen a ion:
An ins ance segmen desc ibes he geome ic shape o one
objec . Ins ances in a poin cloud can be dis inguished by a unique iden i ie . An ins ance is
usually en iched wi h seman ic in o ma ion. Poin s o he same seman ic segmen desc ibe
di e en objec s. Fo example, i wo ables a e in one poin cloud, hen bo h ca y he same
seman ic label. In o de o dis inguish he ables, ins ances mus be c ea ed. Each able is
an ins ance, which usually consis s o a geome ically connec ed poin cloud segmen .
The c ea ion o a digi al win goes beyond his idea. Fo modeling a digi al win,
new pa ame ized objec s ha e o be o med ha desc ibe he poin cloud con en by
gene aliza ions such as a simple geome y.
XLIV
Remo e Sens. 2022,14, 446 4 o 41
2.2. Cap u ed and Syn he ic Poin Clouds
Almos any seman ic 3D poin cloud is de i ed om a syn he ic su ace model o is
cap u ed by con ac less senso s. An o e iew o he me hods is gi en in Figu e 2.
Syn he ic 3D poin clouds a e mos ly gene a ed om la ge collec ions o online model
da abases, such as [
27
]. These poin clouds a e gene a ed e icien ly by ans o ming a
su ace model in o a egula o andom poin cloud. These poin s lie on he su ace o he
p e ious model o ha e syn he ic noise added. Syn he ic 3D poin clouds usually ep esen
only a single objec o a small g oup o objec s. Usually, hey a e used o algo i hm
de elopmen o p o o ype es ing [28,29].
Figu e 2. Cap u ing sys ems and basic da a o he c ea ion o 3D poin clouds.
Any acquisi ion echnique o cap u ing eali y has a ce ain esolu ion, p ecision
and co ec ness, which can be ound in he esul ing poin cloud. These poin cloud
cha ac e is ics o en depend on he su ace o he objec , he acquisi ion dis ance, he
en i onmen al condi ions and he measu emen senso s.
Op ical senso s a e he mos widely used me hod o mapping eali y. Op ical senso s
use ligh o di e en spec al bands o c ea e a 3D poin cloud o eal en i onmen s wi h
pho og amme ic me hods, as desc ibed in [
30
]. In pa icula , dep h imaging came as
and LIDAR sys ems ha e been widely used in he las decade o c ea e poin cloud
da ase s [
23
–
25
]. The easons a e use iendliness, mainly mode a e acquisi ion and
e alua ion cos s [
31
,
32
] and he e icien cap u ing o la ge a eas. In addi ion o op ical
senso s, ada is some imes used o c ea e poin clouds [33,34].
Dep h imaging came as consis o one o mo e came as o di e en spec al anges
and an ac i e emi e . Di e en p inciples o de e mining he image dep hs a e used.
Fo example, he Ma e po P o 3D Came a and he Mic oso Kinec V1 use s uc u ed ligh
(SL) [35] and he Mic oso Kinec V2 uses he ime o ligh (ToF) me hod [36].
Wi h he SL came as, a monoch ome nea -in a ed (NIR) image is cap u ed in addi ion
o a ue-colo image ( ed, g een, blue (RGB)). The scene o be cap u ed is illumina ed by a
p ojec o wi h a known NIR pa e n. The pa e n consis s o a ious b igh and da k do s
ha a e dis ibu ed in a non-co ela ing manne . The p ojec ed pa e n is dis o ed by he
geome y o he objec . The dep h is de e mined in se e al s eps and o each pixel. Fi s ,
he ho izon al displacemen o he do pa e n is de e mined based on he objec dis ance.
Based on he dis o ion, he dep h o he espec i e pixel is hen calcula ed in he nex
s ep using he equa ion o s e eo iangula ion [
37
]. Fo his pu pose, he dis o ion in he
uni o pixels, he base leng h (dis ance p ojec o –came a) and he ocal leng h in pixels
a e used. Fo each pixel, he dis o ion is de e mined using a local, e.g., 9
×
9pixel a ea,
which is compa ed wi h a se o e e ence images o di e en dep hs. The compa ison
is pe o med using c oss-co ela ion. An in e pola ion is pe o med be ween he highes
XLV

Remo e Sens. 2022,14, 446 5 o 41
co ela ion alues o inc ease up o sub-pixel esolu ion [
36
,
38
]. Fo u he in o ma ion on
he SL me hod using he Mic oso Kinec V1 as an example, he eade is e e ed o [39].
In es iga ions o he Mic oso Kinec V1 show he p ecision exp essed by he s anda d
de ia ion (SD) o 1mm a 800mm dis ance and o 11mm a a dis ance o 3000mm [
32
].
Acco ding o [
31
], he co ec ness (o se o he a ge geome y) is up o 40mm o a
cap u ed dis ance o 1600mm (wi hin a ypical wo king ange o 400 o 4000mm). E ec s
such as lying pixels (e oneous poin measu emen in a gap), colo -dependen accu acy
changes and mul ipa h o e laps a edges do no o only occu a a e y low le el [
31
].
Mo eo e , o he Ma e po P o 3D Came a, which was used o online a ailable aining
da ase s by [
40
,
41
], he co ec ness, p ecision and esolu ion ha e been in es iga ed in
di e en s udies. He e, a dis ance-dependen co ec ness o up o 80 mm o he u hes
cap u ing dis ance was also de e mined. A e a scaling ac o is elimina ed, a p ecision o
be e han 10mm SD can be de e mined o he en i e wo king ange [
35
]. A LIDAR
poin cloud was used as a e e ence o he men ioned s udy. The esolu ion o he
Ma e po P o 3D Came a is 5 (ho izon al) and 10 ( e ical) poin s pe deg ee [42].
The ToF echnique is based on measu ing he a el ime o a signal om an emi e
o e lec a an objec ’s su ace and back o a ecei e [
30
]. Pulse modula ion (PM) and
con inuous-wa e (CW) ampli ude modula ion a e he mos common ToF me hods. In mos
dep h imaging came as, such as he Mic oso Kinec V2, CW ampli ude modula ion is used.
In CW ampli ude modula ion, he objec o be cap u ed is con inuously illumina ed wi h
NIR ligh , whose ampli ude changes pe iodically. Because he signal needs a ce ain ime
be ween senso and objec , a phase shi occu s be ween he ansmi ed and ecei ed signal.
This phase shi is p opo ional o he signal p opaga ion ime. I his ime is mul iplied
wi h he known speed o ligh , he double dis ance be ween objec and senso sys em can
be de e mined. The phase di e ence is de e mined o se e al modula ed equencies
by co ela ing he ecei ed signal wi h he emi ed e e ence equencies. As long as he
maximum dis ance is smalle han 2
π
o he equency, a dis ance can be de e mined as
unique [36].
The p ecision o he Mic oso Kinec V2, as wi h he Mic oso Kinec V1, depends on he
acquisi ion dis ance and a ies be ween 1 and 3mm SD o he ypical wo king ange o
800 o3000mm [
31
,
32
]. Recen dep h imaging came as, such as he Mic oso Azu e Kinec ,
ha e a p ecision o less han 1mm o he same wo king ange (s a ic eco ding). Re . [
31
]
obse ed a cons an o se o -18 mm o he whole wo king ange o he Mic oso Kinec V2.
Sys ema ic e oneous measu emen s, such as lying pixels, colo -dependen accu acy
changes o up o 4mm, mul ipa h-e ec s a edges o up o 30mm and a high dependence o
dis ance measu emen s on empe a u e changes, a e he disad an ages o his measu emen
p inciple [
31
,
36
,
43
]. These e ec s can be conside ed o elimina ed in a la e seman ic
segmen a ion.
LIDARsys ems a e used o s a ic and kinema ic eco dings o scenes. LIDAR sys ems
emi a lase beam, which is p ojec ed on o a o a ing mi o . Th ough he o a ion, he beam
is shi ed by a ce ain inc emen . Fo each inc emen , he e ical and ho izon al di ec ions
as well as he dis ance o he su ace a e egis e ed. Toge he wi h he in ensi y alue, and
e en ually wi h u he spec al alues, he 3D poin cloud is c ea ed. Fo he dis ance
measu emen s, he e is he phase di e ence (PD) me hod, which can be used o ealize a
highe measu ing equency, and he PM me hod, which is less objec su ace-sensi i e [
30
].
PM LIDAR sys ems a e p e e ed o kinema ic scanning on mobile pla o ms. Kinema ic
lase scanning usually in ol es measu ing indi idual p o iles, which a e assembled as an
en i e poin cloud using na iga ion da a o algo i hms, as in [
44
]. Mobile LIDAR sys ems a e
mainly used o ou doo applica ions and on obo s. Medium- ange LIDAR sys ems such as
VelodyneHDL-64E a e o en used o c ea ing da ase s in esea ch p ojec s wi h a p ecision o
20mm [
45
]. High-end mobile mapping sys ems (MMS), such as he Riegl VMY-1 [
46
], allow
he su eying o la ge-scale a eas wi h a poin accu acy o 15mm a 50m dis ance and a
p ecision o 10mm. MMS such as he Na Vis M6 a e used in many s udies [47].
XLVI
Remo e Sens. 2022,14, 446 6 o 41
The cu en s a e o he echnology o indoo su eys includes e es ial LIDAR
sys ems (TLS), such as he Leica RTC 360,Z+F-Image 5016 o Fa o Fokus X 3D 330. These
sys ems p edominan ly use he PD me hod and a e used o dis ances sho e han 100m.
Labo a o y and ield in es iga ions show ha , wi h hese measu ing sys ems, 3D poin
clouds wi h p ecision o less han1mm and co ec ness o less han2mm in he nea
ield o up o 20m can be eached [
48
]. Howe e , hese alues e e o op imal s udy
ci cums ances such as ma o homogeneous su aces. In p ac ice, i has been shown o
all LIDAR sys ems ha he accu acy o he poin clouds a ies and scanning a i ac s
occu . Typical scanning a i ac s a e come ails, mixed pixels on edges and mul i-pa h
e ec s on highly e lec i e su aces, as shown in Figu e 1. O he in luencing a iables,
such as he measu emen objec , he se up and he en i onmen , as well as he condi ion
o he measu emen sys ems [
49
], mus be aken in o accoun o he de e mina ion o he
quali y o a cap u ed poin cloud [
50
–
52
]. The esolu ion, he app oxima ed accu acy, he
acquisi ion me hod and he wo king ange a e c ucial pa ame e s ha mus be known o
es ima ed o he la e seman ic segmen a ion o a poin cloud.
2.3. 3D Poin Cloud Da ase s
In a ious e iews [
23
–
25
] and in web da abases (e.g., h ps://pape swi hcode.com/
da ase s accessed on 30 No embe 2021 and h ps://www.seman icschola .o g/ on 30
No embe 2021) on poin cloud da ase s and me hods o poin cloud p ocessing, an
o e iew o mo e han 100 publicly a ailable poin cloud da ase s is gi en. These
con ibu ions summa ize in o ma ion on applica ion a eas, applied senso s, en i onmen al
ci cums ances o ile o ma s. The main goal o hese publica ions is o p o ide benchma ks
o a i hme ic e alua ions. A seman ic segmen a ion is no a ailable o all exis ing da ase s.
A selec ion o seman ic 3D poin clouds is examined in mo e de ail. The ocus will be on
he ini ial human segmen a ion and i s e alua ion. No all da ase s could be documen ed
in he same le el o de ail.
The da ase s in Table 1we e de i ed om syn he ic su ace models. All show one
objec o one known class. In some da ase s, he objec models a e subdi ided so ha hey
can be used o seman ic and ins ance segmen a ion. Since he poin clouds a e de i ed
om syn he ic models, he geome y can be conside ed ee o scanning a i ac s. Howe e ,
e o s can s ill occu du ing anno a ion and alignmen .
Table 1.
Syn he ic da ase s wi h yea o publica ion, da a sou ce, sepa a ion me hod (classi ica ion
(Cls), seman ic segmen a ion (SSeg) and ins ance segmen a ion (ISeg)), numbe o models, numbe o
classes and en i onmen .
Da ase Yea Da a Sou ce Sepa a ion
Me hod
No. o
Models
No. o
Classes En i onmen
ShapeNe [27] 2015 T imble 3D Wa eh.,
Yobi3D Cls, SSeg >220, 000 3135 In-/Ou doo
ModelNe [53] 2015 T imble 3D Wa eh.,
Yobi3D Cls, ISeg 151,128 660 In-/Ou doo
Shape2Mo ion [26] 2019 ShapeNe , T imble
3D Wa eh. ISeg Cls, SSeg 2440 45 In-/Ou doo
An e alua ion me ic o classi ica ions is in oduced by he ShapeNe da ase , which
desc ibes how accu a e o unique a classi ica ion is. Human anno a o s classi y a seman ic
model un il he classi ica ion accu acy a ies by less han 2% [
27
]. The ModelNe da ase
consis s o 3D CAD models aken om web da abases. The anno a ion is pe o med
using Ame zone Mechanical Tu k (AMT). The anno a o s classi y di e en models using a
web-based ool. Fo his, a model and a label a e p oposed. The anno a o s imp o e
he co ec ness o a label o a displayed model by yes-o -no ques ions. An e alua ion
is conduc ed by he da ase designe s o he en mos popula ca ego ies [
53
]. In he
Shape2Mo ion da ase , a seman ic segmen a ion o mo able pa s, such as wheels o ca
XLVII
Remo e Sens. 2022,14, 446 7 o 41
doo s, and hei p ope ies is pe o med. An e alua ion o he classi ica ion is ca ied ou
by simula ing he mo ion di ec ly a e he segmen a ion and classi ica ion [26].
Complex poin cloud simula ion ools, such as he HELIOS++ [
54
] o Gazebo oge he
wi h he Robo ics Ope a ion Sys em [
55
], ha e eached a high le el o de elopmen . These
ools can be used o c ea e poin clouds om su ace and CAD models ha con ain he
cha ac e is ics o speci ic senso s and sys em con igu a ions.
Indoo da ase s a e commonly cap u ed wi h dep h imaging came as. Some o
he mos popula da ase s a e summa ized in Table 2. Fo a la ge numbe o da ase s,
dep h imaging came as a e used in combina ion wi h an ini ial measu emen uni (IMU).
Toge he wi h he poses om he IMU and he images, a Simul aneous Localiza ion and
Mapping (SLAM) p ocedu e is used o compu e a mul i-dimensional ep esen a ion o he
cap u ed scene. The seman ic anno a ion occu s ei he in images, ideos, meshes o in 3D
poin clouds.
Table 2.
Indoo da ase s eco ded by dep h came as wi h yea o publica ion, senso , senso me hod,
sepa a ion me hod (classi ica ion (Cls), objec de ec ion (ObjD) and seman ic segmen a ion (SSeg)),
su ace a ea and numbe o classes.
Da ase Yea Senso Senso
Me hod
Sepa a ion
Me hod
Su ace A ea
Poin s
No. o
Classes
SceneNN [56] 2016 Kinec 2 ToF Cls, SSeg 7078 m²
1,450,748 19
S3DIS [40] 2016 Ma e po SL Cls, SSeg 6020 m² 12
ScanNe [57] 2017 Occipial (iPad) SL ObjD, SSeg 78,595 m² 17
Ma e po 3D [41] 2017 Ma e po SL Cls, SSeg 219,399 m² 40
ScanObjec NN [58] 2019 SceneNN, ScanNe ToF, SL Cls, SSeg 2.971.648 15
The S an o d La ge-Scale 3D Indoo Spaces (S3DIS) da ase is seman ically segmen ed as
a 3D poin cloud using he so wa e Cloud Compa e (CC) [
59
]. Fo he SceneNN,ScanNe
and Ma e po 3D da ase s, a mesh is he segmen a ion base. All anno a ions a e pe o med
wi h cus om ools. The SceneNN da ase is i s au oma ically segmen ed coa sely and hen
inely. The g aph-based segmen a ion algo i hm o [
60
] is adap ed and he segmen a ion is
a e wa ds imp o ed by he ope a o by sepa a ing, me ging and e- o ming he segmen s.
The seman ic anno a ion is pe o med by use s a aching labels o he segmen s [
56
,
61
]. The
seman ic segmen a ion o he ScanNe da ase is pe o med by au oma ic p e-segmen a ion
and a subsequen ine segmen a ion wi h classi ica ion using ools on AMT. In addi ion o
seman ic segmen a ion wi h meshes, CAD models a e i ed in o a mesh and a e a ailable
as a di e en da a o ma [
57
]. The Ma e po 3D da ase is seman ically segmen ed in
wo s ages and e i ied by en expe s. In he i s s age, loo plans a e de i ed using
planes p ojec ed on o he mesh. In he second s age, he meshes o indi idual ooms esp.
egions a e segmen ed acco ding o classes and ins ances using ScanNe ’s ool [
41
]. Fo he
ScanObjec NN da ase , he SceneNN and ScanNe meshes a e he basis. A selec ion om his
da ase is used and imp o ed. Segmen s a e ebuil and ca ego ies a e ha monized. A 3D
poin cloud wi h 1024 poin s is calcula ed ou o each mesh.
The e i ica ion o dep h image da ase s is mainly pe o med by expe s o he
au ho s [
41
,
58
]. Al e na i ely, he same da ase is seman ically segmen ed by di e en
people o iden i y e o anno a ions [56]. No in o ma ion is a ailable abou he alida ion
o he S3DIS da ase [40].
A selec ion o ecen seman ic 3D poin clouds gene a ed wi h LIDAR sys ems is
summa ized in Table 3. These da ase s will be used la e in he quali y model. Mos
3D poin clouds om LIDAR sys ems a e o ou doo scenes and a e cap u ed wi h
mul i-senso sys ems (MSS). Wi h MSS, he cap u ing o la ge a eas is mo e e icien
han wi h TLS. The geome ic accu acy o a ew cen ime e s, which is necessa y o he
majo i y o applica ions in geodesy and ci il enginee ing, is main ained. In addi ion o he
XLVIII
Remo e Sens. 2022,14, 446 8 o 41
LIDAR measu emen s, many MSS cap u e RGB images om he scanned scene o colo ize
he poin cloud. Fu he mo e, hese images can be used o seman ic segmen a ion.
The GT seman ic segmen a ion o he da ase s Pa is-Lille 3D,Seman ic3D,MLS1 TUM
Ci y Campus (MSL1 TUM CC), To on o3D and Complex Scene Poin Cloud (CSPC) is conduc ed
comple ely o in pa s wi h CC. Fo hese da ase s, he 3D poin cloud o ma is he basis o
da a p ocessing. This is also he case o he Seman icKITTI da ase , which is seman ically
segmen ed using a cus om o line ool [
13
]. The Building Indoo Poin Cloud (BIPC) da ase
uses he LabelMe ool [
62
] o he segmen a ion and classi ica ion o 2D images. The 2D
seman ic segmen s a e p ojec ed in o 3D space a e anno a ion. Any inco ec anno a ions
in he poin cloud a e co ec ed using ano he 3D ool [
63
]. Ano he me hod o seman ically
segmen 3D poin clouds is o i geome ies, such as planes o boxes, in o he poin cloud.
This is applied o pa s o he da ase Seman ic3D [
15
]. All poin s wi hin a ce ain dis ance
om he geome y a e selec ed. The esul ing segmen is assigned o a class.
Table 3.
LIDAR- eco ded da ase s wi h yea o publica ion, senso , senso me hod, sepa a ion me hod
(seman ic segmen a ion (SSeg) and ins ance segmen a ion (ISeg)), numbe o poin s, numbe o classes
and en i onmen .
Da ase Yea Senso Senso
Me hod
Sepa a ion
Me hod
No. o
Poin s
No. o
Classes En i onmen
Pa is Lille 3D [64] 2018 Velod.
HDL-32E MMS ca SSeg 1431 M 50
Ou doo
Seman ic3D [15] 2017 Unknown TLS TLS SSeg 4 B 8
Ou doo
Seman icKITTI [13] 2019 Velod.
HDL-64E MMS ca SSeg 4.5 B 28
Ou doo
MSL1 TUM CC [14] 2020 Velod.
HDL-64E MMS ca SSeg, ISeg 1.7M 8
Ou doo
To on o3D [65] 2020 Teled. Op .
Me . MMS ca SSeg 78.3 M 8
Ou doo
CSPC-Da ase [66] 2020 Velod. VLP-16 MMS backp. SSeg 68.3 M 6
Ou doo
BIPC-Da ase [63] 2021 Velod. VLP-16 MMS backp. SSeg - 30
Indoo
Closely ela ed o he seman ic segmen a ion is i s e alua ion. The Seman ic3D da ase
is e alua ed by class compa isons in he o e lapping a eas o he neighbo ing poin clouds.
Fo his pu pose, all poin s in he neighbo hood o an adjacen poin cloud a e selec ed
om a gi en poin wi h a sea ch adius o 50mm. The classes o he selec ed poin s a e
compa ed wi h he class o he ini ial poin [
15
]. The Seman icKITTI and he CSPC da ase s
a e e alua ed and imp o ed by expe s in a second p ocessing s ep [
13
,
66
]. Fo he BIPC
da ase , he segmen s c ea ed in 2D a e e alua ed on he 3D poin cloud [
63
]. S a is ical
e alua ion o seman ic accu acy o all da ase s is no documen ed. No in o ma ion on he
e i ica ion o seman ic segmen a ion is a ailable o he Pa is-Lille 3D,MLS1 TUM CC and
To on o3D da ase s.
Based on he da ase s om he las six yea s, i can be concluded ha mo e and mo e
LIDAR sys ems a e being used. Mainly LIDAR da ase s o ou doo a eas a e c ea ed,
because o he la ge ange and he highe esolu ion o hese sys ems. Fo indoo s, dep h
imaging came as a e s ill commonly used. Since many o hese da a come om he CV
domain, su ace models o oxels a e addi ional ou pu o ma s, along wi h poin clouds
and images. I can be seen ha he da ase s a e no necessa ily la ge in e ms o classes and
poin s, bu he anno a ion is mo e specialized and imp o ed compa ed o ea ly da ase s.
Ea lie da ase s a e e alua ed wi h new ools and op imized o speci ic asks. The manual
anno a ion can be s ill iden i ied as a bo leneck.
2.4. Poin Cloud Anno a ion Tools
Many anno a ion se ices and ools a e used o au onomous d i ing o d i e
assis ance. Fo his applica ion, a ew ou doo classes need o be ( oughly) anno a ed.
An o e iew and compa ison o 33 anno a ion ools o his a ea o applica ion is p esen ed
in [
67
]. These anno a ion ools mainly use simple geome ies, such as bounding boxes,
XLIX
Remo e Sens. 2022,14, 446 15 o 41
in he equi ed quan i y. I is he basis o all u he cha ac e is ics and mus be ul illed in
o de o ca y ou a seman ic segmen a ion and i s e alua ion.
The
p ocess eliabili y
desc ibes how he p ocess was ca ied ou . This cha ac e is ic
can be de e mined by pe o ming he seman ic segmen a ion se e al imes. A e each
segmen a ion, accu acy pa ame e s a e de e mined, which can be used as e mina ing
c i e ia, as in [
27
]. O he a ian s equi e ha a ce ain numbe o i e a ions mus be ul illed
in o de o de e mine a quali y pa ame e . This a ian is p e e ed o he desc ip ion o he
p ocess quali y, since i maps he a iance o he me ic and pa ame e alues. Using his,
he achie able pe o mance can be de e mined by a p ocess se ing. Due o he complexi y
o hese asks, he a e age epe i ion ac o o de e mine his pa ame e is usually e y
small, as shown in [13] ac o 2, in [57] ac o 2.3 and in [68] ac o 4.
Comple eness
gi es he deg ee o which he necessa y in o ma ion, de e mina ions
and execu ion o he wo k s eps o he classi ica ion a e p esen .
Consis ency
is he deg ee o which he measu ed alues ma ch he da a model. He e,
i is necessa y o check whe he he poin cloud ea u es a e p esen and whe he hey ake
he co esponding ange o alues.
Fo he seman ic segmen a ion,
co ec ness
,
p ecision
, and
seman ic accu acy
a e
di e en cha ac e is ics ha ha e di e en unde lying causes and di e en e ec s on
he usabili y o he poin cloud. Mo eo e , hese cha ac e is ics a e o en de ined and
summa ized in di e en ways. Fo example, i only he pe o mance o a seman ic
segmen a ion me hod is o be conside ed, co ec ness and p ecision a e o en combined
wi h accu acy. The accu acy is commonly gi en when ML and AI algo i hms a e used. In
many seman ic segmen a ion applica ions, his de ined cha ac e is ic is exp essed by he
pa ame e IoU, also known as he Jacca d index, o he F1 sco e, also known as Dice’s index.
These pa ame e s a e he weigh ed a e ages o bo h cha ac e is ics. I is ad an ageous
o apply one combined cha ac e is ic o accu acy and i s meaning ul pa ame e , e.g., IoU,
o be e compa ison. O he applica ions use mo e han one pa ame e o desc ibe he
di e en pe spec i es o accu acy o a mo e dis inguished and cause-o ien ed iew.
Fo he analysis o he seman ic segmen a ion p ocess, wo ypes o e o s a e possible:
a poin is e oneously assigned o a class o which i does no belong o a ue poin o
his class is no ecognized as membe o i . These wo e o s a e known as i s - and
second- ype e o s om s a is ical es s [
87
]. The segmen a ion
p ecision
can be conside ed
an e o o he i s ype. This e o speci ies how well an anno a o o an algo i hm can
dis inguish classes— o example, how accu a ely class bounda ies can be d awn. The
segmen a ion
co ec ness
can also be conside ed a second- ype e o . This e o desc ibes
how well a class can be ecognized, e.g., how unique he poin ea u es a e. Thus, he
bes ea u es a e used o ob ain a class o homogenous poin s. This ype o e o can be o
impo ance depending on he analysis in ques ion. Fo example, i may be less c i ical i
no all poin s o a la ge class (such as loo ) a e de ec ed du ing seman ic segmen a ion, as
long as hese poin s a e no classi ied o assigned o a class (e.g., scanning a i ac s) ha is
no u he used. Mo e p oblema ic a e addi ional poin s (e.g., om scanning a i ac s) ha
a e assigned o he class loo , because he poin cloud ep esen s inco ec seman ics.
Thus a , co ec ness and p ecision based on he numbe o poin s desc ibe he quali y
o a seman ic poin cloud. Howe e , hese cha ac e is ics do no gi e any in o ma ion
abou he geome y o he seman ic classes and i s geome ic size changes due o e o s. In
o de o be able o e alua e he geome ic aspec as well, he cha ac e is ics o co ec ness
and p ecision ha e o be ex ended. Geome ic co ec ness can be de e mined i a (dense)
e e ence poin cloud o su ace model is a ailable. The co ec ness can be de e mined o
each indi idual poin . This in o ma ion can no longe be e alua ed o se e al hund ed
housand poin s. The co ec ness o a poin cloud can be de e mined by he mean, a e age
de ia ion o s anda d de ia ion o all poin s in a segmen . In gene al, co ec ness is
he deg ee o which he abs ac model ma ches he achie ed seman ic segmen a ion
esul . This can be di ided in o use -dependen and so wa e-dependen co ec ness. The
use -dependen co ec ness is based on he unde s anding o he CD and he usage o he
LVI

Remo e Sens. 2022,14, 446 16 o 41
so wa e by he use . The so wa e-dependen co ec ness e e s o e o s in he so wa e
(e.g., inco ec pa ame e s o p og amming). Howe e , a sepa a ion is only possible i he
seman ic segmen a ion is ca ied ou se e al imes unde con ollable condi ions.
The quali y cha ac e is ic o p ecision is desc ibed by he pa ame e s o he seman ic
and geome ic p ecision. The e m "p ecision" should be de ined clea ly, because he e
a e di e en de ini ions in use. In he geode ic con ex , p ecision is o en unde s ood as
epea abili y [
87
]. The de ia ion o he esul s o an expe imen o i s mean alue a e n
epe i ions is de e mined. Fo he seman ic segmen a ion p ocess, his de ini ion would
lead o he de e mina ion o how much he indi idual segmen a ions de ia e om each
o he . This shall no be he main subjec o he in es iga ion, since a de ia ion o he mean
o se e al segmen a ions usually has no ele ance o a p ac ical applica ion. Ne e heless,
i makes sense o epea a segmen a ion and o calcula e a join poin cloud om hese
epe i ions in o de o inc ease he eliabili y, as men ioned abo e. Usually, de ia ion om
a e e ence poin cloud is equi ed. This can be exp essed by he a io o ue poin s o
all poin s assigned o a class [
88
]. This e m desc ibes how much o he segmen a ion
is “co ec ” and is commonly used in ML. Mos ly, he in e se p opo ion is o majo
impo ance o he de elopmen o an applica ion, because his desc ibes wha does no
wo k ye [
89
]. This p opo ion is hen he subjec o analysis. In addi ion o he use o he
numbe o poin s, i is ad an ageous o 3D models and poin clouds o also use he a eal
a ios as well as geome ic pa ame e s.
The
seman ic accu acy
desc ibes how well he seman ic label i s o a seman ic poin
cloud segmen . The di icul y is in de ining wha is seman ically co ec , which a ibu es
a e desc ibed and which dep h o desc ip ion and dis inc ion mus be applied. Fo he
de ini ion o wha is seman ically co ec , no uni e sally alid de ini ion can be ound. An
a emp o s anda dize his p oblem was discussed in Sec ion 3.2. Fo he ype o a ibu e
desc ip ion, he IFC s anda d [
80
] can be used. This is designed o he de elopmen and
no o he documen a ion. This can be explained using he example wi h he ables. The
able i sel o ms a seman ic class. These classes can be di e en ia ed du ing he nex
s age in o a ame and able op. As a as we know, he e is no s anda dized scheme
o his de ini ion, so ha an indi idual CD as shown in Appendix Amus be de eloped
and applied.
3.4.2. Quali y Pa ame e s
The se en quali y cha ac e is ics used o seman ic segmen a ion can be desc ibed by
quali y pa ame e s. These pa ame e s desc ibe he p ope y ha an objec has o a ce ain
cha ac e is ics. Fo ins ance, hese pa ame e s a e he p esence o a ce ain da a o ma
as a quali a i e pa ame e o he numbe o poin s (NoP) as a quan i a i e pa ame e . This
will be demons a ed in an example in Sec ion 4.1. The e alua ion o poin clouds by he
quali y model will be co e ed in Sec ion 4.2. Fo he e alua ion, he quali y pa ame e s
mus be de e mined and h eshold alues mus be se . Fu he mo e, he pa ame e s o
he seman ic segmen a ion can be dis inguished in o pa ame e s wi h objec ela ion (O),
conce ning he poin cloud, and p ocess ela ion (P), such as he ime equi ed o an ac ion
o he use o a ce ain CD. All pa ame e s o he seman ic segmen a ion ask a e b ie ly
explained and shown in Tables 5–11. The pa ame e s a e numbe ed in he ex and e e o
he co esponding able en y wi h P#.# o a clea unde s anding.
Quali y pa ame e s o cha ac e is ic
a ailabili y
desc ibe which in o ma ion mus
be a ailable abou he p ocess and he poin cloud o a desc ip ion and an e alua ion
(Table 5). These pa ame e s a e he abs ac model exp essed by he CD (P1.1), he size o
he poin cloud exp essed by he NoP (P1.2) and he a ea size (P1.3), as well as he objec
ea u es (e.g., x-, y-, z-coo dina es) be o e (P1.4) and a e (P1.5) he seman ic segmen a ion.
Fu he mo e, he ile o ma ou pu (P1.6) and use es ic ions (P1.7) mus be in es iga ed. The
use es ic ions e e o he ques ion o whe he a da ase can be used o an applica ion o
p ocessing s ep. Fu he es ic ions a e ha ce ain da ase s may no be used o aining.
LVII
Remo e Sens. 2022,14, 446 17 o 41
The pa ame e P1.7 ensu es an objec i e e alua ion o he da ase s. Thus, i is conside ed
ha any da ase has a ce ain bias, which is lea ned by ML algo i hms [86].
Table 5. Pa ame e s o a ailabili y.
P. No. Pa ame e Name Uni Range P/O
P1 A ailabili y
P1.1 CD exis s yes/no P
P1.2 Numbe o poin s >0 O
P1.3 A ea size m2>0 O
P1.4 Objec cha ac. in yes/no O
P1.5 Objec cha ac. ou . yes/no O
P1.6 File o ma ou e.g., p s O
P1.7 Use es ic ion yes/no O
Table 6. Pa ame e s o p ocess eliabili y.
P. No. Pa ame e Name Uni Range P/O
P2 Reliabili y o P ocess
P2.1 Numbe o
segmen a ions >1 P
P2.2 A e age ime equi ed % 0–100 P
The pa ame e s numbe o segmen a ions (NoS) (P2.1) and a e age ime equi ed (ATR)
(P2.2) desc ibe he
eliabili y o he p ocess
(Table 6). I a poin cloud is independen ly
seman ically segmen ed mo e han once, he eliabili y can be measu ed. The mo e
equen ly a p ocess is ca ied ou , he mo e eliable a e he co ec ness and accu acy.
This is he heo e ical assump ion. The pa ame e NoS desc ibes how o en a segmen a ion
was pe o med wi h a ce ain me hod. I is he basis o he calcula ion o o he pa ame e s
and can also be used as a quali y measu e. The ATR can be used o compa e di e en
seman ic segmen a ion me hods. The ATR is calcula ed o each me hod. The a e age ime
o all anno a o s wi h any me hod is o in e es . The maximum segmen a ion ime o all
me hods is he alue
∆ max
. The ATR is calcula ed om Equa ion (1), whe e
i
s ands o he
espec i e segmen a ion.
∆ i
is he e o e he ime needed o he segmen a ion
i
. Mo eo e ,
he use -dependen segmen a ion ime can be analyzed i all segmen a ions pe o med
wi h a ce ain ool a e compa ed. The pa ame e ATR desc ibes he p ocess and allows he
planning o he wo king ime.
ATR =∑imax
i=1k∆ i∗100
∆ max k
i(1)
Table 7. Pa ame e s o comple eness.
P. No. Pa ame e Name Uni Range P/O
P3 Comple eness
P3.1 Seman ic segmen a ion a e % 0–100 O
P3.2 Numbe o classes >0 O
The
comple eness
o a seman ically segmen ed poin cloud (Table 7) is desc ibed by
he seman ic segmen a ion a e (SSR) (P3.1) and numbe o classes (NoC) (P3.2). The pa ame e
SSR desc ibes how many poin s ha e been assigned o any class. The SSR is he quo ien o
he numbe o classi ied poin s (Pcls) and all poin s (Pall) (Equa ion(2)).
SSR =Pcls
Pall
∗100 (2)
A poin cloud ha is only segmen ed in pa s o en occu s in he applica ion phase. The
seman ically segmen ed pa s o he poin cloud a e used o aining o o he e alua ion
LVIII
Remo e Sens. 2022,14, 446 18 o 41
o an algo i hm. The es o he da a a e hen seman ically segmen ed using he au oma ic
me hod. The pa ame e NoC desc ibes how many classes a e a ailable o a ce ain da ase .
Table 8. Pa ame e s o consis ency.
P. No. Pa ame e Name Uni Range P/O
P4 Consis ency
P4.1 Geome ic Consis ency (GC) o x, y, z m ≥0 O
P4.2 Spec al Consis ency o RGB (SCRGB) 0–255 O
P4.3 Spec al Consis ency o I (SCI) 0–255 O
P4.4 Class equali y 0–1 O
The
consis ency
o he da a (Table 8) is de e mined by he uni s and he scaling
anges o he objec ea u es (P4.1 o P4.3). Each objec pa ame e di ec ly ela es o a
quali y pa ame e . The de e mina ion can be achie ed au oma ically o aken om he
da a (e.g., using a ex edi o ). Fu he mo e, he consis ency is desc ibed by he measu e
o he class equali y (CE) (P4.4). This is calcula ed om he a ge alue o a balanced class
dis ibu ion (
C a ge
). All classes should be ep esen ed by he same amoun o poin s, so
ha , la e , an ML p ocedu e has op imal lea ning condi ions. Howe e , his equi emen is
ne e gi en wi h eal da ase s, because classes such as walls and loo s a e o e ep esen ed
by poin s. The p opo ion o poin s o a class in ela ion o he o al NoP is exp essed
by a a io in he alue ange 0–1. The ac ual dis ibu ions a e hen calcula ed (
Cac
). The
di e ences be ween he a ge and ac ual alues o each class a e de e mined. The sum
o he absolu e di e ences di ided by wo is a measu e o balance (Equa ion(3)), whe e 0
ep esen s a balanced a io and 1 an unbalanced a io.
CE =∑k
i=1k(C a ge −Cac )k
2(3)
Table 9. Pa ame e s o co ec ness.
P. No. Pa ame e Uni Range P/O
P5 Co ec ness
P5.1 Recall o poin s class x % 0–100 O
P5.2 Recall o a ea class x % 0–100 O
The
co ec ness
(Table 9) o he seman ic segmen a ion can be desc ibed by he
pa ame e ecall o poin s (RP) (P5.1). The PR is he a e be ween co ec ly assigned ue
posi i e (TP) poin s and he NoP in he abs ac model o a ce ain class (TP and alse
nega i e (FN) poin s) (Figu e 7). I is exp essed by Equa ion (4). This pa ame e depends
on he size di e ences o he class in he abs ac model. I he classes di e g ea ly, as can
be e alua ed by he pa ame e CE, a compa ison o di e en classes may lose signi icance.
Fo a small se , e en a ew FN poin s can signi ican ly lowe he pa ame e . This p oblem is
discussed in [
89
] and desc ibed by a new pa ame e o in o ma i eness. Fo applica ions
in he con ex o poin clouds, his pa ame e is unsui able due o he i egula dis ibu ion
o he poin s.
RP =TP
TP +FN (4)
RA =TPa ea
TPa ea +FNa ea (5)
To a oid he poin cloud densi y p oblem, he ep esen a ion in he o m o a eas can
be used. He e, he a eas a e calcula ed o he poin cloud segmen s. Ins ead o he NoP,
he TP a ea size can be inse ed in o Equa ion (4). The esul is he ecall o a ea (RA) in
Equa ion (5). The co ec ness is now desc ibed by he a ea ha is co e ed by TP poin s
LIX
Remo e Sens. 2022,14, 446 19 o 41
di ided by he a ea o all e e ence poin s o his class. As an in e media e s ep o calcula e
hese pa ame e s, he a eas ha a e co ec ly and inco ec ly assigned a e calcula ed. In he
case o inco ec assignmen s, he dis inc ion be ween FN and FP a eas is o in e es . The
pa ame e RA exp esses he in luence o FN su aces. The in luence o he alse posi i e
(FP) a eas is desc ibed in he ollowing, among o he s, by he p ecision o a ea (PA). The
FN and FP poin s a e isualized in Figu e 8. This isualiza ion allows an analysis o he
seman ic segmen a ion, e.g., he assignmen o scanning a i ac s o a class o he occu ence
o classi ica ion gaps can be de e mined.
Figu e 7.
Schema ic ep esen a ion o he con usion ma ix o he loo class wi h en ies o TP, FN,
FP and ue nega i e (TN) poin s.
Table 10. Pa ame e s o p ecision.
P. No. Pa ame e Uni Range P/O
P6 P ecision
P6.1 P ecision class x % 0–100 O
P6.2 P ecision a ea class x % 0–100 O
P6.3 MD o FP p s. class x mm ≥0 O
P6.4 SD o FP p s. class x mm ≥0 O
The p ecision is exp essed by he p ecision o poin s (PP) (P6.1) and he PA (P6.2). The
PP is he a io o TP poin s o a class o all poin s assigned by he segmen a ion o his class
(Equa ion (6)). The assigned poin s could also be exp essed as he sum o he TP and he
FN poin s (Figu e 7).
PP =TP
TP +FP (6)
PA =TPa ea
TPa ea +FPa ea (7)
The conside a ion o he cha ac e is ic p ecision based on a eas ha a e spanned by
he poin cloud segmen s can be ad an ageous when using he poin cloud as a model. Fo
LX
Remo e Sens. 2022,14, 446 20 o 41
a geome ic exp ession, Equa ion (7) can be used o de e mine PA. The isualiza ion o he
FP poin s is gi en in Figu e 8, which is a good s a ing poin o he analysis p ocess.
Figu e 8. Segmen ed poin cloud o he class able colo ed by TP, FP and FN poin s.
Thegeome icpa o hep ecisioncanalsobedesc ibed by hepa ame e smaximumde ia ion
(MD) o FPpoin s (P6.3) and SDo FPpoin s (P6.4). The MDo FP and SDo FPpoin s ely on he
FP poin s o he seman ic segmen a ion. They a e he poin s ha change he geome y o
he seman ic class, as shown in Figu e 9. Fo his conside a ion, only classes wi h seman ic
objec s a e conside ed, since, no mally, he goal o seman ic segmen a ion is o ex ac
objec s and o emo e scanning a i ac s. The geome ic de ia ion o he poin cloud
segmen is o majo impo ance o c ea ing a model. I he poin cloud is used o c ea e
a mesh, hen he MD, which is he enla gemen o he class segmen , is decisi e. This
is exp essed by he u hes FP poin . Fo modeling on he basis o poin clouds o he
ep esen a ion o he eco ded objec s by symbols, as is he case a he LoD 100 o a BIM
applica ion [90], he pa ame e SD o FP poin s is mo e meaning ul.
The
seman ic accu acy
(Table 11) is desc ibed by pa ame e s ha can be exp essed by
yes-o -no ques ions. Documen a ion o he p ocess and isual inspec ions can be used o
de e mine he CD applied pa ame e (P7.1) and whe he i is s uc u ed hie a chically (P7.2).
The pa ame e CD applied can be answe ed wi h yes i he CD is used and a leas one class
is segmen ed. The pa ame e Hie a chical CD can be con i med i he used CD has se e al
le els (a leas wo) and so di e en seman ic de ailing le els a e a ailable. The que y
whose class was inally used is exp essed by he pa ame e P7.3. I he class is p esen and
seman ically co ec , he pa ame e is answe ed wi h yes.
Table 11. Pa ame e s o seman ic accu acy.
P No. Pa ame e Uni Range P/O
P7 Seman ic Accu acy
P7.1 CD applied yes/no P
P7.2 Hie a chical CD yes/no O
P7.3 class x used yes/no O
3.4.3. Desc ip i e and E alua i e Func ion
A quali y model such as he one abo e can ha e wo unc ions. One is desc ip i e and
he o he is e alua i e, as desc ibed by ISO 9000 (2015) [73].
LXI

Remo e Sens. 2022,14, 446 21 o 41
Fo he
desc ip i e use
, he aim is o display and analyze how indi idual pa ame e s
(de ined as signi ican by he model) a y when in luences change. Di e en se ings, ools
o wo k p ocesses o a seman ic segmen a ion can be compa ed. Quali y pa ame e s a e
no ans o med in o ano he ep esen a ion o ange o his pu pose. The main in luencing
cha ac e is ics o he de elopmen o a seman ic segmen a ion p ocess a e conside ed and
his is one main applica ion o he quali y model. Mo e p ecisely, he in luence o he ini ial
(manual) segmen a ion o a poin cloud is in es iga ed. Thus, he model also p o ides
he basis o desc ibing an au oma ic (e.g., ML-based) seman ic segmen a ion p ocess, as
conside ed in many wo ks, such as [91–94].
Figu e 9.
Calcula ion o he SD o FP poin s
σ
on he example o a chai . G een TP poin s a e
wi hin he objec bounda ies. The ed FP poin s we e added o he chai class bu ac ually belong o
ano he class.
Fo he
e alua i e use
, he sui abili y o a poin cloud o an applica ion should be
assessed. I should be de i ed om he pa ame e s whe he a poin cloud in combina ion
wi h he segmen a ion me hod is sui able o a ce ain applica ion o no . Fo his pu pose,
he calcula ed pa ame e s o he quali y model a e c ucial. An example applica ion would
be o use a seman ic poin cloud o de e mine he wall su ace a ea, o calcula e he
eno a ion cos s, based on he as-buil wall su ace a ea. Fo his ask, co ec seman ic
segmen a ion is c ucial. The poin cloud should be e alua ed by applying a quali y model
in ad ance. The quali y o he indi idual pa ame e s mus be de ined by limi o a ge
alues. These alues a e de i ed om he applica ion. The e alua ion s eps a e de ined
acco ding o he scheme shown in Figu e 10.
A e he limi o a ge alues ha e been de ined, hey a e compa ed wi h he
de e mined ac ual alues. This adjus men can be ep esen ed in an au oma ic p ocedu e
by one Boolean alue. In he simples o e all e alua ion me hod, all pa ame e s mus
be ue o su icien quali y. The weigh ing o he pa ame e s o special cases p e en s
excessi ely igo ous il e ing. The cen al issue is he limi o a ge alues, which a e no
always known and ha e o be es ima ed based on expe ience.
Figu e 10. E alua ion o he sui abili y o a poin cloud wi h he quali y model.
LXII
Remo e Sens. 2022,14, 446 22 o 41
4. Applying he Quali y Model
The bene i o he quali y model as a basis o desc ibing and e alua ing he p ope ies
o a seman ic 3D poin cloud will now be explained by some examples. The pe o mance
o he quali y model is shown on he basis o wo o ou own indoo poin clouds and o he
publicly a ailable poin cloud da ase s. Ou own poin clouds a e shown in Figu e 11 and
we e seman ically segmen ed independen ly, mul iple imes, using wo di e en seman ic
segmen a ion ools. The quali y o he poin cloud and he seman ic segmen a ion p ocess
a e desc ibed by he quali y pa ame e s. The e alua ion pe o mance o he quali y model
is conside ed o ou own and he publicly a ailable da ase s. The applica ions o in e es
a e he analysis o :
• Seman ic poin cloud as a model;
• Seman ic poin cloud as a modeling basis;
• Seman ic poin cloud as aining da a.
Ta ge alues a e de ined in each case. The geome ic, seman ic and o mal cha ac e is ics
o he poin cloud a e p ocessed and used o an applica ion. Howe e , hese poin cloud
cha ac e is ics ha e a deg ee o unce ain y i he seman ic poin cloud was c ea ed by
cap u ing a eal objec and pe o ming a seman ic segmen a ion. The possible e o s and
he quan i a i e unce ain y o he senso s a e desc ibed in Sec ion 2.2. I can be s a ed ha
he usually esul ing e ec s o cu en ly a ailable and used senso s do no signi ican ly
a ec he indoo modeling applica ions. Ou own poin clouds we e eco ded wi h he
Z+F Image 5016 using a esolu ion o 6mm a 10 m. The quali y was se o high o educe
he noise while s ill ha ing a mode a e (in p ac ice use ul) eco ding ime o 6min [
95
].
The geome ic co ec ness o his poin cloud on a la su ace can be es ima ed as 2mm
o 3mm acco ding o he in es iga ion o [
48
], using he DVW- es - ield-me hod acco ding
o [
96
]. This accu acy a ies due o he di e en su ace shapes and o he objec p ope ies.
In addi ion, scanning a i ac s occu , as shown in Figu e 1and desc ibed in Sec ion 2.2. The
ocus is now on he seman ic segmen a ion, whe e e o s a e caused by ool se ings and
he anno a o .
Figu e 11.
Poin s o be examined wi hou seman ic segmen a ion. Objec s o he chai , able and loo
classes, as well as scanning a i ac s, a e shown.
The poin clouds in Figu e 11 a e e y challenging o seman ic segmen a ion. A
CD was de eloped and applied in o de o in es iga e segmen a ion p oblems. This CD
consis s o i e classes and is pa ly hie a chically s uc u ed. The classes o he i s
le el a e loo , u ni u e and scanning a i ac s. In he second le el, he u ni u e class
LXIII
Remo e Sens. 2022,14, 446 23 o 41
is di ided in o able and chai . Tables and chai s a e wo objec classes ha a e spa ially
and geome ically simila , which makes segmen a ion di icul . The poin s o hese wo
classes also ha e simila spec al p ope ies. Finally, he objec su aces a e highly e lec i e
and he geome ic shape is suscep ible o he occu ence o scanning a i ac s. The loo
class was in eg a ed o simula e scenic segmen a ion wi h o eg ound and backg ound
objec s. The sepa a ion o scanning a i ac s is a complex ask, e en o humans, whe e
subjec i e decisions mus be made and lea ned. The es poin cloud does no ep esen any
eal pa icula ask, bu is in ended o demons a e achie able pe o mance on challenging
cases. The poin clouds show eco dings o a labo a o y (Lab) and a semina oom (Room),
which we e au oma ically segmen ed wi h he PCCT using he spec al pa ame e s colo
and in ensi y. The poin clouds we e p ocessed by up o nine di e en anno a o s. These
a e he es poin clouds Lab RGB,Lab I as well as Room RGB and Room I. Fu he mo e, he
poin clouds Lab and Room we e p ocessed wi h Recap, in which he anno a o s de e mine
he segmen s by hemsel es. These a e he da ase s Lab R and Room R.
4.1. Quali y Model o Desc ibe Seman ic Poin Clouds
The desc ip ion o a seman ic poin cloud and a segmen a ion p ocess is always based
on a selec ion o cha ac e is ics, wi h he goal o being able o answe a speci ic esea ch o
p ac ical ques ion. The esea ch ques ion o he ollowing conside a ion is:
Wha in luence do he segmen a ion ool and di e en anno a ions ha e on he quali y
o he seman ic segmen ed poin cloud?
The mo i a ion o his ques ion is o de elop an e icien , e ec i e and aceable
segmen a ion p ocess. Di e en expe imen al se ings and de elopmen s ages shall be
desc ibed, so ha hei in luences on he p ocess can be analyzed. This should also esul
in mo e con enien poin clouds o models and aining da a, as well as imp o ed p ocess
and algo i hm unde s anding. All cha ac e is ics o he model a e desc ibed in de ail in he
ollowing.
4.1.1. Reliabili y Cha ac e is ics
The eliabili y o a poin cloud can be desc ibed mainly by o mal in o ma ion o me ada a,
as lis ed in Table 12. The c ea ion and use o a CD, which egula es which objec s will be
segmen ed and classi ied, is o p ima y impo ance. A compa ison o seman ic segmen a ion
is only possible i he CD is kep cons an . The pa ame e CD exis s mus be a ailable o
u ilize all o he seman ic-based desc ip ions. The accu acy o he implemen a ion o he
CD is desc ibed by he pa ame e s o seman ic accu acy in Sec ion 4.1.3. Fo he es poin
clouds, a CD exis s, which desc ibes he seman ic classes o loo , u ni u e, chai and able,
and scanning a i ac s.
The size o he poin cloud is ano he o mal pa ame e , which is desc ibed by he
NoP and he su ace a ea. The NoP ha can be p ocessed by segmen a ion ools a ies
widely. Some imes, he poin cloud is au oma ically educed o a maximum NoP. This
il e ing changes he poin cloud s uc u e and, depending on he applica ion, can esul in
unwan ed e ec s, such as he loss o su ace de ails. The Lab and Room poin clouds consis
o 2.7 and 14.5 million poin s. The su ace a ea o he objec s co e ed by poin s is 51 m
2
and
61 m
2
o he Lab and he Room poin clouds, espec i ely. Based on hese wo pa ame e s,
an addi ional use ul pa ame e , he a e age poin cloud densi y, can be calcula ed. The
a e age poin cloud densi y can be used as he esolu ion o he poin cloud. This a ies
wi h he dis ance o he eco ding de ice, and his shows ha he pa ame e s o he quali y
model a e chosen o be undamen al, so ha op ional pa ame e ex ensions a e possible.
The segmen a ion ools equi e ce ain poin cloud ea u es o enable p ocessing.
Spec al ea u es a e o en used o pe o m an au oma ic segmen a ion o o colo he poin
o be e isual di e en ia ion. Mos poin clouds ha e geome ic ea u es (coo dina es)
and spec al ea u es o colo and in ensi y. In addi ion o hese ea u es, no mals (N) a e
calcula ed o c ea e pe spec i e images o o ien he single poin wi hin hei neighbo hood,
as done wi h he PCCT. These ea u es can en ich he poin cloud a e he seman ic
LXIV
Remo e Sens. 2022,14, 446 24 o 41
segmen a ion. The expo ed ea u e can change du ing he seman ic segmen a ion. The
poin clouds in he example a e only ex ended by he ea u e seman ic class. This is
exp essed by expo ing each class as a single p s ile. Closely ela ed o he ea u e
pa ame e s o he poin cloud is he ile o ma ha is a ailable o impo and expo o
so wa e. The p s o ma is suppo ed by all ools being used. This ile o ma co esponds
o he da a model o Sec ion 3.3. The used da a model s a es ha all segmen s should be
a ailable as an indi idual ile. I he da a model equi es ha he seman ics o he poin
cloud ha e o be included in one ile, hen a di e en expo ile o ma mus be used. This
ile o ma mus ha e one addi ional space o he seman ic label. The poin clouds Lab and
Room a e cu en ly no licensed and a e only used in e nally, so he e is no es ic ion on
usage (P1.7). This means ha he use o he da ase s canno be aced.
Table 12.
Calcula ed and de e mined alues o he quali y pa ame e s o a ailabili y and eliabili y
o p ocess. Objec pa ame e s wi h * a e calcula ed in he segmen a ion so wa e.
P. No. Pa ame e Name Lab RGB Lab I Lab R Room RGB Room I Room R
P1 A ailabili y
P1.1 CD exis s yes yes yes yes yes
yes
P1.2 NoP 2,790,352 poin s 14,526,242 poin s
P1.3 A ea size 51m261 m2
P1.4 Objec cha . in. x, y, z, I, R, G, B, xN*, yN*, zN*
P1.5 Objec cha . ou . x, y, z, I, R, G, B, Class
P1.6 File o ma ou . p s/cs p s/cs p s p s/cs p s /cs
p s
P1.7 Use es ic ion no no no no no
no
P2 Reliabili y o P ocess
P2.1 NoS 7 7 9 8 8
8
P2.2 ATR 13% 13% 55% 38% 45%
49%
In addi ion o poin cloud me ada a, me ada a abou he p ocess a e also ep esen ed
by he p ocess eliabili y, as shown in Table 12. Reliabili y can be de e mined i a p ocess
is pe o med independen ly mul iple imes. I can be de e mined by obse ing which
pa ame e s change sys ema ically and which a e andom. Acco ding o he esea ch
ques ion, wo in luences should be analyzed. On he one hand, he in luence o di e en
use s is conside ed, and on he o he hand, ha o di e en ools is assessed. The epea
accu acy o di e en use s is in es iga ed in Sec ion 4.1.4. A his poin , he ocus is on he
wo di e en ools. Fo a s a is ical conside a ion, he numbe o se en o nine anno a ions
pe ool is oo small. Howe e , a quali a i e o compa a i e desc ip ion o he in luences o
he ools in he o m o a endency is possible despi e he small numbe o samples. Fo his
pu pose, he ollowing alues a e no based on he anno a ions o indi idual anno a o s,
bu on a join poin cloud wi h all anno a ions. Fo he de e mina ion o he pa ame e s
o he da ase s Lab RGB and Lab I, se en di e en anno a ions we e pe o med; o he
Lab R da ase , nine anno a ions we e pe o med, and o he da ase s Room RGB,Room I and
Room R, eigh anno a ions we e pe o med.
The ATR is calcula ed based on he longes ime o seman ic segmen a ion o each
poin cloud. The maximum ime is 120 min o he poin cloud Lab and 194 min o he
poin cloud Room. Fo bo h poin clouds, he seman ic segmen a ion wi h Recap akes he
longes . The ATR alues in Table 12 show ha he PCCT p o ides an a e age o only 13%
o he maximum ime o small poin clouds such as Lab. Wi h Recap, he ATR is 55% o he
Lab poin cloud. Fo he la ge da ase , i can be seen ha he PCCT can be used o wo k
as e on a e age, bu he di e ences in ime dec ease wi h inc easing poin cloud size.
The pa ame e s o a ailabili y and p ocess eliabili y a e he basis on which o desc ibe
u he pa ame e s ha ha e a mo e p ac ical meaning o he in es iga ed ques ion. Thus
a , i is desc ibed how a p ocess can be ca ied ou wi h he selec ed da a and esou ces,
how eliable his p ocess and he o he quali y pa ame e s a e, as well as how e icien he
ools and i s usage a e in compa ison o o he s.
LXV