Recei ed 5 Oc obe 2023, accep ed 26 Oc obe 2023, da e o publica ion 14 No embe 2023, da e o cu en e sion 21 No embe 2023.
Digi al Objec Iden i ie 10.1109/ACCESS.2023.3333280
Poin Sampling Ne : Re olu ionizing Ins ance
Segmen a ion in Poin Cloud Da a
NANDHAGOPAL GOMATHI1, KRISHNAMOORTHI RAJATHI1, MIROSLAV MAHDAL 2,
AND MUNIYANDY ELANGOVAN 3,4
1Depa men o Compu e Science and Enginee ing, Vel Tech Ranga ajan D . Sagun hala R&D Ins i u e o Science and Technology, Chennai 600062, India
2Depa men o Con ol Sys ems and Ins umen a ion, Facul y o Mechanical Enginee ing, VSB—Technical Uni e si y o Os a a, 70800 Os a a, Czech
Republic
3Depa men o Resea ch and De elopmen , Bond Ma ine Consul ancy, EC1V 2NX London, U.K.
4Depa men o Biosciences, Sa ee ha Ins i u e o Medical and Technical Sciences, Chennai 602105, India
Co esponding au ho : Mi osla Mahdal (mi osla .mahdal@ sb.cz)
This wo k was suppo ed by he Applica ion o Machine and P ocess Con ol Ad anced Me hods h ough he Minis y o Educa ion, You h
and Spo s, Czech Republic, unde P ojec SP2023/074.
ABSTRACT Today, he e is a g ea need o 3D ins ance segmen a ion, which has se e al uses in obo ics
and augmen ed eali y. Unlike p ojec i e obse a ions like 2D pho og aphs, 3D models o e a me ic
econs uc ion o he scene ies wi hou occlusion o scale ambigui y o he en i onmen . In ag icul u e,
unde s anding Plan g ow h pheno yping enhances comp ehension o complex gene ic ea u es and accele -
a es he ad ancemen o con empo a y b eeding and sma a ming. A educ ion in c op p oduc ion quali y
is caused by lea diseases in ag icul u e. In o de o inc ease p oduc i i y in he ag icul u al indus y, i is
he e o e possible o au oma e he ecogni ion o lea diseases. Di e se lea disease pa e ns a ec he
de ec ion’s accu acy in he majo i y o sys ems. Du ing pheno yping, 3D PCs (PC) o componen s o plan s
like he s ems and lea es a e segmen ed in o de o ollow au onomous g ow h and es ima e he le el o
s ess he c op has expe ienced. This esea ch p oposed a Poin Sampling Me hod wi h occupancy g id
ep esen a ion o segmen ing PCs o di e en plan species, which was de eloped. To handle uno de ed
inpu se s, his app oach mainly elies on he applica ion o he single symme ic unc ion max pooling.
In eali y, a se o op imiza ion unc ions a e used by he ne wo k o choose poin s which is mo e cu ious o
ins uc i e om he PC and encapsula e he selec ion eason, and he ully connec ed laye s, used o shape
classi ica ion o shape segmen a ion, in eg a e hese lea ned ideal signi icances hooked on a global desc ip o
ega ding he o e all shape. A e being ained on he Poin Sampling Ne wo k-c ea ed plan da ase , he
ne wo k can simul aneously ealize seman ic and lea ins ance segmen a ion.
INDEX TERMS Ins ance segmen a ion, PC da a, poin sampling ne , poin clus e ing, seman ic segmen a-
ion, lea segmen a ion.
I. INTRODUCTION
A new ield o s udy called plan pheno yping links gene ics
o ag icul u e, ecology, and plan physiology [1]. In o de
o in ui i ely depic plan g ow h, i in es iga es a g oup o
ma ke s p oduced by he e e -changing in e ac ion o he
en i onmen and genes h oughou de elopmen . S udy [2]
in es iga es he co ela ion be ween pheno ypes and geno-
ypes by compu e ized digi aliza ion. This esea ch endea o
seeks o enhance ou comp ehension o in ica e gene ic
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Abdel-Hamid Soliman .
ea u es and, ul ima ely, expedi e he ad ancemen o con-
empo a y b eeding echniques and p ecision ag icul u e [3],
[4]. In gene al, he in es iga ion o plan pheno ype is p i-
ma ily conce ned wi h he o gans, co e ing aspec s like he
shape and s uc u e o he oo sys em and he cha ac e is ics
o he ui s and lea es. The p ima y si e o pho osyn he-
sis and espi a ion is ound in lea es because hey a e ha
o gan’s la ges su ace a ea [5]. Thus, some o he mos
signi ican pheno ypic cha ac e is ics o a lea a e i s wid h,
leng h, and inclina ion [6]. The s em sys em ac s as he
solid ounda ion o he plan ’s s uc u e and connec s all
o he o gans geog aphically, including he plan ’s lea es,
VOLUME 11, 2023
2023 The Au ho s. This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0 License.
Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ 128875
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
lowe s, and ui s. How much s ess he plan has been
exposed o can be de e mined by he pheno yping o he
s ems [7],[8]. Plan pheno yping success depends on he
e icien and p ecise segmen a ion o plan pa s. The ask
o segmen ing plan o gans has been he subjec o a ious
s udies since he 1990s, pa icula ly segmen ing lea es o
iden i y diseases. The basis o 2D pic u e-based pheno yping
is ypically con en ional algo i hms o pa e n ecogni ion,
image p ocessing, and machine lea ning o segmen a ion
based on a h eshold alue [9], edge de ec ion me hod [10],
egion g owing app oach [11], and clus e ing algo i hm [12].
Con olu ional neu al ne wo k (CNN)–based deep lea ning
algo i hms ha e ecen ly ad anced o s a e-o - he-a le els
in he classi ica ion o images and image segmen a ion [13].
Pic u e ne wo ks wi h deep lea ning we e used in e e -
ences [14] o sepa a e bo h lea es and ui s in pic u es o
plan s.
In spi e o his, he ew monoco yledonous plan s wi h
ewe lea es ha he 2D pheno yping app oaches o en
ope a e wi h a e plan 1 and plan 2. The p ima y cause
o his is ha dep h in o ma ion canno be included in a
2D image because i can only be aken om one iewing
angle. The blocking and o e lap in he canopy be ween
he lea es also subs an ially hinde segmen a ion echniques
based on 2D images [15],[16]. Analysis o he obse ed
beha io al ea u es’ s a is ical signi icance is also weakened
because pic u es do no adequa ely depic he whole loca-
ion dis ibu ion o he plan ’s s uc u e. In addi ion o colo ,
ex u e, and o he isual signals, 3D models can con ey he
mos impo an in o ma ion dep h as compa ed o pic u es.
The co ne s one o e y accu a e pheno ypic assessmen
is dep h di ec ly esol es he p oblems o occlusion and
o e laps. Plan pheno yping echniques based on dep h pic-
u es o PC da a ha e been quickly de eloping in ecen
yea s as a esul o he ad en o inexpensi e and high-
p ecision 3-dimensional imaging echnologies [17],[18].
The Lida app oach, which u ilizes lase -based senso s, has
become a widely employed ool o ‘‘3D econs uc ion’’
and pheno yping. This is mos ly due o i s excep ional p e-
cision in cap u ing de ailed in o ma ion abou la ge ees in
h ee-dimensional imaging [19], monoco plan [20], co on
plan [21], and se e al o he cash c ops [22]. 3D senso s based
on ime-o - ligh (ToF) and s uc u ed ligh & a e essen-
ial ools o Plan 3D pheno yping due o hei excep ional
eal- ime capabili ies [23]. In he e e ence [24], a ange
o g eenhouse and cash c ops we e ec ea ed and analyzed
using binocula s e eo ision. Re e ences [25] pe o med
C op pheno ypic analysis and 3D econs uc ion u ilizing
he mul iple iew s e eo (MVS) me hod. Yang e al. [26]
c ea ed a oolse called Label 3D maize plan ha manu-
ally iden i ies 3D PC da a o maize plan b anches so ha
i may be used o es ing and aining machine lea ning
models.
•This esea ch acknowledges he challenge posed
by di e se lea disease pa e ns in he majo i y o
sys ems. By using 3D ins ance segmen a ion, i seeks o
add ess his issue by enhancing he accu acy o disease
de ec ion.
•The p oposed Poin Sampling Me hod wi h occupancy
g id ep esen a ion is a no el app oach o segmen ing
3D poin clouds o di e en plan species.
•The ne wo k’s in eg a ion o shape classi ica ion and
shape segmen a ion in o a global desc ip o o o e all
shape is a aluable con ibu ion.
•This app oach allows he ne wo k o simul aneously
ealize seman ic and lea ins ance segmen a ion, p o-
iding a comp ehensi e unde s anding o he plan ’s
s uc u e and heal h.
Despi e ha ing good 3D da a, he sepa a ion o plan en i-
ies a e he de eloping chunk and he subsequen di ision
o Pheno yping p ocesses a e challenging when de e min-
ing he pheno ypic pa ame e s o each plan ’s o gans. The
use o 3D PCs o unsupe ised lea segmen a ion has
al eady gained p ominence. To dis inguish lea es om s ems,
enhanced he PC ne iden i ica ion echnique and c ea ed
a mixed segmen a ion algo i hm ha could accoun o he
physical di e ences be ween di e en indi iduals o co on.
He e al. [27] p esen ed an app oach ha was da a-d i en,
dynamic, and p oposal- ee and gene a ed he p ope con-
olu ion ke nels o use depending on he ci cums ances.
They in es iga ed a as con ex by assembling iden ical
poin s ha had wi hin o es o he geome ic cen oids and
sha ed he same seman ic ca ego ies in o de o make he
ke nels disc imina i e. Poux e al. [28] p esen ed a me hod
o he au oma ed ex ac ion o objec s om 3D poin clouds.
Thei app oach in ol ed an unsupe ised segmen a ion ech-
nique and a classi ica ion p ocess based on an on ology,
which was u he enhanced by sel -lea ning mechanisms.
Wang e al. [29] in es iga ed a echnique o accomplish-
ing o gan-le el 3D ins ance segmen a ion in le uce. Thei
objec i e was o ain a neu al ne wo k capable o accu a ely
segmen ing poin clouds o a ious plan lea es in o dis inc
ins ances. Zong e al. [7] employed a no el app oach o
enhance he pe o mance o a 3D poin cloud segmen a ion
model, wi h he aim o guiding he segmen a ion p ocess
o cells and accu a ely i ing he s uc u e o he con aine
ship cell guide. As a esul , i acili a ed he implemen a ion
o he con aine simula ion es box unc ionali y. Following
he segmen a ion o he cell guide, a p ecision inspec-
ion was ca ied ou concu en ly o assess i s p ac icali y.
Wang e al. [30] in oduced a no el unsupe ised pipeline
ha e ec i ely ca ied ou he asks o single- ee sepa a ion
and lea -wood classi ica ion concu en ly. The app oach sug-
ges ed in hei s udy is ounded upon a unique supe poin
g aph ha inco po a es comp ehensi e node and edge p op-
e ies. The u iliza ion o a join dual- ask ne wo k acili a ed
he acquisi ion o class labels and he g ouping o da a poin s
h ough he inco po a ion o g aph ope a o s. Vu e al. [31]
in oduced So G oup, a me hod ha demons a es simplici y
and e ec i eness in he ask o ins ance segmen a ion on 3D
poin clouds. The company So G oup c ea ed a me hod o
g ouping so seman ic sco es in o de o esol e he issue
128876 VOLUME 11, 2023
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
ha a ises om he p ac ice o ha d g ouping on objec s
ha a e locally ambiguous. The sugges ions de i ed om
he g ouping s age we e alloca ed o ei he posi i e o neg-
a i e samples. Nex , a e ining s age was implemen ed using
a op-down app oach o enhance he posi i e aspec s and
mi iga e he un a o able aspec s. Yi e al. [32] in oduced
GSPN, an inno a i e objec sugges ion ne wo k designed o
he pu pose o ins ance segmen a ion in 3D poin cloud da a.
The GSPN algo i hm p oduced sugges ions o objec s o
excellen quali y and objec ness, esul ing in a signi ican
enhancemen o pe o mance o an ins ance segmen a ion
amewo k. Zanjani e al. [33] in oduced a no el comp e-
hensi e lea ning amewo k, e e ed o as Mask-MCNe ,
designed o he pu pose o oo h ins ance segmen a ion
wi hin a h ee-dimensional poin cloud o in a-o al scan da a.
The p ecise segmen a ion o ee h is a c ucial componen in
he de elopmen o au oma ed compu a ional den is y.
Poin cloud ins ance segmen a ion is essen ial o comp e-
hending 3D scenes. The e a e s ill a ew issues ha need o
be esol ed, including (i) he absence o a common down
sampling echnique o poin clouds ha a e speci ically p e-
pa ed o deep lea ning and (ii) he ne wo k a chi ec u e o
mul ipu pose segmen ing a poin cloud is di icul ; o exam-
ple, a ne wo k is di icul o main ain equilib ium be ween
he ins ance segmen a ion ask and he o gan seman ic seg-
men a ion ask. A deep lea ning ne wo k called Poin Ne was
c ea ed o pe o m 2 Plan Phenomics plan body seman ic
segmen a ion and plan ins ance segmen a ion simul ane-
ously using a ully anno a ed poin cloud da ase comp ising
se e al species in o de o o e come he a o emen ioned
di icul ies. Deep lea ning based Down sampled Poin -wise
ins ance labels algo i hm p oposed o segmen a ion on poin
cloud da a.
Fu he , his esea ch wo k is o ganized; sec ion II p o-
ides an explana ion o he ma e ials and ela ed p ocesses.
Compa a i e es s and ou comes a e p o ided in Sec ion III.
Sec ion IV o e s mo e commen s and analyses. The inal
po ion is when he conclusion is eached.
II. MATERIALS AND METHODS
A. DATA PREPROCESSING
Da a p ep ocessing, o ins ance, segmen a ion in poin
cloud da a, in ol es cleaning, ans o ming, and o ganiz-
ing 3D poin clouds o be e model inpu , o en including
noise emo al, downsampling, and ea u e ex ac ion. He e,
a median il e is chosen o p ep ocess he da a.
U ilizing digi al me hods, image noise educ ion mini-
mizes undesi able b igh ness and colo luc ua ions caused
by low ligh o heigh ened senso sensi i i y. Algo i hms
disce n and mi iga e noise while p ese ing i al de ails,
ele a ing image quali y. Employing il e s and s a is ical
app oaches enhances cla i y, ende ing i ap o pho og aphy,
medical imaging, and compu e ision, ensu ing accu acy in
depic ion.
Median(C)=Med {Ci}(1)
=Ci(M+1)/2,Mis Odd (2)
=1/2[Cj(M/2) +Cj(M/2) +1],Mis (3)
B. PC SAMPLING
E icien ly managing complex poin clouds can pose signi -
ican compu a ional challenges due o hei inhe en high
p ecision, equen ly comp ising ens o housands o da a
poin s. Fu he mo e, a majo i y o cu en 3D neu al ne -
wo ks demand s anda dized inpu s ea u ing a ixed numbe
o poin s, hus manda ing he need o subsampling p io
o in-dep h p ocessing and modeling [34]. In his esea ch,
we in oduce no el enhancemen s o he well-es ablished
echniques o ‘‘Voxel-Based Sampling (VBS) and Fa hes
Poin Sampling (FPS)’’.
The sugges ed echnique demons a es e icien p ese a-
ion o local poin cloud (PC) densi y du ing he sampling
p ocess, equi ing low compu a ional e o . While he exis -
ing FPS (Fa hes Poin Sampling) sec ions a e inhe en ly
concise and spa se, he e is a po en ial isk o apid in o ma-
ion loss, which cons i u es a no able d awback. Wi hin he
h ee-dimensional (3D) space o he PC, a oxel is cha ac-
e ized by h ee pa ame e s: heigh , wid h, and leng h, wi h
Voxel-based Sampling (VBS) employed as a down-sampling
me hod. VBS eplaces he o iginal coo dina es wi hin each
oxel wi h he oxel’s cen e o g a i y o achie e his down-
sampling
VBS handles da a quickly, bu i aces wo signi ican
limi a ions: (i) The oxeliza ion scale elies on h ee a iables
ha luc ua e based on he PC’s size, densi y, and he o al
pixel coun pos -VBS sampling, making i unsui able o
p ocessing la ge ba ches o PCs and da ase s due o unp e-
dic abili y. (ii) VBS p oduces PCs wi h an e enly dis ibu ed
da a dis ibu ion, which is no conduci e o e icien deep
lea ning. A e ca e ully conside ing he p os and cons o
bo h FPS and VBS, we in oduce a no el PC downsampling
echnique known as Poin Sampling (PS). Figu e 1shows he
Poin Sampling Technique.
FIGURE 1. Poin sampling echnique.
VOLUME 11, 2023 128877
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
N, he quan i y o down-sampled poin s, comes i s and
mus be es ablished. In o de o un Poin Sampling on he
ini ial PC and c ea e a PC ha con ains a ew ex a poin s
han N, he second s age includes changing he pixel se ings.
The empo a y pixels o he PC a e subjec ed o he pe cen -
age o poin s o be educed, and he esul is a down-sampled
N-poin image. Fo all o he es s in his in es iga ion, N was
se o 4096.
C. NETWORK ARCHITECTURE
The en i e design o he backbone a chi ec u e o Poin Ne
is displayed in Figu e 2. The labeling ne wo k ecei es n
da a poin s as inpu , ans o ms he inpu and ea u es, and
hen pools all o he poin ’s ea u es oge he . The esul s a e
my labeling sco es. The labeling ne is expanded upon by
he segmen a ion ne wo k [35],[36]. I combines local and
global elemen s, as well as ou comes measu ed in poin s. All
laye s using ReLU employ he ba ch no maliza ion me hod.
In a label classi ica ion ne , d opou laye s a e employed o
he inal mlp. The h ee main componen s o an end- o-end
ne wo k. The on hal con ains an encode -like s uc u e
ypical o deep neu al ne wo ks and ou consecu i e Neigh-
bo hood Poin A ibu e Ex ac ion o Modules (APEMs)
o ea u e calcula ion [37],[38]. P io o each APEM, he
p ominen a ea is down-sampled, comp essing he ea u es
and dec easing he numbe o poin s. Deep Fusion Ne [36]
c ea ed a model o he ne wo k’s co e. The lowe decode ,
which is mo e complex han he mo e e icien decode and
indica es a mo e ine-g ained p ocess, is ini ially execu ed
by his componen , which we e e o as he Poin A ibu e
Fusion Module (PAFM).
FIGURE 2. Poin ne a chi ec u e.
A he end o he PAFM, ea u e usion is ealized using
con olu ions and a conca ena ion ope a ion [39]. The ea-
u e lows o he op and bo om b anches a e u ilized o
depic he ins ance segmen a ion p oblem and he seman-
ic segmen a ion challenge co espondingly. In addi ion,
in o de o comple e he p ocess o seman ic segmen a ion,
he seman ic low wi hin he ou pu module o Poin Ne
applies he ‘‘a gmax ope a ion’’ o he las ea u e laye ,
he e o e ex ac ing he p edic ed seman ic labels o each
indi idual poin . Meanwhile, Mean Shi clus e ing is e ec-
i ely employed wi hin he ins ance low’s inal ea u e laye
o achie e p ecise ins ance segmen a ion. The ine-g ained
3D ins ance segmen a ion is qui e di icul . Assigning pa
ca ego y is he ask gi en a 3D scan da a. Label each poin
o ace (e.g., lea ). We es using he Velodyne lida . Mos
ca ego ies o objec s ha e labels ha a pai o componen s.
Anno a ions pe aining o g ound u h a e agged on he
o ms’ sampled poin s. Pa segmen a ion is o mula ed as a
pe -poin classi ica ion p oblem.
Algo i hm 1 Poin Clus e ing
Inpu :G,Pins (ij)
Ou pu : Poin -wise ins ance labels o he en i e PC 1 se
G= −1;
2 Clus e Coun ←0;
3Repea o each block i do 4 check i equals o block1,
hen
5 Repea o e e y poin Pins (ij) in block I do
6 De ine x whe e Pins (ij) is loca ed in he x h cell o G;
7Gx←Pins (ij);
8end o
9else
10 Repea o e e y ins ance yjin block y do
11 De ine Gxpoin s in yja e loca ed in cells Gi;
12 Gi← he cells in Gi ha do no ha e alue −1; 13 check
i he equency o he mode <30, hen
14 Gi←Clus e Coun ;
15 Clus e Coun ←Clus e Coun +1;
16 else
17 Gi← he mode o Gi;
18 endi
19 end o
20 end i
21 end o
22 Repea o e e y poin Pins (ij) in he whole scene do
23 De ine k whe e Pins (ij) is loca ed in he k h cell o G;
24 Gi←Gi;
25 end o
D. ADJACENT POINT EXTRACTION FOR MODULE (APEM)
To imp o e Poin Ne ’s on -end ea u e ex ac ion, APEM
was de eloped. In he on sec ion, he e a e ou APEMs ha
ollow one ano he . APEM is always ollowed by a ea u e
space-down sampling in his on pa ha esembles an
encode . APEM possesses he ollowing wo cha ac e is ics:
(i) I concen a es on simul aneously ex ac ing high-le el
and low-le el in o ma ion, and (ii) The success ul u iliza ion
o he encode acili a es he ex ac ion o in o ma ion ha
is anspo ed om a local scale o a global scale. This
is achie ed by consis en ly ‘‘agg ega ing he neighbo hood
in o ma ion’’ inside he ea u e space.
128878 VOLUME 11, 2023
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
E. POINT ATTRIBUTE FUSION MODULE
Acco ding o Deep Fusion Ne , bo h ine-g ained poin -
le el ea u es and coa se-g ained oxel-le el da a a e e y
bene icial o ea u e lea ning. Combining he da a a wo
di e en g anula i ies enhances his ne wo k’s capaci y o
ex ac cha ac e is ics [40],[41]. We inco po a e a ‘‘Poin
A ibu e Fusion Module (PAFM)’’ in o he co e o ou Poin -
Ne as an homage o Deep Fusion Ne . The PAFM model
employs wo decode s, one o coa se-g ained lea ning and
ano he o ine-g ained lea ning, o gene a e ea u e lay-
e s ha exhibi di e en le els o lea ning g anula i y [42],
[43]. The module i s ob ains he coa se-g ained and ine-
g ained ea u e maps, deno ed as Fc and F , espec i ely.
These ea u e maps a e hen combined using echniques such
as a e age pooling and ea u e conca ena ion. The esul ing
agg ega ed ea u e deno ed as FDGF, is ob ained a e apply-
ing a 1D con olu ion wi h he ec i ied linea uni (ReLU)
ac i a ion unc ion. The esul s o PAFM ha e he po en ial
o enhance he pe o mance o he ollowing seman ic and
ins ance segmen a ion asks [44],[45].
F. INTERNET OF INTENSITY AND LOCATION MODULE
The s udy o a en ion is now a iable subjec despi e being
widely used in 2D image ecogni ion and deep lea ning. I is
s ill ea ly in he de elopmen o 3D PC lea ning ne wo ks
o apply a en ion mechanisms. Based on his, an In ensi y
and Loca ion A en ion Module was de eloped wi h wo
sub-a en ional mechanisms: Loca ion A en ion (LA) and
In ensi y A en ion (IA) [46],[47],[48]. In o de o emphasize
he signi icance o c ucial poin s in he PC ep esen a ion,
Loca ion A en ion (LA) emphasizes he da a poin s ha may
success ully exp ess essen ial a ibu es wi h Neighbo hood
loca ion and gi e g ea e p io i y. The ea u e ec o s wi h he
same in ensi y ha include he mo e in iguing da a a e mo e
likely o be he ocus o in ensi y a en ion (IA). By agg e-
ga ing along he a ibu e channel di ec ion, he signi icance
o each ea u e dis ibu ion is au oma ically de e mined [38].
Then, o e e y channel based on i s signi icance, he sig-
ni ican ea u e dimensions a e ampli ied while he un ela ed
ones a e simul aneously mu ed.
In he a en ion module, he op laye poin is esponsi-
ble o ins ance segmen a ion, and he bo om laye poin is
esponsible o seman ic segmen a ion. The speci ica ions o
concu en modi ica ions a e no he same, e en hough he
p ocessing pa ame e s o bo h lows a e he same. A ec o
o size 4096 is cons uc ed by pe o ming mean pooling on
he 4096 ou pu cha ac e is ic p oduced by he geog aphical
a en i eness module o he PAFM. The esul ing ec o is
hen subjec ed o he sigmoid unc ion o ob ain he posi ion
weigh ec o . To c ea e he ea u e ha is ein o ced by SA,
whose size is 128, he weigh ec o and PAFM ea u e a e
mul iplied [49],[50].
Finally, a 128-channel equency ec o is p oduced using
he sigmoid me hod. The ein o cing a ibu e ec o o he
p ecise loca ion p ocess, which is also an ou come o he
a en ion module, is c ea ed by mul iplying he agg ega ed
ea u e ec o by he weigh ec o . A e applying an addi-
ional 1D con olu ion o he calcula ion o he a en ion
uni , he ins ance low p oduces ‘‘a 4096 5 ea u e map’’ o
he segmen a ion o indi idual PC ins ances. The ins ance
segmen a ion esul is hen p oduced using he cha ac e is ic
map and Mean Shi clus e ing echnique. A e he clus e ing
p ocess, he loss unc ion is compu ed o e alua e he segmen-
a ion accu acy o he examples du ing he aining phase.
By in eg a ing an addi ional one-dimensional con olu ion
and an A gmax algo i hm in o he ou pu ea u e o he a en-
ion mechanism (AM) o seman ic low, whe e C deno es he
numbe o seman ic classes, we can gene a e a 4096. On his
page, he seman ic segmen a ion loss is de e mined.
Each scene is di ided in o 1 ×1 me e s. Also, he en i e
scene di ided in o a 400 ×400 ×400 g id G is. Gi se es
as a isual cue o he cell’s ins ance label. i ∈(0, 400 ×
400 ×400). A poin clus e ing algo i hm is c ea ed o
combine objec ins ances om dis inc blocks gi en G, and
poin -wise ins ance he names o each block Pins anywhe e
Pins (ij) deno es he block I ins ance label o he j h poin .
In Algo i hm 1, he p ocess o clus e ing he poin s om
di e en blocks a e labeling.
G. LOSS FUNCTIONS
Ou Poin Ne con ols nume ous ope a ions simul aneously
using ca e ully buil a ious loss unc ions. Deep lea ning
ne wo ks mus be ained using loss unc ions. The s anda d
c oss-en opy loss unc ion, o Lsem, is used o he seman ic
segmen a ion posi ion and is de ined as ollows.
loSEG = −
n
X
i=0
c
X
j=1
p(X)N
jlogp (X)J(i) (4)
whe e
xj - i is he expec ed likelihood ha ‘‘ he cu en poin pi
belongs o class j.‘‘ xj is he p edic ed p obabili y ha he
cu en poin pi belongs o class j (example lea ) and he one
ho encoding o he poin ’s ac ual seman ic class is ep e-
sen ed by xj. I he a gumen eally alls wi hin ca ego y j,
wi h xj ‘as i s alue I equals 1, hen 0. In he ask o ins ance
segmen a ion.
I he poin ac ually alls wi hin ca ego y j, i e u ns 1 else.
I is 0. The quan i y o PC inpu ins ances o he ins ance
segmen a ion ask can a y. Hence, in o de o supe ise he
aining p ocess, ‘‘a comp ehensi e loss unc ion’’ is u ilized,
consis ing o h ee sub-losses wi h a ying weigh s ac oss an
inde e mina e numbe o ins ances. The ins ance loss loINS
equa ion is p esen ed as ollows:
loINS
=lα∗sl (SEG)+lβ∗sl (INSLBL)+lγ∗sl(REGU) (5)
The weigh s o sub-loss o segmen s om PC epel di -
e en labels om clus e s, and egula iza ion losses aid in
o ming a egula o de ed and e ec i e ea u e limi o com-
pu a ion. He e, l and sl ep esen loss and sub-loss in ins ance
segmen a ion; he symbols α,β,ϒ, espec i ely, show o al
VOLUME 11, 2023 128879

N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
losses. The cen e o i h insa ance is aken om ea u e space
and he ea u e ec o c ea ed. In his esea ch, n o ins ance is
aken o compu ing he loss unc ion. Pa ame e d is aken as
a h eshold o wo neighbo hood clus e s. Finally, Fea u es
ha a e bo h coa se and ine-g ained a e in eg a ed in o de
o ge he ou pu goal.
A Double-hinge Loss (DHL) poin le el was sugges ed
in e e ence o aking in o accoun he limi a ions o he
ins ance ask on he seman ic ask on he ne wo k’s mid-le el
cha ac e is ics. Since DHL was immedia ely ansla ed Based
on he ea u e pa e n (Lo m), he Poin Ne o al loss unc ion
is depic ed as ollows.:
To alLoss =Loseg +Loins +Lo m (6)
III. EVALUATION MEASURES
To assess he e ec i eness o Poin Ne in he seman ic
segmen a ion o he PC, ou quan i a i e me ics o each
seman ic class: P ecision (P e), Recall (Re), In e sec ion o e
Union (IoU), and F1. I is be e i each o he ou seman ic
me ics has a highe sco e. The pe cen age o all poin s
p edic ed by he ne wo k ha we e co ec ly ca ego ized in o
his seman ic class is he measu e o p ecision. IoU measu es
he amoun o o e lap be ween each seman ic ca ego y’s
expec ed and ac ual a eas, whe eas F1 is a ull indica ion
p oduced by ha monically a e aging P e and Re.
The ins ance segmen a ion ou comes we e e alua ed using
he me ics mCo , mWCo , mP ec, and mRec. The ini ial
knowledge poin se o he m h case inside a gi en seman ic
g oup and he an icipa ed poin s se o he n h e en wi hin
he same seman ic class a e collec i ely e e ed o as IGm.
The highes possible alue o he en i e assessmen is deno ed
by max 12. The ne wo k’s p edic ed poin se and he g ound
u h poin se a e he wo inpu s ha he bina y unc ion IoU
uses o build he seman ic IoU equa ion. The pa ame e C
de e mines how many seman ic classes a e u ilized o calcu-
la e Mean Recall (mRec) and Mean P ecision (mP ec).
IV. EXPERIMENTS AND RESULTS
The da a u ilized in his in es iga ion was sou ced om he
plan PC, which was gene a ed h ough lase -scanned poin
cloud da a collec ion. The da ase includes obse a ions o
h ee dis inc c ops h i ing in a ious en i onmen al condi-
ions, wi h each c op unde going epea ed scanning o e a
30-day pe iod. Show he c ops in Figu e 3a a ious g ow h
phases (Gaidon, Wang, Cabon, & Vig, 2016). The da ase
has a 25 m maximum scanning e o . The collec ion has a
o al o 546 unique PCs, including 129 so ghum PCs and
312 oma o PCs. While he lowes PC only has abou 10,000,
he la ges has o e 100,000. In seman ic segmen a ion, he
aining and alida ion p ocess begins wi h a labeled da ase ,
whe e images a e anno a ed wi h pixel-le el class labels. This
da ase is di ided in o aining and alida ion se s, wi h he
o me used o model aining and he la e o moni o -
ing and ine- uning. The accu acy sco e e lec s he a io o
co ec ly classi ied pixels o he o al numbe o pixels in
FIGURE 3. Shows sample PC da a om plan 1 and plan 2.
he alida ion se , helping gauge he model’s pe o mance in
segmen ing objec s and scenes.
V. DATA PREPARATION AND TRAINING DETAILS
This p oposed esea ch wo k is ca ied ou in a ha dwa e
pla o m consis ing o an NVIDIA, an ENVI-suppo ed CPU
wi h 128 GB o RAM, 16 co es, and 32 h eads. Tenso Flow
1.13.1 is he deep lea ning amewo k used. The ba ch size
emains cons an a 8 du ing aining, and he p ocess o
lea ning a e is ini ia ed a 0.002., and a e 10 epochs pe
i e a ion, he gaining knowledge a e is 30% lowe . Since
he da ase is no open sou ce, he da a se s exhibi sol e
ime s. numbe o pose g aph nodes on a huge da ase bu
su ice i o s a e ha he pa ame e s sugges ed a e e ec i e.
7 h gene a ion benchma k on a low-powe compu e . I has
a mode a e CPU and memo y oo p in and can map e y
as egions. De aul se ings inc ease he numbe o pose
g aph membe s by O(N). Ce es and he SCHUR_JACOBI
p econdi ione wi h he SPARSE_NORMAL_CHOLESKY
sol e , based on ex ensi e es ing. Compa able o he dog leg
subspace echnique, using LM a he us egion is a supe io
op ion because i is well-suppo ed.
A. SEMANTIC SEGMENTATION OUTCOMES
Figu e 4displays he quali a i e Poin Ne seman ic c op
species indings. To highligh he ac ual pe o mance,
we p ominen ly exhibi es samples om a ious g owing
se ings and phases o Poin Ne ha appea o be success ul
in di e en ia ing be ween he s em and he lea es because
The line di iding he wo seman ically d i en classes only
con ains a ew in equen inco ec segmen a ions. Table 1
displays he esul s o Poin Ne ’s quan i a i e seman ic di i-
sion o he en i e es se . The majo i y o me ics a e g ea e
han 85 %, sugges ing sa is ac o y seman ic pe o mance.
F om he me ics shown in Table 1, i is clea ha Poin Ne ’s
lea segmen a ion esul s a e supe io o i s s em esul s since
he e a e a ewe poin s connec ed wi h s ems in he aining
da a han he e a e wi h lea es. Ou o he h ee species,
plan 2 o e s he bes seman ic segmen a ion esul s, which
migh be accoun ed o by he aining da ase ’s p edomi-
128880 VOLUME 11, 2023
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
TABLE 1. Quan i a i e measu e o seman ic segmen a ion o plan .
nance o plan 2 PCs. Addi ional in o ma ion om he wo
dis inc species can be added o his unbalanced aining o
make i be e esul s.
B. INSTANCE SEGMENTATION OUTCOMES
Sample PCs om a ious g ow h s ages oge he wi h he
compa a i e examina ion o h ee c ops’ Poin Ne ins ance
esul s [51],[52]. All h ee species ha e e ec i ely seg-
men ed lea specimens [53],[54],[55]. Signi ican di e -
ences exis be ween he h ee species’ lea s uc u es. La ge,
b oad lea es a e ound on obacco plan s, complex lea es
wi h a leas one la ge lea le and wo smalle lea le s a e
ound on oma o plan s, and long, hin lea es a e ound on
so ghum plan s [56],[57]. Poin Ne does well a segmen ing
lea ins ances ac oss all h ee lea a ie ies. The quan i a-
i e assessmen s o Poin Ne ’s lea segmen a ion by ins ance
o he es se a e shown in Table 2. The bulk o me ics
show a e age pe o mance is 85%, which is app op ia e o
ins ance, segmen a ion.
TABLE 2. Quan i a i e measu e, o ins ance, segmen a ion o plan .
C. EVALUATION OF ALTERNATIVE METHODS
On he same plan da ase , he Py hon ool is used, and
ou Poin Ne is compa ed wi h a numbe o well-used PC
segmen a ion ne wo ks in his subsec ion. The only ones
ha can do any hing o he han seman ic segmen a ion a e
Poin Ne ++ and DGCNN. ASIS and Plan Ne can simul-
aneously do bo h he seman ic and ins ance di ision asks,
much like ou ne wo k can. Using he same se o concep ual
and ins ance labels, we de elop all h ee dual- unc ion ne -
wo ks [58],[59],[60]. F om each o hei o iginal a icles,
we adop ed he sugges ed pa ame e combina ions o he
compa ing ne wo ks. In he majo i y o cases, Poin Ne ou -
pe o med he compe i ion and ou pe o med hem on all ou
a e aged quan i a i e c i e ia. Ins ance segmen a ion pe o -
mance compa ison be ween Poin Ne and wo dual- unc ion
segmen a ion ne wo ks is shown in Table 2.
FIGURE 4. Pe o mances o seman ic segmen a ion a e compa ed o
quan i a i e.
The e alua ion me ics conside ed o his compa ison
include Mean P ecision (MPREC), Mean Recall (MREC),
Mean Co e age (MCOV), and Mean Weigh ed Co e age
(MWCOV) o ou dis inc me hods: ASIS, PLANTNET,
PSEGNET, and ou p oposed me hod.
Ou p oposed Poin SamplingNe has ob ained he bes pe -
o mance, aking in o accoun all ou a e aged me ics, wi h
he excep ion o he mCo alues, mRec alues, and mP ec
alues o lea es.
We quali a i ely compa ed Poin Ne o he s a e-o - he-
a on challenges equi ing seman ic di ision and ins ance
segmen a ion. Figu e 5p o ides a de ailed e alua ion o he
quali a i e pe o mance o ins ance segmen a ion and seman-
ic spli ing applied o a ious plan species. The assessmen
is based on se e al key me ics, including Mean P ecision
(MPREC), Mean Recall (MREC), Mean Co e age (MCOV),
and Mean Weigh ed Co e age (MWCOV), o ou dis inc
me hods: ASIS, PLANTNET, PSEGNET, and ou no el p o-
posed me hod. The examples in he wo images demons a e
ha Poin Ne ou pe o ms ne wo ks designed p ima ily o
seman ic segmen a ion as well as ‘‘dual- unc ion segmen a-
ion ne wo ks, such as ASIS and Plan Ne .’’
FIGURE 5. Quan i a i e pe o mances o ins ance segmen a ion.
D. COMPARATIVE ANALYSIS
In his sec ion, we conduc an analysis o he p oposed
me hod wi h espec o mP ec and mRec me ics and p o ide
a compa a i e assessmen agains exis ing me hods, namely
VOLUME 11, 2023 128881
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
DKne and ISBNe . Figu e 6shows he expec ed poin g oup
o he n- h ins ance in he same seman ic class, which e lec s
he g ound u h poin s g oup o he m- h ins ance. The high-
es alue o e e y pa ame e es ed is shown by max. The
bina y unc ion IoU is compu ed p ecisely as he seman ic
IoU equa ion. I akes wo inpu s: he g ound u h poin se
and he an icipa ed poin se om he ne wo k. The numbe
o seman ic classes used o calcula e he Mean P ecision
(mP ec) and Mean Recall (mRec) is deno ed by he pa am-
e e . The esul s o his analysis a e p esen ed in Figu e 7.
FIGURE 6. mP ec compa ison.
FIGURE 7. mRec compa ison.
VI. DISCUSSION
A. GENERALIZATION ABILITY OF POINTNET
He e, we demons a e ha he sugges ed Poin Ne is adap -
able and adequa e o be used wi h u he kinds o PC
da a. We used he S an o d La ge-Scale 3D in e io Spaces
(S3DIS) o p ac icing and e alua ing ou Poin Ne in o de
o con i m i s sui abili y o PCs o in e io sec ions, which
di e g ea ly om 3D c ops. The S3DIS da ase ’s poin s a e
b oken down in o 13 seman ic classi ica ions, including loo ,
able, window, and mo e. Each block se ed as a single PC
inpu and was downscaled o 4096 poin s.
S3DIS A ea 5’s PCs we e employed in sc eening, while he
o he S3DIS egions we e employed in aining. This wo k
ained Poin Ne using he same hype pa ame e se ings used
o aining he plan da ase . Displayed a e he ou comes o
he acu e seman ic and ins ance segmen a ion in Figu e 8,
in u n. Compa ing Poin Ne o he GT, he majo i y o he
poin s we e co ec ly ca ego ized, and ou ne wo k appea s
o be pa icula ly adep a iden i ying u ni u e like ables
and chai s wi h a a ie y o o ms and o ien a ions. The ou
sepa a e ooms exhibi good ins ance segmen a ion om ou
Poin Ne , despi e he ac ha ins ance-based segmen a ion
on he da ase S3DIS is hough o be a di icul p ocess. The
loss o ins ance segmen a ion is compu ed du ing aining jus
a e clus e ing. Using a o he seman ic segmen a ion ask
an A gmax ope a ion and an addi ional 1D con olu ion on
he using AM’s seman ic low ou pu unc ion, we a i e o a
esul o one-ho encoded ea u e o 4096 ×C, whe e C s ands
o how many seman ic classes he e a e. He e, he seman ic
segmen a ion loss is compu ed.
Ins ance segmen a ion on small hings appea s o be mo e
e ec i e o Poin Ne han on la ge objec s.
FIGURE 8. Lea segmen om PC da a (Le ), Lea ins ance segmen
(Righ ).
B. DISCUSSION OF THE EFFECTIVENESS
This pa o he a icle shows mo e de ail abou how his s udy
add esses he h ee p oblems ha exis in he s a e-o - he-a
deep lea ning models o PC segmen a ion. Building a ne -
wo k o Segmen ing mul ipu pose PCs is challenging. Each
segmen a ion ask necessi a es a dedica ed ne wo k segmen
ha is go e ned by a unique loss unc ion, esul ing in wo
causes. In Poin Ne , Lsem and Lins, espec i ely, cons ain
bo h he ins ance and seman ic segmen a ion pa hways. Addi-
ionally, we in oduced he poin -le el losses o he ea u e
a ibu e map in o de o mo e e ec i ely esol e he aining
on he p incipal ne wo k. As a esul , Poin Ne ’s en i e ne -
wo k is cons ained by h ee losses combined. Modi ying An
addi ional design conce n is he po en ial impac o he weigh
assigned o he loss o one b anch on he o e all loss, which
may esul in an imbalance in he load o he o he b anch.
Fo ins ance, i is p ojec ed ha ins ance pe o mance will
su e once he le el o seman ic deg ada ion in Poin Ne is
inc eased. By al e ing he weigh s gi en o each segmen ’s
losses, i is possible o achie e an app op ia e a io be ween
seman ic segmen a ion and ins ance segmen a ion. Poo gen-
e aliza ion o species equen ly leads o wo un a o able
ou comes o a PC-acqui ing model. The i s is ha i is
possible o w ongly label pa s o a plan 2’s s em as ‘‘ he s em
o a plan 1’’ when using seman ic segmen a ion; o ins ance,
128882 VOLUME 11, 2023
N. Goma hi e al.: Poin Sampling Ne : Re olu ionizing Ins ance Segmen a ion in Poin Cloud Da a
i is possible o iden i y elemen s o a plan species ‘‘1’’ on a
PC o a plan species ‘‘2’’. Ano he phenomenon is he possi-
bili y o a signi ican segmen a ion pe o mance di e en ial
be ween monoco yledons and dico yledons as a esul o he
subs an ial a ia ions in 3D s uc u e. Figu e 9shows he
T aining and Valida ion Accu acy o seman ic segmen a ion
on he Le side, and he igh side shows he T aining and
Valida ion Accu acy, o ins ance, segmen a ion.
FIGURE 9. T aining and alida ion accu acy o seman ic segmen a ion
(Le ) and ins ance segmen a ion (Righ ).
Poin Ne ou pe o med well-known ne wo ks on asks
equi ing seman ic and ins ance segmen a ion, and esul s
ha a e bo h quali a i e and quan i a i e in na u e showed
The e we e wo undesi ed phenomena de ec ed in equen ly.
Fu he mo e, we ound ha o he nume ous objec ca -
ego ies seen in PCs, Poin Ne has good gene aliza ion
capabili ies. The ne wo k’s applicabili y o o he disciplines,
such as ‘‘indoo Simul aneous Localiza ion and Mapping
and sel -d i ing ca s,’’ was shown by he Poin Ne es on
he indoo S3DIS da ase . The p oposed algo i hm is also
es ed wi h eal- ime da a, which was eco ded using Velo-
dyne lida . The PAEM independen ly ga he s bo h high and
low-le el ai s om wo nea by loca ions in o de o imp o e
lea ning. Using wo concu en decode s, Fi s , PAFM builds
ea u e laye s using wo dis inc lea ning g anula i ies. Then,
o c ea e ho ough ea u e lea ning, PAFM in eg a es he wo
ea u e g anula i ies. Using wo ypes o in e es (loca ion
and channel), he ocused a en ion componen o Poin -
Ne inc eases he e iciency o ne wo k aining o bo h
segmen a ion asks. The nega i e e ec s on classi ica ion
aining and p ocedu e a e shown in Figu e 10. Nume ous
aining sessions aid in inc easing accu acy and minimizing
losses. The quan i a i e pe o mance compa ison o PSegNe
on ins ance segmen a ion agains he unc ion segmen a ion
ne wo ks. Wi h he excep ion o he plan lea mP ec, mRec,
and mCo , ou PSegNe has demons a ed he bes pe -
o mance ac oss all cases, encompassing all ou a e aged
measu emen s.
Addi ionally, we quali a i ely con as ed PSegNe wi h he
s a e o he a on asks ela ed o seman ic and ins ance
segmen a ion. The Poin Ne ’s quan i a i e seman ic segmen-
a ion indings o he whole es se show sa is ac o y
FIGURE 10. T aining and alida ion loss on segmen a ion p ocess.
seman ic segmen a ion pe o mance, wi h mos measu es
eaching 75.0%.
C. FAILINGS
Poin Ne s ill pe o ms badly on seedlings e en hough i can
do a pai o dis inc kinds o ca ego iza ions on h ee di e en
species wi h a ious g ow h phases. The s age o seeds is
signi ican ly smalle han he ea lie phases o de elopmen
and has a dis inc i e 3D shape o many plan species. Conse-
quen ly, because o he unique cha ac e is ics o he seedling
s age, i may be di icul o ain and e alua e deep lea ning
ne wo ks. Sadly, his is ue o all such ne s. Poin Ne ’s
segmen a ion pe o mance will decline as he deg ee o de ail
o he plan s uc u e inc eases. The e a e wo causes. The
ne wo k canno unc ion immedia ely on examples o ees
because he e a en’ any lea y plan specimens in he a ailable
da ase . Second, he sys em can only ecei e 4096 da a as one
sample sou ce owing o ha dwa e limi a ions. The plan PC
wi h dense oliage is expec ed o ha e a limi ed numbe o
poin s on each o gan, esul ing in inadequa e ea u e lea ning
and hence gene a ing subop imal segmen a ion ou comes.
VII. CONCLUSION
In his s udy, he Poin Down Sampling app oach is in o-
duced. The no el PC down-sampling me hod has ad an ages
o e bo h he con en ional dis ance poin sampling echnique
and he oxel-based sampling s a egy. Gi en ha i can adjus
he numbe o poin s a e downsampling and add andom-
iza ion o he da ase , i is especially highly sui ed o deep
lea ning ne wo k es ing and aining. Addi ionally, using
da a om lase scanning o a ious species, his esea ch
de elops Poin Ne , an inno a i e dual- unc ion classi ica ion
ne wo k app op ia e o c op PCs.
The en i e Poin Ne consis s o he Poin Fusion Module,
he middle elemen , and he on pa , which is o en an
encode in deep neu al ne wo ks. The Poin Fusion Module
decodes and combines wo cha ac e is ics wi h di e en g an-
ula i ies. The ins ance segmen a ion ask is ep esen ed by he
uppe b anch and he lowe b anch o he a en ion modules
o he hi d Poin Ne componen , espec i ely. Ou Poin Ne
ou pe o med a a ie y o well-known ne wo ks, namely
VOLUME 11, 2023 128883