Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
A ailable online 17 Janua y 2023
0168-1699/© 2024 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/).
Con en s lis s a ailable a ScienceDi ec
Compu e s and Elec onics in Ag icul u e
jou nal homepage: www.else ie .com/loca e/compag
O iginal pape s
Weakly and semi-supe ised de ec ion, segmen a ion and acking o able
g apes wi h limi ed and noisy da a✩
Thomas A. Cia uglia a,∗,1, Ionu M. Mo oia,1, Leona do Sa aceni a,1, Mulham Fawakhe ji a,
Albe o San eliu b, Daniele Na di a
aDepa men o Compu e , Con ol and Managemen Enginee ing (DIAG) ‘‘An onio Rube i’’, Sapienza Uni e si y o Rome, ia A ios o 25, Rome, 00185, I aly
bIns i u de Robo ica i In o ma ica Indus ial, CSIC-UPC, Calle Llo ens i A igas 4-6, 08028, Ba celona, Spain
ARTICLE INFO
Keywo ds:
F ui de ec ion and segmen a ion
Yield p edic ion
Compu e ision
Deep lea ning
Sel -supe ised lea ning
ABSTRACT
De ec ion, segmen a ion and acking o ui s and ege ables a e h ee undamen al asks o p ecision
ag icul u e, enabling obo ic ha es ing and yield es ima ion applica ions. Howe e , mode n algo i hms a e
da a hung y and i is no always possible o ga he enough da a o apply he bes pe o ming supe ised
app oaches. Since da a collec ion is an expensi e and cumbe some ask, he enabling echnologies o using
compu e ision in ag icul u e a e o en ou o each o small businesses. Following p e ious wo k in his
con ex (Cia uglia e al.,2022), whe e we p oposed an ini ial weakly supe ised solu ion o educe he da a
needed o ge s a e-o - he-a de ec ion and segmen a ion in p ecision ag icul u e applica ions, he e we imp o e
ha sys em and explo e he p oblem o acking ui s in o cha ds. We p esen he case o ineya ds o able
g apes in sou he n Lazio (I aly) since g apes a e a di icul ui o segmen due o occlusion, colou and gene al
illumina ion condi ions. We conside he case in which he e is some ini ial labelled da a ha could wo k as
sou ce da a (e.g. wine g ape da a), bu i is conside ably di e en om he a ge da a (e.g. able g ape da a).
To imp o e de ec ion and segmen a ion on he a ge da a, we p opose o ain he segmen a ion algo i hm wi h
a weak bounding box label, while o acking we le e age 3D S uc u e om Mo ion algo i hms o gene a e
new labels om al eady labelled samples. Finally, he wo sys ems a e combined in a ull semi-supe ised
app oach. Compa isons wi h s a e-o - he-a supe ised solu ions show how ou me hods a e able o ain new
models ha achie e high pe o mances wi h ew labelled images and wi h e y simple labelling.
1. In oduc ion
De ec ion and acking o ui s and ege ables a e wo undamen al
asks o p ecision ag icul u e, enabling obo ic ha es ing and yield
es ima ion applica ion. As wi h any o he au oma ion ask, de ec ion
o ege ables bene i s om con olled en i onmen s and well known
ield condi ions. The educ ion o a iabili y and unce ain y in ui
posi ion, occlusions, a ie y, illumina ion, o ci e a ew aspec s o he
p oblem, ha e a huge impac on he success ul implemen a ion o a
lea ning based de ec ion sys em. Fo his eason, many de ec ion based
sys ems a e designed wi h he aim o educing he sou ces o a iabili y.
Fo example, Wang e al. (2013) and Nuske e al. (2014) p oposed
de ec ion sys ems o ui coun ing and yield es ima ion using a di ec
illumina ion de ice o con ol ambien ligh . In bo h cases, he p oposed
sys ems need o be un a nigh o be unc ional. In P e o e al. (2021),
✩This wo k ex ends he one i led ‘‘Pseudo-label Gene a ion o Ag icul u al Robo ics Applica ions’’ p esen ed a he 3 d In e na ional Wo kshop on
Ag icul u e-Vision, CVPR 2022, New O leans.
∗Co esponding au ho .
E-mail add ess: [email p o ec ed] (T.A. Cia uglia).
1The au ho s con ibu ed equally o he wo k.
an example o a s addle obo ic pla o m is gi en, which is ano he
common way o con ol he en i onmen al ligh and emo e came a
in insics a iabili ies.
While hese app oaches a e iable, hey a e also di icul o adap
o di e en cul i a ions and equi e conside able economical and ech-
nical in es men , which is o en beyond he capaci ies o small and
medium ag icul u al businesses, which a e o en amily based. While
he economic p oblems hey ace a e gene ally he same as hose o
bigge companies (e.g. lack o manpowe o ha es ege ables), hey
do no ha e he economic s eng h o knowledge o enginee he cul i-
a ion om he g ound up o hea ily au oma ed p ocesses. This means
ha ha ing mo e lexible app oaches ha a e mo e algo i hmic and
da a o ien ed han ha dwa e o ien ed would posi i ely impac hese
h ps://doi.o g/10.1016/j.compag.2023.107624
Recei ed 26 Augus 2022; Recei ed in e ised o m 1 Janua y 2023; Accep ed 2 Janua y 2023
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
2
T.A. Cia uglia e al.
businesses, allowing a ange o possible applica ions o da a d i en
p ecision ag icul u e.
In his espec , mode n compu e ision, mos ly based on deep
lea ning algo i hms, has lowe ed he ini ial in es men needed o in e-
g a e ad anced de ec ion echniques o moni o ing and managing he
c ops. Howe e , as discussed by Koi ala e al. (2019) in hei su ey
on deep lea ning echniques applied o ui coun ing, hese algo i hms
a e da a hung y and ga he ing he co ec and igh amoun o da a is
no always s aigh o wa d.
One way o ace da a sca ci y is he use o algo i hmic echniques
o semi-supe ised, weakly-supe ised and ans e lea ning, whe e
addi ional in o ma ion is added o he aining p ocess as an ex e nal
aining signal, o by le e aging wha was lea ned in a di e en bu
ela ed ask. An example o hese app oaches in ag icul u e is gi en
by Bellocchio e al. (2019) whe e he au ho s p opose an oli e coun ing
solu ion ha is explici ly ained wi h weak labels and consis ency
losses. The auxilia y signal in his case is he labelling ob ained by
an ex e nal classi ie ha de ec s whe he o no he e a e oli es in
he pic u e. This wo k is close o ou s o he ocus on wo king on
da a wi h minimal labelling. Howe e , i is based on simple di ec ui
coun ing, which can lead o huge e o s in cases whe e sel occlusion
is ypical. An example o ans e lea ning ha can be used o educe
he amoun o da a needed o aining is shown in Gülden ing and
Nalpan idis (2021). In his s udy, he au ho s show how p e- aining
using con as i e lea ning as an unsupe ised echnique is able o im-
p o e he pe o mances o de ec ion algo i hms compa ed o s anda d
ImageNe p e- aining. This esul is a good s a ing poin o aining
a deep ne wo k, bu some ask speci ic aining da a is s ill needed.
In his con ex , i is in e es ing o conside he challenges ha
de ec ion and acking algo i hms ace when he ield is no p epa ed
o au oma ion. Some o hese challenges a e: une en dis ibu ion o
ege ables in he ield, in a-species a iabili y, illumina ion, occlusion
and clu e . F om a echnical poin o iew, all hese aspec s ansla e
o co a ia e shi s and lack o labelled samples. Fo example, a s udy
on he in a-species co a ia e shi o swee peppe s and i s impac on
de ec ion and segmen a ion algo i hm is gi en in Hals ead e al. (2020).
The au ho s explo e he issue o gene alizabili y by conside ing a ui
ha is g own using di e en cul i a s and in di e en en i onmen s
( ield s glasshouse). Thei esul s show how in single ask lea ning
he pe o mances d op signi ican ly, as low as 0.323 o F1-sco e, on
c oss da ase de ec ion, and only by se ing up a mul i- ask lea ning
p oblem hey a e able o inc ease his sco e by a good ma gin, hanks
o he mul iple back p opaga ion signals. Le e aging he c oss- ask
co ela ions can be seen in i sel as a o m o sel -supe ised lea ning.
E en wi h hese app oaches, all de ec ion algo i hms need some da a
o he a ge dis ibu ion o ain on, and i is o en di icul o collec
a good amoun o labelled images ha ca ch he ac ual dis ibu ion
a iabili y.
A good example o a c op wi h a wide ange o a ie al a iabili y is
he g ape ine. Wine and able g apes a e di e en in sizes and bunch
s uc u e, and he ines a e ained in di e en ways. E en be ween
di e en a ie ies o each ype o g apes he a iabili y in size, shape,
colou , oliage and ine s uc u e makes de ec ion ela ed asks di icul
o gene alize. Wi h espec o hese conside a ions, a numbe o wo ks
a e ele an o ou discussion. Ea ly app oaches o g ape de ec ion
and coun ing a e cha ac e ized by he use o ine- uned hand c a ed
ea u es. Fo example, an ea ly app oach o g ape de ec ion is p esen ed
in Sk abanek and Maje ík (2016). He e he au ho s use his og am o
o ien ed g adien s (HOG) desc ip o s oge he wi h a Suppo Vec o
Machine o build a whi e wine g ape de ec o . An app oach ha builds
on hese ea ly esul s and da a is he one p esen ed by Pé ez-Za ala
e al. (2018). The au ho s use again a solu ion based on hand c a ed
ea u es, i.e. HOG, as adial symme y ans o m (FRST) and linea
bina y pa e ns (LBP), o eed a suppo ec o machine (SVM) based
de ec o , and use geome ical conside a ions o sepa a e sel -occluding
g ape bunches. The yield es ima ion ask is hen a esul o he com-
pu a ion o he numbe o be ies de ec ed. Bo h hese solu ions show
some obus ness o colou and illumina ion a iabili y, bu equi e a
good deal o uning o he algo i hms, which limi s he euse o ained
sys ems o o he a ie ies. Ano he app oach o g ape yield es ima ion
ha is based on geome ical conside a ions is ha o Liu e al. (2017),
whe e he de ec ion is done on he ea ly s age buds ha shoo om he
b anches in an unsupe ised ashion only by using Gaussian i ing. The
ad an age o his me hod is he independence om labelled da a and
he eliance on a simple came a as inpu . In his sense, his wo k is close
o ou s, bu he app oach is usable only in he ea ly s age season and
does no ake in o accoun he yield loss o mal o med and diseased
g ape bunches. We conside his kind o app oach complemen a y o
ou s, since i can be used o ha e ea ly season p edic ion ha can be
e ined la e by p ope de ec ion based me hods.
Mo e ecen ly, a good numbe o deep lea ning based de ec ion
algo i hms ha e been p oposed. Palacios e al. (2022) p esen an ea ly
yield p edic ion sys em based on be y coun ing using a SegNe seg-
men a ion ne wo k o ex ac ea u es om he bunches and canopy
images, such as he numbe o es ima ed isible be ies, o he a io o
lea es in he de ec ion bounding boxes. The s a is ics on he a iabili y
o he ea u es o he 6 g ape ine a ie ies s esses how c oss- a ie y
dis ibu ion shi s a e signi ican o lea ning echniques. The au ho s
show a no malized oo mean squa ed e o (RMSE) o 23.83% on hei
da a be ween he coun ed and eal numbe o be ies. Howe e , he
whole sys em is s ill un a nigh wi h di ec illumina ion. Ano he be y
de ec ion sys em based on CNNs is p esen ed in Zabawa e al. (2019)
and in he ex ended wo k (Zabawa e al.,2020). In hese wo ks, he
au ho s each a e y good 94.0% and 85.6% o he wo g ape ine
aining sys ems conside ed, bu again he images we e collec ed wi h
a modi ied s addle g ape ine ha es e o illumina ion cons ancy.
This solu ion is hen no applicable o, o example, able g apes, o
which a s addle machine is non iable. In Co iello e al. (2020), a
coun ing ne wo k inspi ed by densi y based c owd coun ing echniques
was p esen ed. The au ho s show e y in e es ing esul s, wi h mean
a e age e o (MAE) anging om 0.85% o Pino G is a ie y, o
11.73% o Ma zemino a ie y. To achie e o hese esul s, hey had
o label mo e han 35000 be ies, a e y ime consuming ope a ion.
All hese app oaches ha e he common cha ac e is ics o le e aging
deep lea ning o inc eased pe o mances, bu limi ing he a iabili y
o he p oblem by using speci ic machines o huge quan i ies o labelled
da a. These wo aspec s limi he applicabili y o such echniques o he
speci ic case o which hey we e concei ed.
The a o emen ioned conside a ions inspi ed us o explo e me h-
ods ha could help in aining de ec ion and ins ance segmen a ion
algo i hms wi h ew labelled da a. We explici ly conside he case
whe e a small amoun o labelled da a om a simila cul i a ion has
been collec ed and labelled (Sou ce Da ase , SD), bu which is no
enough o ge accep able de ec ion and segmen a ion pe o mances on
a di e en o cha d wi h consis en co a ia e dis ibu ion shi (Ta ge
Da ase , TD). We use as ou es a ge da a example able g ape
ineya ds cul i a ed in Ap ilia, sou he n Lazio, while ou sou ce da ase
is he Emb apa Wine G ape Ins ance Segmen a ion Da ase (WGISD)
p esen ed in San os e al. (2020).
We p esen a combina ion o weakly and semi supe ised echniques
ha a e able o signi ican ly inc ease he pe o mance o he algo i hms
and we compa e hese newly ained algo i hms wi h he s a e-o -
he-a app oaches on he example applica ion o acking ui s o
yield es ima ion. The p oposed solu ion is able o p oduce pseudo
labelled da a in o de o b idge he gap o co a ia e shi s ha occu
whene e a new speci ic c op becomes he a ge o a compu e ision
sys em o p ecision ag icul u e. We explici ly ackle he p oblem o
doing so wi h limi ed ha dwa e and so wa e esou ces, o add ess he
needs o small and medium businesses. Fo his eason, all he pseudo
labelling s a egies p esen ed a e based on simple ideos collec ed wi h
a cellphone came a.
Wi h his in mind, he speci ic pseudo labelling s a egies we p o-
pose a e o wo kinds:
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
3
T.A. Cia uglia e al.
Fig. 1. An example o he ou a ie ies p esen in he expe imen al ield. The Black Pizzu ello (d) is he mos in e es ing o his wo k because i p esen s he highes a iabili y
in shape and colou wi h espec o o he ounded be y a ian s.
•Au oma ic bounding boxes gene a ion o objec s con ained in
consecu i e ideo ames, based on a s a ing es ima e and 3D
s uc u e geome ical conside a ions. We show ha , le e aging a
simple ini ial labelling – which could be manual o au oma ic –
and he in o ma ion ha we can ge om ea u e ma ching and
s uc u e om mo ion, we a e able o gene a e new labelled da a
ha g ea ly inc eases he pe o mance o he de ec o .
•Pseudo mask gene a ion o ins ance segmen a ion: we show how,
s a ing om a simple bounding box - which could be he one
au oma ically gene a ed in he p e ious s ep - i is possible o
use a segmen a ion ne wo k oge he wi h a e ining s a egy o
gene a e new mask labels.
Sel -supe ised echniques ha e been equen ly p oposed o sol e he
da a sca ci y p oblem in speci ic scena ios (see, o example, G anland
e al.,2022;Li e al.,2022 and Siddique e al.,2022 o some ecen
de ec ion o segmen a ion app oaches), howe e , o he bes o he
au ho s’ knowledge, his is he i s ime ha a gene al sel -supe ision
echnique o de ec ion and segmen a ion in ag icul u e is p oposed in
o de o ackle a whole ca ego y o p oblems.
The s uc u e o he pape is he ollowing. In Sec ion 2, he pseudo-
label gene a ion sys em (PLG) is de ailed, bo h o he de ec ion (Sec-
ion 2.6) and o he segmen a ion asks (Sec ion 2.7). In addi ion,
a acking ask o yield es ima ion is discussed in Sec ion 2.6.3. In
Sec ion 2.2, we desc ibe he da a collec ed and used, and in Sec ion 2.4
he speci ic me ics o mul iple objec acking (MOT) a e de ined.
Expe imen s and discussion o all hese sys ems a e desc ibed in
Sec ion 3and conclusions a e d awn in Sec ion 4.
2. Ma e ials and me hods
In his Sec ion, we discuss he da a, he gene al a chi ec u e o he
sys em, and he algo i hms on which i is based. We s a by desc ibing
he expe imen al ield whe e he a ge da a has been collec ed, and
hen how i was collec ed, and how i compa es o he sou ce da a ha
was al eady a ailable. Then, we desc ibe he global sys em a chi ec u e
and in oduce i s componen s. The inal sub-sec ions in oduce he
me ics used o ou expe imen al e alua ion, and gi e mo e de ails o
each subsys em.
2.1. Expe imen al ield
The expe imen al ield is loca ed in sou he n Lazio (I aly). The
ineya d is composed o wo plo s app oxima ely 114 m ×51 m
(0,58 ha) and 122 m ×48 m (0,58 ha). Vineya ds a e s uc u ed as a
adi ional ellis sys em called Tendone wi h a wide dis ance be ween
each plan , 3 ×3m2. Plan a ions a e all olde han 3 yea s and so in ull
p oduc ion and heal h, hus ep esen ing a ypical wo king condi ion
o he alida ion o ag onomic ac i i ies such as ui ha es ing o
ine p uning. All s uc u es a e adi ionally co e ed wi h plas ic and
ne o p o ec g apes om ain and hail. The a e age ex ension o each
plo is a ound 1 hec a e and dimensions (leng h and wid h) a e on a e -
age be ween 25 m and 50 m acco ding o plo ex ension and geome y.
The selec ed ineya d in Ap ilia has cu en ly ou di e en able g ape
a ie ies which a e desc ibed in he ollowing: Whi e Pizzu ello, Black
Pizzu ello, Red Globe and Black Magic. Fig. 1 shows some examples
o hese g ape a ie ies while Fig. 2 shows images o he expe imen al
ield as well as he app oxima e ex ension o each g ape a ie y in he
ineya d.
O he ou a ie ies ha we e p esen in he ineya d, Black Magic
was o e y low quali y and hus un ended by he ield owne . Whi e
Pizzu ello is iden ical in shape o he Black one, and he la e has
he same colou as he o me when no ipe. Toge he , whi e and
black Pizzu ello a e a peculia a ie y o he Lazio Region and p esen
he highes a iabili y in shape and colou wi h espec o s anda d
ounded be y a ian s. Fo hese easons, while we collec ed images o
all he a ie ies, we inally concen a ed ou da a labelling e o only
on Black Pizzu ello.
2.2. Da ase
As we men ioned in he in oduc ion, he p oposed sys em deals
wi h he co a ia e shi om a gene ic sou ce da ase o a a ge da ase
ha is ep esen a i e o he images ha could be collec ed on he
ield. We assembled ou a ge da ase wi h wo di e en kinds o
da a. The i s a e ideos eco ded using a mid ange cellphone came a
(Mo oG8 Plus), which simula es a da a collec ion ope a ion ha could
be pe o med by a a me wi h ease. We collec ed ideos mo ing along
he ineya d (i.e. angen ial o he ows), wi hou any equi emen on
dis ance om he ui s o heigh om he g ound. In his wo k, we
use HD (1280 ×720) ideos a 10 Hz wi h a o al o 1469 ames.
Examples o hese ames a e shown in Fig. 3. A sho segmen o 10 s
has been labelled o es use in he case o he acking algo i hm
e alua ion, while he es has been used wi hou labelling hanks o he
semi-supe ised na u e o he sys em. We b ie ly call his a ge ideo
da ase TVid. No e ha we collec ed he images using a neu al g ey
ca d o whi e balance pu poses, bu his is mean o u u e uses, and
i is no a equi emen o he me hods p esen ed in his wo k.
The second kind o da a is composed o s a ic images o Black
Pizzu ello. This da a simula es he images a a me , o a obo , could
collec o pe o m some ag icul u al ac ion on speci ic g ape bunches
(e.g. quali y es ima ion, disease de ec ion, au oma ic ha es ing). The
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
4
T.A. Cia uglia e al.
Fig. 2. (a) Sa elli e iew o he expe imen al ineya d. The image shows he a ie ies ha a e g own on each ow. (b) A pic u e o he ellis (Tendone) s uc u e.
Fig. 3. (a) and (b) Examples o he ideo ames collec ed o ou a ge da ase (TVid): we s ess ha hese a e simple cellphone came a based ideos and a e he only new da a
equi ed o he sys em o be able o p oduce new labels wi hou supe ision. (c) Example o TImg da ase . (d) Example om he sou ce da ase , he WGISD da ase (San os e al.,
2020). I is possible o no e he di e ences in shape, colou and gene al illumina ion condi ions.
da ase consis s o 134 images o 3000 ×4000 esolu ion, collec ed wi h
he same cellphone came a used o he ideos, howe e he op ics
and chip used o ideo and s ill images a e di e en , as is o en he
case wi h cellphones. This is in en ional, since i adds a e y common
sou ce o co a ia e shi ela ed o he de ice and cap u ing mode
(mo ion s s ill images). All he images in his case ha e been labelled
o de ec ion (bounding boxes), while a small subse has also been
labelled o ins ance segmen a ion (70 images), using he Inno escus
labelling applica ion (Inno escus LLC,0000). All hese labels a e used
o alida ion and es ing o he algo i hms desc ibed in his sec ion.
We call his s ill images da ase TImg. Toge he hese da ase s (TVid
and TImg) cons i u e ou Ta ge da ase (TD).
As men ioned abo e, we wo k unde he hypo hesis ha a small
amoun o labelled da a o he same ui exis s, bu ha i has consid-
e able co a ia e shi wi h espec o he TD dis ibu ion. In his wo k,
ou Sou ce Da a is he one p esen ed by San os e al. (2020). Fo he
de ails abou hese da a, he eade can check he ci ed wo k. He e we
gi e a sho summa y o unde line he di e ences be ween SD and TD
in e ms o :
• he g ape a ie ies (wine s able, be y shape and colou )
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
5
T.A. Cia uglia e al.
Fig. 4. This igu e shows he comple e sys em a chi ec u e. The inpu s a e a sou ce da ase ( ed cylinde ) and a ideo collec ed on he ield by he obo o a a me ( i s g een
cylinde , TVid, expanded as a sequence o ames o show he key ame selec ion p ocess). SDe and SSeg (ligh ed blocks) a e he ini ial de ec ion and segmen a ion ne wo ks
ained only on he sou ce da ase . All he in e media e compu ing blocks a e depic ed in o ange, while he in e media e ou pu s a e in blue ci cles. Bo h he pseudo bounding
boxes and pseudo masks p oduced a e depic ed in yellow, while he de ec ion and segmen a ion ne wo ks ained on hese new labels (TDe and TSeg) a e depic ed in ligh g een.
The da a low in he sys em is also colou coded as pe legend.
• he illumina ion condi ions ( ull sun s shadows)
• he came a de ice (Re lex s cellphone came a)
•scale o he images (s anda d scale s a iable scale)
To quan i y he co a ia e shi gap in Sec ion 3, he pe o mance d op
o de ec o s and ins ance segmen a ion ne wo ks ained on SD and
es ed on TD a e gi en.
2.3. Sys em o e iew
An o e iew o he sys em is depic ed in Fig. 4. The main inspi ing
p inciple o his wo k is he economy o da a labelling and da a euse.
Fo his eason, he only wo sou ces o da a a e he sou ce da ase
(da a a ailable om a simila ask) and a ideo collec ed on he a ge
ield. The sou ce da ase is used o ain he ini ial de ec ion and
segmen a ion models, namely he Sou ce De ec o Ne wo k (SDe ) and
he Sou ce Segmen a ion Ne wo k (SSeg). SDe is no pe ec ly uned
on he a ge en i onmen , s ill i can be used on selec ed ames o he
ideo inpu ha we call key ames. To keep his solu ion simple, we
conside equally spaced key ames s a ing om he i s one, bu o he
s a egies could be de ised. A se o ini ial bounding boxes is ex ac ed
om his key ame, using a high con idence h eshold, o limi he alse
posi i es. Then, he whole ideo is passed in a Geome ic Consis ency
block (GC block) ha ex ac s ea u es om each ame and associa es
hem. We es ed wo di e en op ions o his block, as will be shown in
he ollowing sec ions. Using his geome ic in o ma ion, oge he wi h
he ini ial bounding boxes ex ac ed om he key ames, i is possible o
in e pola e he bounding boxes posi ions o he emaining ames wi h
high accu acy. These new bounding boxes a e ou pseudo-labels o
aining he de ec o on he a ge en i onmen , which we call Ta ge
De ec o (TDe ).
The De ec ion Pseudo-Labels Gene a ion (DPLG) sub-sys em could
be used independen ly by he Segmen a ion Pseudo-Labels Gene a ion
(SPLG). To p o e he e ec i eness o he app oach, we compa e he
pe o mance o TDe on he bunches acking p oblem, i.e. coun ing
he numbe o g ape bunches by coun ing he ins ances acked along
a ideo. This p oblem is ele an since i can be used o yield es i-
ma ion pu poses. We es wo di e en acking algo i hms, which a e
desc ibed in Sec ion 2.6.3 and e alua ed in Sec ion 3.2.
The goal o he second pa o he sys em is o gene a e pseudo
masks o aining an ins ance segmen a ion ne wo k. This sub-sys em
can be seen bo h as an independen pseudo label gene a o , o as pa o
a bigge sys em such as he one we desc ibe he e. As men ioned be o e,
he SSeg is ained only on sou ce da a and is no able o p oduce
good segmen a ion masks on he Ta ge Da a. Howe e , i is possible o
gi e he ne wo k some in o ma ion cues ha can g ea ly imp o e he
mask es ima es. The i s one is he bounding box egion in which he
ins ance should be segmen ed. This cue comes easily om he p e ious
s ep o pseudo bounding box gene a ion, bu i could be p oduced
o he wise. This gene a es he ini ial pseudo masks. I is possible o
use hese pseudo masks o aining he Ta ge Segmen a ion Ne wo k
(TSeg) bu his would lead o poo pe o mances due o con i ma ion
bias. We need he e o e o injec ex e nal in o ma ion om o he cues
ha we ha e. In ou sys em, his is he ole o he pseudo masks e ining
block. In Sec ion 2.7.1, h ee di e en solu ions o e inemen will be

Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
6
T.A. Cia uglia e al.
desc ibed. Thanks o hese e ined pseudo masks i is inally possible o
ain he TSeg Ne wo k. Sec ion 3.3 epo s he esul s o he e ining
s a egies and compa es he pe o mance o TSeg wi h SSeg.
Finally, he expe imen s o he whole sys em, ained only on ideos
and es ed o ins ance segmen a ion pe o mance, a e gi en in Sec-
ion 3.4.
2.4. Me ics
In his Sec ion, we desc ibe he me ics used o e alua e and com-
pa e he de ec o s, he acke s and he ins ance segmen a ion al-
go i hms. To e alua e he de ec o s and ins ance segmen a ion algo-
i hms, he s anda d me ics o P ecision, Recall and In e sec ion o e
Union (IoU) ha e been used. In addi ion, o ins ance segmen a ion, he
A e age P ecision, as de ined in he MS COCO challenges (Lin e al.,
2014), has been used.
Usually AP is compu ed o each class and hen a e aged o ob ain
he mean a e age p ecision (mAP). In his wo k, since he e is only one
class (g ape), he AP coincides wi h he mAP. In he MS COCO me ics,
he AP is calcula ed by compu ing he p ecision a e e y ecall le el
om 0 o 1 wi h a s ep size o 0.01. The mAP is hen compu ed by
a e aging he AP o e all he objec ca ego ies and en IoU h esholds
om 0.5 o 0.95 wi h a s ep size o 0.05.
To e alua e he acke s, we ollow he common p ac ice o Mul iple
Objec T acking (MOT) as de ined by Wu and Ne a ia (2006) and
he CLEAR MOT me ics (Be na din and S ie elhagen,2008). MOT
is a di icul ask o e alua e, since he pe o mance me ics should
cap u e bo h he p ecision in de ec ing indi idual ins ances and he
accu acy in acking each ins ance ac oss mul iple ames, wi hou
losing ack o swi ching be ween ins ances. Gi en a numbe o objec s
𝑜𝑗, 𝑗 ∈ [0 … 𝑚], he acke p oduces a numbe o hypo heses ℎ𝑖, 𝑖 ∈
[0 … 𝑛]. The pe o mance o associa ion o hypo hesis and objec s can
be measu ed ame by ame using he classic T ue Posi i e, T ue
Nega i e, False Posi i e and False Nega i e igu es, oge he wi h hei
di ec descendan s P ecision and Recall. Howe e , ecen ly a numbe
o compound indexes ha e been p oposed o be e cap u e he gene al
acke ’s pe o mance. The i s one is he Mul iple Objec T acking
Accu acy (MOTA), de ined as ollows:
𝑀𝑂𝑇 𝐴 = 1 − (𝐹 𝑁 +𝐹 𝑃 +𝐼𝐷𝑠𝑤)
𝐺𝑇 ∈ (−∞,1] (1)
whe e 𝐹 𝑁 and 𝐹 𝑃 a e False Nega i es and False Posi i es, 𝐼𝐷𝑠𝑤
ep esen s he numbe o ins ances whose ID has been e oneously
swi ched, GT is he eal numbe o ins ances in he ideo. This index
accoun s o h ee sou ces o e o , namely he alse posi i e a io, he
alse nega i e a io and he misma ch a io. Toge he , hey gi e an idea
o he gene al acking accu acy. To e alua e he p ecision, a second
index was p oposed:
𝑀𝑂𝑇 𝑃 =∑𝑡,𝑖 𝑑𝑡,𝑖
∑𝑡𝑐𝑡
(2)
whe e 𝑐𝑡deno es he o al numbe o ma ches in ame 𝑡, and 𝑑𝑡,𝑖 in
gene al ep esen he dis ance o he hypo hesis and he objec , bu in
ou case can be compu ed as he o e lap o he g ound u h and he
hypo hesis bounding boxes. This second index gi es only a measu e o
he p ecision in de ec ing he ins ances wi hou gi ing any in o ma ion
on he acking and associa ion capabili y.
2.5. De ec ion and segmen a ion ne wo k a chi ec u es
As explained in Sec ion 2.3, he gene al pseudo label gene a ion
sys em is based on p e- ained de ec ion and segmen a ion ne wo ks
(SDe and SSeg) and is mean o p oduce he pseudo labelled da a o
ain new ne wo ks ha a e able o pe o m be e on TD (TDe and
TSeg).
The main pa ame e s ha in luence he choice o he a chi ec u es
a e speed and accu acy. I is well known (Liu e al.,2020) ha So A
de ec ion ne wo ks can be di ided in o wo main ca ego ies: wo s age
and single s age. The i s kind sepa a es de ec ion in o wo phases,
he i s is called egion p oposal and gi es objec bounding boxes
candida es, while he second il e s and e ines hese candida es o
p oduce he bounding boxes and classi ies he objec s. The second kind
ins ead ex ac s bo h egion p oposal and class p edic ion in one pass.
The main ad an age o he single s age de ec o s is speed, which is
much highe han he wo s age one, bu a he cos o a gene al educed
accu acy. The main examples o single s age de ec o s a e he YOLO
a ian s, in pa icula he ecen YOLO 5 (Redmon and Fa hadi,2018).
One o he bes known and bes pe o ming wo s age a chi ec u es is
Mask R-CNN (He e al.,2017), which is also a segmen a ion ne wo k,
mo e accu a e han any YOLO a ian s, bu slowe and di icul o
weak o eal- ime use.
In his wo k, we use he single s age YOLO 5s a chi ec u e o
he expe imen s on acking, since eal- ime de ec ion is needed o
his kind o applica ion. In addi ion, some o he a ian s ha e a
small numbe o pa ame e s, which makes hem iable o embedded
applica ions, such as obo ic ha es ing. The pseudo bounding box
gene a ion could be pe o med o line, hus allowing use o he be e
pe o ming Mask R-CNN, bu we decided o use he YOLO de ec o o
keep his sub sys em sel -con ained. In addi ion, using an a chi ec u e
wi h lowe de ec ion pe o mance s esses and es s he obus ness o
he gene a ion p ocess. Fo segmen a ion and pseudo mask gene a ion,
a segmen a ion ne wo k is needed, so he choice alls on Mask R-
CNN. The de ails o he p e aining and ine uning o he de ec ion
and segmen a ion ne wo ks a e gi en in Sec ions 3.1.1 and 3.3.1,
espec i ely.
2.6. De ec ion pseudo-label gene a ion sub-sys em and acking applica ion
In his Sec ion, we de ail he elemen s o he DPLG sys em depic ed
in Fig. 5,i.e. he pseudo bounding box gene a ion sys em, oge he
wi h he acking algo i hm used o yield es ima ion as a possible
applica ion.
2.6.1. Geome ic consis ency block
The pu pose o his block is o use geome ical co espondences
ex ac ed h ough epipola geome y o associa e g ape ins ances in
di e en ames o a ideo s eam, i.e. , gi en a de ec ed g ape bunch,
by iden i ying 2D ea u es belonging o i and ma ching, o iangula -
ing, hem ac oss mul iple ames, i is possible o ind he same bunch
ins ance in he ollowing ames. We use his s a egy in wo ways in
his wo k, i s o ex ac pseudo bounding boxes, and hen o acking.
In his Sec ion, we desc ibe he gene al unc ional p inciples o S M
algo i hms and hei compu a ional cos s.
We expe imen ed wi h wo app oaches, he i s is he same used
in San os e al. (2020), which le e ages a S M so wa e applica ion,
namely COLMAP (Schönbe ge e al.,2016;Schönbe ge and F ahm,
2016). Since S M is a well known p oblem, he in e es ed eade can
ind de ails o he solu ions in Ha l ey and Zisse man (2006) and
Szeliski (2022). In b ie , we used he COLMAP modali y ha ex ac s
spa se ea u es om each ame and hen uns a sequen ial all e sus
all sea ch and ma ching o he ea u es ex ac ed om he ideo. These
co espondences a e hen used o iangula e he 3D poin s by minimiz-
ing he 3D o 2D ep ojec ion e o . Howe e , he na u e o he p oblem
is such ha e en wi h he spa se se ing, he compu a ional cos s
inc ease exponen ially wi h he numbe o ames. Ou expe imen s
equi ed 5 h o compu a ion o ideos o 500 o 600 ull HD ames,
on a compu e equipped wi h an In el-Co e i7 3.4 GHz, a N idia GTX
950 m and 16 GB o memo y.
The second app oach we expe imen ed add esses his aspec in
o de o ha e a eal ime solu ion ha can be un on an online acke ,
such as he one ha will be desc ibed in Sec ion 2.6.3. The idea is ha
in ou con ex a ull S M solu ion (i.e. , he 3D posi ion o he ex ac ed
2D ea u es in a wo ld e e ence ame) is no needed, since he kind
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
7
T.A. Cia uglia e al.
Fig. 5. This igu e shows he de ec ion pseudo-label gene a ion (DPLG) sub-sys em alone. The sou ce da ase is used o ain an ini ial coa se bounding box de ec o (SDe ) ha
is hen used, oge he wi h he S M sys em, o gene a e a la ge numbe o new labelled images om he ames o con inuous ideos o he ineya d. This same sys em can be
applied o o he ui s wi h ela i e simplici y.
Fig. 6. Fea u e ma ching and geome ic e i ica ion using RANSAC: COLMAP S M in i s ligh es o m s ill equi es conside able compu a ional ime and i is iable only o o line
elabo a ions, while 2D ea u e ma ching equi es much less compu a ion and po en ially uns in eal ime. (a) The ma ching o su ea u es wi h b u e o ce ma ching (b) he
same ma ching e ined wi h homog aphy compu a ion combined wi h RANSAC selec ion.
o ideos ha a e collec ed in he ineya d a e simple walks wi hou
closed loops. This means ha each able g ape bunch is p esen , a
mos , in a ew consecu i e ames, excep o he occasional occlusion.
Fo his eason, we ound ha ex ac ing 2D ea u es om a ame 𝑖and
om a small numbe o subsequen ames 𝑖+1,…, 𝑖+𝑛, and hen ma ch-
ing hem was enough o map he g ape ins ance co espondences along
he ideo s eam. The ea u es and desc ip o s used a e SURF (Bay
e al.,2006), while he ma ching is a simple b u e o ce (all- s-all)
dis ance compu a ion be ween he co esponding ea u e desc ip o s.
Since hese ini ial ma ch p oposals con ain some misma ched ea u es,
a RANSAC e i ica ion s ep is used o il e hem ou . Gi en ha o
small came a mo ions, image ans o ma ion could be app oxima ed
by a homog aphy ans o ma ion, we epea edly andom sample ou
ma ches, compu e he ela i e homog aphy, and check how many o he
ma ches a e co ec ly p edic ed by i . The homog aphy wi h he highes
consensus is selec ed and all ma ches whose displacemen is no com-
pa ible wi h he selec ed homog aphy, a e disca ded. An example o
his p ocess is gi en in Fig. 6. The pa allel lines le in Fig. 6(b) show he
ma ches ha ag ee wi h he homog aphy es ima ed h ough RANSAC.
Using his app oach, e en using b u e o ce ma ching, we we e able
o each, wi hou pa icula op imiza ions and wo king only on CPU, 3
ames pe second on he a o emen ioned ideo and ha dwa e.
2.6.2. Bounding box in e pola ion and pseudo label gene a ion
Bounding box in e pola ion can be be e unde s ood by looking a
Fig. 7.
S a ing om a bounding box ound by SDe a ame 𝑖, hanks o he
GC Block, i is possible o ha e an associa ion be ween he 2D ea u es
con ained inside he box wi h some ea u es in ame 𝑖+𝑛. Since he
came a is mo ing, bo h he posi ion o he g apes and he illumina ion
condi ions in ame 𝑖+𝑛will be di e en , consequen ly he ea u es
ma ched will ha e a di e en posi ion. The ques ion is hen how o
d aw he new bounding box in ame 𝑖+𝑛. We use he hypo hesis, ha
he came a is slowly mo ing, and ha he mo ion is angen ial o he
di ec ion o he ineya d. Thanks o his hypo hesis we can assume ha
he new bounding box will ha e he same size as he one ound in ame
𝑖.
The posi ion o he new bounding box is compu ed by se ing he
cen e o he box o coincide wi h he cen e o g a i y o he ea u es
in ame 𝑖+𝑛, as depic ed in Fig. 7(a). Ano he aspec o conside in
e alua ing he pseudo bounding box gene a ion scheme is he e ec
o came a eloci y combined wi h ame a e. I he ame a e o
he ideo is high, o he came a eloci y is low, he change in iew
will be minimal, and consequen ly he in o ma ion added by such
a sample will be mino . Fo his eason, we conside ed i use ul o
explo e he e ec o he a io be ween key ames and o he ames.
We call his pa ame e skip alue, since i is he numbe o ames in
which he bounding boxes p edic ed in ame 𝑖a e in e pola ed, be o e
aking a new p edic ion by SDe . Ou abla ion expe imen s showed
ha using skip 1 (i.e. using only SDe o p oduce pseudo-labels) ga e
lowe pe o mance han using skip 2. Howe e , inc easing he skip
alue seems no o gi e mo e ad an ages. This aspec is explo ed in
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
8
T.A. Cia uglia e al.
Fig. 7. The bounding box in e pola ion p ocess. (a) Shows he upda ing p inciple: he bounding box (blue) p edic ed in ame 𝑖is mo ed o ame 𝑖+𝑛; while he size emains
he same, he posi ion o he new bounding box (o ange) is upda ed by compu ing he new cen e o g a i y o he ea u es ex ac ed and making i coincide wi h he box cen e.
(b) Pseudo-labels gene a ed by means o he S M algo i hm: he g een boxes a e he p edic ions p oduced by SDe a ame 𝑖 ansposed in he cu en one (𝑖+𝑛), while he ed
boxes a e in e pola ed ones acco ding o he ea u es ma ched ( ep esen ed as ed poin s).
Fig. 8. This igu e shows he segmen a ion pseudo-label gene a ion (SPLG) sub-sys em.
Sec ion 3.2, whe e we show on he acke applica ion he esul s o
using di e en skip alues.
2.6.3. T acking o yield es ima ion
Mul i Objec T acking o he g ape bunch ins ances is a p elimina y
s ep in yield es ima ion, as i is possible o es ima e he numbe o
bunches by coun ing he numbe o ajec o ies acked by he algo-
i hm. The main app oach o acking by de ec ion ha we conside is
he one p esen ed by San os in San os e al. (2020) which is based on
de ec ion and S M. Howe e , no me ics we e gi en he e o o mally
desc ibe he pe o mances o he app oach. The e o e, we eplica ed
he expe imen s and compu ed he me ics using as a a ge he es
sequence o TVid, desc ibed in Sec ion 2.2 and depic ed in Fig. 3. In
addi ion, we es ed ano he de ec ion based S a e-o - he-A acke ,
DeepSORT (Wojke e al.,2017), designed o wo k in eal- ime using
a deep associa ion me ic. We chose his acke since he compu a ion
in ol ed in es ima ing e en spa se co espondences be ween he ames
using COLMAP (Schönbe ge and F ahm,2016) equi es conside able
ime and a e no easible o edge o obo ic de ices. In addi ion, he
compu a ion o he ull S M solu ion akes a long ime and limi s he
leng h o he ideo o a ew hund ed ames, while o he second
app oach he e is no such limi . In Sec ion 3, we compa e he acking
solu ions using he MOT me ics wi h he acking g aphs o gain mo e
insigh s on wha he acke does and how o imp o e i u he . An
example o hese g aphs is gi en in Fig. 15.
2.7. Segmen a ion pseudo-label gene a ion sub-sys em
While de ec ion is enough o coun ing asks, o quan i a i e yield
es ima ions o o asks ha equi e a physical in e ac ion wi h he
olumes o he g apes (e.g. ha es ing), segmen a ion, and in pa ic-
ula ins ance segmen a ion, is equi ed. Ins ance segmen a ion equi es
labels ha a e ideally pixel pe ec masks, howe e Bellocchio e al.
(2019) showed how, e en wi h minimal labelling signal (e.g. p esence
o absence o an objec in a image), he ask ne wo k is able o lea n
ep esen a ions ha a e close o masks o he objec o in e es . Fo his
eason, we again adop a pseudo-labelling app oach o his p oblem,
s a ing wi h a p e ained ne wo k on he WGISD sou ce da ase and
hen using simple ex e nal cues o wo k as ou ex e nal in o ma ion
signal ha helps in e ining he label. The o e iew o his sub-sys em
is depic ed in Fig. 8.
Ou SSeg ne wo k is Mask R-CNN ained on WGISD, as usual. Mask
R-CNN in i s basic o m ex ac s egion p oposals and uses hem o
p edic bounding boxes and ins ance segmen a ion masks. Howe e ,
i is possible o use he segmen a ion subne wo k o Mask R-CNN
as he pseudo mask ini ial gene a o . In pa icula , Mask R-CNN is
wi ed di e en ly a in e ence ime han a aining ime, since he
bounding boxes p edic ed by he de ec ion head a e di ec ly ed o he
mask head. The ne wo k will use his bounding box as a cue, o as
an a en ion mechanism, which helps he segmen a ion subne wo k o
ou pu a use ul pseudo mask. This is depic ed in Fig. 9. This s a egy
will mi iga e he p oblem o con i ma ion bias, since he box comes
om an ex e nal in o ma ion sou ce. In ou sys em, he bounding boxes
could come om he ou pu o he DPLG sub-sys em. A he same ime,
in Sec ion 3we show he pe o mances o he pseudo mask gene a ion
s a ing om g ound u h bounding boxes so as o be e isola e he
pe o mance con ibu ion o he mask gene a ion p ocess. In his way,
he numbe o pseudo-masks will coincide wi h he ac ual numbe o
g ape clus e s in he image, and he measu ed e o will only be due o
he mask gene a ion p ocess. The quali a i e di e ence o segmen a ion
mask be ween he s anda d wi ing o he Mask R-CNN ne wo k, and he
one wi h an ex e nal a en ion mechanism, is shown in Fig. 10.
2.7.1. Pseudo mask e ining block
To e ine he pseudo masks, in o de o educe o emo e con-
i ma ion bias, an ex e nal sou ce o in o ma ion is needed. Some
Compu e s and Elec onics in Ag icul u e 205 (2023) 107624
9
T.A. Cia uglia e al.
Fig. 9. Mask-RCNN in e nal wi ing a aining and in e ence imes. A aining ime,
he mask p edic ion head uses he same inpu s as he o he wo heads, i.e. he RoI
c opped ea u es. A in e ence ime, he c opping is done only using he bounding
boxes p oposals o he bounding box p edic ion head. Ou sys em uses only he
ea u e ex ac ion pa and d ops he bounding box eg ession, using ins ead ei he
he bounding boxes coming om g ound u h o he bounding boxes pseudo-labels
gene a ed by he DPLG sub-sys em. The dashed blue line in he lowe diag am shows
whe e he wi e is cu and ou bounding boxes p oposals a e injec ed.
Fig. 10. Le image: pseudo mask p oduced by Mask-RCNN ained only on he Sou ce
Da ase and wi hou a bounding box cue. Righ image: same image showing he e ec
o gi ing a bounding box cue a in e ence ime.
ea lie wo ks wo ked on his aspec , such as Kho e a e al. (2017). We
ied h ee di e en s a egies o e ine he ini ial masks, using simple
compu e ision echniques ha wo k on di e en p inciples om he
con olu ional il e s con ained in SSeg and ha use simple geome ical
conside a ions.
•Dila ion: he i s me hod o igina es om he obse a ion ha
SSeg ends o unde es ima e he masks on he a ge da a. Fo
his eason, a simple mo phological dila ion ha expands he
mask un il i ouches he e e ence bounding box is able o add
aluable in o ma ion o he label. The dila ion is applied wi h a
5×5 ci cula -shaped ke nel. An example o he esul is gi en in
Fig. 11(a).
•SLIC: Simple Linea I e a i e Clus e ing (SLIC) (Achan a e al.,
2012) is a me hod o supe -pixel segmen a ion o he image.
Supe -pixels a e con iguous egions o he image ha a e clus-
e ed oge he by a KMeans algo i hm unning on bo h colou
and space (5-dimensional). We apply his supe -pixel di ision o
he en i e image and compa e i wi h each pseudo mask. The
SLIC algo i hm ha was used was he one implemen ed in he
Py hon sciki -image lib a y (Van de Wal e al.,2014) wi h
2000 segmen s and compac ness 0.1. All he supe pixels ha a e
co e ed by mo e han an uppe h eshold 𝑡𝑢= 70% a e added
o he mask, while all he pixels ha a e co e ed by less han
a lowe h eshold 𝑡𝑙= 30% a e emo ed om he mask. The
a ionale is ha in his way we should be able o emo e also
he backg ound pixels e oneously con ained in he ini ial pseudo
mask. An example o he esul is gi en in Fig. 11(b).
•G ub Cu : his is an i e a i e segmen a ion echnique in oduced
by Ro he e al. (2004). I ep esen s he image as a g aph whe e
o eg ound and backg ound pixels a e modelled as Gaussian Mix-
u e Models and ha e o be sepa a ed i e a i ely by cu s o he
g aph edges. We used he OpenCV (B adski,2000) implemen a-
ion whe e i is possible o ini ialize he algo i hm wi h he pseudo
mask de ining ou pixel ca ego ies, i.e. su e o eg ound, su e
backg ound, p obable o eg ound, and p obable backg ound. The
pseudo mask is used as p obable o eg ound. Dila ion is applied
o he pseudo mask o a numbe o i e a ions p opo ional o he
smalles dimension o he e e ence bounding box o ob ain he
p obable backg ound. E osion is applied o he same numbe o
i e a ions o ob ain he su e o eg ound, while he es is se o
su e backg ound. A sample o he e ec s o G ab Cu is shown in
Fig. 12.
In Sec ion 3, we show how each o hese e inemen me hods
pe o ms compa ed o he baseline (pseudo mask wi h no e inemen ).
3. Resul s and discussion
3.1. De ec ion expe imen s
In his Sec ion, we desc ibe he esul s o he de ec ion expe imen s.
Table 1 shows ou p elimina y expe imen s o compa e di e en e -
sions o he de ec o . All models in his ini ial compa ison a e ained
and es ed on WGISD. Resul s show ha he models wi h a la ge
numbe o pa ame e s o e a minimal pe o mance inc ease on basic
de ec ion, compa ed o he ligh weigh e sions S and N. This can be
explained by a gene al homogenei y o he dis ibu ion o he g ape
images in WGISD, which does no equi e huge numbe o pa ame e s
o lea n a good es ima o . This is expec ed, since in ag icul u e we do
no wo k wi h huge amoun s o da a. Fo his eason, we decided o
base all he acke s on he S and 𝑁 a ian s o educe o e i ing.