Weakly and semi-supervised detection, segmentation and tracking of table grapes with limited and noisy data

Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 A ailable online 17 Janua y 2023 0168-1699/© 2024 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/). Con en s lis s a ailable a ScienceDi ec Compu e s and Elec onics in Ag icul u e jou nal homepage: www.else ie .com/loca e/compag O iginal pape s Weakly and semi-supe ised de ec ion, segmen a ion and acking o able g apes wi h limi ed and noisy da a✩ Thomas A. Cia uglia a,∗,1, Ionu M. Mo oia,1, Leona do Sa aceni a,1, Mulham Fawakhe ji a, Albe o San eliu b, Daniele Na di a aDepa men o Compu e , Con ol and Managemen Enginee ing (DIAG) ‘‘An onio Rube i’’, Sapienza Uni e si y o Rome, ia A ios o 25, Rome, 00185, I aly bIns i u de Robo ica i In o ma ica Indus ial, CSIC-UPC, Calle Llo ens i A igas 4-6, 08028, Ba celona, Spain ARTICLE INFO Keywo ds: F ui de ec ion and segmen a ion Yield p edic ion Compu e ision Deep lea ning Sel -supe ised lea ning ABSTRACT De ec ion, segmen a ion and acking o ui s and ege ables a e h ee undamen al asks o p ecision ag icul u e, enabling obo ic ha es ing and yield es ima ion applica ions. Howe e , mode n algo i hms a e da a hung y and i is no always possible o ga he enough da a o apply he bes pe o ming supe ised app oaches. Since da a collec ion is an expensi e and cumbe some ask, he enabling echnologies o using compu e ision in ag icul u e a e o en ou o each o small businesses. Following p e ious wo k in his con ex (Cia uglia e al.,2022), whe e we p oposed an ini ial weakly supe ised solu ion o educe he da a needed o ge s a e-o - he-a de ec ion and segmen a ion in p ecision ag icul u e applica ions, he e we imp o e ha sys em and explo e he p oblem o acking ui s in o cha ds. We p esen he case o ineya ds o able g apes in sou he n Lazio (I aly) since g apes a e a di icul ui o segmen due o occlusion, colou and gene al illumina ion condi ions. We conside he case in which he e is some ini ial labelled da a ha could wo k as sou ce da a (e.g. wine g ape da a), bu i is conside ably di e en om he a ge da a (e.g. able g ape da a). To imp o e de ec ion and segmen a ion on he a ge da a, we p opose o ain he segmen a ion algo i hm wi h a weak bounding box label, while o acking we le e age 3D S uc u e om Mo ion algo i hms o gene a e new labels om al eady labelled samples. Finally, he wo sys ems a e combined in a ull semi-supe ised app oach. Compa isons wi h s a e-o - he-a supe ised solu ions show how ou me hods a e able o ain new models ha achie e high pe o mances wi h ew labelled images and wi h e y simple labelling. 1. In oduc ion De ec ion and acking o ui s and ege ables a e wo undamen al asks o p ecision ag icul u e, enabling obo ic ha es ing and yield es ima ion applica ion. As wi h any o he au oma ion ask, de ec ion o ege ables bene i s om con olled en i onmen s and well known ield condi ions. The educ ion o a iabili y and unce ain y in ui posi ion, occlusions, a ie y, illumina ion, o ci e a ew aspec s o he p oblem, ha e a huge impac on he success ul implemen a ion o a lea ning based de ec ion sys em. Fo his eason, many de ec ion based sys ems a e designed wi h he aim o educing he sou ces o a iabili y. Fo example, Wang e al. (2013) and Nuske e al. (2014) p oposed de ec ion sys ems o ui coun ing and yield es ima ion using a di ec illumina ion de ice o con ol ambien ligh . In bo h cases, he p oposed sys ems need o be un a nigh o be unc ional. In P e o e al. (2021), ✩This wo k ex ends he one i led ‘‘Pseudo-label Gene a ion o Ag icul u al Robo ics Applica ions’’ p esen ed a he 3 d In e na ional Wo kshop on Ag icul u e-Vision, CVPR 2022, New O leans. ∗Co esponding au ho . E-mail add ess: [email p o ec ed] (T.A. Cia uglia). 1The au ho s con ibu ed equally o he wo k. an example o a s addle obo ic pla o m is gi en, which is ano he common way o con ol he en i onmen al ligh and emo e came a in insics a iabili ies. While hese app oaches a e iable, hey a e also di icul o adap o di e en cul i a ions and equi e conside able economical and ech- nical in es men , which is o en beyond he capaci ies o small and medium ag icul u al businesses, which a e o en amily based. While he economic p oblems hey ace a e gene ally he same as hose o bigge companies (e.g. lack o manpowe o ha es ege ables), hey do no ha e he economic s eng h o knowledge o enginee he cul i- a ion om he g ound up o hea ily au oma ed p ocesses. This means ha ha ing mo e lexible app oaches ha a e mo e algo i hmic and da a o ien ed han ha dwa e o ien ed would posi i ely impac hese h ps://doi.o g/10.1016/j.compag.2023.107624 Recei ed 26 Augus 2022; Recei ed in e ised o m 1 Janua y 2023; Accep ed 2 Janua y 2023 Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 2 T.A. Cia uglia e al. businesses, allowing a ange o possible applica ions o da a d i en p ecision ag icul u e. In his espec , mode n compu e ision, mos ly based on deep lea ning algo i hms, has lowe ed he ini ial in es men needed o in e- g a e ad anced de ec ion echniques o moni o ing and managing he c ops. Howe e , as discussed by Koi ala e al. (2019) in hei su ey on deep lea ning echniques applied o ui coun ing, hese algo i hms a e da a hung y and ga he ing he co ec and igh amoun o da a is no always s aigh o wa d. One way o ace da a sca ci y is he use o algo i hmic echniques o semi-supe ised, weakly-supe ised and ans e lea ning, whe e addi ional in o ma ion is added o he aining p ocess as an ex e nal aining signal, o by le e aging wha was lea ned in a di e en bu ela ed ask. An example o hese app oaches in ag icul u e is gi en by Bellocchio e al. (2019) whe e he au ho s p opose an oli e coun ing solu ion ha is explici ly ained wi h weak labels and consis ency losses. The auxilia y signal in his case is he labelling ob ained by an ex e nal classi ie ha de ec s whe he o no he e a e oli es in he pic u e. This wo k is close o ou s o he ocus on wo king on da a wi h minimal labelling. Howe e , i is based on simple di ec ui coun ing, which can lead o huge e o s in cases whe e sel occlusion is ypical. An example o ans e lea ning ha can be used o educe he amoun o da a needed o aining is shown in Gülden ing and Nalpan idis (2021). In his s udy, he au ho s show how p e- aining using con as i e lea ning as an unsupe ised echnique is able o im- p o e he pe o mances o de ec ion algo i hms compa ed o s anda d ImageNe p e- aining. This esul is a good s a ing poin o aining a deep ne wo k, bu some ask speci ic aining da a is s ill needed. In his con ex , i is in e es ing o conside he challenges ha de ec ion and acking algo i hms ace when he ield is no p epa ed o au oma ion. Some o hese challenges a e: une en dis ibu ion o ege ables in he ield, in a-species a iabili y, illumina ion, occlusion and clu e . F om a echnical poin o iew, all hese aspec s ansla e o co a ia e shi s and lack o labelled samples. Fo example, a s udy on he in a-species co a ia e shi o swee peppe s and i s impac on de ec ion and segmen a ion algo i hm is gi en in Hals ead e al. (2020). The au ho s explo e he issue o gene alizabili y by conside ing a ui ha is g own using di e en cul i a s and in di e en en i onmen s ( ield s glasshouse). Thei esul s show how in single ask lea ning he pe o mances d op signi ican ly, as low as 0.323 o F1-sco e, on c oss da ase de ec ion, and only by se ing up a mul i- ask lea ning p oblem hey a e able o inc ease his sco e by a good ma gin, hanks o he mul iple back p opaga ion signals. Le e aging he c oss- ask co ela ions can be seen in i sel as a o m o sel -supe ised lea ning. E en wi h hese app oaches, all de ec ion algo i hms need some da a o he a ge dis ibu ion o ain on, and i is o en di icul o collec a good amoun o labelled images ha ca ch he ac ual dis ibu ion a iabili y. A good example o a c op wi h a wide ange o a ie al a iabili y is he g ape ine. Wine and able g apes a e di e en in sizes and bunch s uc u e, and he ines a e ained in di e en ways. E en be ween di e en a ie ies o each ype o g apes he a iabili y in size, shape, colou , oliage and ine s uc u e makes de ec ion ela ed asks di icul o gene alize. Wi h espec o hese conside a ions, a numbe o wo ks a e ele an o ou discussion. Ea ly app oaches o g ape de ec ion and coun ing a e cha ac e ized by he use o ine- uned hand c a ed ea u es. Fo example, an ea ly app oach o g ape de ec ion is p esen ed in Sk abanek and Maje ík (2016). He e he au ho s use his og am o o ien ed g adien s (HOG) desc ip o s oge he wi h a Suppo Vec o Machine o build a whi e wine g ape de ec o . An app oach ha builds on hese ea ly esul s and da a is he one p esen ed by Pé ez-Za ala e al. (2018). The au ho s use again a solu ion based on hand c a ed ea u es, i.e. HOG, as adial symme y ans o m (FRST) and linea bina y pa e ns (LBP), o eed a suppo ec o machine (SVM) based de ec o , and use geome ical conside a ions o sepa a e sel -occluding g ape bunches. The yield es ima ion ask is hen a esul o he com- pu a ion o he numbe o be ies de ec ed. Bo h hese solu ions show some obus ness o colou and illumina ion a iabili y, bu equi e a good deal o uning o he algo i hms, which limi s he euse o ained sys ems o o he a ie ies. Ano he app oach o g ape yield es ima ion ha is based on geome ical conside a ions is ha o Liu e al. (2017), whe e he de ec ion is done on he ea ly s age buds ha shoo om he b anches in an unsupe ised ashion only by using Gaussian i ing. The ad an age o his me hod is he independence om labelled da a and he eliance on a simple came a as inpu . In his sense, his wo k is close o ou s, bu he app oach is usable only in he ea ly s age season and does no ake in o accoun he yield loss o mal o med and diseased g ape bunches. We conside his kind o app oach complemen a y o ou s, since i can be used o ha e ea ly season p edic ion ha can be e ined la e by p ope de ec ion based me hods. Mo e ecen ly, a good numbe o deep lea ning based de ec ion algo i hms ha e been p oposed. Palacios e al. (2022) p esen an ea ly yield p edic ion sys em based on be y coun ing using a SegNe seg- men a ion ne wo k o ex ac ea u es om he bunches and canopy images, such as he numbe o es ima ed isible be ies, o he a io o lea es in he de ec ion bounding boxes. The s a is ics on he a iabili y o he ea u es o he 6 g ape ine a ie ies s esses how c oss- a ie y dis ibu ion shi s a e signi ican o lea ning echniques. The au ho s show a no malized oo mean squa ed e o (RMSE) o 23.83% on hei da a be ween he coun ed and eal numbe o be ies. Howe e , he whole sys em is s ill un a nigh wi h di ec illumina ion. Ano he be y de ec ion sys em based on CNNs is p esen ed in Zabawa e al. (2019) and in he ex ended wo k (Zabawa e al.,2020). In hese wo ks, he au ho s each a e y good 94.0% and 85.6% o he wo g ape ine aining sys ems conside ed, bu again he images we e collec ed wi h a modi ied s addle g ape ine ha es e o illumina ion cons ancy. This solu ion is hen no applicable o, o example, able g apes, o which a s addle machine is non iable. In Co iello e al. (2020), a coun ing ne wo k inspi ed by densi y based c owd coun ing echniques was p esen ed. The au ho s show e y in e es ing esul s, wi h mean a e age e o (MAE) anging om 0.85% o Pino G is a ie y, o 11.73% o Ma zemino a ie y. To achie e o hese esul s, hey had o label mo e han 35000 be ies, a e y ime consuming ope a ion. All hese app oaches ha e he common cha ac e is ics o le e aging deep lea ning o inc eased pe o mances, bu limi ing he a iabili y o he p oblem by using speci ic machines o huge quan i ies o labelled da a. These wo aspec s limi he applicabili y o such echniques o he speci ic case o which hey we e concei ed. The a o emen ioned conside a ions inspi ed us o explo e me h- ods ha could help in aining de ec ion and ins ance segmen a ion algo i hms wi h ew labelled da a. We explici ly conside he case whe e a small amoun o labelled da a om a simila cul i a ion has been collec ed and labelled (Sou ce Da ase , SD), bu which is no enough o ge accep able de ec ion and segmen a ion pe o mances on a di e en o cha d wi h consis en co a ia e dis ibu ion shi (Ta ge Da ase , TD). We use as ou es a ge da a example able g ape ineya ds cul i a ed in Ap ilia, sou he n Lazio, while ou sou ce da ase is he Emb apa Wine G ape Ins ance Segmen a ion Da ase (WGISD) p esen ed in San os e al. (2020). We p esen a combina ion o weakly and semi supe ised echniques ha a e able o signi ican ly inc ease he pe o mance o he algo i hms and we compa e hese newly ained algo i hms wi h he s a e-o - he-a app oaches on he example applica ion o acking ui s o yield es ima ion. The p oposed solu ion is able o p oduce pseudo labelled da a in o de o b idge he gap o co a ia e shi s ha occu whene e a new speci ic c op becomes he a ge o a compu e ision sys em o p ecision ag icul u e. We explici ly ackle he p oblem o doing so wi h limi ed ha dwa e and so wa e esou ces, o add ess he needs o small and medium businesses. Fo his eason, all he pseudo labelling s a egies p esen ed a e based on simple ideos collec ed wi h a cellphone came a. Wi h his in mind, he speci ic pseudo labelling s a egies we p o- pose a e o wo kinds: Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 3 T.A. Cia uglia e al. Fig. 1. An example o he ou a ie ies p esen in he expe imen al ield. The Black Pizzu ello (d) is he mos in e es ing o his wo k because i p esen s he highes a iabili y in shape and colou wi h espec o o he ounded be y a ian s. •Au oma ic bounding boxes gene a ion o objec s con ained in consecu i e ideo ames, based on a s a ing es ima e and 3D s uc u e geome ical conside a ions. We show ha , le e aging a simple ini ial labelling – which could be manual o au oma ic – and he in o ma ion ha we can ge om ea u e ma ching and s uc u e om mo ion, we a e able o gene a e new labelled da a ha g ea ly inc eases he pe o mance o he de ec o . •Pseudo mask gene a ion o ins ance segmen a ion: we show how, s a ing om a simple bounding box - which could be he one au oma ically gene a ed in he p e ious s ep - i is possible o use a segmen a ion ne wo k oge he wi h a e ining s a egy o gene a e new mask labels. Sel -supe ised echniques ha e been equen ly p oposed o sol e he da a sca ci y p oblem in speci ic scena ios (see, o example, G anland e al.,2022;Li e al.,2022 and Siddique e al.,2022 o some ecen de ec ion o segmen a ion app oaches), howe e , o he bes o he au ho s’ knowledge, his is he i s ime ha a gene al sel -supe ision echnique o de ec ion and segmen a ion in ag icul u e is p oposed in o de o ackle a whole ca ego y o p oblems. The s uc u e o he pape is he ollowing. In Sec ion 2, he pseudo- label gene a ion sys em (PLG) is de ailed, bo h o he de ec ion (Sec- ion 2.6) and o he segmen a ion asks (Sec ion 2.7). In addi ion, a acking ask o yield es ima ion is discussed in Sec ion 2.6.3. In Sec ion 2.2, we desc ibe he da a collec ed and used, and in Sec ion 2.4 he speci ic me ics o mul iple objec acking (MOT) a e de ined. Expe imen s and discussion o all hese sys ems a e desc ibed in Sec ion 3and conclusions a e d awn in Sec ion 4. 2. Ma e ials and me hods In his Sec ion, we discuss he da a, he gene al a chi ec u e o he sys em, and he algo i hms on which i is based. We s a by desc ibing he expe imen al ield whe e he a ge da a has been collec ed, and hen how i was collec ed, and how i compa es o he sou ce da a ha was al eady a ailable. Then, we desc ibe he global sys em a chi ec u e and in oduce i s componen s. The inal sub-sec ions in oduce he me ics used o ou expe imen al e alua ion, and gi e mo e de ails o each subsys em. 2.1. Expe imen al ield The expe imen al ield is loca ed in sou he n Lazio (I aly). The ineya d is composed o wo plo s app oxima ely 114 m ×51 m (0,58 ha) and 122 m ×48 m (0,58 ha). Vineya ds a e s uc u ed as a adi ional ellis sys em called Tendone wi h a wide dis ance be ween each plan , 3 ×3m2. Plan a ions a e all olde han 3 yea s and so in ull p oduc ion and heal h, hus ep esen ing a ypical wo king condi ion o he alida ion o ag onomic ac i i ies such as ui ha es ing o ine p uning. All s uc u es a e adi ionally co e ed wi h plas ic and ne o p o ec g apes om ain and hail. The a e age ex ension o each plo is a ound 1 hec a e and dimensions (leng h and wid h) a e on a e - age be ween 25 m and 50 m acco ding o plo ex ension and geome y. The selec ed ineya d in Ap ilia has cu en ly ou di e en able g ape a ie ies which a e desc ibed in he ollowing: Whi e Pizzu ello, Black Pizzu ello, Red Globe and Black Magic. Fig. 1 shows some examples o hese g ape a ie ies while Fig. 2 shows images o he expe imen al ield as well as he app oxima e ex ension o each g ape a ie y in he ineya d. O he ou a ie ies ha we e p esen in he ineya d, Black Magic was o e y low quali y and hus un ended by he ield owne . Whi e Pizzu ello is iden ical in shape o he Black one, and he la e has he same colou as he o me when no ipe. Toge he , whi e and black Pizzu ello a e a peculia a ie y o he Lazio Region and p esen he highes a iabili y in shape and colou wi h espec o s anda d ounded be y a ian s. Fo hese easons, while we collec ed images o all he a ie ies, we inally concen a ed ou da a labelling e o only on Black Pizzu ello. 2.2. Da ase As we men ioned in he in oduc ion, he p oposed sys em deals wi h he co a ia e shi om a gene ic sou ce da ase o a a ge da ase ha is ep esen a i e o he images ha could be collec ed on he ield. We assembled ou a ge da ase wi h wo di e en kinds o da a. The i s a e ideos eco ded using a mid ange cellphone came a (Mo oG8 Plus), which simula es a da a collec ion ope a ion ha could be pe o med by a a me wi h ease. We collec ed ideos mo ing along he ineya d (i.e. angen ial o he ows), wi hou any equi emen on dis ance om he ui s o heigh om he g ound. In his wo k, we use HD (1280 ×720) ideos a 10 Hz wi h a o al o 1469 ames. Examples o hese ames a e shown in Fig. 3. A sho segmen o 10 s has been labelled o es use in he case o he acking algo i hm e alua ion, while he es has been used wi hou labelling hanks o he semi-supe ised na u e o he sys em. We b ie ly call his a ge ideo da ase TVid. No e ha we collec ed he images using a neu al g ey ca d o whi e balance pu poses, bu his is mean o u u e uses, and i is no a equi emen o he me hods p esen ed in his wo k. The second kind o da a is composed o s a ic images o Black Pizzu ello. This da a simula es he images a a me , o a obo , could collec o pe o m some ag icul u al ac ion on speci ic g ape bunches (e.g. quali y es ima ion, disease de ec ion, au oma ic ha es ing). The Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 4 T.A. Cia uglia e al. Fig. 2. (a) Sa elli e iew o he expe imen al ineya d. The image shows he a ie ies ha a e g own on each ow. (b) A pic u e o he ellis (Tendone) s uc u e. Fig. 3. (a) and (b) Examples o he ideo ames collec ed o ou a ge da ase (TVid): we s ess ha hese a e simple cellphone came a based ideos and a e he only new da a equi ed o he sys em o be able o p oduce new labels wi hou supe ision. (c) Example o TImg da ase . (d) Example om he sou ce da ase , he WGISD da ase (San os e al., 2020). I is possible o no e he di e ences in shape, colou and gene al illumina ion condi ions. da ase consis s o 134 images o 3000 ×4000 esolu ion, collec ed wi h he same cellphone came a used o he ideos, howe e he op ics and chip used o ideo and s ill images a e di e en , as is o en he case wi h cellphones. This is in en ional, since i adds a e y common sou ce o co a ia e shi ela ed o he de ice and cap u ing mode (mo ion s s ill images). All he images in his case ha e been labelled o de ec ion (bounding boxes), while a small subse has also been labelled o ins ance segmen a ion (70 images), using he Inno escus labelling applica ion (Inno escus LLC,0000). All hese labels a e used o alida ion and es ing o he algo i hms desc ibed in his sec ion. We call his s ill images da ase TImg. Toge he hese da ase s (TVid and TImg) cons i u e ou Ta ge da ase (TD). As men ioned abo e, we wo k unde he hypo hesis ha a small amoun o labelled da a o he same ui exis s, bu ha i has consid- e able co a ia e shi wi h espec o he TD dis ibu ion. In his wo k, ou Sou ce Da a is he one p esen ed by San os e al. (2020). Fo he de ails abou hese da a, he eade can check he ci ed wo k. He e we gi e a sho summa y o unde line he di e ences be ween SD and TD in e ms o : • he g ape a ie ies (wine s able, be y shape and colou ) Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 5 T.A. Cia uglia e al. Fig. 4. This igu e shows he comple e sys em a chi ec u e. The inpu s a e a sou ce da ase ( ed cylinde ) and a ideo collec ed on he ield by he obo o a a me ( i s g een cylinde , TVid, expanded as a sequence o ames o show he key ame selec ion p ocess). SDe and SSeg (ligh ed blocks) a e he ini ial de ec ion and segmen a ion ne wo ks ained only on he sou ce da ase . All he in e media e compu ing blocks a e depic ed in o ange, while he in e media e ou pu s a e in blue ci cles. Bo h he pseudo bounding boxes and pseudo masks p oduced a e depic ed in yellow, while he de ec ion and segmen a ion ne wo ks ained on hese new labels (TDe and TSeg) a e depic ed in ligh g een. The da a low in he sys em is also colou coded as pe legend. • he illumina ion condi ions ( ull sun s shadows) • he came a de ice (Re lex s cellphone came a) •scale o he images (s anda d scale s a iable scale) To quan i y he co a ia e shi gap in Sec ion 3, he pe o mance d op o de ec o s and ins ance segmen a ion ne wo ks ained on SD and es ed on TD a e gi en. 2.3. Sys em o e iew An o e iew o he sys em is depic ed in Fig. 4. The main inspi ing p inciple o his wo k is he economy o da a labelling and da a euse. Fo his eason, he only wo sou ces o da a a e he sou ce da ase (da a a ailable om a simila ask) and a ideo collec ed on he a ge ield. The sou ce da ase is used o ain he ini ial de ec ion and segmen a ion models, namely he Sou ce De ec o Ne wo k (SDe ) and he Sou ce Segmen a ion Ne wo k (SSeg). SDe is no pe ec ly uned on he a ge en i onmen , s ill i can be used on selec ed ames o he ideo inpu ha we call key ames. To keep his solu ion simple, we conside equally spaced key ames s a ing om he i s one, bu o he s a egies could be de ised. A se o ini ial bounding boxes is ex ac ed om his key ame, using a high con idence h eshold, o limi he alse posi i es. Then, he whole ideo is passed in a Geome ic Consis ency block (GC block) ha ex ac s ea u es om each ame and associa es hem. We es ed wo di e en op ions o his block, as will be shown in he ollowing sec ions. Using his geome ic in o ma ion, oge he wi h he ini ial bounding boxes ex ac ed om he key ames, i is possible o in e pola e he bounding boxes posi ions o he emaining ames wi h high accu acy. These new bounding boxes a e ou pseudo-labels o aining he de ec o on he a ge en i onmen , which we call Ta ge De ec o (TDe ). The De ec ion Pseudo-Labels Gene a ion (DPLG) sub-sys em could be used independen ly by he Segmen a ion Pseudo-Labels Gene a ion (SPLG). To p o e he e ec i eness o he app oach, we compa e he pe o mance o TDe on he bunches acking p oblem, i.e. coun ing he numbe o g ape bunches by coun ing he ins ances acked along a ideo. This p oblem is ele an since i can be used o yield es i- ma ion pu poses. We es wo di e en acking algo i hms, which a e desc ibed in Sec ion 2.6.3 and e alua ed in Sec ion 3.2. The goal o he second pa o he sys em is o gene a e pseudo masks o aining an ins ance segmen a ion ne wo k. This sub-sys em can be seen bo h as an independen pseudo label gene a o , o as pa o a bigge sys em such as he one we desc ibe he e. As men ioned be o e, he SSeg is ained only on sou ce da a and is no able o p oduce good segmen a ion masks on he Ta ge Da a. Howe e , i is possible o gi e he ne wo k some in o ma ion cues ha can g ea ly imp o e he mask es ima es. The i s one is he bounding box egion in which he ins ance should be segmen ed. This cue comes easily om he p e ious s ep o pseudo bounding box gene a ion, bu i could be p oduced o he wise. This gene a es he ini ial pseudo masks. I is possible o use hese pseudo masks o aining he Ta ge Segmen a ion Ne wo k (TSeg) bu his would lead o poo pe o mances due o con i ma ion bias. We need he e o e o injec ex e nal in o ma ion om o he cues ha we ha e. In ou sys em, his is he ole o he pseudo masks e ining block. In Sec ion 2.7.1, h ee di e en solu ions o e inemen will be Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 6 T.A. Cia uglia e al. desc ibed. Thanks o hese e ined pseudo masks i is inally possible o ain he TSeg Ne wo k. Sec ion 3.3 epo s he esul s o he e ining s a egies and compa es he pe o mance o TSeg wi h SSeg. Finally, he expe imen s o he whole sys em, ained only on ideos and es ed o ins ance segmen a ion pe o mance, a e gi en in Sec- ion 3.4. 2.4. Me ics In his Sec ion, we desc ibe he me ics used o e alua e and com- pa e he de ec o s, he acke s and he ins ance segmen a ion al- go i hms. To e alua e he de ec o s and ins ance segmen a ion algo- i hms, he s anda d me ics o P ecision, Recall and In e sec ion o e Union (IoU) ha e been used. In addi ion, o ins ance segmen a ion, he A e age P ecision, as de ined in he MS COCO challenges (Lin e al., 2014), has been used. Usually AP is compu ed o each class and hen a e aged o ob ain he mean a e age p ecision (mAP). In his wo k, since he e is only one class (g ape), he AP coincides wi h he mAP. In he MS COCO me ics, he AP is calcula ed by compu ing he p ecision a e e y ecall le el om 0 o 1 wi h a s ep size o 0.01. The mAP is hen compu ed by a e aging he AP o e all he objec ca ego ies and en IoU h esholds om 0.5 o 0.95 wi h a s ep size o 0.05. To e alua e he acke s, we ollow he common p ac ice o Mul iple Objec T acking (MOT) as de ined by Wu and Ne a ia (2006) and he CLEAR MOT me ics (Be na din and S ie elhagen,2008). MOT is a di icul ask o e alua e, since he pe o mance me ics should cap u e bo h he p ecision in de ec ing indi idual ins ances and he accu acy in acking each ins ance ac oss mul iple ames, wi hou losing ack o swi ching be ween ins ances. Gi en a numbe o objec s 𝑜𝑗, 𝑗 ∈ [0 … 𝑚], he acke p oduces a numbe o hypo heses ℎ𝑖, 𝑖 ∈ [0 … 𝑛]. The pe o mance o associa ion o hypo hesis and objec s can be measu ed ame by ame using he classic T ue Posi i e, T ue Nega i e, False Posi i e and False Nega i e igu es, oge he wi h hei di ec descendan s P ecision and Recall. Howe e , ecen ly a numbe o compound indexes ha e been p oposed o be e cap u e he gene al acke ’s pe o mance. The i s one is he Mul iple Objec T acking Accu acy (MOTA), de ined as ollows: 𝑀𝑂𝑇 𝐴 = 1 − (𝐹 𝑁 +𝐹 𝑃 +𝐼𝐷𝑠𝑤) 𝐺𝑇 ∈ (−∞,1] (1) whe e 𝐹 𝑁 and 𝐹 𝑃 a e False Nega i es and False Posi i es, 𝐼𝐷𝑠𝑤 ep esen s he numbe o ins ances whose ID has been e oneously swi ched, GT is he eal numbe o ins ances in he ideo. This index accoun s o h ee sou ces o e o , namely he alse posi i e a io, he alse nega i e a io and he misma ch a io. Toge he , hey gi e an idea o he gene al acking accu acy. To e alua e he p ecision, a second index was p oposed: 𝑀𝑂𝑇 𝑃 =∑𝑡,𝑖 𝑑𝑡,𝑖 ∑𝑡𝑐𝑡 (2) whe e 𝑐𝑡deno es he o al numbe o ma ches in ame 𝑡, and 𝑑𝑡,𝑖 in gene al ep esen he dis ance o he hypo hesis and he objec , bu in ou case can be compu ed as he o e lap o he g ound u h and he hypo hesis bounding boxes. This second index gi es only a measu e o he p ecision in de ec ing he ins ances wi hou gi ing any in o ma ion on he acking and associa ion capabili y. 2.5. De ec ion and segmen a ion ne wo k a chi ec u es As explained in Sec ion 2.3, he gene al pseudo label gene a ion sys em is based on p e- ained de ec ion and segmen a ion ne wo ks (SDe and SSeg) and is mean o p oduce he pseudo labelled da a o ain new ne wo ks ha a e able o pe o m be e on TD (TDe and TSeg). The main pa ame e s ha in luence he choice o he a chi ec u es a e speed and accu acy. I is well known (Liu e al.,2020) ha So A de ec ion ne wo ks can be di ided in o wo main ca ego ies: wo s age and single s age. The i s kind sepa a es de ec ion in o wo phases, he i s is called egion p oposal and gi es objec bounding boxes candida es, while he second il e s and e ines hese candida es o p oduce he bounding boxes and classi ies he objec s. The second kind ins ead ex ac s bo h egion p oposal and class p edic ion in one pass. The main ad an age o he single s age de ec o s is speed, which is much highe han he wo s age one, bu a he cos o a gene al educed accu acy. The main examples o single s age de ec o s a e he YOLO a ian s, in pa icula he ecen YOLO 5 (Redmon and Fa hadi,2018). One o he bes known and bes pe o ming wo s age a chi ec u es is Mask R-CNN (He e al.,2017), which is also a segmen a ion ne wo k, mo e accu a e han any YOLO a ian s, bu slowe and di icul o weak o eal- ime use. In his wo k, we use he single s age YOLO 5s a chi ec u e o he expe imen s on acking, since eal- ime de ec ion is needed o his kind o applica ion. In addi ion, some o he a ian s ha e a small numbe o pa ame e s, which makes hem iable o embedded applica ions, such as obo ic ha es ing. The pseudo bounding box gene a ion could be pe o med o line, hus allowing use o he be e pe o ming Mask R-CNN, bu we decided o use he YOLO de ec o o keep his sub sys em sel -con ained. In addi ion, using an a chi ec u e wi h lowe de ec ion pe o mance s esses and es s he obus ness o he gene a ion p ocess. Fo segmen a ion and pseudo mask gene a ion, a segmen a ion ne wo k is needed, so he choice alls on Mask R- CNN. The de ails o he p e aining and ine uning o he de ec ion and segmen a ion ne wo ks a e gi en in Sec ions 3.1.1 and 3.3.1, espec i ely. 2.6. De ec ion pseudo-label gene a ion sub-sys em and acking applica ion In his Sec ion, we de ail he elemen s o he DPLG sys em depic ed in Fig. 5,i.e. he pseudo bounding box gene a ion sys em, oge he wi h he acking algo i hm used o yield es ima ion as a possible applica ion. 2.6.1. Geome ic consis ency block The pu pose o his block is o use geome ical co espondences ex ac ed h ough epipola geome y o associa e g ape ins ances in di e en ames o a ideo s eam, i.e. , gi en a de ec ed g ape bunch, by iden i ying 2D ea u es belonging o i and ma ching, o iangula - ing, hem ac oss mul iple ames, i is possible o ind he same bunch ins ance in he ollowing ames. We use his s a egy in wo ways in his wo k, i s o ex ac pseudo bounding boxes, and hen o acking. In his Sec ion, we desc ibe he gene al unc ional p inciples o S M algo i hms and hei compu a ional cos s. We expe imen ed wi h wo app oaches, he i s is he same used in San os e al. (2020), which le e ages a S M so wa e applica ion, namely COLMAP (Schönbe ge e al.,2016;Schönbe ge and F ahm, 2016). Since S M is a well known p oblem, he in e es ed eade can ind de ails o he solu ions in Ha l ey and Zisse man (2006) and Szeliski (2022). In b ie , we used he COLMAP modali y ha ex ac s spa se ea u es om each ame and hen uns a sequen ial all e sus all sea ch and ma ching o he ea u es ex ac ed om he ideo. These co espondences a e hen used o iangula e he 3D poin s by minimiz- ing he 3D o 2D ep ojec ion e o . Howe e , he na u e o he p oblem is such ha e en wi h he spa se se ing, he compu a ional cos s inc ease exponen ially wi h he numbe o ames. Ou expe imen s equi ed 5 h o compu a ion o ideos o 500 o 600 ull HD ames, on a compu e equipped wi h an In el-Co e i7 3.4 GHz, a N idia GTX 950 m and 16 GB o memo y. The second app oach we expe imen ed add esses his aspec in o de o ha e a eal ime solu ion ha can be un on an online acke , such as he one ha will be desc ibed in Sec ion 2.6.3. The idea is ha in ou con ex a ull S M solu ion (i.e. , he 3D posi ion o he ex ac ed 2D ea u es in a wo ld e e ence ame) is no needed, since he kind Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 7 T.A. Cia uglia e al. Fig. 5. This igu e shows he de ec ion pseudo-label gene a ion (DPLG) sub-sys em alone. The sou ce da ase is used o ain an ini ial coa se bounding box de ec o (SDe ) ha is hen used, oge he wi h he S M sys em, o gene a e a la ge numbe o new labelled images om he ames o con inuous ideos o he ineya d. This same sys em can be applied o o he ui s wi h ela i e simplici y. Fig. 6. Fea u e ma ching and geome ic e i ica ion using RANSAC: COLMAP S M in i s ligh es o m s ill equi es conside able compu a ional ime and i is iable only o o line elabo a ions, while 2D ea u e ma ching equi es much less compu a ion and po en ially uns in eal ime. (a) The ma ching o su ea u es wi h b u e o ce ma ching (b) he same ma ching e ined wi h homog aphy compu a ion combined wi h RANSAC selec ion. o ideos ha a e collec ed in he ineya d a e simple walks wi hou closed loops. This means ha each able g ape bunch is p esen , a mos , in a ew consecu i e ames, excep o he occasional occlusion. Fo his eason, we ound ha ex ac ing 2D ea u es om a ame 𝑖and om a small numbe o subsequen ames 𝑖+1,…, 𝑖+𝑛, and hen ma ch- ing hem was enough o map he g ape ins ance co espondences along he ideo s eam. The ea u es and desc ip o s used a e SURF (Bay e al.,2006), while he ma ching is a simple b u e o ce (all- s-all) dis ance compu a ion be ween he co esponding ea u e desc ip o s. Since hese ini ial ma ch p oposals con ain some misma ched ea u es, a RANSAC e i ica ion s ep is used o il e hem ou . Gi en ha o small came a mo ions, image ans o ma ion could be app oxima ed by a homog aphy ans o ma ion, we epea edly andom sample ou ma ches, compu e he ela i e homog aphy, and check how many o he ma ches a e co ec ly p edic ed by i . The homog aphy wi h he highes consensus is selec ed and all ma ches whose displacemen is no com- pa ible wi h he selec ed homog aphy, a e disca ded. An example o his p ocess is gi en in Fig. 6. The pa allel lines le in Fig. 6(b) show he ma ches ha ag ee wi h he homog aphy es ima ed h ough RANSAC. Using his app oach, e en using b u e o ce ma ching, we we e able o each, wi hou pa icula op imiza ions and wo king only on CPU, 3 ames pe second on he a o emen ioned ideo and ha dwa e. 2.6.2. Bounding box in e pola ion and pseudo label gene a ion Bounding box in e pola ion can be be e unde s ood by looking a Fig. 7. S a ing om a bounding box ound by SDe a ame 𝑖, hanks o he GC Block, i is possible o ha e an associa ion be ween he 2D ea u es con ained inside he box wi h some ea u es in ame 𝑖+𝑛. Since he came a is mo ing, bo h he posi ion o he g apes and he illumina ion condi ions in ame 𝑖+𝑛will be di e en , consequen ly he ea u es ma ched will ha e a di e en posi ion. The ques ion is hen how o d aw he new bounding box in ame 𝑖+𝑛. We use he hypo hesis, ha he came a is slowly mo ing, and ha he mo ion is angen ial o he di ec ion o he ineya d. Thanks o his hypo hesis we can assume ha he new bounding box will ha e he same size as he one ound in ame 𝑖. The posi ion o he new bounding box is compu ed by se ing he cen e o he box o coincide wi h he cen e o g a i y o he ea u es in ame 𝑖+𝑛, as depic ed in Fig. 7(a). Ano he aspec o conside in e alua ing he pseudo bounding box gene a ion scheme is he e ec o came a eloci y combined wi h ame a e. I he ame a e o he ideo is high, o he came a eloci y is low, he change in iew will be minimal, and consequen ly he in o ma ion added by such a sample will be mino . Fo his eason, we conside ed i use ul o explo e he e ec o he a io be ween key ames and o he ames. We call his pa ame e skip alue, since i is he numbe o ames in which he bounding boxes p edic ed in ame 𝑖a e in e pola ed, be o e aking a new p edic ion by SDe . Ou abla ion expe imen s showed ha using skip 1 (i.e. using only SDe o p oduce pseudo-labels) ga e lowe pe o mance han using skip 2. Howe e , inc easing he skip alue seems no o gi e mo e ad an ages. This aspec is explo ed in Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 8 T.A. Cia uglia e al. Fig. 7. The bounding box in e pola ion p ocess. (a) Shows he upda ing p inciple: he bounding box (blue) p edic ed in ame 𝑖is mo ed o ame 𝑖+𝑛; while he size emains he same, he posi ion o he new bounding box (o ange) is upda ed by compu ing he new cen e o g a i y o he ea u es ex ac ed and making i coincide wi h he box cen e. (b) Pseudo-labels gene a ed by means o he S M algo i hm: he g een boxes a e he p edic ions p oduced by SDe a ame 𝑖 ansposed in he cu en one (𝑖+𝑛), while he ed boxes a e in e pola ed ones acco ding o he ea u es ma ched ( ep esen ed as ed poin s). Fig. 8. This igu e shows he segmen a ion pseudo-label gene a ion (SPLG) sub-sys em. Sec ion 3.2, whe e we show on he acke applica ion he esul s o using di e en skip alues. 2.6.3. T acking o yield es ima ion Mul i Objec T acking o he g ape bunch ins ances is a p elimina y s ep in yield es ima ion, as i is possible o es ima e he numbe o bunches by coun ing he numbe o ajec o ies acked by he algo- i hm. The main app oach o acking by de ec ion ha we conside is he one p esen ed by San os in San os e al. (2020) which is based on de ec ion and S M. Howe e , no me ics we e gi en he e o o mally desc ibe he pe o mances o he app oach. The e o e, we eplica ed he expe imen s and compu ed he me ics using as a a ge he es sequence o TVid, desc ibed in Sec ion 2.2 and depic ed in Fig. 3. In addi ion, we es ed ano he de ec ion based S a e-o - he-A acke , DeepSORT (Wojke e al.,2017), designed o wo k in eal- ime using a deep associa ion me ic. We chose his acke since he compu a ion in ol ed in es ima ing e en spa se co espondences be ween he ames using COLMAP (Schönbe ge and F ahm,2016) equi es conside able ime and a e no easible o edge o obo ic de ices. In addi ion, he compu a ion o he ull S M solu ion akes a long ime and limi s he leng h o he ideo o a ew hund ed ames, while o he second app oach he e is no such limi . In Sec ion 3, we compa e he acking solu ions using he MOT me ics wi h he acking g aphs o gain mo e insigh s on wha he acke does and how o imp o e i u he . An example o hese g aphs is gi en in Fig. 15. 2.7. Segmen a ion pseudo-label gene a ion sub-sys em While de ec ion is enough o coun ing asks, o quan i a i e yield es ima ions o o asks ha equi e a physical in e ac ion wi h he olumes o he g apes (e.g. ha es ing), segmen a ion, and in pa ic- ula ins ance segmen a ion, is equi ed. Ins ance segmen a ion equi es labels ha a e ideally pixel pe ec masks, howe e Bellocchio e al. (2019) showed how, e en wi h minimal labelling signal (e.g. p esence o absence o an objec in a image), he ask ne wo k is able o lea n ep esen a ions ha a e close o masks o he objec o in e es . Fo his eason, we again adop a pseudo-labelling app oach o his p oblem, s a ing wi h a p e ained ne wo k on he WGISD sou ce da ase and hen using simple ex e nal cues o wo k as ou ex e nal in o ma ion signal ha helps in e ining he label. The o e iew o his sub-sys em is depic ed in Fig. 8. Ou SSeg ne wo k is Mask R-CNN ained on WGISD, as usual. Mask R-CNN in i s basic o m ex ac s egion p oposals and uses hem o p edic bounding boxes and ins ance segmen a ion masks. Howe e , i is possible o use he segmen a ion subne wo k o Mask R-CNN as he pseudo mask ini ial gene a o . In pa icula , Mask R-CNN is wi ed di e en ly a in e ence ime han a aining ime, since he bounding boxes p edic ed by he de ec ion head a e di ec ly ed o he mask head. The ne wo k will use his bounding box as a cue, o as an a en ion mechanism, which helps he segmen a ion subne wo k o ou pu a use ul pseudo mask. This is depic ed in Fig. 9. This s a egy will mi iga e he p oblem o con i ma ion bias, since he box comes om an ex e nal in o ma ion sou ce. In ou sys em, he bounding boxes could come om he ou pu o he DPLG sub-sys em. A he same ime, in Sec ion 3we show he pe o mances o he pseudo mask gene a ion s a ing om g ound u h bounding boxes so as o be e isola e he pe o mance con ibu ion o he mask gene a ion p ocess. In his way, he numbe o pseudo-masks will coincide wi h he ac ual numbe o g ape clus e s in he image, and he measu ed e o will only be due o he mask gene a ion p ocess. The quali a i e di e ence o segmen a ion mask be ween he s anda d wi ing o he Mask R-CNN ne wo k, and he one wi h an ex e nal a en ion mechanism, is shown in Fig. 10. 2.7.1. Pseudo mask e ining block To e ine he pseudo masks, in o de o educe o emo e con- i ma ion bias, an ex e nal sou ce o in o ma ion is needed. Some Compu e s and Elec onics in Ag icul u e 205 (2023) 107624 9 T.A. Cia uglia e al. Fig. 9. Mask-RCNN in e nal wi ing a aining and in e ence imes. A aining ime, he mask p edic ion head uses he same inpu s as he o he wo heads, i.e. he RoI c opped ea u es. A in e ence ime, he c opping is done only using he bounding boxes p oposals o he bounding box p edic ion head. Ou sys em uses only he ea u e ex ac ion pa and d ops he bounding box eg ession, using ins ead ei he he bounding boxes coming om g ound u h o he bounding boxes pseudo-labels gene a ed by he DPLG sub-sys em. The dashed blue line in he lowe diag am shows whe e he wi e is cu and ou bounding boxes p oposals a e injec ed. Fig. 10. Le image: pseudo mask p oduced by Mask-RCNN ained only on he Sou ce Da ase and wi hou a bounding box cue. Righ image: same image showing he e ec o gi ing a bounding box cue a in e ence ime. ea lie wo ks wo ked on his aspec , such as Kho e a e al. (2017). We ied h ee di e en s a egies o e ine he ini ial masks, using simple compu e ision echniques ha wo k on di e en p inciples om he con olu ional il e s con ained in SSeg and ha use simple geome ical conside a ions. •Dila ion: he i s me hod o igina es om he obse a ion ha SSeg ends o unde es ima e he masks on he a ge da a. Fo his eason, a simple mo phological dila ion ha expands he mask un il i ouches he e e ence bounding box is able o add aluable in o ma ion o he label. The dila ion is applied wi h a 5×5 ci cula -shaped ke nel. An example o he esul is gi en in Fig. 11(a). •SLIC: Simple Linea I e a i e Clus e ing (SLIC) (Achan a e al., 2012) is a me hod o supe -pixel segmen a ion o he image. Supe -pixels a e con iguous egions o he image ha a e clus- e ed oge he by a KMeans algo i hm unning on bo h colou and space (5-dimensional). We apply his supe -pixel di ision o he en i e image and compa e i wi h each pseudo mask. The SLIC algo i hm ha was used was he one implemen ed in he Py hon sciki -image lib a y (Van de Wal e al.,2014) wi h 2000 segmen s and compac ness 0.1. All he supe pixels ha a e co e ed by mo e han an uppe h eshold 𝑡𝑢= 70% a e added o he mask, while all he pixels ha a e co e ed by less han a lowe h eshold 𝑡𝑙= 30% a e emo ed om he mask. The a ionale is ha in his way we should be able o emo e also he backg ound pixels e oneously con ained in he ini ial pseudo mask. An example o he esul is gi en in Fig. 11(b). •G ub Cu : his is an i e a i e segmen a ion echnique in oduced by Ro he e al. (2004). I ep esen s he image as a g aph whe e o eg ound and backg ound pixels a e modelled as Gaussian Mix- u e Models and ha e o be sepa a ed i e a i ely by cu s o he g aph edges. We used he OpenCV (B adski,2000) implemen a- ion whe e i is possible o ini ialize he algo i hm wi h he pseudo mask de ining ou pixel ca ego ies, i.e. su e o eg ound, su e backg ound, p obable o eg ound, and p obable backg ound. The pseudo mask is used as p obable o eg ound. Dila ion is applied o he pseudo mask o a numbe o i e a ions p opo ional o he smalles dimension o he e e ence bounding box o ob ain he p obable backg ound. E osion is applied o he same numbe o i e a ions o ob ain he su e o eg ound, while he es is se o su e backg ound. A sample o he e ec s o G ab Cu is shown in Fig. 12. In Sec ion 3, we show how each o hese e inemen me hods pe o ms compa ed o he baseline (pseudo mask wi h no e inemen ). 3. Resul s and discussion 3.1. De ec ion expe imen s In his Sec ion, we desc ibe he esul s o he de ec ion expe imen s. Table 1 shows ou p elimina y expe imen s o compa e di e en e - sions o he de ec o . All models in his ini ial compa ison a e ained and es ed on WGISD. Resul s show ha he models wi h a la ge numbe o pa ame e s o e a minimal pe o mance inc ease on basic de ec ion, compa ed o he ligh weigh e sions S and N. This can be explained by a gene al homogenei y o he dis ibu ion o he g ape images in WGISD, which does no equi e huge numbe o pa ame e s o lea n a good es ima o . This is expec ed, since in ag icul u e we do no wo k wi h huge amoun s o da a. Fo his eason, we decided o base all he acke s on he S and 𝑁 a ian s o educe o e i ing.