Recei ed June 9, 2021, accep ed June 26, 2021, da e o publica ion July 1, 2021, da e o cu en e sion July 12, 2021.
Digi al Objec Iden i ie 10.1109/ACCESS.2021.3093978
A Gene ic ROS-Based Con ol A chi ec u e o
Pes Inspec ion and T ea men in G eenhouses
Using a Mobile Manipula o
JON MARTIN 1, ANDER ANSUATEGI1, IÑAKI MAURTUA1, AITOR GUTIERREZ1,
DAVID OBREGÓN2, OSKAR CASQUERO 3, AND
MARGA MARCOS 3, (Senio Membe , IEEE)
1Au onomous and In elligen Sys ems Uni , Fundación Teknike , 20600 Eiba , Spain
2Cen o Tecnológico CTC, 39011 San ande , Spain
3Sys ems Enginee ing and Au oma ic Con ol Depa men , Facul y o Enginee ing, Uni e si y o he Basque Coun y (UPV/EHU), 48940 Bilbao, Spain
Co esponding au ho : Jon Ma in (jon.ma in@ eknike .es)
This wo k was suppo ed in pa by he G eenPa ol Eu opean P ojec h ough he Eu opean GNSS Agency by he Eu opean Union’s (EU)
Ho izon 2020 Resea ch and Inno a ion P og am unde G an 776324 [11].
ABSTRACT To mee he demands o a ising popula ion g eenhouses mus ace he challenge o p oducing
mo e in a mo e e icien and sus ainable way. Inno a i e mobile obo ic solu ions wi h lexible na iga ion
and manipula ion s a egies can help moni o he ield in eal- ime. Guided by In eg a ed Pes Managemen
s a egies, obo s can pe o m ea ly pes de ec ion and selec i e ea men asks au onomously. Howe e ,
combining he di e en obo ic skills is an e o p one wo k ha equi es expe ience in many obo ic
ields, usually de i ing on ad-hoc solu ions ha a e no eusable in o he con ex s. This wo k p esen s
Robo amewo k, a gene ic ROS-based a chi ec u e which can easily in eg a e di e en na iga ion, manipu-
la ion, pe cep ion, and high-decision modules leading o a as e and simpli ied de elopmen o new obo ic
applica ions. The a chi ec u e includes gene ic eal- ime da a collec ion ools, diagnosis and e o handling
modules, and use - iendly in e aces. To demons a e he bene i s o combining and easily in eg a ing
di e en obo ic skills using he a chi ec u e, wo lexible manipula ion s a egies ha e been de eloped
o enhance he pes de ec ion in i s ea ly s a e and o pe o m a ge ed sp aying in simula ed and ield
comme cial g eenhouses. Besides, an addi ional use-case has been included o demons a e he applicabili y
o he a chi ec u e in o he indus ial con ex s.
INDEX TERMS P ecision ag icul u e, obo ic con ol a chi ec u e, mobile manipula o , pes de ec ion and
ea men , g eenhouse.
I. INTRODUCTION
The Eu opean ag icul u e land su ace is dec easing due o
de o es a ion and u baniza ion while popula ion con inues
inc easing. In o de o achie e a mo e sus ainable business
model, p o ec he c ops om ad e se wea he condi ions and
con ol he empe a u e and wa e o he plan , g eenhouse
p oduc ion is g owing a 22% accumula ed inc ease in a ea
since 2011 [1]. Howe e , he p esence o wa m, humidi y
condi ions and abundan ood unde p o ec ed s uc u es p o-
ide a o able habi a s o pes de elopmen , his being he
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Vincenzo Con i .
main h ea o p oduc ion and p oduc i i y o g eenhouse
c ops wo ldwide [2]. Digi al a ming [3]–[5] can help h ough
senso s, obo ics and da a analysis o au oma ically main ain
and moni o g eenhouses, making c opping sys em sma
and, hus, enhancing he ag icul u al p oduc i i y.
T adi ional pes de ec ion me hods in oma o c ops ely
on a me s obse ing skills, which a e e y ime-consuming
and ine icien in la ge c ops. Nowadays, obo ic solu ions
combined wi h compu e ision can be used o au oma e
his epe i i e inspec ion ask, inc easing he eliabili y, max-
imizing he heal h o c ops and op imizing he use o pes-
icides o as li le as 5%-10% [6]. Fo ha pu pose, obo s
need o implemen plen y o di e en asks such as localize
VOLUME 9, 2021 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ 94981
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
hemsel es [7] and na iga e inside g eenhouses [8], [9];
acqui e quali y pic u es o iden i y pes s and hei loca-
ions [10]; o p ocess he ob ained esul s o gene a e e icien
high-le el ins uc ions o command he obo acco ding o an
In eg a ed Pes Managemen (IPM) sys em [11]. Howe e ,
mos esea ch wo ks ocus on indi idual p oblems neglec -
ing i s in eg a ion wi hin a single comple e solu ion. The
combina ion o di e en obo ic skills can be di icul and
usually de i e o ad-hoc solu ions, bu his is necessa y o
pe o m ea ly pes de ec ion. The insec in hei ea ly eggs
s a e can measu e as less as 0.3 mm and, o de ec hem,
ad ance pe cep ion and dex e i y skills need o be me ged o
au oma ically ob ain close and good quali y pic u es o he
pes s om di e en sides o he lea es.
This wo k p esen s Robo amewo k, a no el obo ic a chi-
ec u e ha in eg a es na iga ion, manipula ion, and pe cep-
ion skills while ollowing high le el ins uc ions om an
IPM decision suppo sys em o ea ly pes de ec ion and
ea men in g eenhouses. The a chi ec u e includes addi-
ional ea u es ha makes i easily applicable o simila
p ecision ag icul u e applica ions whe e obo na iga ion,
manipula ion and pe cep ion skills a e equi ed. This gene ic
a chi ec u e can ema kably educe he de elopmen ime
equi ed o pe o m Robo Ope a ing Sys em (ROS) based
ield obo ic expe imen s due o e icien euse o common
modules ac oss p ojec s and obo pla o ms. To demons a e
he easy in eg a ion and he bene i s o combining di e en
obo ic skills wi hin he a chi ec u e, lexible manipula ion
s a egies o enhance pes de ec ion and a ge ed sp aying
ha e been de eloped. Finally o e alua e he a chi ec u e, se -
e al es s in simula ed and ield comme cial g eenhouses ha e
been pe o med in he con ex o he Eu opean G eenPa ol
p ojec [13].
This pape is s uc u ed as ollows: The ela ed wo k
is analyzed in Sec ion 2. Sec ion 3 in oduces he chal-
lenges o pe o ming au onomous pes de ec ion and ea -
men and he main obo ic sys em equi emen s. Sec ion 4
desc ibes he de eloped Robo amewo k a chi ec u e while
Sec ion 5 ocuses on he manipula ion s a egies o enhanc-
ing pes de ec ion and ea men . Sec ion 6 in oduces he
simula ed and ield es s ca ied ou o e alua e he sys em in
g eenhouse and indus ial scena ios. Finally, he conclusions
ob ained om he assessmen a e discussed and he u u e
wo k is p esen ed.
II. RELATED WORK
Resea ch on p ecision ag icul u e obo ics has ecen ly
ocused on wo a eas: (i) weed inspec ion and a ge ed sp ay-
ing and (ii) ui and ege ables ha es ing obo s [5]. The i s
a ea is mos ly ep esen ed by ou doo obo s o weed con ol
such as he G aph Weeds Ne [14], he RHEA p ojec cen-
e ed on bo h ag icul u e and o es y [15], BoniRob p ojec
dedica ed o mul ipu pose a ming [16], o CROPS p ojec
ocused on p ecision sp aying in ineya ds [17]. The na i-
ga ion o hese ou doo obo s is la gely based on he use o
sa elli e localiza ion sys ems and hei signal is much weake
and unp ecise in indoo en i onmen s, making hem less sui -
able o g eenhouses [18]. Mo eo e , g eenhouses a e spe-
cially challenging o simul aneous localiza ion and mapping
solu ions, as hey a e pa ially s uc u ed en i onmen s wi h
cons an ly g owing plan s [19]. Tha is why mos o he obo s
ound in g eenhouses use ails o na iga e on i [20]. Some
examples a e he oma o ha es ing obo [21], he peppe ha -
es ing obo [22] o he che y oma o ha es ing obo [23].
O he examples a e AURORA, a sp aying obo ha imple-
men s a wall ollowing algo i hm o na iga ion [24] o a
g eenhouse sp aying obo ha ollows lines and QR codes
o na iga ing [25]. The use o ixed pa hs such as ails o
na iga ion in g eenhouses has esul ed in decoupling na iga-
ion om he manipula ion and inspec ion asks. This is he
case o he CROPS obo amewo k [26], whe e he con ol
a chi ec u e co e s only he ui localiza ion and a m con ol
unc ionali ies. As demons a ed by he open sou ce con ol
a chi ec u e F oboMind [12], a common eusable a chi ec u e
ha combines di e en obo ic skills and ailo ed o p ecision
ag icul u e obo s can signi ican ly dec ease de elopmen
ime and esou ces due o e icien euse o exis ing wo k
ac oss p ojec s. Howe e , despi e using ROS as communica-
ion middlewa e, BoniRob is ou da ed as i does no in eg a e
he s a e-o - he-a accep ed na iga ion_s ack [27] o na -
iga ion o Mo eI ! [28] o manipula ion. The i s package
akes in o ma ion om odome y, senso s eams, and a goal
pose and ou pu s sa e eloci y commands ha a e sen o he
mobile base. The second one is he mos widely used so wa e
o manipula ion and p o ides he la es ad ances in mo ion
planning, manipula ion, o 3D pe cep ion, among o he s.
In o de o combine bo h algo i hms o mobile manipula o s,
some au ho s ha e ied o simul aneously plan and execu e
mobile manipula ion goals [29], [30] bu hese a e compu-
a ionally complex me hods es ed in simula ion o labo a-
o y condi ions and cu en ly un easible o g eenhouse-like
uns uc u ed en i onmen s.
De eloping an e ec i e IPM equi es equen and p e-
cise obse a ions o plan s. To build an ea ly pes de ec-
ion i may no be enough o ocus on plan s o lea es
ha a e al eady in ec ed wi h insec s a adul s ages as
in [32]–[34], bu i is necessa y o de ec he cause o he
in ec ion. In o de o enhance he ea ly pes de ec ion, i is
necessa y o go a s ep u he and de ec he insec s also
in hei egg and la a s ages [10], [35]. Mo eo e , mos
pes de ec ion wo ks ocus on he de ec ion and classi ica-
ion o pes s on al eady acqui ed pic u es da ase neglec -
ing he di icul ies o au oma ically ob aining hem wi h
enough quali y and closeness. In his sense, his wo k p esen s
he manipula ion s a egies de eloped o ge close pic-
u es o he su aces o he lea es om abo e and om
below, so as o inspec he su aces o he lea es om bo h
sides.
Finally, he e a e se e al Eu opean p ojec s as DROPSA
[36], ISEFOR [37], PALMPROTECT [38] o EMPHASIS
[39] ocused on he de elopmen o new igh ing s a egies
agains some speci ic pes s, bu he b idge be ween new pes
94982 VOLUME 9, 2021
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
FIGURE 1. Toma o c op g eenhouse e olu ion a he beginning (up) and
a he end (down) o he season.
de ec ion s a egies and au oma ed and obus managemen is
ba ely add essed.
This wo k p esen s, simila ly o [31], a decoupled mobile
manipula ion con ol o g eenhouse ela ed asks using
he ROS de- ac o algo i hms [27] and [28]. The na iga-
ion is based on la es obo ic solu ions which ha e p o en
o success ully use Galileo Sa elli es combined wi h IMU,
odome y and ange lase senso s o localiza ion [8]. The
con ol a chi ec u e ollows he hyb id pa adigm p esen ed
in [40], whe e a ional and e icien delibe a i e decisions
ep esen ed by an IPM s a egy a e combined wi h eac i e
beha io s ep esen ed by he di e en na iga ion, manipula-
ion and ision modules.
III. PROBLEM DESCRIPTION AND ARCHITECTURE
REQUIREMENTS
The e a e se e al challenges o de eloping a obo ic sys em
able o pe o m au onomous and con inuous moni o ing in
g eenhouses o he de ec ion, iden i ica ion, and con ol o
pes s. As shown in Figu e 1, he plan s g ow ema kably
du ing he g owing season a ec ing: (1) he localiza ion and
na iga ion sys ems because o a cons an change o he en i-
onmen and he na owing o he co ido s; (2) he manipula-
ion s a egy, as he a m needs o app oach he lea es o ob ain
good quali y pic u es while a oiding damaging he c ops; and
(3) he ision modules dealing wi h changes in illumina ion
and ocus dis ance. In addi ion, he sys em mus be able o
execu e high-le el ins uc ions p oposed by he IPM s a egy,
p o iding diagnosis and logging capabili ies and o e ing an
easy- o-use use in e ace.
The main obo ic sys em equi emen s p esen ed in
Table 1 ha e been iden i ied by obse ing a single obo needs
o he G eenPa ol applica ion. I is howe e no iceable ha
mos equi emen s ema ked in bold a e desi able o almos
TABLE 1. Main mobile manipula o sys em equi emen s.
any o he mobile manipula o sys em. Robo amewo k akes
all hese equi emen s in o accoun and p esen s an a chi ec-
u e ha is no only alid o he cu en applica ion bu also
o o he ag icul u al o e en indus ial applica ions.
IV. SYSTEM ARCHITECTURE
This Sec ion con ains a desc ip ion o he gene al con ol
a chi ec u e p esen ed in Figu e 2. The ou -laye ed a chi ec-
u e seeks he easy in eg a ion o he di e en obo unc ion-
ali ies ensu ing he sys em equi emen s p esen ed in Table 1.
I ollows a dis ibu ed compu ing design allowing se e al
asks o un in di e en compu e s while s ill appea ing o
i s use s as a single cohe en sys em and allowing an easy
ex ensibili y.
ROS [41] is p oposed as co e communica ion middle-
wa e among he di e en modules. In ecen yea s, ROS has
become he de ac o s anda d amewo k o he de elopmen
o so wa e in obo ics. ROS is a lexible open-sou ce ame-
wo k o w i ing obo so wa e ha p o ides, collec ion o
communica ion mechanisms, ools, lib a ies, and ules ha
aim o simpli y he ask o c ea ing obo so wa e o a wide
a ie y o obo ic pla o ms.
In he a chi ec u e, he e a e se e al modules ha a e com-
mon o any applica ion. These a e ep esen ed in u quoise
colo and include he obo use in e ace, as well as appli-
ca ion laye , he e o managing and logging modules and
some common pa s o he abili ies laye . Some o he modules
composed by s anda d ROS modules o packages de eloped
and es ed by he G eenPa ol p ojec a e a ailable in he
a chi ec u e, bu i is up o he use o use hem o implemen
new modules using he a ailable ones as empla es. The
d i e s laye , o ins ance, depends on he obo used. Also,
he high-le el decision modules, he e ep esen ed as an IPM
sys em, depends on he applica ion.
A. DECISION LAYER
The decision laye con ains he high-le el decision modules
ha gene a e new plans o he applica ion. In his case,
VOLUME 9, 2021 94983
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
FIGURE 2. Fou -laye Robo amewo k con ol a chi ec u e. Modules ep esen ed wi h u quoise
colo a e common o any applica ion. G ey modules a e s anda d ROS modules while he es ha e
been de eloped and es ed du ing he G eenPa ol p ojec and could be used as empla es.
FIGURE 3. G eenhouse ep esen a ion wi h a obo app oaching he ed
ci cles ep esen ing he a ge s wi h na iga ion and pes inspec ion asks
(le ). Rep esen a ion o an IPM gene a ed T1-T4 plan ( igh ).
an IPM s a egy gene a es pes scou ing and ea men plans
based on domain expe knowledge, c ops dis ibu ion in
he g eenhouse and in o ma ion ob ained om p e ious plan
execu ions as seen in he op laye o Figu e 2. The plans a e
composed by a ge s ha con ain a na iga ion goal o mo e
he obo o a desi ed posi ion and a ask o be pe o med
he e. An example plan o he cu en applica ion can be seen
in Figu e 3 whe e he obo mus na iga e o ou di e en
g eenhouse zones and pe o m he e an inspec ion ask.
The Robo GUI module in Figu e 2 is common o any
applica ion and p o ides an easy- o-use use in e ace o load,
FIGURE 4. Robo GUI in e ace used o load and s a a json plan (le )
and he applica ion wo k low messages once he plan has been
ini ialized ( igh ).
execu e o cancel plans ul illing sys em equi emen SR7.
The plan is hen sen o he applica ion laye o be in e p e ed
and execu ed by he obo while he GUI displays he cu en
s a e o he sys em including s a us, ale s, o he ba e ies
le el as shown in Figu e 4.
The plans a e implemen ed using a Ja aSc ip Objec No a-
ion (JSON) o ma , which is a e y common open s anda d
94984 VOLUME 9, 2021
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
FIGURE 5. S a e machine ep esen ing he ope a ion mode a ge s plan and i s na iga ion, asks, and e o -handling
beha io s a es.
and language-independen , ha uses human- eadable ex o
s o e and ansmi da a objec s consis ing o a ibu e– alue
pai s and a ay da a.
The JSON keywo ds ela ed o na iga ion a e:
•na iga e: Indica es whe he he a ge con ains a na -
iga ion s ep. The ollowing na iga ion keys a e only
conside ed i na iga e is ue.
•na iga ionType: Th ee ypes o obo na iga ion a e
a ailable. (1) Na u al na iga ion making use o he
well-known na iga ion_s ack om ROS; (2) Rela i e
na iga ion o pe o m a con inuous mo ion o each
a posi ion ela i e o he obo ’s cu en posi ion; and
(3) P ecise na iga ion o gene a e a con inuous mo ion
o posi ion he obo accu a ely wi h espec o an a i i-
cial ma k. Only he i s ype is used in he G eenPa ol
con ex .
•na iga ionT ials: Numbe o ials o na iga ion in
case o ailu e.
• a ge Pose: Na iga ion des ina ion (x, y, he a) in he
gi en ame_id sen o he na iga ion node.
The JSON keywo ds ela ed o ask de ini ion:
• asks: An a ay o asks o be execu ed.
– name: Name o he ask.
– ype: Type o he ask plugin ha will be loaded and
execu ed. In his case i may be an inspec ion o a
sp aying ask.
– pa ams: Necessa y pa ame e s o pe o m he ask.
This enables a high con igu abili y o he IPM
s a egy o de ine he zones o be inspec ed o he
amoun o pes icide o use depending on he in ec-
ion le el o he plan .
This me hod pe mi s o de s pa ame iza ion ha co e s a
wide ange o mobile obo ics applica ions. The plans can be
gene a ed manually o au oma ically by di e en high-le el
decision suppo sys ems, add essing sys em equi emen
SR4. Sec ion 5 p esen s wo simple plan examples o
pes inspec ion and ea men ope a ions and subsec ion 6-C
p esen s an addi ional plan o an aile on inspec ion in an
indus ial use case.
B. APPLICATION LAYER
The applica ion laye includes he obo manage module,
which is esponsible o con olling he o e all obo ic sys-
em, main aining i s s a us con inuously. This laye also
includes he ope a ion mode, which in e p e s he high-le el
plans and implemen s a speci ic obo ic applica ion. The
ope a ion mode is designed o be as gene al as possible, easily
con igu able o a a ie y o obo ic p ocesses and, hus,
a oiding an ad-hoc implemen a ion only use ul o speci ic
wo kspace con igu a ions. A plan can be speci ied by a se o
a ge s composed by a na iga ion des ina ion o he mobile
pla o m and a se o asks o be pe o med a each des ina ion.
Figu e 5 shows he s a e machine implemen ed o he ope -
a ion mode. I s a s in a Wai ing s a e un il a new-plan e en
indica es he beginning o he ope a ion. The sys em swi ches
hen o Checking nex a ge s a e, analyzing he nex a ge
in he sequence.
The Na iga ing s a e is esponsible o coo dina ing he
au onomous mo emen s o he mobile pla o m. The a chi-
ec u e pe mi s an easy in eg a ion o di e en na iga ion
modules and p o ides he managemen o hei esul s. I he
na iga ion is no able o each he a ge pose due o obs acles
on he way o localiza ion p oblems, his will be no i ied in
he esul and, i equi ed, a Na iga ion Reco e y Beha io is
igge ed. Due o he c i icali y o he sa elli e-based localiza-
ion in g eenhouses o each a posi ion accu a ely, i has been
VOLUME 9, 2021 94985
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
necessa y o include an addi ional ea u e in he na iga ion
s ack o p o ide in o ma ion abou he localiza ion quali y.
In case o a ema kable localiza ion quali y loss, a speci ic
e o eco e y beha io can be igge ed. This beha io con-
sis s in sending he obo o a well-known g eenhouse posi ion
whe e he localiza ion signal is known o be s ong and e y-
ing om he e he p e ious na iga ion goal. The e is a second
eco e y beha io igge ed when, despi e ha ing a p ope
localiza ion signal, he obo does no each he des ina ion
wi h enough accu acy. This can happen because a sligh ly
be e localiza ion is needed o because he e a e obs acles
on he way ha he na iga ion module canno o e come.
In bo h cases, he eco e y beha io consis s o wai ing o a
p ede ined ime s ill, while playing an ad e isemen sound.
Wai ing may help imp o ing he localiza ion while he sound
no i ies he ope a o s in he icini y abou he cu en obo
s a e and, i needed, abou he need o emo ing he obs acle
on he way.
The numbe o ials o pe o m he na iga ion a e con-
igu able. I he e a e no mo e ials le , meaning he obo
ailed o each he des ina ion, he ailu e is no i ied in he
unning na iga ion ailu e beha io s a e, and he asks o be
pe o med a his poin a e skipped, add essing he ollowing
a ge . The na iga ion s a e has been de eloped in a gene ic
way o suppo di e en global, ela i e, and p ecise na iga-
ion modules as i will be explained la e .
Once he na iga ion inishes co ec ly, he con igu ed se
o asks a e execu ed in he Doing Tasks s a e. A Task is
he implemen a ion o a obo s’ speci ic se o ac ions. The
p oposed a chi ec u e is designed o implemen new obo ic
asks by using he ROS pluginlib mechanism. The asks a e
de eloped as plugins which a e pa ame ized, dynamically
loadable and execu ed om a un ime lib a y. The plan gene -
a ed by he high-le el decision module mus con ain enough
in o ma ion o he s a e machine o unde s and whe e o go
and which ac ions o ake a each place. The e is he e o e no
need o ouch o ecompile he co e o he amewo k. This is
use ul o ex ending/modi ying he applica ion and p o ide a
g ea ex endibili y o he sys em. Sec ion V p esen wo ask
implemen a ions in he con ex o p ecision ag icul u e o
pes de ec ion and ea men while Sec ion VI-C illus a es an
addi ional ask example o an aile on inspec ion in an indus-
ial con ex . The bene i s and eusabili y o he a chi ec u e
is inally desc ibed in he discussions sec ion. Mo eo e , he
he e p esen ed ask plugins can be used as empla es and be
adap ed o u u e asks.
C. ABILITIES LAYER
This laye is composed by he ROS nodes in ol ed in he
basic con ol unc ionali ies o a obo . These nodes manage
he senso and ac ua o componen s, and p o ide obo capa-
bili ies such as au onomous na iga ion, manipula ion, and
inspec ion. A his le el, ROS p o ides a wide ange o s a e-
o - he-a obo ic algo i hms: GMaping [42] o gene a ing
maps using he on boa d 2D lase scanne s. The maps can be
manually modi ied o include, o ins ance, o bidden a eas
FIGURE 6. G eenPa ol obo desc ip ion in ROS- isualiza ion RViz.
I p esen s he main pla o m componen s (mobile pla o m, a m,
senso s...) and he ans o ma ion links be ween hem.
o he obo ; he Na iga ion S ack used in he Na iga ion
S a e o planning global and local pa hs. I uses combined
2D lase scanne s and sa elli e based localiza ion o gene a e
he eloci y commands o he mobile base while a oiding he
obs acles on he uns uc u ed g eenhouse en i onmen ; he
Uni ied Robo Desc ip ion URDF o gene a ing a combined
obo desc ip ion as p esen ed in Figu e 6; Mo eI ! is used
o gene a ing and execu ing collision ee manipula ion a-
jec o ies in he Doing Tasks S a e as i will be la e p esen ed
in Sec ion V. Mo eI ! ools can be also used, o in eg a e 3D
poin cloud based obs acles o use ul simula ion ools among
o he u ili ies.
On op o hem, se e al addi ional nodes ha e been de el-
oped. To ensu e he sys em equi emen SR1 and eely na -
iga e wi hin he g eenhouse, he localiza ion module bene i s
om he mul iple signal equencies and he highe accu-
acy p o ided by he Eu opean Global Na iga ion Sa elli e
Sys em (EGNSS) o he Galileo cons ella ion as explained
in [8]. The sys em equi emen SR3 is achie ed using a
deep lea ning model o de ec ing he mos ha m ul pes s
in g eenhouse oma o c ops: Bemisia Tabaci, Tu a Absolu a
and Whi e ly [10]. In addi ion, a lea de ec ion deep lea ning
model has been implemen ed o sa ely and accu a ely de ec
and app oach indi idual lea es using a 3D came a. Then,
close and high esolu ion pic u es o he pes s a e aken as
p esen ed in Sec ion 5, answe ing o sys em equi emen SR2.
The a chi ec u e includes addi ional modules o ela i e and
p ecise na iga ion which a e aluable o a wide ange o
mobile obo ic applica ions as shown in subsec ion 6-C.
D. DRIVERS LAYER
The d i e s laye includes he modules ha allow in e ac ing
wi h he obo pla o m senso s and ac ua o s. An o e iew
o he speci ic obo ic sys em used o alida ion pu poses is
shown in Figu e 7.
94986 VOLUME 9, 2021
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
FIGURE 7. G eenPa ol obo ic pla o m en e ing he g eenhouse.
FIGURE 8. G eenPa ol pes inspec ion and ea men ools moun ed a
he obo a ms end-e ec o .
The mobile pla o m consis s on he Segway
RMP 440 Omni Flex [43] wi h mecanum wheels o imp o e
mobili y in g eenhouse na ow co ido s. The pla o m is
equipped wi h an on-boa d PC, a Velodyne 3D lase scan-
ne [44] o obs acle de ec ion and wo OS32C sa e y lase
scanne s [45] o obs acle de ec ion, mapping and na iga ion.
The absolu e localiza ion uni consis o a mul i-cons ella ion,
GNSS ecei e , IMU and odome y. A KUKA LBR iiwa
manipula o [46] has been moun ed on he middle o he
pla o m o allow inspec ing he lea es on he igh and le
sides. The ision sys em consis s o a 3D RealSense cam-
e a [47] o ind lea es posi ions and an IDS RGB au o ocus
came a [48] o acqui e good quali y pic u es o he pes s as
seen in Figu e 8.
The sp aying equipmen consis s o a plas ic ank on he
back- igh co ne o he pla o m, an elec ic comp esso wi h
a pipe and a sp aying nozzle a he a m’s end-e ec o shown
in Figu e 8.
A bene i o using ROS is he a ailabili y o a wide a ie y
o obo ic componen s d i e s such as mobile obo s, manipu-
la o s, came as and, lase s. This makes he a chi ec u e ha d-
wa e agnos ic enabling he possibili y o eplace hem wi hou
a ec ing he es o he a chi ec u e.
E. MONITORING
The h ee modules shown on he le side o Figu e 2 a e
a ailable wi h he a chi ec u e o moni o he unc ional s a e
o he sys em. The Diagnos ics module has been designed
o collec ing and p ep ocessing speci ic da a om d i e s
and abili ies laye s which a e hen passed o he Diagnos-
ics Manage o au oma ic decision making and inciden s
no i ica ion ul illing sys em equi emen SR5. These wo
modules mus be adap ed o he applica ion on demand.
Mo eo e , he gene a ed DEBUG, INFO, WARNING and
ERROR messages a e eco ded by he Logging module using
a Rabbi MQ [49] queue ha implemen s he Ad anced Mes-
sage Queuing P o ocol (AMQP). The logs a e used o eco d
his o ical ack o he p ocess, ensu ing SR6, and can ei he
be s o ed locally in he obo o in he cloud using a non-
ela ional Elas icsea ch da abase [50]. This da a has been la e
used o ob ain es s esul s and s a is ics.
The ollowing Sec ion shows how o include new ad-hoc
modules and asks wi hin he a chi ec u e. In pa icula , he
in eg a ion o manipula ion s a egies o enhancing pes
de ec ion and ea men ope a ions a e p esen ed.
V. MANIPULATION STRATEGIES FOR PEST DETECTION
AND TREATMENT
The high-le el decision suppo sys em (in his case he IPM
s a egy) de ines he manipula ion, inspec ion, and ea men
asks o be pe o med. Fi s , he mobile pla o m needs o na -
iga e o he a ge plan s as seen in Sec ion 4-A. Once in on
o he plan , he obo ic a m moun ed on he middle- op o he
mobile pla o m pe o ms he co esponding pes inspec ion
o ea men ask on igh and le sides o he pla o m. This
Sec ion p esen s he s a egies aken, he execu ion wo k low
and examples o simple plans o each manipula ion ask.
The asks ha e been de eloped as plugins and ep esen s an-
dalone in eg a ion cases wi hing he a chi ec u e p esen ed
he e.
A. PEST INSPECTION TASK
The plan zones o be inspec ed and he numbe o pic u es
ha need o be aken a each zone a e ep esen ed as he
Pes Moni o ing Index (PMI) in Table 2 and Figu e 9. Lowe
and da ke zones end o p o ide mo e sui able habi a s o
he pes s, esul ing on a highe numbe o pic u es equi ed.
As an example, in he high-up zone he obo mus inspec
lea es abo e 1m (PMI6) and equi es wo pic u es o be aken,
while in he middle-bo om zone he obo mus inspec lea es
om bellow in be ween 0.5 m and 1 m (PMI2) and equi es
4 pic u es o be aken. A simple inspec ion plan is shown in
Figu e 11.
VOLUME 9, 2021 94987
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
TABLE 2. Pes moni o ing index o de ining he numbe o pic u es o
ake a each plan zone.
FIGURE 9. The G eenPa ol obo appea s acing he plan a he high-up
zone inspec ion posi ion in Gazebo simula ion.
The manipula ion s a egy o pes de ec ion consis s o
he wo k low de ined in Figu e 10 (up). Fi s , he a m
is mo ed o he nex inspec ion zone. Second, he lea
de ec o model and he RGBD image a e used o ind
lea es poses. I no lea is ound, he a m is mo ed o
he ollowing inspec ion zone. Thi d, he a m app oaches
he lea es ound in he p e ious s ep and akes close pic-
u es o hem using he RGB au o ocus came a. An algo-
i hm de e mines he quali y o he pic u e. I he quali y
is no good enough, he a m makes a p ede ined small
mo emen , and akes a new pic u e om he e. This p o-
cess is epea ed un il all equi ed plan zones ha e been
inspec ed.
The pic u es aken in his p ocess a e sa ed locally on
he obo . A e comple ing he plan, he pic u es a e sen
o he cloud, whe e a Deep Lea ning (DL) model has been
deployed o iden i y in ec ion a eas in he g eenhouse o line.
The IPM s a egy module uses he DL module esul s along
FIGURE 11. Example o a simple G eenPa ol inspec ion plan.
FIGURE 12. Example o a simple G eenPa ol sp aying plan.
wi h addi ional in o ma ion such as he cu en ha es season
condi ions, he wo king a ea size, he size o he plan o legal
aspec s on pes icides on he wo king coun y. As a esul , new
inspec ion (Figu e 11) and ea men (Figu e 12) plans a e
gene a ed.
FIGURE 10. Manipula ion s a egies wo k lows o pes de ec ion ask (up) and pes ea men ask (down).
94988 VOLUME 9, 2021
J. Ma in e al.: Gene ic ROS-Based Con ol A chi ec u e o Pes Inspec ion and T ea men
B. PEST TREATMENT TASK
The pes ea men p ocess can be de ined as he p ecise
sp aying o pes icide on di e en plan zones (high, middle
and low), being he pes icide sp aying dose a each plan
de e mined by IPM s a egy as shown in he pa ame e s ield
in Figu e 12.
The manipula ion s a egy is ep esen ed by he wo k low
de ined in Figu e 10 (down). Fi s , he a m is mo ed o he
nex sp aying zone. Second, he sp aye is ac i a ed and in
o de o co e he whole plan zone, he manipula o pe o ms
small, con olled mo emen s un il he comple e dose has been
sp ayed. This p ocess is epea ed un il all equi ed plan zones
ha e been sp ayed.
VI. SYSTEM VALIDATION
This sec ion p esen s he alida ion es s pe o med wi hin he
simula ed and eal g eenhouses o 52×30m and 31 co ido s
FIGURE 13. De ails o he simula ed en i onmen in Gazebo simula o
wi h he obo pe o ming na iga ion and inspec ion asks.
be ween plan s shown in Figu e 13 and Figu e 1 espec i ely.
The aim o hese es s has been he assessmen o he ollow-
ing ea u es: i s , he co ec in eg a ion o Robo amewo k
wi h he di e en obo ic modules; second, success ul execu-
ion o pes inspec ion and ea men plans; hi d, he logging
capabili ies o he sys em o gene a e and use he collec ed
da a; inally, he sys em equi emen s p oposed in Table 1.
The esul s and he mos ema kable conclusions a e
de ailed a he end o each es . Fu he mo e, he use o
Robo amewo k in an indus ial applica ion is p esen ed o
demons a e i s adap abili y and gene aliza ion.
A. GREENHOUSE SIMULATION TESTS
Gazebo simula o [51] has been used o simula e he
c ops, obo senso y in o ma ion (lase , images...), physics
in ol ed (collisions, ine ia...) and localiza ion da a (global
coo dina es, e o s...). The lea es, despi e ealis ic, do no
pe ec ly ep esen he eal wo ld and do no con ain insec s
on hem. Thus, a simula ed ision module o lea de ec-
ion p o ides hei posi ion. Also, he images used o al-
ida ing he pes de ec ion and iden i ica ion modules a e
semi- andomly acqui ed om ou cus om da ase o labelled
images (a se no used o aining he model) wi h in ec ed
and heal hy images. The same so wa e as in he eal scena io
has been used.
The simula ion es p esen ed in Figu e 14 consis s o he
ollowing s eps: Fi s , an IPM algo i hm gene a es a new semi
andom scou ing plan based on he g eenhouse dimensions,
which in his case co esponds o 31 ows (R1-R31) in he
ho izon al axis and 6 e ical zones ha , in u n, consis o
FIGURE 14. Rep esen a ion o he g eenhouse inspec ed/sp ayed zones du ing he di e en simula ion es -s eps: Ini ial semi andom scou ing plan (up).
Scou ing esul s ep esen ing he in ec ed and heal hy zones (middle). Sp aying plan gene a ed o in ec ed zones (down).
VOLUME 9, 2021 94989
