Methodology for car location and motion determination using camera sensors and image segmentation and classification

1
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
MÁSTER UNIVERSITARIO EN
INGENIERÍA INDUSTRIAL
TRABAJO FIN DE MÁSTER
METHODOLOGY FOR CAR LOCATION AND
MOTION DETERMINATION USING CAMERA
SENSORS AND IMAGE SEGMENTATION AND
CLASSIFICATION
Documen 1 – Memo y
Documen 2 – Anex I: Posi ion es ima o code
Documen 3 – Anex II: Road classi ie code
Es udian e Ande Ramos Mo eno
Di ec o Mikel Díez Sánchez
Co-Di ec o Asie Zubiza e a Picó
Depa amen o Au omá ica y con ol
Cu so académico 2019-2020
Bilbao, 12, Sep iemb e, 2020
2
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
ABSTRACT
In his documen a s udy o he di e en me hodologies o ob ain he posi ion o an
au onomous ehicle by using came a images is ca ied ou . The objec i e o he p ojec is no
only o calcula e he posi ion change o he ca om one ame o he ollowing one, bu also
o s udy he di e en sys ems o eloca e he ca in a map by de ec ing cha ac e is ic ea u es
o he en i onmen in he image, allowing o co ec he accumula ed de ia ion. Among he
di e en op ions o doing his elocaliza ion, i is de eloped a oad segmen a ion and
classi ica ion sys em, as i is conside ed o be he mos in e es ing op ion o he de ia ion
co ec ion p oblem.
RESUMEN
En es e documen o se lle a a cabo un es udio de las di e en es me odologías pa a ob ene la
posición de un ehículo au ónomo usando imágenes ob enidas median e cáma as. El obje i o
del p oyec o no es solo calcula el cambio de posición de una cap u a a o a, sino ambién
es udia los di e en es sis emas pa a elocaliza el coche en un mapa de ec ando de alles
ca ac e ís icos en la imagen, pe mi iendo co egi la des iación acumulada. En e las
di e en es opciones pa a hace es a elocalización, se desa olla un sis ema de segmen ación y
clasi icación de ca e e a, ya que se conside a que es la opción más in e esan e pa a el
p oblema de co ección de la des iación.
LABURPENA
Dokumen u hone an kame en e abile a i eske lo u ako i udiak e abili a ibilgailu au ónomo
ba en posizioa lo zeko me odología ezbe dinen az e ke a bu u zen da. P oiek ua en helbu ua
i udi ba e ik hu engo ako posizio di e en zia kalkula zea baka ik ez da, bai a mapa ba ean
au o ba bi lokaliz zeko sis ema ezbe dinen az e ke a e e. Ho e a ako i udien xehe asun
be eizga iak de ek a u egi en da, desbide apen me a ua zuzen dai zan. Bi lokalizazioa
lo zeko auke a ezbe dinen a ean, segmen azio sis ema e a e epide sailkapen ba ga a u da,
desbide apena en zuzenke a lo zeko auke a hobe ena bai a.
3
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
MÁSTER UNIVERSITARIO EN
INGENIERÍA INDUSTRIAL
TRABAJO FIN DE MÁSTER
METHODOLOGY FOR CAR LOCATION AND
MOTION DETERMINATION USING CAMERA
SENSORS AND IMAGE SEGMENTATION AND
CLASSIFICATION
MEMORY
Es udian e Ande Ramos Mo eno
Di ec o Mikel Díez Sánchez
Co-Di ec o Asie Zubiza e a Picó
Depa amen o Au omá ica y con ol
Cu so académico 2019-2020
Bilbao, 12, Sep iemb e, 2020
4
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
INDEX
FIGURES INDEX .............................................................................................................................. 5
GRAPHS INDEX .............................................................................................................................. 7
TABLES INDEX ................................................................................................................................ 8
LIST OF ACRONIMS ........................................................................................................................ 9
1. MEMORY ................................................................................................................................. 10
1.1. In oduc ion ..................................................................................................................... 10
1.2. Con ex ............................................................................................................................. 10
1.3. Objec i es and scope o he wo k .................................................................................... 15
1.4. Bene i s i b ings o wo k ................................................................................................. 15
1.5. Requi emen desc ip ion and analysis o he s a e-o - he-a ........................................ 16
1.5.1. Da abase sea ch ........................................................................................................ 16
1.5.2. Posi ion es ima ion algo i hms ................................................................................. 19
1.5.3. S udy o he me hods o elocaliza ion .................................................................... 20
1.6. Desc ip ion o he p oposed solu ion. Design .................................................................. 23
1.6.1. Da ase selec ion ....................................................................................................... 24
1.6.2. Selec ion o localiza ion me hod ............................................................................... 24
1.6.3. Selec ion o a me hod o elocaliza ion ................................................................... 25
2. METODOLOGY CARRIED OUT IN THE WORK DEVELOPMENT ................................................. 26
2.1. Tasks, phase, eams and p ocedu e desc ip ion .............................................................. 26
2.2. Gan diag am .................................................................................................................. 29
2.3. Calcula ions and algo i hms ............................................................................................. 29
2.3.1. Posi ion es ima o ..................................................................................................... 29
2.3.2. Road classi ie ............................................................................................................ 54
3. ECONOMIC ASPECTS ............................................................................................................... 65
4. CONCLUSIONS ......................................................................................................................... 67
5. BIBLIOGRAPHY ......................................................................................................................... 68
5
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
FIGURES INDEX
Figu e 1: LiDAR de ec ion ............................................................................................................ 12
Figu e 2: Came a o au onomous ca ........................................................................................ 13
Figu e 3: Ca e e ence sys em ................................................................................................... 13
Figu e 4: Came a e e ence sys em ............................................................................................ 14
Figu e 5: Le el 5 ca o da a collec ion ...................................................................................... 17
Figu e 6: PandaSe ca o da a collec ion .................................................................................. 18
Figu e 7: KITTI benchma k ........................................................................................................... 19
Figu e 8: 3D objec de ec ion ...................................................................................................... 21
Figu e 9: Seman ic segmen a ion................................................................................................ 21
Figu e 10: Image o oad a e isola ion o whi e colo .............................................................. 22
Figu e 11: SVM o classi ica ion ................................................................................................. 23
Figu e 12: Came a calib a ion chessboa d .................................................................................. 30
Figu e 13: 2D image pai ............................................................................................................. 31
Figu e 14: Scene econs uc ion ................................................................................................. 31
Figu e 15: Co ne desc ip o ....................................................................................................... 32
Figu e 16: Ma ching o wo consecu i e ames ......................................................................... 33
Figu e 17: RANSAC algo i hm applied o s aigh line app oxima ion ........................................ 33
Figu e 18: Possible came a posi ions .......................................................................................... 37
Figu e 19: Mul iplying he eloci y o he di ec ion ec o ........................................................ 38
Figu e 20: Ca in highway ............................................................................................................ 46
Figu e 21: Ca exi ing highway .................................................................................................... 46
Figu e 22: Ca u ning igh ......................................................................................................... 47
Figu e 23: Ca en e ing highway ................................................................................................. 47
Figu e 24: Ca abou o c oss pe pendicula ly he i s oad sec ion ......................................... 47
Figu e 25: Ca going s aigh ....................................................................................................... 49
Figu e 26: Ca going s aigh ....................................................................................................... 50
Figu e 27: Ca u ning igh ......................................................................................................... 50
Figu e 28: Ca going s aigh ....................................................................................................... 50
Figu e 29: Ca abou o u n le ................................................................................................. 51
Figu e 30: KITTI labeled image be o e p ep ocessing ................................................................. 54
Figu e 31: KITTI labeled image a e p ep ocessing .................................................................... 54
Figu e 32: P e ained CNNs accu acy s p edic ion speed ......................................................... 55
Figu e 33: Road segmen a ion CNN ............................................................................................ 56
Figu e 34: Lane segmen a ion CNN ............................................................................................. 56
Figu e 35: Noise co up ed de ec ion ......................................................................................... 56
Figu e 36: Noise il e ed auxilia image ...................................................................................... 57
Figu e 37: Classi ica ion lea ne app ........................................................................................... 58

6
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 38: Con usion ma ix o models 3 and 4 ......................................................................... 60
Figu e 39: Ca going s aigh ....................................................................................................... 61
Figu e 40: Ca eaching c oss oad ............................................................................................... 61
Figu e 41: Ca u ning le ........................................................................................................... 61
Figu e 42: Ca going s aigh ....................................................................................................... 62
Figu e 43: Ca s opping a a ic ligh ......................................................................................... 62
7
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
GRAPHS INDEX
G aph 1: "Main" diag am ............................................................................................................ 39
G aph 2: "Posi ion es ima o " diag am ....................................................................................... 41
G aph 3: "Edi image" diag am ................................................................................................... 43
G aph 4: "Es ima e ela i e posi ion" diag am ........................................................................... 44
G aph 5: Yaw measu emen esul s ............................................................................................ 48
G aph 6: Yaw esul s wi h di e en senso s ............................................................................... 51
G aph 7: Displacemen o no h wi h di e en senso s ............................................................. 52
G aph 8: Displacemen o eas wi h di e en senso s ............................................................... 52
G aph 9: Budge summa y .......................................................................................................... 66
8
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
TABLES INDEX
Table 1: Gan diag am ................................................................................................................ 29
Table 2: Road classi ie hi pe cen ages ...................................................................................... 63
Table 3: In e nal hou s ................................................................................................................ 65
Table 4: Amo iza ions ................................................................................................................ 65
Table 5: Budge summa y ........................................................................................................... 65
9
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
LIST OF ACRONIMS
• 2D: Two Dimensions
• 3D: Th ee Dimensions
• CNN: Con olu ional Neu al Ne wo k
• FAST: Fea u es om Accele a ed Segmen Tes
• GPS: Global Posi ioning Sys em
• HLS: Hue-Ligh -Sa u a ion
• IMU: Ine ial Measu emen Uni
• KNN: K-Nea es Neighbou s
• LiDAR: Ligh De ec ion And Ranging
• RANSAC: Random Sample Consensus
• RGB: Red-G een-Blue
• RMS: Roo Mean Squa e
• SURF: Speeded-Up Robus Fea u es
• SVD: Singula Value Decomposi ion
• SVM: Suppo Vec o Machine
16
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
1.5. REQUIREMENT DESCRIPTION AND ANALYSIS OF THE STATE-OF-THE-ART
1.5.1. DATABASE SEARCH
As we do no ha e he possibili y o c ea ing ou own da abase o images by means o a ca wi h
coupled senso s, he i s s ep is o look o he a ailable da abases o de eloping he p ojec .
In he As yx HiRes2019 Da ase , as s a ed in [3], we ound a da ase con aining 500 ames
ob ained by means o a colo came a o 30 Hz wi h a esolu ion o 2048 by 618 pixels, oge he
wi h a 13 Hz ada and a 10 Hz LiDAR. Despi e o he di e ence be ween he equencies o he
senso s, he 500 ames a e synch onized in ad ance and ha e o e 3000 3D objec anno a ions.
The wo k p oposed in he pape o he da ase , is ocused on ob aining high esolu ion ada
da a, howe e , came as a e also used o es ima e he ela i e posi ion o he ca , based on leas
squa es app oxima ion and applying he RANSAC algo i hm o ou lie ejec ion. I also pe o ms
de ec ion o di e en objec s on he oad (mainly ca s) by means o deep lea ning algo i hms
using no only he ada , bu also he came a and he LiDAR.
In he Be keley DeepD i e da ase , as s a ed in [4], we ound a da ase con aining 100,000 ideo
sequences in ol ing di e en imes o he day, wea he condi ions o d i ing scena ios. The
images con ain lane anno a ions oge he wi h bounding boxes anno a ions o di e en
ca ego ies and some pixel-le el anno a ions. The da ase is mean o be used o ul illing se en
asks ha a e needed o success ul au onomous d i ing, which a e: image agging, objec
de ec ion, lane ma king, d i able a ea de ec ion, seman ic segmen a ion, mul iple objec
acking and imi a ion lea ning.
As s a ed in [5], he Le el 5 da ase owned by Ly is public and con ains o e 100,000 hou s o
ideo sequences wi h he measu emen s o h ee 10 Hz LiDAR de ices, one on he op o he ca
and he o he wo placed nea each o he on wheels. I also con ains he measu emen s o
six 360 deg ees came as ha a e synch onized wi h he LiDAR de ices. The objec i e o he
da ase is o join o ces o accele a e he de elopmen o au onomous d i ing by sha ing he
da a ob ained by hei ca lee . They also sha e hei knowledge in o m o online webina s and
compe i ions.

17
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 5: Le el 5 ca o da a collec ion
The nuScenes da ase , as s a ed in [6], con ains he measu emen s o one LiDAR, 5 ada s, 6
colo came as, an IMU and a GPS o a o al o 1,400,000 images. The came as a e o 12 Hz and
ha e a esolu ion o 1600x900 pixels. The LiDAR has a equency o 20 Hz while he ada s ha e
a equency o 13 Hz. The IMU and GPS ha e a equency o 1000 Hz. I also has manual
anno a ions o 23 di e en objec ca ego ies. The da ase di e en ia es om o he exis ing
da ase s due o he 360 deg ees co e age o i s senso s and o ha ing da a ob ained a nigh
ime and ainy condi ions. Ca localiza ion is pe o med by c ea ing a map om he ob ained
poin s om he LiDAR. Then, Mon eCa lo Localiza ion scheme is applied by using he LiDAR and
came a measu emen s.
The Ox o d Rada Robo Ca da ase , as s a ed in [7], is ocused on alida ing he measu emen s
ob ained wi h he F equency-Modula ed Con inuous-Wa e class o ada , allowing obus
de ec ion in ad e se en i onmen al condi ions. Apa om he ada , he da ase con ains he
measu emen s o wo 3D LiDAR de ices and ano he wo 2D LiDAR de ices, a s e eo came a,
h ee monocula came as and an IMU/GPS sys em. The e is a o al amoun o 280 kilome es o
u ban d i ing in di e en wea he condi ions.
The PandaSe , as s a ed in [8], is made a ailable o bo h academic and comme cial use. I
con ains, apa om he GPS and IMU measu es, 48,000 came a images, 16,000 LiDAR sweeps,
ob ained om 6 came as and 2 LiDAR de ices. I also makes a ailable he labels o 37 ca ego ies
applying seman ic segmen a ion and 28 ca ego ies applying objec de ec ion. The objec i e o
he PandaSe is o p omo e and ad ance in he esea ch o au onomous d i ing and machine
lea ning by making hei in o ma ion public.
18
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 6: PandaSe ca o da a collec ion
In he KITTI Vision Benchma k Sui e, as s a ed in [2], i can be ound a p ojec by he Ka ls uhe
Ins i u e o Technology and he Toyo a Technological Ins i u e a Chicago. This benchma k
p o ides a lo o di e en da ase s o aw senso measu emen s, among which i can be ound
he images ob ained by ou came as, o which, wo a e g ayscale came as and he o he wo
a e colo came as. The image se s consis o di e en scena ios ha include ci y, esiden ial and
oad en i onmen s, including he co esponding calib a ion pa ame e s o each da ase . The
images ha e a heigh o 375 pixels and a wide o 1242 pixels.
Apa om he images ob ained by he ou came as, he measu emen s co esponding o he
LiDAR (Ligh De ec ion and Ranging), he IMU (Ine ial Measu emen Uni ) and he GPS. The
LiDAR and he came as wo k a 10 Hz while he es o he senso s wo k a 100 Hz.
The se con ains a o al amoun o six hou s o a ic scena ios, which suppose a o al numbe
o 216,000 images, wi h he objec i e o p o iding adequa e da a o de elop mul iple asks
necessa y o success ul au onomous d i ing. Among hese asks i can be ound s e eo ision,
objec de ec ion o op ical low wi h he app op ia e labels o de eloping he co esponding
Con olu ional Neu al Ne wo k i needed.
19
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 7: KITTI benchma k
1.5.2. POSITION ESTIMATION ALGORITHMS
Once he da ase s udy is done, a sea ch in he li e a u e o posi ion es ima ion me hods mus
be done.
Mos de eloped algo i hms ega ding image p ocessing o came a posi ion es ima ion a e
desc ibed on he book called “Mul iple- iew geome y in compu e ision”, as s a ed in [9]. In
i , we can ind heo e ical explana ions o how o analyse scenes ha ing di e en numbe o
images o hem om di e en poin s o iew. In his p ojec we will ocus mainly in he analysis
o scenes om wo poin s o iew, also called s e eo ision.
In he chap e desc ibing he s e eo ision concep and among u ili ies o ela ing wo images
and wha we can ge wi h hem, appea s he posi ion es ima ion by ea u e ma ching applying
he RANSAC algo i hm o ou lie ejec ion. The e a e some subchap e s showing s ep by s ep
he de elopmen o he ull me hod and he op ions i leads o in he posi ion es ima ion subjec .
Howe e , i is no he only me hod o ob ain he ca posi ion. In [14] a simple iangula ion
me hod is desc ibed o ind he ela i e posi ion om he loca ion o a pai o came as o he
objec i e ca in he image. This me hod, e en i i is a om ou objec i e, can be modi ied o
ins ead o calcula ing he dis ance o a mo ing objec (in his case, a ca ), he posi ion o any
20
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
ixed objec in he image. I he posi ion is again calcula ed o ha same objec in he ollowing
ame, he ela i e displacemen o he ca is ob ained.
In [15], i is desc ibed a sys em based on a monocula came a and an IMU ha can make a 6
deg ees o eedom posi ion es ima ion. The sys em is used o ecognize an image ha has
p e iously been seen and eloca ing he ca based on he loca ion o ha image. In addi ion, i
also uses he me hod o calcula e he ela i e pose be ween one ame o ano he , which is he
objec i e o ou p ojec . This is done by calcula ing he undamen al ma ix ha ela es each
image pai and applying he RANSAC algo i hm o ejec he possible ou lie s. This me hod o
posi ion es ima ion was also p oposed in [3] and is explained in [9]. This sys em can be ound
also de eloped in o he pape s such as [16], whe e he came a and he IMU a e used once again
o es ima e he ca loca ion.
1.5.3. STUDY OF THE METHODS FOR RELOCALIZATION
The second objec i e o he p ojec is he s udy o he di e en me hods o ex ac aluable
in o ma ion om he image such as a ic signals, oad shape o cha ac e is ic ea u es o
objec s. Fi s o all, he me hods o de ec ing objec s o pa s o he oad mus be ound so we
can selec he app op ia e one.
The mos used applica ions o his pu pose a e based on deep lea ning, which a e algo i hms
o au oma ic lea ning. Among he di e en ypes o a chi ec u es o deep lea ning, he CNN
(Con olu ional Neu al Ne wo k) is he mos app op ia e one o image p ocessing. To ob ain
he desi ed CNN o ou applica ion a aining p ocess mus be ca ied ou by using se e al
labelled images. I he aining is success ul he ne wo k is able o eplica e he p ocess o
de ec ion by i sel in images ha a e di e en om he ones o he aining se .
The mos di ec way o using deep lea ning o ou objec i e is using i o objec de ec ion,
which allows o de ec and ma k objec s in an image, in oducing hem in a bounding box. This
echnology can be e y use ul, as s a ed in [10], no only o de ec ing objec s such as ca s,
people o signals, bu also o inding i s posi ion and dimensions using he LiDAR and came a
measu emen s. In he pape i is p oposed a mul i- iew 3D objec de ec ion ne wo k ha
p edic s he 3D bounding boxes using a LiDAR on iew, a LiDAR bi ds-eye and an RGB image
as inpu s.
21
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 8: 3D objec de ec ion
I he ex ension o wha we wan o de ec is la ge o needs o be de ec ed in a mo e p ecise
way han in oducing i in a bounding box, a seman ic segmen a ion ne wo k could be ained,
which allows o make a pixel-le el classi ica ion o he image. The possibili ies o his me hod
a e bigge as enables he de ec ion o bigge ins ances such as buildings o he oad and, based
on i s shape, ins ances wi h complex geome y such as people o cyclis s. In [11] i can be ound
an expe imen al s udy o he applica ion o di e en segmen a ion ne wo ks in a d i ing
scena io. Bo h accu acy and aining ime a e ex ac ed as he mos impo an esul s and
pa ame e s o conside when choosing a CNN.
Figu e 9: Seman ic segmen a ion
Apa om he applica ions o deep lea ning algo i hms desc ibed be o e, he e a e many o he s
such as simple classi ica ion o using i o pe o m eg ession. This las one is also a e y use ul
ool o au onomous d i ing applica ions, as s a ed in [12], whe e a eg ession CNN is used o
de ec ing he lanes in he oad. In his pos , he concep o ans e lea ning is in oduced, being
a e y in e es ing me hod o sa e wo k and ime. T ans e lea ning consis s o e aining an
al eady ained ne wo k wi h many images wi h ou aining se , which leads o less compu a ion
and wo k as we do no ha e o build a cus omized ne wo k o ou applica ion.

22
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Howe e , he de ec ion o ea u es in an image is no only possible by applying deep lea ning
algo i hms, he e a e o he me hods, ha ha e he ad an age o a oiding he aining s ep. As
s a ed in [13], an RGB image can be con e ed o HLS (Hue-Ligh -Sa u a ion), which allows us o
isola e he pixels in an image wi h a de e mined colo despi e o he ligh ness. By applying i we
can isola e he cha ac e is ic colo o he lines in he oad and hen iden i y he shape o he
lines by using he Hough ans o m. This me hod could also be applied o he ecogni ion o
objec s o known colo and shape such as a ic signs.
Figu e 10: Image o oad a e isola ion o whi e colo
Once he esea ch o he exis ing de ec ion me hods has concluded, an in es iga ion o he
ecogni ion o classi ica ion me hods mus be done. As men ioned be o e, bo h objec de ec ion
and segmen a ion ne wo ks ha e he abili y o classi ying he ins ances hey de ec , howe e ,
he e a e o he classi ica ion me hods ha migh be be e and, he e o e, ha e o be
conside ed.
Suppo Vec o Machines (SVM) can be used as classi ie s, bu also o eg ession o o he asks
such as ou lie ejec ion. They a e supe ised lea ning models ha , gi en a se o aining
examples, each ma ked as belonging o one ca ego y, a aining algo i hm builds a model ha
assigns new examples o one ca ego y o he o he . As i can be seen, he s eps o gene a e he
classi ica ion model is simila o he one equi ed o ain a CNN, howe e , he compu a ion
ime equi ed o ain he SVM is much lowe .
23
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
The da a used o aining he model is used o de ine a hype plane ha de e mines in which
ca ego y alls a new example. In igu e 11 i can be seen how he hype plane is de ined acco ding
o he bes ma gin om he aining da a. In he example, a linea hype plane (linea SVM) can
be seen, bu he o de o he hype plane can be inc eased de ining, o example, a cubic SVM.
Figu e 11: SVM o classi ica ion
The K-Nea es Neighbou s (KNN) algo i hm can be used o classi ica ion o eg ession, being a
e y simple me hod in e ms o how i wo ks. When a new example is in oduced in he model,
he K closes aining examples a e analysed and seen which ca ego y hey belong o, assigning
he mos appea ing ca ego y o he new example. The alue o K, ep esen ing he neighbou s
ha a e conside ed, is de e mined acco ding o he ype o KNN algo i hm, o example, a Fine
KNN only conside s he closes neighbou o de e mining he ca ego y. Like he SVM, he ime
needed o aining a KNN model is much lowe han he one needed o a CNN.
1.6. DESCRIPTION OF THE PROPOSED SOLUTION. DESIGN
In his sec ion, we will analyse he di e en al e na i es discussed in he s a e-o - he-a sec ion
and choose he ones ha bes i ou objec i es. Fi s , among he p oposed da ase s he mos
app op ia e one o he p ojec will be chosen. Then, a me hod o ob aining he ca posi ion by
means o he came a will be de ined and inally, he de elopmen o a model wi h he abili y o
ex ac ing in o ma ion om a d i ing en i onmen will be de ined.
24
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
1.6.1. DATASET SELECTION
The i s s ep is he selec ion o he da abase o use. I is conside ed ha all he s udied da ase s
ha e enough da a o de eloping he equi ed asks. Howe e , he As yx HiRes2019 Da ase [3],
con ains jus 500 ames, which, e en i seem o be enough o he de elopmen , migh no be
enough o he alida ion.
Among he op ions, he nuScenes da ase [6] and he Ox o d Rada Robo Ca da ase [7] a e
ocused on alida ing he ada pe o mance in bad isibili y condi ions, which is a om ou
scope o he p ojec . The KITTI Vision Benchma k Sui e [2], is he selec ed one due o se e al
easons ha make i s and o e he es o he da ase s. The main ac o o i s selec ion is i s
use in he pa allel asks ha a e ca ied ou in he doc o al hesis p ojec . The use o a di e en
da ase s could lead o p oblems in he compa ibili y when using he measu emen s o he
senso s o he necessi y o modi ica ions o he algo i hms. Apa om ha , he KITTI da ase
con ains measu emen s ob ained wi h bo h colo and g ey came as, being he colo came as
necessa y o he second objec i e o he p ojec bu no as good as he g ey ones o he i s
pa due o he less con as o hei pic u es. I also has simila esou ces o he o he da ase s
in e ms o ma e ial o he de elopmen o di e en au onomous d i ing asks such as objec
de ec ion o seman ic segmen a ion.
1.6.2. SELECTION OF LOCALIZATION METHOD
Nex , he selec ion o he me hod o localize he ca is done. The ma ching me hod is selec ed
wi h ou lie ejec ion by using he RANSAC algo i hm used in [15], due o he de ailed
desc ip ion a ailable in [9] and he good esul s ob ained by i in some o he p e iously
men ioned pape s. The disad an age o he me hod is ha o ob aining he displacemen o
he ca , i is necessa y o use he measu emen s ob ained by he IMU, which educes, in pa ,
he edundancy o he sys em, being his one o he equi emen s o he p ojec .
The disca ded op ions include he iangula ion algo i hm used in [14] because e en i i is easy
o ind he posi ion di e ence o he ca , i is necessa y o de elop p e iously a de ec ion
algo i hm, being his he ha des pa o he p ocess. In ou case, o use his me hod, would
imply he de elopmen o a CNN o de ec ing a ixed poin and he de elopmen o some
me hod o make su e ha he poin we a e de ec ing is he same in all images. In addi ion, he
dep h es ima ion is one o he highes limi a ions o a came a, ob aining, in gene al, inaccu a e
esul s.
Also, he Mon e Ca lo localiza ion algo i hm desc ibed in [6] is an in e es ing op ion due o he
accu acy i has (10 cm e o ), howe e , he inpu s o he algo i hm a e composed, apa om
he images o he came a, o a poin cloud ob ained by he LiDAR and a map. The necessi y o
using he LiDAR o posi ion es ima ion would educe he edundancy o he sys em, educing
25
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
he da a o he senso usion. Anyway, in his p ojec we do no ha e any map a ailable, so he
de elopmen is no possible.
1.6.3. SELECTION OF A METHOD FOR RELOCALIZATION
A e s udying he exis ing me hods o de ec ing ins ances o elocaliza ion on he oad, i has
been decided o de elop a sys em able o de ec he oad and classi y i acco ding o he
ins ance he ca is a (s aigh , c oss oad, le cu e, igh cu e). One o he main easons o
selec ing his op ion is he da a p o ided by he KITTI da ase o building a CNN o segmen he
oad. As explained be o e, oad de ec ion can only be achie ed by using his ype o CNN due o
he size occupied by he oad in he images. The KITTI da ase p o ides a se o labelled images
o aining a CNN o segmen bo h he oad and he lane he ca is a .
I is also possible o ob ain a classi ica ion CNN ha analyses he aw image and ob ains di ec ly
he ca ego y o he image. The p oblem is ha we would ha e o label all he aining se and
ga he a big numbe o images. This is due o he ac ha mos o he ins ances on he oad
co espond o s aigh s, being necessa y o a good aining o ha e a balanced numbe o
aining in o ma ion pe ca ego y.
I is decided o ain he oad segmen a ion ne wo k wi h he KITTI da a and a e wa ds,
pos p ocess he ob ained in o ma ion and eed i o a classi ie . By doing his, we ha e he
possibili y o emo ing he noise ha migh appea in he ou pu o he segmen a ion ne wo k.
As he aining da a o he classi ie is e y li le o ain a classi ica ion CNN, i is decided o
ain a simple classi ie , such as an SVM o a KNN, which do no need a big amoun o aining
in o ma ion o pe o m well and equi e a e y educed ime o be ained.
Me hods like he one desc ibed in [13], a e no chosen due o he lack o obus ness compa ed
o a segmen a ion CNN and also because i he aining se is gi en, hey equi e, he same, o
e en mo e wo k han he selec ed op ion o i s de elopmen .
Finally, a so wa e ha inco po a es all he p e iously men ioned algo i hms mus be selec ed.
The a ailable p og ams o doing i we e Ma Lab and Py hon. Ma Lab was chosen due o he
highe knowledge o he use o he p og am and due o he big amoun o applica ions
inco po a ed in he p og am o isual analysis and au onomous d i ing asks. Addi ionally, i is
chosen o compa ibili y wi h he doc o al hesis de eloped in pa allel which uses Ma Lab.
32
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
The hi d possibili y is o calcula e he ela i e posi ion o he came as by knowing he came a
pa ame e s and he scene, which, using a single came a in mo ion a wo di e en ime ames,
is he same as calcula ing he came a posi ion change, which is ou goal.
O cou se, we do no need o know e e y single poin o he scene and i s loca ion in he eal
wo ld, which will make he p ocess oo complex. I is enough o ecognize some “pai s” o poin s
in he images, his is, he p ojec ion in he wo 2D images o a eal wo ld poin . This me hod is
called ma ching, and i will p o ide empi ical in o ma ion o ind a ela ion be ween he wo
images in he o m o a ma ix called he undamen al ma ix, as i will be explained la e .
2.3.1.2. Fea u e de ec ion and ma ching
Fo de ec ing a pai o poin s, as he scene is no known dimensionally speaking, we would need
o ma ch hese poin s by pe o ming a isual app oach, which is he same as ecognizing he
same poin in bo h images. Fi s , we need o de ec co ne s o ea u es in he image. We de ine
co ne s as poin s whose neighbou pixels s and in wo dominan and di e en edge di ec ions,
a be e unde s anding can be eached by looking a igu e 15.
Figu e 15: Co ne desc ip o
Fea u es e e o bounda ies be ween g oups o pixels such as sudden changes in colo o
geome y o objec s and, in gene al, co ne s a e a pa o hem. Ma Lab p o ides di e en
algo i hms o de ec he ea u es in an image. Each one has a di e en way o measu ing he
con as o a de e mined pixel and i s neighbou s. I is chosen he SURF (Speeded-Up Robus
Fea u es) de ec ion algo i hm due o he be e esul s ob ained expe imen ally agains o he
ea u e de ec o s such as FAST (Fea u es om Accele a ed Segmen Tes ) o he minimum
eigen alue ea u e de ec ion algo i hm. Once he ea u es o he pai o images is de ec ed, he
ma ching p ocess is ca ied ou , joining he ea u es o each o he images oge he .

33
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Howe e , among hese ma chings, he e will be some w ong ones, which implies ha poin s
ha a e ma ched oge he migh no ep esen he same ins ance in bo h images. These w ong
ma chings a e called ou lie s, while he igh ones a e called inlie s. In igu e 16 wo consecu i e
images aken om a ca a e shown o e laid wi h he ea u es co esponding o each image ( ed
ci cles and g een c osses) de ec ed and ma ched wi h each o he . Two ou lie s can clea ly be
seen ma ked in ed as hey a e ma ching poin s ha a e a away om each o he in he images.
Figu e 16: Ma ching o wo consecu i e ames
2.3.1.3. RANSAC algo i hm o ou lie ejec ion
The appea ance o ou lie s is a no mal hing in he ma ching p ocess, as he e a e many di e en
poin s in an image ha migh ha e simila cha ac e is ics ega ding he con as p ope ies o
hei neighbou pixels. The ejec ion o ou lie s is a p ocess wi h high impo ance as he use o
an ou lie in he calcula ion will de i e in a w ong solu ion o he p oblem.
The main p ocess o ejec ing ou lie s is he use o he RANSAC algo i hm. Fo a be e
unde s anding o how he algo i hm wo ks i will be explained o i s use in he simple case o
app oxima ing a s aigh line wi h a se o poin s.
Figu e 17: RANSAC algo i hm applied o s aigh line app oxima ion
34
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
In igu e 17, he solid poin s would be he a o emen ioned inlie s, while he open ones would
be he ou lie s. In g aph a i can be seen ha i he ou lie s a e no ejec ed, high de ia ion in
he inal esul appea s. The RANSAC algo i hm andomly selec s pai s o poin s and joins hem
wi h a line, de ining a e wa ds a h eshold dis ance om i . I he numbe o poin s wi hin he
de ined h eshold is low (2 o case c-d) i means ha a leas one o he poin s is an ou lie , in
his case, c. On he o he hand, i he numbe o poin s wi hin he h eshold is high (10 o case
a-b), i means ha he poin s a e inlie s.
A cu ious cha ac e is ic o he RANSAC algo i hm is ha , unlike o he app oxima ion algo i hms
ha y o use as much in o ma ion as possible o ob ain he solu ion and a e wa ds y o
elimina e he in alid da a, his me hod does he opposi e. I es ima es a solu ion wi h as small
da a as i can o he solu ion o be easible and hen enla ges he se wi h consis en da a i
possible.
2.3.1.4. Ob aining he undamen al ma ix
Once we ha e done he ou lie ejec ion, we need o ansla e he ma ched pai o poin s in o
an algeb aic exp ession. This algeb aic exp ession is wha we call he undamen al ma ix, a 3x3
ma ix ha ela es he poin s o an image wi h i s ma ches. The undamen al ma ix sa is ies he
ollowing exp ession:
𝒙′𝑇∙ 𝑭 ∙ 𝒙=0 (1)
Whe e 𝐹 is he undamen al ma ix, 𝑥 is a poin in one o he images and 𝑥′ is he ma ched poin
in he o he image. By using his exp ession we will be able o es ima e he undamen al ma ix.
I can be clea ly seen he impo ance o he ou lie ejec ion s ep, as he use o an ou lie o
es ima ion would mean in oducing a “ alse” pai o poin s in he equa ion and he e o e he
ob en ion o a w ong esul .
As 𝑥 and 𝑥′a e poin s o he o m (x,y,1), each poin ma ch a ises one linea equa ion including
he unknown en ies o 𝐹. I we de elop he p e ious equa ion we ob ain:
𝒙′𝒙𝑓11+𝒙′𝒚𝑓12+𝒙′𝑓13+𝒚′𝒙𝑓21+𝒚′𝒚𝑓22+𝒚′𝑓23+𝒙𝑓31+𝒚𝑓32+𝑓33=0 (2)
The n pai o poin s ob ained by he ma ching me hod de i e in n linea equa ions o es ima e
he undamen al ma ix. Howe e , one o he p ope ies o he undamen al ma ix, ha mus
be necessa ily ul illed is ha i is o ank 2. The solu ion ob ained by he se o linea equa ions
shown in equa ion (2), will be, in gene al, a ma ix o ank 3. The e o e, he de e minan o he
ob ained ma ix mus be o ced o be ze o once we ha e ob ained a solu ion based on equa ion
(2).
35
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
The minimum numbe o poin ma ches we need o ind he undamen al ma ix is se en, as i
will be explained now. Ha ing a se o linea equa ions o he o m A* = 0, o med by se en
equa ions as he one s a ed in (2), he solu ion is a wo dimensional space o he o m:
𝑭=𝛼∙𝑭𝟏+(1−𝛼)∙𝑭𝟐 (3)
Whe e 𝛼 is a scala a iable and he ma ices F1 and F2 a e ob ained as he ma ices
co esponding o he gene a o s 𝑓1 and 𝑓2 o he igh null-space o A. By o cing he
de e minan o be ze o as explained be o e we ob ain:
de (𝛼∙𝑭𝟏+(1−𝛼)∙𝑭𝟐)=0 (4)
Since 𝐹1 and 𝐹2 a e known, his leads o a cubic polynomial equa ion in 𝛼. When sol ing he
equa ion, we can ob ain one o h ee solu ions ( ejec ing he complex ones), which leads, when
subs i u ing in equa ion (3), also o one o h ee solu ions o he undamen al ma ix.
I he numbe o poin s a ailable is highe han se en, he so called 8-poin algo i hm can be
used, being a simple algo i hm han he explained p e iously. In his case, he esolu ion o he
linea equa ions leads o a unique solu ion by applying he equa ion used be o e and o cing he
de e minan o he undamen al ma ix o be ze o. Howe e , be o e sol ing he linea equa ions
i is necessa y o apply a no maliza ion o he inpu poin s, imp o ing he s abili y o he esul .
The needed no maliza ion is a ansla ion and scaling o each image placing a he o igin o he
coo dina es he cen oid o he e e ence poin s so ha he RMS dis ance o he poin s om he
o igin is equal o he squa e oo o 2.
As explained, assuming we ha e a la ge amoun o poin ma chings (mo e han 8), o es ima ing
he undamen al ma ix i is possible o use he 7-poin algo i hm o he 8-poin algo i hm. The
applica ion o each o he me hods implies some ad an ages and disad an ages, howe e , he
unc ion used by Ma Lab o es ima e he undamen al ma ix is he 8-poin algo i hm, being i s
modi ica ion oo complex and ou o he scope o he p ojec . The e o e, he undamen al
ma ix is ob ained by applying he 8-poin algo i hm.
2.3.1.5. Rela i e posi ion calcula ion
Once he undamen al ma ix is known, we wan o ind he ela i e posi ion o he came a in
he second pic u e compa ed o he i s one. This will be gi en by a o a ion ma ix (𝑅), and a
ansla ion ec o (𝑡), which is uni a y, acco ding o he ollowing exp ession:
𝑬=𝒕 ∙ 𝑹= 𝑲𝑇∙ 𝑭 ∙ 𝑲 (5)
Whe e F is he undamen al ma ix and E is he essen ial ma ix, which is also a ep esen a ion
o he ela ion o he ma ched poin s bu in eg a ing he came a pa ame e s. K is he in insic
ma ix, o he ollowing o m:
36
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
𝑲= [𝑓𝑥𝑠 𝑥0
0 𝑓𝑦𝑦0
0 0 1] (6)
Whe e 𝑓𝑥 and 𝑓𝑦 a e he ocal leng hs o he came a in x and y espec i ely, 𝑥0 and 𝑦0 ep esen
he o se o he p incipal poin o he came a and 𝑠 is he axis skew, which ep esen s he skew
coe icien be ween he x and y axis, which is no mally ze o.
The in insic ma ix is gi en in he da ase s ha a e being used, so he Essen ial ma ix can be
easily ob ained. To ob ain he ansla ion ec o and he o a ion ma ix, 𝐸 mus be
decomposed by SVD, ob aining:
𝑬≂𝑼𝑫𝑽𝑇 (7)
Whe e U and V a e o hogonal 3x3 ma ices while D is a diagonal ma ix o he o m:
𝑫=(ℰ 0 0
0 ℰ 0
0 0 0) (8)
Whe e ℰ a e he eigen alues o 𝐸 which mus be wo iden ical alues and a null one. Assuming
he ollowing ma ix o help sol e he p oblem, which is la e e i ied o be pa o he esul :
𝑾=(0 −1 0
100
001) (9)
The o a ion ma ix and he ansla ion ec o a e de ined as:
𝑹=𝑼∙𝑾−1∙𝑽𝑇 (10)
𝒕=𝑼∙𝑾∙𝑫∙𝑼𝑇 (11)
Howe e , he e a e wo possible solu ions o he o a ion ma ix and ano he wo o he
ansla ion ec o , which implies ou possible loca ions o he came a as i can be seen in he
ollowing image, whe e he poin s A and B ep esen he cen es o each came a wi h i s
co esponding possible 2D p ojec ions (images) o a eal wo ld 3D poin , ep esen ed as a big
black poin .
37
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 18: Possible came a posi ions
As i can be seen, he only possible solu ion is (a) as he poin s a e in on o he came as. This
is checked by analysing he posi ion o a single poin wi h espec o bo h came as and checking
i i is posi i e. The ejec ion o he h ee o he solu ions implies also he ejec ion o some pai
poin s ha a e ound o be ou lie s wi h he new si ua ion. O cou se, i is impo an o check
which is he pe cen age o ejec ed poin s as i i is high (mo e han 20 %), he undamen al
ma ix should be ecalcula ed, as i means ha i migh ha e no been calcula ed co ec ly.
Once hese pa ame e s a e ob ained, i is jus needed o add he in o ma ion p o ided by he
IMU, in his case, he eloci y, o ind how much he ca is displaced pe ame and mul iply i
by 𝑡 so we ob ain he absolu e displacemen o he ca .

38
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 19: Mul iplying he eloci y o he di ec ion ec o
2.3.1.6. Func ions
Once he algo i hm has been explained om a heo e ical poin o iew, he s eps ollowed by
each o he unc ions o he code a e desc ibed below oge he wi h a diag am so i can be
clea ly unde s ood. The cells in blue ep esen ac ions wi h no u he de elopmen , while he
cells in g een ep esen ac ions de ined by a sub unc ion ha will be explained in a mo e de ailed
diag am la e .
39
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
2.3.6.1.1. Main
G aph 1: "Main" diag am
40
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Fi s , he image da abase o be s udied is loaded by in oducing he pa h whe e he images a e
loca ed. Apa om i , he calib a ion pa ame e s o each o he ou came as will be needed
and also he eloci ies de ec ed by he IMU, which will be used la e .
The images a e picked in pai s (1-2, 2-3, 3-4…) and in oduced in he posi ion es ima o o each
o he came as. A e wa ds, hey go h ough an e o il e ha co ec s calcula ions ha could
no be made because he e was no enough da a o es ima e he undamen al ma ix o he
da a used is ound o be un eliable. Finally, we ob ain a se o da a co esponding o he yaw
a ia ion and ano he se co esponding o he uni a y ansla ion ec o om ame o ame.
To ob ain an absolu e ansla ion ec o we will ha e o make use o he IMU da a.
Be o e in oducing he IMU da a he ob ained ou pu goes h ough an e o il e . The e o
il e is esponsible o de ec ing he ames whe e he calcula ion could no be done o he
poin s whe e he calcula ion was w ong. I he calcula ion could no be done, he il e will ind
a ze o and will subs i u e i by he a e age alue ob ained om he wo closes alues o i ( he
one be o e and he one a e in gene al). To de e mine i he e is an e o , he maximum possible
alue o he yaw a ia ion has been selec ed by s udying he changes in he da a om ame o
ame and de ec ing isually, possible anomalies. A e he s udy, i is decided ha he yaw
a ia ion om one ame o he ollowing one whose alue is highe han 0.33 deg ees will be
conside ed as a w ong calcula ion, ca ying ou he same p ocess as be o e o i s co ec ion.
The IMU has a equency en imes highe han he came a, he e o e, i gi es en imes mo e
measu es. To adjus he equencies, he a e age alue o he en eloci ies co esponding o
each ame is calcula ed. Then, he dis ance a elled pe ame is calcula ed as:
𝑑𝑖𝑠𝑝𝑙𝑎𝑐𝑒𝑚𝑒𝑛𝑡 (𝑚)= 1
10(𝑠)∗ 𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦 (𝑚
𝑠) (12)
The en co esponds o he equency o he came a. Finally, he ob ained dis ance is mul iplied
o he uni a y ansla ion ec o o make i absolu e.
41
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
2.3.1.6.2. Posi ion es ima o
G aph 2: "Posi ion es ima o " diag am
48
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
The yaw is calcula ed om ame o ame and hen all he ob ained measu emen s a e added
o ob ain which is he o al u n desc ibed by he ca in he chosen da ase . The calcula ion is
done o he ou came as a ailable in he ca , con e ing, as explained, he colo images o he
wo colo came as o g ey. I is impo an o ake in o accoun ha he esul s ob ained by he
colo came as a e expec ed o be wo se han he ones ob ained wi h he g ey ones because he
images aken by hem ha e less con as , being his a key ac o in he ea u e de ec ion p ocess.
The esul s ob ained by each o he came as ega ding he yaw a ia ion a e plo ed in he
ollowing g aph, whe e he came a 0 ep esen s he le g ey came a, he came a 1 ep esen s
he igh g ey came a, he came a 2 ep esen s he le colo came a and he came a 3
ep esen s he igh colo came a.
G aph 5: Yaw measu emen esul s
-300
-290
-280
-270
-260
-250
-240
-230
-220
-210
-200
-190
-180
-170
-160
-150
-140
-130
-120
-110
-100
-90
-80
-70
-60
-50
-40
-30
-20
-10
0
10
20
010 20 30 40 50
Yaw a ia ion (deg ees)
Time (seconds)
Came a 0
Came a 1
Came a 2
Came a 3

49
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
The o al yaw calcula ed by each o he came as 0, 1, 2 and 3 a e -277.14 deg ees, -277.69
deg ees, -279.39 deg ees and -280.67 deg ees espec i ely. We can see ha he esul s ob ained
a e e y close o each o he and also close o he 270 deg ees app oxima e esul ha we
expec ed. Howe e , i is impo an o be awa e ha i he calcula ion is epea ed wi h exac ly
he same inpu da a, he esul s will a y due o he andom selec ion o poin s o he RANSAC
algo i hm. Anyway, he o al alue ob ained should be close o he ob ained ones in his
expe imen .
By looking a he g aph, i is in e es ing o ake a look a he smoo h a ia ion be ween he s a
o he da ase and he en h second and be ween he seconds 35 and he end, ep esen ing in
bo h cases a change o lane. I can also be clea ly seen he sho s aigh pa done when he ca
was exi ing he highway, which demons a es ha he algo i hm is wo king well.
2.3.1.7.2. Valida ion by compa ison
Once i has been checked ha he sys em is wo king p ope ly and gi ing p ope esul s, we can
compa e he esul s wi h he measu emen s ob ained by he GPS o quan i y he accu acy o
he sys em.
In his case, he da ase used con ains a se o 666 images in a esiden ial en i onmen . The ca
is inishing a le u n be o e acing a long s aigh . Then i u ns 90 deg ees o he igh ,
con inuing s aigh and inishing wi h a le u n o , again, 90 deg ees.
Figu e 25: Ca going s aigh
50
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 26: Ca going s aigh
Figu e 27: Ca u ning igh
Figu e 28: Ca going s aigh
51
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 29: Ca abou o u n le
The yaw has been calcula ed wi h di e en senso s and plo ed in he same g aph as i can be
seen in he ollowing pic u e:
G aph 6: Yaw esul s wi h di e en senso s
The line in ed ep esen s he measu emen s ob ained wi h he IMU, he yellow line ep esen s
he measu emen s ob ained wi h he GPS, he g een line ep esen s he measu emen s
ob ained wi h he came a and inally, he blue line ep esen s he measu emen s ob ained wi h
he LiDAR. The wo 90 deg ees cu es a e clea ly de ec ed as i can be seen in he pa s wi h
high slope o he g aph, which indica e a yaw a ia ion.
As i can be seen, he esul s ob ained wi h he came a a e logical compa ed o o he senso s
and s ays close o he GPS measu emen s han any o he o he wo. The GPS is assumed o be
52
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
he mos accu a e senso o he p esen ed ones. Howe e , his is no always ue, so i is
impossible o us o measu e wi h a pe cen age he accu acy o he algo i hm o show i s e o .
The posi ion esul s a e compa ed in wo di e en g aphs. One o hem shows he displacemen
o he no h agains ime and he o he one shows he displacemen o he eas agains ime.
G aph 7: Displacemen o no h wi h di e en senso s
G aph 8: Displacemen o eas wi h di e en senso s
The senso s a e ep esen ed wi h he same colo s as in he yaw g aph. I can be clea ly seen ha
he IMU and came a esul s a e almos he same in all he g aph, implying ha he e ec o he
uni a y ec o ob ained in he ma ching p ocess is minimum compa ed o he e ec gene a ed
53
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
by he IMU eloci y magni udes, being an in e es ing poin o wo k on in u u e p ojec s.
Howe e , as he esul s o he came a de ine he wo cu es in he se (which is, mainly,
esponsibili y o he uni a y ec o s), we can s a e ha he esul s ob ained a e logical.
To summa ize, i can be conclude ha he algo i hm is qui e good es ima ing he yaw a ia ion
o he ca , bu ha he limi a ion o needing a second senso o calcula e he displacemen om
ame o ame de i es in wo se esul s in his pa s, o a leas no use ul esul s o he senso
usion due o he dependence o he IMU.

54
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
2.3.2. ROAD CLASSIFIER
The posi ion es ima o is a g ea ool o analyse he posi ion o he ca in sho dis ances,
howe e , as ime goes on, he e o ha he es ima o migh ha e keeps inc easing. Fo ha
eason, i is in e es ing o eloca e he ca somehow om ime o ime. Assuming ha he maps
a e known, i we could de ec ha he ca has eached a c oss oad o a cu e, we could eplace
i s posi ion o he loca ion o he ins ance de ec ed.
The i s ask is o sepa a e he oad om he es o he image. The bes way o do his, as
explained in sec ion 1.6.3, is by aining a segmen a ion CNN, which will allow us o classi y he
pixels o he image in wo ca ego ies: “Road” and “Res ”. No mally, wha mos ime akes when
de eloping a CNN is ob aining p ope images and labelling hem. Howe e , we can ake
ad an age o he aining se p o ided in he KITTI da abase and skip his s ep.
The KITTI aining images make a dis inc ion be ween he oad co esponding o he sense he
ca is going and he coun e sense oad, he e o e, hey ha e h ee labels. As we a e no
in e es ed in de ec ing he coun e sense, we ha e o p ep ocess he labels be o e aining and
educe he ca ego ies o wo.
Figu e 30: KITTI labeled image be o e p ep ocessing
Figu e 31: KITTI labeled image a e p ep ocessing
55
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Ins ead o c ea ing ou own CNN o segmen ing he images, we c ea e a segmen a ion ne wo k
based on he p e ained ne wo k Resne 18, using he ans e lea ning echnique men ioned in
1.5.3 and appea ing in [12]. This educes he aining ime and complexi y. We choose Resne 18
as i can be easily modi ied o a segmen a ion ne wo k and due o i s good accu acy-p edic ion
speed a io ela ion, as i can be seen in he ollowing image.
Figu e 32: P e ained CNNs accu acy s p edic ion speed
As in he KITTI da abase labels a e gi en o ain also a lane segmen a ion ne wo k, we ake
ad an age o i and c ea e his ne wo k as well. The aining code used o he oad de ec o can
be eused o he lane de ec o , so i does no suppose an inc ease o wo k. Howe e , his CNN
was no ound use ul in his p ojec o ul il i s objec i es, so i will no be used in u he
calcula ions. Ne e heless, i s de elopmen migh be use ul in di e en asks such as inding
wha lane o he oad he ca is a , by o e lapping he wo CNNs.
Fo he oad segmen a ion CNN, 289 aining images a e used o h ee di e en ypes o oads:
u ban unma ked, u ban ma ked and u ban mul iple ma ked. Fo he lane segmen a ion CNN, 95
aining images o u ban ma ked oads a e used. Bo h CNNs a e es ed wi h a se o 290 images
wi h success ul esul s.
56
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Figu e 33: Road segmen a ion CNN
Figu e 34: Lane segmen a ion CNN
Once we a e able o di ide he oad om he es o he image, he de ec ion is pos p ocessed
wi h he objec i e o elimina ing he noise ha appea s in some o he images and o ob ain
nume ical da a abou he oad shape ha will be hen used o ain he oad classi ie .
Figu e 35: Noise co up ed de ec ion
The i s s ep is o cu he lowe pa o he image, whe e he ne wo k does no de ec well he
i s pixels o oad. Then, an auxilia image is c ea ed whe e he oad de ec ion appea s in whi e
57
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
while he es appea s black. A e ha , we smoo h he edge whe e black and whi e colou s
collide wi h a gauss il e and dele e he pixel g oups o he same colo ha a e smalle han
15000. This numbe has been selec ed acco ding o he maximum pixels ha can ga he
oge he as noise and in oducing a secu i y ma gin o 5000 pixels. In gene al, he g oup o pixels
belonging o he oad de ec ion wi hou noise is a o e his numbe .
Figu e 36: Noise il e ed auxilia image
As i can be seen in he image, he small g oups o pixels de ec ed as “Road” ha e disappea ed
as hey ep esen a smalle numbe han he speci ied. I is impo an o make hese wo s eps
as hey could lead o big e o s when ex ac ing da a om he de ec ion.
The se o images needed o he classi ie is no as easy o ob ain as he one used o he
segmen a ion ne wo k because, in his case, any image whe e a oad appea s is no alid. In
gene al, mos o he images co espond o s aigh s, while le cu es a e he less common ones.
Howe e , by lipping he images co esponding o igh cu es, which a e mo e common, we
can ob ain he same numbe o le ones. By applying he lipping me hod we inally ob ain 32
c oss oads, 33 le cu es, 33 igh cu es and 74 s aigh s.
The op ion o aining ano he CNN o he classi ica ion has been hough , which, in ac , would
be he mos obus op ion. Howe e , as men ioned in 1.6.3, he images a ailable o aining a e
oo ew o a CNN o each a p ope accu acy in de ec ion. Tha is he eason why ins ead o a
CNN i is decided o make a simple classi ie by using he Classi ica ion Lea ne app o Ma Lab.
The Classi ica ion Lea ne app allows us o ain di e en models wi h he da a we impo . Unlike
a CNN aining, whe e we could use he me ely pos p ocess image as aining, o aining he
di e en models in he app we need o gi e nume ical da a. The e o e, he nex s ep is o ex ac
cha ac e is ic poin s om he auxilia image o ob ain da a o he classi ie . The pa ame e s we
ex ac and he p ocedu es ollowed o ob ain hem a e desc ibed he e:
a) Pa ame e 1: Highes ow whe e a whi e pixel appea s.
b) Pa ame e 2: Middle poin o oad in he lowes ow.
64
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
E en i i does no appea in he se he e is also an impo an limi a ion o he sys em ha mus
be aken in o accoun . As he segmen a ion CNN is only ained o de ec he oad, i does no
de ec he ca s in he images, conside ing hose pa s ins ances o he images whe e he e is no
oad. This di ec ly implies ha he oad classi ie could no wo k in si ua ions wi h high a ic o
could show p oblems i a ca co e s a big pa o he oad appea ing in he image.

65
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
3. ECONOMIC ASPECTS
In ables 3 and 4 i can be seen he budge o he p ojec di ided in in e nal hou s and
amo iza ions. As i is an in es iga ion p ojec he e a e no ma e ials o p ocesses ha ha e o
be aken in o accoun . In he amo iza ion, i has been conside ed no only he cos o he
Ma lab so wa e i sel , bu also he ex a oolboxes ha a e needed o do his pa icula p ojec .
Table 3: In e nal hou s
Table 4: Amo iza ions
By aking in o accoun he o al cos s ob ained be o e we can build able 5, which leads o he
inal cos o he p ojec .
Table 5: Budge summa y
As i can clea ly be seen, mos o he cos s o he p ojec a e due o he human esou ces. In
g aph 9 i can be seen wha ac ion o he o al cos ep esen s each o he pa s.
Concep Hou s Uni a y p ice (€/h) Sub o al
Con ol enginee 450 50 22.500 €
TOTAL 22.500 €
In e nal hou s
Concep P ice (€) Use (hou s) Li e (hou s) Sub o al (€)
Ma Lab license 800 275 2400 91,67 €
Compu e ision oolbox license 500 250 2400 52,08 €
Image p ocessing oolbox license 400 250 2400 41,67 €
Deep lea ning oolbox license 460 175 2400 33,54 €
Compu e 1000 450 5000 90,00 €
TOTAL 308,96 €
Amo iza ions
Concep Cos (€)
In e nal hou s 22.500,00 €
Amo iza ions 308,96 €
Sub o al 22.808,96 €
Indi ec cos s (5%) 1.140,45 €
VAT (21%) 4.789,88 €
TOTAL 28.739,29 €
Summa y
66
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
G aph 9: Budge summa y
67
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
4. CONCLUSIONS
In his p ojec , a me hod o de e mining he ca loca ion based on came a images and aided by
he IMU and he di e en possibili ies o ex ac ea u es om an image ha e been explained.
Among hem, a simple oad segmen a ion CNN has been de eloped oge he wi h a classi ie
model o ob ain he oad ype he ca is a , wi h he objec i e o using i o eloca ing he ehicle
in a map.
The dependence o he IMU could be co ec ed by de eloping a ixed objec de ec ion ne wo k
and by applying iangula ion o he de ec ed objec s, which would allow o ob ain he dis ance
om he ehicle o he objec and om ame o ame, i s displacemen .
Also, he classi ie has been ained wi h e y ew images due o he educed ime o de elop
he p ojec and only ou ca ego ies a e in oduced. An inc ease in he numbe o aining
images would make possible o de elop a classi ica ion CNN, being his a much mo e obus
op ion han he de eloped KNN o SVM models. The inc ease in he ca ego ies is necessa y o
ecognize en i onmen s such as a c oss oad whe e he ca can go ei he s aigh o o he igh .
I is clea ha many o he esou ces o compu e ision p esen ed in he s a e-o - he-a
(sec ion 1.5) a e e y use ul ools o ca loca ion sys ems and in gene al, o au onomous
d i ing. In his p ojec , only a small app oach o hose me hods has been done, being he i s
s ep o ob ain a obus localiza ion sys em based no only in came a measu emen s bu also in
he o he desc ibed senso s.
68
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
5. BIBLIOGRAPHY
[1] E ik Coelingh, Lo a Jakobsson, Hen ik Lind, Magdalena Lindman. Collision wa ning wi h au o
b ake, a eal-li e sa e y pe spec i e. Vol o Ca co po a ion, Sweden, 2007.
[2] And eas Geige , Phillip Lenz, Ch is oph S ille , Raquel U asun. Vision mee s Robo ics: The
KITTI Da ase . In e na ional Jou nal o Robo ics Resea ch (IJRR), 2013.
[3] "Au omo i e Rada Da ase o Deep Lea ning Based 3D Objec De ec ion"; Meye , Michael
and Kuschk, Geo g; Eu opean Rada Con e ence 2019.
[4] Fishe Yu, Hao eng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashish
Madha an, T e o Da ell. BDD100K: A Di e se D i ing Da ase o He e ogeneous Mul i ask
Lea ning. UC Be keley, Co nell Uni e si y, UC San Diego, Elemen , Inc.
[5] h ps://sel -d i ing.ly .com/le el5/da a/?sou ce=pos _page---------------------------
[6] H. Caesa , V. Banki i, A. H. Lang, S. Vo a, V. E. Liong, Q. Xu, A. K ishnan, Y. Pan, G. Baldan and
O. Beijbom, “nuScenes: A mul imodal da ase o au onomous d i ing”, In a Xi p ep in
a Xi :1903.11027.
[7] D. Ba nes, M. Gadd, P. Mu cu , P. Newman, and I. Posne , “The Ox o d Rada Robo Ca
Da ase : A Rada Ex ension o he Ox o d Robo Ca Da ase ,” in P oceedings o he IEEE
In e na ional Con e ence on Robo ics and Au oma ion (ICRA), Pa is, 2020.
[8] h ps://scale.com/open-da ase s/pandase
[9] R. Ha ley, and A. Zisse man, “Mul iple View Geome y in Compu e Vision”. Camb idge
Uni e si y P ess, New Yo k, NY, USA, second edi ion, (2003).
[10] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia. Mul i-View 3D Objec De ec ion Ne wo k
o Au onomous D i ing. Depa men o Elec onic Enginee ing, Tsinghua Uni e si y. Baidu Inc.
[11] Çağ ı Kaymak, Ayşegül Uça . A B ie Su ey and an Applica ion o Seman ic Image
Segmen a ion o Au onomous D i ing. Fi a Uni e si y, Mecha onics Eng. Dep . 23119, Elazig,
Tu key.
[12] h ps://blogs.ma hwo ks.com/deep-lea ning/2017/11/17/deep-lea ning- o -au oma ed-
d i ing-pa -2-lane-de ec ion/
[13] h ps:// owa dsda ascience.com/ inding-lane-lines-simple-pipeline- o -lane-de ec ion-
d02b62e7572b
69
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
[14] Abdelmoghi Zaa ane, Ib issam Slimani, Wahban Al Okaishi, Issam A ou , Abdella i
Hamdoun. Dis ance measu emen sys em o au onomous ehicles using s e eo came a. LTI Lab,
Depa men o Physics, Facul y o Sciences Ben M’Sik, Uni e si y Hassan II O Casablanca,
Mo occo.
[15] Tong Qin, Peiliang Li, and Shaojie Shen. Relocaliza ion, Global Op imiza ion and Map
Me ging o Monocula Visual-Ine ial SLAM.
[16] Shujun Ma, Xinhui Bai, Yinglei Wang and Rui Fang. Robus S e eo Visual-Ine ial Odome y
Using Nonlinea Op imiza ion. School o Mechanical Enginee ing and Au oma ion, No heas e n
Uni e si y, China.
[17] h ps://semcon.com/o e ings/applied-au onomy/lida - s- ada - o -applied-au onomy/
[18] h ps://en.wikipedia.o g/wiki/Suppo _ ec o _machine#De ini ion
[19] h ps://en.wikipedia.o g/wiki/K-nea es _neighbo s_algo i hm#Algo i hm
[20] h ps://www.ma hwo ks.com/help/ ision/examples/seman ic-segmen a ion-using-deep-
lea ning.h ml
[21] h ps://medium.com/da a-b each/in oduc ion- o- ea u e-de ec ion-and-ma ching-
65e27179885d

1
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
MÁSTER UNIVERSITARIO EN
INGENIERÍA INDUSTRIAL
TRABAJO FIN DE MÁSTER
METHODOLOGY FOR CAR LOCATION AND
MOTION DETERMINATION USING CAMERA
SENSORS AND IMAGE SEGMENTATION AND
CLASSIFICATION
ANEX I – POSITION ESTIMATOR CODE
Es udian e Ande Ramos Mo eno
Di ec o Mikel Díez Sánchez
Co-Di ec o Asie Zubiza e a Picó
Depa amen o Au omá ica y con ol
Cu so académico 2019-2020
Bilbao, 12, Sep iemb e, 2020
2
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
INDEX
INDEX ............................................................................................................................................. 2
6. ANEX I: Posi ion es ima o code................................................................................................ 3
6.1. Main ................................................................................................................................... 3
6.2. Posi ion es ima o .............................................................................................................. 7
6.3.Edi image ............................................................................................................................ 8
6.4. Es ima e ela i e posi ion .................................................................................................. 9
3
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
6. ANEX I: POSITION ESTIMATOR CODE
6.1. MAIN
clc
close all
clea all
%In oduce image pa h he e:
imagePa h = 'J: Sc ip ImageSe s 0042 2011_10_03_d i e_0042_ex ac ';
%The da a co esponding o he IMU measu emen s is loaded
load IMU_0042.ma
%To i he IMU measu emen s o he came a measu emen s we ha e o
%make he a e age alue o each 10 measu emen s mo e o less
x = IMU(1:end-15,1);
y = IMU(1:end-15,2);
z = IMU(1:end-15,3);
n=0;
k=0;
o i=1:11760
n=n+1;
i n>28
n=0;
k=k+1;
end
k=k+1;
xmod(i,1) = x(k,1);
ymod(i,1) = y(k,1);
zmod(i,1) = z(k,1);
end
c=0;
o h=1:1175
xa g=0;
ya g=0;
za g=0;
o =1:10
c=c+1;
xaux = xmod(c,1);
xa g = xa g+ xaux;
yaux = ymod(c,1);
ya g = ya g+ yaux;
zaux = zmod(c,1);
za g = za g+ zaux;
end
xa g = xa g/10;
ya g = ya g/10;
za g = za g/10;
4
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
(h,1) = ( xa g^2+ ya g^2+ za g^2)^0.5; %Ex ac ion o he modulus
end
d = *0.1; %The magni ude o he ad anced dis ance is calcula ed
%The in o ma ion o each came a is ex ac ed om he da ase
se 0 = s ca (imagePa h,'image_00 da a');
se 0 = se 0( ind(~isspace(se 0)));
se 0 = imageDa as o e( ull ile(se 0));
se 1 = s ca (imagePa h,'image_01 da a');
se 1 = se 1( ind(~isspace(se 1)));
se 1 = imageDa as o e( ull ile(se 1));
se 2 = s ca (imagePa h,'image_02 da a');
se 2 = se 2( ind(~isspace(se 2)));
se 2 = imageDa as o e( ull ile(se 2));
se 3 = s ca (imagePa h,'image_03 da a');
se 3 = se 3( ind(~isspace(se 3)));
se 3 = imageDa as o e( ull ile(se 3));
%Calib a ion pa ame e s
%K = In insic/Calib a ion ma ix
%D = Dis o sion coe icien s
K0 = [9.799200e+02 0.000000e+00 6.900000e+02;
0.000000e+00 9.741183e+02 2.486443e+02;
0.000000e+00 0.000000e+00 1.000000e+00]';
D0 = [-3.745594e-01 2.049385e-01 1.110145e-03 1.379375e-03 -7.084798e-
02];
K1 = [9.903522e+02 0.000000e+00 7.020000e+02;
0.000000e+00 9.855674e+02 2.607319e+02;
0.000000e+00 0.000000e+00 1.000000e+00]';
D1 = [-3.712084e-01 1.978723e-01 -3.709831e-05 -3.440494e-04 -
6.724045e-02];
K2 = [9.601149e+02 0.000000e+00 6.947923e+02;
0.000000e+00 9.548911e+02 2.403547e+02;
0.000000e+00 0.000000e+00 1.000000e+00]';
D2 = [-3.685917e-01 1.928022e-01 4.069233e-04 7.247536e-04 -6.276909e-
02];
K3 = [9.049931e+02 0.000000e+00 6.957698e+02;
0.000000e+00 9.004945e+02 2.389820e+02;
0.000000e+00 0.000000e+00 1.000000e+00]';
D3 = [-3.735725e-01 2.066816e-01 -6.133284e-04 -1.193269e-04 -
7.600861e-02];
2
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
INDEX
INDEX ............................................................................................................................................. 2
7. ANEX II: Road classi ie code ..................................................................................................... 3
7.1. Coun e oad dele e ........................................................................................................... 3
7.2. Segmen a ion CNN aining ............................................................................................... 4
7.3. Pa ame e ex ac ion ( o aining and o ob aining esul s) ........................................... 6
7.4. Flipe ................................................................................................................................. 13

3
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
7. ANEX II: ROAD CLASSIFIER CODE
7.1. COUNTER ROAD DELETE
clc
close all
clea all
%Changes he labeled pixels as "coun e lane" (black) o
% he " es " ca ego y ( ed)
%In oduce image olde pa h
labDi = ull ile('G: TFM Sc ip 6-Road Ki i da a_ oad lab');
sds=imageDa as o e(labDi );
labDi Aux = di ('G: TFM Sc ip 6-Road Ki i da a_ oad lab *.png');
Red = [255 0 0];
Black = [0 0 0];
o z=1:289 %Se he e he numbe o labels
im = eadimage( sds,z);
o i=1:375
o j=1:1242
i im(i,j,:)==Black
im(i,j,:) = Red;
end
end
end
ilename = s ca ('G: TFM Sc ip 6-
Road Ki i da a_ oad lab_2ca '...
,labDi Aux(z).name);
imw i e(im, ilename,'png');
end
4
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
7.2. SEGMENTATION CNN TRAINING
clc
close all
clea all
%images
imgDi = ull ile('G: TFM Sc ip 6-Road Ki i da a_ oad img');
imds = imageDa as o e(imgDi );
classes = [
"Road"
"Res "];
%labels
labelDi = ull ile('G: TFM Sc ip 6-Road Ki i da a_ oad lab');
Red = [255 0 0;];
Pink = [255 0 255;];
pxds = pixelLabelDa as o e(labelDi ,classes,{Pink Red});
% Speci y he ne wo k image size. This is ypically he same as he
aining image sizes.
imageSize = [375 1242 3];
% Speci y he numbe o classes.
numClasses = numel(classes);
ne = esne 18;
%C ea e ne
lg aph = deeplab 3plusLaye s(imageSize, numClasses, " esne 18");
%Se augmen e o inc ease aining in o
pixelRange = [-30 30];
augmen e = imageDa aAugmen e ( ...
'RandXRe lec ion', ue, ...
'RandXT ansla ion',pixelRange, ...
'RandYT ansla ion',pixelRange);
%C ea e image and label da as o e
pximds = pixelLabelImageDa as o e(imds,pxds, ...
'Da aAugmen a ion',augmen e );
% De ine aining op ions.
op ions = ainingOp ions('sgdm', ...
'Lea nRa eSchedule','piecewise',...
'Lea nRa eD opPe iod',10,...
'Lea nRa eD opFac o ',0.3,...
'Momen um',0.9, ...
'Ini ialLea nRa e',1e-3, ...
'L2Regula iza ion',0.005, ...
'MaxEpochs',30, ...
5
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
'MiniBa chSize',8, ...
'Shu le','e e y-epoch', ...
'Checkpoin Pa h', empdi , ...
'Ve boseF equency',2,...
'Plo s',' aining-p og ess');
[ne , in o] = ainNe wo k(pximds,lg aph,op ions);
6
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
7.3. PARAMETER EXTRACTION (FOR TRAINING AND FOR OBTAINING
RESULTS)
clc
close all
clea all
load 'RoadDe ec o 2.ma ' %In oduce segmen a ion ne name
load 'Model5.ma ' %In oduce p edic o name
%In oduce image pa h
es Di = ull ile('J: TFM Calculos Sc ip ImageSe s 0009 2011_09_26_d i
e_0009_ex ac image_02 da a'); %In oduce images olde pa h
sds=imageDa as o e( es Di );
Ini ialize = 0;
o s=1:453 %In oduce he numbe o pic u es in ile
disp(['Pic u e ',num2s (Ini ialize +s)])
I = eadimage( sds,Ini ialize +s); %Read image
%Adjus men o image size
I = im esize(I,[375,1242]);
[heigh ,wide,channels] = size(I);
C = seman icseg(I, ne ); %Segmen image
B = labelo e lay(I,C,'T anspa ency',0.4); %Visualize segmen ed
image
% % igu e
% % imshow(B)
%Pos -p ocess
Iaux = gb2g ay(I); %C ea e auxilia black and whi e image
%Cu he lowe pa o he image whe e dis o ion is high
heigh =heigh -60;
Iaux = imc op(Iaux,[0 0 wide heigh ]);
o i=1:heigh %Pain oad whi e and es in black
o j=1:wide
i C(i,j)=='Road'
Iaux(i,j) = 1000;
else
Iaux(i,j) = 0;
end
7
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
end
end
Iaux = imgauss il (Iaux); %Apply Gauss il e o smoo h
Iaux = bwa eaopen(Iaux,15000); %Dele e pixel g oups
%smalle han 15000, changes o logical
%pixelAmoun calcula ion
pixelAmoun =0;
o i=1:heigh
o j=1:wide
i Iaux(i,j)==1
pixelAmoun = pixelAmoun +1;
end
end
end
%Calcula ion o highes poin in image o oad
i=1;
j=1;
while Iaux(i,j)==0
i j==wide
j=0;
i=i+1;
end
i i==heigh
maxHeigh = i;
b eak
end
maxHeigh = i;
j=j+1;
end
%I he e is no de ec ion, we in oduce a ze o in all
pa ame e s
i maxHeigh == heigh
Va 1(s,1) = maxHeigh ;
Va 2(s,1) = 0;
Va 3(s,1) = 0;
Va 4(s,1) = 0;
Va 5(s,1) = 0;
Va 6(s,1) = 0;
Va 7(s,1) = 0;
Va 8(s,1) = 0;
Va 9(s,1) = 0;
Va 10(s,1) = 0;
Va 11(s,1) = 0;
Va 12(s,1) = 0;
Va 13(s,1) = 0;
Va 14(s,1) = 0;
Va 15(s,1) = 0;
Va 16(s,1) = 0;
Va 17(s,1) = 0;

8
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Va 18(s,1) = 0;
Va 19(s,1) = 0;
con inue
end
%Region o analysis de ini ion
egionTop = heigh - ound((heigh -maxHeigh )*0.8);
egionBo om = heigh ;
%Calcula ion o middle poin in lowe ow
=0;
a g=0;
coun e =0;
o j=1:wide
i Iaux(heigh ,j) == 1
a g=a g+j;
coun e =coun e +1;
end
end
i a g ~= 0
a g= ound(a g/coun e );
else
disp('ERROR: a g=0');
end
%Le and igh poin s a ay ob en ion
%Le
z=0;
o i= egionTop: egionBo om
z=z+1;
o j=1:wide
i Iaux(i,j)==1
le Poin s(z,1)=j;
le Poin s(z,2)=-i;
b eak
end
end
le Dis ance(z,1)=a g-j;
end
%Righ
z=0;
o i= egionTop: egionBo om
z=z+1;
o j=1:wide
i Iaux(i,wide+1-j)==1
igh Poin s(z,1)=wide+1-j;
igh Poin s(z,2)=-i;
b eak
end
end
igh Dis ance(z,1)=wide+1-j-a g;
9
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
end
ba ch = ound(z/4);
%Cen al poin s a ay
a gPoin s(:,2) = -( egionTop: egionBo om);
a gPoin s(:,1) = a g;
%Line app oxima ion and angle calcula ion o le and igh
poin s
le Line = i (le Poin s(:,1),le Poin s(:,2),'poly1');
igh Line = i ( igh Poin s(:,1), igh Poin s(:,2),'poly1');
le Angle = angleCalcula ion(le Line);
igh Angle = angleCalcula ion( igh Line);
% %Poin ep esen a ion
% igu e
% sca e (le Poin s(:,1),le Poin s(:,2));
% % hold on
% % plo (le Line)
% hold on
% sca e ( igh Poin s(:,1), igh Poin s(:,2));
% % hold on
% % plo ( igh Line)
% hold on
% sca e (a gPoin s(:,1),a gPoin s(:,2));
% %Bo de ep esen a ion(no used o calcula ions)
% Iaux = edge(Iaux,'Canny',0.2,0.5); %Ex ac whi e and black
edges
% Iaux=bwa eaopen(Iaux,10);
% Iaux=imdila e(Iaux, ue(3));
%
% igu e
% imshow(Iaux)
%A e age alues o le and igh dis ances
le A gDis ance = mean(le Dis ance);
le Dis ance1 = le Dis ance(1,1);
le Dis ance2 = le Dis ance(1+ba ch,1);
le Dis ance3 = le Dis ance(1+2*ba ch,1);
le Dis ance4 = le Dis ance(1+3*ba ch,1);
le Dis ance5 = le Dis ance(end,1);
igh A gDis ance = mean( igh Dis ance);
igh Dis ance1 = igh Dis ance(1,1);
igh Dis ance2 = igh Dis ance(1+ba ch,1);
igh Dis ance3 = igh Dis ance(1+2*ba ch,1);
igh Dis ance4 = igh Dis ance(1+3*ba ch,1);
igh Dis ance5 = igh Dis ance(end,1);
10
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
%Pa ame e in oduc ion
Va 1(s,1) = maxHeigh ;
Va 2(s,1) = a g;
Va 3(s,1) = le Angle;
Va 4(s,1) = igh Angle;
Va 5(s,1) = le A gDis ance;
Va 6(s,1) = le Dis ance1;
Va 7(s,1) = le Dis ance2;
Va 8(s,1) = le Dis ance3;
Va 9(s,1) = le Dis ance4;
Va 10(s,1) = le Dis ance5;
Va 11(s,1) = igh A gDis ance;
Va 12(s,1) = igh Dis ance1;
Va 13(s,1) = igh Dis ance2;
Va 14(s,1) = igh Dis ance3;
Va 15(s,1) = igh Dis ance4;
Va 16(s,1) = igh Dis ance5;
Va 17(s,1) = pixelAmoun ;
ela ionHeigh Dis anceLe = le A gDis ance/maxHeigh ;
Va 18(s,1) = ela ionHeigh Dis anceLe ;
ela ionHeigh Dis anceRigh = igh A gDis ance/maxHeigh ;
Va 19(s,1) = ela ionHeigh Dis anceRigh ;
%In oducing pa ame e s in able and applying sliding window.
%This is done when calcula ing, no when ex ac ing
in o ma ion
% o aining models
i s>4
Taux = able(Va 1(end-4:end,1),Va 2(end-4:end,1),Va 3(end-
4:end,1)...
,Va 4(end-4:end,1),Va 5(end-4:end,1),Va 6(end-
4:end,1),Va 7(end-4:end,1)...
,Va 8(end-4:end,1),Va 9(end-4:end,1),Va 10(end-
4:end,1),Va 11(end-4:end,1)...
,Va 12(end-4:end,1),Va 13(end-4:end,1),Va 14(end-
4:end,1),Va 15(end-4:end,1)...
,Va 16(end-4:end,1),Va 17(end-4:end,1),Va 18(end-
4:end,1),Va 19(end-4:end,1));
Taux.P ope ies.Va iableNames =
{'maxHeigh ','a g','le Angle',...
' igh Angle','le A gDis ance','le Dis ance1','le Dis ance2',...
'le Dis ance3','le Dis ance4','le Dis ance5',' igh A gDis ance',...
' igh DIs ance1',' igh Dis ance2',' igh Dis ance3',' igh Dis ance4',..
.
' igh Dis ance5','pixelAmoun ',' ela ionLe ',' ela ionRigh '};
esul = Model5.p edic Fcn(Taux);
inalResul (s,1) = pas F ames( esul )
else
11
BILBOKO
INGENIARITZA ESKOLA
ESCUELA
DE INGENIERÍA DE BILBAO
Taux = able(Va 1,Va 2,Va 3...
,Va 4,Va 5,Va 6,Va 7...
,Va 8,Va 9,Va 10,Va 11...
,Va 12,Va 13,Va 14,Va 15...
,Va 16,Va 17,Va 18,Va 19);
Taux.P ope ies.Va iableNames =
{'maxHeigh ','a g','le Angle',...
' igh Angle','le A gDis ance','le Dis ance1','le Dis ance2',...
'le Dis ance3','le Dis ance4','le Dis ance5',' igh A gDis ance',...
' igh DIs ance1',' igh Dis ance2',' igh Dis ance3',' igh Dis ance4',..
.
' igh Dis ance5','pixelAmoun ',' ela ionLe ',' ela ionRigh '};
esul = Model5.p edic Fcn(Taux);
i esul =='S aigh '
inalResul (s,1)='S';
elsei esul =='C oss oad'
inalResul (s,1)='C';
elsei esul =='Le cu e'
inalResul (s,1)='L';
else
inalResul (s,1)='R';
end
end
clea a gPoin s igh Poin s le Poin s maxheigh a g le Angle
igh Angle...
le Dis ance igh Dis ance le Dis ance1 igh Dis ance1
le Dis ance2 igh Dis ance2...
le Dis ance3 igh Dis ance3 le Dis ance4 igh Dis ance4
le Dis ance5 igh Dis ance5...
ba ch Taux esul
end
% %In oduc ion o da abase in Excel o aining
% T =
able(Va 1,Va 2,Va 3,Va 4,Va 5,Va 6,Va 7,Va 8,Va 9,Va 10,Va 11,Va 12,V
a 13,Va 14,Va 15,Va 16,Va 17,Va 18,Va 19);
% ilename = 'imageIn o.xlsx';
% ini ialCell = 'F88';
%
w i e able(T, ilename,'Shee ',1,'Range',ini ialCell,'W i eVa iableName
s',0);
%In oduc ion o da a in Excel o esul s
Taux = able(Va 1,Va 2,Va 3...
,Va 4,Va 5,Va 6,Va 7...
,Va 8,Va 9,Va 10,Va 11...