Machine Learning algorithm evaluation on advanced driver assistance

EVALUACIÓN DE ALGORITMOS DE MACHINE LEARNING PARA CONDUCCIÓN MACHINE LEARNING ALGORITHM EVALUATION ON ADVANCED DRIVER ASSISTANCE TRABAJO FIN DE GRADO CURSO 2020-2021 BY WENBO SUN DIRECTOR CARLOS GARCÍA SÁNCHEZ GUILLERMO BOTELLA JUAN GRADO EN INGENIERÍA INFORMÁTICA FACULTAD DE INFORMÁTICA UNIVERSIDAD COMPLUTENSE DE MADRID THANKS I am e y g a e ul o he wo eache s o hei help in his g. F om he beginning, I didn' know and didn' know how o do i , no only ga e me he gene al di ec ion o he p ojec , bu also ga e me some necessa y in o ma ion. In he la e pe iod, Ca los poin ed me o he sho comings o he p ojec and encou aged me o comple e he p ojec . Fo his I mus say ha he eache s a e e y good. I am e y g a e ul and g a e ul o hem. A he same ime, I am e y g a e ul o he school o he help I lea ned du ing he epidemic. ABSTRACT Machine lea ning algo i hm e alua ion on ad anced d i e assis ance In his esea ch and de elopmen p ojec , ou main pu pose is o s udy ou deep lea ning a chi ec u es o eal- ime objec de ec ion o people and bicycles encoun e ed in on o d i ing. We use 4 di e en algo i hms o he same da a se , and compa e he mAPs ob ained a e aining. And discuss which me hod is he mos accu a e, bu also conside he ime i akes o ge wha is sui able o wha kind o scene. The p ojec I came up wi h would like o be used in a d i ing assis ance sys em. The sys em uses came a senso s o ge inpu , and hen uses algo i hms o assis , so ha he sa e y o he ca is gua an eed when d i ing. A he same ime, i can un on a low- pe o mance e sion o he machine and compa e he ps o di e en algo i hms. Keywo ds Compu e ision, au onomous d i ing, con olu ional neu al ne wo k, objec de ec ion, deep lea ning, machine lea ning, ea u e selec ion and ex ac ion, d i ing assis ance INDEX OF CONTENTS Chap e 1 -!In oduc ion ............................................................................................... 1! Chap e 1.1 -!Mo i a ion ........................................................................................ 2! Chap e 1.2 -!Objec i e ......................................................................................... 4! Chap e 2 -!Con ibu ion o s uden ............................................................................ 5! Chap e 2.1 -!Plan o Wo ks .................................................................................... 5! Chap e 2.2 -!Flow Cha ........................................................................................ 6! Chap e 3 -!Ad anced and a chi ec u e ................................................................... 7! Chap e 3.1 -!Con olu ional Neu al Ne wo k, CNN ............................................ 7! Chap e 3.2 -!Ta ge de ec ion ne wo k ............................................................ 10! Chap e 3.2.1 -!Two s age ............................................................................. 10! 1.!Fas e -RCNN[4] ................................................................................... 10! 2.!Fpn[5] ................................................................................................... 13! Chap e 3.2.2 -!One s age ............................................................................ 17! 1.!SSD[6] ................................................................................................... 17! 2.!VGG16[8] ............................................................................................ 18! 3.!MobileNe V3[9] ................................................................................... 20! 4.!YOLO 4[7] ........................................................................................... 23! Chap e 4 -!Da a Se ................................................................................................... 30! Chap e 4.1 -!Tshingua-Daimle [13] .................................................................... 30! Chap e 4.2 -!INRIA Pe son Da ase [14] .............................................................. 30! Chap e 4.3 -!C ea ed da a se .......................................................................... 30! Chap e 5 -!Ha dwa e ................................................................................................. 31! Chap e 5.1 -!In el(R) Co e(TM) i7-9700F CPU .................................................... 31! Chap e 5.2 -!GEFORCE RTX 2080 Ti .................................................................... 32! Chap e 5.3 -!Je son Nano 2GB De elope Ki [28] ........................................... 32! Chap e 6 -!Ta ge de ec ion e alua ion ................................................................. 33! Chap e 6.1 -!In e sec ion O e Union (IOU) ...................................................... 33! Chap e 6.2 -!T ue Posi i e, False Posi i e, False Nega i e and T ue Nega i e 33! Chap e 6.3 -!P ecision ......................................................................................... 34! Chap e 6.4 -!Recall .............................................................................................. 34! Chap e 6.5 -!AP calcula ion ............................................................................... 35! Chap e 7 -!F amewo ks .............................................................................................. 35! Chap e 7.1 -!Cuda ............................................................................................... 35! Chap e 7.2 -!Py o ch ........................................................................................... 36! Chap e 7.3 -!MMCV ............................................................................................. 37! Chap e 8 -!Resul s o he expe imen ....................................................................... 37! Chap e 8.1 -!Resul s o VGG16 ........................................................................... 38! Chap e 8.2 -!Resul s o SSD MobileNe .............................................................. 40! Chap e 8.3 -!Resul s o YOLO 4 .......................................................................... 41! Chap e 8.4 -!Resul o FASTER RCNN+FPN ......................................................... 42! Chap e 9 -!Compa ison o expe imen al esul s ..................................................... 43! Chap e 9.1 -!Compa ison o esul s o high-pe o mance machines ............ 43! Chap e 9.2 -!Compa ison o esul s o low-pe o mance machines ............. 45! Chap e 10 -!Conclusions and u u e wo k ............................................................... 46! Chap e 10.1 -!Conclusions .................................................................................. 46! Chap e 10.2 -!Fu u e Wo k .................................................................................. 47! BIBLIOGRAPHY ................................................................................................................ 49! Appendix ........................................................................................................................ 51! 1 Chap e 1 - In oduc ion Pedes ian De ec ion has always been a ho and di icul poin in compu e ision esea ch. The p oblem o be sol ed by pedes ian de ec ion is o ind all pedes ians in an image o ideo ame, including hei posi ion and size, which a e gene ally ep esen ed by ec angula boxes, simila o ace de ec ion, which is also a ypical a ge de ec ion p oblem. Pedes ian de ec ion echnology has a s ong use alue. I can be combined wi h pedes ian acking, pedes ian e-iden i ica ion and o he echnologies. I can be used in au omo i e unmanned d i ing sys ems (ADAS), in elligen obo s, in elligen ideo su eillance, human beha io analysis, passenge low s a is ics sys ems, and in elligence T anspo a ion and o he ields. Since he human body is qui e lexible, he e will be a ious pos u es and shapes, and i s appea ance is g ea ly a ec ed by wea ing, pos u e, iewing angle, e c., and i also aces he in luence o ac o s such as occlusion and illumina ion. This makes pedes ian de ec ion a compu e ision A e y challenging subjec . The main p oblems o be sol ed in pedes ian de ec ion a e: The appea ance is e y di e en . Including iewing angle, pos u e, clo hing and a achmen s, ligh ing, imaging dis ance, e c. Looking a he pas om di e en angles, he appea ance o pedes ians is e y di e en . Pedes ians in di e en pos u es ha e e y di e en appea ances. Due o he di e en clo hes ha people wea , as well as he in luence o umb ellas, ha s, sca es, luggage and o he a achmen s, he appea ance is e y di e en . The di e ence in ligh ing also caused some di icul ies. The human body a a dis ance and he human body a a close dis ance a e also e y di e en in appea ance. 2 Occlusion p oblem. In many applica ion scena ios, pedes ians a e e y dense and he e a e se ious occlusions. We can only see a pa o he human body, which b ings se ious challenges o he de ec ion algo i hm. The backg ound is complica ed. Whe he indoo o ou doo , pedes ian de ec ion gene ally aces e y complica ed backg ounds. The appea ance and shape, colo , and ex u e o some objec s a e e y simila o human bodies, which makes he algo i hm unable o dis inguish accu a ely. De ec ion speed. Pedes ian de ec ion gene ally uses a complex model wi h a la ge amoun o calcula ions. I is e y di icul o achie e eal- ime and gene ally equi es a lo o op imiza ion. Chap e 1.1 - Mo i a ion One day in he u u e, new ene gy ehicles will e en ually eplace uel ehicles, and au onomous d i ing will also eplace he d i e one day in he u u e. The cu en le el o au onomous d i ing is o mula ed by he SAE In e na ional[27], which is di ided in o 6 le els (L0-L5). -Le el L0: The d i e is in ull con ol o he ehicle; -Le el L1: The au oma ic sys em can some imes assis he d i e o comple e ce ain d i ing asks; -Le el L2 assis ed d i ing: The au oma ic sys em can comple e ce ain d i ing asks, bu he d i e needs o moni o he d i ing en i onmen and comple e he es , while ensu ing ha p oblems occu and ake o e a any ime. A his le el, he w ong pe cep ion and judgmen o he au oma ic sys em can be co ec ed by he d i e a any ime, and mos ca companies can p o ide his sys em. L2 can be di ided in o di e en usage 3 scena ios based on speed and en i onmen , such as low-speed a ic jams on he loop, as d i ing on highways, and au oma ic pa king by he d i e in he ca ; -Le el L3 semi-au onomous d i ing: The au oma ic sys em can no only comple e ce ain d i ing asks, bu also moni o he d i ing en i onmen unde ce ain condi ions, bu he d i e mus be eady o egain d i ing con ol (when he au oma ic sys em eques s i ). The e o e, a his le el, he d i e s ill canno sleep o ake a deep es . A e he comple ion o L2, he esea ch ield o ca companies is ex ended om he e. Due o he pa icula i y o L3, he mos meaning ul deploymen cu en ly seen is o upg ade on he high-speed L2; he di e ence be ween L3 and L2 is ha he ehicle is esponsible o pe iphe al moni o ing, and he human d i e only needs o main ain a en ion o eme gencies. -Le el L4 highly au oma ed d i ing: Au oma ed sys ems can comple e d i ing asks and moni o he d i ing en i onmen unde ce ain en i onmen s and speci ic condi ions; cu en ly, he deploymen o L4 is mos ly based on ci y use, which can be ully au oma ed ale pa king. I can also be di ec ly combined wi h axi se ices. A his s age, wi hin he scope o au onomous d i ing, all asks ela ed o d i ing ha e no hing o do wi h he d i e and passenge s. The pe cep ion o ex e nal esponsibili y lies in he au onomous d i ing sys em, and he e a e di e en design and deploymen ideas he e; -Le el L5 ully au oma ed d i ing: all d i ing asks ha he au oma ed sys em can comple e unde all condi ions. 10 Figu e[3.1.3] Fully connec ed laye ou pu esul Chap e 3.2 - Ta ge de ec ion ne wo k -Commonly used de ec ion can be oughly di ided in o wo ca ego ies: wo s age and one s age. Two s age means ha he de ec ion ne wo k is di ided in o wo s eps. The i s s ep is o ame he objec , and he second s ep is o de e mine he classi ica ion o he objec . One s age is o di ec ly use he eg ession ne wo k o ge i s classi ica ion and de ec ion ame based on he ex ac ed ea u es. Among hem, Fas e R-CNN belongs o wo s age, SSD and YOLO belong o one s age. -The ad an age o one s age is as , and he ad an age o wo s age is accu acy. A e ge ing he de ec ion ame, i is much easie o classi y his ask han di ec ly e u ning o he ne wo k o ge he classi ica ion. Chap e 3.2.1 - Two s age 1. Fas e -RCNN[4] The Fas e -RCNN model in oduces RPN (Region P oposal Ne wo k) o di ec ly gene a e candida e egions. Fas e -RCNN can be seen as a combina ion o RPN and Fas RCNN models, ha is, Fas e -RCNN = RPN + Fas -RCNN. 11 Fo he RPN ne wo k, a CNN model (gene ally called a ea u e ex ac o ) is i s used o ecei e he en i e pic u e and ex ac he ea u e map. Then an N×N (3×3 in he ex ) sliding window is used on his ea u e map, and a low-dimensional ea u e (such as 256-d) is mapped o each sliding window posi ion. Then his ea u e is sen o wo ully connec ed laye s, one o classi ica ion p edic ion and he o he o eg ession. Gene ally, k a p io i boxes (ancho s, de aul bounding boxes) o di e en sizes o p opo ions a e se o each window posi ion, which means ha k candida e egions ( egion p oposals) a e p edic ed o each posi ion. Fo he classi ica ion laye , i s ou pu size is 2k, which means ha each candida e a ea con ains he objec o backg ound p obabili y alue, while he eg ession laye ou pu s 4k coo dina e alues, which indica e he posi ion o each candida e a ea ( ela i e o each a p io i box). Fo each sliding window posi ion, hese wo ully connec ed laye s a e sha ed. The e o e, RPN can be implemen ed using con olu ional laye s: i s , an n×n con olu ion o ob ain low-dimensional ea u es, and hen wo 1×1 con olu ions, which a e used o classi ica ion and eg ession espec i ely. Figu e[3.2.1.1] CNN model (gene ally called a ea u e ex ac o ) RPN uses wo classi ica ions, only dis inguishing backg ound and objec s, bu does no p edic he ca ego y o he objec , ha is, class-agnos ic. Since he coo dina e alues need o be p edic ed a he same ime, du ing aining, he a p io i box mus be ma ched wi h he g ound- u h box. The p inciple is: (1) he a p io i box wi h he highes 12 IoU o a ce ain g ound- u h box; (2) and A g ound- u h box wi h an IoU alue g ea e han 0.7 a p io i box, as long as one is sa is ied, he a p io i box can ma ch a g ound- u h, so ha he a p io i box is a posi i e sample (belonging o he objec ), and he g ound- u h Fo he e u n goal. Fo hose a p io i boxes whose IoU alue wi h any g ound- u h box is lowe han 0.3, hey a e conside ed as nega i e samples. The RPN ne wo k can be ained sepa a ely, and he sepa a ely ained RPN model gi es many egion p oposals. Due o he la ge numbe o a p io i boxes, many o he candida e egions p edic ed by RPN o e lap. We mus i s pe o m NMS (non-maximum supp ession, IoU h eshold is se o 0.7) o educe he numbe o candida e egions, and hen a ange hem in descending o de o con idence, selec op -N egion p oposals a e used o ain he Fas R-CNN model. The ole o RPN is o eplace he ole o Selec i e sea ch, bu i is as e , so Fas e R-CNN can accele a e bo h aining and p edic ion. Figu e[3.2.1.2] RPN s uc u e 13 Fas e -RCNN ollows he ollowing aining p ocess: S ep 1: Use he p e- ained model on ImageNe o ini ialize he ea u e ex ac ion ne wo k and ain he RPN ne wo k; S ep 2: Use he model p e- ained on ImageNe o ini ialize he Fas -RCNN ea u e ex ac ion ne wo k, and use he candida e box gene a ed by he ained RPN ne wo k in s ep one as inpu o ain a Fas -RCNN ne wo k. So a , each o he wo ne wo ks The pa ame e s o he i s laye a e no sha ed a all; S ep 3: Use he Fas -RCNN ne wo k pa ame e s o s ep wo o ini ialize a new RPN ne wo k, bu se he lea ning a e o he ea u e ex ac ion ne wo k pa ame e s sha ed by RPN and Fas -RCNN o 0, e en i he unique pa ame e s o he RPN ne wo k a e lea ned, i is ixed Fea u e ex ac ion ne wo k. A his poin , he wo ne wo ks ha e sha ed all common con olu ional laye s; S ep 4: Fo hose ne wo k laye s ha a e s ill sha ed, add Fas -RCNN-speci ic ne wo k laye s, con inue aining, and ine- une Fas -RCNN-speci ic ne wo k laye s. Up o his poin , RPN and Fas -RCNN ne wo ks comple ely sha e pa ame e s. Use Fas -RCNN o comple e candida e ame ex ac ion and a ge de ec ion unc ions a he same ime. 2. Fpn[5] In he pas as e cnn o a ge de ec ion, no ma e i is pn o as cnn, oi ac s on he las laye . This is no p oblem o he de ec ion o la ge a ge s, bu he e a e some p oblems o de ec ion o small a ge s. Because o small a ge s, when con olu ional pooling is pe o med o he las laye , he seman ic in o ma ion is ac ually gone, because we all know ha he me hod o mapping a oi o a ea u e map is o di ec ly di ide he 14 unde lying coo dina es by s ide, Ob iously, he la e , he smalle he map will be, and i may e en disappea . The e o e, in o de o sol e he p oblem o mul i-scale de ec ion, a ea u e py amid ne wo k is in oduced. Figu e[3.2.1.3] Mul i-scale py amid Figu e[3.2.1.3] (a) is a ai ly common mul i-scale me hod called a ea u e image py amid. This me hod was widely used in he ea lie a i icial design ea u es (DPM), and i was also used in CNN. I is o mul i-scale he inpu image by se ing di e en zoom a ios. This can sol e mul iple scales, bu i is equi alen o aining mul iple models (assuming a ixed inpu size is equi ed). E en i he inpu size is no allowed, i also inc eases he memo y space o s o ing images o di e en scales. Figu e[3.2.1.3] (b) is CNN. Compa ed wi h a i icially designed ea u es, cnn can lea n mo e ad anced seman ic ea u es by i sel . A he same ime, CNN is obus o scale changes. The e o e, as shown in he igu e, he ea u es calcula ed om he inpu o a single scale can also be used o iden i y , Bu when encoun e ing ob ious mul i-scale a ge de ec ion, he py amid s uc u e is s ill needed o u he imp o e he accu acy. Judging om some o he leading me hods on he imageNe and COCO da a se s, he ea u e image py amid me hod is used in he es , ha is, he combina ion o Figu e[3.2.1.3] (a) and Figu e[3.2.1.3] (b). Explains ha he ad an age o each le el o he cha ac e ized image py amid is ha i p oduces a mul i-scale ea u e ep esen a ion, 15 and he ea u es o each le el ha e s ong seman ics (because all ea u es gene a ed by cnn), including he high- esolu ion le el (The la ges scale inpu image). Howe e , his mode has ob ious d awbacks. Compa ed wi h he o iginal me hod, he ime has inc eased by 4 imes, and i is di icul o use in eal- ime applica ions. Simila ly, i also inc eases he s o age cos , which is why he image py amid is only used in he es ing phase. Bu i i is only used in he es ing phase, hen aining and es ing will be inconsis en in in e ence. The e o e, some ecen me hods ha e simply abandoned he image py amid. Figu e[3.2.1.3] (c), SSD ied o use CNN py amid-shaped hie a chical ea u es ea lie . Ideally, he SSD-s yle py amid euses he mul i-scale ea u e maps om mul iple laye s calcula ed by he o wa d p ocess, so his o m does no consume addi ional esou ces. Howe e , in o de o a oid he use o low-le el ea u es, SSD abandoned he shallow ea u e map, bu s a ed building py amids om con 4_3, and added some new laye s. The e o e, SSD has gi en up on eusing highe - esolu ion ea u e maps, bu hese ea u e maps a e e y impo an o de ec ing small a ge s. This is he di e ence be ween SSD and FPN. Figu e[3.2.1.3] (4) is he s uc u e o FPN. FPN is a py amid o m o na u al use o CNN hie a chical ea u es, while gene a ing ea u e py amids wi h s ong seman ic in o ma ion a all scales. The e o e, he s uc u e o FPN is designed wi h a op-down s uc u e and a ho izon al connec ion o in eg a e a shallow laye wi h high esolu ion and a deep laye wi h ich seman ic in o ma ion. In his way, i is possible o quickly build a ea u e py amid wi h s ong seman ic in o ma ion on all scales om a single inpu image a a single scale, wi hou incu ing signi ican cos s. FPN is no a comple e a ge de ec ion ne wo k, bu a ea u e py amid ne wo k. The FPN men ioned is ac ually a ea u e py amid ex ac ion ea u e. The e o e, he idea o Fas e RCNN plus FPN essen ially changes he ea u e ex ac ion pa because he e a e mo e ea u e laye s. So ROIpooling has also inc eased. 16 Figu e[3.2.1.4] as e _ cnn+ pn s uc u e ① The backbone ne wo k gene a es ou -scale ea u e maps, and hen sequen ially passes h ough hei espec i e La e al_con o make hei channel numbe s consis en . ②The deep ea u e maps a e down-sampled o adjacen laye scales s ep by s ep, and hen he wo a e added and used. ③Ou pu he ea u e map a e usion The low-le el ea u e seman ic in o ma ion is ela i ely small, bu he a ge loca ion is accu a e; he high-le el ea u e seman ic in o ma ion is iche , bu he a ge loca ion is ela i ely ough. Fpn is independen ly p edic ed in di e en ea u e maps. 17 Figu e[3.2.1.5] Independen p edic ion in di e en ea u e maps Chap e 3.2.2 - One s age 1. SSD[6] SSD uses he idea o meshing, and unlike Fas e RCNN, i in eg a es all ope a ions in o a con olu ional ne wo k. In o de o de ec a ge s o di e en scales, SSD pe o ms sliding window scanning on he ea u e images o di e en con olu ional laye s; small a ge s a e de ec ed in he ea u e images ou pu by he p e ious con olu ional laye , and la ge a ge s a e de ec ed in he ea u e images ou pu by he subsequen con olu ional laye . The goal. I s main ea u es a e: 1.1. De ec ion based on mul i-scale ea u e images: P edic ion on mul i-scale con olu ion ea u e maps o de ec a ge s o di e en sizes, which imp o es he de ec ion accu acy o small a ge objec s o a ce ain ex en . 1.2. D awing on he idea o Ancho boxes in Fas e R-CNN, sampling candida e egions on ea u e maps o di e en scales, which imp o es he ecall a e o de ec ion and he de ec ion e ec o small a ge s o a ce ain ex en . The ollowing igu e shows he p inciple o SSD: 18 Figu e[3.2.2.1] Schema ic o SSD 2. VGG16[8] VGG was p oposed in 2014 by he Visual Geome y G oup, Depa men o Science and Enginee ing, Ox o d Uni e si y. The main wo k is o p o e ha inc easing he dep h o he ne wo k can a ec he inal pe o mance o he ne wo k o a ce ain ex en . VGG has wo s uc u es, namely VGG16 and VGG19. Excep o he di e ence in ne wo k dep h, he e is no di e ence in essence be ween he wo. Compa ed wi h AlexNe in 2012, a high ad ance o VGG is o use con inuous 3x3 small con olu ion ke nels o eplace he la ge ones in AlexNe (AlexNe uses 11x11, 7x7 and 5x5 con olu ion ke nels). The supe posi ion o wo 3x3 con olu ion ke nels wi h a s ep leng h o 1, i s ecep i e ield is equi alen o a 5x5 con olu ion ke nel. Howe e , he use o s acked small con olu ion ke nels is due o he la ge con olu ion ke nel, because he inc ease in he numbe o laye s 19 inc eases he nonlinea i y o he ne wo k, which allows he ne wo k o lea n mo e complex models, and he small con olu ion ke nel has ewe pa ame e s. Figu e[3.2.2.2] VGG16 model VGG16 con ains: 13 con olu ional laye s (Con olu ional Laye ), espec i ely ep esen ed by con 3- XXX 3 ully connec ed laye s (Fully connec ed Laye ), espec i ely ep esen ed by FC- XXXX 5 pooling laye s (Pool laye ), espec i ely ep esen ed by maxpool The ou s anding ea u e o VGG16 is simplici y, which is e lec ed in: 3. The con olu ional laye s all use he same con olu ion ke nel pa ame e s The con olu ional laye s a e all exp essed as con 3-XXX, whe e con 3 indica es ha he ke nel size o he con olu ional laye used by he con olu ional laye is 3, ha is, he wid h and heigh a e 3, and 3*3 is e y Small con olu ion ke nel size, combined wi h o he pa ame e s (s ide=1, padding=same), so ha each con olu ional laye ( enso ) can main ain he same wid h and wid h as he 26 a)Combine he o iginal Da kne 53 wi h CSPNe . YOLO 3 is composed o a se ies o esidual s uc u es. A e he combina ion, he main job o CSPne is o spli he s ack o he o iginal esidual block, and spli i in o wo pa s: he main pa con inues o s ack he o iginal esidual block, and he b anch pa is equi alen o a esidual edge. A e a small amoun o p ocessing, i is di ec ly connec ed o he end. b)Use he MIsh ac i a ion unc ion o eplace he o iginal Leaky ReLU. In YOLO 3, each con olu ional laye includes a ba ch no maliza ion laye and a Leaky ReLU. In he backbone ne wo k CSPDa kne 53 o YOLO 4, Mish is used ins ead o he o iginal Leaky ReLU. (2)SPP[11] The o iginal design pu pose o SPP is o make he con olu ional neu al ne wo k no es ic ed by he ixed inpu size. In YOLO 4, he au ho in oduced SPP because i signi ican ly inc eases he ecep i e ield, isola es he mos impo an con ex ea u es, and ha dly educes he unning speed o YOLO 4. As shown in he igu e below, i is he classic spa ial py amid pooling laye in SPP. Figu e[3.2.4.3] spp space py amid pooling laye In YOLO 4, he speci ic me hod is o use he maximum pooling o ou di e en scales o p ocess he ea u e map ou pu by he uppe laye . The maximum pooling co e size is 13x13, 9x9, 5x5, 1x1, and 1x1 is equi alen o no p ocessing. 27 (3)PANe [12] PANe as a whole can be seen as a numbe o imp o emen s on Mask R-CNN, making ull use o ea u e usion, such as in oducing Bo om-up pa h augmen a ion s uc u e, making ull use o ne wo k shallow ea u es o segmen a ion; in oducing Adap i e ea u e pooling o make he ex ac ed ROI Fea u es a e mo e abundan ; Fully- conneFc ed usion is in oduced, and a mo e accu a e segmen a ion esul is ob ained by using he ou pu o a on -backg ound bina y classi ica ion b anch. Figu e[3.2.4.4] PANe schema ic (a)FPN backbone,(b)Bo om-up pa h augmen a ion,(c) Adap i e ea u e pooling,(d)Box b anch,(e)Fully-connec ed usion In YOLO 4, he au ho uses PANe ins ead o FPN in YOLO 3 as he me hod o pa ame e agg ega ion, and pe o ms pa ame e agg ega ion om di e en backbone laye s o di e en de ec o le els. And modi ied he o iginal PANe me hod, using enso connec ion (conca ) ins ead o he o iginal sho cu connec ion (sho cu connec ion). (4) YOLO 3 Head In YOLO 4, he Head ha inhe i s YOLO 3 pe o ms mul i-scale p edic ion, which imp o es he de ec ion pe o mance o a ge s o di e en sizes.YOLO 4 lea ns he 28 YOLO 3 me hod, uses h ee di e en le els o ea u e maps o usion, and inhe i s he Head o YOLO 3. (5)T icks To ob ain be e accu acy wi hou inc easing he in e ence cos , bu only change he aining s a egy o only inc ease he aining cos me hod, he au ho calls i "Bag o eebies"; i only inc eases a small amoun o in e ence cos bu can signi ican ly imp o e he accu acy o a ge de ec ion Plug-in modules and pos -p ocessing me hods, called "Bag o specials" - Bag o eebies Random zoom Flip, o a e Image dis u bance, noise, occlusion Change b igh ness, con as , sa u a ion, andom e ase(Cu ou MixUp Cu Mix) Common egula iza ion me hods a e: D opOu D opConnec D opBlock The me hods o balance posi i e and nega i e samples a e: Focal loss OHEM (online ha d- o-sepa a e sample mining) In addi ion, he e a e imp o emen s in e u n loss: GIOU DIOU CIOU - Bag o specials Inc ease he ecep i e ield skills: SPP ASPP RFB A en ion mechanism: Squeeze-and-Exci a ion (SE) Spa ial A en ion Module (SAM) Fea u e usion in eg a ion: FPN SFAM ASFF BiFPN ( om he amous E icien De ) Be e ac i a ion unc ion: ReLU LReLU PReLU ReLU6 SELU Swish ha d-Swish Pos -p ocessing non-maximum supp ession algo i hm: so -NMS DIoU NMS (6)Ways o imp o e In addi ion o he a ious T icks men ioned abo e, in o de o make he a ge de ec o easie o ain on a single GPU, he au ho also p oposes 5 imp o ed me hods: 29 - Mosaic This is a new da a enhancemen me hod p oposed by he au ho , which d aws on he idea o Cu Mix da a enhancemen me hod. Cu Mix da a enhancemen me hod uses wo pic u es o s i ching, bu Mosaic uses ou pic u es o s i ching. - SAT SAT is a sel -ad e sa ial aining da a enhancemen me hod, which is a new ad e sa ial aining me hod. In he i s s age, he neu al ne wo k changes he o iginal image wi hou changing he ne wo k weigh s. In his way, he neu al ne wo k conduc s an ad e sa ial a ack on i sel , changing he o iginal image o c ea e a decep ion ha he e is no desi ed objec on he image. In he second s age, he no mal me hod is used o ain he neu al ne wo k o de ec he a ge . - CmBN The ull name o CmBN is C oss mini-Ba ch No maliza ion, which is de ined as C oss Mini- Ba ch No maliza ion (CmBN). CmBN is an imp o ed e sion o CBN, which is used o collec s a is ical da a in mul iple mini-ba ches in a ba ch. - SAM The au ho modi ied he o iginal SAM (Spa ial A en ion Module) me hod, changing SAM om spa ial a en ion o poin a en ion. As shown in he igu e below, o he con en ional SAM, he maximum pooling laye and he a e age pooling laye ac on he inpu ea u e maps espec i ely o ob ain wo se s o ea u e maps wi h he same shape, and hen inpu he esul s in o a con olu ional laye , ollowed by Sigmoid unc ion o c ea e spa ial a en ion. - PAN The au ho modi ied he o iginal PAN (Pa h Agg ega ion Ne wo k) me hod, using enso connec ion (conca ) ins ead o he o iginal sho cu connec ion (sho cu connec ion) 30 Chap e 4 - Da a Se Chap e 4.1 - Tshingua-Daimle [13] Tsinghua-Daimle Cyclis De ec ion con ains 9741 images wi h anno a ions only o "cyclis ". Only cyclis s which a e ully isible (occlusion<10%) and highe han 60 pixels ha e been labeled he e. Chap e 4.2 - INRIA Pe son Da ase [14] This da ase has c opped he image o pedes ians so ha he pe son can be mo e p ominen , and he pos u e is no pa icula ly s ange. Only a ew pic u es a e ob ained om Google. Chap e 4.3 - C ea ed da a se I did no selec pedes ians om Tsinghua's da a se , because he size o pedes ian images in Tsinghua's da a se is 64x64, which is oo small o expe imen s, so I did no choose. Fo he KITTI da a se , when I go hei da a se , I ound ha i was all ca s, so I judged ha i did no mee my opic. Finally. Cyclis da a se adop s Tsinghua-Daimle Cyclis De ec ion pa o he da a se ,1507 image esolu ion 2048*1024. The Pe son da a se uses he 614 aining se and 288 es se o he INRIAPe son pedes ian da a se . A o al o 2088 shee s o all da a se s a e di ided in o aining se (1671) and es se (417) a a a io o 8:2. 31 All he pic u es used need o use he .jpg o ma . The c ea ed da a se uses codes o dis inguish he aining se and he alida ion se , and hen uses he code o con e he VOC o ma . They a e 1.Anno a ions (.xml, each pic u e co esponds o a ile, which s o es he name o he pic u e, he long sec ion, he uppe le and lowe igh coo dina es o he a ge ame, he ca ego y name o he a ge ame) 2.ImagesSe (. x , sa e The name o he image ha needs o be ained) 3.JPEGImage ( he o iginal image) Chap e 5 - Ha dwa e The ha dwa e I used o comple e his p ojec . Knowing his is necessa y, because di e en ha dwa e will a ec he aining p ocess, especially he aining ime, o e iciency. Chap e 5.1 - In el(R) Co e(TM) i7-9700F CPU l Co es: 8 l Th eads: 8 l P ocesso Base F equency: 3.00GHz l Max Tu bo F equency: 4.7GHz l TDP: 65W 32 Chap e 5.2 - GEFORCE RTX 2080 Ti l NVIDIA CUDA ® Co es: 4352 l Boos Clock (MHz): 1545 l Base Clock (MHz): 1350 l S anda d Memo y Con ig: 11 GB GDDR6 l G aphics Ca d Powe (W): 250W Chap e 5.3 - Je son Nano 2GB De elope Ki [28] l CPU Quad-co e ARM® A57 @ 1.43 GHz l GPU 128-co e NVIDIA Maxwell™ l MEMORY 2 GB 64-bi LPDDR4 25.6 GB/s Since i s elease in 2019, NVIDIA Je son Nano has c ea ed a enzy in he ield o AIOT edge compu ing applica ions wo ldwide, He was also named "Bes AI P ocesso " in he 2020 Bes Visual P oduc s awa d lis by Edge AI and Vision Alliance. This is mainly because he NVIDIA Je son Nano chip ully mee s he six challenges o AIOT chip: high pe o mance, small size, low powe consump ion, ull in e ace, ecological in eg i y, and excellen cos . The eason o using his edge de ice was o es a ious models on a low-powe machine. Fo compa ison, es he FPS on a low-pe o mance machine. 33 Chap e 6 - Ta ge de ec ion e alua ion The classi ica ion ask will be judged by he p ecision/ ecall cu e. The p incipal quan i a i e measu e used will be he a e age p ecision (AP).[15] The e a e some impo an hings o know abou calcula ing AP.[16] Chap e 6.1 - In e sec ion O e Union (IOU) Figu e[6.1.1] IOU o mula To pu i simply, we equi e ha he IOU o he wo boxes is di ided by he o e lapping pa o he wo boxes by he union o he wo boxes. I is clea om he second o mula abo e. Pay a en ion o he wo boxes when calcula ing. The a ea o he union is equal o he sum o he a eas o he wo minus he a ea o he in e sec ion o he wo. Chap e 6.2 - T ue Posi i e, False Posi i e, False Nega i e and T ue Nega i e T ue Posi i e (TP): A co ec posi ioning esul is ha he IOU be ween you p edic ed box and ou g ound u h can be g ea e han ou speci ied h eshold, we gene ally ake his h eshold o be 0.5 34 False Posi i e (FP): I is a w ong esul , ha is, he IOU o he box and g ound u h you p edic ed is less han he h eshold False Nega i e (FN): We o iginally had an objec he e, so he e should be a box in ou place, bu you did no p edic i , hen his g ound u h is a FN o you model T ue Nega i e (TN): This is ha ou g ound u h has ound a p edic ion box whose IOU is g ea e han he h eshold, hen he g ound is conside ed o be success ully de ec ed Chap e 6.3 - P ecision P ecision is he abili y o a model o iden i y only he ele an objec s. I is he pe cen age o co ec posi i e p edic ions and is gi en by: Chap e 6.4 - Recall Recall is he abili y o a model o ind all he ele an cases (all g ound u h bounding boxes). I is he pe cen age o ue posi i e de ec ed among all ele an g ound u hs and is gi en by: 35 Chap e 6.5 - AP calcula ion (1) Be o e VOC2010, you only need o selec he maximum P ecision when Recall >= 0, 0.1, 0.2, ..., 1 o al 11 poin s, and hen AP is he a e age o hese 11 P ecision. Figu e[6.5.1] AP calcula ion Calcula ing he o al a ea, we ha e he AP. (2) In VOC2010 and la e , o each di e en Recall alue (including 0 and 1), selec he maximum P ecision when i is g ea e han o equal o hese Recall alues, and hen calcula e he a ea unde he PR cu e as he AP alue. Finally he a e age AP alue o all classes is mAP. Chap e 7 - F amewo ks Chap e 7.1 - Cuda CUDA (an ac onym o Compu e Uni ied De ice A chi ec u e) is a pa allel compu ing pla o m and applica ion p og amming in e ace (API) model c ea ed by N idia.[17]I allows so wa e de elope s and so wa e enginee s o use a CUDA-enabled g aphics 42 Figu e[8.3.2] loss line cha o YOLO 4 The same is a single-laye a ge de ec ion, yolo 4's pe o mance is signi ican ly be e han be o e. Chap e 8.4 - Resul o FASTER RCNN+FPN 43 Figu e[8.4.1] mAP line cha o FASTER RCNN+FPN Figu e[8.4.2] loss line cha o FASTER RCNN+FPN I am e y sa is ied wi h he pe o mance o as e cnn+ pn, because he same da a se has a e y big di e ence h ough he algo i hm. Below I will compa e all he abo e esul s oge he . Chap e 9 - Compa ison o expe imen al esul s Chap e 9.1 - Compa ison o esul s o high-pe o mance machines Model Inpu Size FPS mAP Cyclis Pe son VGG16 512*512 47.3 0.81 0.85 0.77 MobileNe 3 320*320 43.1 0.67 0.54 0.80 YOLO 4 608*608 19.6 0.78 0.75 0.81 FASTER RCNN+FPN 1333*800 12 0.9 0.899 0.902 Table[9.1] Compa ison o algo i hm esul s-high 44 The able shows us he a ious da a o each algo i hm in he high-pe o mance compu e . F om he expe imen al esul s, we can see ha FASTER RCNN+FPN pe o ms bes in e ms o de ec ion accu acy, bu he sho comings a e also e y ob ious. The ps only eached 12. This is due o i s wo-s age ne wo k. I has a egion p oposal ne wo k RPN, his pa uses CNN, and he egion p oposal ne wo k and classi ica ion ne wo k sha e con olu ional ea u es. And he e is FPN, independen p edic ion o di e en ea u e laye s. These can e ec i ely a ge de ec ion, bu he speed is slowe . We can easily ind his om he able. The abo e SSD-MobileNe inpu esolu ion is ela i ely small, and he ecogni ion e ec o he pe son class wi h he o iginal image size o abou 800 is be e . The inpu esolu ion o SSD-VGG16 is ela i ely la ge, and i can be e ecognize he cyclis class wi h he o iginal image size o 2048*1024. The eason why mobileNe 's mAP is oo low is because o o e i ing in he pe son pa . I is because he da a se o he pe son pa is oo small. The way o sol e his p oblem and ge he pe o mance i should be is o inc ease he amoun o da a, as I explained in he esul s sec ion. YOLO 4 is al eady e y good as a non-ligh weigh pe o mance. I shows a e y a e age alue, and he e a e some aspec s ha mus be explained. Compa ed wi h i s p e ious e sion, i has added many imp o emen s, such as adding ea u e py amids (SPP, PAN) o he las con olu ional laye , and modi ying he ac i a ion unc ion o Mish, lea ning a e annealing algo i hm. F om he s uc u al poin o iew, YOLO 4 also has an FPN s uc u e. Compa ed wi h FASTER RCNN+FPN, YOLO 4 has a highe ps pe o mance. Al hough mAP did no each he wo s age, I go a be e esul in he one s age. 45 Chap e 9.2 - Compa ison o esul s o low-pe o mance machines Model FPS VGG16 0.98 MobileNe 3 7.61 YOLO 4 -- FASTER RCNN+FPN -- Table[9.2] Compa ison o algo i hm esul s-low On low-pe o mance compu e s, I ocus on he pe o mance o ps. Because he machine I used is he je son nano 2gb e sion o NVIDIA. I means ha in he case o suppo ing a se ies o en i onmen s, he heo e ical ideo memo y o he machine is only 2gb. In ac ual use, he ideo memo y is only abou 1.3gb. This has ac ually eached a limi alue. F om he able, we can see ha in he case o e y low ideo memo y, only ligh weigh algo i hms can ac ually un no mally and pe o m a ge ecogni ion. The no mal e sion o YOLO 4 canno un on a machine wi h only 2gb o ideo memo y. The wo-s age FASTER RCNN+FPN is e en mo e impossible. The e o e, o a low-pe o mance machine, i is e y impo an o choose a co ec algo i hm. F om he able, as a simple con olu ional neu al model, VGG16 can ope a e co ec ly and ecognize a ge s no mally. Bu I hink i s ps is no good enough. As a ligh weigh neu al ne wo k, MobileNe can un no mally and has a ps o abou 7 I hink i is unde s andable. The e may be doub s ha his ps is no e y high, why is i s ill a ligh weigh neu al ne wo k? This is because we a e using he La ge e sion, which has 3 46 bo leneck laye s and a poin -wise con olu ional laye mo e han he Low e sion. I s Dep hwise Sepa able Con olu ion is e y sui able o low-pe o mance e sions. Chap e 10 - Conclusions and u u e wo k Chap e 10.1 - Conclusions Fo hese ou algo i hms and he selec ed da a se , om he esul s, i he machine pe o mance is no conside ed, hen FASTER RCNN+FPN is undoub edly he bes auxilia y sys em. Because i has highe accu acy. I is undeniable ha wo s age is be e han one s age in e ms o accu acy. Bu he sho comings a e also e y ob ious. The aining ime o he wo s age is much longe han ha o he one s age. And in ac ual use, he ps o he wo s age applica ion on high-pe o mance machines is no high enough, and i canno un smoo hly on low-pe o mance machines. The speed o yolo 4 is much as e han as e - cnn on high-pe o mance machines. And because he ea u e ex ac ion laye o yolo 4 uses a ea u e py amid s uc u e, i can also ge e y good accu acy. SSD-VGG16 uses ea u e maps o di e en scales o de ec ion. This may also be he eason why in his da a se , o small a ge s such as people and bicycles, and he e may be obs uc ions on he oad, i has a highe accu acy a e han he YOLO algo i hm. Fo his ime, MobileNe 3, which pe o med bes in low-pe o mance machines. I hink he ligh weigh neu al ne wo k can be used as a solu ion in sol ing business p oblems, o in ehicle sa e y sys ems. Because he speed o he ca is as and he neu al ne wo k needs o be as , hen a ligh weigh neu al ne wo k is a e y good choice. Video memo y, memo y, and cpu will be ela i ely small. O cou se, his is jus a s uc u al poin o iew o hese ou algo i hms. 47 I hink he choice o he aining da a se is e y impo an . F om he inal esul , he choice o he pic u e size is ac ually e y impo an , because he e a e ac ually es ic ions on he pic u es on he In e ne o he pic u es aken by you sel . Today's came a equipmen is ge ing be e and be e , o example, he pixels o mobile phones a e ge ing be e and be e . The equi emen s o he aining se a e also e y high. Maybe a ce ain a ge de ec ion algo i hm is excellen , bu due o he inapp op ia e choice o he da a se , he inal esul is bad. So in he end, because o he di e en models o di e en models and he di e en came a op ions in he ac ual use, he inal esul s will appea o be di e en . I inally used all he algo i hms o make ac ual de ec ions o assis ed d i ing. Using he ideo o lea ning o d i e in a d i ing school, a no ice may no no ice pedes ians and bicycles on he oad. 4 algo i hms a e used o he sc eensho s o he ideo in he appendix. I am sa is ied wi h he aining esul s and can achie e a ge de ec ion o mos o he scenes. Chap e 10.2 - Fu u e Wo k Fo u u e wo k, om he a ge de ec ion amewo k. F om he Fas e R-CNN ela ed se ies ocusing on accu acy o he YOLO se ies ocusing on speed, he u u e a ge de ec ion esea ch di ec ion will ocus mo e on he balance o accu acy and speed. The e o e, many ne wo k s uc u es a e gene a ed on he SSD amewo k. I hink ha i a ge de ec ion is o be used as an in- ehicle sa e y sys em, a ious ligh weigh neu al ne wo ks a e he mos p e e ed in ecen yea s. The e a e epo s ha YOLO 4- iny pe o ms e y well, we can also y i in low-pe o mance machine. The new YOLO 5 has appea ed, we can y o use i in high-pe o mance machine. 48 I ha e seen some esea ch pape s ha show ha he o a ion in a iance o con olu ional neu al ne wo ks is ob ained h ough da a enhancemen and la ge sample lea ning, and i does no ha e his ea u e i sel . Maybe con inue o s udy his will change he basic amewo k o he a ge de ec ion algo i hm. The use o 3D lida will be mo e e ined han he image de ec ion. Be e senso s will make a ge de ec ion mo e e ined, and now mo e au onomous d i ing applica ions on he ma ke a e 3D lida . O cou se, some con en ional ideas a e o imp o e he loss unc ion, ac i a ion unc ion, and ea u e usion, which can also be conside ed. 49 BIBLIOGRAPHY [1] Good ellow, I., Bengio, Y., Cou ille, A.．Deep lea ning (Vol. 1)．Camb idge：MIT p ess，2016 [2] Gu, J., Wang, Z., Kuen, J., Ma, L., Shah oudy, A., Shuai, B., Liu, T., Wang, X., Wang, L., Wang, G. and Cai, J., 2015. Recen ad ances in con olu ional neu al ne wo ks. a Xi p ep in a Xi :1512.07108. [3] Zhang, W., 1988. Shi -in a ian pa e n ecogni ion neu al ne wo k and i s op ical a chi ec u e. In P oceedings o annual con e ence o he Japan Socie y o Applied Physics. [4] Shaoqing Ren, Kaiming He, Ross Gi shick, and Jian Sun.Fas e R-CNN: Towa ds Real-Time Objec De ec ion wi h Region P oposal Ne wo ks [5] Tsung-Yi Lin1,2, Pio Dolla ´ 1 , Ross Gi shick1 , Kaiming He1 , Bha a h Ha iha an1 , and Se ge Belongie.Fea u e Py amid Ne wo ks o Objec De ec ion [6] Wei Liu1 , D agomi Anguelo 2 , Dumi u E han3 , Ch is ian Szegedy3 , Sco Reed4 , Cheng-Yang Fu1 , Alexande C. Be g.SSD: Single Sho Mul iBox De ec o [7] Alexey Bochko skiy,Chien-Yao Wang,Hong-Yuan Ma k Liao.YOLO 4: Op imal Speed and Accu acy o Objec De ec ion [8] Simonyan,Zisse man.Ve y Deep Con olu ional Ne wo ks o La ge Scale Image Recogni ion [9] And ew Howa d, Ma k Sandle , G ace Chu, Liang-Chieh Chen, Bo Chen,Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasude an, Quoc V. Le, Ha wig Adam,Sea ching o MobileNe V3 [10] Chien-Yao Wang,Hong-Yuan Ma k Liao,I-Hau Yeh,Yueh-Hua Wu,Ping-Yang Chen,Jun-Wei Hsieh,CSPNET: A NEW BACKBONE THAT CAN ENHANCE LEARNING CAPABILITY OF CNN [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,Spa ial Py amid Pooling in Deep Con olu ional Ne wo ks o Visual Recogni ion [12] Shu Liu, Lu Qi, Hai ang Qin, Jianping Shi, Jiaya Jia,Pa h Agg ega ion Ne wo k o Ins ance Segmen a ion [13] Y. Yang H. Xiong M. B aun S. Pan K. Li X. Li, F. Floh and D. M. Ga ila. A new benchma k o ision-based cyclis de ec ion, June 2016. [14] Na nee DALAL,Finding People in Images and Videos [15] Ma k E e ingham, John Winn, The PASCAL Visual Objec Classes Challenge 2012 (VOC2012) De elopmen Ki [16] h ps://gi hub.com/ a aelpadilla/Objec -De ec ion-Me ics [17] h ps://de elope .n idia.com/cuda-zone [18] Abi-Chahla, Fedy (June 18, 2008). "N idia's CUDA: The End o he CPU?". Tom's Ha dwa e. Re ie ed May 17, 2015. [19] Yegulalp, Se da (19 Janua y 2017). "Facebook b ings GPU-powe ed machine lea ning o Py hon". In oWo ld. Re ie ed 11 Decembe 2017 [20] Lo ica, Ben (3 Augus 2017). "Why AI and machine lea ning esea che s a e beginning o emb ace PyTo ch". O'Reilly Media. Re ie ed 11 Decembe 2017 50 [21] Ke ka , Nikhil (2017). "In oduc ion o PyTo ch". Deep Lea ning wi h Py hon. Ap ess, Be keley, CA. pp. 195–208. doi:10.1007/978-1-4842-2766-4_12. ISBN 9781484227657. [22] Pa el, Mo (2017-12-07). "When wo ends use: PyTo ch and ecommende sys ems". O'Reilly Media. Re ie ed 2017-12-18. [23] Mannes, John. "Facebook and Mic oso collabo a e o simpli y con e sions om PyTo ch o Ca e2". TechC unch. Re ie ed 2017-12-18. FAIR is accus omed o wo king wi h PyTo ch – a deep lea ning amewo k op imized o achie ing s a e o he a esul s in esea ch, ega dless o esou ce cons ain s. Un o una ely in he eal wo ld, mos o us a e limi ed by he compu a ional capabili ies o ou sma phones and compu e s.” [24] A akelyan, Sophia (2017-11-29). "Tech gian s a e using open sou ce amewo ks o domina e he AI communi y". Ven u eBea . Re ie ed 2017-12-18. [25] "The C++ F on end". PyTo ch Mas e Documen a ion. Re ie ed 2019-07-29. [26] h ps://gi hub.com/open-mmlab/mmc /blob/mas e /README.md [27] SAE. Taxonomy and de ini ions o e ms ela ed o d i ing au oma ion sys ems o on- oad mo o ehicles. h ps://www.sae.o g/misc/pd s/au oma ed_d i ing.pd ,2018. [28] h ps://de elope .n idia.com/embedded/je son-nano-2gb-de elope -ki 51 Appendix Table[1] SSD-MobileNe mAP Epoch mAP Cyclis pe son 1 0.612 0.425 0.798 2 0.649 0.491 0.807 3 0.658 0.504 0.811 4 0.669 0.540 0.799 5 0.657 0.499 0.814 6 0.652 0.498 0.805 7 0.663 0.519 0.808 8 0.648 0.494 0.802 9 0.668 0.534 0.803 10 0.665 0.533 0.797 11 0.670 0.541 0.798 12 0.667 0.537 0.797