A SVM and k-NN Restricted Stacking to Improve Land Use and Land Cover Classification

A SVM and k-NN Res ic ed S acking o Imp o e Land Use and Land Co e Classi ica ion Jo ge Ga cia-Gu ie ez, Daniel Ma eos-Ga cia, and Jose C. Riquelme-San os Depa men o Compu e Science E.T.S.I.I. - Uni e si y o Se ille, Spain {jga cia,ma eos, iquelme}@lsi.us.es Abs ac . Land use and land co e (LULC) maps a e emo e sensing p oduc s ha a e used o classi y a eas in o diffe en landscapes. The newes echniques ha e been applied o imp o e he final LULC classifi- ca ion and mos o hem a e based on SVM classifie s. In his pape , a new me hod based on a mul iple classifie s ensemble o imp o e LULC map accu acy is shown. The me hod builds a s a is ical as e om LI- DAR and image usion da a ollowing a pixel-o ien ed s a egy. Then, he pixels om a aining a ea a e used o build a SVM and k-NN es ic ed s acking aking in o accoun he special cha ac e is ics o spa ial da a. A compa ison be ween a SVM and he es ic ed s acking is ca ied ou . The esul s o he es s show ha ou app oach imp o es he esul s in he con ex o he eal da a om a ipa ian a ea o Huel a (Spain). 1 In oduc ion Remo e sensing has become a e y impo an ool o ca y ou many diffe en asks o he Na u al En i onmen . In his way, emo e sensing has success ully been applied o impo an ac i i ies like flood con ol, o es al in en o ies o in asi e species con ol in p o ec ed o specially in e es ing a eas. Al hough emo e sensing usually wo ks wi h images exclusi ely, da a usion has been o high in e es since he appea ance o new ac i e senso s (i.e., da a is p oduced as a esponse o a s imulus which is no he sola ligh ). They complemen images and o e come some o hei limi a ions, e.g., he p oblems associa ed o shadows. These limi a ions cause usion o senso s can be ound as a p ope echnique specially in e es ing o imp o e he esul s o he classical emo e sensing app oaches. One o he mos ac i e esea ch lines has been based on LIDAR (LIgh De ec ion And Ranging) echnology. This echnology is able o egis e objec heigh s and i is specially ecommended o be applied on complex landscapes like ipa ian zones. Thus, Ve els e al.[1] use LIDAR o s udy ege al species communi ies and An ono akis e al.[2] de elop a new me hodology o iden i y diffe en ypes o comme cial wood in ipa ian zones using only LIDAR. An au oma ic pixel classifica ion which is gene ally supe ised is usually he fi s s ep o ex ac knowledge om emo e sensing da a. Se e al echniques om machine lea ning ha e been used wi h sa is ac o y esul s hough suppo ec o machines (SVM) a e he p edominan echnique o ob ain he bes esul s E.S. Co chado Rod iguez e al. (Eds.): HAIS 2010, Pa II, LNAI 6077, pp. 493–500, 2010. 494 J. Ga cia-Gu ie ez, D. Ma eos-Ga cia, and J.C. Riquelme-San os in mos cases [3]. Despi e he SVM’s high accu acy, imp o emen is needed o each he s anda ds o p oduc s like land uses and land co e (LULC) maps [4]. LULC maps a e emo e sensing p oduc s ha a e used o classi y a eas in o diffe en landscapes subjec o hei own cha ac e is ics o unc ionali y. The newes echniques ha e been applied o imp o e he final classifica ion o de elop LULC maps. Fau el e al. [5] apply an SVM o classi y he pixels depending on mo phologic and hype spec al da a. In Mi akis e al. [6], a neu onal ne wo k wi h weigh s de e mined by a gene ic algo i hm ob ains he final classifica ion using usion ope a o s and uzzy logic. I is impo an o unde line ha ensembles a e one o he mos powe ul ools in machine lea ning and so hey a e in emo e sensing whe e hey ha e also been applied p o usely. A e y clea example can be seen in [7] whe e an s acking o se e al SVM’s and a andom o es is used o ca y ou he pixel classifica ion. This wo k explo es he applica ion o ensembles on emo e sensing aking ad an age o con ex ual in o ma ion [8] om mul i-sou ce (LIDAR and ae ial images) da a. Thus, a no el supe ised me hod called R-STACK (based on a s acking o a SVM and mul iple NN classifie s) is shown wi h wo pu poses: –Show an easy way o imp o e he quali y o models when in elligen ech- niques a e applied on LIDAR and image y usion da a. –Imp o e he gene al accu acy o an au oma ically gene a ed LULC map. The es o he pape is o ganized as ollows. Sec ion 2 p o ides a desc ip ion o he da a used in his wo k. Sec ion 3 desc ibes he me hodology used, highligh - ing he ea u e se and he model ex ac ion p ocess. The esul s achie ed a e shown in sec ion 4 and, finally, sec ion 5 is de o ed o summa y he conclusions and o discuss u u e lines o wo k. 2Da aDesc ip ion A LIDAR sys em is an op ical senso echnology ha measu es p ope ies o sca e ed ligh (usually lase ) o find ange and/o o he in o ma ion o a dis an a ge . The whole p ocess s a s wi h he emission o pola ized ligh , ypically, in he ul a iole isible o nea in a ed. Then, LIDAR ca ches he eflec ed signal om he opog aphic su ace and measu es he ime employed o each e u n o es ablish he dis ance be ween he emi e and he objec ha p oduced he e u n. This p ocess is helped by a global posi ioning sys em (GPS) o gi e ise o a cloud poin da abase in which o e e y poin , i is possible o find: spa ial posi ion(i.e., x, y and z coo dina es), in ensi y o e u n, numbe o he e u n in a sequence (i a pulse caused mul iple impac s), e c. This ea u es and he RGB alues in an o hopho o a e used in his wo k o ob ain s a is ical measu es on which he me hod is based and hey will be explained in sec ion 3. The LIDAR da a was aken in coas al a eas o he p o ince o Huel a. The pulses we e geo- e e enced and co ec ly alida ed by he dis ibu o o he da a and ha ing 1,384,875 eco ds o an a ea o 1.5 km2. The epo ed p ecision indica es a maximum e o o 0.5 m in he x-y posi ions, and o 0.15 m in he A SVM and k-NN Res ic ed S acking o Imp o e LULC Classifica ion 495 z posi ion. Along wi h he LIDAR fligh , he ae ial pho og aphs we e aken o he a ea wi h a esolu ion o 0.5 m2. The s udy a ea is si ua ed in he sou h o Spain, in he mou h o he Tin o and Odiel i e s. This a ea is nea he ci y o Huel a and p esen s a mix o u ban and na u al a eas. The na u al a eas can be classified in fi e subclasses: wa e ed zones, ma shland and ege a ion (low, middle and high). The high ege a ion is o med by sca ce ees o he genus eucalyp us in hea ea.Themiddle ege a ion is o med by diffe en ypes o Medi e anean bushes ha p incipally su ound oads and u ban a eas. Pas u es a e classified as low ege a ion and include ba e ea h a eas. In addi ion, he u ban a eas a e also classified in fi e subclasses: oads and ailways, buildings, coal deposi s, dumps and mixed a eas. 3Me hod Ou LULC de elopmen me hod (see Figu e 3) ollows a pixel-o ien ed s a egy which obliges us o c ea e a ma ix o as e whe e each elemen is a pixel. Each pixel ep esen s an a ea in unc ion o he esolu ion. The alue o esolu ion mus be p o ided by he use as a me hod pa ame e o de e mine he a ea wi hin each pixel. The esolu ion depends on he LIDAR poin densi y and he o hopho o esolu ion. In ou case, he selec ed esolu ion is se a 3 m2. Lesse esolu ion could damage he smalles classes ( oads) classifica ion and bigge esolu ion canno be possible due o he LIDAR esolu ion (0.5 poin s/m2). Apa om he esolu ion, i is necessa y o supply a digi al ele a ion model (DEM) o ex ac he ac ual heigh s o he LIDAR e u ns. In ou case, his p ocess is ca ied ou by an adap a i e mo phologic fil e [9]. In addi ion, expe knowledge has been applied o manually classi y o e a 2% o o al da a (7399 ins ances). Expe knowledge leaned on he pho og aphs aken in he same fligh as LIDAR da a and p e ious in o ma ion om he Regional Minis y o Andalu- sia (LULC map om 2003) was collec ed by an ope a o o build he aining se . The nex s ep (s ep 2 in Figu e 3) is o calcula e a se o a iables om image RGB alues, LIDAR in ensi y, heigh s and dis ibu ion o he LIDAR e u ns o each pixel (a o al o 500,000 pixels). In his manne , six y-one diffe en measu es we e calcula ed o e e y pixel. Mos o a iables used ha e been ex ac ed om li e a u e [10][2]. In Table 1, a summa y o hese ea u es can be seen. Specially in e es ing is he case o he no malized diffe ence ege a ion index (NDVI). The classical NDVI is gene a ed om nea in a ed band (NIR) and he ed band (R) as can be seen in Equa ion 1. In ou case, i canno be calcula ed since he NIR band is no a ailable in LIDAR o o hopho og aphy. Thus, he new a ibu e SNDVI has been used o simula e he NDVI using he in ensi y (I) om LIDAR (Equa ion 2) as nea -in a ed alue which app oxima es he eal NIR alue. NDV I =NIR −R NIR +R(1) SNDV I =I−R I+R(2) 496 J. Ga cia-Gu ie ez, D. Ma eos-Ga cia, and J.C. Riquelme-San os A new me hod called R-STACK based on a modified s acking o wo well-known classifie s (SVM and k-NN) has been de eloped o he model gene a ion. The Weka [11] implemen a ion o SOM and a ad-hoc k-NN implemen a ion we e used o each classifie espec i ely. Mo eo e , he gene al scheme o s acking has been modified o adap i o geog aphic da a. In his way, he fi s le el (s eps 5 and 6) consis s in a SVM which akes e e y ea u e om he pixels in he aining a ea o build an ini ial model which classifies e e y pixel om he s udy zone. A ha poin , a classical SVM applica ion on images is esul ed. Inpu l: LIDAR da a o: O hopho og ahy da a Ou pu m: LULC map Begin 1. De elop a ma ix as e in which e e y cell in ol es a physical posi ion 2. Add he co esponding s a is ics om land o o each pixel in as e 3. Selec a aining se om as e , called ain 4. Label each pixel in ain using expe knowledge 5. Build a SVM model, s m, om ain 6. Use s m o classi y e e y pixel in as e 7. Fo each pixel pin as e 7.1. Collec he neigbou hood o pin a se s 7.2. Build a k-NN model, knn, oms 7.3. Use knn o classi y p 8. Re u n a map mwi h e e y pixel spa ial posi ion and i s label End Fig. 1. The LULC classifica ion me hod based on a R-STACK algo i hm (s eps 6 o 8) The no el y o he R-STACK me hod se les on he second le el (s ep 7). Pa icula ly, on he applica ion o se e al classifie s (k-NN) and he way hey a e ained. Thus, a k-NN is build o each pixel aking i s neighbou s in he as e as aining se which in ol es a s ong ela ion (physical dependence) among he aining pixels and he cu en pixel o classi y. Fo he s udy a ea, we wo k wi h k= 3 and an 8-adjacency ha is, each 3-NN is de eloped wi h jus 8 ins ances o he pixel su ounding a ea. Fo his eason, he p ocess can be ackled om he poin o iew o efficiency and complexi y. In he end, he k-NN classifies he cu en pixel again ha ing used he model buil by i s neighbou s. In his way, possible inconsis ences and non-desi ed effec s can be emo ed. I is impo an o poin ou ha i is necessa y o make a as e copy be o e his las p ocess and whils he classes in he o iginal as e a e modified, e e y k-NN has o be build aking he neighbou s om he as e copy in o de o a oid colla e al A SVM and k-NN Res ic ed S acking o Imp o e LULC Classifica ion 497 Table 1. Six y-one candida e a iables. Va iables wi h (*) a e calcula ed o each band o a pixel: Heigh (H), In ensi y(I), Red(R), G een(G) and Blue(B). Va iable Desc ip ion Va iable Desc ip ion SNDVIMIN SNDVI minimum ICV In ensi y coefficien o a ia ion SNDVIMAX SNDVI maximum HCV Heigh coefficien o a ia ion SNDVISTD SNDVI S anda d de ia ion SLP Slope SNDVIAVG SNDVI a e age CRR Canopy elie a io MIN(*) Minimum PEC Pene a ion coefficien MAX(*) Maximum TOTALR To al o e u ns STD(*) S anda d de ia ion PCTN1 Unique e u n pe cen age AVG(*) A e age PCTN2 Double e u n pe cen age VAR(*) Va iance PCTN3 Th ee o mo e e u ns pe cen age SKEW(*) Skewness PCTR1 Fi s e u n pe cen age KURT(*) Ku osis PCTR2 Second e u n pe cen age RANGE(*) Range PCTR3 Thi d o la e e u n pe cen age NOTFIRST Second o la e e u n PCTR31 PCTR3 o e PCTR1 EMP Emp y neighbou s PCTR21 PCTR2 o e PCTR1 PCTR32 PCTR3 o e PCTR2 effec s. O he wise, he new classifica ion sequence would affec he esul o he emaining pixels. 4Resul s Two kinds o es ing ha e been ca ied ou o compa e he efficiency o ou ap- p oach agains a classical SVM. The fi s es is based on s a is ical echniques. Since emo e sensing da a is expensi e o gene a e, he compa ison has o es on an a ificial da a spli . In ou case, 100 spli s a e c ea ed om he o iginal da a so ha each spli con ains abou 740 ins ances. Then, a 10- old-c oss- alida ion p ocess is made o e e y spli . The esul s a e egis e ed o he ollowing com- pa ison p ocess. We ha e used he p ocedu e sugges ed in se e al wo ks [12] o obus ly com- pa ing classifie s ac oss mul iple da ase s in o de o e alua e he s a is ical sig- nificance o he measu ed diffe ences in algo i hm anks. The chosen p ocedu e in ol es he use o a s a is ical es o compa e classifie s one each o he . Ou objec i e was o compa e a classical SVM o ou app oach in e ms o accu acy. Thus, he Wilcoxon p ocedu e was selec ed as he app op ia e es . A ai compa ison o he algo i hms is ob ained by a e age anks and in his case, a e he p e ious 100 10- old-c oss- alida ion esul s, ou app oach anks fi s . Wi h he measu ed a e age anks, he Wilcoxon es checks whe he he a e age anks a e significan ly diffe en om he mean ank = 1.5 expec ed unde he null hypo hesis. Leaning on a s a is ical package (MATLAB), p alue o he Wilcoxon es ha e esul ed on a alue less han 5.72e−06 so he null 498 J. Ga cia-Gu ie ez, D. Ma eos-Ga cia, and J.C. Riquelme-San os Table 2. A summing up o he hold-ou es o he SVM classical app oach Use class Wa e Ma sh Roads o Low Middle High Buildings Coal Dumps Mixed sample ailways Veg. Veg. Veg. deposi s a eas Wa e 178 5 0 0 0 0 0 0 0 2 Ma shland 0 100 1 2 2 1 0 2 0 1 Roads o ailways 0 4 69 0 6 0 0 0 1 0 Low Veg. 0 4 2 50 1 0 0 0 0 0 MiddleVeg.0 9 2 2213 0 0 0 0 High Veg. 0 0 0 0 0 26 0 0 0 0 Buildings 0 2 3 0 2 1 31 0 0 0 Coal deposi s01 0 410 0 10 0 0 Dumps10 0 000 0 0 219 Mixed a eas 0 0 17 0 0 0 0 0 1 2 TP Ra e 0.962 0.917 0.863 0.877 0.568 1.0 0.795 0.625 0.677 0.1 FP Ra e 0.002 0.051 0.048 0.015 0.021 0.009 0 0.003 0.004 0.021 P ecision 0.994 0.8 0.734 0.862 0.636 0.839 1 0.833 0.913 0.143 KIA 0.815 Co ec ly classified 0.846 Table 3. A summing up o he hold-ou es o he SVM + k-NN es ic ed s acking Use class Wa e Ma sh Roads o Low Middle High Buildings Coal Dumps Mixed sample ailways Veg. Veg. Veg. deposi s a eas Wa e 181 3 0 0 0 0 0 0 0 1 Ma shland 1 98 1 5 1 1 0 0 0 2 Roads o ailways 0 4 72 0 2 0 0 0 0 2 Low Veg. 0 2 2 53 0 0 0 0 0 0 MiddleVeg.0 4 0 5253 0 0 0 0 High Veg. 0 0 0 0 0 26 0 0 0 0 Buildings 0 2 2 0 2 2 31 0 0 0 Coal deposi s01 1 400 0 10 0 0 Dumps10 0 000 0 0 246 Mixed a eas 0 0 15 1 0 0 0 0 0 4 TP Ra e 0.978 0.899 0.9 0.93 0.676 1.0 0.795 0.625 0.774 0.2 FP Ra e 0.005 0.033 0.04 0.028 0.009 0.01 0 0 0 0.019 P ecision 0.989 0.86 0.774 0.779 0.833 0.813 1 1 1 0.267 KIA 0.847 Co ec ly classified 0.873 A SVM and k-NN Res ic ed S acking o Imp o e LULC Classifica ion 499 hypo hesis is ejec ed. Ha ing ound ha he measu ed a e age anks a e sig- nifican ly diffe en (a α= 0.05), ou analysis based on anks e eals ha he accu acy o classical SVM is significan ly wo se han ha o ou app oach o his kind o da a. The second ype o es ing is a hold-ou p ocess wi h da a p e iously classified. This is he common es ing in emo e sensing. The es da a se (600 ins ances) was selec ed om he o iginal da a se because o i s special difficul y o be classified and i is no pa o he aining se . In Table 3 and Table 2, esul s o ou app oach and classic SVM a e shown when he hold-ou p ocess is ca ied ou . The gene al imp o emen is a 3% which is a e y impo an ad ance. 5 Conclusions In his pape , a new me hod based on a mul iple classifie s ensemble was used o imp o e LULC map accu acy. The me hod buil a s a is ical as e om LIDAR and image usion da a ollowing a pixel-o ien ed s a egy. Then, he pixels om a aining a ea we e used o ain a SVM and k-NN es ic ed s acking (called R-STACK) aking in o accoun he special cha ac e is ics o spa ial da a. A compa ison be ween a SVM and he R-STACK me hod was ca ied ou . The esul s in a ipa ian a ea o Huel a (Spain) showed a global accu acy o 84.6% o he classical SVM and 87.6% o he new app oach which means a significan ad ance. E en hough esul s a e sa is ac o y, he e a e s ill se e al p oblems o fix. Some o hem a e ela ed o shadows om images and i s weigh on he final classifica ion which has o be aken in o accoun . Hence, a con ol o weigh s o each ea u e has o be implemen ed in o de o a oid hei misclassifica ion effec s. Gene ic algo i hms could be a e y sui able ool o sol e his p oblem. In addi ion, dependence on he aining se can be a mo e impo an p oblem. Some imes, he aining se can be incomple e o no enough o desc ibe he eal space. These p oblems a e ha de o fix. Despi e he ac ha a semi-supe ised app oach seems o be mo e sui able o so ou his kind o p oblems, e y ew semi-supe ised p oposals can be ound ye and mo e esea ch is needed in o de o de elop hem wi h he equi ed accu acy. Finally, some p oblems a e inhe en in pixel-o ien ed app oaches such as he de ec ion o pa ial a ificial s uc u es. In he u u e, i would be e y in e es ing o apply a p io phase in which a low addi ion o he compu a ional cos , an objec -o ien ed segmen a ion and classifica ion could be ca ied ou o ex ac he mos difficul s uc u es o classi y, using isual ecogni ion echniques om he compu e ision wo ld. Acknowledgmen s. We would like o hank he Regional Minis y o Andalu- sia o all he suppo ecei ed in he de elopmen o his wo k and especially, o hank I ene Ca pin e o, Juan Jos´e Vales and Daniel Laguna o hei e y app ecia ed commen s. We would also like o hank F ancisco Ma ´ınez-´ Al a ez and Luis Gon¸cal es-Seco o all he ime hey in es ed ha allowed his wo k o be comple ed. 500 J. Ga cia-Gu ie ez, D. Ma eos-Ga cia, and J.C. Riquelme-San os Re e ences 1. Ve els , J., Gee ling, G., Syko a, K., Cle e s, J.: Mapping o agg ega ed floodplain plan communi ies using image usion o casi and lida da a. In e na ional Jou nal o Applied Ea h Obse a ion and Geoin o ma ion (11), 83–94 (2009) 2. An ona akis, A., Richa ds, K., B asing on, J.: Objec -based land co e classifi- ca ion using ai bo ne LIDAR. Remo e Sensing o En i onmen (112), 2988–2998 (2008) 3. Dalpon e, M., B uzzone, L., Vesco o, L., Damiano, G.: The ole o spec al es- olu ion and classifie complexi y in he analysis o hype spec al images o o es a eas. Remo e Sensing o En i onmen (113), 2345–2355 (2009) 4. Shao, G., Wu, J.: On he accu acy o landscape pa e n analysis using emo e sensing da a. Landscape Ecology (23), 505–511 (2008) 5. Fau el, M., Benedik sson, J., Chanusso , J., S einsson, J.: Spec al and spa ial classifica ion o hype spec al da a using SVMs and mo phological p ofiles. IEEE T ansac ions on Geoscience and Remo e Sensing 46(11), 3804–3814 (2008) 6. Mi akis, N., Topaloglou, C., Alexand idis, T., Theocha is, J., Zalidis, G.: Decision usion o GA sel -o ganizing neu o- uzzy mul ilaye ed classifie s o land co e clas- sifica ion using ex u al and spec al ea u es. IEEE T ansac ions on Geoscience and Remo e Sensing 46(7), 2137–2152 (2008) 7. Waske, B., an de Linden, S.: Classi ying mul ile el image y om sa and op- ical senso s by decision usion. IEEE T ansac ions on Geoscience and Remo e Sensing 46(5), 1457–1466 (2008) 8. Co ijo, F.J., Blanca, N.P.D.L.: Imp o ing classical con ex ual classifica ions. In- e na ional Jou nal o Remo e Sensing 19(8) (1998) 9. Goncal es-Seco, L., Mi anda, D., C ecen e, R., Fa o, J.: Digi al e ain model gene a ion using ai bo ne LIDAR in flo es ed a ea o Galicia. In: P oceedings o 7 h In e na ional Symposium on Spa ial Accu acy Assessmen in Na u al Resou ces and En i onmen al Sciences, Spain, pp. 169–180 (2006) 10. Hudak, A.T., C ooks on, N.L., E ans, J.S., Halls, D.E., Falkowski, M.J.: Nea es neighbo impu a ion o species-le el, plo -scale o es s uc u e a ibu es om lida da a. Remo e Sensing o En i onmen 112, 2232–2245 (2008) 11. Hall, M., F ank, E., Holmes, G., P ah inge , B., Reu emann, P., Wi en, I.H.: The WEKA da a mining so wa e: An upda e. SIGKDD Explo a ions 11(1) (2009) 12. Ga cia, S., He e a, F.: An ex ension on s a is ical compa isons o classifie s o e mul iple da a se s o all pai wise compa isons. Jou nal o Machine Lea ning Re- sea ch 9, 2677–2694 (2008)