An extensive analysis of the interaction between missing data types, imputation methods, and supervised classifiers

An ex ensi e analysis o he in e ac ion be ween
missing da a ypes, impu a ion me hods, and
supe ised classi ie s
Unai Ga cia ena∗
, Robe o San ana∗
Facul y o In o ma ics, Uni e si y o he Basque Coun y. P Manuel La dizabal, 1 - 20018
Donos ia-San Sebas i´an, Gipuzkoa, Spain.
Abs ac
When da a-mining eal-wo ld da a, we o en ind ou sel es acing obse a ions
ha ha e no alue eco ded o some a ibu es. This can be caused by se e al
phenomenon, such as a machine’s incapabili y o eco d ce ain cha ac e is ics
o a pe son e using o answe a ques ion in a poll. Depending on ha mo i a-
ion, alues gone missing may ollow one kind o pa e n o ano he , o desc ibe
no egula i y a all. One app oach o pallia e he e ec o missing da a on
machine lea ning asks is o eplace he missing obse a ions. Impu a ion algo-
i hms a emp o calcula e a alue o a missing gap, using in o ma ion a ound
i , i.e., he a ibu e and/o he alue o he obse a ion. While se e al impu-
a ion me hods ha e been p oposed in he li e a u e, ew wo ks ha e add essed
he ques ion o he ela ionship be ween he ype o missing da a, he choice o
he impu a ion me hod, and he e ec i eness o classi ica ion algo i hms ha
used he impu ed da a. In his pape we add ess he ela ionship among hese
h ee ac o s. By cons uc ing a benchma k o hund eds o da abases con ain-
ing di e en ypes o missing da a, and applying se e al impu a ion me hods
and classi ica ion algo i hms, we empi ically show ha a in e ac ion be ween
impu a ion me hods and supe ised classi ica ion can be deduced. Besides, di -
e ences in e ms o classi ica ion pe o mance o he same impu a ion me hod
∗Co esponding au ho
Email add esses: [email p o ec ed] (Unai Ga cia ena),
[email p o ec ed] (Robe o San ana)
This is he accep ed manusc ip o he a icle ha appea ed in inal o m in Expe Sys ems wi h Applica ions 89 : 52-65
(2017), which has been published in inal o m a h ps://doi.o g/10.1016/j.eswa.2017.07.026. © 2017 Else ie unde CC
BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/)
in di e en missing da a pa e ns ha e been ound. This poin s o he con e-
nience o conside ing he combined choice o he impu a ion me hod and he
classi ie algo i hm acco ding o he missing da a ype.
Keywo ds: missing da a, impu a ion me hods, supe ised classi ie s, machine
lea ning
1. In oduc ion
Missing alues a e ubiqui ous in almos e e y ype o eal-wo ld da ase s.
They can be pa icula ly de imen al o ce ain applica ions o he da ase s,
especially when he dis ibu ion o he missing da a (MD) is no uni o m and a
possible mechanism ha could explain he los alues is unknown. Pe haps he
mos used among he non- i ial al e na i es o deal wi h MD a e impu a ion
me hods (IMs). These me hods eplace he missing alues by es ima es ha can
be aken om he da abase (DB), de i ed om s a is ics o known alues (e.g.,
he mean o a gi en a iable), o ob ained using mo e sophis ica ed algo i hms.
The e is consensus on he impo ance o he applica ion o IMs, especially
when DBs wi h MD a e used as a basis o lea ning supe ised classi ie s. How-
e e , he choice o he IM, and i s impac on he classi ie pe o mance can be
e y dependen on he MD ype. Fo example, an imp ope choice o he IM can
bias he lea ned classi ie , p oducing a low classi ica ion quali y on es da a.
When he p oblem o supe ised classi ica ion is conside ed, hese h ee el-
emen s a e s ongly in e wined. In his pape we analyze his ela ionship by
in es iga ing p oblems wi h di e en ypes o MD, add essed using a se o IMs
wi h he inal goal o supe ised classi ica ion by means o di e en ypes o
classi ie s. Ou aim is o de e mine o wha ex en he e is a ela ionship be-
ween he choice o he IM and he p ecision o he classi ie s when conside ing
DBs ha exhibi di e en ypes o MD.
P e ious wo k (Ba is a & Mona d, 2003; Luengo e al., 2012) has analyzed
he ela ionship be ween he IMs used o ea ing MD and classi ie s. Ba is a
& Mona d (2003) e alua ed ou IMs o wo di e en classi ie s concluding ha
2
he choice o he IM in luences he pe o mance o he classi ie s. A mo e in-
dep h s udy on he ela ionship be ween IMs and classi ie s was p esen ed by
Luengo e al. (2012). Au ho s conduc ed an ex ensi e e alua ion o classi ie s
and IMs on eal-wo ld DBs and concluded ha he choice o he IM should
indeed be condi ioned on he ype o classi ica ion me hod used.
In his pape we go beyond he analysis o he ela ionship be ween IMs and
classi ica ion algo i hms, and conside as ano he ac o he pa icula cha ac-
e is ics o he MD. We hypo hesize ha he h ee p e iously men ioned ac o s
can in luence he classi ica ion esul s and should be conside ed in hei in e ac-
ion. We in es iga e his hypo hesis by de ising p ocedu es ha gene a e DBs
wi h di e en ypes o MD, and using hem as a benchma k, we e alua e he
e ec o he MD ype and he IMs on he pe o mance p o ided by he clas-
si ie . Ano he con ibu ion o ou wo k is he simul aneous use o eal-wo ld
DBs, which a e used as a basis o cons uc he benchma k, wi h an a i icially
gene a ed MD ype which is in oduced in he o iginal DB. Following his s a -
egy, we can con ol he cha ac e is ics o he MD and e alua e he e ec on
he o he ac o s analyzed. In ou in es iga ion we also e alua e an ex ensi e
numbe o classi ie s, including many o hose in es iga ed in p e ious wo k and
some o he mo e ecen classi ica ion app oaches.
The pape is o ganized as ollows. In he nex sec ion, some essen ial back-
g ound on he main concep s co e ed in he pape is gi en. Rela ed wo k, em-
phasizing he connec ion wi h ou p oposal, is discussed in Sec ion 3. Sec ion 4
gi es a o mal p esen a ion o he me hods used o gene a e he di e en ypes
o MD. This sec ion also desc ibes he da abases selec ed o e alua e he ela-
ionships be ween he me hods and algo i hms. Sec ions 5 and 6 espec i ely
p esen he impu a ion and classi ica ion me hods in es iga ed. In Sec ion 7
we desc ibe he expe imen al amewo k, he esul s o he expe imen s, and
discuss some o ou indings. Sec ion 8 concludes he pape and p esen s some
lines o u u e esea ch.
3
2. Backg ound
2.1. Missing da a ypes
Many di e en easons can cause missing da a (MD) in eal-wo ld da abases.
Iden i ying any pa e n in he MD alues is a key aspec a he ime o con-
cei ing me hods o deal wi h he missing obse a ions. In pa icula , he ype
o MD can di ec ly impac he quali y o he p edic ions o he classi ica ion
me hods applied o he da a. The e o e, se e al wo ks ha e been de o ed o
cha ac e ize he ypes o MD, p oposing me hods o de ec hese ypes and sug-
ges ing algo i hms o impu a ion. In his sec ion we e iew he mos common
accep ed classes o MD and hei expec ed e ec on he beha io o supe -
ised classi ica ion echniques (Ba is a & Mona d, 2003; Gelman & Hill, 2006;
He nndez-Pe ei a e al., 2015; Blombe g & Ruiz, 2013; Luengo e al., 2012):
•Missing Comple ely A Random (MCAR): When he da abase’s measu e-
men ailu es occu andomly, he e is no speci ic pa e n o be iden i ied.
The impac o MCAR on a classi ica ion algo i hm will depend on he MD
dis ibu ion o e he da a. The mo e uni o m he dis ibu ion o he MD
is, he less bias is expec ed o be in oduced in he da abase.
•Missing A Random (MAR): MD is ca aloged as MAR when a pa e n
can be iden i ied, i.e., we can ind a common ac o in all he obse a ions
wi h missing alues. Fo example, we ind ha when a ce ain a iable
(wi h no MD) akes ex eme alues o an obse a ion, wo o he a iables
end o be missing o ha same obse a ion.
•Missing No A Random (MNAR): This MD ype is simila o MAR.
Howe e , in his case he alues causing o he s o be missing a e no
known, his can ha e wo o igins:
–Missingness depending on unobse ed Va iables (MuOV): One o he
easons hese alues a e no known can be ha simply hey we e no
obse ed.
4
–Missingness depending on i s Value I sel (MIV): An elemen can
be missing depending on i s alue i sel . This could happen when a
a iable akes a alue ou o i s ep esen a ion ange.
In gene al, i is no possible o iden i y he MCAR ype o MD since in eal
da abases he e is no way o ack he cause o his MD. MCAR can be caused
by a huge a ie y o easons, om da a loss du ing an in o ma ion ans e ence,
o a pe son’s e usal o p o ide pe sonal da a in a poll, e c. As s a ed abo e,
assuming ha he missing alues a e uni o mly dis ibu ed, he da ase will
no expe ience a conside able loss o in o ma ion. As long as he amoun o
missing alues is no signi ican , e en disca ding obse a ions con aining MD
will no necessa ily ha e an impac on pos e io classi ica ion. Howe e , e en i
he amoun o in o ma ion los ega ding missing obse a ions may be small1,
IM quali y could be as impo an as in o he MD ype.
The MAR kind o MD is no as common as MCAR, bu i is easie o in e i s
o igin by s udying o he a iables o he da ase . Fo example, in a si ua ion in
which people a e asked abou hei habi s and heal h, some in o ma ion abou
seden a y li es yle migh be a ailable. Howe e , while some subjec s may be
open o sha e in o ma ion abou hei weigh , o he subjec s (mo e likely hose
wi h an o e weigh condi ion) migh be mo e eluc an o disclose his ype o
in o ma ion. This example illus a es si ua ions in which a cause o MAR can
be in e ed om an analysis o he cha ac e is ics o he da abase.
MAR ype o MD can be a po en ial sou ce o p oblems o he pe o mance
o e ed by he classi ica ion algo i hms. Since in his case he e is an unde lying
eason o he MD, i is likely ha obse a ions con aining MD will be simila
o each o he and will be agged in he same class. This could lead o an
unbalanced da abase ha will po en ially a ec he classi ica ion. In his case,
disca ding da a is no an ad isable op ion, and he use o IMs is a equi emen .
Finally, MNAR p esen s a conside ably mo e di icul si ua ion. Following
1Johnny, I’m no su e i his sen ence is clea and co ec
5

he p e ious example, he ask would become much mo e edious i we had no
asked abou o he medical and li es yle pa ame e s (MuOV). Ano he scena io
needs o be add essed when an indi idual is ashamed and e uses o disclose
in o ma ion abou he amoun o money he o she spends on d ugs. This a iable
mos likely depends only on i sel . In his second case we would ha e (MIV).
These wo si ua ions a e he mos p oblema ic ones, since i can be as ha m ul
as MAR o he da a, bu i can be easily misiden i ied as MCAR, as a esul
o he impossibili y o iden i y a pa e n in he unobse ed a iables.
2.2. Impu a ion me hods
The e a e wo main ways o handling MD. Igno ing obse a ions wi h missing
da a is gene ally a good choice when he ca dinal o missing alues is ela i ely
small and he MD is homogeneously dis ibu ed. Ne e heless, when hese mea-
su emen ailu es a e concen a ed in a single a iable, o when igno ing hem
would suppose a big loss o in o ma ion, we conside echniques o ill he gaps.
This is essen ially wha impu a ion does. Se e al s a egies ha e been p oposed
o his pu pose and hey can exhibi impo an di e ences in e ms o com-
plexi y, and ou pu quali y (B owns one & Valle a, 2001; Liu & B own, 2013;
Ba is a & Mona d, 2003; Lakshmina ayan e al., 1999). Sec ion 5 will p esen
a numbe o impu a ion me hods ele an o ou wo k.
2.3. Classi ica ion p oblems and supe ised classi ica ion algo i hms
Classi ica ion is he ask o lea ning a a ge unc ion ha maps an a ibu e
collec ion o a p ede ined class. Algo i hms ha sol e his p oblem could be
de ined as p ocesses ha gene alize pa e ns om a se o gi en obse a ions.
These ope a ions, which a e able o bo h p edic and desc ibe da a, can be
di ided in o wo majo b anches: Supe ised and Unsupe ised Classi ica ion
(SC and UC). In a supe ised classi ica ion p oblem (Tan e al., 2006; Hilbe
& Lpez, 2011), a se o ( aining) da a is gi en and, o each case in he aining
se , he associa ed class (o label) is known. The classi ica ion p oblem consis s
6
o using he in o ma ion he aining da a con ains o in e o p edic he class
o new cases ha a e no labeled.
We ocus on he analysis o SC algo i hms and how hei beha io is impac ed
by MD and impu a ion me hods. This supe ised classi ica ion pa adigm can
be e y sensi i e o he MD p oblem. We expec ha i a conside able amoun
o he da a is missing o a ce ain a iable in a ce ain class, he classi ie may
s uggle o o e an accep able beha io when agging an obse a ion wi h ha
men ioned class. Howe e , he way in which di e en ypes o MD in luence he
beha io o he classi ie s has no been p e iously s udied in de ail.
We no ice ha , conside ing he classi ica ion scena io, MD can be p esen
in he aining da a, o which we know he class and also in he es da a, o
which in o ma ion abou he label is unknown. In his pape , we conside hese
wo si ua ions as he same scena io, i.e., we do no conside me hods ha use
class in o ma ion a he ime o implemen ing he impu a ion algo i hm.
In Sec ion 6, we p esen he classi ie s ha will be used in he expe imen al
pa o his pape .
3. Rela ed wo k
No many ecen s udies ha e esea ched he in e ac ion be ween missing
da a con igu a ions, MD handling me hods, and supe ised classi ica ion algo-
i hms om his app oach. In he ollowing pa ag aphs, some pape s ha ha e
in es iga ed his ques ion a e e iewed.
3.1. Gene al da a
The mos exhaus i e in es iga ion on he join beha io o IMs and classi-
ica ion algo i hms is he pape by Luengo e al. (2012). They pe o med an
analysis wi h 23 classi ica ion algo i hms (g ouped in h ee classes) combined
wi h 14 ways o dealing wi h MD. They selec ed 21 DBs, all o hem con aining
na u al missing alues, anging om 0.06 o 21.82 pe cen om he o al alues
in he ables. Since hey ha e no knowledge abou he MD dis ibu ion, hey
7
assumed i ollowed a MAR pa e n. This is a undamen al di e ence wi h he
app oach we ollow in his pape , since ou ocus is on he in luence o he MD
ype and he e o e he me hods we p opose o c ea e he es ing benchma k
a e comple ely di e en and mo e sophis ica ed o he one p oposed in Luengo
e al. (2012). Howe e , he expe imen s conduc ed by he au ho s p o ed he
posi i e e ec o impu ing da a, and ound e idence o he ela ion be ween
he classi ie ype and he IM. The same au ho s de eloped he wo k p esen ed
in (Luengo e al., 2010), which ocuses on he Radial Basis Func ion Ne wo k
(RBFN) classi ie , poin ing ou he imp o emen o he classi ie esul s when
combined wi h an E en Co e ing impu e , p oposed by Wong & Chiu (1987).
Ba is a & Mona d (2003) s udied he e ec s o impu a ion o e he C4.5
and CN2 classi ie s. They chose ou da abases om which 3 had no MD, and
inse ed a ying pe cen ages o missing alues (10, 20, 30, 40, 50 and 60%)
and hen p oceeded o classi y hem. The obse a ions con aining MD we e
emo ed, as hey ep esen ed only abou 2% o he o al o ha DB. The
DBs wi h missing alues we e classi ied wi h no impu a ion (no e ha C4.5
and CN2 ha e hei own way o wo king wi h MD), mean/mode impu a ion,
and K-nea es neighbo s impu a ion (KNNI) (wi h k= 10). 10-NNI showed
be e pe o mance han he o he simple ea men s, bu he expe imen s ound
limi a ions in he way MD was inse ed, since only some o all he a ibu es
chosen we e a ec ed. Also, only wo basic IMs we e conside ed in Ba is a &
Mona d (2003).
Acuna & Rod iguez (2004) e alua ed he e ec o h ee me hods o dealing
wi h MD on he misclassi ica ion e o a e. They used he mean, median and
KNNI IMs, and also in es iga ed he esul s o case dele ion. Then hey es ed
he esul s wi h wo classi ie s, Linea Disc iminan Analysis (LDA) and K-
nea es neighbo s (KNN). This wo k adop ed 12 da ase s, 4 o which con ained
na u al MD. These ou DBs we e p e ea ed in o de o make hei s a ing
poin s simila by applying he IM. The esul s achie ed by Acu˜na and Rod ´ıguez
showed ha he IM e ec on accu acy has a highe dependence on he p oblem
a he han on he classi ica ion algo i hm. These esul s poin in he di ec ion
8
o he ques ion we in es iga e in his pape , since one o he ea u es cha ac-
e izing a gi en DB is he MD ype. Howe e , he wo k p esen ed by Acuna &
Rod iguez (2004) did no add ess his ques ion explici ly and he numbe o IMs
and classi ica ion me hods was limi ed in compa ison o he s udy we p esen
he e.
Fa hang a e al. (2008) examined he impac o pe o ming MD impu a ion
as a p ep ocessing s ep o pos e io classi ica ion. They combined di e en e -
sions o ou impu a ion me hods (Ho Deck, Na¨ı e Bayes Polynomial mul iple
Reg ession and Mean) wi h six classi ie s (Rippe 2, C4.5, RBFN, Suppo Vec-
o Machine (SVM), KNN and na¨ı e Bayes). They selec ed 15 DBs and consid-
e ed 5, 10, 20, 30, 40 and 50% o MCAR- ype MD in hei expe imen s. They
concluded ha he imp o emen in he accu acies ob ained om he da ase
wi h MD and he impu ed one had no ela ion wi h he pe cen age o MD.
Also, di e ences in he imp o emen we e shown o di e en IM-classi ica ion
algo i hm combina ions, which led hem o conclude ha no uni e sally bes
IM exis s. The pape inally esol es ha impu a ion is bene icial o machine
lea ning asks, o e all.
H uschka J e al. (2007) in oduced wo IMs based on Bayesian Ne wo ks
and con as ed hem wi h ou o he classic me hods, namely, Expec a ion-
Maximiza ion, Da a Augmen a ion, Decision T ees, and Mean/Mode. Fou
da ase s we e used as a benchma k, which combined na u al and a i icial miss-
ing alues in oduced in he expe imen s. Fou classi ie s we e conside ed in he
expe imen s. This wo k concluded ha IM ob aining close alues o o iginal
da a (compa ing impu a ions in a i icially gene a ed MD o alues o iginally
in hose cells) did no necessa ily p oduce be e esul s when classi ying.
Song e al. (2008) pe o med an analysis simila o he one p esen ed by
Ba is a & Mona d (2003), in which he e ec o KNN-impu a ion o e C4.5
classi ica ion was in es iga ed. Howe e , in hei pape hey conside ed h ee
MD ypes (MCAR, MAR and NMAR). Thei conclusions ega ding IM-C4.5
coincide wi h hose ound by Ba is a & Mona d (2003) , and also s a e ha
he MD mechanism in luences he classi ying ask. Ne e heless, an in-dep h
9
4.1.3. MIV
This algo i hm is e y simila o he one used o c ea e he MAR pa e n
o MD, bu ins ead o gene a ing a iables ha will lose hei alues depending
on he alues o a causa i e a iable, i will be he causa i e i sel which loses
a iables. The pseudocode is shown in Algo i hm 3
Inpu :
da a: Da abase
mdp: MD pe cen age
nV: numbe o a iables losing hei alues
Ou pu :Da abase wi h mdp% gene a ed MD
begin
x = numObse a ions(da a)
y = numVa iables(da a)
causa i es = andom([0,y], nV)
o i∈[0,leng h(causa i es)] do
aux = da a[:,causa i es[i]]
o j∈[0, x ·y·mdp/100//nV] do
obse a ions[j] = minIndex(aux)
aux[obse a ion[j]]=maxIn
end
o j∈[0,leng h(obse a ions)] do
da a[obse a ions[j], causa i e] = “NaN”
end
e u n (da a)
end
end
Algo i hm 3: MIV gene a ing algo i hm.
4.1.4. MuOV
This algo i hm is also qui e simila o he one p oposed o gene a ing MAR
MD ype, bu in his case he causa i e a iable is unobse ed. The e o e,
16

he obse a ions ha will ha e missing alues will be chosen andomly. The
pseudocode is shown in Algo i hm 4.
Inpu :
da a: Da abase
mdp: MD pe cen age
nV: numbe o a iables losing hei alues
Ou pu :Da abase wi h mdp% gene a ed MD
begin
x = numObse a ions(da a)
y = numVa iables(da a)
MDVa iables = andom([0,y], nV)
o i∈[0, x ·y·mdp/100)//nV] do
obse a ions[i] = andom([0,x])
end
o i∈[0,leng h(obse a ions)] do
o j∈[0, leng h(MDVa iables)] do
da a[obse a ions[i], MDVa iables[j] = “NaN”
end
end
e u n (da a)
end
Algo i hm 4: MuOV gene a ing algo i hm.
As p e iously indica ed, his algo i hm ollows he same s uc u e, bu since
in his case he causa i e a iables a e unknown, he obse a ions o lose hei
alues will be chosen andomly.
Some o ou me hods o in oduce he missing da a seek o simula e he
di e en ypes o MD which depend on isible o hidden dependencies be ween
he a iables in he DB. The e o e, some o he algo i hms selec he obse a ions
whe e MD will be in oduced based on he alues o he causa i e a iable. We
no ice ha o he c i e ia o de ine he MD ypes ha e been p oposed in he
li e a u e. In pa icula , he MD pa e ns could be de ined in e ms o he
pe cen age o MD wi hin a single obse a ion (Deb & Liew, 2016).
17
4.2. Desc ip ion o he da abases in es iga ed
We ha e selec ed a se o 10 da ase s om he UCI Machine Lea ning Repos-
i o y (Lichman, 2013) buil in comple ely di e en con ex s in o de o achie e
esul s which a e as gene ic as possible, a oiding biasing he IMs and classi ie s
wi h a speci ic beha io om he da a. We ha e da abases obse ing di e se
na u al aspec s (Fo es , Biodeg, Clima e, Lea ), medical ma e measu emen s
(Diabe ic, BUPA, Tho acic), c edi denial/app o al (Ge man), ehicle dimen-
sion analysis (Vehicle) and image pixel in e p e a ion (Segmen a ion).
Table 1 con ains a s uc u al desc ip ion o he DBs. The i s h ee columns
(Ca ego ical, In ege and Real) p o ide in o ma ion abou he ype o he ele-
men s con ained in he DB. The ollowing wo columns, (N. Va iables, N. Cases)
e e o he numbe o measu emen s o each obse a ion (including he class
ag) and he amoun o obse a ions he DB con ains. Then, he index each
DB has been assigned in o de o simpli y u u e e e ences o dis inc da ase s.
The las wo columns e e exclusi ely o he class o he da a. n is he numbe
o classes, while Hn ep esen s he no malized en opy, compu ed as:
Hn(p) = −P
i
pi·logb(pi)
logb(n)
whe e piis he p obabili y o class i,bis he loga i hm base (in ou case we
used ebase, he na u al loga i hm), and nis he numbe o classes in he da a.
The esul o his p ocedu e p oduces a numbe be ween 0 and 1, which may be
in e p e ed as he nea e o 1, he mo e unce ain y we ind in a class, and hus,
he be e dis ibu ed is. And he nea e o 0, he mo e unbalanced owa ds
one ce ain class i is. This coe icien is no malized by he numbe o classes.
We used his measu e in o de o equilib a e measu emen s be ween bina y and
mul i-class da abases.
18
Da abase Ca eg. In . Real N.Fea s. N.Cases DB Index n Hn
Biodeg 7 3 3 41 1055 5 2 0.92
BUPA 3 3 3 7 345 4 2 0.98
Clima e 7 7 3 18 540 8 2 0.42
Diabe ic 7 3 3 20 1151 7 2 1
Fo es 3 3 3 27 326 6 4 0.91
Ge man 3 3 7 20 1000 3 2 0.88
Lea 7 7 3 16 340 1 30 1
Segmen a ion 7 7 3 19 2310 9 7 1
Tho acic 3 3 7 17 470 10 2 0.61
Vehicle 7 3 7 18 946 2 4 1
Table 1: Desc ip ion o he o iginal da ase s used o gene a e he benchma k.
5. Impu a ion me hods
IMs di e in he class o DBs hey can be applied o, hei compu a ional
complexi y, and he sophis ica ion o he me hods used o eplace he MD.
The e a e s aigh o wa d echniques o eplace missing obse a ions (Ba is a &
Mona d, 2003; Lakshmina ayan e al., 1999; Gelman & Hill, 2006), ha simply
copy alues om o he obse a ions (using simila i y as a c i e ion, o example).
O he mo e elabo a ed impu a ion algo i hms use Pa ame e Es ima ion (PE)
(Yuan, 2010; Honake & King, 2010) o eplace he missing alues om he
a ailable da a. The i s class o me hods equi e less compu a ional ime and
hei esul s could be su icien ly good o some DBs. When hese simple
s a egies do no p oduce esul s ha i he expec a ions, PE can p esumably
p o ide mo e accu a e esul s.
In his pape we ha e selec ed 8 impu a ion me hods, including some o he
mos equen ly used and mo e sophis ica ed app oaches:
The ini ial h ee me hods conside ed a e Mean-Mode,Median and Mos
F equen Value impu a ion. These me hods ha e sel -explana o y names, as
hey calcula e he s a is ics and simply asc ibe hem o gaps.
Las Value Ca ied Fo wa d impu a ion is also sel -explana o y. This unc-
ion sea ches he a iable o he las a ailable alue, and uses i as i s guess o
19
he missing alue.
In e pola ion’s wo king sys em can also be easily deduced om i s name,
bu his me hod conside s a pa ame e ha may change i s complexi y, hus
a ec ing i s compu a ional cos . The in e pola ion me hod calcula es a unc ion
ha i s bo h he p e ious and la e alues o a missing alue s e ch and ills
i using he unc ion.
Ho Deck impu a ion can also ha e a high compu a ional cos , since i de-
pends on he numbe o obse a ions and a iables he DB has. This me hod
compu es he dis ance (e.g., euclidean dis ance) be ween an obse a ion wi h
one (o mo e) missing alue(s) and eplica es he alue o ha ce ain a iable
in he nea es obse a ion.
I e a i e Impu a ion basically impu es he same missing alues mul iple
imes. The common ope a ion o hese me hods is o make ini ial simple guesses
o missing alues (e.g., mean impu a ion) and eimpu a es he same missing
alues using o he (mo e complex, usually) me hods. Once all he a iables
ha e been eimpu ed, he cycle is epea ed. An i e a ion limi o a ole ance
h eshold could be se as a hal ing condi ion.
Mul iple Impu a ion also belongs o he compu a ionally complex me hod
spec um. This s a egy duplica es he DB m imes and impu es hem sepa-
a ely. Subme hods used o pe o m impu a ion on hese eplicas may o may
no be di e en s a egies. Nex , one o wo s a egies can be ollowed. The i s
one applies he analysis algo i hm (Supe ised Classi ica ion in ou case) and
hen combines hei esul s. The second op ion i s pe o ms he combina ion
s ep, o ming one single DB again, being able his way o know he a iance o
he impu ed alues. Then he analysis is applied. The ecommended m alue
used o his p ocedu e has (and s ill is) inc eased as compu a ion capaci y has
isen, being he classical ad ice 3 ≤m≤5, ha ing eached o 20 ≤m≤100 by
some au ho s, i.e., Van Buu en (2012).
This wo k has used wo impu a ion me hods ha pe o m Mul iple Im-
pu a ion; MICE (Mul i a ia e Impu a ion by Chained Equa ions) (Buu en &
G oo huis-Oudshoo n, 2011) and Amelia (Honake e al., 2011).
20
We no ice ha some o he IMs conside ed he e a e mo e app op ia e o a
pa icula ype o da a (e.g., he in e pola ion me hod is concei ed o da a wi h
a empo al componen ). Howe e , a he ime o applying he IMs we do no
ake in o accoun he cha ac e is ics o he da a. We expec ha i a gi en IM
is no app op ia e o one DB, his ac will be ansla ed in he esul ob ained
by he classi ie o ha pa icula DB.
6. Supe ised classi ica ion me hods
In his sec ion we desc ibe he classi ica ion me hods e alua ed in he pape .
De ails abou he implemen a ion a e desc ibed in Sec ion 7.1.2. When no de-
ails abou he pa ame e s used by he classi ie s a e p o ided, i is assumed ha
hey we e applied wi h hei de aul pa ame e s in he used implemen a ions.
We use he ollowing classi ie s:
1. Regula ized logis ic eg ession wi h no m l1 (Ll1) (Yu e al., 2011)
2. Regula ized logis ic eg ession wi h no m l2 (Ll2) (Yu e al., 2011)
3. Linea disc iminan analysis (LDA) (Leas Squa es) (Fishe , 1936)
4. Quad a ic disc iminan analysis (QDA) ( eg pa am = 0.01)
(F iedman, 1989)
5. Deep Neu al Ne wo k (hidden laye s = 2, op imize = Adag ad,
ac i a ion = ReLu, max s eps = 20,000) (Bishop, 1995; G on, 2017)
(Mo e de ails a ailable in Sec ion 6.1)
6. Suppo Vec o Machine (ke nel = linea ,
max i e = 5000000) (Joachims, 1998)
7. Suppo Vec o Machine (ke nel = Polynomic,
deg ee = 3, gamma = 0.01, max i e = 5000000) (Joachims, 1998)
8. Radial Basis Func ion (gamma = 0.1, max i e = 5000000)
(Pa k & Sandbe g, 1991)
9. Gaussian Na¨ı e Bayes classi ie (GNB) (Manning e al., 2008)
10. G adien boos ing (GB) (max dep h = 11) (F iedman, 2001) wi h
numbe o ees n = 100 and maximum dep h o ees maxd= 11
21

11. Random o es s (RF) (B eiman, 2001) wi h n = 100 and maxd= 11
12. Decision ee (DT) (Olshen e al., 1984) maxd=n
13. k-nea es neighbo classi ie (1NN) algo i hm (Aha e al., 1991)
wi h k= 1 and using he Euclidean dis ance
14. k-nea es neighbo classi ie (3NN) algo i hm (Aha e al., 1991)
wi h k= 3 and using he Euclidean dis ance
Some o hese classi ie s conside in e ac ions be ween he ea u es, some o h-
e s inco po a e egula iza ion echniques, o ake in o accoun simila i y me ics
be ween he da a. Acco ding o Luengo e al. (2012), classi ie s 13 and 14 a e
lazy classi ie s, while 11 and 12 a e agged in he decision ee class. The es ,
1-10, a e model cons uc ing classi ie s.
6.1. Deep Neu al Ne wo k Classi ie
The i h classi ie used in his expe imen a ion is based on an a i icial neu-
al ne wo k a chi ec u e, a mul i laye pe cep on (MLP), o be mo e speci ic.
These classi ie s a e composed o neu ons, which a e no hing mo e han simple
unc ions ha ecei e mul iple inpu s and p oduce one ou pu . These neu ons
a e o ganized in laye s, which a e connec ed wi h each o he , in a sequen ial
manne . Two neu ons in one laye canno be connec ed while all he nodes in
wo adjacen laye s a e ully in e connec ed ia a ying weigh s.
Finally, he inpu laye mus ha e he same amoun o neu ons as a iables
a da abase has, while he ou pu laye should ha e as many as classes in he
da a, ollowing a so max a chi ec u e. The alues p oduced by hese neu ons
a e o be ea ed as he p obabili ies o an obse a ion in oduced in he inpu
laye belonging o a ce ain class. The aining p ocess consis s on ecompu ing
he p e iously men ioned weigh s based on he knowledge o a desi ed ou pu
gi en a ce ain inpu (Pal & Mi a, 1992; Bishop, 1995; G on, 2017).
In ou case, he s uc u e was composed o wo hidden laye s wi h 2n
3and
n
3nodes espec i ely, being n he amoun o a iables in a da abase.
22
6.2. F-1 Sco e
To measu e he esul s o he classi ica ion me hods, we use he F-1 sco e.
This me ic is compu ed as ollows:
F1= 2 ·1
1
+1
p
whe e s ands o ecall, which is he esul o di iding he labels co ec ly
p edic ed as ue by all he ac ual ue labels. And p ep esen s p ecision, com-
pu ed by di iding he co ec ly p edic ed ue labels by all he labels p edic ed
as ue (including he w ongly p edic ed ones):
=T ueP osi i e
O iginal uelabels , p =T ueP osi i e
Alllabelsp edic edas ue
As i can be seen, his me ic is only in ended o bina y classi ica ion p ob-
lems, while ou benchma k con ains mul i-label DBs. Fo non-bina y classi ica-
ion p oblems, we ha e compu ed he weigh ed a e age o he me ic o each
class, sepa a ely. Mo e de ails abou he implemen a ion o his me hod a e
p o ided a he end o Sec ion 7.1.2.
7. Expe imen s
In his sec ion we conduc an in-dep h empi ical in es iga ion o he ela-
ionship among he MD ypes, IMs and classi ica ion me hods. Among he
ques ions we add ess in his sec ion a e he ollowing:
1. Do di e en MD ypes p oduce di e en e ec s on he p edic ion quali y
o he classi ie s?
2. Wha is he o e all beha io o he IMs when all DBs, MD ypes, and
classi ica ion me hods a e conside ed?
3. Is i possible o iden i y any join e ec o IMs and classi ie s on he sco e
achie ed o di e en DBs? Does his po en ial e ec depend on he ype
o MD?
23
7.1. Expe imen al F amewo k
To s udy he MD-IM/PE-SC ela ion, en non- empo al da abases om he
UCI Machine Lea ning Reposi o y (Lichman, 2013) wi hou missing alues we e
chosen. Using hese DBs, we applied he me hods p oposed o inse ing di e -
en ypes o MD as p esen ed in Sec ion 4. The nex s ep consis ed o impu ing
he missing alues using eigh di e en IM. Finally, he benchma k o new DBs
wi h he inse ed MD was used o apply he di e en classi ie s, which we e
alida ed using s a i ied 5- old c oss alida ion.
To gua an ee a ep esen a i e sample o he possible DBs wi h di e en MD
ypes, we c ea ed, o each o he 10 DBs, i e di e en a ian s, using a 5- old
s a i ied me hodology. We hus sepa a ed he aining and es ing pa s a he
beginning o he algo i hms. We did his in o de o limi he in o ma ion used
by he impu e o he aining and es ing se s sepa a ely and o o ally con ol
he pe cen age o missing da a p esen in each pa i ion. Fo each one o hese 5
a ian s p oduced by he 5- old s a i ied s a egy and each o he 4 MD ypes, 30
di e en DBs con aining MD we e c ea ed. This was done using Algo i hms 1-4.
The e o e, we ob ained 10DBs ×5 olds ×4MD ypes×30ins ances = 6,000DBs
wi h missing alues. F om now on, when we e e o he “benchma k” we e e
o his se o 6,000 DBs, 1,500 o each o he ou MD ypes. Due o he
s ochas ic na u e o he algo i hms used o gene a e he MD, all DBs we e
di e en . E en i hey come om only 10 o iginal DBs, he me hods used o
inse missing alues make each DB unique. Again, no e ha MD was sepa a ely
in oduced on bo h aining and es ing pa i ions, in o de o gua an ee ha
he pe cen age o missingness is exac ly he same.
In he nex phase we ocus on s udying he beha io o he IMs and clas-
si ie s on he benchma k. The eigh di e en IMs desc ibed in Sec ion 5 we e
applied o he benchma k, o ob ain a o al o 6,000DB ×8IM = 48,000 di e -
en , comple e DBs. Finally, all hese da abases we e classi ied by applying he
14 classi ica ion algo i hms desc ibed in Sec ion 6, ob aining a inal ou pu o
48,000DBs/5 olds ×14Classi ie s = 134,400 dis inc sco es. Figu e 1 cap u es
his comple e p ocess o one single DB.
24
Figu e 1: Gene al scheme desc ibing he expe imen al amewo k used o e alua e he in luence o MD ypes, IMs, and classi ica ion algo i hms on
he classi ie pe o mance.
25
IMs Mean Median M . F eq. LVCF In e p. HD MICE EM To al
Mean 0 2 2 16 17 9 9 9 64
Median 20 0 1 19 20 12 11 14 97
M . F eq. 68 53 0 47 49 29 31 32 309
LVCF 36 30 13 0 9 11 12 10 121
In e p. 38 32 16 12 0 12 10 11 131
HD 68 62 50 61 59 0 15 16 331
MICE 68 63 48 65 60 29 0 6 339
EM 65 63 50 66 67 40 28 0 379
To al 363 305 180 286 281 142 116 98 1771
Table 5: Table 4 esul s il e ed by MIV MD ype.
IMs Mean Median M . F eq. LVCF In e p. HD MICE EM To al
Mean 0 2 18 20 29 14 14 11 108
Median 13 0 15 21 31 14 15 13 122
M . F eq 14 6 0 11 22 11 13 11 88
LVCF 16 15 22 0 19 11 11 10 104
In e p. 16 16 16 10 0 11 11 11 91
HD 61 61 59 65 70 0 8 11 335
MICE 61 61 58 65 70 27 0 13 355
EM 61 61 62 68 71 15 15 0 353
To al 242 222 250 260 312 103 87 80 1556
Table 6: Table 4 esul s il e ed by MAR MD ype.
IMs Mean Median M . F eq. LVCF In e p. HD MICE EM To al
Mean 0 4 31 41 54 20 20 14 184
Median 15 0 38 44 53 18 18 15 201
M . F eq. 8 5 0 10 28 13 13 13 90
LVCF 11 11 18 0 27 14 13 13 107
In e p. 14 9 13 9 0 13 12 13 83
HD 56 56 59 59 64 0 12 32 338
MICE 59 58 61 62 64 17 0 29 350
EM 58 55 65 65 67 15 12 0 337
To al 221 198 285 290 357 110 100 129 1690
Table 7: Table 4 esul s il e ed by MuOV MD ype.
32

IMs Mean Median M . F eq. LVCF In e p. HD MICE EM To al
Mean 0 7 45 56 59 19 16 12 214
Median 19 0 51 58 61 24 17 12 242
M . F eq. 12 8 0 44 45 12 12 9 142
LVCF 17 13 26 0 36 19 12 11 134
In e p. 11 11 22 9 0 17 12 10 92
HD 49 47 59 58 64 0 11 21 309
MICE 63 60 67 63 68 40 0 38 399
EM 59 52 70 68 69 39 15 0 372
To al 230 198 340 356 402 170 95 113 1904
Table 8: Table 4 esul s il e ed by MCAR MD ype.
The analysis o Table 4 e eals some ac s abou he beha io o he IMs:
•Th ee clea g oups o IMs can be dis inguished when obse ing o e all
a ings.
•The complex me hods ha e a la gely posi i e sco e when sub ac ing (o−)
om (o+), wi h alues nea 800-1,000.
•The second g oup, composed by he simple me hods, p oduced nega i e
alues when pe o ming he same sub ac ion, by a la ge ma gin. Rega d-
ing solely his g oup, we see ha median impu a ion s ands ou om he
es , as i s ands in a nega i e assessmen o -261, while he o he wo
ob ained a ound -500.
•Finally, he empo al me hods pe o med he wo s , by a la ge ma gin.
Analysis o Tables 5-8 shows how he pe o mance o he IMs is ela ed o
he MD ypes.
•S a ing om he mos biased MD ype, MIV (Table 5), we see how i s
esul s a y om wha could be obse ed in he o e all Table 4. To s a ,
he empo al impu e s seemed o pe o m be e han he simple me hods,
opposed o wha he i s Table 4 showed. Addi ionally, Mos F equen
impu a ion excelled in his ca ego y, as i ob ained almos as good esul s
as he complex me hods, which, as expec ed, opped he anking.
33
•Rega ding MAR (Table 6), mo e simila esul s o he global sco es we e
ob ained. In his case, LVCF was he ou lie , as i could be g ouped wi h
he simple me hods.
•As o MuOV (Table 7), he esul s also ollow a simila pa e n o he
gene al ankings. In his case, a ou h clus e could be dis inguished,
as Mos F equen and LVCF s and conside ably close o each o he , and
ela i ely away om hei o iginal g oups.
•Finally, he absolu ely andom MCAR (Table 8) o e ed conside ably di -
e en esul s. The esul dis ibu ion changed subs an ially, as MICE and
EM emained in he op, bu he o he complex me hod, HD, s ood close
o Median impu a ion a he han he op g oup i is supposed o belong
o. Mean appea s o s and in no-man’s land, while Mos F equen and
LVCF could be g ouped oge he . Ending wi h an isola ed In e pola ion.
The pai -wise compa ison be ween IMs e eals only a pa ial pic u e o he
o e all beha io o he IMs. To u he unde s and he ela ion be ween IM and
MD ype, we designed ano he expe imen o e alua e he ank o he IMs. Fi s ,
all he sco es we e di ided in 40 g oups. Each g oup con ains all classi ica ion
esul s o he combina ions o he 10 o iginal DBs and he 4 MD ypes. Each
g oup comp ises 8 ×14 ×30 = 3360 classi ica ion sco es. In each g oup, he
sco es we e so ed om he highes o he lowes , and spli in o 3 equally sized
g oups: High: he expe imen s wi h he highes sco e; Low: expe imen s wi h
he lowes sco e; Medium: he es o he expe imen s. By inspec ing each g oup
we can de e mine, o each o iginal DB and MD me hod, which combina ion o
IMs and classi ie s we e he mos equen in each o he g oups, allowing us o
de ec high pe o ming IMs, classi ie s, and combina ions o he wo.
In he ollowing analysis, he 10 High g oups (one o each DB) ha ha e a
common MD ype a e joined in a single “supe g oup”. Simila ly, a single Low
supe g oup is c ea ed o each MD ype. We hen compu e he p opo ion o
each IM included in he High and Low supe g oups. This way, we ob ain he
34
IMs ha had he bes pe o mance independen ly o he DB (since hey we e
no me ged un il he inal s ep), and MD ype.
Figu e 2 shows he pola cha s desc ibing he equency o he IMs in he
supe g oups High and Low acco ding o he MD ype. Each sec ion o he
cha is p opo ional o he equency o a gi en IM in he cases whe e he
classi ie s achie ed high ( espec i ely low) sco es. A egula ci cula shape in
hese cha s indica es ha all IMs whe e p esen wi h a simila equency in
high and low quali y cases, hus indica ing no signi ican impac o he IM in
he sco e. Con e sely, mo e i egula shapes indica e ha , o he gi en MD
ype, he e is an in luence o he IMs in he sco es. Fu he mo e, a big gap
be ween he size o he high and low sec ions o a gi en IM clea ly indica es
whe he o no he applica ion o he IM con ibu es o imp o e he F1 sco es.
An analysis o Figu e 2 shows some clea egula i ies be ween di e en MD
ypes, as all h ee complex IMs had a much la ge appea ance a e in he op
g oup a he han in he low one. Mo eo e , all MD con igu a ions also con e ge
in he null e ec i eness o he empo al me hods o non- empo al da a.
Howe e , when i comes o he simple me hods, MIV (Figu e 2a) and he
o he MD ypes (MAR, Figu e 2b, MuOV, Figu e 2c and MCAR, Figu e 2d)
con ain subs an ial di e ences. MIV g an s Mos F equen impu a ion an al-
mos op pe o ming deg ee, p oducing simila esul s o he complex me hods.
Addi ionally, he highe appea ance equency o his me hod, p oduced ha
i s pee s in he simple IM clus e su e ed a dec ease o hei appea ance a io
in he high pe o mance g oup, while he appea ance in he low g oup inc eased
subs an ially.
7.3. In e ac ions be ween MD ypes, impu a ion me hods, and classi ie s
In his sec ion we add ess he ollowing ques ions: Is i possible o iden i y
any join e ec o IMs and classi ie s on he sco e achie ed o di e en DBs?
Does his po en ial e ec depend on he ype o MD ?
To in es iga e his ques ion we use he same supe g oups o High and Low
con igu a ions bu in his case we compu e he equencies o all 8 ×14 = 112
35
mean
High
mean
Low
median
High
median
Low
m. eq.
High
m. eq.
Low
l c
High
l c
Low
in e .
High
in e .
Low
HD
High HD
Low mice
High
mice
Low
EM
High
EM
Low
(a) MIV
mean
High
mean
Low
median
High
median
Low
m. eq.
High
m. eq.
Low
l c
High
l c
Low
in e .
High
in e .
Low
HD
High HD
Low mice
High
mice
Low
EM
High
EM
Low
(b) MAR
mean
High
mean
Low
median
High
median
Low
m. eq.
High
m. eq.
Low
l c
High
l c
Low
in e .
High
in e .
Low
HD
High HD
Low mice
High
mice
Low
EM
High
EM
Low
(c) MuOV
mean
High
mean
Low
median
High
median
Low
m. eq.
High
m. eq.
Low
l c
High
l c
Low
in e .
High
in e .
Low
HD
High HD
Low mice
High
mice
Low
EM
High
EM
Low
(d) MCAR
Figu e 2: F equency o he IMs in he con igu a ion wi h highes (High) and lowes (Low)
classi ica ion sco e.
36
SVM−3
RBFN
3−NN
1−NN
LDA
QDA
C−SVM
Nai eB
CART
Reg .L2
Reg .L1
G ad.B.
Rnd.F .
DNN
Mean
Median
M.F eq.
LVCF
In e p.
HD
MICE
EM
0
50
100
150
200
250
Classi ie
Impu e
50
100
150
200
250
(a) MIV
RBFN
SVM−3
3−NN
1−NN
QDA
LDA
Nai eB
CART
C−SVM
Reg .L2
G ad.B.
Rnd.F .
Reg .L1
DNN
Mean
Median
M.F eq.
LVCF
In e p.
HD
MICE
EM
0
50
100
150
200
250
Classi ie
Impu e
50
100
150
200
250
(b) MAR
3−NN
RBFN
SVM−3
1−NN
Nai eB
CART
LDA
QDA
C−SVM
Reg .L2
G ad.B.
Rnd.F .
Reg .L1
DNN
Mean
Median
M.F eq.
LVCF
In e p.
HD
MICE
EM
0
50
100
150
200
250
Classi ie
Impu e
50
100
150
200
250
(c) MuOV
3−NN
SVM−3
RBFN
1−NN
CART
QDA
Nai eB
LDA
Reg .L2
C−SVM
Rnd.F .
Reg .L1
G ad.B.
DNN
Mean
Median
M.F eq.
LVCF
In e p.
HD
MICE
EM
0
50
100
150
200
250
Classi ie
Impu e
50
100
150
200
250
(d) MCAR
Figu e 3: Amoun o IM-Classi ie pai s p esen in he High sco e sec ion.
37

pai s o IMs and classi ie s. This in o ma ion is shown in Figu e 3, in which
alues a e anked by classi ica ion pe o mance (x axis).
F om Figu e 3a we can see how he pa ial conclusions abou he posi i e
ela ionship be ween his MDT and Mos F equen impu a ion a e co obo a ed.
Fo an ex ensi e se o classi ie s, his impu e has p o ided esul s in a simila
ange o hose ob ained by he complex IMs. Besides, i is no iceable ha he
esul s a e mo e balanced be ween classi ie s han in he es o igu es.
Figu e 3b ep esen s he equency o he pai s conside ing only MAR MD
ype. The i s di e ence obse able be ween his and he p e ious Figu e 3a is
he scale i is d awn. In his case, he op classi ie appea ed mo e equen ly,
which deno es a wo se ela ion be ween he DNN and MIV compa ed o MAR.
Addi ionally, in his case Mos F equen impu a ion did no s and ou om he
me hods o i s same ca ego y. This causes a be e isualiza ion o he classi ie s
ha equi e a complex impu a ion o p oducing an op imal pe o mance. Good
examples o his collec ion a e L2-penalized Reg ession, o C-SVM. On he con-
a y, we also ha e some o he classi ie s ha a e una ec ed by he IM applied.
Random Fo es and G adien Boos ing classi ie s a e excellen exempla s o his
se .
Resul s il e ed by MuOV MD ype a e p esen ed by Figu e 3c. Again,
we dis inguish be ween classi ie s ha p oduce simila esul s ega dless he
impu e ha p ecedes i . Howe e , his ime mo e con as can be ound in his
aspec . The di e ence be ween he simple and complex IMs o LDA and QDA
classi ie suppo s his dis inc ion be ween SCs.
Finally, Figu e 3d, ep esen ing MCAR, shows an almos non exis ing se o
mid ield classi ie s. The di e ences be ween con iguous classi ie s in he p e ious
igu es we e quasi-cons an . This ins ance, howe e , p esen s a conside able gap
in e ms o high g oup appea ance a e be ween Na¨ı e Bayes and LDA classi ie ,
specially o he complex me hods.
Conside ing all ou igu es, we can conclude ha a s ong ela ion be ween
IM and MD ype exis s (MIV and Mos F equen impu a ion, o example).
Addi ionally, we dis inguish be ween classi ie s ha need quali y impu a ion
38
o p oduce a op pe o mance (Deep Neu al Ne wo k and Logis ic Reg ession
classi ie s) and o he s ha do no (G adien Boos ing and Random Fo es Clas-
si ie s). The e seems o be a ce ain egula i y on he classi ie s anking, as
he e is no signi ican a ia ion be ween any wo MD ypes in he classi ie s axis
(x axis) in Figu e 3.
7.4. Seconda y expe imen a ion
Fo his seconda y expe imen a ion, we add ess he ques ion: How does he
amoun o MD a ec he pe o mances o bo h classi ie s and impu e s?
To answe his ques ion, we ha e conduc ed a simila se o expe imen as
in he p e ious expe imen a ion Sec ion 7.1. As explained in Sec ion 7.1.1, we
selec ed wo da abases, and inse ed MCAR MD wi h a ying pe cen ages (7-
42%) o u he impu a ion. These da abases we e subjec ed o 5 old c oss
alida ion using wo classi ie s and F1 sco e as a me ic. To isually in e p e
hese esul s, Figu e 4 has been c ea ed. These hea maps show he mean o all
F1 sco es compu ed in all 30 uns. Each sub igu e p esen s he da a ob ained
om a ce ain da abase, and a ce ain classi ie . Acco dingly, Figu e 4a shows
he mean F1 sco e o each pai o impu e and MD pe cen age ob ained in he
Lea da abase, wi h he G adien Boos ing classi ie . In he igu es a da k colo
ep esen s a be e F1 sco e, and, he e o e, a be e pe o mance.
F om his Figu e 4 we can lis he ollowing indings:
•As expec ed, he mo e MD, he wo se pe o mance o e ed by he classi-
ie s.
•Again ollowing expec a ions, he classi ie mo e sensi i e o impu a ion
quali y (Reg ession) showed an inc eased need o quali y impu a ion as
he amoun o MD ose.
•G adien Boos ing classi ie was selec ed due o i s ela i ely educed need
o a good impu a ion o o e ing a good pe o mance. Howe e , as he
MD inc eased in a DB, his changes, as complex impu e s (i.e., MICE)
p oduce mo e s able esul s han he es .
39
7 14 21 28 35 42
Pe cen ages
EM
MICE
HD
In e p.
LVCF
M .F eq.
Median
Mean
Impu a ionMe hods
MeanF1sco es
0.18
0.24
0.30
0.36
0.42
0.48
(a) Lea DB, G adien Boos ing classi ie .
7 14 21 28 35 42
Pe cen ages
EM
MICE
HD
In e p.
LVCF
M .F eq.
Median
Mean
Impu a ionMe hods
MeanF1sco es
0.12
0.15
0.18
0.21
0.24
0.27
(b) Lea DB, Logis ic Reg ession classi ie .
7 14 21 28 35 42
Pe cen ages
EM
MICE
HD
In e p.
LVCF
M .F eq.
Median
Mean
Impu a ionMe hods
MeanF1sco es
0.52
0.56
0.60
0.64
0.68
(c) Vehicle DB, G adien Boos ing classi-
ie .
7 14 21 28 35 42
Pe cen ages
EM
MICE
HD
In e p.
LVCF
M .F eq.
Median
Mean
Impu a ionMe hods
MeanF1sco es
0.48
0.54
0.60
0.66
0.72
(d) Vehicle DB, Logis ic Reg ession classi-
ie classi ie .
Figu e 4: Mean F1 sco es o he 30 uns o he expe imen a ion.
40
•Compa ing he wo pai s gene a ed om he same da abase, we see how
he pa e ns a e e y simila , which is indica i e o a s ong dependency
o he esul s on he p oblem.
•O e all, complex impu e s (specially MICE, bu also Ho Deck and EM,
in a lesse ex en ) keep epo ing accep able esul s as MD inc eases in a
DB, while he accu acy o o he me hods (i.e., Median impu a ion) seems
o be keep up he pace.
8. Conclusions
Al hough i is gene ally accep ed ha di e en pa e ns o MD can p oduce
a di e en e ec on he da a, and some esea che s ha e sugges ed ha hese
pa e ns should be aken in o accoun a he ime o selec ing he IM, an in-
dep h analysis o his ques ion had no been p e iously app oached. This may
be due o he complexi y o de ining an expe imen al amewo k ha acili a es
in es iga ing all he ac o s in ol ed. In his pape we ha e p oposed a way o
s udy he in e ac ions be ween algo i hms dealing wi h missing da a, supe ised
classi ica ion me hods and missing da a i sel , aking in o accoun i s di e en
con igu a ions (MCAR, MIV, MAR and MuOV).
A key cha ac e is ic o ou app oach has been he combina ion o o iginal
DBs wi hou MD wi h specially de ised me hods o injec ing MD consis en
wi h he desi ed MD ypes: MCAR, MIV, MAR and MuOV. Ten da ase s wi h
no missing alues we e chosen, and we e cloned 30 imes. Then he ou di e -
en MD pa e ns we e applied o hem, sepa a ely. The esul ing benchma k
comp ising 1,200 da ase s was hen used o apply he 8 di e en IMs. Fou een
classi ica ion algo i hms we e applied o he impu ed DBs.
S a is ical signi ican di e ences we e de ec ed in he pe o mance o he
classi ie s when using di e en IMs. The pa e ns o di e ences we e dependen
on he MD ype. Resul s showed signi ican ela ion be ween IM complexi y and
classi ica ion algo i hms, since MICE (which implemen s Mul iple Impu a ion)
and Ho Deck (also Expec a ion-Maximiza ion, o a lesse ex en ) we e mo e
41
Pal, S. K., & Mi a, S. (1992). Mul ilaye pe cep on, uzzy se s, and classi ica-
ion. IEEE T ansac ions on neu al ne wo ks,3, 683–697.
Pa k, J., & Sandbe g, I. W. (1991). Uni e sal app oxima ion using adial-basis-
unc ion ne wo ks. Neu al compu a ion,3, 246–257.
Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O.,
Blondel, M., P e enho e , P., Weiss, R., & Dubou g, V. (2011). Sciki -lea n:
Machine lea ning in Py hon. The Jou nal o Machine Lea ning Resea ch,12,
2825–2830.
Sil a, P. F., Ma cal, A. R., & da Sil a, R. M. A. (2013). E alua ion o ea-
u es o lea disc imina ion. In In e na ional Con e ence Image Analysis and
Recogni ion (pp. 197–204). Sp inge .
Song, Q., Sheppe d, M., Chen, X., & Liu, J. (2008). Can k-NN impu a ion
imp o e he pe o mance o C4. 5 wi h small so wa e p ojec da a se s? A
compa a i e e alua ion. Jou nal o Sys ems and so wa e,81, 2361–2370.
Tan, P.-N., S einbach, M., Kuma , V., & o he s (2006). In oduc ion o da a
mining olume 1. Pea son Addison Wesley Bos on.
Twala, B. (2009). An empi ical compa ison o echniques o handling incom-
ple e da a using decision ees. Applied A i icial In elligence,23 , 373–405.
Van Buu en, S. (2012). Flexible impu a ion o missing da a. CRC p ess.
Wong, A. K., & Chiu, D. K. (1987). Syn hesizing s a is ical knowledge om
incomple e mixed-mode da a. IEEE T ansac ions on Pa e n Analysis and
Machine In elligence, .
Yu, H.-F., Huang, F.-L., & Lin, C.-J. (2011). Dual coo dina e descen me hods
o logis ic eg ession and maximum en opy models. Machine Lea ning,85,
41–75.
Yuan, Y. C. (2010). Mul iple impu a ion o missing da a: Concep s and new
de elopmen (Ve sion 9.0). SAS Ins i u e Inc, Rock ille, MD,49.
48

Ziba, M., Tomczak, J. M., Lubicz, M., & wi ek, J. (2014). Boos ed SVM o
ex ac ing ules om imbalanced da a in applica ion o p edic ion o he pos -
ope a i e li e expec ancy in he lung cance pa ien s. Applied so compu ing,
14, 99–108.
49