Recei ed 8 Oc obe 2023, accep ed 22 Oc obe 2023, da e o publica ion 26 Oc obe 2023, da e o cu en e sion 6 No embe 2023.
Digi al Objec Iden i ie 10.1109/ACCESS.2023.3327870
Towa d an Op imal and S uc u ed Fea u e
Subse Selec ion o Mul i-Ta ge Reg ession
Using Gene ic Algo i hm
FARRUKH HASAN SYED1, MUHAMMAD ATIF TAHIR 1,
JAROSLAV FRNDA 2,3, (Senio Membe , IEEE),
MUHAMMAD RAFI 1, MUHAMMAD SHAHID ANWAR 4,
AND JAN NEDOMA 3, (Senio Membe , IEEE)
1School o Compu ing, Depa men o Compu e Science, Na ional Uni e si y o Compu e and Eme ging Sciences, Islamabad 44000, Pakis an
2Depa men o Quan i a i e Me hods and Economic In o ma ics, Facul y o Ope a ion and Economics o T anspo and Communica ions, Uni e si y o Žilina,
01026 Žlina, Slo akia
3Depa men o Telecommunica ions, Facul y o Elec ical Enginee ing and Compu e Science, VSB—Technical Uni e si y o Os a a, 70800 Os a a,
Czech Republic
4Depa men o AI and So wa e, Gachon Uni e si y, Seongnam-si 13120, Sou h Ko ea
Co esponding au ho s: Muhammad Ra i ([email p o ec ed]) and Muhammad Shahid Anwa (shahidanwa [email protected] )
This wo k was suppo ed in pa by he Minis y o Educa ion, You h and Spo s o he Czech Republic conduc ed by he VSB—Technical
Uni e si y o Os a a, Czechia, unde G an 2023/39 and G an 2023/42; and in pa by he Eu opean Union wi hin he REFRESH
P ojec —Resea ch Excellence Fo Region Sus ainabili y and High-Tech Indus ies, o he Eu opean Jus T ansi ion Fund’’ unde
G an ‘‘CZ.10.03.01/00/22003/0000048.’’
ABSTRACT Mul i Ta ge Reg ession (MTR) is a machine lea ning me hod ha simul aneously p edic s
mul iple eal- alued ou pu s using a se o inpu a iables. A lo o eme ging applica ions ha can be mapped
o his class o p oblem. In MTR me hod one o he c i ical aspec is o handle s uc u al in o ma ion like
ins ance and a ge co ela ion. MTR algo i hms a emp o exploi hese in e dependences when building a
model. This esul s in inc eased model complexi ies, which in u n, educe he in e p e abili y o he model
h ough manual analysis o he esul . Howe e , da a d i en eal-wo ld applica ions o en equi e models ha
can be used o analyze and imp o e eal-wo ld wo k lows. Le e aging dimensionali y educ ion echniques
can educe model complexi y while e aining he pe o mance and boos in e p e abili y. This esea ch
p oposes mul iple ea u e subse al e na i es o MTR using gene ic algo i hm, and p o ides a compa ison
o he di e en ea u e subse selec ion al e na i es in conjunc ion wi h MTR algo i hms. We p oposed
a gene ic algo i hm based ea u e subse selec ion wi h all a ge s and wi h indi idual a ge keeping he
s uc u al in o ma ion in ac in he selec ion p ocess. Expe imen s a e pe o med on eal wo ld benchma ked
MTR da a se s and he esul s indica e ha a signi ican imp o emen in pe o mance can be ob ained wi h
compa a i ely simple MTR models by u ilizing op imal and s uc u ed ea u e selec ion.
INDEX TERMS Mul i- a ge eg ession, ea u e selec ion, gene ic algo i hm, single a ge , mul iple
objec i es.
I. INTRODUCTION
Mul i- a ge Reg ession (MTR) has been ecei ing inc easing
a en ion in he esea ch communi y [1],[2],[3]. Wi h
he exponen ial inc ease in au oma ion and da a collec ion,
da a-d i en decision-making is becoming a no m in almos
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Wen ao Fan .
all a eas o li e. Inc easing eal-wo ld si ua ions a e being
iden i ied whe e he objec i e is o make in e ence abou
se e al a ge a iables a he same ime. Reg ession asks
in gene al and mul i- a ge eg ession asks in pa icula
ha e a as numbe o applica ions anging om economics,
business and inance o heal h, enginee ing and space
esea ch, e c. [4],[5],[6]. Since MTR p oblems deal
wi h mul iple a ge a iables, la ge sys ems wi h complex
121966
2023 The Au ho s. This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0 License.
Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ VOLUME 11, 2023
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
FIGURE 1. Classi ica ion o sola la e in ensi ies ([7]).
in e ela ed componen s can be modeled using mul i- a ge
eg ession algo i hms.
Fo ins ance, Figu e 1(a) shows sola la es o di e en
in ensi ies while Figu e 1(b) shows hei i adiance in ensi-
ies. The in ensi ies o he emissions a e used o ca ego ize he
sola la es. A sola e en can esul in emissions o mul iple
in ensi ies, and is he e o e a classical mul i- a ge p oblem.
The al eady challenging p oblem o mul i- a ge p edic ion
and analysis is u he agg a a ed by (i) complex in e depen-
dencies be ween he a ge a iables, and (ii) issues a ising
om high dimensionali y o he ea u e space, popula ly
e e ed o as he ‘‘Cu se o Dimensionali y’’. To he bes o
ou knowledge, mos o he esea ch un il now has ocused
on he a ge in e dependency modeling p oblem, bu e y
li le wo k has been done on add essing he dimensionali y
issues o imp o e model pe o mance. This esea ch aims
o in es iga e ea u e selec ion as a iable pe o mance
imp o emen app oach o mul i- a ge eg ession p oblems.
This esea ch is ela ed o i s semi-supe ised coun e pa
p oposed in [8].
The p oblem o cu se o dimensionali y is inhe en in all
machine lea ning asks. I collec i ely e e s o he p oblems
which a ise in modeling o high dimension da a [9],[10].
As a esul , models become complex [11]. This complexi y
leads o low p edic i e accu acy and in e p e abili y, and an
inc eased un ime. This p oblem is compounded when he
numbe o a ailable examples o lea ning is ew, which
is usually he case in p ac ical cases [12]. To sol e hese
p oblems, a numbe o echniques exis , which gene ally do
so by educing he dimensions o he da a. Fea u e selec ion
echniques [13] help ule ou ea u es which a e i ele an
o edundan . The o iginal ea u es a e no ans o med o
modi ied in any way; hus, he unde s andabili y and he
con ex is p ese ed.
The issue wi h ea u e selec ion is o decide which
ea u es o keep and which ones o d op om he se
o a ailable ea u es. Gi en he la ge s a e space usually
p esen ed by he ea u es, an exhaus i e sea ch is almos
always ou o ques ion. One me hod ha has been used
ex ensi ely o he pu pose o ea u e selec ion is he
use o E olu iona y Algo i hms which a e me aheu is ic
algo i hms inspi ed by he e olu iona y p ocesses in na u e
[14],[15]. One commonly used e olu iona y algo i hm is he
Gene ic Algo i hm (GA) [16]. GAs wi h hei ope a o s o
c osso e and mu a ion, allow a as e explo a ion o di e en
ea u e combina ions in an e ec i e manne . They a e mos
commonly used algo i hms o op imiza ion p oblems, gi en
hei lexible na u e, ease o adap a ion and a iabili y o
pa ame e uning possibili ies [17],[18].
The p ocess o selec ing ea u es o mul i- a iable
lea ning aces added complica ions since he e a e mul iple
dependen a iables. Fo example, does he e exis a common
ea u e subse which can be used o all a ge a iables? O ,
is using a di e en subse o ea u es o each a ge he way o
go? I a common ea u e subse is used, dec easing e o o
one a ge may inc ease he e o o some o he a ge and
ice e sa. The e a e a huge numbe o eal-wo ld analy ics
and op imiza ion p oblems ha ace his issue [19],[20].
In such a case, e en i a e age e o is used as an e alua ion
me ic, he e o in some a ge s migh educe while o some
a ge s, he e o may inc ease o an unaccep able le el.
Keeping he abo e ques ions in mind, his pape in es i-
ga es he e ec o ea u e selec ion on mul i- a ge eg ession
asks using gene ic algo i hms. The objec i e o his esea ch
is as ollows:
OI: To show ha by educing ea u es using ea u e
selec ion, a pa o be e esul s can be ob ained
compa ed o exis ing MTR me hods.
OII: To alle ia e he issues collec i ely e e ed o as he
cu se o dimensionali y issues o high dimension MTR
p oblems.
OIII: To imp o e in e p e abili y and pe o mance o exis -
ing MTR models.
Fo any mul i- a ge ask, he e a e wo basic app oaches
a ailable.
1) To app oach he p oblem as independen modeling
p oblems o each o he a ge s.
2) To use mul i- a ge eg ession algo i hms ha can also
model he in e ac ions be ween he a ge s in o de o
explo e in e a ge dependencies.
Based on he abo e men ioned app oaches, we p opose wo
a ia ions as pa o his esea ch. Fo he i s app oach,
we will be using he baseline app oach called Single
Ta ge (ST) [21], while o he second app oach, we will
be using wo me hods mul i- a ge eg ession algo i hms,
S acked Single Ta ge (SST) and Ensemble o Reg esso
Chains (ERC) [22]. Al hough a numbe o o he mul i- a ge
eg ession algo i hms exis , we ha e chosen hese h ee
algo i hms due o he ac ha he same base eg esso can be
used o all h ee algo i hms. Thus only he e ec o ea u e
selec ion can be isola ed and in es iga ed.
VOLUME 11, 2023 121967
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
Based on he abo e men ioned pa ame e s, he ollowing
s eps shown in igu e 2 o iden i y and p opose ele an
solu ion:
FIGURE 2. S udy p ocess lowcha .
The es o he pape is o ganized as ollows: Sec ion II
p esen s a discussion o he exis ing easea ch in he a ea o
mul i- a ge eg ession and ea u e selec ion. I also gi e a
b ie o e iew o using GA as a ool o ea u e selec ion.
Sec ion III desc ibes he de ails o he p oposed me hod,
along wi h a desc ip ion o he algo i hms and echniques
used in he expe imen s. I also lis s he da ase s used in
he expe imen s. Sec ion IV p esen s comp ehensi e esul s
o he expe imen s and a discussion on he ex en o which
he objec i es o he esea ch ha e been achie ed. Sec ion V
ends he pape wi h a discussion on he ad an ages and
disad an ages o he p oposed app oach and some u u e
di ec ions o he esea ch.
II. RELATED WORK
Mul i- a ge eg ession [23],[24] has gained ocus ecen ly
wi h many p ac ical applica ions and p oblems being
unco e ed which equi e p edic ion o alues o mul iple
a ge a iables ha ing con inuous ou pu s. Some in e es ing
examples include p edic ing cus ome p epaid op-ups [25],
p edic ion o d ug e icacy [26], p edic ion o lou
quali y [27], human b ain ac i i y p edic ion [28] whe e
he ac i i y o adjacen a eas o he b ain need o be
mapped simul aneously o a be e unde s anding o b ain
ac i i y, ecosys em s udy such as in [29] whe e eadings
o a ious a ibu es o ege a ion a a ce ain loca ion
need o be s udied oge he , in epidemiological s udies,
by s udying mul iple a iables o a disease o diseases, and
hei in e dependence [30], and many mo e.
Up o a ce ain ex en , he echniques de eloped o
Mul ilabel Classi ica ion (MLC) can be adap ed o MTR
p oblems as well [21], while some me hods, o example,
Mul i-Objec i e Decision T ee [31], Mul i-Objec i e Ran-
dom Fo es (MORF) [32], Ensembles o Mul i-Objec i e
Decision T ees [33] ha e been de eloped speci ically o
Mul i- a ge Reg ession (MTR).
Al hough a lo o wo k has been done on ea u e selec ion
o adi ional machine lea ning algo i hms, no much wo k
has been done o imp o e MTR algo i hm pe o mance
using ea u e selec ion and mos wo k has ocused on model
de elopmen . Table 1p esen s a summa y demons a ing his
concep .
TABLE 1. MTR model de elopmen .
As can be om able 1, mos g ound-b eaking MTR
pape s discuss algo i hm de elopmen and adap ing MTC
algo i hms o MTR. Fo all machine lea ning asks, he
quali y o ea u es plays he mos impo an ole. E en
he pe o mance o he mos powe ul algo i hms su e s
i a sui able se o ea u es is no used. In his ega d,
ea u e selec ion me hods play a e y impo an ole. Fea u e
selec ion can help in no only imp o ing he esul s by
educing he p oblems associa ed wi h cu se o dimension-
ali y and o e i ing, bu also in dec easing he un ime o
he model [34]. Fea u e selec ion app oaches a e gene ally
di ided in o h ee ca ego ies: Fil e app oach, W appe
app oach and Embedded app oach. E olu iona y algo i hms,
and especially GA and i s a ia ions ha e ound hei way
in o a numbe o ea u e selec ion applica ions and ha e been
used since long. They a e used as w appe me hods, which
means hey use he unde lying machine lea ning algo i hm
o e alua e subse pe o mance. GA wi h i s c osso e and
mu a ion ea u es o e s a good balance o exploi a ion and
explo a ion in o de o sea ch h ough a la ge space o ea u e
combina ions ha make up he sea ch space.
III. METHODOLOGY
A. DATASETS
Fo expe imen s and e alua ion, expe imen s we e pe o med
using benchma k mul i- a ge eg ession da ase s.1
1h p://mulan.sou ce o ge.ne /da ase s-m .h ml
121968 VOLUME 11, 2023
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
TABLE 2. Mul i- a ge Reg ession Da ase s. ddeno es he numbe o
dimensions o ea u es, while mdeno es he numbe o a ge s o each
da a se .
The da ase s a e a ailable o download publicly. A sum-
ma y o he da ase s is gi en in Table 2.
B. EVALUATION MEASURE
Fo mul i- a ge eg ession, Rela i e Roo Mean Squa ed
E o (RRMSE) [22] is used as he e alua ion measu e.
RRMSE is de ined as:
RRMSE =
u
u
P(x,y)ϵD es (ˆyj−yj)2
P(x,y)ϵD es (¯
Yj−yj)2(1)
In he abo e gi en equa ion (1),yjis he g ound u h alue,
ˆyjis he p edic ed alue and ¯
Yjis he mean alue o he a ge
a iable Yjob ained om he aining se .
C. BENCHMARK MTR METHODS
A b ie desc ip ion o he algo i hms is gi en below:
•Single Ta ge : Single Ta ge me hod is he baseline
me hod o mul i- a ge eg ession and igno es a ge
in e dependencies. I is inspi ed by i s mul ilabel
coun e pa called Bina y Rele ance [42]. The idea is o
decompose he ‘m’ a ge p oblem in o ‘m’ independen
single a ge p oblems and hen applying any sui able
adi ional eg ession algo i hm.
•S acked Single Ta ge : In SST, he aining phase has
wo s ages. In he i s s age, ‘m’ independen models
a e ained, one o each a ge (whe e ‘m’ is he numbe
o a ge a iables). In he second s age, he ‘m’ models
a e used o p edic he alues o he ‘m’ a ge s. Then,
he inpu ea u es a e augmen ed using hese p edic ions.
Now, a second se o ‘m’ models, i.e. one o each a ge ,
is lea ned using he augmen ed ea u e space. A es
ime, he models ained in he i s phase a e i s used o
ou pu ‘m’ p edic ions, and based on hese p edic ions,
he ea u e space is augmen ed and inpu o he second
phase om which he inal ou pu is ob ained.
•Ensemble o Reg esso Chains: ERC is based on
he Reg esso Chains [23] concep and consis s o
an ensemble o Reg esso Chains (RC). A chain o
a ge a iables in cons uc ed by o de ing he a iables
andomly. A sepa a e eg ession model is de eloped o
each a ge by using a chain o a ge s o augmen he
ea u e space. Since using only one chain can cause
bias due o chaining o de , in ERC an ensemble o such
chains is used. Final p edic ion is ob ained by a e aging
he p edic ions om each chain. Since a p edic ion
ime, he ac ual alues o he a ge s a e no known, he
p edic ed alues a e ob ained simila o SST.
D. WRAPPER BASED FEATURE SELECTION APPROACH
Fo any ea u e selec ion echnique, he gene al p ocess is
ela i ely he same as shown in Figu e 3:
FIGURE 3. The p ocess o ea u e selec ion.
Since he objec i e o his esea ch was o in es iga e
ea u e selec ion in conjunc ion wi h MTR algo i hms,
he e o e, a w appe app oach was used. The eason o
including MTR algo i hms was ha each algo i hm di e s
in how he a ge s a e included in he inal model, so using
he algo i hms hemsel es as he e alua ion unc ion will s ill
allow any use ul a ge in e ac ion o be cap u ed. The e o e,
he powe o MTR algo i hms will be e ained wi h he added
bonus o a educed ea u e coun .
E. FEATURE SELECTION ALGORITHM
Gene ic algo i hm (GA) uses he ope a o s o selec ion,
c osso e and mu a ion. Each o he ope a o s pe o m
hei espec i e ope a ions on a se o candida e solu ions
( ea u e subse s) called ‘popula ion’. The selec ion ope a o
selec s he candida e solu ions o c osso e based on
hei pe o mance. The c osso e ope a o combines wo
(o some imes mo e) candida e solu ions o p oduce new
candida e solu ions. The in ui ion is ha he child solu ions
om good pa en solu ions will combine he s eng hs o he
pa en solu ions. Since con inuous c osso e s can esul in
con e gence o he popula ion owa ds one op imum esul ,
possibly a local op imum, as simila solu ions a e epea edly
combined, mu a ion ope a o is used o add di e si y o he
popula ion by andomly changing some indi iduals p esen
in he popula ion. Mu a ion allows he solu ions o escape
om local op ima by c ea ing solu ions ha a e di e en
om he cu en popula ion. GA allow as e explo a ion o
he s a e space along wi h ele an exploi a ion o p omising
solu ions. Ano he ad an age o GA is ha hese beha iou s
can be modi ied by uning o he GA pa ame e s which a e
gene ally simple, and hus, do no equi e ex ensi e addi ional
knowledge. GA is also domain independen and can be
applied o p oblems o da a se i espec i e o hei domains.
Howe e , nume ous modi ica ions a e also possible in case
VOLUME 11, 2023 121969
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
domain knowledge o heu is ics need o be inco po a ed o
ce ain si ua ions.
F. METHODOLOGY
The p oposed me hod uses he Single Ta ge (ST) me hod
o ind sui able ea u e subse s o each a ge . Fo GA-FS-
Ta ge , a ea u e selec ion o each a ge is ound indepen-
den ly by conside ing he m- a ge mul i- a ge eg ession
ask as msepa a e single a ge eg ession asks. Fo GA-FS-
MTR, a common ea u e subse is ound by inding a ea u e
subse which minimizes he a e age e o o all he a ge s.
A s ep by s ep explana ion o he whole p ocess is gi en
below:
•Encoding and Popula ion Ini ializa ion: Fea u e sub-
se s a e encoded in he o m o s ings o 0’s and 1’s. The
1’s deno e he p esence o a ea u e in a subse , while 0’s
deno e i s absence. A numbe o candida e solu ions o
ea u e subse s in he o m o s ings o 0’s and 1’s a e
gene a ed andomly.
•Fi ness Calcula ion: Each candida e solu ion is passed
o he eg esso and he e o is ob ained. In he case
o GA-FS-Ta ge , he candida e solu ion co esponds o
one a ge a a ime, while in he case o GA-FS-MTR,
he candida e solu ion co esponds o all he a ge s
oge he . The e o e, o he o me case, Rela i e Roo
Mean Squa ed E o (RRMSE) o one a ge ha is
unde conside a ion is conside ed as he i ness o he
candida e solu ion, while o he la e case, he A e age
Rela i e Roo Mean Squa ed E o s (ARRMSE) o all
a ge s is conside ed as he i ness o he candida e
solu ion.
•Selec ion: The bes candida e solu ions i.e. he ea u e
subse s wi h he lowes e o a e selec ed o u he
p ocessing.
•C osso e : C osso e ope a o is used o gene a e new
candida e solu ions by combining any wo candida e
solu ions as depic ed in Figu e 4.
FIGURE 4. Single poin c osso e .
The bi s om Pa en 1 and Pa en 2 and combined using
he c osso e ope a o o ob ain Child 1 and Child 2.
The c osso e p obabili y is dependen on he i ness
o solu ions. Candida e solu ions wi h highe i ness a e
mo e likely o be selec ed as pa en s o c osso e .
•Mu a ion: Mu a ion ope a o is used o main ain a le el
o di e si y in he solu ions. In his case, his change
consis ed o changing a ‘0’ in he candida e solu ion
s ing o a ‘1’ o ice e sa as depic ed in Figu e 5.
FIGURE 5. One bi mu a ion.
A a iable a e o mu a ion was used o hese expe -
imen s. Fo he i s ew gene a ions, he mu a ion
a e was kep e y high, as mo e explo a ion was
equi ed. The mu a ion a e was dec eased a e a ew
gene a ions, and hen dec eased u he a e a ew mo e.
To compensa e o he high a e o mu a ion and a oid
mu a ion o good solu ions, he mu a ion p ocess was se
up such ha he bes solu ions in he popula ion we e
ne e mu a ed.
•Gene a ion: This cycle con inues un il he s opping
c i e ia is ob ained. Fo hese expe imen s, he e we e
wo s opping c i e ia: ei he i a be e solu ion was
no ound o wen y consecu i e gene a ions, o a
p ede ined numbe o gene a ions was eached.
Complexi y Analysis:
•I M is he dimension o he da a se , hen o gene a e
N indi iduals, he ime complexi y would be close o
O(M*N).
•I F is he ime aken o e alua e a i ness unc ion, hen
he ime complexi y o popula ion would be O(F*N)
•Time complexi y o c osso e O(L), mu a ion O(MU)
•I G is he numbe o gene a ions, hen he o al ime
complexi y o he algo i hm may be deno ed as:
Complexi yGA =G∗O(M∗N∗F∗L∗MU) (2)
Equa ion 2can be used o ob ain a ough es ima e o he
unning ime o he p oposed app oach as well.
Gene ic algo i hms o en p o ide he mos ad an age o
op imiza ion p oblems whe e he popula ion size is ela i ely
small compa ed o he size o he p oblem. The s a e space
o ea u e selec ion can be deno ed as: S=2m, whe e mis
he numbe o ea u es. In he case o ea u e selec ion, he
numbe o ea u es can exceed hund eds o housands which
esul s in a huge s a e space and he numbe o popula ion (N)
is ex emely small as compa ed o (S).
Since one o he objec i es o using GA o ea u e
selec ion was o explo e la ge s a e space o ea u es quickly,
he e o e, apa om mu a ion, we ha e also in oduced one
mo e mechanism o inc ease he p opo ion o s a e space
explo a ion. I he numbe o candida e solu ion goes below a
h eshold due o he selec ion p ocess, new andom candida e
solu ions a e gene a ed and added o he popula ion. This
allows o he numbe o solu ions o emain a a ce ain
numbe , while also in oducing di e se candida e solu ions
in o he popula ion.
Two a ia ions o p oposed app oach ha e been e alua ed
o hese expe imen s. Figu es 6(a) and 6(b) show he
121970 VOLUME 11, 2023

F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
p oposed me hods g aphically and hei desc ip ion is gi en
below:
FIGURE 6. G aphical ep esen a ions o he p oposed me hods.
1) GENETIC ALGORITHM FEATURE SELECTION
MULTI-TARGET (GA-FS-MTR)
The p oposed me hod GA-FS-MTR wo ks by inding a
common ea u e subse which educes he a e age e o o
all a ge s. Fo hese expe imen s, he ST app oach was
used o ind he common ea u e subse which was hen
e alua ed using di e en MTR algo i hms. A popula ion o
candida e solu ions was ini ialized andomly. One by one,
each ea u e subse was passed o he ST algo i hm. Since
ST e alua es he ea u es pe a ge , he ob ained esul s we e
a e aged o ind he ARRMSE. The ea u e subse s wi h he
lowes ARRMSE we e selec ed o u he p ocessing. A e a
p econ igu ed numbe o gene a ions was eached, he subse
wi h he lowes ARRMSE was conside ed as he bes one.
Finally, he ob ained ea u e subse s we e used in conjunc ion
wi h he MTR algo i hms; Single Ta ge (GA-FS-MTR(ST)),
S acked Single Ta ge (GA-FS-MTR(SST)) and Ensemble o
Reg esso Chains (GA-FS-MTR(ERC)), o ob ain he inal
esul s. The ad an age o using his app oach is ha any a ge
in e dependencies a e also inco po a ed in he inal p edic ion
model.
2) GENETIC ALGORITHM FEATURE SELECTION USING
SEPARATE FEATURES (GA-FS-TARGET)
The p oposed me hod GA-FS-Ta ge wo ks by inding a ge
speci ic ea u es o each a ge sepa a ely. One ad an age
o using his echnique is ha no MTR algo i hms a e
needed. The e o e, he complexi y o he model is dec eased.
Ano he si ua ion whe e his me hod may also be use ul
is when o any MTR p oblem, he e a e some a ge s
ha a e mo e impo an han o he s. When using MTR
algo i hms, each a ge is conside ed equally. This p oposed
me hod can p o ide an imp o emen o e baseline me hods
while allowing lexibili y o handling each a ge di e en ly.
Simila o he i s a ia ion, a popula ion o candida e
solu ions we e ini ialized andomly. The candida e solu ions
we e e alua ed using ST, bu he RRMSE o a single a ge
was conside ed each ime. E en ually, he ea u e subse
which minimized he RRMSE o a ge ‘mi’ was conside ed
he bes ea u e subse o a ge ‘mi’. The p ocess was hen
epea ed o each a ge . In his way, o a da ase wi h ‘m’
a ge s, ‘m’ ea u e subse s we e ound.
I would be pe inen o men ion he e ha in each case, he
ST me hod was used o ind he ea u e subse . The eason
is ha since he MTR algo i hms also inco po a e a ge
in o ma ion when gene a ing he models, only he ea u e
subse s e u ned by using MTR algo i hms would no ha e
gi en an accu a e pic u e o he ea u e impo ance.
G. EXPERIMENTAL SETTINGS
All expe imen s we e ca ied ou using Mulan and Weka [43].
The da a was di ided in o wo equal hal . 15- old c oss
alida ion was used o ea u e selec ion expe imen s using
hal o he da a. The expe imen s we e pe o med using
‘ ep ee’ as base eg esso . Once he bes ea u e subse s we e
iden i ied, he educed ea u es we e used o aining and
es ing on he second hal o he da a o ob ain inal esul s
which a e epo ed in he pape . This was also ca ied ou
using 10- old c oss alida ion. Fo he gene ic algo i hm,
a s anda d single poin c osso e was used. The mu a ion a e
was se andomly. Ini ially, a highe mu a ion a e was se o
encou age explo a ion. I was dec eased o e he cou se o
he subsequen ly gene a ions.
IV. RESULTS AND DISCUSSIONS
This sec ion p esen s he esul s o he expe imen s and
sepa a e discussions o each compa ed me hod. The esul s
a e p esen ed o show a compa ison o he p oposed
me hods wi h hei espec i e baseline me hods. In each
case, he p oposed a ia ion can be seen o ha e imp o ed
he pe o mance o he he baseline me hod. In he Pe
Ta ge me hod, a Single Ta ge (ST) app oach is employed.
The e o e, his p oposed me hod is compa ed wi h all he
baseline me hods using Table 12.
A. COMPARISON WITH (GA-FS-MTR)
GA-FS-MTR-SO: The p oposed GA-FS-MTR-SO me hod
p o ides an imp o emen in pe o mance compa ed o he
benchma k me hods in all ins ances excep a ew. Tables 3,
4and 5show he esul s.
The ST me hod achie es an a e age pe o mance imp o e-
men o a ound 7.42% compa ed o i s baseline by using he
p oposed app oach in he majo i y o he da a se s, as well as
on a e age. The s anda d de ia ion is also less, showing ha
esul s do no a y e y la gely.
The SST me hod also achie es a pe o mance imp o e-
men o a ound 14.5% compa ed o i s baseline me hod by
using he p oposed app oach in he majo i y o he da a se s.
The a e age pe o mance and he s anda d de ia ion also
e lec he pe o mance imp o emen .
VOLUME 11, 2023 121971
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
TABLE 3. Table showing he compa ison be ween he ARRMSE ob ained
om baseline me hod ST [21] wi h he ARRMSE ob ained using he
p oposed Single objec i e me hod GA-FS-MTR(ST)-SO.
TABLE 4. Table compa ing he ARRMSE ob ained om baseline me hod
SST [22], wi h he ARRMSE ob ained using he p oposed Single objec i e
me hod GA-FS-MTR(SST)-SO.
TABLE 5. Table compa ing he ARRMSE ob ained om baseline me hod
ERC [22], wi h he ARRMSE ob ained using he p oposed Single objec i e
me hod GA-FS-MTR(ERC)-SO.
The ERC me hod also achie es a pe o mance imp o e-
men o a ound 11% compa ed o i s baseline me hod by
using he p oposed app oach in he majo i y o he da a se s.
In each case, in addi ion o he dec ease e o , he s anda d
de ia ion is also educed, hus leading o a mo e con iden
p edic ion using he p oposed app oach.
To es i he di e ences in pe o mances we e signi ican ,
pai ed samples - es was ca ied ou be ween each MTR
algo i hm and i s p oposed coun e pa . Table 6p esen s he
esul ing p- alues o Single Objec i e me hods.
The signi icance es shows ha he p oposed me hods
ha e signi ican ly be e pe o mance a alpha alue o 0.05 in
TABLE 6. Signi icance es esul s showing p- alues be ween Benchma k
me hods ST, SST and ERC e sus p oposed me hods GA-FS-MTR(ST)-SO,
GA-FS-MTR(SST)-SO and GA-FS-MTR(ERC)-SO.
some cases, while a he alpha alue o 0.1, he p oposed
me hods ha e signi ican ly be e pe o mance in almos all
cases.
Fu he , F iedman es was ca ied ou o be e unde s and
he pe o mance di e ences be ween he me hods. Figu e 7
shows c i ical dis ance diag am o he compa ed echniques
a alpha=0.1.
FIGURE 7. C i ical dis ance diag am compa ing Single Objec i e based
GA-FS-MTR me hods wi h he baseline me hods o alpha=0.1.
Lowe anks a e be e . I can be seen om Figu e 7 ha he
p oposed me hods gene ally ou pe o m he baseline me hods
and he e o e ha e lowe anks in he diag am. (Objec i es I
and I)
GA-FS-MTR-MO: I he ea u e selec ion p oblem is
o mula ed as a mul i-objec i e (MO) p oblem, a mul i-
objec i e i ness unc ion is used in he GA. The GA(MO)
e u ns a Pa e o on , om which he ea u e subse ha
educes he e o he mos is selec ed. Tables 7,8and9p esen
he esul s o hese expe imen s.
TABLE 7. Table compa ing he ARRMSE ob ained om baseline me hod
ST [21] e sus he ARRMSE ob ained using he p oposed Mul i-objec i e
me hod GA-FS-MTR(ST)-MO.
The p oposed GA-FS-MTR-Mul i Objec i e me hods also
p o ides an o e all pe o mance imp o emen as shown in
Tables 7,8and 9. In he a e age case an imp o emen o
a ound 4% o e ST, 15.5% o e SST and 10.5% o e ERC
is ob ained. In his case as well, he s anda d de ia ion ac oss
all da a se s is also lowe o he p oposed me hod.
To es i he di e ences in pe o mances we e signi ican ,
pai ed samples - es was ca ied ou be ween each MTR
121972 VOLUME 11, 2023
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
TABLE 8. Table compa ing he ARRMSE ob ained om baseline me hod
SST [22] e sus he ARRMSE ob ained using he p oposed Mul i-objec i e
me hod GA-FS-MTR(SST)-MO.
TABLE 9. Table compa ing he ARRMSE ob ained om baseline me hod
ERC [22] e sus he ARRMSE ob ained using he p oposed Mul i-objec i e
me hod GA-FS-MTR(ERC)-MO.
algo i hm and i s p oposed coun e pa . Table 10 p esen s he
esul ing p- alues o Mul i-Objec i e me hods.
TABLE 10. Signi icance es esul s showing p- alues be ween
Benchma k me hods ST, SST and ERC e sus p oposed me hods
GA-FS-MTR(ST)-MO, GA-FS-MTR(SST)-MO and GA-FS-MTR(ERC)-MO.
The p oposed me hods show signi ican di e ences a
alpha alue o 0.1 indica ing ha he p oposed app oach
imp o es he pe o mance conside ably.
F iedman es was also ca ied ou be ween he p oposed
and baseline me hods. Figu e 8shows c i ical dis ance
diag am o he compa ed echniques a alpha=0.1 o
Mul i-Objec i e based GA-FS-MTR me hods. (Objec i es I
and I).
FIGURE 8. C i ical dis ance diag am compa ing Mul i-Objec i e based
GA-FS-MTR me hods wi h he baseline me hods o alpha=0.1.
Lowe anks a e be e . I is seen om he igu es ha
he p oposed me hods gene ally ou pe o m he baseline
me hods and he e o e ha e lowe anks in he diag am. The
GA-FS-MTR(ERC) me hod pe o ms bes among he p o-
posed me hods o bo h he cases.
In o de o analyze how much ea u e educ ion he p o-
posed app oaches p o ide, he numbe o ea u es ob ained
a e ea u e selec ion a e analyzed. Table 11 shows he
pe cen age educ ion in he numbe o ea u es when he
p oposed GA-FS-MTR (GA-FS-MTR-Single Objec i e o
GA-FS-MTR-Mul i-Objec i e) me hods a e used.
TABLE 11. Pe cen age educ ion in numbe o ea u es using
GA-FS-MTR-Single Objec i e and GA-FS-MTR-Mul i-Objec i e.
The esul s show ha he p oposed me hods p o ide a
ema kable educ ion in he numbe o ea u es. Wi h a
minimum o 38 pe cen educ ion, ea u e selec ion p o ides
a subs an ial imp o emen in in e p e abili y by iden i ying
he ea u es which a ec he a ge a iables he mos .
Ano he poin o in e es in he esul s is ha e en o he
lowe dimensional da ase s, he p oposed me hods s ill allow
o a leas an a pa pe o mance wi h he educed ea u e
se . This scale o educ ion allows o a g ea e possibili y
o isualizing he esul s using g aphical echniques and hus
gaining mo e insigh in o he da a.
In o de o inspec he e ec o ea u e selec ion on each
a ge , Figu es 9(a) and 9(b), show a compa ison o he
Rela i e Roo Mean Squa ed E o o each a ge when
ea u es a e educed using GA-FS-MTR-Single Objec i e.
Figu es 10(a) and 10(b) show a compa ison o he Rela i e
Roo Mean Squa ed E o o each a ge when ea u es a e
educed using GA-FS-MTR-Mul i-Objec i e.
As can be seen om he igu es, he pe o mance
imp o emen is obse ed in almos all o he a ge s using
he p oposed app oaches.
Del ing u he in o he ea u e selec ion p ocess o ind
how o en any ea u e is used when making p edic ions.
In o de o do ha , Local in e p e able model-agnos ic
explana ions LIME [44] was used o e alua e he ea u e
con ibu ions o wo da a se s ‘ju a’ and ‘edm’ and he
p edic ions we e made using a Decision T ee eg esso and a
Ridge eg esso implemen ed using py hon’s sklea n lib a y.
Figu es 11(a) and 11(b) show he pe cen age o he numbe
o a imes a ea u e was used by he model o make a
p edic ion. As can be seen om he igu es, al hough some
ea u es a e used mo e equen ly in making he p edic ions,
i is no possible o decide he ex en o which he models will
be a ec ed i one o he ea u es is d opped. The e o e, any
da a d i en ac i i y will equi e an assessmen o he ea u es
indi idually. Fea u e selec ion iden i ies he mos impo an
VOLUME 11, 2023 121973
F. H. Syed e al.: Towa d an Op imal and S uc u ed Fea u e Subse Selec ion o MTR
FIGURE 9. Compa ison o Rela i e Roo Mean Squa ed E o o each
a ge in he da a se o da a se s ‘and o’ and ‘ju a’ when ea u e
educ ion is ob ained using GA-FS-MTR-Single Objec i e.
FIGURE 10. Compa ison o Rela i e Roo Mean Squa ed E o o each
a ge in he da a se o da a se s ‘and o’ and ‘ju a’ when ea u e
educ ion is ob ained using GA-FS-MTR-Mul iObjec i e.
ea u es and he e o e allows inc eased in e p e abili y and
unde s anding in o he wo king o he model.
B. COMPARISON WITH (GA-FS-TARGET)
The esul s o he p oposed me hod GA-FS-Ta ge a e
p esen ed in he Table 12. Al hough his p oposed echnique
igno es he possible a ge in e dependencies, i s ill p o ides
an a e age pe o mance imp o emen o oughly 8.6% in
compa ison o all he MTR me hods.
FIGURE 11. Pe cen age o imes a ea u e con ibu ed o a p edic ion o
wo machine lea ning models.
TABLE 12. Table compa ing he ARRMSE ob ained om baseline
me hods ST [21], SST and ERC [22], e sus he ARRMSE ob ained using he
p oposed Pe a ge me hod.
Table 13 shows he p- alues o - es be ween he
benchma ked me hods and he p oposed me hod GA-FS-
Ta ge . The esul s how a signi icance in all cases a alpha
alue 0.1 and a signi icance in all bu one case o alpha 0.05.
TABLE 13. Signi icance es esul s showing p- alues be ween Benchma k
me hods ST, SST and ERC e sus p oposed me hod GA-FS-Ta ge .
Figu e 12 shows he c i ical dis ance diag am o
GA-FS-Ta ge me hod a alpha=0.1. The p oposed me hod
121974 VOLUME 11, 2023