Ensemble methods and semi-supervised learning for information fusion: A review and future research directions

In o ma ion Fusion 107 (2024) 102310 A ailable online 16 Feb ua y 2024 1566-2535/© 2024 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by- nc-nd/4.0/). Con en s lis s a ailable a ScienceDi ec In o ma ion Fusion jou nal homepage: www.else ie .com/loca e/in us Ensemble me hods and semi-supe ised lea ning o in o ma ion usion: A e iew and u u e esea ch di ec ions José Luis Ga ido-Lab ado , Ana Se ano-Mamola ∗, Jesús Maudes-Raedo, Juan J. Rod íguez, Césa Ga cía-Oso io Uni e sidad de Bu gos, A da. Can ab ia s/n, Bu gos, 09006, Spain ARTICLE INFO Keywo ds: Semi-supe ised lea ning Ensemble lea ning In o ma ion usion Semi-supe ised ensemble classi ica ion Label sca ci y Bibliog aphic e iew Resea ch ends Expe imen al p o ocols ABSTRACT Ad ances o e he pas decade a he in e sec ion o in o ma ion usion me hods and Semi-Supe ised Lea ning (SSL) a e in es iga ed in his pape ha g apple wi h challenges ela ed o limi ed labelled da a. To do so, a bibliog aphic e iew o pape s published since 2013 is p esen ed, in which ensemble me hods a e combined wi h new machine lea ning algo i hms. A o al o 128 new p oposals using SSL algo i hms o ensemble cons uc ion a e iden i ied and classi ied. All he me hods a e ca ego ised by app oach, ensemble ype, and base classi ie . Expe imen al p o ocols, p e-p ocessing, da ase usage, unlabelled a ios, and s a is ical es s a e also assessed, unde lining he majo ends, and some sho comings o pa icula s udies. I is e iden om his li e a u e e iew ha ounda ional algo i hms such as sel - aining and co- aining a e in luencing cu en de elopmen s, and ha inno a i e ensemble echniques a e con inuing o eme ge. Addi ionally, aluable guidelines a e iden i ied in he e iew o imp o ing esea ch in o in insically semi-supe ised and unsupe ised p e-p ocessing me hods, especially o eg ession asks. 1. In oduc ion In o ma ion usion, an impo an p ocess ha in ol es in eg a ing and assimila ing da a om a ious sou ces and combining di e en lea ne pe spec i es, has become a c i ical componen in a a ie y o ields such as na u al language p ocessing, compu e ision, and a ec i e compu ing [1]. This comp ehensi e app oach yields a be e unde s anding o complex phenomena and acili a es he c ea ion o mo e accu a e and eliable decisions. Howe e , acqui ing labelled da a o implemen supe ised lea ning me hods emains a majo obs acle in se e al in o ma ion usion applica ions. The bo leneck pe sis s, due o high cos s, ime cons ain s, and qui e o en he shee imp ac icali y o labelling ex ensi e da ase s, all o which is compounded by he inhe en na u e o he da a, and in many a eas a sho age o domain expe s. Addi ionally, ecen ad ances wi hin he ield o in o ma ion u- sion ha e led o inc eased in e es in objec i e me hodologies such as ensemble lea ning, which amalgama es di e se models o enhance p e- dic i e pe o mance, and Semi-Supe ised Lea ning (SSL) ha uses bo h labelled and unlabelled da a o aining. Ensemble lea ning augmen s decision-making by combining mul iple models, e ec i ely mi iga ing he bias o indi idual models and imp o ing o e all accu acy and eliabili y. In u n, SSL u ilises unlabelled da a o supplemen he sca ci y o labelled samples, add essing he limi a ions o he all- oo- o en labo ious labelling p ocess. These echniques o e p omising ∗Co esponding au ho . E-mail add ess: [email p o ec ed] (A. Se ano-Mamola ). possibili ies in in o ma ion usion, p o iding iable solu ions o o e - come he cons ain s imposed by he lack o labelled da a in a ious applica ion domains. SSL has eme ged as an ou s anding pa adigm o add essing he challenges o pa ially labelled da a [2]. The semi-supe ised app oach akes ad an age o he abundance o unlabelled da a in many con ex s, oge he wi h a small se o labelled examples, in o de o imp o e lea ning pe o mance. Fu he mo e, ensemble lea ning is used o im- p o e gene alisa ion, which o en sol es he p oblems linked o he poo adap abili y o single-lea ned app oaches. Thus, ensemble me h- ods, which combine mul iple models o make collec i e p edic ions, ha e shown g ea po en ial in SSL. Semi-supe ised ensembles can exploi inhe en s uc u es and ela ions wi hin he da a, by agg ega ing he ou pu s o di e se models ained on bo h labelled and unlabelled da a, o achie e enhanced p edic i e accu acy and obus ness; a p o- cess ha is also known as in o ma ion usion in he ield o machine lea ning. A que y sea ch on he Scopus da abase, using ‘‘semi-supe ised’’ and ‘‘ensemble’’ as he sea ch e ms, o pape s published be ween 2013 and 2023, e u ned 450 pape s, which a e add essed in his e iew. Among hose pape s, 128 new semi-supe ised ensemble me hods we e iden i ied in 127 s udies. The me hods we e ca ego ised acco ding o he semi-supe ised app oach (w appe , p e-p ocessing, in insically), h ps://doi.o g/10.1016/j.in us.2024.102310 Recei ed 29 Sep embe 2023; Recei ed in e ised o m 14 Feb ua y 2024; Accep ed 15 Feb ua y 2024 In o ma ion Fusion 107 (2024) 102310 2 J.L. Ga ido-Lab ado e al. Fig. 1. VOSViewe keywo d co-occu ence clus e ing iew. Each e m is ep esen ed by a ci cle, whe e i s diame e and he size o i s label ep esen s he equency o occu ence o he e m. The lines in he isualisa ion indica e ha he wo keywo ds a e p esen in he same pape , and he wid h o he line is p opo ional o he numbe o pape ha sha e hese keywo ds. he ensemble ype (Bagging, Boos ing...) and he base classi ie ype. Mo eo e , he expe imen al p o ocol, se ings, p e-p ocessing me hods, numbe o da a se s employed in he expe imen a ion, unlabelled a ios, and s a is ical es s we e all analysed. A keywo d co-occu ence map was gene a ed, based on he se o esul s ound in Scopus using VOSViewe so wa e [3]. A hesau us was also compiled, in o de o pe o m da a cleaning: linking up di e en spellings o he same e m, abb e ia ed keywo ds wi h ull keywo ds, and synonyms. A o al o 31 keywo ds we e iden i ied ha me he c i e ion o ha ing a minimum equency o 3 occu ences om he ini ial pool o 210 keywo ds. Then, a co-occu ence analysis was pe o med on hese 31 keywo ds, as shown in Fig. 1. In his map, he 5 clus e s ha we e iden i ied a e shown in di e en colou s. Clus e ing in VOS iewe in ol es a p ocess o minimising dis ances be ween keywo ds and placing he mos closely ela ed keywo ds in one clus e [4]. The node a ea and he on size depends on he weigh o he keywo d: he highe he alue, he mo e equen he keywo d, and he e o e he la ge he co esponding node and label. The connec ing line be ween nodes ep esen s a sha ed occu ence o a keywo d wi h ano he keywo d. The keywo d co-occu ence s eng h is ep esen ed by he hickness o he connec ing line: he hicke he connec ing line, he mo e equen he co-occu ences be ween bo h keywo ds. In he ed-colou ed clus e , he mos equen ly epea ed keywo ds ‘‘en- sembles’’ and ‘‘semi-supe ised lea ning’’ a e ep esen ed, ollowed by he e m ‘‘classi ica ion’’, oge he wi h machine lea ning me hods, and algo i hms such as ‘‘co- aining’’, ‘‘adaboos ’’, and ‘‘ andom o es ’’. The yellow g oup includes he e m ‘‘high-dimensional da a’’ and ela ed applica ions, such as ‘‘social-ne wo ks’’ and ‘‘spamme de ec ion’’. The blue and pu ple clus e s include machine lea ning e ms and some ensemble algo i hms, such as ‘‘sel - aining’’ and ‘‘ i- aining’’. Te ms ela ed o ‘‘imbalanced lea ning’’ can be ound in he g een clus e , oge he wi h applica ions whe e imbalanced da a is a common issue, such as ‘‘in usion de ec ion’’ and ‘‘anomaly de ec ion’’. Al hough he e a e s a e-o - he-a e iews on his opic, none o e comp ehensi e analyses o ensembles used in combina ion wi h SSL. The main objec i e o his e iew is o o e some aluable insigh o he scien i ic communi y in o he use o ensemble me hods o SSL. Techniques employed o combining base classi ie s, such as Bagging and Vo ing, a e explo ed. Addi ionally, an analysis o commonly used base classi ie s including T ee and Suppo Vec o Machine (SVM) classi ie s a e also examined. Likewise, expe imen al aspec s, including he numbe o da ase s and s a is ical es ou ines, a e employed. A c i ical pe spec i e is also p o ided in his comp ehensi e e iew ha cla i ies he use o exis ing me hods and hei cha ac e is ics. Ano he objec i e o his wo k is o p esen a comp ehensi e ca alogue o ecen semi-supe ised ensemble lea ning me hodologies o he sci- en i ic communi y, enabling u u e esea ch o conside he cu en s a e-o - he-a . Consequen ly, he e alua ion o o hcoming li e a u e p oposing no el algo i hms can be pe o med agains he mos up- o- da e echniques. Mo eo e , ce ain inadequacies o cu en esea ch a e ou lined in his pape , so ha such sho comings may be emedied in u u e s udies. Al hough he e a e al eady o he e iew a icles on SSL me hods, ce ain di e en ia ing cha ac e is ics jus i y he inclusion o his e iew in he li e a u e on SSL: 1. Fi s , his e iew is pa icula ly ocused on he in e sec ion o SSL and in o ma ion usion me hods, speci ically ensemble me h- ods. By ocusing on his in e sec ion, his s udy goes beyond a- di ional e iews and p o ides aluable insigh in o he syne gies and no el app oaches ha eme ge when bo h me hodologies a e combined. 2. I also add esses a c ucial gap in p e ious ones, in so a as i p o ides an upda ed analysis ha inco po a es ecen ad ances o SSL, ensu ing ha eade s a e awa e o he la es me hods p oposed in he ield. 3. A no able ea u e o his e iew, unlike p e ious e iews, is he analysis o he numbe o da ase s, me ics, and me hodologies used in each s udy, whene e new algo i hms a e compa ed wi h exis ing ones. In mos pape s, he compa isons be ween p oposed In o ma ion Fusion 107 (2024) 102310 3 J.L. Ga ido-Lab ado e al. and exis ing me hods a e no me hodologically subs an ia ed which migh o he wise add eliabili y o hei claims ha he new me hods ac ually ep esen imp o emen s. 4. Las ly, an analysis o he keywo ds used in he pape s ha a e unde e iew, he g owing in e es in he di e en SSL me hods, and he ci a ion g aph and ne wo k o links be ween he pape s a e all ea u es ha canno be ound in o he e iews. The emainde o his pape is o ganised as ollows. The basic concep s, assump ions, and axonomy o SSL me hods a e co e ed in Sec ion 2. In Sec ion 3, he e iew o wo ks on ensembles o SSL and hei analysis is o ganised in o h ee empo al s ages: pas , p esen , and u u e. Finally, he conclusions o he e iew a e gi en in Sec ion 4. 2. Semi-supe ised lea ning As he name implies, semi-supe ised lea ning is a machine lea ning echnique somewhe e be ween supe ised lea ning, which equi es ha ing a labelled da ase , and unsupe ised lea ning, which aims o disco e pa e ns and in e es ing s uc u es in unlabelled da ase s. In semi-supe ised lea ning he ul ima e goal is he same as in supe - ised lea ning: o ob ain a p edic i e model ha can assign labels o unlabelled ins ances. Howe e , in addi ion o he labelled ins ances ha a e a ailable, semi-supe ised lea ning also u ilises ins ances o which hei labels a e unknown. The in o ma ion con ained in hese unlabelled ins ances can be exploi ed in a ious ways. In he simples app oach, he unlabelled ins ances a e used only a he beginning o he lea ning p ocess o ini ialise pa ame e s o a supe ised algo i hm o o iden i y g oups on which he algo i hm will ac . Ano he app oach in ol es i e a i ely and inc emen ally expanding he se o labelled ins ances by adding new ins ances as he models a e e ined and con idence in he p edic ion o labels o unlabelled ins ances inc eases. Finally, he e a e me hods ha di ec ly use bo h labelled and unlabelled ins ances in he compu a ion o he loss unc ion. This sec ion p esen s he axonomy o all hese semi-supe ised lea ning me hods, bu i s , le us conside he cha ac e is ics ha a da ase should ha e o ensu e ha he use o unlabelled ins ances can bene i he lea ning p ocess. 2.1. Unde lying assump ions I is impo an o no e ha semi-supe ised lea ning is no always gua an eed o imp o e a supe ised model. Fo unlabelled da a o help build a be e classi ie i is impo an ha su icien unlabelled da a is a ailable and ha he dis ibu ion o he unlabelled da a mee s some assump ions [5]: •The smoo hness assump ion: I wo ins ances look simila , hey should ac ually be o he same class. •The low-densi y assump ion: Class decision bounda ies should a oid a eas o high densi y and p e e a eas whe e he e a e ew ins ances. •The mani old assump ion: Ins ances appea ing in he same low- dimensional mani old should ha e he same class. •The clus e assump ion: Ins ances ha a e clus e ed oge he should be assigned he same class. This p inciple is a gene ali- sa ion o he o he h ee. 2.2. Taxonomy and SSL me hods classi ica ion SSL has adi ionally been di ided in o induc i e and ansduc i e lea ning, depending on he p ima y goal. While he aim o ansduc- i e me hods is o ob ain he labels o unlabelled da a poin s wi hin a da ase , he aim o an induc i e me hod is o ind a gene alised model ha can gene a e p edic ions o any objec in he inpu space. Van Engelen and Hoos [2] p esen ed a no el axonomy o semi- supe ised me hods (see Fig. 4). The i s di ision sepa a es he p e- iously p esen ed induc i e and ansduc i e me hods. The me hods Fig. 2. I e a i e p ocess o a w appe model [6]. 𝐿is he supe ised se ,𝑈 he unsupe ised se ,𝑈′is a pa o 𝑈wi h pseudo-labels. Fig. 3. Gene al s uc u e o a semi-supe ised algo i hm based on unsupe ised p e-p ocessing. 𝐿is he supe ised se , 𝑈 he unsupe ised se . used in he ansduc i e b anch a e called g aph-based. They ha e o unde go h ee phases: he i s conce ns he way ha he g aph is buil ; he second, he way in which weigh s a e a ached o he links; and he hi d, he way ha he class o he unlabelled nodes can be in e ed. Induc i e me hods a e di ided in o h ee b anches depending on how hey exploi he unlabelled da a. Fi s ly, w appe me hods, which use an i e a i ely ained supe ised me hod whose aining da a in- cludes bo h ins ances om he unlabelled da a and pseudo-labels (i.e., labels p edic ed om p io i e a ions o he model). These echniques can be sel - aining, i a single classi ie is used, co- aining, i se e al a e simul aneously employed, and boos ing, i se e al a e sequen ially u ilised. The main di e ence be ween hese app oaches s ems om he way ha pseudo-labels a e included in he labelled subse . An example o his p ocess can be ound in Fig. 2. In he unsupe ised p e-p ocessing me hods, he labelled and un- labelled da a a e sepa a ely used, commonly using he unlabelled da a o ex ac ion o ans o ma ion o he da ase , and o ini ialise some o he algo i hm pa ame e s. In mos o hose me hods, a he han modi ying he labelled se by adding new ins ances and pseudo-labels, unsupe ised echniques a e used on he unlabelled (o he comple ed aining se ). Typical models include hose ha ex ac ea u es, hose ha apply clus e ing o p opaga e classes, and hose ha include p e- aining using au oencode s. Fig. 3 p esen s a gene al o e iew o he abo e-men ioned me hods. Addi ionally, in insically semi-supe ised me hods a e cha ac- e ised by di ec ly exploi ing all he assump ions. Deep-lea ning me hods, in which he objec i e unc ions o he unlabelled ins ances a e di ec ly conside ed, cons i u e mos o he in insically semi-supe ised me hods. The e a e se e al me hods o SSL wi h deep lea ning. A ecen e iew can be ound in [7]. I is no ewo hy ha deep lea ning me hods a e in en ionally omi ed om his s udy. A decision ha was based on wo p ima y conside a ions: i s , image p ocessing is p ima ily a ge ed in mos o he deep- lea ning app oaches, a e y dis inc domain qui e unlike he o he me hods unde s udy; second, hei ope a ional cha ac e is ics di e signi ican ly om he o he algo i hms ha a e included in he s udy. Hence, i would be ad isable o conduc a dedica ed e iew exclusi ely In o ma ion Fusion 107 (2024) 102310 4 J.L. Ga ido-Lab ado e al. Table 1 P os and cons o induc i e me hods. Sou ce: Ramí ez-Sanz e al. [8]. Licensed CC-BY. Me hod P os Cons W appe me hods 1. Easy o implemen . 2. Con igu able, easy o change base es ima o /s. 3. Can be used wi h almos any supe ised me hod. 1. P one o add noise. 2. Dependen on supe ised me hods. Unsupe ised p e-p ocessing 1. Can be used wi h almos any supe ised me hod. 1. Less impac o unlabelled da a. In insically semi-supe ised 1. Unlabelled da a is used on he lowe le el (objec i e unc ion o op imiza ion p ocedu e). 2. Usually easy o de elop om i s supe ised e sion. 1. Mo e complex models ∼Ha de o ain. 2. Mos o hem equi e la ge amoun s o da a. Fig. 4. The axonomy o semi-supe ised me hods as p oposed in [2]. In addi ion o he b anches wi hin he axonomy, some ep esen a i e me hods a e also shown in each sub-ca ego y (some in g ey because hei ca ego isa ion is ques ionable, o because hey could be assigned o mo e han one ca ego y). is used o ep esen he se o unlabelled ins ances, and 𝜃 o ep esen he pa ame e s o algo i hms. o he examina ion o semi-supe ised me hods ha employ deep lea ning. The e a e eno mous di e ences be ween in insically semi-supe ised me hods when le e aging he same assump ions. Fou ca ego ies a e iden i ied in he li e a u e, including ma gin maximise s which u ilise he low-densi y assump ion, o c ea e di iding lines ia clus e ing and SVM. Mani old me hods exploi he same assump ion, so as o gene a e opological a ia ions ha help achie e be e da a sepa- a ion. Gene a i e models ely on ad e sa ial models, and inally, pe u ba ion-based me hods usually inco po a e neu al ne wo ks ha al e he accep ance unc ion o he loss unc ion. Ramí ez-Sanz e al. [8] unde lined he p os and cons o all o hese me hods, which can be seen in Table 1. 3. Ensembles o semi-supe ised lea ning Ensemble me hods ha e eme ged as e ec i e s a egies o im- p o ing he gene alisabili y and obus ness o p edic i e models by in eg a ing he p edic ions o mul iple base es ima o s cons uc ed using a gi en lea ning algo i hm o combina ion he eo [9]. In he a o emen ioned pape , o example, an o e iew o ensemble me hods used in semi-supe ised en i onmen s can be consul ed. 3.1. Pas In 2013, T igue o e al. [10] published a e iew o echniques o SSL wi hin w appe me hods. They del ed in o he s a e-o - he-a a ha ime, ocusing on w appe s, mos o which employ ensemble lea n- ing echniques in he well-known co- aining app oach. A axonomy o w appe me hods was in oduced, encompassing he ca ego ies o single/mul i iew, single/mul i lea ning, and single/mul i classi ie . The au ho s desc ibed mul i-lea ne as he combina ion o a ious ech- niques o build he base classi ie s, whe eas mul i classi ie in ol ed he use o mul iple classi ie s o c ea e an ensemble. I sugges s ha he e a e no mul i-lea ne me hods ha unc ion as single classi ie s, bu he e a e single lea ning me hods ha unc ion as mul i classi ie s. In ha s udy, he pe o mance o he selec ed me hods was exhaus- i ely explo ed using i y- i e UCI classi ica ion da ase s and a ying he a io o labelled ins ances in he ange o 10 o 40 pe cen o he o al ins ances in he aining se s. A o al o 18 me hods we e in es iga ed, o which 14 we e co- aining me hods, hus u ilising en- semble echniques. Al hough he explo a ion was qui e in-dep h o he analysis o each me hod’s pe o mance, i was no no ably ex ensi e, as only 18 me hods p oposed since he eme gence o semi-supe ised me hods o e he pas 15 yea s we e examined. Ne e heless, hei ocus emains on he mos signi ican algo i hms ha ha e made an impac , as is e iden in hei con inued conside a ion In o ma ion Fusion 107 (2024) 102310 5 J.L. Ga ido-Lab ado e al. in mo e ecen e iews [7,11]. Addi ionally, compa isons we e d awn wi h o he me hods in oduced in ecen yea s [12,13]. O e he pas 10 yea s, addi ional e iews on SSL ha e been pub- lished, such as he one by Ning e al. [14], which was speci ically ocused on Co-T aining. The au ho s p o ided de ailed in o ma ion on he undamen al s ages o such an algo i hm: iew acquisi ion, lea ne di e en ia ion, and label con idence assessmen . In e ms o iew acquisi ion, hey highligh ed i e me hods [14] o in oducing di e si y in o classi ie s: gene a ing andom subspaces, wi h he RASCO [15] and he Rel-RASCO [16] algo i hms as p ominen examples; adop ing independen iews, as ini ially p oposed by Blum and Michel in he o iginal Co-T aining algo i hm [17]; ensu ing he su iciency o iews, as demons a ed in RSCO [18]; u ilising au oma ed pa i ioning, as ound in he CODA algo i hm [19]; and he mo e ecen s a egy o segmen ing iews based on knowledge space, an app oach ha only has one e e ence speci ically o ien ed owa ds deep lea ning: he DeCoTa algo i hm [20]. Rega ding lea ne di e en ia ion, he use o di e en base lea ne s was conside ed in such me hods as Democ a ic CoT aining [21]. Mo e- o e , he use o di e en op imisa ion algo i hms was likewise p oposed in [22] whe e gene ic algo i hms and pa icle swa m op imisa ion we e used o ob ain sligh ly di e en SVM; and di e en ways o se ing he pa ame e s o he base lea ne , o example, in [23] whe e he Minkowski dis ance exponen was changed o ob ain wo di e en eg ession models wi hin a co- aining algo i hm. Addi ionally, hey analysed di e en ways wi h which o measu e and o main ain lea ne di e ences. The inal pa ame e aken in o conside a ion was he e alua ion o pseudo-label con idence. The same au ho s d ew a dis inc ion be- ween implici con idence de i ed om classi ie ce ain y and explici con idence using en- old c oss- alida ion echniques, among o he s. Despi e such an in e es ing pe spec i e in hei p oposal o cha ac- e ise he di e en co- aining me hods, hey only conside ed some o he ea ly me hods, neglec ing he mos ecen . A mo e ecen e iew ocused on sel - aining was au ho ed by Amini e al. in 2022 [24]. Among he me hods hey conside ed, only ou used ensemble echniques. Much like he e iew o T igue o e al. [10], whose mo e in-dep h han b oad analysis included an empi ical s udy, while only ocusing on a ew selec ed me hods. 3.2. P esen In he las 10 yea s he e ha e been many new con ibu ions o semi-supe ised lea ning ha ha e no been co e ed by p e ious li e a- u e e iews, making hem ou da ed. In his sec ion, we e iew a icles published since 2013, ocusing on he combina ion o semi-supe ised lea ning wi h ensemble cons uc ion me hods. In wha ollows, he sea ch and il e ing p ocess o he e iewed a icles is explained, as well as he iden i ica ion o subg oups depending on he base me hod, he use o p e-p ocessing echniques o he size o he da a se s. Finally, a s udy o he expe imen al alida ion ca ied ou in he a icles is p esen ed. 3.2.1. Sea ch and il e ing p ocess Resea ch in o semi-supe ised ensembles o e he pas decade is ho oughly explo ed in his e iew pape . I s aim is o p o ide a comp ehensi e unde s anding o he cu en s a e o esea ch and i s e olu iona y ajec o y. To do so, he ollowing sea ch e ms we e used o ex ac a se o pape s om he Scopus da abase: (KEY (ensemble) AND KEY (semi-supe ised)) AND PUB- YEAR > 2012 Fig. 5. Numbe o algo i hms in each o he semi-supe ised ypes p oposed by Van Engelen and Hoos [2], some o which appea in mo e han one ca ego y. This ini ial sea ch yielded a o al o 450 pape s, and a p elimina y il e ing p ocess was ca ied ou o include only hose ha in oduced new me hods, while excluding e iews and applica ions. As a esul o his ini ial il e , a o al o 186 new semi-supe ised ensemble me hods we e iden i ied, spanning a ious ca ego ies. Two addi ional s udies ha i ed he sea ch pa ame e s we e also included, which we e e e enced in some o he sample pape s. Subsequen ly, o e ine he analysis owa ds hese algo i hms, he ocus was na owed down o hose associa ed wi h classi ica ion and eg ession, lea ing a o al o 128 SSL me hods. I equi ed he exclusion o ce ain b anches o SSL, including ansduc i e lea ning, which lacks p edic i e capaci y, despi e unc ioning wi h bo h classi ie s and eg esso s, and ac i e lea ning, due o i s eliance on use in e en ion. The 127 pape s o he e iew we e classi ied acco ding o he axonomy o Van Engelen and Hoos [2]. The esul s showed ha 94 o he pape s p oposed w appe me hods, 16 we e on in insically semi- supe ised me hods, and 21 on unsupe ised p e-p ocessing me hods. A ce ain subjec i i y may be no ed in his ca ego isa ion, in so a as ou o he pape s ell in o mo e han one ca ego y, hence he sum o he pape s wi hin each ca ego y was no equal o he o al numbe o pape s. An UpSe plo [25] wi h in e sec ions is shown in Fig. 5. The in e sec ions a e shown wi hin a ma ix, wi h he ma ix ows co e- sponding o he se s, and he columns co esponding o he in e sec ions be ween hose se s. The size o he se s and he in e sec ions a e shown as ho izon al and e ical ba cha s, espec i ely. The dis ibu ion o he pape s o e he yea s and he h ee main ca ego ies a e shown in Fig. 6. As expec ed, mos me hods we e w appe me hods and he la ge numbe o pape s published be ween 2018 and 2021 is s iking. 3.2.2. Analysis o me hods Wi h a speci ic ocus on ensemble me hods, his s udy encompasses a wide ange o echniques unc ioning in acco dance wi h he co- aining and he boos ing pa adigms, bo h o which a e amewo ks embedded wi hin w appe me hods. Ano he analysis was conduc ed wi h ega d o he ensemble ech- niques and he mos common base classi ie s. These echniques a e o en combined. The mos widely used echniques a e weigh ed and majo i y o ing (bo h ep esen ed as ‘‘Vo ing’’ in Fig. 7), ollowed In o ma ion Fusion 107 (2024) 102310 6 J.L. Ga ido-Lab ado e al. Fig. 6. E olu ion in he las en yea s o he p oduc ion o algo i hm g ouped by ype and yea . Fig. 7. Numbe o algo i hms ha use he di e en ypes o ensembles o combina ions he eo ha we e desc ibed in he sample o pape s. Ano he six combina ions we e used in only one a icle. All possible combina ions can be seen in he Appendix A wi h hei espec i e pape s. The Se Size shows he global size, including he combina ions ha a e no displayed. by me hods which employed Bagging, Fo es ,1and Boos ing. Apa om he mino i y me hods g ouped in he ca ego y ‘‘O he ’’ (which includes he use o he so max unc ion and Bayesian ne wo ks, among o he s), s acking was he leas used me hod. The UpSe plo in Fig. 7 shows he di e en combina ions o echniques. In 101 o he 127 pape s, (80%), he au ho s p e e ed o use a single echnique ins ead o a combina ion. A o al o 2 o mo e echniques we e combined in 26 o he 127 pape s, he mos commonly used combina ion being Bagging-Vo ing. The mos widely used base classi ie s we e decision ees, ollowed by he SVM algo i hm. Those esul s we e expec ed, as he SVM was one o he i s algo i hms adap ed o semi-supe ision [26] and deci- sion ees we e among he mos widely used algo i hms o ensemble classi ica ion [27,28]. Ex ensi e usage o A i icial Neu al Ne wo ks was also no ed, such as Mul ilaye Pe cep ons, and lazy echniques, such as KNN. To a lesse ex en , some echniques we e based on Bayes’ heo em and linea me hods, he la e being mainly o eg ession p oblems. The dis ibu ion diag am o hese base classi ie s can be ound in Fig. 8. The combina ion o lea ne s ob ained wi h di e en 1Fo his s udy, Random Fo es s, despi e being a sub- ype o bagging, has been ea ed as a dis inc ca ego y wi h o he me hods such as Ro a ion Fo es and CoFo es as ‘‘Fo es ’’. me hods is a common p ac ice in SSL whe e i is usually e e ed o as Co-Lea ning [10,21]. The ype o base es ima o de e mines whe he i is a classi ie o a eg esso . O e he pas en yea s, algo i hms o semi-supe ised eg ession we e p esen ed in only 13 a icles; he emaining 114 algo i hms p esen ed o e ha same pe iod we e o classi ica ion. An in e es ing poin o e alua ion is whe he he algo i hms in he pape s unde e iew u ilised any o m o p e-p ocessing. A o al o 45 algo i hms made use o some ype o p e-p ocessing, among which 7 employed wo o mo e me hods. Those me hods p ima ily included subspaces and ea u e ex ac ion, along wi h a wide a ie y o echniques such as no malisa ion, ea u e selec ion, mani olds, and noise educ ion. The dis ibu ion can be obse ed in Fig. 9. 3.2.3. E alua ing he expe imen al alida ion The way ha he expe imen a ion was conduc ed o e alua e he p oposed algo i hms is ano he de ail o assessmen . I includes he numbe o da ase s used, labelling a ios, whe he ain– es spli ing o c oss- alida ion was employed, and he numbe o execu ions, as well as whe he s a is ical es s we e conduc ed in compa ison wi h o he models, speci ying he s a is ical me hods used o such compa isons. In Fig. 10, a his og am is p esen ed in in e als o i e, illus a ing he numbe o da ase s used o expe imen a ion. The mos common In o ma ion Fusion 107 (2024) 102310 7 J.L. Ga ido-Lab ado e al. Fig. 8. Dis ibu ion o base lea ne me hods used in he p oposed ensembles. The e a e se en een o he combina ions, which can be ei he Co-Lea ning, o es ing wi h a ious echniques, used only in one a icle. All possible combina ions can be seen in Appendix A alongside hei espec i e e e ences. The Se Size shows he global size, including he combina ions ha a e no displayed. Fig. 9. P e-p ocessing echniques used in some algo i hms. The e a e a u he se en combina ions, only used in one pape . All possible combina ions can be seen in he supplemen a y ma e ial. The Se Size shows he global size, including he combina ions ha a e no displayed. app oach is o use less han en da ase s. In ela ion o he labelling a io, a ixed numbe o labels we e u ilised in wel e algo i hms o he en i e da ase o a ixed numbe pe class. One labelling a io was es ed in a o al o 28 pape s. The mos common minimum and maximum labelling anges we e bo h wi hin 0.1−0.15. T ain– es spli ing was he mos ypical e alua ion a e c oss- alida ion echniques, wi h a a io o 63 ain– es spli s o 37 c oss- alida ions. Conce ning he numbe o execu ions, only one was epo ed in mos pape s, wi h mul iple execu ions desc ibed in only 40 o he pape s. The mean was 21 execu ions and he mos equen alue was 10 o he pape s epo ing mo e han one execu ion ou ine. The gene al end is un o una ely no o employ any s a is ical es , so compa isons wi h o he me hods a e complica ed. In Fig. 11, i can be obse ed ha no es s o any ype we e used in mos o he pape s, o alling 73 (57%). In dec easing o de o usage, he es s we e as ollows: F iedman es , S uden - es , Wilcoxon es , Nemenyi es , Holm es , Bon e oni es , and Bayesian es . Addi ionally, he e a e some o he s, such as Iman–Da enpo o Finne used in ce ain a icles. I was no ewo hy ha Bayesian es s, p oposed by Bena oli e al. in 2017 [29], we e used in only one o he pape s ound in his s udy. Demša also p oposed many s a is ical es s in 2006 [30], so i is all he mo e ema kable ha a e so many yea s, algo i hms wi hou s a is ical es s a e s ill employed in many s udies ha migh o he wise ha e yielded some s a is ical e i ica ion o he imp o emen s ha we e p oposed in hose s udies. In cases whe e s a is ical es s we e u ilised, he ypical alpha alues we e 0.01 and 0.05. Las ly, i is in e es ing o discuss which me ics a e mos commonly used o assess he pe o mance o he algo i hms. He e, a dis inc ion is made be ween classi ica ion and eg ession, as he me ics di e . The in e sec ion diag ams can be obse ed in Figs. 12 and 13. The me ics used in classi ica ion a e qui e di e se, wi h a no able p e alence o accu acy, ound in 84 pape s and exclusi ely used in up o 49 pape s, o en p esen ed as i s in e se, he e o a e. Tha me ic is ollowed by he F-Sco e me ics (p ima ily F1), p ecision, ecall, he a ea unde he ROC cu e, and Cohen’s kappa. Addi ionally, o he me ics ha a e less commonly used a e he geome ic mean and speci ici y. In con as , he Roo Mean Squa e E o was he mos common eg ession me ic, ollowed by he coe icien o de e mina ion and some o he a ia ions, such as he Rela i e Roo Mean Squa e E o , and he Mean Absolu e E o . Nume ous s udies ha e been conduc ed, e ealing wo main ypes o esea ch. The i s ype in ol es gene al me hods ha a e es ed on a la ge numbe o da ase s. The second ype in ol es me hods c ea ed o sol e speci ic p oblems by exploi ing a pa icula da ase . This g oup includes se e al no ewo hy examples ha highligh he p oposed solu ions o a ious p oblems. One new me hod o de ec ing ad e se e ec s in d ugs is he SSEL-ADE [31]. This ecen s udy p oposes a Co-T aining and andom subspaces-based app oach, using a da ase ha combines a ious da a sou ces. Ano he example wo h highligh ing is he EnSSL algo i hm [32, 33], which has been es ed o medical p oblems such as blood and lung diseases. The algo i hm combines se e al classical semi-supe ised echniques, including T i-T aining, Co-T aining, and Sel -T aining. In o ma ion Fusion 107 (2024) 102310 8 J.L. Ga ido-Lab ado e al. Fig. 10. His og am o a icles using di e en numbe s o da ase s. Fig. 11. S a is ical es s applied o e alua e he pe o mance o he p oposed algo- i hms. The e a e a u he eigh combina ions only used in one pape . All combina ions can be seen in he supplemen a y ma e ial. The Se Size shows he global size, including he combina ions ha a e no displayed. 3.3. Discussion and u u e guidelines A e ex ensi e analysis, ce ain aspec s o cu en machine lea ning echniques may now be discussed. The i s aspec conce ns he p e a- lence o w appe me hods, speci ically hose u ilising co- aining ech- niques. I p omp s he ques ion as o why unsupe ised p e-p ocessing and in insically semi-supe ised echniques a e no as well-de eloped. One explana ion is ha ensembles, which a e closely ela ed o co- aining echniques, a e pe haps easie o design and o adap . In any case, explo ing he de elopmen o a ying ypes o me hods would be an in e es ing a enue o pu sue. Ano he poin o discussion is he ype o base es ima o o be used. SVMs a e a less widely used han ees. On he one hand, i may be expec ed and, on he o he hand, i dese es some discussion. Ensemble me hods p ima ily consis o decision ees [27], so hei widesp ead usage is expec ed. Howe e , i is su p ising how he low-densi y as- sump ion can be exploi ed in SVMs, one o he oldes semi-supe ised me hods, by maximising he dis ances be ween he on ie da a [5]. Ano he no ewo hy aspec is Co-Lea ning [10]; gi en he cu en end o combining me hods, u he explo a ion o hose echniques is a compelling a ea o esea ch. One obse able end is he signi ican numbe o s udies ha ha e no expe imen al e alua ion. Al hough all he pape s p esen ed expe imen s and esul s, o e hal included no s a is ical compa ison. Addi ionally, many p o ided no implemen a ions, posing challenges o new me hods o es ablish hemsel es as obus echniques. Conduc - ing s a is ical es s ha show a p oposed me hod pe o ming signi i- can ly be e han o he compa a i e me hods o e s ample e idence o o he esea che s wishing o apply i o simila p oblems. Fu he mo e, i hese schola ly wo ks included access o he ull implemen a ion o he esea che s, i migh acili a e implemen a ion in p oduc ion en i onmen s and encou age u he esea ch. An analysis o he ci a ion g aph o he pape s and he empo al e olu ion in Fig. 14 led o u he explo a ion o he opic. I is wo h no ing ha 81 ou o he 127 pape s lacked e e ences o we e no e e enced in he o he pape s ha o med he sample o his s udy. Mo eo e , i was ound ha mos pape s ha did e e ence each o he showed minimal c oss- e e encing. Conside ing he in o ma ion p esen ed in he p e ious pa ag aph, i becomes clea why s a is i- cal assu ance and accessible implemen a ions a e lacking and why ela ionships a e no es ablished. A esea che encoun e s di icul ies when compa ing hei me hod o o he s wi h no e idence o signi ican pe o mance, and he implemen a ion p ocess equi es in e p e a ion. I should be ei e a ed ha nume ous me hods canno be consolida ed in he cu en s a e o he a . Fu he mo e, while e iews p o e use ul in e alua ing me hod pe o mance, such as he one conduc ed by T igue o e al. [10], hei implemen a ion p o es ime-consuming o each me hod. Based on he abo e discussion, se e al u u e di ec ions can be iden i ied. 1. The e is signi ican po en ial o c ea e me hods ha u ilise semi- supe ised and unsupe ised p e-p ocessing echniques, as well as explo ing combina ions o hose me hods. 2. The use o mo e s a is ical me hods o e alua e newly de eloped me hods is ecommended, gi en hei ecen g ow h and unde - u ilisa ion in his ield, and u he explo a ion o he Bayesian es s is sugges ed [29]. 3. S a is ical compa isons o SSL echniques wi h hei supe ised coun e pa s is essen ial when employing supe ised me hods as baseline lea ne s. 4. I is also conside ed essen ial o p o ide accessible implemen- a ions o each new me hod ha is p esen ed, which can be included in such lib a ies as Weka [34], LAMDA-SSL [35], and he SSL Lib a y (sslea n) [36], and o ollow he API indica ions In o ma ion Fusion 107 (2024) 102310 9 J.L. Ga ido-Lab ado e al. Fig. 12. Classi ica ion me ics o e alua e he pe o mance o he p oposed algo i hms. The e a e ano he nine een combina ions, used only in one a icle. All combina ions can be seen in he supplemen a y ma e ial. The Se Size shows he global size, including he combina ions ha a e no displayed. Fig. 13. Reg ession me ics o e alua e he pe o mance o he p oposed algo i hms. The Se Size shows he global size, including he combina ions ha a e no displayed. o mo e es ablished lib a ies such as SciKi -Lea n [37]. Addi- ionally, o g ea e accessibili y, any such implemen a ions can be included in pla o ms such as Pape s wi h code.2 Beyond hese p ac ical ecommenda ions, he e a e some desi able objec i es ha could be conside ed in u u e wo k. Ra he han guide- lines, hese objec i es a e o anyone wishing o del e deepe in o he subjec . Fi s ly, conduc ing mo e speci ic s udies on which ypes o p oblems align be e wi h pa icula ypes o da a, especially conce ning ields o knowledge (indus ial, medical, ecological, e c.), da a ypes ( ex , images, sounds, abula , e c.), and hei combina ions. Secondly, s a is ical assessmen s o he pe o mance o a ious me hods can be pe o med on speci ic da ase s. I would be pa icu- la ly bene icial o ha e well-popula ed lib a ies o well-implemen ed 2h ps://pape swi hcode.com. me hods and speci ic semi-supe ised da ase s, mo ing beyond he yp- ical ‘‘unlabelling’’ app oach. This would ha e signi ican implica ions, equi ing he p oposal o new me ics o e alua e he pe o mance o algo i hms in s ic ly semi-supe ised se ings, unde s ood as hose in which he eal class is impossible o de e mine wi hou manual labelling. Las ly, me hods mus be c ea ed o asce ain how necessa y SSL is o speci ic da ase s and o disce n quickly whe he u ilising supe ised me hods migh be a be e op ion. 4. Conclusions A comp ehensi e o e iew o he combined use o SSL and in- o ma ion usion me hods, speci ically ensemble me hods, has been p esen ed in his e iew pape , ocusing on ecen de elopmen s since 2013. Se e al isualisa ion ools ha e been used o analyse he ela ion be ween he sample o pape s o e iew, such as a ci a ion g aph, as well as a keywo d co-occu ence map ha e ealed he main opics and ends wi hin he ield. Fu he mo e, UpSe plo s ha e been used o analyse he equency o use o he ensemble me hods, base classi ie s, p e-p ocessing echniques, s a is ical es s, and pe o mance me ics. A c i ical e alua ion has been p esen ed o pas and p esen esea ch in o semi-supe ised ensembles, highligh ing he s eng hs and weak- nesses o di e en app oaches, as well as he expe imen al aspec s and alida ion me hods. Some o he main indings o his e iew a e as ollows: •Semi-supe ised ensembles ha e shown hei g ea po en ial o imp o ing he lea ning pe o mance and obus ness o p edic i e models by exploi ing he unlabelled da a and he di e si y o base classi ie s. •The e was a ema kable inc ease in pape s published on he opic be ween 2018 and 2020. •W appe me hods we e he mos popula and widely s udied ca ego y o semi-supe ised ensembles, especially he co- aining and he sel - aining a ian s. •Bagging and o ing we e he mos common echniques o com- bining base classi ie s, while decision ees and suppo ec o machines we e he mos equen ly used base lea ne s. •The e was a lack o ex ensi e and igo ous expe imen a ion and alida ion in many s udies, as well as a need o mo e ci a ions and in luence o he p oposed me hods. In o ma ion Fusion 107 (2024) 102310 16 J.L. Ga ido-Lab ado e al. [144] O. Gha oudi, H. Elghazel, A. Aussem, A semi-supe ised ensemble app oach o mul i-label lea ning, in: 2016 IEEE 16 h In e na ional Con e ence on Da a Mining Wo kshops, ICDMW, IEEE, 2016, pp. 1197–1204, h p://dx.doi.o g/10. 1109/icdmw.2016.0173. [145] Y. Liu, X. Li, Z. Zhang, A new app oach in ejec in e ence o using ensemble lea ning based on global semi-supe ised amewo k, Fu u e Gene . Compu . Sys . 109 (2020) 382–391, h p://dx.doi.o g/10.1016/j. u u e.2020.03.047. [146] X. Zhang, F. Jiang, R. Zhang, S. Li, Y. Zhou, Social spamme de ec ion based on semi-supe ised lea ning, in: 2021 IEEE 20 h In e na ional Con e ence on T us , Secu i y and P i acy in Compu ing and Communica ions, T us Com, IEEE, 2021, pp. 849–855, h p://dx.doi.o g/10.1109/ us com53373.2021.00120. [147] C. Li, Y. Xie, X. Chen, Semi-supe ised ensemble classi ica ion me hod based on nea neighbo and i s applica ion, P ocesses 8 (4) (2020) 415, h p://dx. doi.o g/10.3390/p 8040415. [148] X. Niu, W. Ma, Semi-supe ised classi ie ensemble model o high-dimensional da a, In o m. Sci. 643 (2023) 119203, h p://dx.doi.o g/10.1016/j.ins.2023. 119203. [149] C. Jian, Y. Ao, Imbalanced aul diagnosis based on semi-supe ised ensemble lea ning, J. In ell. Manu . 34 (7) (2022) 3143–3158, h p://dx.doi.o g/10.1007/ s10845-022-01985-2. [150] C. Vo, P. Nguyen, ST_OS: An e ec i e semisupe ised lea ning me hod o cou se-le el ea ly p edic ions, IEEE T ans. Lea n. Technol. 14 (2) (2021) 238–256, h p://dx.doi.o g/10.1109/ l .2021.3072995. [151] V.T.N. Chau, N.H. Phung, An e ec i e semi-supe ised lea ning amewo k o empo al s uden classi ica ion, in: 2019 6 h NAFOSTED Con e ence on In o ma ion and Compu e Science, NICS, IEEE, 2019, pp. 363–369, h p: //dx.doi.o g/10.1109/nics48868.2019.9023795. [152] S. Zhao, J. Li, A semi-supe ised sel - aining me hod based on densi y peaks and na u al neighbo s, J. Ambien In ell. Humaniz. Compu . 12 (2) (2020) 2939–2953, h p://dx.doi.o g/10.1007/s12652-020-02451-8. [153] Y. Wang, T. Li, Imp o ing semi-supe ised co- o es algo i hm in e ol ing da a s eams, Appl. In ell. 48 (10) (2018) 3248–3262, h p://dx.doi.o g/10.1007/ s10489-018-1149-7. [154] M.-L. Zhang, Z.-H. Zhou, Exploi ing unlabeled da a o enhance ensemble di e si y, Da a Min. Knowl. Disco . 26 (1) (2011) 98–129, h p://dx.doi.o g/ 10.1007/s10618-011-0243-9. [155] R. Lin, O. Li, Q. Li, Y. Liu, Unknown ne wo k p o ocol classi ica ion me hod based on semi-supe ised lea ning, in: 2015 IEEE In e na ional Con e ence on Compu e and Communica ions, ICCC, IEEE, 2015, pp. 300–308, h p://dx.doi. o g/10.1109/compcomm.2015.7387586. [156] R. Thomas, J.E. Judi h, Vo ing-based ensemble o unsupe ised ou lie de ec- o s, in: Lec u e No es in Elec ical Enginee ing, Sp inge Singapo e, 2020, pp. 501–511, h p://dx.doi.o g/10.1007/978-981-15-3992-3_42. [157] I. Li ie is, A. Kana os, V. Tampakas, P. Pin elas, A weigh ed o ing ensemble sel -labeled algo i hm o he de ec ion o lung abno mali ies om X-Rays, Algo i hms 12 (3) (2019) 64, h p://dx.doi.o g/10.3390/a12030064. [158] Y. Zhao, M.K. H yniewicki, XGBOD: Imp o ing supe ised ou lie de ec ion wi h unsupe ised ep esen a ion lea ning, in: 2018 In e na ional Join Con e ence on Neu al Ne wo ks, IJCNN, IEEE, 2018, pp. 1–8, h p://dx.doi.o g/10.1109/ ijcnn.2018.8489605. [159] X. Li, W. Ma, Z. Zhou, C. Xu, XSS a ack de ec ion model based on semi- supe ised lea ning algo i hm wi h weigh ed neighbo pu i y, in: Ad-Hoc, Mobile, and Wi eless Ne wo ks, Sp inge In e na ional Publishing, 2020, pp. 198–213, h p://dx.doi.o g/10.1007/978-3-030-61746-2_15.