Flow Cytometry-Based Classification in Cancer Research: A View on Feature Selection

75CanCe In o ma ICs 2015:14(s5) In oduc ion Flow cy ome y enables quan i a i e measu emen o single- cell p ope ies h ough isible and luo escen ligh in a high- h oughpu manne . The measu ed signals include luo escence emission and ligh sca e . Flow cy ome y has been ou inely used o de ec ing malignancies om blood samples.1 Th ough echnological ad ances, measu ing he combina ion o luo- escen signals om se e al di e en channels has enabled he use o high-dimensional da a o s udies, such as cy ome ic inge p in ing2 and la ge-scale analysis o cell ypes.3 In his pape , we s udy he analysis o low cy ome y da a om he ea u e selec ion poin o iew. Mo e speci i- cally, low cy ome y is able o p oduce la ge quan i ies o pa ially edundan measu emen da a, and he selec ion o impo an quan i ies wi hin he la ge body o measu emen s is o in e es . Mo eo e , a ypical scena io con ains la ge quan- i ies o measu emen da a bu may be limi ed o only a ew pa ien s. Thus, an ideal me hod would dis il only he essen ial pa s o he measu emen s om each pa ien , while p oduc- ing eliable and well gene alizing esul s when only a small amoun o indi iduals is a ailable in he aining da a. We will concen a e ou a en ion on wo pa icula se s o low cy ome y da a. The i s se o igina es om he acu e myeloid leukemia (AML) p edic ion challenge o he DREAM ini ia i e in 2013.4 The compe i ion a ac ed a numbe o eams, and as se e al esea che s use he da a as pa o hei wo k, he challenge da a ha e become a s anda d benchma k wi hin he ield. Fo example, Aghaeepou e al.4 p esen s a la ge pool o analysis app oaches om he DREAM challenge. Among classi ica ion me hods p esen ed in he li e a u e, he e a e se e al sophis ica ed machine lea ning app oaches, such as lea ning ec o quan iza ion,5 co ela i e ma ix map- ping, and ela i e en opy di e ences.6 The s eng h o da a- d i en app oaches elying on supe ised classi ica ion is hei abili y o handle high-dimensional da a wi hou equi ing p io knowledge o he biological applica ion. The DREAM AML da a ep esen a ela i ely la ge-scale expe imen consis ing o al oge he 179 pa ien s. Al hough i is a small numbe o adi ional machine lea ning p oblems, he numbe o pa ien s is unusually la ge o a biological s udy. To his aim, we use ano he da ase ha ep esen s a mo e commonly encoun e ed sample size o 16 samples ex ac ed om a p os a e cance cell line. This da ase , wi h wo di - e en ea men s and a low numbe o samples, p esen s a non i ial bu common challenge o p edic ion and ela ed ea u e selec ion. Mo e in o ma ion on he wo da ase s is p o ided in Da a sec ion. In ou ea lie wo k,7 we p esen ed a supe ised classi ica- ion pipeline based on linea disc iminan analysis (LDA) and logis ic eg ession (LR) classi ie s. B ie ly, he me hod i s Flow Cy ome y-Based Classi ica ion in Cance Resea ch: A View on Fea u e Selec ion s. saki a Hassan1, Pekka uusu uo i2,3, Leena La onen3 and Heikki Hu unen1 1Depa men o Signal P ocessing, Tampe e Uni e si y o Technology, Tampe e, Finland. 2Po i Depa men , Tampe e Uni e si y o Technology, Po i, Finland. 3BioMediTech, Uni e si y o Tampe e, Tampe e, Finland. Supplemen a y Issue: S a is ical Sys ems Theo y in Cance Modeling, Diagnosis, and The apy Abs Ac : In his pape , we s udy he p oblem o ea u e selec ion in cance - ela ed machine lea ning asks. In pa icula , we s udy he accu acy and s abili y o di e en ea u e selec ion app oaches wi hin simplis ic machine lea ning pipelines. Ea lie s udies ha e shown ha o ce ain cases, he accu acy o de ec ion can easily each 100% gi en enough aining da a. He e, howe e , we concen a e on simpli ying he classi ica ion models wi h and seek o ea u e selec ion app oaches ha a e eliable e en wi h ex emely small sample sizes. We show ha as much as 50% o ea u es can be disca ded wi hou comp omising he p edic ion accu acy. Mo eo e , we s udy he model selec ion p oblem among he  1 egula iza ion pa h o logis ic eg ession classi ie s. To his aim, we compa e a mo e adi ional c oss- alida ion app oach wi h a ecen ly p oposed Bayesian e o es ima o . Keywo ds: AML, leukemia, low cy ome y, logis ic eg ession, e o es ima ion, model selec ion SUPPLEMENT: s a is ical sys ems heo y in Cance modeling, Diagnosis, and he apy CITATIoN: Hassan e al. Flow Cy ome y-Based Classi ica ion in Cance Resea ch: A View on ea u e selec ion. Cance In o ma ics 2015:14(s5) 75–85 doi: 10.4137/CIn.s30795. TYPE: o iginal esea ch RECEIVED: no embe 18, 2015. RESUBMITTED: eb ua y 01, 2016. ACCEPTED FoR PUBLICATIoN: eb ua y 07, 2016. ACADEMIC EDIToR: J. T. E i d, Edi o in Chie PEER REVIEw: six pee e iewe s con ibu ed o he pee e iew epo . e iewe s’ epo s o aled 1860 wo ds, excluding any con iden ial commen s o he academic edi o . FUNDINg: Au ho s disclose no ex e nal unding sou ces. CoMPETINg INTERESTS: Au ho s disclose no po en ial con lic s o in e es . CoRRESPoNDENCE: [email p o ec ed] CoPYRIghT: © he au ho s, publishe and licensee Libe as academica Limi ed. his is an open-access a icle dis ibu ed unde he e ms o he C ea i e Commons CC-BY-NC 3.0 License. Pape subjec o independen expe blind pee e iew. all edi o ial decisions made by independen academic edi o . Upon submission manusc ip was subjec o an i- plagia ism scanning. P io o publica ion all au ho s ha e gi en signed con i ma ion o ag eemen o a icle publica ion and compliance wi h all applicable e hical and legal equi emen s, including he accu acy o au ho and con ibu o in o ma ion, disclosu e o compe ing in e es s and unding sou ces, compliance wi h e hical equi emen s ela ing o human and animal s udy pa icipan s, and compliance wi h any copy igh equi emen s o hi d pa ies. This jou nal is a membe o he Commi ee on Publica ion E hics (COPE). Published by Libe as academica. Lea n mo e abou his jou nal. Hassan e al 76 CanCe In o ma ICs 2015:14(s5) ans o ms he measu emen da a in o highe dimensional space by gene a ing combined ea u es wi h mul iplica ions and di isions be ween measu emen s. Following his mapping in o highe dimensional space, LDA is used o lowe ing he dimen- sionali y in o a single alue pe measu emen . Then, empi ical dis ibu ion unc ions (EDFs) a e cons uc ed om LDA esul s o AML-posi i e and AML-nega i e sample classes and com- pa ed o aining EDFs o bo h classes. The compa ison esul s in wo simila i y alues pe g oup o measu emen s, and hese esul s a e ed o he LR classi ie o a inal AML p edic ion esul . Ou app oach, oge he wi h al e na i e well-pe o ming app oaches,4,5 ep esen s a ela i ely complica ed pipeline o somewha a bi a y compu a ion s eps. Thus, ou in e es is o simpli y hese pipelines in o a simple collec ion o ob ious ea- u es, while s ill e aining a good accu acy. Ou app oach he e is o use LR classi ie applied o sum- ma ize ea u es, which a e he mean and s anda d de ia ion o he measu emen s ins ead o he comple e da a. This educes he numbe o ea u es used in classi ica ion and, subsequen ly, also he model complexi y. An essen ial pa o classi ie design is e o es ima ion, which guides model selec ion.8 Ou s a egy o model selec ion is o apply he ecen ly in oduced Bayesian e o es ima o (BEE).9 We compa e BEE model selec ion wi h a adi ional 10- old c oss- alida ion (CV-10) e o es ima ion, as well as wi h Bayesian in o ma ion c i e- ion (BIC)-based model selec ion, and conclude ha he p o- posed app oach enables accu a e p edic ion o low cy ome y da a wi h ewe measu emen s and a less complex classi ie model han hose p e iously p esen ed in he li e a u e. The es o his pape is o ganized as ollows. In Ma e ials and Me hods sec ion, we desc ibe he da a and me hods used in his s udy and b ie ly discuss how ea u e selec ion is com- monly done in machine lea ning. Expe imen al Resul s sec ion p esen s he esul s o ou expe imen s wi h di e en model and ea u e selec ion c i e ia o he ma e ials in oduced in Ma e ials and Me hods sec ion. Finally, in Conclusions sec- ion, we summa ize he wo k and discuss he conclusions o he esul s. Ma e ials and Me hods In his sec ion, we desc ibe he da ase s used in his pape . We also gi e a b ie o e iew o modeling me hod o ea u e selec ion. In addi ion o his, we in oduce he s a e-o - he-a Bayesian e o es ima o (BEE) o model pa ame e selec- ion. Finally, we p esen an example whe e he pe o mance o BEE is benchma ked agains o he model selec ion c i e ia. da a. In his wo k, we s udy wo da ase s: A la ge se wi h 179 samples and a smalle se wi h 16 samples. These wo case s udies ep esen di e en classi ica ion challenges in e ms o bo h applica ion and sample size. AML da ase . The low cy ome y da ase o he AML expe imen has been collec ed om he DREAM6- FlowCAP2 challenge, which was o ganized by he DREAM p ojec and he FlowCAP ini ia i e (DREAM challenge AML da ase can be accessed om Aghaeepou e al).4 We use he aining da ase ha consis s o low cy ome y measu emen s o 179 pa ien s. Among hem, 23 pa ien s a e AML posi i e and he emaining 156 pa ien s a e AML nega i e. The low cy ome y measu emen o each pa ien co esponds o se en join ly measu ed g oups (he ea e called ubes) o se en quan i ies wi h a o al o 49 bioma ke measu emen s pe cell. The bioma ke s a e summa ized in Table 1 and include Fo wa d Sca e in linea scale (FS Lin), Sidewa d Sca e in loga i hmic scale (SS Log), and i e luo- escence in ensi ies (FL1–FL5) in loga i hmic scales. Fo calib a ion pu poses, FS Lin, SS Log, and CD45-ECD we e measu ed o all ubes and he o he 28 bioma ke s we e measu ed only in one ube. Cance cell line da ase . As ano he case s udy, we use low cy ome y da a om a small sample se ing. The da a come om a p os a e cance cell line 22R 1 s ained wi h p opi dium iodide o cell cycle analysis.10 The cells a e ans ec ed wi h miRNAs (ei he con ol o miR-193b) and induced o p oli e a e by o e exp ession o cyclin D. The da a consis o 16 samples, wi h 8 samples (wi hou cyclin D o e exp ession) wi h ela i ely consis en cell cycle p o ile and 8 samples (o e exp essing cyclin D) wi h an al e ed cell cycle p o ile, ie, induced cell cycle ac i i y wi h an inc ease in cells in DNA syn hesis phase. The samples o bo h classes include ou epe i ions o wo ea men s, which a e consid- e ed he e o ep esen he same class. Each sample con ains 12 measu ed channels, consis ing o wo sca e measu e- men s and ou luo escence channels, bo h as a ea and heigh measu emen s. Table 1. Lis o se en ubes wi h bioma ke s p o ided in DREAM6 AML p edic ion da a. FL1 Log FL2 Log FL3 Log FL4 Log FL5 Log ube 1 s Lin ss Log IgG1- I C IgG1-Pe CD45-eCD IgG1-PC5 IgG1-PC7 ube 2 s Lin ss Log Kappa- I Lambda-Pe CD45-eCD CD19-PC5 CD20-PC7 ube 3 s Lin ss Log CD7- I C CD4-Pe CD45-eCD CD8-PC5 CD2-PC7 ube 4 s Lin ss Log CD15- I C CD13-Pe CD45-eCD CD16-PC5 CD56-PC7 ube 5 s Lin ss Log CD14- I C CD11c-Pe CD45-eCD CD64-PC5 CD33-PC7 ube 6 s Lin ss Log HLa-D - I C CD117-Pe CD45-eCD CD34-PC5 CD38-PC7 ube 7 s Lin ss Log CD5- I C CD19-Pe CD45-eCD CD3-PC5 CD10-PC7 Flow cy ome y-based classi ica ion in cance esea ch 77CanCe In o ma ICs 2015:14(s5) Fea u e ex ac ion. Se e al ea u e ex ac ion me hods can be used o ob ain meaning ul ea u es om aw low cy ome y measu emen s. Fo ins ance, among widely used ea u e ex ac ion echniques a e me hods based on p incipal componen analysis and his og am compu a- ion. Biehl e al p oposed s a is ical di e gences o ex ac ea u es ha include momen s, median, and in e qua ile ange.5 The leng h o he ea u e ec o was 186 in his case. Ano he well-pe o med model was based on mul idimen- sional en opic dis ance-based ea u es.4,5 Manninen e al.7 expanded he cell measu emen s o each ube o a highe dimensional space. Following his ans o ma ion, LDA is used o lowe he dimensionali y in o a single alue o each measu emen . These p e ious s udies a e summa ized in Table 2. Table 2 also abula es he es accu acy in e ms o he a ea unde he ecei e ope a ing cha ac e is ics (ROC) cu e (AUC) measu e o e a single ain/ es spli , which should no be in e p e ed as a de ini i e measu e o accu acy, as he spli o he samples is jus one ins ance o all possible spli s. In his pape , we use one o he simples ea u e ex ac ion echniques ha include only he mean and he s anda d de ia- ion o he each measu emen . Fo he i s da ase , he leng h o his ex ac ed ea u e ec o is 98, comp ising 49 mean alues and 49 s anda d de ia ions. As seen in he expe imen s o Expe imen al Resul s sec ion, hese ea u es a e su icien o sepa a e he classes wi hou comp omising he p edic ion accu acy. We will conside wo e sions o hese basic ea- u es: he i s ea u e se con ains only he 49 mean alues o he measu emen s, while he second ea u e ec o conside s bo h mean alues and s anda d de ia ions, wi h al oge he 98 ea u es. The same app oach is used wi h he smalle da a- se , hus p oducing wo di e en expe imen al cases. Be o e aining he classi ie s, we no malized all ea u es o he in e - al (0, 1). L and egula iza ion. LR is a disc imina i e me hod o modeling he class condi ional p obabili y densi ies by he logis ic unc ion. Gi en an obse a ion ma ix wi h N obse a ions, P ea u es, and co espond- ing class labels y ∈1,…,C, we de ine LR model o he bina y classi ica ion as, (1) and (2) He e, x ep esen s a ea u e ec o in he ea u e space co esponding o class label y ∈ {0,1}, β0 is he in e cep , and β ep esen s coe icien s o he logis ic model. We can de e mine he model pa ame e s β0 and β om he aining da ase by sol ing he  1 penalized LR p oblem, (3) whe e λ . 0 is he egula iza ion pa ame e . When he num- be o aining da a is no la ge compa ed o he numbe o ea u es, ie, P  N, egula iza ion is used o sol e he o e i - ing p oblem.11 In egula iza ion, an ex a e m, λ, is added, which con ols he ade-o be ween he loss unc ion and he size o he coe icien s. Mo e ecen ly, in ea u e selec- ion,  1- egula ized LR has ecei ed much a en ion, as i yields a spa se solu ion ha has ela i ely ew nonze o coe i- cien s.12 This minimiza ion ask is analogous o leas absolu e sh inkage and selec ion ope a o (Lasso) algo i hm p oposed by Tibshi ani.13 In addi ion o his, se e al ex ensions o Lasso ha e also been de eloped, such as g ouped Lasso,14,15 Dan zig selec o ,16 elas ic ne ,17 and g aphical Lasso.18 In his pape , Table 2. S udies based on ea u e ex ac ion s a egies o DREAM amL challenge da ase . ACCURACY SIzE oF FEATURE VECToR BRIEF DESCRIPTIoN Biehl e al.51.00 186 Ex ac ion o ea u es wi h momen s, median and in e qua ile and lea ning ec o quan iza ion is used o p edic ion Vila e al.41.00 31 Ex ac ion o ea u es wi h en opies and his og am based classi ie is used o p edic ion manninen e al.7 1.00 (# o e en s) x 84 Expand ea u es o highe dimension and hen mapping o 1-D using linea disc iminan analysis; logis ic eg ession is used o p edic ion ou solu ion ( his s udy) 0.9989 49 Ex ac ion o ea u e ec o om means o measu emen s and applying egula ized logis ic eg ession o p edic ion ou solu ion ( his s udy) 0.9992 98 Ex ac ion o ea u e ec o om means and s anda d de ia ion o measu emen s and applying egula ized logis ic eg ession o p edic ion No e: The accu acy is measu ed in e ms o he AUC o a single ain/ es spli . Hassan e al 78 CanCe In o ma ICs 2015:14(s5) we use he GLMNET algo i hm by F iedman e al.19 ha combines he  2 and  1 penal ies: (4 ) whe e λ . 0 and α ∈[0,1]. The pa ame e α is a comp omise be ween he  1 and  2 penal ies, he eby de e mining he ype o egula iza ion. On he o he hand, he egula iza ion pa am- e e λ con ols he amoun o egula iza ion. A e y la ge λ will comple ely sh inks he coe icien s o ze o and may yield a null o emp y model. In gene al, he model pa ame e s λ and α a e selec ed using he CV app oach.20 The da ase is andomly spli in o K mu u- ally exclusi e subse s o app oxima ely equal sized. In K- old CV, he p ocess is i e a ed k imes. A he k h i e a ion, he K h old is e ained as es se and he emaining K − 1 olds a e used as aining se o ain he model. Each o he K- olds is es ed exac ly once. The es se assesses he quali y o he ained model. Then, he K esul s a e combined o a e aged o p oduce a single es i- ma ion o he model. The mos commonly used alues o K a e 5 and 10. In his expe imen , we se he alue o α = 1 and CV-10 is used o he selec ion o he model pa ame e λ and assessmen o he model. As he ype o egula iza ion is de e mined by α, se ing α = 0 p o ides  2 penal y ha is use ul in cases, whe e he ea u es a e mu ually co ela ed. On he o he hand, α = 1 p o- ides spa se solu ion wi h ewe coe icien s and, in u n, his is sui able o implici ea u e selec ion. We ha e also expe imen ed wi h 5- old CV, bu he esul s do no imp o e signi ican ly. bayesian e o es ima o . A Bayesian app oach o e o es ima ion was ecen ly in oduced in he con ex o disc e e classi ie s21 and linea classi ie s.22 The Bayesian e o es ima- o (BEE) es ima es he classi ica ion e o di ec ly om he aining se and has shown o imp o e bo h he accu acy and speed o he ac ual e o es ima e21,22 compa ed o adi ional coun ing-based app oaches, such as CV. In ou ea lie pape s, we ha e shown ha BEE has imp o ed he s abili y and speed o compu a ion in he model selec ion con ex as well.9,23 We will nex b ie ly e iew he de ini ion o BEE o a ixed linea wo-class classi ie speci ied by he pa ame e s β and β0. The Bayesian e o es ima o o linea classi ica ion assumes ha he samples om each class a e independen and iden ically dis ibu ed Gaussian andom a iables. Fo he wo classes, he pa ame e s (mean and co a iance) o he Gaussian model a e deno ed as θ0 and θ1 and he co esponding p io s o he pa a me e s a e deno ed as p0(θ) and p1(θ). Then, he pos e io p obabili y densi y unc ions (PDFs) o pa ame e s o class c ∈{0,1} a e gi en by he Bayes’ ule: (5) whe e is he Gaussian class condi ional densi y o c ∈{0,1}. The Bayesian e o es ima o (BEE) is de ined as he minimum mean squa ed es ima o by minimizing he expec a- ion be ween he e o es ima e and he ue e o . This quan- i y is composed o class-speci ic condi ional expec ed e o s balanced by he p io s p(c) o he wo classes c ∈{0,1}22: (6) wi h he expec ed classi ica ion e o o samples om class c gi en by (7) whe e εc(θ) deno es he ue classi ica ion e o . The in eg al o Equa ion (7) can be e alua ed by assuming an in e se Wisha p io o he class condi ional densi y: (8) whe e υ ∈∈∈∈ × ,κ,Sm PP , and P a e he hype pa am- e e s o he Bayesian model and is he pa ame e o he Gaussian dis ibu ion. Di e en choices o he alues o hype pa ame e s lead o di e en e o es ima o s, bu we will concen a e on a speci ic choice shown o be success ul in ea lie wo ks9,22: κ = P + 2, ν = 0.5, s = I, and m = 0. Fo he esul ing simpli ied closed- o m solu ion, e e o Re .9 Ma lab and Py hon implemen a ions o BEE a e a ailable o down- load (h ps://si es.google.com/si e/bayesiane o es ima e/). Model selec ion. Model selec ion is a c i ical aspec in classi ie design. Mo eo e , mos mode n classi ie s a e uned by a se o hype pa ame e s, whose selec ion has a subs an ial e ec on he esul ing accu acy as well. Thus, he selec ion o an app op ia e model amily and he associa ed hype pa a- me e s equi es an accu a e measu e o compa ing he accu- acies o he model candida es. In ou wo k, we a e p ima ily in e es ed in he selec ion o he egula iza ion pa ame e λ o an LR classi ie . Howe e , i is o be no ed ha he me hodol- ogy applies o any linea classi ie . The p edic ion accu acy and selec ion o he bes model can be quan i ied by e o es ima o s. CV es ima o is o en used o selec he bes alue o he model selec ion pa ame e λ along a egula iza ion pa h. As an example, e o cu es o di e en alues o λ a e illus a ed in Figu e 1. Fo his pu pose, we used he low cy ome y aining da a o 49 ea u es and 179 obse a ions. Fo an indi idual ube, each ea u e ep esen s he a e age o he bioma ke in ensi ies. The e o cu es a e es i- ma ed o di e en alues o λ anging om 10−9 o 100. In he example o Figu e 1 (le panel), a 5- old CV p o- cedu e is epea ed 100 imes and each s ep includes i e ain- ing i e a ions on pa ial da a. The e o cu es ob ained o 100 i e a ions o 5- old CV illus a e he signi ican de ia ion Flow cy ome y-based classi ica ion in cance esea ch 79CanCe In o ma ICs 2015:14(s5) o he egula iza ion pa hs om one i e a ion o ano he . The de ia ion is due o he andomness in spli ing he aining da a in o olds, which esul s in an indi idual e o es ima e o each spli . Mo eo e , o a e y small numbe o samples, such as 5 o 10, he spli o alida ion and aining se s o he K- old CV es ima o may no be app op ia e. In ac , in his expe imen , he K- old CV app oach ails o es ima e he e o s o smalle λ, as he numbe o samples spli by CV is insu i- cien . On he o he hand, Figu e 1 ( igh panel) illus a es he e o es ima e o BEE, which is a single de e minis ic e o cu e. I is o be no ed ha he cu e ecognizes model o e - i ing (e o es ima e s a s o inc ease o small egula iza- ion e ms λ), al hough he e o is es ima ed di ec ly om he aining se . No spli ing o i e a i e esampling is equi ed, which in u n accele a es he compu a ion. expe imen al esul s In he ollowing sec ion, we p esen he expe imen al esul s. Fi s , we demons a e di e en model selec ion c i e ia o es i- ma e he signi ican ea u es. Then, we assess he pe o mances o hose me hods in he AML classi ica ion case. Finally, we p esen he esul s o he second, small sample case. compa ison o model selec ion c i e ia. Typical app oach o he selec ion o model pa ame e is CV.13 In his pape , we also conside Bayesian e o es ima o (see Bayesian E o Es ima o sec ion) and BIC24 as al e na i e app oaches o es ima e he egula ized pa ame e . In o de o s udy he beha io o di e en pa ame e selec ion c i e ia, we i s ain he LR classi ie wi h he aining da a along he dec easing sequence o egula iza ion pa h wi h log10 (λ)∈{0,–0.05,–0.1, –0.15,…, –8.90, –8.95, –9.00}. Then, again he whole aining da a a e used o es ima e he e o a e o each λ. Finally, o each es ima o , we selec he model wi h λ alue ha achie es he minimum e o a e. As esampling in CV-10 in oduces andomness, in his case, we i e a e 200 imes and he esul is a e aged. The de e minis ic na u e in BEE and BIC will p oduce he same esul on he aining da a a each i e a ion. The esul s a e summa ized in Table 3. Fo all me hods, minimum e o a es, AUC, and he numbe o selec ed ea- u es a e es ima ed om he whole aining da a. I is o be no ed ha he epo ed AUC is compu ed om he aining o emphasize ha all ea u e se s a e enough o pa i ion he ea u e space in o wo ca ego ies pe ec ly. The es e o is epo ed la e . The esul s indica e ha he numbe o ea u es selec ed by BEE me hod is lowe compa ed o hose o CV and BIC. Fo he i s ea u e ec o wi h size 49, BEE selec s only 14 ea u es as signi ican , while o he second ea u e ec o wi h −6 −5 −4 −3 −2 −1 0 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Log10 (λ) 5− old c oss alida ion e o −6 −5 −4 −3 −2 −1 0 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 Log10 (λ) BEE Figu e 1. Le : Examples o egula iza ion pa h e o cu es o 5- old CV o ou low cy ome y da a wi h heal hy and AML posi i e classes. Righ : The co esponding Bee cu e. Table 3. Pa ame e selec ion by di e en es ima o s: a e age numbe o selec ed ea u es, λ, aUC, and hei s anda d de ia ions wi h aining da a. METhoD FEATURE TYPE NUMBER oF SELECTED FEATURES SELECTED Log10 (λ) AUC (TRAININg) CV-10 mean 19.72 ± 2.41 −2.95 ± 2.30 0.9997 ± 0.0017 CV-10 mean and s d 23.91 ± 0.80 −4.14 ± 3.00 1 ± 0.0000 Bee mean 15 ± 0.00 −2.05 ± 0.00 0.9989 ± 0.0000 Bee mean and s d 13 ± 0.00 −1.80 ± 0.00 0.9992 ± 0.0000 BIC mean 20 ± 0.00 −5.85 ± 0.00 1 ± 0.0000 BIC mean and s d 24 ± 0.00 −5.70 ± 0.00 1 ± 0.0000 Hassan e al 80 CanCe In o ma ICs 2015:14(s5) leng h 98, BEE selec s only 12 ea u es. Tables 4 and 5 lis he selec ed ea u es, ie, signi ican bioma ke s along wi h he co esponding coe icien alues. Due o he andomness in CV-10, we only p esen he esul s o one i e a ion as an illus- a ion: he e is a signi ican a ia ion o he selec ed ea u es depending on he chosen CV spli . Howe e , i is o be no ed ha he coe icien s o BIC and BEE a e no speci ic o his pa icula i e a ion, as hey do no include he andom spli . Pe o mance assessmen o he model selec ion c i e ia. The pe o mances o he model selec ion me hods a e s udied in he ollowing sec ion. The classi ica ion e o is conside ed as he pe o mance c i e ion, and bo h alse posi i es (heal hy con ol classi ied as AML) and alse nega i es (AML classi ied as heal hy con ol) a e coun ed wi h equal weigh . The pe o - mance o he Bayesian e o es ima o is benchma ked agains hose o CV-10 and BIC o a di e en numbe o sample sizes. Fo his pu pose, a andomly selec ed p opo ion o 10%, 15%, 20%–90%, and 95% is selec ed o aining he classi ie , while he emaining da a a e used o pe o mance assessmen . Fo each aining sample, he expe imen is execu ed 200 imes by gene a ing a new aining se each ime. Classi ica ion e o s. The e o cu es o di e en sample sizes a e shown in Figu e 2. The p ocedu e is epea ed 200 imes o each aining sample size, and he a e age o clas- si ica ion e o is compu ed o each model selec ion c i e ion. Wi h a e y small numbe o aining samples, such as 10% o 15% o he da ase , BEE p o ides imp o ed accu acy o e CV-10 and BIC (Fig. 2 le and igh panels). Fo ins ance, wi h 10% aining samples, he classi ica ion e o s o he model selec ed by CV-10 a e 7.56% (Fig. 2 le panel) and 7.41% (Fig. 2 igh panel) highe han hose o BEE. In case o BIC, he classi ica ion e o is 7.81% highe han ha o BEE (Fig. 2 le panel). As he numbe o aining samples inc eases, o example, abo e 60%, he pe o mance o BIC exceeds han ha o BEE (Fig. 2 le panel). Howe e , he pe o mance o BIC is simila o ha o BEE when mo e ea- u es a e in ol ed in he expe imen (Fig. 2 igh panel). Table 4. The nonze o coe icien s o ea u es wi h mean. TUBE FEATURE 10-FoLD CV BEE BIC Cons an −13.38 −3.50 −13.54 ube 1 sLin 0.75 00.78 ube 1 ssLog −5.92 −0.73 −6.01 ube 1 L1:IgG1- I C −0.46 −0.19 −0.46 ube 1 L4:IgG1-PC5 −2.08 0−2.13 ube 1 L5:IgG1-PC7 −3.07 −0.19 −3.14 ube 2 sLin 0.001 0 0 ube 2 L5:CD20-PC7 2.76 02.78 ube 3 ssLog 0−0.77 0 ube 3 L4:CD8-PC5 −1.94 −0.10 −1.97 ube 4 sLin 0.97 00.97 ube 4 L1:CD15 - I C −4.77 0−4.82 ube 4 L2:CD13-Pe 3.44 0.21 3.48 ube 4 L4:CD16-PC5 0−0.09 0 ube 4 L5:CD56-PC7 3.29 0.75 3.35 ube 5 sLin 2.35 02.40 ube 5 L2:CD11c-Pe −0.15 0−0.16 ube 5 L3:CD45-eCD −1.84 −0.02 −1.84 ube 5 L4:CD64-PC5 1.66 01.69 ube 5 L5:CD33-PC7 0.75 0.60 0.76 ube 6 L2:CD117-Pe 4.82 0.89 4.88 ube 6 L4:CD34-PC5 6.88 0.72 6.99 ube 6 L5:CD38-PC7 00.41 0 ube 7 L1:CD5 - I C −4.68 −0.19 −4.74 No e: The size o he ea u e se s is 49, o which he CV, BEE, and BIC selec 20, 14, and 19 ea u es, espec i ely. Table 5. The nonze o coe icien s o ea u es wi h mean and s anda d de ia ion. TUBE FEATURE 10-FoLD CV BEE BIC Cons an −13.47 −3.27 −13.47 ube 1 sLin mean 0.027 00.027 ube 1 ssLog mean −4.49 −0.47 −4.49 ube 1 L1:IgG1- I C s d −3.80 −0.22 −3.80 ube 1 L5:IgG1-PC7 s d −1.60 −0.16 −1.60 ube 2 L5:CD20-PC7 mean 0.50 00.50 ube 3 ssLog mean 0−0.29 0 ube 3 L5:CD2-PC7 mean −0.48 0−0.48 ube 3 L5:CD2-PC7 s d −1.21 0−1.21 ube 4 sLin mean 0.05 00.05 ube 4 L1:CD15 - I C mean −0.14 0−0.14 ube 4 L2:CD13-Pe mean 2.50 02.50 ube 4 L4:CD16-PC5 mean −1.32 0−1.32 ube 4 L4:CD16-PC5 s d 0−0.39 0 ube 4 L5:CD56-PC7 s d 6.07 0.45 6.07 ube 5 L1:CD14- I C s d −0.004 0−0.004 ube 5 L3:CD45-eCD mean −2.51 0−2.51 ube 5 L5:CD33-PC7 mean 1.47 0.32 1.47 ube 5 L5:CD33-PC7 s d −0.70 0−0.70 ube 6 ssLog s d 0−0.23 0 ube 6 L2:CD117-Pe mean 3.19 0.46 3.19 ube 6 L2:CD117-Pe s d 2.19 02.19 ube 6 L4:CD34-PC5 s d 1.88 0.75 1.88 ube 6 L5:CD38-PC7 mean 1.31 0.44 1.31 ube 7 sLin mean 1.39 01.39 ube 7 sLin s d 0−0.16 0 ube 7 L1:CD5 - I C s d −1.16 0−1.16 ube 7 L5:CD10-PC7 s d 0.04 00.04 No e: The size o he ea u e se s is 98, o which he CV, BEE, and BIC selec 23, 12, and 23 ea u es, espec i ely. Flow cy ome y-based classi ica ion in cance esea ch 81CanCe In o ma ICs 2015:14(s5) A ea unde he ROC cu e. In his sec ion, we e alua e he pe o mance in e ms o AUC. Figu e 3 illus a es he a e age o AUC o di e en aining sample sizes. He e, he BEE me hod achie es imp o emen o e he o he me hods. Wi h small ain- ing samples, o ins ance, 10%, he a e age AUC o BEE is 1.11% (Fig. 3 le panel) and 1.30% (Fig. 3 igh panel) highe han ha o CV-10. As he numbe o aining samples inc eases, CV-10 and BIC also con e ge owa d he esul s o BEE; howe e , he BEE selec ed model consis en ly esul s in he highes AUC sco e. Wi h he la ge ea u e ec o ha includes he measu e- men s o mean and s anda d de ia ion, he a e age AUC cu es o BEE and BIC ollow he simila pa e n (Fig. 3 igh panel). Numbe o selec ed ea u es. We u he assess he pe - o mances o he es ima o s using ea u e selec ion c i e ia. A each i e a ion, we de e mine he o al numbe o selec ed ea u es ha ha e nonze o alues o a di e en numbe o aining samples. Then, we compu e he a e age and he a iabili y (ie, s anda d de ia ion) o he selec ed ea u es o di e en aining samples. The esul s a e illus a ed in Figu e 4. Fo BEE, he a e age numbe o selec ed ea u es is lowe in amoun compa ed o hose o CV and BIC (Fig. 4 op panel). Fo ins ance, wi h 95% aining samples, BEE equi es 36.49% and 33.89% less ea u es han CV and BIC, espec i ely, o model p edic ion (Fig. 4 op- igh panel). Mo eo e , he a iabili y in selec ed ea u es using BEE is also compa able (Fig. 4 bo om panel). The CV-10 has he wo s pe o mance. Al hough BIC shows ha he de ia- ion in ea u e selec ion a di e en i e a ions is smalle , he 0 20 40 60 80 100 120 140 160 180 0.02 0.03 0.04 0.05 0.06 0.07 Numbe o samples used o aining Classi ica ion e o s CV-10 BEEp BIC 0 20 40 60 80 100 120 140 160 180 0.02 0.03 0.04 0.05 0.06 0.07 Numbe o samples used o aining Classi ica ion e o s CV-10 BEEp BIC Figu e 2. The a e age classi ica ion e o cu es o CV-10, BEE wi h p ope p io (BEEp), and BIC. No es: Le : ea u e ec o o mean alues o measu emen s. Righ : ea u e ec o o mean and s anda d de ia ion o measu emen s. 0 20 40 60 80 100 120 140 160 180 0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 Numbe o samples used o aining AUC CV-10 BEEp BIC 0 20 40 60 80 100 120 140 160 180 0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 Numbe o samples used o aining AUC CV-10 BEEp BIC Figu e 3. The a e age AUCs o CV-10, BEE wi h p ope p io (BEEp), and BIC. No es: Le : ea u e ec o o mean alues o measu emen s. Righ : ea u e ec o o mean and s anda d de ia ion o measu emen s. Hassan e al 82 CanCe In o ma ICs 2015:14(s5) a e age numbe o selec ed ea u es is highe han ha o o he s (Fig. 4 op panel). Simila i y o he selec ed ea u e se s. Ano he pe o mance measu emen is he s abili y o selec ing he same ea u e a di e en i e a ions. Fo his pu pose, Sø ensen–Dice coe - icien 25 is used, which measu es he deg ee o simila i y be ween selec ed ea u es o wo di e en i e a ions. The anges can a y om 0 o 1. The alues closes o 1 indica e a high-deg ee o simila i y. Fo di e en aining samples, we i s de e mine which ea u es a e selec ed a each i e a ion. As he model selec ion p ocess is epea ed 200 imes, we es ima e he simila i y as he mean dice coe icien o each o he 200!/ (2! × (200 − 2)!) = 19,900 possible pai s o selec ed ea u e se s. The esul s a e shown in Figu e 5. In e ms o s abil- i y, he pe o mance o BEE is subs an ially be e han hose o he o he me hods, as he selec ed ea u e se s a e mos simila wi h ha c i e ion – a signi ican issue when ying o unde s and he biological mechanisms behind he da a. Fo example, wi h 60% aining samples, he dice coe icien o BEE is 6.03% highe han ha o CV (Fig. 5 le panel). On he o he hand, wi h 90% aining samples, he dice coe - icien o BEE is 5.81% highe han ha o CV and 4.90% highe han ha o BIC (Fig. 5 igh panel). Indeed, he dice coe icien is un a o able o CV wi h small aining samples: The dice coe icien is lowes among he al e na i es, indica - ing ha he selec ed ea u e se s wi h he CV c i e ion ha e high a iabili y. small sample case wi h a cance cell line. Fo u he con idence on he p esen ed me hod, we analyze da a om a cance cell line in a small sample se ing. As desc ibed p e i- ously, we conside ed he classi ica ion accu acy, AUC mea- su e, and he numbe o selec ed a iables bo h wi h and wi hou s anda d de ia ion ea u es (Figs. 6–8). In his case, 0 20 40 60 80 100 120 140 160 180 8 10 12 14 16 18 20 22 24 Numbe o samples used o aining A e age numbe o ea u es CV-10 BEEp BIC 0 20 40 60 80 100 120 140 160 180 8 10 12 14 16 18 20 22 24 Numbe o samples used o aining A e age numbe o ea u es CV-10 BEEp BIC 0 20 40 60 80 100 120 140 160 180 1 1.5 2 2.5 3 3.5 4 Numbe o samples used o aining S anda d de ia ion o numbe o ea u es CV-10 BEEp BIC 0 20 40 60 80 100 120 140 160 180 1.5 2 2.5 3 3.5 4 4.5 Numbe o samples used o aining S anda d de ia ion o numbe o ea u es CV-10 BEEp BIC Figu e 4. Compa isons o he numbe o selec ed ea u es o CV-10, BEE wi h p ope p io (BEEp), and BIC. No es: Le : ea u e ec o o mean alues o measu emen s. Righ : ea u e ec o o mean and s anda d de ia ion o measu emen s. Top: a e age numbe o selec ed ea u es. Bo om: s anda d de ia ion o numbe o he selec ed ea u es. Flow cy ome y-based classi ica ion in cance esea ch 83CanCe In o ma ICs 2015:14(s5) 0 20 40 60 80 100 120 140 160 180 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Numbe o samples used o aining Dice index CV-10 BEEp BIC 0 20 40 60 80 100 120 140 160 180 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Numbe o samples used o aining Dice index CV-10 BEEp BIC Figu e 5. Compa ison o he s abili y o selec ing ea u es o CV-10, BEEp, and BIC. No es: Le : ea u e ec o o mean alues o measu emen s. Righ : ea u e ec o o mean and s anda d de ia ion o measu emen s. 2 4 6 8 10 12 14 0.2 0.25 0.3 0.35 0.4 0.45 Numbe o samples used o aining Classi ica ion e o s LOOCV BEEp BIC 2 4 6 8 10 12 14 0.2 0.25 0.3 0.35 0.4 0.45 Numbe o samples used o aining Classi ica ion e o s LOOCV BEEp BIC Figu e 6. The a e age classi ica ion e o cu es o LOOCV, BEEp, and BIC. No es: Le : ea u e ec o o mean alues o measu emen s. Righ : ea u e ec o o mean and s anda d de ia ion o measu emen s. 2 4 6 8 10 12 14 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 Numbe o samples used o aining AUC LOOCV BEEp BIC 2 4 6 8 10 12 14 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Numbe o samples used o aining AUC LOOCV BEEp BIC Figu e 7. The a e age AUC cu es o LOOCV, BEEp, and BIC. No es: Le : ea u e ec o o mean alues o measu emen s. Righ : ea u e ec o o mean and s anda d de ia ion o measu emen s.