Comparing Feature Selection Methods in Clinical Data Modeling: LASSO, Ridge, Elastic Net, SCAD, Boruta, Best Subset, and Stepwise Regression

Compa ing Fea u e Selec ion Me hods in Clinical Da a Modeling: LASSO, Ridge, Elas ic Ne , SCAD, Bo u a, Bes Subse , and S epwise Reg ession Yumeng Wang1, Iulia C is ina Tudo 1* 1Co cep The apeu ics, Redwood ci y, CA 94065 *Co esponding au ho : c udo @co cep .com Abs ac Accu a e a iable selec ion is essen ial o p edic i e modeling in clinical esea ch, ye guidance on me hod choice unde a ying da a condi ions is limi ed. We sys ema ically e alua ed s epwise eg ession, bes subse selec ion, Bo u a, and ou egula iza ion-based me hods (LASSO, Ridge, Elas ic Ne , SCAD) using simula ed da ase s and a p ospec i e s udy o endogenous hype co isolism in pa ien s wi h di icul - o-con ol ype 2 diabe es. Key me ics included p edic i e pe o mance, pa ame e es ima ion, and ea u e selec ion accu acy. S epwise and bes subse me hods pe o med well in low-dimensional, low-collinea i y se ings, o e ing in e p e able and pa simonious models wi hou hype pa ame e uning. In medium- o-high dimensional o highly co ela ed se ings, egula ized me hods yielded mo e s able and accu a e p edic ions, while Bo u a iden i ied a complemen a y se o ea u es o e looked by linea models. These esul s p o ide p ac ical guidance o choosing a iable selec ion me hods and sugges ha combining complemen a y app oaches can enhance bo h ea u e disco e y and in e p e abili y in clinical da ase s. Key Wo ds: LASSO, s epwise eg ession, simula ion, hype co isolism. 1. In oduc ion Fea u e selec ion is a c i ical s ep in clinical da a modeling, ye he e emains a lack o p ac ical guidance on when o use speci ic me hods. Each app oach has i s own s eng hs and limi a ions, making he choice highly con ex -dependen . Clinical da ase s o en p esen unique challenges: a la ge numbe o candida e p edic o s, many o which may be i ele an o noisy; ela i ely small sample sizes due o he high cos o pa ien ec ui men ; and s ong co ela ions among a iables. These ac o s inc ease he isk o o e i ing and poo gene alizabili y i no p ope ly add essed. Mo eo e , in e p e abili y is essen ial in clinical esea ch, which explains why linea eg ession emains widely used despi e he a ailabili y o mo e complex modeling echniques. Taken oge he , hese cha ac e is ics unde sco e he impo ance o obus ea u e selec ion s a egies in building eliable and clinically meaning ul models. A commonly used classical app oach o a iable selec ion is s epwise eg ession, which i e a i ely adds o emo es p edic o s based on in o ma ion c i e ia o hypo hesis es s. While s epwise me hods ha e been widely applied in clinical esea ch o hei simplici y and in e p e abili y, hey a e known o su e om se e al limi a ions in high-dimensional se ings. These include model ins abili y, and a p opensi y o o e i , pa icula ly when he numbe o ea u es (p) is la ge ela i e o he sample size (n) (De ksen & Keselman, 1992; Ha ell, 2015). Ano he commonly used classical app oach is bes subse eg ession, which a emp s o iden i y he combina ion o p edic o s ha p o ides he bes i acco ding o a c i e ion such as AIC, BIC, o adjus ed R2. Unlike s epwise p ocedu es, bes subse eg ession conduc s an exhaus i e sea ch ac oss all possible p edic o combina ions, which gua an ees ha he iden i ied model is globally op imal unde he chosen me ic. The main ad an ages o his app oach a e i s in e p e abili y and anspa ency. Howe e , i quickly becomes compu a ionally p ohibi i e as he numbe o p edic o s g ows, and like s epwise eg ession, i is p one o ins abili y and o e i ing in small-sample, high- dimensional se ings (Mille , 2002). To add ess hese challenges, egula iza ion me hods ha e eme ged as powe ul al e na i es by imposing penal ies on model complexi y. The Leas Absolu e Sh inkage and Selec ion Ope a o (LASSO) (Tibshi ani, 1996) applies an L1 penal y, encou aging spa si y by sh inking some coe icien s exac ly o ze o and he eby pe o ming a iable selec ion. Ridge eg ession (Hoe l & Kenna d, 1970), which uses an L2 penal y, sh inks coe icien s owa d ze o bu e ains all p edic o s, o e ing imp o ed s abili y when p edic o s a e highly co ela ed, as co ela ed ea u es end o sh ink oge he . The Elas ic Ne (Zou & Has ie, 2005) combines L1 and L2 penal ies, p o iding a balance be ween spa si y and s abili y, and is pa icula ly e ec i e when g oups o co ela ed ea u es a e p esen . In addi ion, noncon ex penal ies such as he Smoo hly Clipped Absolu e De ia ion (SCAD) (Fan & Li, 2001) ha e been p oposed o o e come he es ima ion bias in oduced by LASSO, especially o p edic o s wi h la ge ue e ec s, while s ill p oducing spa se and in e p e able models. Beyond eg ession-based me hods, ea u e selec ion app oaches le e aging ee-based models ha e gained popula i y in clinical and biomedical esea ch. Me hods such as Bo u a (Ku sa & Rudnicki, 2010) build upon Random Fo es s o assess he impo ance o each a iable in p edic ing he ou come. Bo u a i e a i ely compa es he impo ance o eal ea u es o ha o pe mu ed ea u es, selec ing only hose ha consis en ly show s onge p edic i e powe . The ad an age o ee-based app oaches lies in hei abili y o na u ally cap u e nonlinea i ies and in e ac ions among p edic o s, wi hou he need o s ong modeling assump ions. Howe e , hey o en sac i ice in e p e abili y compa ed o linea models. Taken oge he , hese di e se me hods highligh he ade-o s be ween in e p e abili y, s abili y, and p edic i e pe o mance ha a e cen al o ea u e selec ion in clinical da a modeling. Despi e he wide a ay o a ailable app oaches, he e emains limi ed p ac ical guidance o clinical esea che s on when o use which me hod. This s udy aims o ill ha gap by sys ema ically compa ing s epwise eg ession, bes subse eg ession, LASSO, Ridge, Elas ic Ne , SCAD, and Bo u a, using bo h simula ed da a and eal clinical da ase s. 1.1 S epwise Reg ession ia F-Tes -Based Fea u e Selec ion Va iable selec ion in s epwise eg ession is based on null hypo hesis es ing, mos commonly h ough he F- es . In o wa d selec ion, he p ocedu e begins wi h an emp y model (o only he in e cep ). A each s ep , he imp o emen in model i om adding a candida e p edic o is assessed using he pa ial F-s a is ic. The p edic o wi h he smalles p- alue is added, p o ided i alls below a p especi ied inclusion h eshold α. A la ge α inc eases he likelihood o including mo e p edic o s bu also aises he alse posi i e a e. In con as , backwa d elimina ion s a s wi h he ull model and sequen ially emo es he p edic o wi h he la ges p- alue, con inuing un il no a iable exceeds he elimina ion h eshold. The bidi ec ional (s epwise) p ocedu e combines bo h app oaches, allowing a iables o be added o emo ed a each s ep, so ha p edic o s p e iously included can la e be excluded. The algo i hm e mina es once no u he p edic o s mee he en y o emo al c i e ia, yielding a inal model de e mined by sequen ial hypo hesis es ing. 1.2 Bes subse eg ession In bes subse eg ession, he goal is o iden i y he model o size 𝒌𝒌∈{𝟏𝟏,⋯,𝒑𝒑} ha minimizes a chosen selec ion c i e ion, such as AIC, BIC, Mallows’Cp, o adjus ed R2. A naï e implemen a ion equi es i ing all 2p possible models, which is compu a ionally in easible when p is la ge. To add ess his, he “leaps and bounds” algo i hm (Fu ni al & Wilson, 1974) was de eloped, which p unes he sea ch space by disca ding subse s ha canno possibly imp o e he c i e ion alue. This app oach subs an ially educes he compu a ional bu den while s ill gua an eeing ha he op imal subse is iden i ied o each model size. P ac ical implemen a ions a e widely a ailable, including he leaps package in R o linea eg ession, and he glmul i package, which gene alizes bes subse selec ion o gene alized linea models such as logis ic eg ession (Calcagno & de Mazancou , 2010). Despi e i s ad an ages in e ms o in e p e abili y and op imali y unde he chosen c i e ion, bes subse eg ession can emain compu a ionally demanding in mode a e- o-high dimensions and is sensi i e o collinea i y among p edic o s. 1.3 LASSO LASSO adds an ℓ1 penal y o he o dina y leas squa es (OLS) objec i e: 𝜷𝜷�=𝒂𝒂𝒂𝒂𝒂𝒂𝒎𝒎𝒎𝒎𝒎𝒎 𝜷𝜷�𝟏𝟏 𝟐𝟐𝒎𝒎��𝒚𝒚𝒎𝒎−𝒙𝒙𝒎𝒎𝑻𝑻𝜷𝜷�𝟐𝟐 𝒎𝒎 𝒎𝒎=𝟏𝟏 +𝝀𝝀��𝜷𝜷𝒋𝒋� 𝒑𝒑 𝒋𝒋=𝟏𝟏 � whe e λ ≥ 0 is a uning pa ame e ha con ols he s eng h o egula iza ion. The ℓ1 penal y encou ages spa si y, sh inking some coe icien s exac ly o ze o, he eby pe o ming a iable selec ion. As λ inc eases, mo e coe icien s a e sh unk o ze o, yielding simple models; when λ=0, he solu ion educes o o dina y leas squa es. In p ac ice, λ is ypically chosen ia K- old c oss- alida ion, minimizing p edic ion e o on held-ou olds. E icien algo i hms make LASSO compu a ionally easible e en when p ≫ n, which has con ibu ed o i s widesp ead use in clinical applica ions. 1.4 Ridge Ridge eg ession uses an ℓ2 penal y ins ead: 𝜷𝜷�=𝒂𝒂𝒂𝒂𝒂𝒂𝒎𝒎𝒎𝒎𝒎𝒎 𝜷𝜷�𝟏𝟏 𝟐𝟐𝒎𝒎��𝒚𝒚𝒎𝒎−𝒙𝒙𝒎𝒎𝑻𝑻𝜷𝜷�𝟐𝟐 𝒎𝒎 𝒎𝒎=𝟏𝟏 +𝝀𝝀�𝝆𝝆𝒋𝒋𝟐𝟐 𝒑𝒑 𝒋𝒋=𝟏𝟏 � Unlike LASSO, Ridge does no p oduce spa se models. All coe icien s a e sh unk owa d ze o bu emain nonze o. This is bene icial when all a iables con ibu e meaning ully o he ou come, such as in medical diagnosis models whe e bioma ke s collec i ely imp o e disease p edic ion. Howe e , i does no pe o m a iable selec ion in he s ic sense. 1.5 Elas ic Ne Elas ic Ne combines he s eng hs o LASSO and Ridge by inco po a ing bo h ℓ1 and ℓ2 penal ies: 𝜷𝜷�=𝒂𝒂𝒂𝒂𝒂𝒂𝒎𝒎𝒎𝒎𝒎𝒎 𝜷𝜷�𝟏𝟏 𝟐𝟐𝒎𝒎��𝒚𝒚𝒎𝒎−𝒙𝒙𝒎𝒎𝑻𝑻𝜷𝜷�𝟐𝟐 𝒎𝒎 𝒎𝒎=𝟏𝟏 +𝝀𝝀[𝜶𝜶��𝜷𝜷𝒋𝒋� 𝒑𝒑 𝒋𝒋=𝟏𝟏 +(𝟏𝟏−𝜶𝜶)�𝝆𝝆𝒋𝒋𝟐𝟐 𝒑𝒑 𝒋𝒋=𝟏𝟏 ]� whe e α ∈ [0,1] con ols he mixing be ween LASSO (α=1) and Ridge (α=0). The dual egula iza ion p o ides a lexible comp omise be ween spa si y and s abili y. Tuning in ol es selec ing bo h λ and α, usually ia a wo-dimensional g id sea ch wi h c oss- alida ion. 1.6 SCAD The Smoo hly Clipped Absolu e De ia ion (SCAD) penal y was designed o o e come he bias o LASSO o la ge coe icien s while s ill yielding spa se solu ions. The SCAD penal y is de ined by a piecewise unc ion ha beha es like LASSO nea ze o (encou aging spa si y), bu la ens ou o la ge coe icien s, he eby p o iding unbiased es ima es o la ge coe icien s. The SCAD es ima o sol es: 𝜷𝜷�=𝒂𝒂𝒂𝒂𝒂𝒂𝒎𝒎𝒎𝒎𝒎𝒎 𝜷𝜷�𝟏𝟏 𝟐𝟐𝒎𝒎��𝒚𝒚𝒎𝒎−𝒙𝒙𝒎𝒎𝑻𝑻𝜷𝜷�𝟐𝟐 𝒎𝒎 𝒎𝒎=𝟏𝟏 +�𝒑𝒑𝝀𝝀(�𝜷𝜷𝒋𝒋�) 𝒑𝒑 𝒋𝒋=𝟏𝟏 � whe e 𝒑𝒑𝝀𝝀 (⋅) is he SCAD penal y unc ion depending on λ and a uning pa ame e a (commonly se o 3.7). Compa ed o LASSO and Ridge, SCAD’s noncon exi y makes he op imiza ion mo e challenging han o con ex penal ies. 1.7 Bo u a algo i hm The Bo u a algo i hm (Ku sa & Rudnicki, 2010) is a w appe me hod buil a ound Random Fo es s o pe o m all- ele an ea u e selec ion. Unlike LASSO, Ridge, o SCAD, which impose penal ies in eg ession models, Bo u a le e ages he a iable impo ance measu e om Random Fo es s. A each i e a ion, a Random Fo es model is i ed, and he impo ance sco e o each eal ea u e is compa ed agains he maximum impo ance among he pe mu a ed ea u es (shadows). Fea u es consis en ly ou pe o ming he shadows a e deemed “con i med”, while hose consis en ly unde pe o ming a e “ ejec ed”. Fea u es wi h in e media e pe o mance a e le “ en a i e”. The main ad an age o Bo u a is i s abili y o de ec weak bu ele an ea u es, hough a he cos o highe compu a ional bu den compa ed o single-model me hods. To summa ize, he me hods discussed abo e di e in hei unde lying selec ion mechanisms, compu a ional e iciency, and beha io unde mul icollinea i y o high-dimensional se ings. Table 1 p o ides a side-by-side compa ison o hese se en app oaches, highligh ing hei key ea u es, ad an ages, and limi a ions. Table 1: Compa ison o key ea u es o he selec ed me hods Me hod Type Fea u e Selec ion App oach Mul icollinea i y In e p e abili y Scalabili y S epwise Reg ession Linea (GLM) G eedy (Bo h di ec ions) Poo ly High Mode a e Bes Subse Linea (GLM) Exhaus i e Sea ch Poo ly High Slow LASSO Linea (GLM) Sh inkage (L1 penal y) Good High Good Ridge Linea (GLM) Sh inkage (L2 penal y) Good High Good Elas ic Ne Linea (GLM) Sh inkage (L1 + L2 penal y) Good High Good SCAD Linea (GLM) Sh inkage (Non- con ex penal y) Good High Mode a e Bo u a T ee- based Impo ance om Random Fo es Gene ally obus Mode a e Mode a e In his pape , we compa e s epwise eg ession, bes subse , LASSO, idge eg ession, elas ic ne , and Bo u a ac oss bo h simula ed da ase s and eal-wo ld clinical da a. Ou e alua ion ocuses on p edic i e pe o mance, pa ame e es ima es, and ea u e selec ion accu acy unde a ying le els o sample size, p edic o dimensionali y, signal- o-noise a io, spa si y, and collinea i y. The emainde o he a icle is o ganized as ollows. In Sec ion 2, we p esen a simula ion s udy designed o sys ema ically compa e he pe o mance o hese me hods ac oss con olled scena ios. Sec ion 3 applies he me hods o an empi ical s udy aimed a iden i ying isk ac o s o hype co isolism, illus a ing hei p ac ical di e ences in a eal clinical con ex . We conclude in Sec ion 4 wi h a discussion o key indings and p ac ical conside a ions o applying hese echniques in clinical esea ch. 2. Simula ion S udy 2.1 Simula ion Design To sys ema ically assess model pe o mance unde di e se clinical da a scena ios, we conduc ed a comp ehensi e simula ion s udy based on a i e- ac o ac o ial design. The pa ame e s we e selec ed o mimic a wide ange o condi ions commonly encoun e ed in clinical esea ch. Speci ically, we a ied he ollowing ac o s: 1. Sample size (N): Values o 100, 500, and 1000 we e conside ed o ep esen small, medium, and la ge s udy sizes ypical o clinical se ings. 2. Numbe o candida e p edic o s (p): We examined p edic o se s o size 15, 30, 50, 100, and 1000, spanning low-, mode a e-, and high-dimensional se ings. 3. Signal- o-noise a io (SNR): le ’s deno e he model as: 𝒚𝒚𝟎𝟎 =𝒇𝒇(𝒙𝒙𝟎𝟎)+ 𝜺𝜺𝟎𝟎 whe e 𝒙𝒙𝟎𝟎 ∈ Rp is he independen a iable and 𝒚𝒚𝟎𝟎 ∈ R is he esponse a iable. Then, he SNR is de ined as: 𝑺𝑺𝑺𝑺𝑺𝑺=𝒗𝒗𝒂𝒂𝒂𝒂(𝒇𝒇(𝒙𝒙𝟎𝟎)) 𝒗𝒗𝒂𝒂𝒂𝒂(𝜺𝜺𝟎𝟎) SNR alues anged om 0.05 o 4.5, co esponding o 2–80% o a iance explained, wi h SNR = 1 ep esen ing 50% explained a iance. 4. Spa si y le el: Spa si y was de ined as he p opo ion o uly nonze o coe icien s in he unde lying model. To app oxima e ealis ic condi ions, i e p edic o s we e assigned coe icien s o 1, while he emaining coe icien s decayed exponen ially owa d ze o: 𝜷𝜷𝟎𝟎𝒎𝒎=𝟎𝟎.𝟓𝟓𝒎𝒎−𝒔𝒔, o 𝒎𝒎=6, … ,p. A h eshold o 1e–2 dis inguished ue coe icien s om negligible ones. By a ying he o al numbe o p edic o s, his se up cap u ed di e ing assump ions abou he numbe o ele an ea u es. 5. Collinea i y le el: P edic o s we e gene a ed om a mul i a ia e no mal dis ibu ion wi h i s o de au o eg essi e co ela ion s uc u es. Co ela ion alues o 0.2, 0.5, and 0.8 we e used o ep esen weak, mode a e, and s ong collinea i y, espec i ely. Each simula ion scena io was eplica ed 50 imes o es ima e he mean and s anda d de ia ion o each e alua ion me ic. Fo each eplica ion, we gene a ed bo h a aining se and a es se o equal size unde he same pa ame e con igu a ions. The aining se was used o model i ing, while he es se se ed o e alua e p edic i e accu acy, as desc ibed below. 2.2 E alua ion Me ics To e alua e and compa e he pe o mance o each eg ession me hod ac oss he simula ed scena ios, we conside ed h ee aspec s: p edic i e pe o mance, pa ame e es ima es and a iable selec ion accu acy. These me ics we e chosen o e lec he u ili y o he i ed model o p edic ion, he es ima ed coe icien alues, and i s abili y o eco e he ue unde lying s uc u e o he da a. • P edic i e pe o mance was measu ed using he no malized oo mean squa ed e o (NRMSE) on an independen es se : 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁=�1 𝑛𝑛𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡∗𝑠𝑠𝑠𝑠(𝑦𝑦𝑖𝑖)�(𝑦𝑦𝚤𝚤 �−𝑦𝑦𝑖𝑖)2 𝑛𝑛𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑖𝑖=1 whe e 𝑦𝑦𝑖𝑖 a e he obse ed esponses, 𝑦𝑦𝚤𝚤 � is he p edic ed esponses, 𝑠𝑠𝑠𝑠(𝑦𝑦𝑖𝑖) is he s anda d de ia ion o he ou come in he es se , and 𝑛𝑛𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 is he numbe o obse a ions in he es se . No maliza ion by ou come a iabili y allows compa abili y ac oss scena ios. • Pa ame e es ima es we e assessed by ela i e es e o (RTE) (Has ie, Tibshi ani, & Tibshi ani, 2017), which measu es he expec ed es e o ela i e o he Bayes e o a e: 𝑁𝑁𝑅𝑅𝑁𝑁(𝛽𝛽󰆹) = 𝑬𝑬�𝑦𝑦0−𝑥𝑥0𝑇𝑇𝛽𝛽󰆹�2 𝜎𝜎2 =(𝛽𝛽󰆹−𝛽𝛽0)𝑇𝑇 𝛴𝛴(𝛽𝛽󰆹−𝛽𝛽0) + 𝜎𝜎2 𝜎𝜎2 A pe ec sco e o 1 indica es ha he model exac ly eco e s he ue coe icien s, whe eas he null sco e is gi en by 𝛽𝛽0𝑇𝑇𝛴𝛴𝛽𝛽0+𝜎𝜎2 𝜎𝜎2= 𝑁𝑁𝑁𝑁𝑁𝑁+ 1. • Va iable selec ion accu acy was e alua ed by compa ing he es ima ed suppo se (nonze o coe icien s) agains he g ound u h. We compu ed he Ma hews Co ela ion Coe icien (MCC) (Ma hews, 1975), which p o ides a balanced summa y o all ou classi ica ion ou comes (TP, TN, FP, FN): 𝑁𝑁𝑀𝑀𝑀𝑀= 𝑅𝑅𝑇𝑇∗𝑅𝑅𝑁𝑁−𝐹𝐹𝑇𝑇∗𝐹𝐹𝑁𝑁 �(𝑅𝑅𝑇𝑇+𝐹𝐹𝑇𝑇)(𝑅𝑅𝑇𝑇+𝐹𝐹𝑁𝑁)(𝑅𝑅𝑁𝑁+𝐹𝐹𝑇𝑇)(𝑅𝑅𝑁𝑁+𝐹𝐹𝑁𝑁) TP ( ue posi i es) a e co ec ly selec ed nonze o coe icien s, TN ( ue nega i es) a e co ec ly excluded ze o coe icien s, FP ( alse posi i es) a e ze o coe icien s inco ec ly selec ed, and FN ( alse nega i es) a e nonze o coe icien s missed. The MCC anges om -1 (comple e disag eemen ) o 1 (pe ec ag eemen ), wi h 0 indica ing andom pe o mance. • Posi i e a iable selec ion accu acy was quan i ied using he F1 sco e ( an Rijsbe gen, 1979), which balances sensi i i y and p ecision in iden i ying ue p edic o s: 𝐹𝐹1 = 2∗𝑇𝑇𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑠𝑠𝑃𝑃𝑃𝑃𝑛𝑛∗𝑁𝑁𝑃𝑃𝑃𝑃𝑅𝑅𝑅𝑅𝑅𝑅 𝑇𝑇𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑠𝑠𝑃𝑃𝑃𝑃𝑛𝑛∗𝑁𝑁𝑃𝑃𝑃𝑃𝑅𝑅𝑅𝑅𝑅𝑅 , whe e 𝑇𝑇𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑠𝑠𝑃𝑃𝑃𝑃𝑛𝑛=𝑇𝑇𝑇𝑇 𝑇𝑇𝑇𝑇+𝐹𝐹𝑇𝑇 and 𝑁𝑁𝑃𝑃𝑃𝑃𝑅𝑅𝑅𝑅𝑅𝑅=𝑇𝑇𝑇𝑇 𝑇𝑇𝑇𝑇+𝐹𝐹𝐹𝐹 A key limi a ion o F1 sco e is i s bias owa d la ge models: adding mo e a iables ypically inc eases ecall ( educing alse nega i es) and hus in la es F1, e en i many i ele an p edic o s a e included. To add ess his, we applied an adjus ed F1 sco e ha di ec ly penalizes model size: 𝐴𝐴𝑠𝑠𝐴𝐴𝐴𝐴𝑠𝑠𝐴𝐴𝑃𝑃𝑠𝑠 𝐹𝐹1 = 𝐹𝐹1∗(1 −𝑁𝑁𝐴𝐴𝑁𝑁𝑁𝑁𝑃𝑃𝑃𝑃 𝑃𝑃𝑜𝑜 𝑜𝑜𝑃𝑃𝑅𝑅𝐴𝐴𝐴𝐴𝑃𝑃𝑃𝑃𝑠𝑠 𝑠𝑠𝑃𝑃𝑅𝑅𝑃𝑃𝑃𝑃𝐴𝐴𝑃𝑃𝑠𝑠 𝑁𝑁𝐴𝐴𝑁𝑁𝑁𝑁𝑃𝑃𝑃𝑃 𝑃𝑃𝑜𝑜 𝑜𝑜𝑃𝑃𝑅𝑅𝐴𝐴𝐴𝐴𝑃𝑃𝑃𝑃𝑠𝑠 ) This adjus men educes sco es o unnecessa ily la ge models, p o iding a ai e compa ison ac oss me hods wi h di e en le els o spa si y. 2.3 Pa ame e uning S epwise eg ession was pe o med wi h a p- alue cu o o 0.10 o a iable en y and 0.05 o e en ion. Fo he bes subse eg ession model, he Bayesian In o ma ion C i e ion (BIC) was used o de e mine he op imal model size. All c oss- alida ion p ocedu es we e conduc ed using 10- old c oss- alida ion. The LASSO was uned using c oss- alida ion o selec he penal y pa ame e λ. Two e sions o he LASSO model we e conside ed: one using λmin, he alue ha minimizes he c oss- alida ed p edic ion e o , and ano he using λ1se, he la ges alue o λ wi hin one s anda d e o o he minimum. These a e e e ed o as LASSO (min) and LASSO (1se), espec i ely, in he ollowing sec ions. Ridge eg ession was uned in a simila manne , wi h λ selec ed acco ding o he λ1se ule. Elas ic ne eg ession was i wi h equal weigh ing o he ℓ1 and ℓ2 penal ies (α=0.5), and λ uning was pe o med ollowing he λ1se ule. Fo SCAD, he conca i y pa ame e a was se o he commonly used de aul alue o 3.7, and λ was chosen as λmin. In he Bo u a p ocedu e, only ea u es con i med as impo an we e e ained as selec ed p edic o s. The inal andom o es model was hen buil and uned h ough a g id sea ch o e i e candida e alues o he numbe o p edic o s conside ed a each spli , wi h c oss- alida ion used o selec he op imal con igu a ion. Fo s epwise eg ession and bes subse selec ion, compu a ional bu den inc eased subs an ially as he ea u e space g ew. To add ess his, we implemen ed p edic o p e il e ing based on he ma ginal co ela ion be ween each p edic o and he esponse. Speci ically, when he numbe o p edic o s was ≥ 1000, s epwise eg ession was es ic ed o he op 100 ea u es. Fo bes subse selec ion, when he numbe o p edic o s was ≥ 100, he analysis was es ic ed o he op 50 ea u es. 2.4 Simula ion S udy Resul s 2.4.1 P edic i e Pe o mance P edic i e pe o mance was e alua ed using no malized RMSE, wi h esul s summa ized in Figu e 1. Each ba ep esen s a me hod, whe e scena ios wi h p < 100 a e conside ed low-dimensional and hose wi h SNR < 0.5 a e conside ed low-signal se ings. Do s indica e model size, mapped o he seconda y y-axis on he igh . Model size o Ridge eg ession is no shown, since all coe icien s emain nonze o and i s model size is equal o he ull ea u e space. As expec ed, pe o mance imp o ed wi h highe SNR ac oss all me hods, yielding smalle RMSE alues. In low-dimensional se ings (small numbe o p edic o s), di e ences be ween s epwise eg ession, bes subse , and egula iza ion-based me hods we e minimal, and became negligible as sample size inc eased. Nei he SNR le el no p edic o co ela ion subs an ially al e ed his pa e n. Fo LASSO, we compa ed wo uning s a egies: λmin which minimizes c oss- alida ion e o and esul s in la ge models, and λ1se, which applies s onge egula iza ion and p oduces mo e pa simonious models. In mos cases, λmin achie ed be e p edic i e accu acy, while λ1se emained smalle in size, o e ing a ade-o be ween accu acy and pa simony. In high-dimensional se ings, he pic u e was ma kedly di e en . Regula iza ion me hods subs an ially ou pe o med s epwise and bes subse app oaches, pa icula ly when sample size was small. These pe o mance gaps diminished as sample size inc eased. Ridge eg ession, howe e , exhibi ed no able o e i ing when SNR was high, since i sh inks coe icien s bu does no elimina e i ele an p edic o s, leading o in la ed p edic ion e o . T ee-based models displayed dis inc beha io . Random o es s pe o med compa ably well unde low SNR, bu unde high SNR hey unde pe o med ela i e o s epwise eg ession. This sugges s ha hei inhe en bias, esul ing om bagging and andom spli s, can hinde pe o mance when he unde lying signal is pu ely linea . In e es ingly, p edic o collinea i y imp o ed he p edic i e accu acy o andom o es s: when mul iple co ela ed p edic o s ca ied he ue signal, he p obabili y o cap u ing i a a spli inc eased, he eby enhancing model pe o mance. O e all, egula iza ion-based me hods showed he mos s able p edic i e pe o mance ac oss simula ion scena ios, pa icula ly in high-dimensional se ings. While SNR had li le impac on he ela i e anking o me hods, collinea i y modes ly bene i ed ee-based app oaches. These esul s sugges ha egula iza ion me hods a e gene ally p e e able o clinical da ase s wi h many p edic o s, whe eas s epwise eg ession and bes subse emain a ac i e op ions in se ings wi h la ge samples and small ea u e spaces, as hey equi e no hype pa ame e uning. Figu e 1. P edic i e pe o mance measu ed by no malized RMSE ac oss simula ion se ings. Ba s ep esen me hods, wi h low-dimensional se ings de ined as p < 100 and low-SNR se ings as SNR < 0.5. Do s indica e model size ( igh y-axis). 2.4.2 Pa ame e Es ima es Pa ame e es ima es we e e alua ed using ela i e es e o , which quan i ies how closely he es ima ed eg ession coe icien s align wi h he ue coe icien s. A sco e o 1 indica es pe ec eco e y o he ue coe icien s, and lowe sco es e lec smalle es ima ion e o . Figu e 2 summa izes he esul s, wi h he do ed black cu e ep esen ing he null model. Bo u a is excluded om his me ic because ee-based me hods do no yield explici coe icien es ima es. The ela i e es e o pa e ns a e p ima ily de e mined by he sample size- o-p edic o a io. When p = 15 (Figu e 2a), me hods clus e in o h ee g oups. The i s g oup is LASSO (min), which consis en ly achie es he lowes ela i e es e o ac oss condi ions. The second g oup includes s epwise eg ession, bes subse , and SCAD. These me hods pe o m well when co ela ion is low and SNR is high, bu hei pe o mance wo sens as p edic o co ela ion inc eases. The hi d g oup is LASSO (1se), Ridge, and Elas ic Ne . These me hods pe o m easonably when SNR is low bu de e io a e sha ply as SNR inc eases, making hem he leas accu a e in his se ing. As sample size inc eases, he pe o mance o he i s and second g oups con e ges, while he hi d g oup emains subop imal. The di e ence be ween LASSO (min) and LASSO (1se) highligh s he c i ical ole o uning egula iza ion pa ame e s. Al hough he cu es appea well sepa a ed in Figu e 2a, he ac ual di e ences a e modes , on he o de o 0.05 on he y-axis. When p = 50 (Figu e 2b), s epwise eg ession and bes subse a e he wo s -pe o ming me hods a small sample sizes such as N = 100. Wi h la ge sample sizes, hei accu acy imp o es and app oaches ha o LASSO (min). A he same ime, Elas ic Ne , Ridge, and LASSO (1se) appea ela i ely wo se as N g ows. The o e all pa e n esembles he case wi h p = 15, excep ha s epwise eg ession and bes subse ne e ully each he accu acy o LASSO (min). The case wi h p = 30 exhibi s a pa e n simila o ha obse ed o p = 50 and is he e o e no shown. When p = 1000 (Figu es 2c-d), Ridge eg ession domina es he plo wi h e y high ela i e es e o s, comp essing he lines o o he me hods. A e emo ing Ridge om he display, i becomes clea ha s epwise eg ession and bes subse pe o m poo ly in high dimensions, al hough hei accu acy imp o es owa d ha o he egula ized me hods as SNR and sample size inc ease. O e all ela i e es e o dec eases wi h la ge sample sizes, showing ha su icien da a can o se he challenges o high dimensionali y. Ac oss hese scena ios, LASSO (min) and SCAD p o ide he mos accu a e coe icien es ima es. SCAD pe o ms bes when co ela ion is low, whe eas LASSO (min) excels when co ela ion is high. The case wi h p = 100 exhibi s a simila pa e n o p = 1000. In summa y, model accu acy is p ima ily de e mined by he sample size o p edic o a io. High SNR imp o es he pe o mance o s epwise eg ession and bes subse bu has limi ed bene i o egula ized me hods. P edic o co ela ion des abilizes s epwise eg ession and bes subse while lea ing egula iza ion-based app oaches ela i ely una ec ed. LASSO (min) consis en ly p oduces he mos accu a e coe icien es ima es, al hough in se ings wi h la ge sample size ela i e o he numbe o p edic o s, s epwise eg ession and bes subse achie e simila accu acy wi hou equi ing hype pa ame e uning. Figu e 6. Hea map o ea u e selec ion ac oss me hods. Each ow ep esen s a ea u e and each column a me hod. Me hods a e o de ed by hie a chical clus e ing. Figu e 6 p esen s he ea u e selec ion hea map, whe e each ow ep esen s a ea u e and each column a me hod, wi h me hods o de ed by hie a chical clus e ing. Fou ea u e clus e s, ou lined by black boxes, highligh g oups o me hods wi h simila selec ion pa e ns. LASSO (min) and SCAD p oduced he la ges models, e lec ing weake egula iza ion, whe eas LASSO (1se) and Elas ic Ne selec ed mo e simila and pa simonious se s o ea u es unde s onge egula iza ion. S epwise eg ession and bes subse iden i ied la gely o e lapping ea u es, wi h s epwise eg ession selec ing mo e. Bo u a, which pe o med bes in simula ion s udies, displayed he mos dis inc selec ion pa e n. To explo e hese di e ences in de ail, Table 3 con as s ea u es selec ed by s epwise eg ession (Buse e al., 2025) e sus Bo u a. S epwise eg ession uniquely iden i ied p io use o SGLT2 inhibi o s, analgesics, i zepa ide, maximum dose o GLP-1 ecep o agonis s, and BMI, ea u es p ima ily ela ed o hype glycemia. In con as , Bo u a uniquely selec ed p io use o diu e ics and be a-blocke s, ea u es mo e e lec i e o hype ension. Gi en he s ong co ela ion be ween hype glycemia- and hype ension- ela ed a iables, hese di e ences sugges ha dis inc modeling app oaches may emphasize di e en aspec s o pa ien clinical p o iles. S epwise eg ession p o ides a mo e s aigh o wa d in e p e a ion, as hype glycemia is di ec ly linked o poo ly con olled diabe es, bu Bo u a aises he possibili y ha hype ension- ela ed ac o s may also play an impo an ole in his popula ion. Table 3. Fea u es Selec ed by S epwise Reg ession Compa ed wi h Bo u a Analysis Fea u e selec ed by S epwise eg ession (Buse e al., 2025) Fea u e selec ed by Bo u a Times selec ed in 1000 eplica es E hnici y = Hispanic/La ino 1000 Numbe o An ihype ensi e Classes 999 Age 807 Fib a es 846 Diu e ics 988 Region 909 Be a Blocking Agen s 595 SGLT2 inhibi o 471 Analgesics 203 Ti zepa ide 171 BMI >= 30 34 Took Maximum Dose o Any GLP-1 3 To be e isualize equen ly selec ed ea u es (de ined as appea ing in a leas 50% o eplica es) ac oss me hods, an UpSe plo was gene a ed (Figu e 7). The LASSO (min) model p oduced he la ges se , mo e han wice as la ge as he o he me hods, wi h 5 unique ea u es and 5 ea u es sha ed only wi h SCAD. As no ed ea lie , ou ea u es we e consis en ly iden i ied by all me hods. Bes subse yielded he mos pa simonious model, selec ing only hese ou ea u es, while LASSO (1se) added one addi ional p edic o (SGLT2 inhibi o ). Bo u a, s epwise eg ession, and Elas ic Ne p oduced models o compa able size, la gely selec ing subse s o LASSO (min) ea u es. O e all, he e was subs an ial o e lap in selec ed ea u es, hough he ela i e impo ance anking a ied ac oss me hods. Figu e 7. UpSe plo o ea u es selec ed ≥50% o he ime ac oss di e en me hods. In summa y, he eal-da a analyses ein o ce ou simula ion indings: while mul iple me hods achie e s ong p edic i e pe o mance, hey di e in he balance be ween pa simony and s abili y. S epwise eg ession p o ides in e p e able, compac models wi hou he need o hype pa ame e uning and p o es o be a p ac ical and e ec i e op ion in his se ing. Meanwhile, Bo u a iden i ies complemen a y signals ha may cap u e al e na i e clinical pa hways, o e ing addi ional insigh s beyond hose o adi ional linea app oaches. 4. Discussion In his s udy, we sys ema ically e alua ed he pe o mance o s epwise eg ession, bes subse , Bo u a, and ou egula iza ion-based me hods, LASSO, Ridge, SCAD and Elas ic Ne , ac oss simula ed scena ios and eal-wo ld clinical da a om he CATALYST s udy. Ou aim was o assess p edic i e pe o mance, ea u e selec ion s abili y, and in e p e abili y unde a ying sample sizes, ea u e dimensionali y, spa si y pa e ns, and collinea i y le els. F om a p ac ical s andpoin , ou esul s sugges ailo ing me hod choice o da a cha ac e is ics may be wa an ed. When he numbe o p edic o s is small and collinea i y is low, s epwise and bes subse me hods o e good p edic i e pe o mance and a iable selec ion accu acy, wi h he added ad an age o in e p e abili y and no need o hype pa ame e uning. In medium- o-la ge ea u e spaces o when collinea i y is high, egula ized me hods, pa icula ly Elas ic Ne and LASSO (1se), p o ide mo e s able and pa simonious models. Bo u a is especially use ul in co ela ed se ings, whe e i s abili y o exploi edundancy among p edic o s yields s ong ea u e selec ion pe o mance. By con as , LASSO (min) and SCAD end o o e -selec ea u es, sac i icing in e p e abili y and selec ion accu acy despi e compe i i e aw p edic i e pe o mance. Applica ion o he CATALYST s udy con i med hese pa e ns. LASSO (min) and Ridge achie ed he highes p edic i e accu acy, while s epwise eg ession anked second bu yielded a much mo e pa simonious model. LASSO and s epwise eg ession la gely con e ged on a co e se o p edic o s o hype co isolism, whe eas Bo u a iden i ied addi ional ea u es ha we e missed by he linea models. These indings ha e impo an implica ions. Fi s , hey unde sco e he ac ha he e is no “one size i s all” model. Classic me hods such as s epwise eg ession demons a e good p edic i e powe when he ea u e numbe is small and ea u e in e ac ion is low. Regula ized me hods a e mo e s able unde high ea u e dimensions, bu hype pa ame e uning is c i ical, and small di e ences in hese pa ame e s may lead o e y di e en conclusions and models. Second, ou esul s highligh he alue o combining complemen a y app oaches. Me hods such as Bo u a may e eal al e na i e o co ela ed p edic o s ha linea models unde -emphasize, o e ing a b oade pe spec i e on po en ial clinical pa hways. The e a e se e al limi a ions o no e. Ou simula ion se ings, while ex ensi e, canno cap u e all complexi ies o eal-wo ld da ase s. Fo ins ance, nonlinea ela ionships a e common in clinical se ings, ye ou simula ion amewo k was es ic ed o linea s uc u es. Fu he mo e, ou e alua ion emphasized linea modeling s a egies, while mo e lexible app oaches such as g adien boos ing o neu al ne wo ks may deli e addi ional p edic i e gains, albei a he expense o in e p e abili y. In conclusion, his s udy highligh s he p ac ical ade-o s among a iable selec ion me hods in clinical esea ch. These insigh s can help guide he choice o analy ic s a egy in u u e disco e y and isk p edic ion e o s, balancing p edic i e accu acy, model s abili y, and in e p e abili y in complex clinical popula ions. Acknowledgemen s The au ho s hank Daniel Einho n o his hough ul ad ice and con ibu ions, which ha e s eng hened his wo k. The au ho s a e also g a e ul o Nina Pasho a o he suppo in conduc ing he analyses. In addi ion, he au ho s acknowledge he Co cep biome ics eam o hei insigh ul commen s and discussions, which imp o ed he cla i y and igo o his wo k. Re e ences Buse, J. B., e al. (2025). P e alence o hype co isolism in di icul - o-con ol ype 2 diabe es. Diabe es Ca e, dc242841. Calcagno, V., & de Mazancou , C. (2010). glmul i: An R package o easy au oma ed model selec ion wi h (gene alized) linea models. Jou nal o S a is ical So wa e, 34(12), 1–29. De ksen, S., & Keselman, H. J. (1992). Backwa d, o wa d and s epwise au oma ed subse selec ion algo i hms: F equency o ob aining au hen ic and noise a iables. B i ish Jou nal o Ma hema ical and S a is ical Psychology, 45(2), 265–282. Fan, J., & Li, R. (2001). Va iable selec ion ia nonconca e penalized likelihood and i s o acle p ope ies. Jou nal o he Ame ican S a is ical Associa ion, 96(456), 1348–1360. Fu ni al, G. M., & Wilson, R. W. (1974). Reg ession by leaps and bounds. Technome ics, 16(4), 499–511. Ha ell, F. E. (2015). Reg ession modeling s a egies: Wi h applica ions o linea models, logis ic and o dinal eg ession, and su i al analysis (2nd ed.). Sp inge . Has ie, T., Tibshi ani, R., & Tibshi ani, R. J. (2017). Ex ended compa isons o bes subse selec ion, o wa d s epwise selec ion, and he lasso. a Xi p ep in a Xi :1707.08692. Hoe l, A. E., & Kenna d, R. W. (1970). Ridge eg ession: Biased es ima ion o nono hogonal p oblems. Technome ics, 12(1), 55–67. Ku sa, M. B., & Rudnicki, W. R. (2010). Fea u e selec ion wi h he Bo u a package. Jou nal o S a is ical So wa e, 36(11), 1–13. Ma hews, B. W. (1975). Compa ison o he p edic ed and obse ed seconda y s uc u e o T4 phage lysozyme. Biochimica e Biophysica Ac a, 405(2), 442–451. Mille , A. (2002). Subse selec ion in eg ession (2nd ed.). Chapman and Hall/CRC. Tibshi ani, R. (1996). Reg ession sh inkage and selec ion ia he Lasso. Jou nal o he Royal S a is ical Socie y: Se ies B (Me hodological), 58(1), 267–288. an Rijsbe gen, C. J. (1979). In o ma ion Re ie al (2nd ed.). Zou, H., & Has ie, T. (2005). Regula iza ion and a iable selec ion ia he Elas ic Ne . Jou nal o he Royal S a is ical Socie y: Se ies B (S a is ical Me hodology), 67(2), 301–320.