Exploring Gender Bias In Remote Pair Programming Among Software Engineering Students: The twincode Original Study And First External Replication

Empi ical So wa e Enginee ing (2024) 29:40 h ps://doi.o g/10.1007/s10664-023-10416-6 Explo ing Gende Bias In Remo e Pai P og amming Among So wa e Enginee ing S uden s: The wincode O iginal S udy And Fi s Ex e nal Replica ion Amado Du án To o1,2 ·Pablo Fe nández1,2 ·Bea iz Be ná dez1,2 · Na haniel Weinman3·Aslıhan Akalın3·A mando Fox3 Accep ed: 18 Oc obe 2023 / Published online: 1 Feb ua y 2024 © The Au ho (s) 2024 Abs ac Con ex Women ha e his o ically been unde ep esen ed in So wa e Enginee ing, due in pa o he s e eo yped assump ion ha women a e less echnically compe en han men. Pai p og amming is bo h widely used in indus y and has been shown o inc ease s uden in e es in So wa e Enginee ing, pa icula ly among women; bu i hose same gende biases a e also p esen in pai p og amming, i s po en ial o a ac ing women o he ield could be hwa ed. Objec i e We aim o explo e he e ec s o gende bias in pai p og amming. Speci ically, in a emo e se ing in which s uden s canno di ec ly obse e he gende o hei pee s, we s udy whe he he pe cep ion o he pa ne , he beha io du ing p og amming, o he s yle o communica ion o So wa e Enginee ing s uden s di e depending on he pe cei ed gende o hei emo e pa ne . To ou knowledge, his is he i s s udy speci ically ocusing on he impac o gende s e eo ypes and bias wi hin pai s in pai p og amming. Me hod We ha e de eloped an online pai -p og amming pla o m ( wincode) ha p o- ides a collabo a i e edi ing window and a cha pane, bo h o which a e hea ily ins umen ed. S uden s in he con ol g oup had no in o ma ion abou hei pa ne ’s gende , whe eas s u- den s in he ea men g oup could see a gende ed a a a ep esen ing he o he pa icipan as a man o as a woman. The gende o he a a a was swapped be ween p og amming asks o analyze 45 a iables ela ed o he collabo a i e coding beha io , cha u e ances, and ques ionnai e esponses o 46 pai s in he o iginal s udy a he Uni e si y o Se ille, and 23 pai s in he ex e nal eplica ion a he Uni e si y o Cali o nia, Be keley. Resul s We did no obse e any s a is ically signi ican e ec o he gende bias ea men , no any in e ac ion be ween he pe cei ed pa ne ’s gende and subjec ’s gende , in any o he 45 esponse a iables measu ed in he o iginal s udy. In he ex e nal eplica ion, we obse ed Communica ed by: Ma ia Te esa Baldassa e, Je ey Ca e , Neil E ns This a icle belongs o he Topical Collec ion: Special issue on Regis e ed Repo s BAmado Du án To o [email p o ec ed] Ex ended au ho in o ma ion a ailable on he las page o he a icle 789().: V,- ol 40 Page 2 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 s a is ically signi ican e ec s wi h mode a e o la ge sizes in ou dependen a iables wi hin he expe imen al g oup, compa ing how subjec s ac ed when hei pa ne s we e ep esen ed as a man o a woman. Conclusions The esul s in he o iginal s udy do no show any clea e ec o he ea men in emo e pai p og amming among cu en So wa e Enginee ing s uden s. In he ex e nal eplica ion, i seems ha s uden s dele e mo e sou ce code cha ac e s when hey ha e a woman pa ne , and communica e using mo e in o mal u e ances, e lec ions and yes/no ques ions when hey ha e a man pa ne , al hough hese esul s mus be conside ed inconclusi e because o he small numbe o subjec s in he eplica ion, and because when mul iple es co ec ions a e applied, only he esul abou in o mal u e ances emains signi ican . In any case, mo e mixed me hods eplica ions a e needed in o de o con i m o e u e he esul s in he same and o he So wa e Enginee ing s uden s popula ions. Keywo ds Gende bias · Pai p og amming · Remo e pai p og amming · Dis ibu ed pai p og amming · So wa e Enginee ing educa ion · Expe imen eplica ion 1 In oduc ion Besides being widely used in indus y, pai p og amming is becoming inc easingly common in So wa e Enginee ing educa ion because o i s demons a ed posi i e in luence on g ades, class pe o mance, con idence, p oduc i i y, and mo i a ion o s ay in So wa e Enginee ing and Compu e Science academic majo s (da Sil a Es ácio and P ikladnicki 2015), especially o women, as epo ed by We ne e al. (2004). In pai p og amming, wo pa ne s wo k closely oge he o sol e a p og amming ask, in which hei abili y o engage collabo a i ely wi h each o he is essen ial. Howe e , hese collabo a i e in e ac ions can be in luenced by implici gende bias (Ho e 2015), which is a widely obse ed phenomenon e en in highly-s uc u ed and p o essional se ings, such as hose epo ed by Ja a e al. (2019) and da Sil a Es ácio and P ikladnicki (2015), and which is based on he s e eo yped assump ion ha women a e less echnically compe en han men (Ma ell e al. 1996; Fishe and Cox 2006; Medel and Pou naghshband 2017; Te ell e al. 2017; Allai e-Duque e e al. 2022). Ou s udy is based on he hypo hesis ha gende bias will lead o obse able di e ences based on subjec s’ pe cep ions o he gende o hei pai p og amming pa ne s, i.e. hey will sco e men and women di e en ly on simila asks, and hey will also beha e and communica e di e en ly depending on whe he hey pe cei e hei pa ne as a man o as a woman, e en hough hei pa ne emains he same on all asks. Speci ically, in a non-coloca ed, i.e. emo e, pai p og amming se ing in which pee gende canno be di ec ly obse ed, ou goal is o iden i y he po en ial e ec s o gende bias by obse ing s uden pai s when he pe cei ed gende o one o he pee s changes. To s udy ou hypo hesis, we ha e applied me hodological iangula ion (Denzin 2006), using se e al me hods o collec da a and app oaching a complex phenomenon like human beha io om mo e han one s andpoin (Cohen e al. 2018). In ou case, h ee di e en da a sou ces ha e been used: (1) ques ionnai es o measu e changes in subjec s’ pe cep ions, (2) da a collec ed au oma ically du ing he pai p og amming asks o measu e beha io al changes, and (3) da a p oduced by se e al expe imen e s analyzing he message in e change du ing he pai p og amming asks o measu e changes in communica ion. Empi ical So wa e Enginee ing (2024) 29 :40 Page 3 o 44 40 Assuming a emo e pai p og amming se ing, which has been p o ed o ha e simila esul s han co-loca ed pai p og amming as epo ed by S o s e al. (2003) and Al-Ja ah and Pon elli (2016), ou esea ch ques ions wi h espec o subjec s’ pe cep ions a e he ollowing: RQ1Does gende bias a ec pe cei ed p oduc i i y compa ed o solo p og amming? Tha is, do pe cei ed di e ences be ween in-pai and solo p oduc i i y depend on he pe cei ed pa ne ’s gende ? RQ2Does gende bias a ec he pa ne ’s pe cei ed echnical compe ency compa ed o one’s own echnical compe ency? Tha is, do pe cei ed di e ences be ween one’s own and pa ne s’ echnical compe ency depend on he pe cei ed pa ne ’s gende ? RQ3Does gende bias a ec he pa ne ’s pe cei ed posi i e and nega i e aspec s? Tha is, do pe cei ed posi i e and nega i e aspec s o hei pa ne s depend on he pe cei ed pa ne ’s gende ?1. RQ4Does gende bias a ec how pa ne s’ skills a e compa ed? Tha is, do pe cei ed pa - ne s’ skills depend on he pe cei ed pa ne ’s gende when hey a e compa ed? Wi h espec o he subjec s’ beha io du ing emo e pai p og amming, and conside ing ha women a e some imes pe cei ed as less compe en in coding because hey o en adop less isky app oaches (Fishe and Cox 2006; Te ell e al. 2017), we assume ha gende bias could cause a subjec o be mo e o less p oac i e on he p og amming ask, i.e., aking mo e o less isks, depending on hei pe cep ion o hei sel -e icacy and hei pe cep ion o he compe ency o hei pa ne (Allai e-Duque e e al. 2022). Thus, ou ela ed esea ch ques ion—based on wha we can au oma ically measu e—is he ollowing: RQ5Does gende bias a ec he equencies o ela i e equencies wi h which each pa ne p oduces sou ce code addi ions, sou ce code dele ions, success ul alida ions, ailed alida- ions, and cha u e ances? Tha is, do hese equencies depend on he pe cei ed pa ne ’s gende ? Rega ding subjec s’ communica ion du ing emo e pai p og amming, we a e in e es ed in knowing whe he gende bias a ec s how subjec s communica e wi h hei pa ne s, i.e., whe he hey use a mo e o mal o in o mal s yle, and whe he hey use some ypes o cha u e ances mo e han o he s. This in e es is mo i a ed by p e ious esea ch whe e i is epo ed ha (i) women and men communica e online di e en ly (Ha sell 2005); (ii) he combina ion o women’s lowe ed pe cep ion o sel , wi h he lowe ed expec a ions om o he s can cause hem o lowe hei a es o pa icipa ion (Medel and Pou naghshband 2017); and (iii) as epo ed by Oda e al. (2022), he pe cei ed gende o he pa ne can exe s imulus con ol o e hei communica ion beha io . Thus, ou ela ed esea ch ques ions a e he ollowing: RQ6Does gende bias a ec he ela i e equency o o mal and in o mal cha u e ances? Tha is, does he o mali y o he messages depend on he pe cei ed pa ne ’s gende ? 1This esea ch ques ion, and i s associa ed a iables, we e added a e he p esen a ion o he ela ed egis e ed epo a ESEM’2021 (Du án e al. 2021). We hough ha including an open ques ion could imp o e he da a collec ion p ocess. 40 Page 4 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 RQ7 Does gende bias a ec he equency o ela i e equency o he di e en ypes o cha u e ances? Tha is, do he equencies o he di e en ypes o messages depend on he pe cei ed pa ne ’s gende ? 1.1 The wincodePla o m To suppo ou s udy, we ha e de eloped he wincode emo e pai p og amming pla o m (El-Re ai e al. 2023), which manages (i) he egis a ion o s uden s collec ing demog aphic da a; (ii) he andom alloca ion in o expe imen al and con ol g oups balancing gende p o- po ions, i.e. ying o ha e he same numbe o pe sons o he same gende in bo h g oups; (iii) he andom alloca ion in o expe imen al-con ol pai s; (i ) he andom assignmen o p o- g amming exe cises o indi idual subjec s and pai s; ( ) he swapping o gende ed a a a s be ween pai p og amming exe cises o hose subjec s in he expe imen al g oup; and ( i) he au oma ic collec ion o in e ac ion me ics and cha u e ances. AsshowninFig. 1, wincode o e s a sou ce code edi o whe e he s uden s concu en ly de elop he solu ion o a p oposed p og aming exe cise in Ja asc ip and can alida e i agains se e al es cases. No e ha , o os e communica ion, only one pa ne can alida e he sou ce code a he same ime and see alida ion esul s, which should be communica ed o he o he pa ne using he cha window, whe e hey a e ins uc ed o collabo a e o sol e he p oposed exe cises. No e also ha a gende ed a a a is displayed only o he s uden in he expe imen al g oup (see Fig. 1a) bu no o he one in he con ol g oup (see Fig. 1b). Expe imen e s can use wincode o c ea e new expe imen al sessions whe e hey can con igu e, among o he aspec s, he ype, numbe , and du a ion o he p og amming exe cises, and he ins uc ional messages shown o he s uden s. I needed, hey can also de elop new p og amming exe cises and hei co esponding es cases. The wincode pla o m is in pe manen e olu ion, and se e al imp o emen s we e inco po a ed o sa is ying some eme ging equi emen s du ing ou s udy, such as allowing he use o Py hon as an al e na i e p og amming language o Ja asc ip o he p og aming exe cises, changing he images used as gende ed a a a s (see Fig. 9), and imp o ing he use in e ace wi h ins uc ions and a gende ed message in he cha window (see Fig. 16aand16b in Appendix B). As a companion ool o wincode,weha ealsode eloped ag-a-cha , a ool ha help expe imen e s code cha u e ances using di e en se s o ags, as shown in Fig. 17 in Appendix B. To assis expe imen e s du ing he aining s age o he coding, ag-a-cha au oma i- cally compu es me ics such as Cohen’s kappa ( o wo code s) and Fleiss’s kappa ( o h ee o mo e code s) in hose dialogs ha a e being coded by se e al expe imen e s o achie e in e -code eliabili y assessmen (O’Conno and Jo e 2020; Syed and Nelson 2015)2. 1.2 Pilo S udies A e p esen ing a e y ini ial app oach o ou s udy (Akalın e al. 2021), and o ge ea ly eedback on (i) he comp ehensibili y and in e nal consis ency o he scales used in he 2 Al hough comme cial quali a i e analysis ools such as MAXQDA (h ps://www.maxqda.com) o A las. i (h ps://a las i.com) a e a ailable, we decided o de elop ag-a-cha because hey a e no speci ically designed o coding cha u e ances, he suppo o in e -code eliabili y me ics is limi ed, and we p e e o be able o expand i s unc ionali y o ou u u e needs and le o he esea che s use i ee o cha ge. Empi ical So wa e Enginee ing (2024) 29 :40 Page 5 o 44 40 (a) Expe imen al g oup — gende ed a a a (b) Con ol g oup — no a a a Fig. 1 wincode use in e ace o subjec s in he expe imen al and con ol g oups (o iginal s udy e sion) 40 Page 6 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 Table 1 Cha u e ance ags by Rod íguez e al. (2017) augmen ed wi h o hogonal in o mal/ o mal ags Tag Desc ip ion Examples I In o mal LOL! Hahaha! F Fo mal All messages excep in o mal S S a emen o in o ma ion o explana ion We need o c ea e a p og am o kids o lea n ma h U Opinion o indica ion o unce ain y Unsu e how o add s ings oge he D Explici ins uc ion Wai pu he i back SU Poli e o indi ec ins uc ion Maybe we can do i use choice = + ACK Acknowledgemen Oh ok go cha M Me a-commen o e lec ion Hmmm QYN Yes/no ques ion Can he answe be nega i e? QWH Wh- ques ion (who, wha , whe e, when, why, and how) How do I ake in hei inpu ? AYN Answe o yes/no ques ion Yea AWH Answe o wh-ques ion The p og am should be able o gene a e e oneous ques ions FP Posi i e ask eedback Oh nice FNON Non-posi i e ask eedback Tha s wei d O O - ask Wow i s swee in his oom ques ionnai es; (ii) he usabili y and pe o mance o he wincode pla o m; and (iii), he applicabili y o he cha u e ance coding based on he one p oposed by Rod íguez e al. (2017) and shown in Table 1, wo pilo s udies wi h a limi ed numbe o s uden s we e ca ied ou a he Uni e si y o Se ille and Uni e si y o Cali o nia, Be keley (UC Be keley) du ing he 2020–21 academic yea . As a esul , he ques ionnai es we e eo ganized in o h ee scales ha we e assessed o in e nal consis ency (see Appendix A), he ini ial se o cha u e ance codes was augmen ed wi h o mali y codes, and he pe o mance and eliabili y o he wincode pla o m was imp o ed. 1.3 O he Gende Iden i ies While we ecognize ha many So wa e Enginee ing s uden s may no iden i y as ei he men o women, ou ini ial explo a ion ocuses p ima ily on in e ac ions be ween s uden s who iden i y as one o hese. The po en ial biases in in e ac ions in ol ing gende - luid, gende - noncon o ming, and nonbina y s uden s is a complex opic dese ing i s own subsequen s udy. 1.4 S uc u e o he Pape The es o he pape is o ganized as ollows. Sec ion 2 e iews ela ed wo k, al hough o ou knowledge, his is he i s s udy speci ically ocusing on he impac o gende bias wi hin pai s in pai p og amming. Sec ions 3 and 4 desc ibe he o iginal s udy ca ied ou a he Empi ical So wa e Enginee ing (2024) 29 :40 Page 7 o 44 40 Uni e si y o Se ille (Decembe 2021) and i s i s ex e nal eplica ion pe o med a UC Be keley (May 2022) espec i ely. Sec ion 5discusses he wo s udies and he h ea s o hei expe imen al alidi y. Finally, Sec ion 6d aws conclusions and p oposes u u e wo k. 2 Rela ed Wo k Se e al sys ema ic li e a u e e iews (SLR’s), which a e summa ized in Table 2,ha ecom- piled he empi ical esea ch on pai p og amming in highe educa ion, including (da Sil a Es ácio and P ikladnicki 2015), which is ocused on dis ibu ed pai p og amming om a eaching pe spec i e. The SLR by Salleh e al. (2010) e eals ha he mos impo an ac o unde s udy is solo e sus pai p og amming in e ms o e ec i eness, quali y o code, and sa is ac ion while s u- den s a e p og amming, concluding ha pai p og amming is mo e e ec i e and sa is ac o y han solo p og amming. Howe e , wi h espec o quali y, indings a e inconclusi e. O he SLR’s, such as he ones by Hanks e al. (2011), Kau Chahal e al. (2021), and Hawli schek e al. (2022), show ha he ocus o he s udies is b oadened, including ac o s such as pe sonali y, mo i a ion, p oblem sol ing, oubleshoo ing, e iciency, con idence, sel -es eem, skill le el, gende , o enjoymen bu no gende bias. In gene al, s uden s a e pai p og amming posi i ely compa ed o solo p og amming. Ne e heless, pai p og amming is e ec i e bu no always e icien , as i may ake longe . By means o con olled expe imen s, emo e and co-loca ed pai p og amming a e com- pa ed by S o s e al. (2003) and Al-Ja ah and Pon elli (2016), showing simila esul s. In mos cases, he analyzed a iables a e ela ed o pe o mance in e ms o ime, quali y, o code es s passed. S uden s pe cep ions ha e also been analyzed in e ms o con idence, sa is ac ion, mo i a ion, o pe sonali y by Salleh e al. (2014). Rega ding p ima y s udies, Table 3summa izes he empi ical s udies on he in luence o gende in pai p og amming, including indings such as (i) same-gende pai s a e mo e “democ a ic”; (ii) women wo king in pai s we e mo e con iden han hose wo king solo; and iii) in mixed-gende pai ings, women a e less con iden compa ed o same-gende pai ings, and epo no inc ease in enjoymen o pai p og amming compa ed o solo p og amming, an e ec ha is signi ican ly obse ed in men (Kau Chahal e al. 2021). Al hough such s udies e eal ha gende seems o be a key ac o , none o hem s udy gende bias in pai p og amming. Many ac o s o he han gende may a ec he ou comes o emo e p og amming sessions (Chapa o e al. 2005; Thomas e al. 2003). P e ious esea ch on p oduc i e pai ing looked a ac o s such as skill le els, au onomy in choosing one’s pa ne (Xinogalos e al. 2017), and di e en pe sonali ies (Hannay e al. 2010). Ne e heless, he wo k on gende composi ion o pai s ound con lic ing esul s abou whe he same-gende o mixed-gende pai ings a e mo e e ec i e (Choi 2015,2013;Ho e 2015; Kau Ku al e al. 2019). One possible explana ion is ha gende co ela es wi h o he dimensions ha may a ec he pai s’ collabo a ion, bu hese co ela ions may a y be ween di e en en i onmen s. Fo example, women in a class may, on a e age, ha e highe skill le el han men because hey had o ace mo e socie al ba ie s o en e he class. On he o he hand, hey may, on a e age, ha e lowe skill le el i women wi h no backg ound a e mo e ac i ely ec ui ed. 40 Page 8 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 Table 2 Summa y o seconda y s udies (SMS o SLR) in pai p og amming in ch onological o de Re e ence Selec ed Pape s Main Fac o s Analyzed Conclusions Salleh e al. (2011) 74 pape s selec ed om 1999 - 2000 pe iod; Con olled, longi udinal, obse a ional s ud- ies ei he in es iga ing ac o s impac ing e ec i eness o PP (23%) o hose measu ing e ec i eness (90%) ia a ious me ics such as quali y (44%) 14 compa ibili y ac o s (pe sonali y ype, ac ual and pe cei ed skill le el, communi- ca ion skills, sel -es eem, gende , e hnici y, lea ning s yle, wo k e hic, ime manage- men abili y, eel-good ac o , con idence le el, ype o ole and ype o asks) and 4 main measu es o e ec i eness in PP: echnical p oduc i i y ( ime spen , knowl- edge/skill ans e , ask pe o mance, code accu acy, numbe and ypes o p oblem, num- be o solu ions o pass es cases), p og am design and quali y (expe opinion, s d qual- i y model, code co e age, numbe o es s passed/ ailed, LOC, design sco es/quali y, numbe o code de ec s), academic pe - o mance (assignmen , inal, mid e m quiz, p ojec and es sco es, cou se g ade, cou se comple ion a e, e en ion a e), sa is ac ion (pai o ma ion, inc eased knowledge and con idence, posi i e a i ude abou collabo a- ion, enjoymen and social in e ac ion) Pai ed s uden s epo highe sa is ac ion and achie ep oduc i i ysimila o be e han solo s uden s. Implemen ing PP in he class oom o lab does no lead o any de imen al e ec on s uden s’ academic pe o mance. While PP had no signi ican ad an age in imp o ing s u- den s’ pe o mance in inal exams o e solo p og amming (e ec size = 0.16) i was e ec- i e in helping s uden s ge be e sco es in hei assignmen s (e ec size = 0.67). Pe son- ali y ype, ac ual and pe cei ed skill le els a e in es iga ed he mos in PP s udies bu e ec s o pe sonali y we e inconclusi e. Pai wo ks well when bo h s uden s ha e simila abili ies and mo i a ion. S uden s p e e o pai wi h someone o simila skills, and s uden s’ skill le el is he mos impo an ac o in luencing e ec i eness o PP. Mos popula me ic o measu e p oduc i i y is ime spen on com- ple ing asks. Code quali y is ano he common me ic o measu e p oduc i i y ha can be measu ed as in e nal, ex e nal o gene al ca e- go ies. When quali y was measu ed acco ding o academic pe o mance and expe opin- ion (ex e nal), s uden s who pai -p og ammed p oduced a be e quali y p og am compa ed o s uden s who p og ammed alone. Howe e , when he quali y o he wo k p oduced by pai and solo s uden s was measu ed using me ics a he in e nal code le el, esul s we e con a- dic o y. Empi ical So wa e Enginee ing (2024) 29 :40 Page 9 o 44 40 Table 2 con inued Re e ence Selec ed Pape s Main Fac o s Analyzed Conclusions Saini e al. (2021) 68 pape s selec ed ia hema ic analysis Compa ing PP s. SP se ings, s uden pe o - mance, s uden a i ude and enjoymen PP has mixed e ec s on s uden s’ pe o - mance, bu almos uni e sally posi i e e ec s on s uden s’ a i udes (i.e. enjoymen ) owa d p og amming. Analysis poin ou he p ob- lems such as small sample size in majo i y o he s udies p e en ing gene aliza ion o he esul s; he lack o con ex in epo ed PP expe imen s obscu es alidi y o esul s, and ha longi udinal analysis o PP expe imen s wi h lea ning asks o inc easing size and complexi y is impo an o build eal-wo ld e idence o he p ac ice. Ko be and Mo schnig (2021) 41 pape s published a e 2000; Compa ing solo s pai p og amming, pe - sonali y, mo i a ion, p oblem sol ing, ou- bleshoo ing, e ec i eness, e iciency, con i- dence, sel es eem, skill le el, gende , enjoy- men PP posi i ely impac s mo i a ion, sel -es eem and con idence o lea ne s. S uden s epo mo e un sol ing he assignmen s and hink PP helps hem sol e p oblems as e . Pai p og amming s uden s epo ed mo e e ec- i e p og amming in bo h isual and ex - based languages bu ea ned mo e achie emen poin s in in he ex -based language (py hon) compa ed o he isual. PP is e ec i e, bu no always e icien . S uden s who ind in o- duc o y p og amming o lea ning a ex -based language especially bene i om he posi i e e ec s o pai p og amming: an app ecia i e and clea communica ion ega ding possible mis akes and misunde s andings is one o he key ac o s. Social ac o s such as gende , pe - sonal ela ionships, e ec s o successes and ailu es (a i ude), dis ibu ion o wo kload and he in luence o pa ne changes mus be conside ed. 40 Page 16 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 hei pa ne s in he p e iously pe o med pai p og amming exe cise (see Sec ion A.3 in he Appendix). They a e ins uc ed o p e ix posi i e aspec s wi h a plus sign (+) and nega i e ones wi h a minus sign (-). This a iable is he esul o au oma ically coun ing he numbe o plus signs in he ex o he open ques ion. pna a io a iable coun ing he numbe o pa ne ’s nega i e aspec s iden i ied by he subjec a e each in-pai ask (see RQ3). In a simila way o he ppa a iable, his a iable is he esul o au oma ically coun ing he numbe o minus signs in he ex o he a o emen ioned open ques ion (see also Sec ion A.3 in he Appendix). ppgende nominal a iable measu ing he pe cei ed pa ne ’s gende du ing he in-pai asks. To measu e his a iable, subjec s a e asked in ques ionnai e #3 whe he hey emembe i hei pa ne s showed some a a a s in cha windows o no . I he answe is no o I don’ emembe (id ), his a iable is assigned he none o id le els a 1and 2. I he answe is yes, hen he subjec s a e asked o he a a a s o he i s and second pa ne , ha ing man, woman,o id as op ions, as shown in Fig. 6. cps in e al a iable composed o i e 0–10 nume ical esponse i ems (cps1...5) measu ing whe he he subjec pe cei ed be e skills in hei i s o second pa ne in he in-pai asks, i.e., compa ed pa ne s’ skills (see RQ4). Low alues co espond o he i s pa ne , whe eas high alues co espond o he second pa ne (see Sec ion A.4 in he Appendix o all he esponse i ems). In he case o he expe imen al g oup only, his a iable is ans o med a e collec ion in such a way ha low alues co espond o he pa ne o whom he induced gende was man, and high alues o he pa ne o whom he induced gende was woman, in o de o analyze whe he he e is a gende bias in he sco ing. 3.4.2 Beha io -Rela ed Va iables ( wincode Pla o m) The esponse a iables au oma ically collec ed by he wincode pla o m and ela ed o he beha io du ing he in-pai p og amming exe cises (see RQ5) a e lis ed below. E e y a iable ep esen s a equency, i.e., a coun , and i s associa ed ela i e equency is compu ed wi h espec o he he sum o he equencies o he wo subjec s in a pai . Fo example, le us suppose ha subjec s iand ja e he wo membe s o a pai , and iand ja e he co esponding alues o he a iable. In his case, he ela i e equencies o each subjec would be i i+ j and j i+ j, espec i ely. sca / sca_ Ra io scale a iables ep esen ing he coun and ela i e equency o cha ac e s added by a subjec o he sou ce code window du ing an in-pai ask (sou ce code addi ions). scd / scd_ Ra io scale a iables ep esen ing he coun and ela i e equency o cha ac e s dele ed by a subjec om he sou ce code window du ing an in-pai ask. (sou ce code dele ions). ok / ok _ Ra io scale a iables ep esen ing he coun and ela i e equency o success ul (ok) alida ions o he sou ce code pe o med by a subjec du ing an in-pai ask. ko / ko _ Ra io scale a iables ep esen ing he coun and ela i e equency o unsuc- cess ul (ko) alida ions o he sou ce code pe o med by a subjec du ing an in-pai ask. Empi ical So wa e Enginee ing (2024) 29 :40 Page 17 o 44 40 dm /dm_ Ra io scale a iables ep esen ing he coun and ela i e equency o dialog messages (cha u e ances) sen by a subjec du ing an in-pai ask. 3.4.3 Communica ion-Rela ed Va iables (U e ance Tagging) The cha u e ances egis e ed in he wincode pla o m du ing he in-pai asks we e manually agged acco ding o wo o hogonal dimensions. The i s dimension uses he 13 ags ( om S o Oin Table 1) p oposed by Rod íguez e al. (2017). The second dimension classi ies each message as o mal o in o mal, conside ing as o mal he usual way in which a uni e si y s uden would communica e ex ually o a p o esso and in o mal o he wise. Fo he agging p ocess, we ollowed a p ocess inspi ed by he wo k o O’Conno and Jo e (2020), in which wo esea che s each agged 60% o he da a, co e ing all dialogue messages. The o e lapping subse o 20%, which was used o he ini ial aining, es ablished he in e -code eliabili y using Cohen’s kappa,whichwasκ= 0.796 o he o mal/in o mal ags, and κ= 0.754 o Rod íguez e al. ags, bo h indica ing subs an ial ag eemen and su icien eliabili y o u he coding acco ding o Syed and Nelson (2015). Fig. 6 Sec ion in ques ionnai e #3 o pa ne ’s pe cei ed gende (ppgende ) a iable 40 Page 18 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 The esponse a iables ela ed o he manual agging o he cha u e ances (see RQ6and RQ7) co espond o he ags in Table 1and a e lis ed below. E e y a iable ep esen s a equency, i.e., a coun , and i s associa ed ela i e equency is compu ed wi h espec o he numbe o cha u e ances gene a ed by he subjec du ing an in-pai ask, which is de ined by he dm a iable speci ied in p e ious sec ion. i/i_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o in o mal messages gene a ed by a subjec du ing an in-pai ask. / _ Ra io scale a iables ep esen ing he absolu e and ela i e equency o o mal messages gene a ed by a subjec du ing an in-pai ask. s/s_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o s a emen o in o ma ion o explana ion messages gene a ed by a subjec du ing an in-pai ask. u/u_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o opinion o indica ion o unce ain y messages gene a ed by a subjec du ing an in-pai ask. d / d_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o explici o di ec ins uc ion messages gene a ed by a subjec du ing an in-pai ask. su / su_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o poli e o indi ec ins uc ion o sugges ion messages gene a ed by a subjec du ing an in-pai ask. ack / ack_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o acknowledgmen messages gene a ed by a subjec du ing an in-pai ask. m/m_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o me a– commen o e lec ion messages gene a ed by a subjec du ing an in-pai ask. qyn / qyn_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o yes/no ques ion messages gene a ed by a subjec du ing an in-pai ask. qwh / qwh_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o wh- ques ion (who, wha , whe e, when, why, and how) messages gene a ed by a subjec du ing an in-pai ask. ayn / ayn_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o answe o yes/no ques ion messages gene a ed by a subjec du ing an in-pai ask. awh / awh_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o answe o wh- ques ion messages gene a ed by a subjec du ing an in-pai ask. p / p_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o posi i e ask eedback messages gene a ed by a subjec du ing an in-pai ask. non / non_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o non–posi i e ask eedback messages gene a ed by a subjec du ing an in-pai ask. o / o_ Ra io scale a iables ep esen ing he absolu e and ela i e equency o o – ask messages gene a ed by a subjec du ing an in-pai ask. 3.5 Con ounding Va iables The con ounding a iables ha we e con olled du ing bo h s udies a e desc ibed below. Empi ical So wa e Enginee ing (2024) 29 :40 Page 19 o 44 40 Table 4 Con ingency able o induced pa ne ’s gende (ipgende ) s. pe cei ed pa ne ’s gende (ppgende ) Pe cei ed Gende Induced Gende man woman none id man 28 (60.87%) 1 (2.17%) 6 (13.05%) 11 (23.91%) woman 1 (2.17%) 27 (58.70%) 6 (13.05%) 12 (26.09%) none 0 (0.00%) 0 (0.00%) 52 (56.52%) 40 (43.48%) 3.5.1 Subjec ’s echnical skills To con ol he a iabili y caused by each subjec on hei pa ne , pai s we e kep he same du ing he en i e expe imen , al hough he subjec s we e no in o med abou his ac . Ideally, his would make he condi ions o he wo in-pai asks he same excep o he p og amming exe cises (see below) and o he induced gende in he case o he expe imen al g oup. 3.5.2 P og amming exe cises In o de o a oid po en ial di e ences among he p og amming exe cises used du ing in-pai asks, hey we e all o simila complexi y and we e andomly assigned. 3.6 Da a Analysis The da a analysis was pe o med only o hose subjec s conside ed as alid acco ding o he ollowing c i e ia: (i) o ha e illed in bo h ques ionnai es; (ii) o ha e hei me ics co ec ly collec ed by he wincode pla o m; (iii) o ha e been pai ed wi h ano he alid subjec ; and (i ) no o ha e disclosed hei gende o hei pa ne ’s du ing he in-pai exe cises; This esul ed in 46 pai s, i.e. 92 alid subjec s, wi h only 9 subjec s d opped because o echnical p oblems wi h hei connec ions o he wincode pla o m, as p e iously men ioned in Sec ion 3.1. 3.6.1 Co ela ion be ween Induced and Pe cei ed Gende Be o e analyzing be ween and wi hin-g oup ela ionships, he co ela ion o he induced and pe cei ed gende in bo h g oups was analyzed in o de o know whe he he ea men had been e ec i ely adminis e ed o he subjec s7. Fo ha pu pose, he esul s o he con ingency able in Table 4we e analyzed obse ing ha he pe cen age o subjec s who we e induced o hink ha hei pa ne was a man and ha e ec i ely emembe ed hey saw a man a a a was close o 61%, whe eas in he case o woman a a a s he pe cen age was close o 59%. Al hough C ame ’s V o Table 4showed ala ge e ec (0.709) acco ding o G a e e and Wallnau (2004), we decided o exclude om he emaining analyses hose subjec s in he expe imen al g oup o whom he induced and pe cei ed gende did no ma ch, because we conside ed ha he ea men had no been 7Theanalysis o he co ela ion be ween induced and pe cei edgende was no included in he egis e ed epo o iginally submi ed o ESEM’2021 (Du án e al. 2021). We included i hanks o he e iewe s’ commen s, whose sugges ion has de ini ely imp o ed ou analysis. 40 Page 20 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 su icien ly e ec i e in hei cases8. On he o he hand, we kep hose subjec s in he con ol g oup who did no pe cei ed any gende ed a a a o did no emembe i , disca ding he es . As a esul , we kep all he subjec s in he con ol g oup (39 men, 6 women, 1 non-bina y) bu only 27 (21 men, 6 women) in he expe imen al g oup. 3.6.2 Be ween-G oups Analysis In he analysis be ween he con ol and expe imen al g oups, o e e y esponse a iable excep o cps9, we compu ed he dis ance be ween he wo in-pai asks as he absolu e alue o he di e ence, i.e. | ( 2)− ( 1)|, since he sign o ha di e ence was no ele an in ou case. In ou esea ch hypo hesis, his dis ance should be smalle o he s uden s in he con ol g oup, who ecei ed no in o ma ion abou hei pa ne s’ gende s i.e. no ea men , han o hose in he expe imen al g oup who e ec i ely pe cei ed wo di e en pa ne s’ gende s a 1and 2. The e o e, o e e y esponse a iable excep o cps, we pe o med a one- ailed unpai ed mean di e ence es be ween g oups, applying a - es o a Mann-Whi ney U es (also known as Wilcoxon es ), depending on he esul s o he no mali y assump ion es s. In he case o he cps a iable, o he con ol g oup we expec ed he mean o be close o he middle poin (5) be ween he i s and second pa ne , as hey we e unconsciously compa ing he skills o he same pe son. Fo he expe imen al g oup, we expec ed he mean o be skewed owa ds 0 (pa ne pe cei ed as a man) o 10 (pa ne pe cei ed as a woman) due o he e ec o he ea men . The e o e, o de ec di e ences be ween g oups o he cps esponse a iable, we pe o med an unpai ed wo- ailed - es because da a dis ibu ion was no signi ican ly di e en om no mal dis ibu ion. Con a y o ou esea ch hypo hesis, no signi ican di e ences we e obse ed a α=0.05 be ween he con ol and expe imen al g oups o any o he 45 esponse a iables desc ibed in Sec ion 3.4, including cps. The co esponding boxplo s a e depic ed in Fig. 7, whe e i can be seen ha he di e ence be ween means— he ci cles in he boxes—in bo h g oups we e e y small. Pos hoc powe analyses using G*Powe (Faul e al. 2007) yielded a s a is ical powe (1−β) o he applied es s o 0.132 o small e ec sizes (d≤0.2), 0.548 o medium e ec sizes (d≤0.5), and 0.915 o la ge e ec sizes (d≤0.8). 3.6.3 Wi hin-g oups Analysis Wi hin he expe imen al g oup, we wan ed o analyze whe he he e we e di e ences be ween he esponse a iables when he same subjec s pe cei ed hei s pa ne s as men o women acco ding o ou esea ch hypo hesis. We also wan ed o s udy he possible in e ac ion be ween he pe cei ed pa ne ’s gende and he subjec ’s gende . Fo hose pu poses, we pe o med a wo-sided pai ed mean di e ence es o e e y esponse a iable excep o cps, using he pe cei ed gende (ppgende ) as a wi hin-subjec s a iable, and applying a - es o a Wilcoxon es depending on he esul s o he no mali y assump ion es s. Fo s udying he in e ac ion, we pe o med he co esponding mixed-model 8 We applied his s ic selec ion o subjec s in he expe imen al g oup in a manne consis en wi h he esul s o he co ela ion analysis, conside ably educing he numbe o subjec s, especially in he eplica ion epo ed in Sec ion 4. 9 As commen ed in i s desc ip ion in Sec ion 3.4.1, he cps a iable is measu ed only once a he end o he expe imen al p ocess, since i compa es i s and second pa ne s’ skills. Empi ical So wa e Enginee ing (2024) 29 :40 Page 21 o 44 40 Fig. 7 Boxplo s o he 45 esponse a iables o be ween-g oups analysis in he o iginal s udy 40 Page 22 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 Table 5 Es ima ed e ec s on expe imen al alidi y o he changes in oduced in he eplica ion E ec on expe imen al alidi y Change desc ip ion Cons uc In e nal Ex e nal Conclusion Thi d o i s yea s uden s – – +2– Spanish s uden s o U.S. s uden s – – +2– Highe pe cen age o women – – +2– Numbe o subjec s educed – – – −2 5% g ade bonus o $15 Amazon gi ca d – – – – Remo e loca ion o subjec s +2−1– – Highe numbe o sessions – −1– – Reduced ime o asks −1−1– – Di e en a a a s −1–– – Blocked exe cise assignmen – +2– – Di e en p og amming language – – – – Legend: –: i does no a ec ; −1/+1: sligh ly inc eases/dec eases; −2/ + 2: mode a ely inc eases/dec eases; −3/ + 3: subs an ially inc eases/dec eases wo-way ANOVA’s wi h he pe cei ed gende (ppgende ) as a wi hin-subjec s a iable and he subjec ’s gende (gende ) as a be ween-subjec s a iable. Fo he cps a iable, which passed he Shapi o-Wilk no mali y es s, we analyzed whe he he subjec ’s gende had any e ec when compa ing pa ne s pe cei ed as man o woman by means o a wo- ailed unpai ed - es be ween g oups, using gende as a be ween-subjec s a iable. Con a y o ou esea ch hypo hesis, no signi ican di e ences we e obse ed a α=0.05 be ween he wo le els o he ppgende a iable o any o he 44 esponse a iables desc ibed in Sec ion 3.4. None o he 44 ANOVA es s de ec ed any signi ican in e ac ion ei he , and no e ec o he subjec ’s gende on he cps a iable was de ec ed. Pos hoc powe analyses using G*Powe (Faul e al. 2007) yielded a s a is ical powe (1 − β) o he applied es s o 0.263 o small e ec sizes (d ≤ 0.2), 0.811 o medium e ec sizes (d ≤ 0.5), and 0.991 o la ge e ec sizes (d ≤ 0.8). As depic ed in Fig. 8, he co esponding boxplo s show e y small di e ences be ween means when pa ne s a e pe cei ed as men o women in he expe imen al g oup. 4 Fi s Replica ion (Be keley May, 2022) In his sec ion, he i s eplica ion ca ied ou a he Uni e si y o Cali o nia Be keley in May 2022 is epo ed ocusing mainly on he changes in he pa icipan s and he expe imen execu ion wi h espec o he o iginal expe imen , since he esea ch ques ions and a iables we e he same in bo h s udies. Fo each change, an es ima ion o hei impac on he ou ypes o expe imen al alidi y desc ibed by Wohlin e al. (2012) is included, ollowing he ecommenda ions by C uz e al. (2023) abou epo ing he impac o changes in eplica ions using a 7-poin disc e e scale om −3 o+3. A summa y o he impac o hose changes is p esen ed in Table 5, including he labels o he a o emen ioned scale in i s legend. Empi ical So wa e Enginee ing (2024) 29 :40 Page 23 o 44 40 Fig. 8 Boxplo s o he 45 esponse a iables o wi hin-g oups analysis in he o iginal s udy 40 Page 24 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 4.1 Pa icipan s In he eplica ion ca ied ou a he Uni e si y o Cali o nia, Be keley, he pa icipan s we e mainly i s yea s uden s en olled in he CS61A (The S uc u e and In e p e a ion o Com- pu e P og ams) and CS88 (Compu a ional S uc u es in Da a Science) cou ses. Applying he same c i e ia han o he o iginal expe imen , he inal numbe o alid subjec s was 46, a anged in 23 pai s. Only 6 s uden s, i.e. 3 pai s, we e excluded om he ini ial 52 pa icipan s. One pai was d opped due o he disclosu e o hei iden i ies du ing he pai p og amming asks; ano he pai was d opped because one o i s pa ne s did no ac i ely pa icipa e in he expe imen al asks; and he hi d pai was excluded because hey los hei connec ion o he wincode pla o m epea edly and hei me ics could no be p ope ly collec ed. Among he emaining 46 alid subjec s, 26 iden i ied as woman (56.52%) and he es as man (43.48%) du ing he egis a ion p ocess10. No e ha , con a y o he o iginal expe imen , he pe cen age o women is abo e ha o men because he CS61A and CS88 in oduc o y cou ses a e aken also by s uden s om o he majo s, usually wi h a highe p esence o women han in Compu e Science majo s, whe e is a ound 25% (Uni e si y o Cali o nia, Be keley 2021). No e also ha despi e he 6 d opped subjec s, he pe cen age o women in he con ol (12 women, 52.17%) and expe imen al (14 women, 60.87%) g oups we e close o each o he . F om ou poin o iew, his change in he sampled popula ion om hi d-yea Spanish s uden s o i s -yea U.S. s uden s, and he highe pe cen age o women, inc eased ex e nal alidi y, bu he educ ion in 50% o he numbe o subjec s (46 pai s o 23 pai s) educed conclusion alidi y. 4.2 Expe imen Execu ion The expe imen execu ion a he Uni e si y o Cali o nia, Be keley ollowed he same p ocess han ha pe o med a he Uni e sidad de Se illa wi h some changes, which a e desc ibed in he ollowing sec ions. 4.2.1 Bonus o Pa icipa ing in he S udy As commen ed in Sec ion 3.2, in he o iginal expe imen he pa icipa ion in he s udy coun ed o a 5% bonus on s uden s’ g ades in he Requi emen s Enginee ing cou se hey we e en olled in o p e en d opou . In he eplica ion, conside ing ha he s uden s we e en olled in wo di e en cou ses wi h di e en p o esso s, hey we e o e ed a $15 Amazon gi ca d o pa icipa ing ac i ely in he s udy ins ead o a g ade bonus which would ha e been di icul o manage. In ou opinion, his change did no a ec any ype o expe imen al alidi y. 4.2.2 Loca ion o S uden s and Numbe o Sessions In he o iginal expe imen , he expe imen al execu ion ook place du ing one o he labo a o y sessions o he Requi emen s Enginee ing cou se, as shown in Fig. 4. The h ee g oups o he cou se had he labo a o y sessions he same day a di e en hou s, wi h 30 s uden s pe session on a e age. In he eplica ion, he s uden s pe o med he expe imen al asks 10 The only s uden who epo ed a non-bina y gende desc ibed as “who ca es” was one o he 6 excluded s uden s. Empi ical So wa e Enginee ing (2024) 29 :40 Page 25 o 44 40 emo ely, coo dina ed by one o he expe imen e s using Zoom. The e we e ou sessions ha ook place du ing a week wi h 10 s uden s pe session on a e age. We hink ha his change inc eased cons uc alidi y wi h espec o he o iginal s udy, since he se ing was s ic ly emo e a he han being co-loca ed in a labo a o y oom, bu i also dec eased in e nal alidi y because o he lack o con ol o he subjec ’s en i onmen , in which in e ac ions wi h a hi d pe son, in e up ions, o dis ac ion could occu . On he o he hand, ha ing mul iple sessions o e a week a he han ha ing h ee consecu i e sessions on he same day also dec eased in e nal alidi y due o he possibili y o some s uden s disclosing he pu pose o he s udy o hei pee s despi e being ins uc ed no o do so. 4.2.3 Timing o he Tasks In he o iginal expe imen , he s uden s we e gi en 20 minu es o he pai p og amming asks, 10 minu es o he solo ask, 10 minu es o he i s ques ionnai e, and 15 minu es o he second and hi d ques ionnai es. In he eplica ion, he s uden s we e gi en 15 minu es o he in-pai asks, 10 minu es o he solo ask, 10 minu es o he i s ques ionnai e, and 10 minu es o he second and hi d ques ionnai es, due o he cons ain s imposed by hei busy schedule. We hink ha he sho ened du a ion o he in-pai asks and he second and hi d ques ion- nai es may ha e comp omised cons uc alidi y by educing he ime span o measu ing he esponse a iables, he in e ac ion ime o assessing he pa ne s’ skills, and he e lec ion ime be o e answe ing each esponse i em. Mo eo e , i may ha e weakened he e ec o he ea men o e con ounding a iables, hus dec easing also in e nal alidi y. 4.2.4 Gende ed A a a s In he o iginal expe imen , he gende ed a a a s used in he cha windows o he subjec s in he expe imen al g oup we e he silhoue es shown in Fig. 9a, whe eas in he eplica ion he a a a s we e hose shown in Fig. 9b, which we e gene a ed a h ps://ge a a aaa s.com/.The subjec s in he eplica ion we e also shown a gende ed message a he op o he cha window indica ing ha hei pa ne was connec ed, e.g. “You pa ne (she/he ) is connec ed” (see Figs. 16aand16b in Appendix B). In p inciple, changing he gende ed silhoue e a a a s by mo e explici ones and adding a gende ed message in he cha window would ha e inc eased cons uc alidi y, bu he co ela ion be ween induced gende and pe cei ed gende in he eplica ion wo sened wi h (a) O i g inal expe imen (b) Replica ion Fig. 9 Gende ed a a a s used in he o iginal expe imen and he eplica ion 40 Page 32 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 Wi h espec o he ex e nal eplica ion, we only obse ed s a is ically signi ican e ec s wi hin he expe imen al g oup, i.e. compa ing how subjec s ac ed when hey hough hei pa ne was a man o a woman, in ou dependen a iables. One a iable was ela ed wi h changes in he beha io (sou ce code dele ions), and he o he h ee we e ela ed wi h he ela i e equency o di e en ype o cha u e ances (in o mal messages, e lec ions, and yes/no ques ions). In he case o he sou ce code dele ions, subjec s dele ed mo e cha ac- e s when hey pe cei ed hei pa ne s as a woman, bu he ela i e equency o in o mal messages, e lec ions, and yes/no ques ions was highe when hey pe cei ed hei pa ne s as a man. We also obse ed a lowe e ec i eness o he ea men in he eplica ion, ha could be caused by he changes in he gende ed a a a s bu also o ha ing used a emo e se ing ins ead o a con olled en i onmen like a labo a o y session, ee o dis ac ions and in e up ions. Tha lowe e ec i eness o he ea men led o a small numbe o selec ed subjec s in he expe imen al g oup, hus leading o conside he eplica ion esul s inconclusi e because o he small sample hey a e based on, and because when mul iple es co ec ions a e applied, only he esul o he ela i e equency o in o mal messages emains signi ican . These ou comes ha e aised a numbe o po en ial esea ch ques ions ha we plan o add ess in he u u e and ha a e b ie ly desc ibed in he nex subsec ions. Take away messages •No e ec o he gende bias ea men was obse ed in he o iginal s udy. •Di e ences in ou a iables we e obse ed in he eplica ion. Only one ( ela i e equency o in o mal messages) p esen ed s as is ically signi ican di e ences a e mul iple es co ec ions. •S a is ical powe was low due o he educed numbe o pa icipan s. •Possible non-exclusi e explana ions o he esul s a e: –Low ea men e ec i eness: 60% in he o iginal s udy (pe o med in a labo a o y) bu only 40% in he eplica ion (pe o med emo ely). –Me ics no su icien ly sensi i e o he ea men e ec . –In e gene a ional communica ion di e ences in he cha u e ance coding p ocess in he o iginal expe imen . –No so s ong gende bias among cu en So wa e Enginee ing s uden s. –Sel -censo ship o pa icipan s due o socio-poli ical p essu es. 6.1 Replica ion in Di e en Cul u al Backg ound The cul u al di e ences be ween Spanish and U.S. s uden s could ha e also in luenced he ou comes o bo h s udies, so we would like o eplica e i o he coun ies and analyze hose po en ial di e ences caused by cul u al backg ounds. Empi ical So wa e Enginee ing (2024) 29 :40 Page 33 o 44 40 6.2 Using Cha bo s as Pa ne s and AI-based U e ance Coding Ano he wo esea ch lines we would like o explo e in he u u e a e he use o cha bo s as pai p og amming pa ne s and he use o deep lea ning o au oma ically code cha u e ances, hus educing he manual e o o ca ying ou a eplica ion. Inspi ed by cu en ends in Psychology (Bendig e al. 2019; G ee e al. 2019) and aking in o accoun no only he absence o signi ican di e ences be ween g oups in he o iginal s udy and he eplica ion, bu also he di icul ies in ec ui ing a ele an numbe o subjec s, we a e conside ing he possibili y o changing om a be ween-g oups design o a wi hin- subjec design in which each subjec pe o ms he pai p og amming asks wi h a cha bo simula ing being a man o a woman ins ead o wi h ano he human subjec . Ob iously, de eloping such a cha bo is no a i ial ask, bu cu en ad ances in he a ea, such as LaMDA (Collins and Ghah amani 2021), BERT (De lin e al. 2019), o GPT-3 (Lim e al. 2021), make his app oach a echnical challenge wo h explo ing. A e y ele an aspec in he de elopmen o such a cha bo is a oiding gende bias in he aining da a, as ecen ly s udied by McAuli e e al. (2022). On he o he hand, now ha we ha e a ele an numbe o coded cha u e ances in Spanish and English, we could use ha labeled da ase o ine ain a la ge language model sys em simila o hose used in cha bo s o classi y use in en s and apply i o he au oma ic coding o cha u e ances, which is one o he mos ime-consuming asks we ha e had o pe o m as expe imen e s in ou explo a o y s udy. I he esul s o such a ine ained sys em we e accu a e, u u e eplica ions would equi ed much less e o han he wo p esen ed in his a icle and expe imen e bias would be conside ably mi iga ed. Appendix A Ques ionnai e #1 and #2 Response I ems In his sec ion, he esponse i ems o he scales used in ques ionnai es #1 and #2 a e enu- me a ed. Those scales we e analyzed o in e nal consis ency using he da a collec ed du ing he pilo s udies, and he esul s o hose analysis consis ing in he Pea son’s co ela ions, C onbach’s α, and p incipal componen s sc ee plo a e also epo ed (UCLA: S a is ical Con- sul ing G oup 2022), indica ing whe he some esponse i ems we e d opped o no acco ding o he ob ained esul s. A.1 Response I ems o Pe cei ed P oduc i i y Scale (pp) All he i ems in his ques ionnai e sec ion, en i led as “Solo p og amming o pai p og am- ming?”, a e 0–10 nume ical esponse i ems in which 0 means “p og amming solo”, 5 means “ he same in bo h cases”, 10 means “p og amming in pai s”. pp1Rega ding he p og amming exe cises you jus did, how do you hink you would ha e been mo e p oduc i e, p og amming solo o p og amming wi h he pa ne assigned o you? pp2Rega ding he p og amming exe cises you jus did, how do you hink you would ha e achie ed a be e p og am quali y, p og amming solo o p og amming wi h he pa ne assigned o you? 40 Page 34 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 pp3Rega ding he p og amming exe cises you jus did, how do you hink you would ha e de eloped a mo e eliable p og am, i.e., a p og am mo e likely o un wi hou ailu es, p og amming solo o p og amming wi h he pa ne assigned o you? pp4Rega ding he p og amming exe cises you jus did, how do you hink you would ha e enjoyed mo e, p og amming solo o p og amming wi h he pa ne assigned o you? AsshowninFig.12, all he i ems p esen ed high Pea son co ela ions wi h C onbach’s α=0.83, and he sc ee plo con i med hey we e unidimensional acco ding o he Kaise c i e ion. As a esul , all o hem we e kep a e he eliabili y analysis on he da a om he pilo s udies. Fig. 12 Pea son co ela ions and sc ee plo o pp scale i ems A.2 Response I ems o Pa ne ’s Pe cei ed Technical Compe ency (pp c) All he i ems in his ques ionnai e sec ion, en i led as “My pa ne o me?”, a e 0–10 nume i- cal esponse i ems in which 0 means “me”, 5 means “bo h equally”, 10 means “my pa ne ”. pp c1 Du ing he p og amming exe cises you jus did, who do you hink had mo e knowledge and echnical skills, you o he pa ne assigned o you? pp c2 Du ing he p og amming exe cises you jus did, who do you hink has been mo e coope a i e, you o he pa ne assigned o you? pp c3 Du ing he p og amming exe cises you jus did, who do you hink has had a as e pace a sol ing he exe cises, you o he pa ne assigned o you? pp c4 Du ing he p og amming exe cises you jus did, who do you hink has led mo e o he solu ions, you o he pa ne assigned o you? AsshowninFig.13, in he ini ial e sion o he scale used in he pilo s udies, he pp c5 i em, which asked whe he he assigned pa ne had been condescending, p esen ed low co ela ions wi h he es o he i ems in he scale and he sc ee plo indica ed wo ac o s. A e emo ing ha unco ela ed i em, he C onbach’s α inc eased om 0.73 o 0.85, and he sc ee plo indica ed only one ac o , as shown in Fig. 14. Empi ical So wa e Enginee ing (2024) 29 :40 Page 35 o 44 40 Fig. 13 Pea son co ela ions and sc ee plo o he ini ial e sion o pp c scale i ems Fig. 14 Pea son co ela ions and sc ee plo a e d opping pp c5 om pp c scale A.3 Response I em o Pa ne ’s Pe cei ed Posi i e and Nega i e Aspec s (ppa and pna) The only i em in his ques ionnai e sec ion, en i led as “Desc ibe you pa ne ”, is a ee ex ield in which subjec s a e ins uc ed o desc ibe he mos posi i e and mos nega i e aspec s o he pa ne assigned o hem in he p og amming exe cises hey jus did, indica ing he posi i e ones wi h a "+" sign and he nega i e ones wi h a "-" sign in on o each aspec . A.4 Response I ems o Compa ed Pa ne s’Skills (cps) All he i ems in his ques ionnai e sec ion, en i led as “Fi s o second pa ne ?”, a e 0–10 nume ical esponse i ems in which 0 means “ i s pa ne ”, 5 means “bo h equally”, 10 means “second pa ne ”. cps1Compa ing you assigned pa ne s in sessions 1 and 3, who do you hink p o ided mo e clea and cons uc i e eedback, you i s pa ne o you second pa ne ? cps2Compa ing you assigned pa ne s in sessions 1 and 3, who do you hink was easie o communica e wi h, you i s pa ne o you second pa ne ? 40 Page 36 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 cps3Compa ing you assigned pa ne s in sessions 1 and 3, who do you hink who do you hink was mo e knowledgeable abou he subjec ma e ial, you i s pa ne o you second pa ne ? cps4Compa ing you assigned pa ne s in sessions 1 and 3, who do you hink would be a be e p ojec pa ne , you i s pa ne o you second pa ne ? cps5Compa ing you assigned pa ne s in sessions 1 and 3, who do you hink would be a be e eaching assis an , you i s pa ne o you second pa ne AsshowninFig.15, all he i ems p esen ed high Pea son co ela ions wi h C onbach’s α=0.88, and he sc ee plo con i med hey we e unidimensional acco ding o he Kaise c i e ion. As a esul , all o hem we e kep a e he eliabili y analysis on he da a om he pilo s udies. Fig. 15 Pea son co ela ions and sc ee plo o cps scale i ems B E olu ion o he wincode Use In e ace The wincode use in e ace used in he ex e nal eplica ion a UC Be keley is shown in Fig. 16aand16b. Empi ical So wa e Enginee ing (2024) 29 :40 Page 37 o 44 40 (a) Expe imen al g oup — gende ed a a a (b) Con ol g oup — no a a a Fig. 16 wincode use in e ace o subjec s in he expe imen al and con ol g oups ( eplica ion e sion) 40 Page 38 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 C Use In e ace o ag-a-cha The use in e ace o he ag-a-cha ool used o collabo a i ely coding cha u e ances isshowninFig.17. Fig. 17 Use in e ace o he ag-a-cha ool Acknowledgemen s We would like o hank he s uden s who olun ee ed o pa icipa e in he pilo s udies, he o iginal expe imen and he i s eplica ion a he Uni e si ies o Se ille (US) and Cali o nia Be keley (UCB). We also wan o hank Da id B incau (unde g adua e s uden a US) o hei suppo in he de elopmen o he wincodepla o m; José Sando al (Mas e ’s s uden a US) o de eloping ag-a-cha , he collabo a i e ool o agging cha u e ances; and Daewon Kwon and Ka im el Re ai (unde g adua e s uden s a UCB) o hei suppo in he e olu i e changes o he wincodepla o m and in he expe imen execu ion a UCB. We pa icula ly acknowledge V on Vance (UCB alumnus, Da a Analys a Google) o hei assis ance ega ding inclusi e language a ound gende iden i y. Las bu no leas , we would like o hank he anonymous e iewe s o hei aluable commen s and sugges ions ha helped us imp o e he quali y and cla i y o his a icle. Funding Funding o open access publishing: Uni e sidad de Se illa/CBUA. This wo k has been pa ially suppo ed by g an s PID2021-126227NB-C21, PID2021-126227NBC22 unded by MCIN/AEI/10.13039/ 501100011033/FEDER and Eu opean Union “ERDF a way o making Eu ope”; TED2021-131023B-C21, TED2021-131023B-C22 unded by MCIN/AEI/10.13039/501100011033 and Eu opean Union “Nex Gen- e a ionEU”/PRTR; EKIPMENT-PLUS (P18-FR-2895), MEMENTO (US-1381595) unded by Jun a de Andalucía/ERDF, Eu opean Union; and Uni e sidad de Se illa unde he 2021 and 2023 G an s o he Exchange Mobili y o P o esso s, Resea che s, and PhD S uden s be ween he Uni e si y o Se ille and he Uni e si y o Cali o nia. Da a A ailabili y The da ase s gene a ed and analyzed du ing he cu en s udy a e a ailable in he Zenodo eposi o y, h ps://doi.o g/10.5281/zenodo.6783717. Decla a ions Con lic s o in e es The au ho s decla ed ha hey ha e no con lic o in e es wi h any aspec o he epo ed s udies. Empi ical So wa e Enginee ing (2024) 29 :40 Page 39 o 44 40 E hical s anda d The expe imen p o ocols we e app o ed by he Ins i u ional Re iew Boa d (IRB) a UC Be keley. A he Uni e si y o Se ille, only s udies in ol ing expe imen a ion animals o biomedical expe - imen s in ol ing humans need o be app o ed by he E hics Commi ee on Expe imen a ion, so no app o al was equi ed in his case. Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Commons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission di ec ly om he copy igh holde . To iew a copy o his licence, isi h p://c ea i ecommons.o g/licenses/by/4.0/. Re e ences AAUW (2002) The STEM gap: Women and gi ls in science, echnology, enginee ing and ma hema ics. Ame - ican Associa ion o Uni e si y Women. h ps://www.aauw.o g/ esou ces/ esea ch/ he-s em-gap/ Akalın A, Weinman N, S asaski K, Fox A (2021) Explo ing he impac o gende bias on pai p og amming. In: P oceedings o he 17 h ACM con e ence on in e na ional compu ing educa ion esea ch, p 435–437 Al-Ja ah A, Pon elli E (2016) On he e ec i eness o a collabo a i e i ual pai -p og amming en i onmen . In: In e na ional con e ence on lea ning and collabo a ion echnologies, p 583–595 Allai e-Duque e G, Chas enay P, Bou a d T, Bélange SA, He nandez O, Mahhou MA, Gi oux P, McMullin S, Desja lais E (2022) Gende di e ences in sel -e icacy o p og amming na owed a e a 2-h science museum wo kshop. Can J Sci Ma h Techn Educ 22:87–100 Bendig E, E b B, Schulze-Thuesing L, Baumeis e HH (2019) The nex gene a ion: Cha bo s in clinical psychology and psycho he apy o os e men al heal h - a scoping e iew. Ve hal ens he apie. h ps://doi. o g/10.1159/000501812 Chapa o EA, Yuksel A, Rome o P, B yan S (2005) Fac o s a ec ing he pe cei ed e ec i eness o pai p o- g amming in highe educa ion. In: P oceedings o he 17 h wo kshop o he psychology o p og amming in e es g oup Choi KS (2013) E alua ing gende signi icance wi hin a pai p og amming con ex . In P oceedings o he hawaii in e na ional con e ence on sys em sciences, p 4817–4825 Choi KS (2015) A compa a i e analysis o di e en gende pai combina ions in pai p og amming. Beha In Technol 34(8):825–837 Cohen L, Manion L, Mo ison K (2018) Resea ch Me hods in Educa ion, 8 h edn. Rou ledge Collins E, Ghah amani Z (2021) LaMDA: ou b eak h ough con e sa ion echnology. Google Resea ch. h ps:// blog.google/ echnology/ai/lamda/ C uz M, Be ná dez B, Du án A, Gue a a-Vega C, Ruiz-Co és A (2023) A model-based app oach o speci ying changes in eplica ions o empi ical s udies in compu e science. Compu ing 105:1189–1213 da Sil a Es ácio BJ, P ikladnicki R (2015) Dis ibu ed pai p og amming: A sys ema ic li e a u e e iew. In So w Technol 63:1–10 de Oli ei a Ne o FG, To ka R, Feld R, G en L, Fu ia CA, Huang Z (2019) E olu ion o s a is ical analysis in empi ical so wa e enginee ing esea ch: Cu en s a e and s eps o wa d. J Sys So w 156:246–267 Denzin NK (2006) Sociological Me hods: A Sou cebook. 5 h ed. Aldine T ansac ion De lin J, Chang M-W, Lee K, Tou ano a K (2019) BERT: P e- aining o deep bidi ec ional ans o me s o language unde s anding. In P oceedings o he 2019 con e ence o he no h ame ican chap e o he associa ion o compu a ional linguis ics: human language echnologies, Volume 1 (Long and Sho Pape s), p 4171–4186 Dimock M (2019) De ining gene a ions: Whe e millennials end and gene a ion z begins. h ps://pew s .ch/ 2szq Jz Du án A, Fe nández P, Be ná dez B, Weinman N, Akalın A, Fox A (2021) Gende bias in emo e pai p og am- ming among so wa e enginee ing s uden s: The wincode explo a o y s udy. In P oceedings o ESEM 2021 – egis e ed epo ack. a Xi :2110.01962 Eckles D, Kizilcec R, Bakshy E (2016) Es ima ing pee e ec s in ne wo ks wi h pee encou agemen designs. P oc Na l Acad Sci 113(27):7316–7322 40 Page 40 o 44 Empi ical So wa e Enginee ing (2024) 29 :40 El-Re ai K, Kwon D, B incau D, Akalın A, Fox A, Fe nández P, Du án A (2023) Twincode: An ins umen ed pla o m o pai p og amming esea ch. In P oceedings o he 54 h ACM echnical symposium on compu e science educa ion . 2, p 1264 Falessi D, Ju is o N, Wohlin C, Tu han B, Münch J, Jedli schka A, Oi o M (2018) Empi ical so wa e enginee - ing expe s on he use o s uden s and p o essionals in expe imen s. Empi ical So w Eng 23(1):452–489 Faul F, E d elde E, Lang A-G, Buchne A (2007) G*Powe 3: A lexible s a is ical powe analysis p og am o he social, beha io al, and biomedical sciences. Beha Res Me hods 39:175–191 Fishe M, Cox A (2006) Gende and p og amming con es s: Mi iga ing exclusiona y p ac ices. In Educ 5(1):47–62 Galdo AC, Celepkolu M, Ly le N, Boye KE (2022) Pai p og amming in a pandemic: Unde s anding middle school s uden s’ emo e collabo a ion expe iences. In P oceedings o he 53 d ACM echnical symposium on compu e science educa ion V. 1, p 335–341 Gómez O, Sola i M, Cal ache C, Ledezma-Ca izalez A (2017) A con olled expe imen on p oduc i i y o pai p og amming gende combina ions: P elimina y esul s. In P oceedings o he XX Ibe o–Ame ican con e ence on so wa e enginee ing, p 197–210 G aphPad (2023) Wha is he di e ence be ween o dinal, in e al and a io a iables? Why should I ca e?. h ps:// .ly/ xCW G a e e FJ, Wallnau LB (2004) S a is ics o he Beha iou al Sciences. 6 edn. Wadswo h/Thompson Lea n- ing G ee S, Ramo D, Chang Y-J, Fu M, Moskowi z J, Ha i a os J (2019) Use o he cha bo “ i ibo ” o deli e pos- i i e psychology skills and p omo e well-being among young people a e cance ea men : Randomized con olled easibili y ial. JMIR Mheal h Uheal h, 7(10) Hanks B, Fi zge ald S, McCauley R, Mu phy L, Zande C (2011) Pai p og amming in educa ion: A li e a u e e iew. Compu Sci Educ 21(2):135–173 Hannay JE, A isholm E, Eng ik H, Sjobe g DIK (2010) E ec s o pe sonali y on pai p og amming. IEEE T ans So w Eng 36(1):61–80. h ps://doi.o g/10.1109/TSE.2009.41 Ha sell T (2005) Who’s alking online? a desc ip i e analysis o gende & online communica ion. In J In Commun Technol Educ 1(1):42–54 Hawli schek A, Be nd S, Schulz S (2022) Empi ical esea ch on pai p og amming in highe educa ion: a li e a u e e iew. Compu e science educa ion, p 1–29 Ho e SI (2015) S udying gende bias in physics g ading: The ole o eaching expe ience and coun y. In J Sci Educ 37(17):2879–2905 Hoppe J (2014) How o label you 10-poin scale. Ve s a Resea ch. h ps:// e s a esea ch.com/blog/how- o- label-you -10-poin -scale/ Ja a L, Bowman NA, Cul e KC, Seg e AM (2019) A la ge-scale expe imen al s udy o gende and pai composi ion in pai p og amming. In P oceedings o he ACM con e ence on inno a ion and echnology in compu e science educa ion, p 176–181 Ka i a N, Williams L, Osbo ne J (2005) Towa ds inc easing he compa ibili y o s uden pai p og amme s. In: In e na ional con e ence on so wa e enginee ing, p 625–626. h ps://doi.o g/10.1109/ICSE.2005. 1553618 Kau Chahal K, Kau A, Saini M (2021) Resea ch and e idence in so wa e enginee ing: om empi ical s udies o open sou ce a i ac s, chap e empi ical s udies on using pai p og amming as a pedagogical ool in highe educa ion cou ses: A sys ema ic li e a u e e iew, p 251–287. Taylo & F ancis G oup Kau Ku al S, Ge s ne K, Beja ano A (2019) Remo e pai p og amming in online cs educa ion: In es iga ing h ough a gende lens. In 2019 IEEE symposium on isual languages and human-cen ic compu ing (VL/HCC), p 75–85. h ps://doi.o g/10.1109/VLHCC.2019.8818790 Ki chenham BA, P leege SL, Hoaglin DC, Emam KE, Rosenbe g J (2002) P elimina y Guidelines o Empi - ical Resea ch in So wa e Enginee ing. IEEE T ans So w Eng 28(8):721–734 Ko be P, Mo schnig R (2021) The e ec s o pai -p og amming in in oduc o y p og amming cou ses wi h isual and ex -based languages. In IEEE on ie s in educa ion con e ence, p 1-9 Lim R, Wu M, Mille L (2021) Cus omizing GPT-3 o you applica ion. OpenAI. h ps://openai.com/blog/ cus omized-gp -3/ Ma ell RF, Lane DM, Em ich C (1996) Male- emale di e ences: A compu e simula ion. Am Psychol 51(2):157–158 McAuli e A, Ha J, Ku al SK (2022) E alua ing gende bias in pai p og amming con e sa ions wi h an agen . In 2022 IEEE symposium on isual languages and human-cen ic compu ing (VL/HCC), p 1–4. h ps://doi.o g/10.1109/VL/HCC53370.2022.9833146 Medel P, Pou naghshband V (2017) Elimina ing gende bias in compu e science educa ion ma e ials. In P oceedings o he 2017 ACM SIGCSE echnical symposium on compu e science educa ion, p 411–416 Empi ical So wa e Enginee ing (2024) 29 :40 Page 41 o 44 40 Na a o D (2018) Lea ning s a is ics wi h R: A u o ial o psychology s uden s and o he beginne s ( e sion 0.6).h ps://lea nings a is icswi h .com/ Newse (2023) This uni e si y has he mos s essed-ou s uden s. h ps://www.newse .com/s o y/330315/10- mos -leas -s essed-college- owns.h ml O’Conno C, Jo e H (2020) In e code eliabili y in quali a i e esea ch: Deba es and p ac ical guidelines. In J Qual Me hods 19:1–13 Oda F, Lechago SA, da Sil a BE, Hun JC (2022) An expe imen al analysis o gende -biased e bal beha io and sel -edi ing using an online cha analog. J Exp Anal Beha 118(1):24–45 Pe Runeson (2003) Using s uden s as expe imen subjec s - an analysis on g adua e and eshmen s uden da a. In P oceedings 7 h In e na ional con e ence on empi ical assessmen & e alua ion in so wa e enginee ing, p 95–102 Po e AA, Vo a LG, Basili VR (1999) Building Knowledge h ough Families o Expe imen s. IEEE T ans So w Eng 25(4):456–473 RDocumen a ion (2023) p.adjus : Adjus p- alues o mul iple compa isons. h ps://www. documen a ion.o g/ packages/s a s/ e sions/3.6.2/ opics/p.adjus Rod íguez FJ, P ice KM, Boye KE (2017) Explo ing he pai p og amming p ocess: Cha ac e is ics o e ec i e collabo a ion. In P oceedings o he 2017 ACM SIGCSE echnical symposium on compu e science educa ion, p 507–512 Saini M, Chahal KK, Kau A (2021) Empi ical s udies on using pai p og amming as a pedagogical ool in highe educa ion cou ses: A sys ema ic li e a u e e iew. Aue bach Publica ions Salleh N, Mendes E, G undy J (2011) Empi ical s udies o pai p og amming o cs/se eaching in highe educa ion: A sys ema ic li e a u e e iew. IEEE T ans So wa e Eng 37:509–525. h ps://doi.o g/10. 1109/TSE.2010.59 Salleh N, Mendes E, G undy J (2014) In es iga ing he e ec s o pe sonali y ai s on pai p og amming in a highe educa ion se ing h ough a amily o expe imen s. Empi ical So Eng 19(3):714–752 Salleh N, Mendes E, G undy J, Bu ch G (2010) The e ec s o neu o icism on pai p og amming: an empi ical s udy in he highe educa ion con ex . In P oceedings o he 2010 ACM-IEEE in e na ional symposium on empi ical so wa e enginee ing and measu emen , p 1–10 Sama a O, Monzon A (2021) Zoom bu nou amids a pandemic: Pe spec i e om a medical s uden and lea ne . The apeu ic Ad ances in In ec ious Disease, 8 S e sos P, S amelos I, Angelis L, Deligiannis I (2009) An expe imen al in es iga ion o pe sonali y ypes impac on pai e ec i eness in pai p og amming. Empi ical So w Eng 14(2):187–226 STEM Women (2021) Pe cen ages o women in STEM s a is ics. STEM Women. h ps://www.s emwomen. com/women-in-s em-pe cen ages-o -women-in-s em-s a is ics S e ens SS (1946) On he heo y o scales o measu emen . Science 103(2684):677–680 S o s D, Williams L, Nagappan N, Bahe i P, Jen D, Jackson A (2003) Vi ual eaming: Expe imen s and expe- iences wi h dis ibu ed pai p og amming. In: Con e ence on ex eme p og amming and agile me hods, p 129–141 S udy In e na ional (2016) S uden s a hese U.S. uni e si ies a e unde he mos s ess. h ps://www. s udyin e na ional.com/news/s uden s-men al-heal h-us-uni e si ies-s ess/ Syed M, Nelson SC (2015) Guidelines o es ablishing eliabili y when coding na a i e da a. Eme ging Adul hood 3(6):375–387 Te ell J, Ko ink A, Middle on J, Rainea C, Mu phy-Hill E, Pa nin C, S allings J (2017) Gende di e ences and bias in open sou ce: pull eques accep ance o women e sus men. Pee J Compu e Science, 3(e111) Thomas L, Ra cli e M, Robe son A (2003) Code wa io s and code-a-phobes: A s udy in a i ude and pai p og amming. In P oceedings o SIGCSE, p 363–367 UCLA: S a is ical Consul ing G oup (2022) Wha does c onbach’s alpha mean?. Accessed 29-June-2022. h ps://s a s.oa c.ucla.edu/spss/ aq/wha -does-c onbachs-alpha-mean/ Uni e si y o Cali o nia, Be keley (2021) Demog aphic in o ma ion ( es ic ed access). h ps://calanswe s. be keley.edu/home Uni e si y o Se ille (2021) S a is ical yea book 2020–2021. h ps://se icio.us.es/splanes u/WS/ Anua io2021/AESY20-21.h ml. English e sion s a s a page 400 We ne LL, Hanks B, McDowell C (2004) Pai -p og amming helps emale compu e science s uden s. J Educ Resou Compu , 4(1) Wohlin C, Runeson P, Hös M, Ohlsson MC, Regnell B, Wesslén A (2012) Expe imen a ion in So wa e Enginee ing: an In oduc ion. Sp inge Xinogalos S, Sa a zemi M, Cha zigeo giou A, Tsompanoudi D (2017) S uden pe cep ions on he bene i s and sho comings o dis ibu ed pai p og 4amming assignmen s. 2017 IEEE global enginee ing educa ion con e ence (EDUCON), p 1513–152