A Web Simulator to Assist in the Teaching of Bayes’ Theorem

Full Te ms & Condi ions o access and use can be ound a h ps://www. and online.com/ac ion/jou nalIn o ma ion?jou nalCode=ujse20 Jou nal o S a is ics Educa ion ISSN: (P in ) 1069-1898 (Online) Jou nal homepage: h ps://www. and online.com/loi/ujse20 A Web Simula o o Assis in he Teaching o Bayes’ Theo em M. J. Bá cena, M. A. Ga ín, A. Ma ín, F. Tusell & A. Unzue a To ci e his a icle: M. J. Bá cena, M. A. Ga ín, A. Ma ín, F. Tusell & A. Unzue a (2019) A Web Simula o o Assis in he Teaching o Bayes’ Theo em, Jou nal o S a is ics Educa ion, 27:2, 68-78, DOI: 10.1080/10691898.2019.1608875 To link o his a icle: h ps://doi.o g/10.1080/10691898.2019.1608875 © 2019 The Au ho (s). Published wi h license by Taylo & F ancis G oup, LLC Published online: 12 Jun 2019. Submi you a icle o his jou nal A icle iews: 2034 View ela ed a icles View C ossma k da a Ci ing a icles: 1 View ci ing a icles JOURNAL OF STATISTICS EDUCATION 2019, VOL. 27, NO. 2, 68–78 h ps://doi.o g/10.1080/10691898.2019.1608875 A Web Simula o o Assis in he Teaching o Bayes’Theo em M. J. Bá cenaa,M.A.Ga ín a,A.Ma ín a,F.Tusell a, and A. Unzue ab aFacul ad de Economía y Emp esa, UPV/EHU, Bilbao, Spain; bEscuela de Ingenie ía, UPV/EHU, Bilbao, Spain ABSTRACT Teaching some concep s in s a is ics g ea ly bene i s om indi idual p ac ice wi h immedia e eedback. In o de o p o ide such p ac ice o a la ge numbe o s uden s we ha e w i en a simula o based on an his o ical e en : he loss in May 22, 1968, and subsequen sea ch o he nuclea subma ine USS Sco pion. S uden s wo k on a simpli ied e sion o he sea ch and can see p obabili ies change in esponse o new e idence. The simula o is designed o assis in he eaching o Bayesian concep s, in pa icula Bayesian upda ing. I has been deployed in ou cou ses and ou expe ience and esul s a e desc ibed, as well as he eac ions o ou s uden s o i s use. The simula o is open sou ce, eely a ailable and easy o implemen and un, as i only equi es a machine o se e web pages. We explain in de ail ou expe ience wi h i s deploymen and use. KEYWORDS Bayesian s a is ics; S a is ics; Simula o 1. In oduc ion Bayes’ heo em, in i s simples o m exp essible as P(A|B)=P(B|A)P(A) P(B),(1) p o ides a way o “in e ” p obabili ies: om he condi ional p obabili y o Bgi en Aand he espec i e ma ginals, he p ob- abili y o Agi en Bcan be compu ed. Al hough i is simple and canbep esen edandp o edinama e o minu es, hisisa concep ha equi es ime o s uden s o g asp. We ha e ound use ul o e he yea s o p esen some exam- ples ha help enhance s uden ’s comp ehension o he p ac ical implica ions o (1). Medical diagnosis is one: he p obabili y P(A|B)o ha ing a sickness Agi en he p esence o symp om Bcanbeob ainedin e mso hep obabili yo hesymp om gi en hesicknessand he espec i ema ginalp obabili ieso sickness and symp om. This nicely illus a es he way o e ise a p io P(A)in he ligh o newly a ailable in o ma ion B. Acqui ing amilia i y wi h he concep s in ol ed equi es, much as he acquisi ion o a new language, epea ed in e ac ion wi h (1), beyond he ew examples ha can be p esen ed in class. Such amilia i y can be os e ed by assigning homewo k o be done by s uden s and la e g aded, bu his imposes a consid- e able bu den on he eache s and p o ides, a bes , delayed eedback o he s uden . We hough ha a simple simula o , p esen ing each s uden a unique ins ance o a p oblem, wi h immedia e eedback and au oma ic g ading, would be much p e e able. This a icle epo s on ou wo k in his di ec ion. Sec ion 2 e iews some wo k which add esses simila goals as ou s o which we ha e o he wise used o inspi a ion. Sec ion 3 b ie ly desc ibes he s o y we ha e used, in a simpli ied ec ea ion, omo i a eagameinwhichs uden s CONTACT F. Tusell e nando[email p o ec ed] Facul ad de Economía y Emp esa, UPV/EHU, A da. Lehendaka i Agui e, 83, Bilbao, Spain. Supplemen a y ma e ials o his a icle a e a ailable online. Please go o www. and online.com/ujse. a e equi ed oloca eamissingsubma ine.Sec ion 4 desc ibes he implemen a ion. We es ed he use o ou simula o on ou in oduc o y cou se on s a is ics o sophomo es, when Bayes’ heo em is i s p esen ed; Sec ion 5 gi es some de ails abou he esul s ob ained. Sec ion 6 closes wi h some commen s. 2. Mo i a ion and A ailable Resou ces 2.1. Mo i a ion In he las ew decades, he e has been a clea end owa d Bayesian s a is ics which was p e iously almos en i ely neglec ed by p ac icing s a is icians: McG ayne (2012) ells he ascina ing his o y. Howe e , his end seems o ha e been much slowe in s a is ical eaching in spi e o igo ous allega ions ad oca ing o change (Cobb 2015;Wi me 2017). In ou cou ses, Bayes’ heo em is in oduced a a e y ea ly s age, jus h ee weeks a e s a ing he i s in oduc o y cou se o S a is ics and Da a Analysis. We conside his ea ly in o- duc ion o he u mos impo ance, e en i s a is ical echniques augh la e a e (s ill) classical in he main. I gi es s uden s a b oade pe spec i e which helps hem unde s and he equen- is in e p e a ion o p obabili y i s , hen o in e ence (Page and Sa ake 2017, p. 263, u he elabo a es his poin ). On he o he hand, a leas o some s uden s, i is h illing o ind igh a he e ybeginningo hesubjec as illcon o e sial ques ionabou wha i is he“ igh ”way olea n omda a. 2.2. A ailable Resou ces In an a emp o in oduce some p ac ice in Bayesian s a is ics beyond simple class oom examples, we sea ched he In e ne o eaching aids. We did no ind any hing co e ing, in a sim- ple way, he p ecise opic we wan ed ou s uden s o p ac ice © 2019 The Au ho s. Published wi h License by Taylo & F ancis G oup, LLC. This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e Commons A ibu ion License (h p://c ea i ecommons.o g/licenses/by/4.0/), which pe mi s un es ic ed use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed. The mo al igh s o he named au ho (s) ha e been asse ed. JOURNAL OF STATISTICS EDUCATION 69 (Bayesian upda ing o p obabili ies), al hough he e a e abun- dan esou ces which make use o games o simula ions o some so . Mos examples we ha e ound a e ela ed o expe imen al design and some ha e a his o y ha goes back o pionee - ing pape s Mead and S e n (1973)andPike(1974); see S e n, La ham, and S e n (2009) o ins ance. Close o ou goal o in oducing s uden s o he udimen s o Bayesian hinking is E ickson (2017), which p oposes wo examples o ac i i ies. I emphasizes g aphical aids in he o m o mosaic plo s o help build in ui ion. Downey (2012)isawon- de ul book wi h lo s o wo ked examples ha guide he eade who is easonably p o icien in Py hon. I could be adap ed o usewi hR,whichisp e alen a ou ins i u ion,bu s illwould equi e mo e skills in p og amming han we can assume o mos o ou s uden s. Wi me (2017), in u n, p esen s an expe ience o in oducing Bayesian ideas h ough Ma ko chain Mon e Ca lo (MCMC) a he unde g adua e le el. Ou goalislessambi ious:wewan edasimple eachingaid o help unde s and he e y ounda ions o Bayesian hinking, namely how a p io i p obabili ies a e upda ed o a pos e io i p obabili ies in he ligh e idence, and how hese a pos e io i become es ablished a p io i knowledge o be used a a nex s ep. We did no ind a ool o do exac ly ha , which led us o de elop he simula o desc ibed in Sec ion 4 a ound he s o y in Sec ion 3 nex . 3. The Loss o he USS SCORPION 3.1. The His o y The USS Sco pion was a nuclea subma ine in he U.S. Na y. I disappea ed on May 22, 1968, close o he Azo es a chipelago, when e u ning o i s base in No olk om a mission. The easons a e as ye unce ain. The e was specula ion abou an explosion, acciden al ac i a ion o a oul o pedo, a So ie a ack, and a ious mal unc ions. A e se e al days elapsed wi hou con ac , he sea ch o he subma ine s a ed. The icini y o he las known posi ion o he ship was di ided in 400 sec o s. An a p io i p obabili y o con aining he emaining o he ship, using a ailable in o - ma ion and expe ’s assessmen s, was asc ibed o each such sec o . The sea ch was conduc ed using me hods o Bayesian sea ch heo y, on he ad ice o s a is ical expe s. These me hods had been used wi h success in he sea ch o a hyd ogen bomb acciden ally d opped by a B52 bombe o he sou he n coas o Spain, nea Paloma es, in 1966. The sea ch o he USS Sco pion also ended wi h success in Oc obe 1968, when pa s o he subma ine we e ound in he sea bed unde 3000 m o wa e , some 400 nau ical miles sou hwes o he Azo es. Bo hC essieandWikle(2011) and McG ayne (2012)con ain s a is ically o ien ed accoun s o he USS Sco pion sea ch. Wikipediaalsohasagoodaccoun andanumbe o poin e s o o he sou ces o in o ma ion. An in e es ing ollow-up, mo e echnical, is Da ey e al. (2016), p esen ing Bayesian sea ch echniques in he case o he los Malaysian Ai Lines ligh MH370, in 2014. We closely ollow he i s e e ence in he sho summa y o ele an heo y nex . 3.2. Bayesian App oxima ion Le Yibe a andom a iable wi h wo s a es: Yi=0means“The subma ine is no p esen in sec o i,” w hil e Yi=1means he opposi e. Likewise, le Xibe a andom a iable coding he ou come o sea ching a sec o i.Le Xi=0i hesubma ineisno oundin said sec o and Xi=1i i is. Clea ly Xiis dependen on Yi: • I he subma ine is no p esen in he i h sec o , i canno possibly be loca ed in ha sec o , so: P(Xi=1|Yi=0)=0. • On he o he hand, i i is indeed p esen in he i h sec o , he p obabili y ha i willbe oundisp: P(Xi=1|Yi=1)=p. A sea ch is no gua an eed o be success ul, so p<1: he e is a nonze o p obabili y ha we ail o de ec he subma ine in asea cho hei h sec o , e en hough i is eally he e. Assume ha he a p io i p obabili y o he subma ine being in he i h sec o is πi.I wesea ch ha sec o onoa ail, he p obabili y a pos e io i ha he ship is he e is, using (1) P(Yi=1|Xi=0)=P(Xi=0|Yi=1)P(Yi=1) P(Xi=0) =P(Xi=0|Yi=1)P(Yi=1) P(Xi=0|Yi=0)P(Yi=0) +P(Xi=0|Yi=1)P(Yi=1) =(1−p)πi 1×(1−πi)+(1−p)πi =πi (1−p) 1−pπi (2) We call he a en ion o s uden s on he ac ha , as in ui ion sugges s, ailu e o loca e he subma ine in a sea ch o sec o i does no p eclude he possibili y ha i is he e, bu makes he pos e io p obabili y smalle han he p io p obabili y: he a io (1−p) 1−pπi which mul iplies πiin (2) is less han one, he mo e so he g ea e pis. As a consequence o a ui less sea ch o sec o i, hep oba- bili ies o all o he s sec o s a e also modi ied. Fo j= i,weha e P(Yj=1|Xi=0)=P(Xi=0|Yj=1)P(Yj=1) P(Xi=0) =1×πj 1−pπi =πj 1 1−pπi .(3) Again as in ui ion sugges s, he ac ha he subma ine is no loca ed by a sea ch o sec o ienhances ou belie ha i is in any o he sec o s j= i, o he ac o ha mul iplies he p io p obabili y πjin (3)isg ea e hanone. 70 M.J. BÁRCENA ET AL. Figu e 1. Ini ial sc een o simula o . 4. The Simula o : Use and Implemen a ion 4.1. Design Goals We did no seek a ool o in oduce Bayes’ heo em, bu a he a ool o p ac ice heo y p e iously lea ned al hough pe haps no ully in e nalized. Consequen ly, Bayes’ heo em and i s applica ion o he p oblem a hand is augh in class, oughly along he lines o Sec ion 3.2, and a handou desc ibing he p ac ice and how o use he simula o is gi en in ad ance o s uden s. We had o se e a la ge numbe o s uden s, no all o hem in one loca ion. This, in p ac ice, educed he choices o a web- basedsimula o , equi ingno hingelseon heclien sideo he han a Ja asc ip -enabled web b owse . S uden s can use he simula o om any compu e oom on campus o om hei own compu e s a home. The implemen a ion is ligh , uns o a single se e and can be easily unde s ood and changed. Simu- la ion pa ame e s like he numbe o ini ial egions, pa ame e p, gene a ion o ini ial a p io i p obabili ies, e c. equi e ai ly simple changes o he sou ce. 4.2. Se up Ou simula o aces he s uden wi h he same decisions ha he sea ch eam o he USS Sco pion had o make, bu in a a he simpli ied se ing: ins ead o 400 sec o s only nine a e p esen ed in a map (see Figu e 1). P io o use o he simula o , s uden s a e gi en a w i e-up con aining essen ially he in o ma ion gi en in Sec ion 3 o he p esen a icle. To s a using he simula o , he s uden only has o p ess he bu on S a in he bo om le co ne . The simula o hen gene a es a andom ins ance o he game assigning a p io i p ob- abili ies o all nine sea chable sec o s and places a sea ch essel nex o he sou hwes co ne o he sea chable a ea. Clicking on any one sec o gi es in o ma ion on i s a p io i p obabili y a any ime; his p obabili y is also encoded as colo sa u a ion in apale eo g eens 1(Figu e 2). The i s sea ch is i ial: jus go o he sec o wi h highe a p io i p obabili y. In o de o do ha , he s uden jus has o d ag wi h he mouse he sea ch essel o he sec o chosen and ei he click on i o on he bu on Sea ch in he sou hwes co ne ; he la e al e na i e has been ound necessa y o playe s using small sc eens such as able s o cellula phones. A e each s uden ’s choice, he simula o upda es he coun- e s a he op o he page (las sec o sea ched, numbe o sea ches, “co ec ” and inco ec sea ches, poin s ea ned). A sea ch is “co ec ” i he sec o wi h he la ges p obabili y is chosen. Poin s ea ned a e en imes he a io o co ec o o al sea ches, so a sco e o 10 means ha he s uden chose e e y ime o sea ch he mos likely ec angle. The simula o will ell he s uden whe he he subma ine is ound,inwhichcase hegameends,o elseupda e hea p io i p obabili ies o each o he sec o s, in ligh o he las ui less sea ch. These upda ed p obabili ies, hough, a e nei he displayed no colo -encoded in he sc een, which always shows p obabili ies p io o he las sea ch: he πio Equa ions (2) 1A di e en pale e, less isually pleasing, is a ailable o colo blind s uden s, should he need a ise. JOURNAL OF STATISTICS EDUCATION 71 Figu e 2. Simula o sc een a e one co ec bu unsuccess ul sea ch. and (3).I is hes uden ’s ask odo heupda ingusing hese o mulas. S uden sa e old he alueo p— he p obabili y ha a sea ch o he igh ec angle will un eil he subma ine—which in he expe imen desc ibed la e was se a p=0.60. They can be old o allowed o disco e by hemsel es ha , as hey p oceed wi h he game, hey only ha e o upda e wo p obabili ies: ha o he ecen ly sea ched sec o , which migh emain he mos p omising in spi e o a ailed a emp , and ha o he p e iously mos likely sec o —since he Bayesian upda ing mul iplies he a p io i p obabili y o all sec o s di e en om he one jus sea ched by he same ac o and so p ese es hei o de ; see Equa ion (3). Thegameendswhen hesubma ineis inallyloca ed,and he poin s ea ned a e sa ed. I is up o he ins uc o s o le he s uden s play once o (ou choice) as o en as hey wish, keeping only hei las sco e. 4.3. Implemen a ion Aspec s Thesimula o iscodedinJa asc ip using helib a ylea le 2 o he map p esen a ion. We chea ed a bi in he implemen a ion. In ac , he subma ine is no andomly alloca ed o any sec o . Wha we andomize a he s a o he game is he minimum numbe o ials he s uden will ha e o go h ough: his is oa oide en slike inding hesubma inea he i s ial, ha would gi e he maximum sco e wi hou a eco d o consis en ly good sea ch choices. No ma e wha , he s uden willha e omakeanumbe o sea ches(a leas sixin he cu en implemen a ion,bu hisiseasy ochange),sowe 2See h p://lea le js.com. canbeassu ed ha asco eo 10meansconsis en good useo Bayesupda ingandno jus aluckysinglechoice ha inds he subma ine on he i s o second andom a emp . Ano he aspec ha eache s migh wan o ine une is he pa ame e p— he p obabili y o success when sea ching he co ec sec o . I ollows om he p e ious pa ag aph ha i has no in luence in he leng h o he game, bu i does ha e a la ge in luence in he upda ing o he a p io i p obabili ies. I se oohigh,anunsuccess ulsea chwilld ama icallylowe he pos e io p obabili y o he sea ched sec o : he subsequen choice will hen almos in a iably be he sec o ha had he la ges a p io i p obabili y be o e he las sea ch. S uden s migh soon no ice he pa e n and play wi h no eso a all o o - mulas in Equa ions (2) and (3)—which de ea s he pu pose o he simula o . I is he e o e ad isable o se pa a mod- e a e alue. We ha e ied in he icini y o 0.6 wi h good esul s. 5. Deploymen and Resul s We ha e es ed he simula o wi h unde g adua e s uden s ak- ing a i s qua e in s a is ics, co e ing p obabili y, andom a iables, densi y and dis ibu ion unc ions, momen s, cha - ac e is ic unc ion, e c. which lay he ounda ion o a second qua e on In e en ial S a is ics. I is in his cou se ha Bayes’ heo em is i s discussed. Ou s uden s a e all in he sophomo e yea o Business, Economics o Business and Law deg ees. The las g oup (double majo in Business and Law) end o be composed o be e pe - o ming s uden s, as he en ance equi emen s a e s ic e . The ma hema ical backg ound o all g oups is simila : wo qua e s o Calculus and Algeb a. 72 M.J. BÁRCENA ET AL. 5.1. Tes We de ised a sho exam wi h ques ions in which s uden s we e equi ed o ecognize whe he he use o he wo d “p obabili y” had a Bayesian o equen is la o . Fo ins ance, When we say ha he p obabili y ha he Malaysian Ai lines plane los in 2014 ( ligh MH370) is in a gi en a ea in he Paci ic ocean wi h p obabili y p=0.01, a e we using he wo d ‘p obabili y’ in a Bayesian o in a equen is sense?. We also included he simples p oblem we could hink o which equi ed sequen ial applica ion o Bayes’ heo em—wha he simula o is designed o p o ide aining o . I ead, (a) In a oggy day, a hike su e s an acciden when e u ning om a moun ain. He belie es ha wi h p obabili y 0.60 he is in he No h slope, bu wi h p obabili y 0.40 he migh ha e ended in he Sou h slope. Be o e using his cellula phone o ask o help,hewouldlike obe e ixhisloca ion.He emembe s ha in he No h slope beech ees a e p e alen (70% o he o al) wi h he es being oak ees (15%) and yew ees (15%), while in he Sou h slope he p opo ions a e 30% beech ees, 60% oak ees and 10% yew ees. He app oaches he nea es ee and ealizes ha i is an oak ee. Using his in o ma ion, he p obabili y ha he is in he No h slope is, app oxima ely... This is a simple example in which s uden s can use he a ailable in o ma ion o e ise hei p io p obabili y and ob ain apos e io p obabili y.Thiswas ollowedby: (b)A e ha ing ound he eemen ionedin he p e ious ques ion, he walks u he and inds ano he ee, which he ecognizes as being a yew ee.He hen alls o heg ound,exhaus ed.Whe e will he ell he escue b igade o sea ch o him, in he No h o Sou h slope? The in en is o go one s ep u he . He e, we wan he s uden o ecognize ha he pos e io om (a) can be used as a p io in (b)— he essence o Bayesian lea ning ha ou simula o a ge s. 5.2. Assessmen Me hod Ou i s hough was o de ise a s anda d expe imen , ei he andomizing he “ ea men ” (= use o simula o ) o aking pai s o s uden s ma ched acco ding o hei abili y (measu ed by hei g ade poin a e age, o ins ance) and assigning wi hin each pai one o he g oup o simula o use s and he o he o he g oup o nonuse s. We would hen compa e pe o mance o bo h g oups when aking he es desc ibed in he p e ious sec ion. Howe e ,since he es was ogi ec edi owa d hecou se g ade his would c ea e an un ai ness owa d he s uden s ha we eno assigned o hesimula o g oup.This,inou con ex , canno be con empla ed. On he o he hand, he e is no way in which we could ensu e ha he un ea ed o con ol g oup was eallyun ea ed:ou o cu iosi yo o he wiseanys u- den could use he simula o and hus con amina e he con ol g oup. We he e o e decided ha we would adminis e he es wice, be o e and a e gi ing a chance o use he simula o . We we e ully awa e ha he second ime he es is aken a be e pe o mance is o be expec ed, e en o nonuse s o he simula o . Bu we coun ed on measu es o use o he simula o ( ime spen , sco e ob ained when using i ) as well as on ha ing some acciden al “con ols”: s uden s who o a ious easons wouldno use hesimula o .Tha wouldenableus odisen angle he e ec o using he simula o and he e ec o me e epe i ion o he es . As i happens, he con ol g oup was la ge han we expec ed. A e co e ing Bayes’ heo em in class, we equi ed s u- den s o ake he es desc ibed abo e. They we e hen encou - aged o eely use he simula o o a pe iod o i e days: his eeusepolicyand he ac ha only helas sco ewouldbe eco ded was made clea o hem be o ehand, as well as he ac ha use o he simula o migh be o some help in a second es . Then, he same es was gi en again, complemen ed wi h a ew ques ions ega ding whe he hey had ound he use o he simula o obeeasy, ewa ding,howmuch ime heyhadspen on i , e c. The ac ha we ha e he same s uden s ake he i s and second es s enables us o accoun o di e ences in s uden abili y.Thep oblem,o cou se,is ha he ecanbeasel - selec ion e ec : mo e mo i a ed s uden s migh choose o use he simula o in g ea e p opo ion han he o he s. Below we epo on some e idence o his e ec in he esul s: i may ha e been coun e ed by he ac ha be e pe o ming s u- den s, a e ob aining a good g ade in he i s es , did no see oom o imp o emen and he e o e neglec ed he use o he simula o . 5.3. Resul s and Modeling Bo h es s we e adminis e ed in he Fall Te m o 2017 in he o m o mul iple choice ques ionnai es, o a oid any subjec i e biases om he g ade s. As Figu es 3 and 4imply, heywe eg aded on a 0–10 scale ( he colo -coded simula o sco es we e also in a 0-10 scale). A o al o 241 s uden s pa icipa ed, bu 52 we e absen in ei he he i s o second es (p e- and pos - es in he sequel). The e o e, we obse ed 189 s uden s who ook bo h he i s and second es plus 52 who only ook one o he es s. Figu e 3 shows he b eakdown o g ades o he common se o ques ions in he p e- and pos - es . (Subjec i e ques ions in he pos - es as o whe he he simula o had been use ul, un, e c. we e no g aded.) Clea ly, he e is an imp o emen , pa icula ly appa en in he la ge median g ade o he subse o s uden s ha used he simula o . Also, g ades below 2.5 poin s we e en i ely absen among he simula o use s. A di e en , mo e insigh ul iew, is p o ided by Figu e 4.The pe o mance o each s uden is shown by a poin , whose coo - dina es a e he g ades in he i s and second es . (Poin s ha e JOURNAL OF STATISTICS EDUCATION 73 Used simula o Did no use simula o Fi s es Second es Fi s es Second es 0.0 2.5 5.0 7.5 10.0 Tes G ade Tes Fi s es Second es Figu e 3. B eakdown on g ades in p e- and pos - es acco ding o use o simula o . been ji e ed, o educe o e plo ing.) The wo panels show he pe o mance o s uden s who did and did no use he simula o . Fu he , o hose who did use he simula o , he sco e ob ained is colo -coded in he le panel. The ed lines ma k show equal g ades in he i s and second es s: poin s abo e he ed line indica e an imp o emen . We expec ed all sco es using he simula o o be equal o close o he maximum 10, gi en ou policy o “use o as long as you wish,keeponlyyou las g ade.”Asama e o ac , hishasno been he case: some s uden s abandoned he ask ea lie wi h low o e en ze o sco es. Howe e , al hough imp o emen in he second es is no clea ly ela ed o he simula o sco e, i seems much mo e consis en among s uden s who used he simula o , wha e e sco e hey ob ained. When we b eak down he esul s by g oup, Figu e 5,we ind ha s uden s in he ADEDE g oup (double deg ee in Business and Law) did much be e han s uden s in ei he o he single deg ees o Business (ADE) o Economics (ECO). I is appa en also ha ADEDE s uden s no only ob ained highe sco es when using he simula o , bu also used i in a la ge p opo ion (mo e colo ed poin s in Figu e 5), which poin s o a possible p oblem o sample sel -selec ion add essed ea lie . To ob ain a mo e o mal assessmen o he e ec o he sim- ula o , we ha e i ed se e al linea models. A na u al app oach would be o conside o each s uden he di e ence in g ades ob ained in he i s and second es as a esponse a iable, and ela e ha di e ence o he use (o no ) o he simula o , and possible o he ac o s. In o he wo ds, o use a pai ed compa isons app oach. Howe e , a o al o 52 obse a ions a e no pai ed: some s uden s ook he i s es and no he second o ice e sa. In o de o use also hese obse a ions, we ha e i ed se e al linea mixed models, Demidenko (2004), in which he esponse a iable is he g ade ob ained in ei he o he es s. Di e ences in he indi idual pe o mance o s uden s a e accoun ed o by a andom e ec , as indi iduals a e o no in e es in hemsel es. The simula o e ec , “ epea ” e ec , and g oup e ec a e in o- duced as ixed e ec s. All models ha e been i ed in R, R Co e Team (2017), using package lme4 (see Ba es e al. 2015). Model 1 is, in he usual no a ion, G ade =1+(1|ID) +Rep; a iable Rep is a dicho omous a iable aking alue 0 o 230 obse a ions co esponding o he i s es and 1 o 200 obse - a ions co esponding o he second. (1|ID) is a andom e m, di e en o each s uden and e lec ing his o he indi idual abili y. I is o no in e es in i sel —we know s uden s o be 74 M.J. BÁRCENA ET AL.                                                                                                                                                                                Used simula o Did no use simula o 0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0 G ade i s es G ade second es 0.0 2.5 5.0 7.5 10.0 Sco e using simula o Figu e 4. Indi idual pe o mances in p e- and pos - es acco ding o use o simula o and sco e ob ained. Each poin is colo -coded e lec ing he sco e ob ained when using he simula o ( igh legend), wi h s uden s who did no use i shown in g ay. di e en —bu use ul o alloca e he pa o a iance ha is explained by s uden ’s he e ogenei y. The coe icien o Rep ( he “ epea ” e ec ) is posi i e and highly signi ican ( e e o column Model 1 in Table 1). I s alue in his model e lec s he (possible) e ec o he use o he simula o o some s uden s, as well as he ac ha ( o all) be e pe o mance should be expec ed he second ime he s uden s ook he es , i espec i e o whe he o no hey used he simula o . To disen angle he e ec a ibu able o he simula o om ha o me e epe i ion o he exam, we can i he model G ade =1+(1|ID) +Rep +Used.sim, whe e Used.sim is a dicho omous a iable aking alue 1 o obse a ions co esponding o he second es o s uden s who did use he simula o . The es ima ion esul s can be seen in column “Model 2” o Table 1. A es o Model 2 e sus Model 1, see Table 2, shows a dec ease o 5.922 in de iance, highly sig- ni ican (p=0.015). (The e is a e y small misma ch wi h he alues o he log-likelihood and model c i e ia AIC and BIC epo ed in Table 1, possibly consequence o di e en me hods o compu ing he log-likelihood and deg ees o eedom.) Howe e , when we i he model G ade =1+(1|ID) +Rep +Used.sim +G oup ( esul s in column Model 3 o Table 1), he coe icien o Used.sim becomes non-signi ican : i seems ha i is pa ly con ounded wi h he G oup e ec . This was o be expec ed since, as i is appa en om Figu e 5, s uden s in g oup ADEDE, who a e clea ly be e pe o me s, ha e also used he simula o in g ea e p opo ion. The G oup e ec is o pa amoun impo ance and accoun s o a dec ease o 19.288 in de iance, educing he a iance accoun ed by he andom e ec ID om 2.52 o 2.13, a educ ion o abou 15%: pa o he he e ogenei y among s uden s is in ac a di e ence be ween g oups. 5.4. S uden ’s Feedback The second es included a ew ques ions ega ding he expe i- ence wi h he simula o . S uden s we e old ha hese ques ions hadno e lec wha soe e in hei g ades,bu esponsewas none heless o al among s uden s ha ing used he simula o . Thei pe cep ion o whe he playing wi h he simula o was o any help did no clea ly co ela e wi h hei pe o mance—see JOURNAL OF STATISTICS EDUCATION 75                                                                                                                                                                                    ADE ADEDE ECO 0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0 G ade i s es G ade second es 0.0 2.5 5.0 7.5 10.0 Sco e using simula o Figu e 5. B eakdown o g ades in p e- and pos - es acco ding o use o simula o pe g oup. Each poin is colo -coded e lec ing he sco e ob ained when using he simula o ( igh legend), wi h s uden s who did no use i shown in g ay. Table 1. E ec on G ade o use o simula o . Model 1 Model 2 Model 3 (In e cep ) 5.60∗∗∗ 5.61∗∗∗ 5.25∗∗∗ (0.18)(0.18)(0.22) RepYES 1.24∗∗∗ 0.99∗∗∗ 1.04∗∗∗ (0.22)(0.24)(0.24) Used.simYES 1.08∗0.75 (0.44)(0.45) G oupADEDE 2.12∗∗∗ (0.48) G oupECO 0.40 (0.32) AIC 2083.28 2079.17 2064.15 BIC 2099.53 2099.49 2092.60 Log-likelihood −1037.64 −1034.58 −1025.08 Num. obs. 430 430 430 Num. g oups: ID 241 241 241 Va : ID (In e cep ) 2.66 2.52 2.13 Va : Residual 5.04 5.06 5.07 NOTE: ∗∗∗p<0.001, ∗∗p<0.01, and ∗p<0.05. Figu e 6. Whe he hey said i had been o no, li le, mode a e o signi ican help, hei pe o mance in he second es appea s o be be e —excep pe haps, qui e pa adoxically, o hose who we e mo e con inced o he use ulness o he simula o . This may be due o he ac ha hey we e good pe o me s in he i s es and he e o e wi h li le oom o imp o emen . On he o he hand, when asked abou how much ime hey spen playing wi h he simula o , he e seems o be a clea pa e n o g ea e imp o emen o hose who spen mo e han 30 minu es on he simula o : see Figu e 7. The median g ade wen om 6.4 o he maximum o 10; o all he o he ca ego ies, an upwa d shi in g ades is isible, bu —excep o he “None” ca ego y— he medians emain he same. I is also appa en ha hose ha used he simula o had a highe median g ade in he i s es . 6. Discussion The e idence we can p esen on he impac o he simula o is no conclusi e: he pa ial con ounding wi h g oup e ec p e- en sus ommakings ongclaimsabou heuse ulnesso he simula o . Howe e , wha we can lea n om he obse a ional