Full Te ms & Condi ions o access and use can be ound a
h ps://www. and online.com/ac ion/jou nalIn o ma ion?jou nalCode=ujse20
Jou nal o S a is ics Educa ion
ISSN: (P in ) 1069-1898 (Online) Jou nal homepage: h ps://www. and online.com/loi/ujse20
A Web Simula o o Assis in he Teaching o Bayes’
Theo em
M. J. Bá cena, M. A. Ga ín, A. Ma ín, F. Tusell & A. Unzue a
To ci e his a icle: M. J. Bá cena, M. A. Ga ín, A. Ma ín, F. Tusell & A. Unzue a (2019) A Web
Simula o o Assis in he Teaching o Bayes’ Theo em, Jou nal o S a is ics Educa ion, 27:2, 68-78,
DOI: 10.1080/10691898.2019.1608875
To link o his a icle: h ps://doi.o g/10.1080/10691898.2019.1608875
© 2019 The Au ho (s). Published wi h
license by Taylo & F ancis G oup, LLC
Published online: 12 Jun 2019.
Submi you a icle o his jou nal
A icle iews: 2034
View ela ed a icles
View C ossma k da a
Ci ing a icles: 1 View ci ing a icles
JOURNAL OF STATISTICS EDUCATION
2019, VOL. 27, NO. 2, 68–78
h ps://doi.o g/10.1080/10691898.2019.1608875
A Web Simula o o Assis in he Teaching o Bayes’Theo em
M. J. Bá cenaa,M.A.Ga ín
a,A.Ma ín
a,F.Tusell
a, and A. Unzue ab
aFacul ad de Economía y Emp esa, UPV/EHU, Bilbao, Spain; bEscuela de Ingenie ía, UPV/EHU, Bilbao, Spain
ABSTRACT
Teaching some concep s in s a is ics g ea ly bene i s om indi idual p ac ice wi h immedia e eedback.
In o de o p o ide such p ac ice o a la ge numbe o s uden s we ha e w i en a simula o based on an
his o ical e en : he loss in May 22, 1968, and subsequen sea ch o he nuclea subma ine USS Sco pion.
S uden s wo k on a simpli ied e sion o he sea ch and can see p obabili ies change in esponse o new
e idence. The simula o is designed o assis in he eaching o Bayesian concep s, in pa icula Bayesian
upda ing. I has been deployed in ou cou ses and ou expe ience and esul s a e desc ibed, as well as he
eac ions o ou s uden s o i s use. The simula o is open sou ce, eely a ailable and easy o implemen
and un, as i only equi es a machine o se e web pages. We explain in de ail ou expe ience wi h i s
deploymen and use.
KEYWORDS
Bayesian s a is ics; S a is ics;
Simula o
1. In oduc ion
Bayes’ heo em, in i s simples o m exp essible as
P(A|B)=P(B|A)P(A)
P(B),(1)
p o ides a way o “in e ” p obabili ies: om he condi ional
p obabili y o Bgi en Aand he espec i e ma ginals, he p ob-
abili y o Agi en Bcan be compu ed. Al hough i is simple and
canbep esen edandp o edinama e o minu es, hisisa
concep ha equi es ime o s uden s o g asp.
We ha e ound use ul o e he yea s o p esen some exam-
ples ha help enhance s uden ’s comp ehension o he p ac ical
implica ions o (1). Medical diagnosis is one: he p obabili y
P(A|B)o ha ing a sickness Agi en he p esence o symp om
Bcanbeob ainedin e mso hep obabili yo hesymp om
gi en hesicknessand he espec i ema ginalp obabili ieso
sickness and symp om. This nicely illus a es he way o e ise a
p io P(A)in he ligh o newly a ailable in o ma ion B.
Acqui ing amilia i y wi h he concep s in ol ed equi es,
much as he acquisi ion o a new language, epea ed in e ac ion
wi h (1), beyond he ew examples ha can be p esen ed in class.
Such amilia i y can be os e ed by assigning homewo k o be
done by s uden s and la e g aded, bu his imposes a consid-
e able bu den on he eache s and p o ides, a bes , delayed
eedback o he s uden . We hough ha a simple simula o ,
p esen ing each s uden a unique ins ance o a p oblem, wi h
immedia e eedback and au oma ic g ading, would be much
p e e able. This a icle epo s on ou wo k in his di ec ion.
Sec ion 2 e iews some wo k which add esses simila goals
as ou s o which we ha e o he wise used o inspi a ion.
Sec ion 3 b ie ly desc ibes he s o y we ha e used, in a
simpli ied ec ea ion, omo i a eagameinwhichs uden s
CONTACT F. Tusell e nando[email p o ec ed] Facul ad de Economía y Emp esa, UPV/EHU, A da. Lehendaka i Agui e, 83, Bilbao, Spain.
Supplemen a y ma e ials o his a icle a e a ailable online. Please go o www. and online.com/ujse.
a e equi ed oloca eamissingsubma ine.Sec ion 4 desc ibes
he implemen a ion. We es ed he use o ou simula o on ou
in oduc o y cou se on s a is ics o sophomo es, when Bayes’
heo em is i s p esen ed; Sec ion 5 gi es some de ails abou
he esul s ob ained. Sec ion 6 closes wi h some commen s.
2. Mo i a ion and A ailable Resou ces
2.1. Mo i a ion
In he las ew decades, he e has been a clea end owa d
Bayesian s a is ics which was p e iously almos en i ely
neglec ed by p ac icing s a is icians: McG ayne (2012) ells
he ascina ing his o y. Howe e , his end seems o ha e
been much slowe in s a is ical eaching in spi e o igo ous
allega ions ad oca ing o change (Cobb 2015;Wi me 2017).
In ou cou ses, Bayes’ heo em is in oduced a a e y ea ly
s age, jus h ee weeks a e s a ing he i s in oduc o y cou se
o S a is ics and Da a Analysis. We conside his ea ly in o-
duc ion o he u mos impo ance, e en i s a is ical echniques
augh la e a e (s ill) classical in he main. I gi es s uden s a
b oade pe spec i e which helps hem unde s and he equen-
is in e p e a ion o p obabili y i s , hen o in e ence (Page and
Sa ake 2017, p. 263, u he elabo a es his poin ).
On he o he hand, a leas o some s uden s, i is h illing o
ind igh a he e ybeginningo hesubjec as illcon o e sial
ques ionabou wha i is he“ igh ”way olea n omda a.
2.2. A ailable Resou ces
In an a emp o in oduce some p ac ice in Bayesian s a is ics
beyond simple class oom examples, we sea ched he In e ne
o eaching aids. We did no ind any hing co e ing, in a sim-
ple way, he p ecise opic we wan ed ou s uden s o p ac ice
© 2019 The Au ho s. Published wi h License by Taylo & F ancis G oup, LLC.
This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e Commons A ibu ion License (h p://c ea i ecommons.o g/licenses/by/4.0/), which pe mi s un es ic ed use, dis ibu ion,
and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed. The mo al igh s o he named au ho (s) ha e been asse ed.
JOURNAL OF STATISTICS EDUCATION 69
(Bayesian upda ing o p obabili ies), al hough he e a e abun-
dan esou ces which make use o games o simula ions o some
so . Mos examples we ha e ound a e ela ed o expe imen al
design and some ha e a his o y ha goes back o pionee -
ing pape s Mead and S e n (1973)andPike(1974); see S e n,
La ham, and S e n (2009) o ins ance.
Close o ou goal o in oducing s uden s o he udimen s
o Bayesian hinking is E ickson (2017), which p oposes wo
examples o ac i i ies. I emphasizes g aphical aids in he o m
o mosaic plo s o help build in ui ion. Downey (2012)isawon-
de ul book wi h lo s o wo ked examples ha guide he eade
who is easonably p o icien in Py hon. I could be adap ed o
usewi hR,whichisp e alen a ou ins i u ion,bu s illwould
equi e mo e skills in p og amming han we can assume o mos
o ou s uden s. Wi me (2017), in u n, p esen s an expe ience
o in oducing Bayesian ideas h ough Ma ko chain Mon e
Ca lo (MCMC) a he unde g adua e le el.
Ou goalislessambi ious:wewan edasimple eachingaid
o help unde s and he e y ounda ions o Bayesian hinking,
namely how a p io i p obabili ies a e upda ed o a pos e io i
p obabili ies in he ligh e idence, and how hese a pos e io i
become es ablished a p io i knowledge o be used a a nex
s ep. We did no ind a ool o do exac ly ha , which led us o
de elop he simula o desc ibed in Sec ion 4 a ound he s o y in
Sec ion 3 nex .
3. The Loss o he USS SCORPION
3.1. The His o y
The USS Sco pion was a nuclea subma ine in he U.S. Na y. I
disappea ed on May 22, 1968, close o he Azo es a chipelago,
when e u ning o i s base in No olk om a mission. The
easons a e as ye unce ain. The e was specula ion abou an
explosion, acciden al ac i a ion o a oul o pedo, a So ie a ack,
and a ious mal unc ions.
A e se e al days elapsed wi hou con ac , he sea ch o
he subma ine s a ed. The icini y o he las known posi ion
o he ship was di ided in 400 sec o s. An a p io i p obabili y
o con aining he emaining o he ship, using a ailable in o -
ma ion and expe ’s assessmen s, was asc ibed o each such
sec o .
The sea ch was conduc ed using me hods o Bayesian sea ch
heo y, on he ad ice o s a is ical expe s. These me hods had
been used wi h success in he sea ch o a hyd ogen bomb
acciden ally d opped by a B52 bombe o he sou he n coas
o Spain, nea Paloma es, in 1966. The sea ch o he USS
Sco pion also ended wi h success in Oc obe 1968, when
pa s o he subma ine we e ound in he sea bed unde
3000 m o wa e , some 400 nau ical miles sou hwes o he
Azo es.
Bo hC essieandWikle(2011) and McG ayne (2012)con ain
s a is ically o ien ed accoun s o he USS Sco pion sea ch.
Wikipediaalsohasagoodaccoun andanumbe o poin e s
o o he sou ces o in o ma ion. An in e es ing ollow-up, mo e
echnical, is Da ey e al. (2016), p esen ing Bayesian sea ch
echniques in he case o he los Malaysian Ai Lines ligh
MH370, in 2014. We closely ollow he i s e e ence in he sho
summa y o ele an heo y nex .
3.2. Bayesian App oxima ion
Le Yibe a andom a iable wi h wo s a es: Yi=0means“The
subma ine is no p esen in sec o i,” w hil e Yi=1means he
opposi e.
Likewise, le Xibe a andom a iable coding he ou come o
sea ching a sec o i.Le Xi=0i hesubma ineisno oundin
said sec o and Xi=1i i is.
Clea ly Xiis dependen on Yi:
• I he subma ine is no p esen in he i h sec o , i canno
possibly be loca ed in ha sec o , so:
P(Xi=1|Yi=0)=0.
• On he o he hand, i i is indeed p esen in he i h sec o , he
p obabili y ha i willbe oundisp:
P(Xi=1|Yi=1)=p.
A sea ch is no gua an eed o be success ul, so p<1: he e is
a nonze o p obabili y ha we ail o de ec he subma ine in
asea cho hei h sec o , e en hough i is eally he e.
Assume ha he a p io i p obabili y o he subma ine being
in he i h sec o is πi.I wesea ch ha sec o onoa ail, he
p obabili y a pos e io i ha he ship is he e is, using (1)
P(Yi=1|Xi=0)=P(Xi=0|Yi=1)P(Yi=1)
P(Xi=0)
=P(Xi=0|Yi=1)P(Yi=1)
P(Xi=0|Yi=0)P(Yi=0)
+P(Xi=0|Yi=1)P(Yi=1)
=(1−p)πi
1×(1−πi)+(1−p)πi
=πi
(1−p)
1−pπi
(2)
We call he a en ion o s uden s on he ac ha , as in ui ion
sugges s, ailu e o loca e he subma ine in a sea ch o sec o i
does no p eclude he possibili y ha i is he e, bu makes he
pos e io p obabili y smalle han he p io p obabili y: he a io
(1−p)
1−pπi
which mul iplies πiin (2) is less han one, he mo e so he g ea e
pis.
As a consequence o a ui less sea ch o sec o i, hep oba-
bili ies o all o he s sec o s a e also modi ied. Fo j= i,weha e
P(Yj=1|Xi=0)=P(Xi=0|Yj=1)P(Yj=1)
P(Xi=0)
=1×πj
1−pπi
=πj
1
1−pπi
.(3)
Again as in ui ion sugges s, he ac ha he subma ine is no
loca ed by a sea ch o sec o ienhances ou belie ha i is in
any o he sec o s j= i, o he ac o ha mul iplies he p io
p obabili y πjin (3)isg ea e hanone.
70 M.J. BÁRCENA ET AL.
Figu e 1. Ini ial sc een o simula o .
4. The Simula o : Use and Implemen a ion
4.1. Design Goals
We did no seek a ool o in oduce Bayes’ heo em, bu a he
a ool o p ac ice heo y p e iously lea ned al hough pe haps
no ully in e nalized. Consequen ly, Bayes’ heo em and i s
applica ion o he p oblem a hand is augh in class, oughly
along he lines o Sec ion 3.2, and a handou desc ibing he
p ac ice and how o use he simula o is gi en in ad ance o
s uden s.
We had o se e a la ge numbe o s uden s, no all o hem
in one loca ion. This, in p ac ice, educed he choices o a web-
basedsimula o , equi ingno hingelseon heclien sideo he
han a Ja asc ip -enabled web b owse . S uden s can use he
simula o om any compu e oom on campus o om hei
own compu e s a home. The implemen a ion is ligh , uns o a
single se e and can be easily unde s ood and changed. Simu-
la ion pa ame e s like he numbe o ini ial egions, pa ame e
p, gene a ion o ini ial a p io i p obabili ies, e c. equi e ai ly
simple changes o he sou ce.
4.2. Se up
Ou simula o aces he s uden wi h he same decisions ha he
sea ch eam o he USS Sco pion had o make, bu in a a he
simpli ied se ing: ins ead o 400 sec o s only nine a e p esen ed
in a map (see Figu e 1). P io o use o he simula o , s uden s a e
gi en a w i e-up con aining essen ially he in o ma ion gi en in
Sec ion 3 o he p esen a icle.
To s a using he simula o , he s uden only has o p ess
he bu on S a in he bo om le co ne . The simula o hen
gene a es a andom ins ance o he game assigning a p io i p ob-
abili ies o all nine sea chable sec o s and places a sea ch essel
nex o he sou hwes co ne o he sea chable a ea. Clicking on
any one sec o gi es in o ma ion on i s a p io i p obabili y a
any ime; his p obabili y is also encoded as colo sa u a ion in
apale eo g eens
1(Figu e 2).
The i s sea ch is i ial: jus go o he sec o wi h highe
a p io i p obabili y. In o de o do ha , he s uden jus has o
d ag wi h he mouse he sea ch essel o he sec o chosen and
ei he click on i o on he bu on Sea ch in he sou hwes co ne ;
he la e al e na i e has been ound necessa y o playe s using
small sc eens such as able s o cellula phones.
A e each s uden ’s choice, he simula o upda es he coun-
e s a he op o he page (las sec o sea ched, numbe o
sea ches, “co ec ” and inco ec sea ches, poin s ea ned). A
sea ch is “co ec ” i he sec o wi h he la ges p obabili y is
chosen. Poin s ea ned a e en imes he a io o co ec o o al
sea ches, so a sco e o 10 means ha he s uden chose e e y
ime o sea ch he mos likely ec angle.
The simula o will ell he s uden whe he he subma ine
is ound,inwhichcase hegameends,o elseupda e hea
p io i p obabili ies o each o he sec o s, in ligh o he las
ui less sea ch. These upda ed p obabili ies, hough, a e nei he
displayed no colo -encoded in he sc een, which always shows
p obabili ies p io o he las sea ch: he πio Equa ions (2)
1A di e en pale e, less isually pleasing, is a ailable o colo blind s uden s,
should he need a ise.
JOURNAL OF STATISTICS EDUCATION 71
Figu e 2. Simula o sc een a e one co ec bu unsuccess ul sea ch.
and (3).I is hes uden ’s ask odo heupda ingusing hese
o mulas.
S uden sa e old he alueo p— he p obabili y ha a sea ch
o he igh ec angle will un eil he subma ine—which in he
expe imen desc ibed la e was se a p=0.60. They can be
old o allowed o disco e by hemsel es ha , as hey p oceed
wi h he game, hey only ha e o upda e wo p obabili ies: ha
o he ecen ly sea ched sec o , which migh emain he mos
p omising in spi e o a ailed a emp , and ha o he p e iously
mos likely sec o —since he Bayesian upda ing mul iplies he
a p io i p obabili y o all sec o s di e en om he one jus
sea ched by he same ac o and so p ese es hei o de ; see
Equa ion (3).
Thegameendswhen hesubma ineis inallyloca ed,and
he poin s ea ned a e sa ed. I is up o he ins uc o s o le he
s uden s play once o (ou choice) as o en as hey wish, keeping
only hei las sco e.
4.3. Implemen a ion Aspec s
Thesimula o iscodedinJa asc ip using helib a ylea le 2 o
he map p esen a ion.
We chea ed a bi in he implemen a ion. In ac , he
subma ine is no andomly alloca ed o any sec o . Wha
we andomize a he s a o he game is he minimum
numbe o ials he s uden will ha e o go h ough: his is
oa oide en slike inding hesubma inea he i s ial,
ha would gi e he maximum sco e wi hou a eco d o
consis en ly good sea ch choices. No ma e wha , he s uden
willha e omakeanumbe o sea ches(a leas sixin he
cu en implemen a ion,bu hisiseasy ochange),sowe
2See h p://lea le js.com.
canbeassu ed ha asco eo 10meansconsis en good
useo Bayesupda ingandno jus aluckysinglechoice
ha inds he subma ine on he i s o second andom
a emp .
Ano he aspec ha eache s migh wan o ine une is he
pa ame e p— he p obabili y o success when sea ching he
co ec sec o . I ollows om he p e ious pa ag aph ha i
has no in luence in he leng h o he game, bu i does ha e
a la ge in luence in he upda ing o he a p io i p obabili ies.
I se oohigh,anunsuccess ulsea chwilld ama icallylowe
he pos e io p obabili y o he sea ched sec o : he subsequen
choice will hen almos in a iably be he sec o ha had he
la ges a p io i p obabili y be o e he las sea ch. S uden s migh
soon no ice he pa e n and play wi h no eso a all o o -
mulas in Equa ions (2) and (3)—which de ea s he pu pose
o he simula o . I is he e o e ad isable o se pa a mod-
e a e alue. We ha e ied in he icini y o 0.6 wi h good
esul s.
5. Deploymen and Resul s
We ha e es ed he simula o wi h unde g adua e s uden s ak-
ing a i s qua e in s a is ics, co e ing p obabili y, andom
a iables, densi y and dis ibu ion unc ions, momen s, cha -
ac e is ic unc ion, e c. which lay he ounda ion o a second
qua e on In e en ial S a is ics. I is in his cou se ha Bayes’
heo em is i s discussed.
Ou s uden s a e all in he sophomo e yea o Business,
Economics o Business and Law deg ees. The las g oup (double
majo in Business and Law) end o be composed o be e pe -
o ming s uden s, as he en ance equi emen s a e s ic e . The
ma hema ical backg ound o all g oups is simila : wo qua e s
o Calculus and Algeb a.
72 M.J. BÁRCENA ET AL.
5.1. Tes
We de ised a sho exam wi h ques ions in which s uden s we e
equi ed o ecognize whe he he use o he wo d “p obabili y”
had a Bayesian o equen is la o . Fo ins ance,
When we say ha he p obabili y ha he
Malaysian Ai lines plane los in 2014 ( ligh
MH370) is in a gi en a ea in he Paci ic ocean
wi h p obabili y p=0.01, a e we using he wo d
‘p obabili y’ in a Bayesian o in a equen is sense?.
We also included he simples p oblem we could hink o
which equi ed sequen ial applica ion o Bayes’ heo em—wha
he simula o is designed o p o ide aining o . I ead,
(a) In a oggy day, a hike su e s an acciden
when e u ning om a moun ain. He belie es ha
wi h p obabili y 0.60 he is in he No h slope, bu
wi h p obabili y 0.40 he migh ha e ended in he
Sou h slope. Be o e using his cellula phone o ask
o help,hewouldlike obe e ixhisloca ion.He
emembe s ha in he No h slope beech ees a e
p e alen (70% o he o al) wi h he es being oak
ees (15%) and yew ees (15%), while in he Sou h
slope he p opo ions a e 30% beech ees, 60% oak
ees and 10% yew ees. He app oaches he nea es
ee and ealizes ha i is an oak ee. Using his
in o ma ion, he p obabili y ha he is in he No h
slope is, app oxima ely...
This is a simple example in which s uden s can use he
a ailable in o ma ion o e ise hei p io p obabili y and ob ain
apos e io p obabili y.Thiswas ollowedby:
(b)A e ha ing ound he eemen ionedin
he p e ious ques ion, he walks u he and inds
ano he ee, which he ecognizes as being a yew
ee.He hen alls o heg ound,exhaus ed.Whe e
will he ell he escue b igade o sea ch o him, in
he No h o Sou h slope?
The in en is o go one s ep u he . He e, we wan he s uden
o ecognize ha he pos e io om (a) can be used as a p io in
(b)— he essence o Bayesian lea ning ha ou simula o a ge s.
5.2. Assessmen Me hod
Ou i s hough was o de ise a s anda d expe imen , ei he
andomizing he “ ea men ” (= use o simula o ) o aking pai s
o s uden s ma ched acco ding o hei abili y (measu ed by
hei g ade poin a e age, o ins ance) and assigning wi hin
each pai one o he g oup o simula o use s and he o he o
he g oup o nonuse s. We would hen compa e pe o mance
o bo h g oups when aking he es desc ibed in he p e ious
sec ion.
Howe e ,since he es was ogi ec edi owa d hecou se
g ade his would c ea e an un ai ness owa d he s uden s ha
we eno assigned o hesimula o g oup.This,inou con ex ,
canno be con empla ed. On he o he hand, he e is no way
in which we could ensu e ha he un ea ed o con ol g oup
was eallyun ea ed:ou o cu iosi yo o he wiseanys u-
den could use he simula o and hus con amina e he con ol
g oup.
We he e o e decided ha we would adminis e he es
wice, be o e and a e gi ing a chance o use he simula o .
We we e ully awa e ha he second ime he es is aken a
be e pe o mance is o be expec ed, e en o nonuse s o he
simula o . Bu we coun ed on measu es o use o he simula o
( ime spen , sco e ob ained when using i ) as well as on ha ing
some acciden al “con ols”: s uden s who o a ious easons
wouldno use hesimula o .Tha wouldenableus odisen angle
he e ec o using he simula o and he e ec o me e epe i ion
o he es . As i happens, he con ol g oup was la ge han we
expec ed.
A e co e ing Bayes’ heo em in class, we equi ed s u-
den s o ake he es desc ibed abo e. They we e hen encou -
aged o eely use he simula o o a pe iod o i e days: his
eeusepolicyand he ac ha only helas sco ewouldbe
eco ded was made clea o hem be o ehand, as well as he ac
ha use o he simula o migh be o some help in a second
es .
Then, he same es was gi en again, complemen ed wi h a
ew ques ions ega ding whe he hey had ound he use o he
simula o obeeasy, ewa ding,howmuch ime heyhadspen
on i , e c.
The ac ha we ha e he same s uden s ake he i s and
second es s enables us o accoun o di e ences in s uden
abili y.Thep oblem,o cou se,is ha he ecanbeasel -
selec ion e ec : mo e mo i a ed s uden s migh choose o use
he simula o in g ea e p opo ion han he o he s. Below we
epo on some e idence o his e ec in he esul s: i may
ha e been coun e ed by he ac ha be e pe o ming s u-
den s, a e ob aining a good g ade in he i s es , did no see
oom o imp o emen and he e o e neglec ed he use o he
simula o .
5.3. Resul s and Modeling
Bo h es s we e adminis e ed in he Fall Te m o 2017 in he o m
o mul iple choice ques ionnai es, o a oid any subjec i e biases
om he g ade s. As Figu es 3 and 4imply, heywe eg aded
on a 0–10 scale ( he colo -coded simula o sco es we e also in a
0-10 scale).
A o al o 241 s uden s pa icipa ed, bu 52 we e absen in
ei he he i s o second es (p e- and pos - es in he sequel).
The e o e, we obse ed 189 s uden s who ook bo h he i s and
second es plus 52 who only ook one o he es s.
Figu e 3 shows he b eakdown o g ades o he common se
o ques ions in he p e- and pos - es . (Subjec i e ques ions in
he pos - es as o whe he he simula o had been use ul, un,
e c. we e no g aded.)
Clea ly, he e is an imp o emen , pa icula ly appa en in he
la ge median g ade o he subse o s uden s ha used he
simula o . Also, g ades below 2.5 poin s we e en i ely absen
among he simula o use s.
A di e en , mo e insigh ul iew, is p o ided by Figu e 4.The
pe o mance o each s uden is shown by a poin , whose coo -
dina es a e he g ades in he i s and second es . (Poin s ha e
JOURNAL OF STATISTICS EDUCATION 73
Used simula o Did no use simula o
Fi s es Second es Fi s es Second es
0.0
2.5
5.0
7.5
10.0
Tes
G ade
Tes Fi s es Second es
Figu e 3. B eakdown on g ades in p e- and pos - es acco ding o use o simula o .
been ji e ed, o educe o e plo ing.) The wo panels show he
pe o mance o s uden s who did and did no use he simula o .
Fu he , o hose who did use he simula o , he sco e ob ained
is colo -coded in he le panel. The ed lines ma k show equal
g ades in he i s and second es s: poin s abo e he ed line
indica e an imp o emen .
We expec ed all sco es using he simula o o be equal o close
o he maximum 10, gi en ou policy o “use o as long as you
wish,keeponlyyou las g ade.”Asama e o ac , hishasno
been he case: some s uden s abandoned he ask ea lie wi h
low o e en ze o sco es. Howe e , al hough imp o emen in he
second es is no clea ly ela ed o he simula o sco e, i seems
much mo e consis en among s uden s who used he simula o ,
wha e e sco e hey ob ained.
When we b eak down he esul s by g oup, Figu e 5,we ind
ha s uden s in he ADEDE g oup (double deg ee in Business
and Law) did much be e han s uden s in ei he o he single
deg ees o Business (ADE) o Economics (ECO). I is appa en
also ha ADEDE s uden s no only ob ained highe sco es when
using he simula o , bu also used i in a la ge p opo ion (mo e
colo ed poin s in Figu e 5), which poin s o a possible p oblem
o sample sel -selec ion add essed ea lie .
To ob ain a mo e o mal assessmen o he e ec o he sim-
ula o , we ha e i ed se e al linea models. A na u al app oach
would be o conside o each s uden he di e ence in g ades
ob ained in he i s and second es as a esponse a iable,
and ela e ha di e ence o he use (o no ) o he simula o ,
and possible o he ac o s. In o he wo ds, o use a pai ed
compa isons app oach.
Howe e , a o al o 52 obse a ions a e no pai ed: some
s uden s ook he i s es and no he second o ice e sa.
In o de o use also hese obse a ions, we ha e i ed se e al
linea mixed models, Demidenko (2004), in which he esponse
a iable is he g ade ob ained in ei he o he es s. Di e ences
in he indi idual pe o mance o s uden s a e accoun ed o by
a andom e ec , as indi iduals a e o no in e es in hemsel es.
The simula o e ec , “ epea ” e ec , and g oup e ec a e in o-
duced as ixed e ec s. All models ha e been i ed in R, R Co e
Team (2017), using package lme4 (see Ba es e al. 2015).
Model 1 is, in he usual no a ion,
G ade =1+(1|ID) +Rep;
a iable Rep is a dicho omous a iable aking alue 0 o 230
obse a ions co esponding o he i s es and 1 o 200 obse -
a ions co esponding o he second. (1|ID) is a andom e m,
di e en o each s uden and e lec ing his o he indi idual
abili y. I is o no in e es in i sel —we know s uden s o be
74 M.J. BÁRCENA ET AL.
Used simula o Did no use simula o
0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0
0.0
2.5
5.0
7.5
10.0
G ade i s es
G ade second es
0.0 2.5 5.0 7.5 10.0
Sco e using simula o
Figu e 4. Indi idual pe o mances in p e- and pos - es acco ding o use o simula o and sco e ob ained. Each poin is colo -coded e lec ing he sco e ob ained when
using he simula o ( igh legend), wi h s uden s who did no use i shown in g ay.
di e en —bu use ul o alloca e he pa o a iance ha is
explained by s uden ’s he e ogenei y.
The coe icien o Rep ( he “ epea ” e ec ) is posi i e and
highly signi ican ( e e o column Model 1 in Table 1). I s
alue in his model e lec s he (possible) e ec o he use o
he simula o o some s uden s, as well as he ac ha ( o
all) be e pe o mance should be expec ed he second ime he
s uden s ook he es , i espec i e o whe he o no hey used
he simula o .
To disen angle he e ec a ibu able o he simula o om
ha o me e epe i ion o he exam, we can i he model
G ade =1+(1|ID) +Rep +Used.sim,
whe e Used.sim is a dicho omous a iable aking alue 1 o
obse a ions co esponding o he second es o s uden s who
did use he simula o . The es ima ion esul s can be seen in
column “Model 2” o Table 1. A es o Model 2 e sus Model 1,
see Table 2, shows a dec ease o 5.922 in de iance, highly sig-
ni ican (p=0.015). (The e is a e y small misma ch wi h he
alues o he log-likelihood and model c i e ia AIC and BIC
epo ed in Table 1, possibly consequence o di e en me hods
o compu ing he log-likelihood and deg ees o eedom.)
Howe e , when we i he model
G ade =1+(1|ID) +Rep +Used.sim +G oup
( esul s in column Model 3 o Table 1), he coe icien o
Used.sim becomes non-signi ican : i seems ha i is pa ly
con ounded wi h he G oup e ec . This was o be expec ed
since, as i is appa en om Figu e 5, s uden s in g oup ADEDE,
who a e clea ly be e pe o me s, ha e also used he simula o
in g ea e p opo ion. The G oup e ec is o pa amoun
impo ance and accoun s o a dec ease o 19.288 in de iance,
educing he a iance accoun ed by he andom e ec ID om
2.52 o 2.13, a educ ion o abou 15%: pa o he he e ogenei y
among s uden s is in ac a di e ence be ween g oups.
5.4. S uden ’s Feedback
The second es included a ew ques ions ega ding he expe i-
ence wi h he simula o . S uden s we e old ha hese ques ions
hadno e lec wha soe e in hei g ades,bu esponsewas
none heless o al among s uden s ha ing used he simula o .
Thei pe cep ion o whe he playing wi h he simula o was
o any help did no clea ly co ela e wi h hei pe o mance—see
JOURNAL OF STATISTICS EDUCATION 75
ADE ADEDE ECO
0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0 0.0 2.5 5.0 7.5 10.0
0.0
2.5
5.0
7.5
10.0
G ade i s es
G ade second es
0.0 2.5 5.0 7.5 10.0
Sco e using simula o
Figu e 5. B eakdown o g ades in p e- and pos - es acco ding o use o simula o pe g oup. Each poin is colo -coded e lec ing he sco e ob ained when using he
simula o ( igh legend), wi h s uden s who did no use i shown in g ay.
Table 1. E ec on G ade o use o simula o .
Model 1 Model 2 Model 3
(In e cep ) 5.60∗∗∗ 5.61∗∗∗ 5.25∗∗∗
(0.18)(0.18)(0.22)
RepYES 1.24∗∗∗ 0.99∗∗∗ 1.04∗∗∗
(0.22)(0.24)(0.24)
Used.simYES 1.08∗0.75
(0.44)(0.45)
G oupADEDE 2.12∗∗∗
(0.48)
G oupECO 0.40
(0.32)
AIC 2083.28 2079.17 2064.15
BIC 2099.53 2099.49 2092.60
Log-likelihood −1037.64 −1034.58 −1025.08
Num. obs. 430 430 430
Num. g oups: ID 241 241 241
Va : ID (In e cep ) 2.66 2.52 2.13
Va : Residual 5.04 5.06 5.07
NOTE: ∗∗∗p<0.001, ∗∗p<0.01, and ∗p<0.05.
Figu e 6. Whe he hey said i had been o no, li le, mode a e
o signi ican help, hei pe o mance in he second es appea s
o be be e —excep pe haps, qui e pa adoxically, o hose who
we e mo e con inced o he use ulness o he simula o . This
may be due o he ac ha hey we e good pe o me s in he
i s es and he e o e wi h li le oom o imp o emen .
On he o he hand, when asked abou how much ime hey
spen playing wi h he simula o , he e seems o be a clea
pa e n o g ea e imp o emen o hose who spen mo e han
30 minu es on he simula o : see Figu e 7. The median g ade
wen om 6.4 o he maximum o 10; o all he o he ca ego ies,
an upwa d shi in g ades is isible, bu —excep o he “None”
ca ego y— he medians emain he same. I is also appa en ha
hose ha used he simula o had a highe median g ade in he
i s es .
6. Discussion
The e idence we can p esen on he impac o he simula o is
no conclusi e: he pa ial con ounding wi h g oup e ec p e-
en sus ommakings ongclaimsabou heuse ulnesso he
simula o . Howe e , wha we can lea n om he obse a ional
