ONLINE ASSESSMENT: MAKE IT SIMPLE
Ma ga ida Ama al
Uni e sidade do Po o
Po o / Po ugal
mama al@ ei .up.p
Abs ac
The e is an ongoing p ojec on he Uni e si y o Po o (U.PORTO) ha aims he
associa ion o wo didac ical ools o in o ma ion and communica ion echnologies: e-
lea ning and compu e based assessmen (CBA).
In o de o make he con e gence o e-lea ning wi h CBA he e a e se e al issues ha
ha e o be add essed: pedagogical p oblems in he assessmen me hods, echnical
p oblems in he implemen a ion p ocess and o ganiza ional p oblems ha a e pa icula ly
complex in an uni e si y campus like U.PORTO: la ge and sp ead. The e-Assessmen
opic is o gene al in e es o hose who wo k in e-lea ning, specially now in he e a o
Web 2.0, whe e many ques ions a ise due o so many di e en app oaches o
me hodology, s a egies and e alua ion. U.PORTO is no excep ion. The need o being
mo e accu a e in he assessmen p ocess when using he web, e en i he eache s use a
blended-lea ning app oach, is eal and wo k is being de eloped in o de o y o espond
o his need.
Al hough he e-lea ning Uni (GATIUP) wo ks owa ds di e en assessmen me hods, his
pa icula pape ocus on he implemen a ion o online es s on campus using he
Uni e si y LMS – Moodle.
Keywo ds - Assessmen , online, mul iple choice ques ions.
1 INTRODUCTION AND CONTEXT
The main pu pose o his pape is o gi e a b ie desc ip ion o an ongoing p ojec a he Uni e si y o
Po o, which aims o a icula e wo a eas o educa ion and echnology: e-lea ning and compu e -based
es s. The cu en si ua ion o he Uni e si y conce ning he global e-lea ning P ojec is desc ibed below.
The ocus will mainly be on he ou comes and no pa icula ly in he implemen a ion s a egy as i is no
pe inen o he unde s anding o ou cu en p ojec on compu e based assessmen - CBA. The main
ocused howe e is he iden i ica ion o some cons ain s ha a ise om an ins i u ional app oach o
online assessmen implemen a ion.
1.1 U.PORTO e-Lea ning P ojec
A. Implemen a ion and Ou pu s
Implemen ing an e-lea ning p ojec in a la ge, sp ead and adi ional uni e si y like he Uni e si y o
Po o, is a complex ask ha mus in ol e a conside able numbe o ac o s, such as acul y manage s,
eache s, in o ma ion sys ems adminis a o s, lib a ians, da a p ocessing depa men s and, o cou se,
s uden s. Al hough we can now easily ind epo s o expe iences ca ied ou in sca e ed and
decen alized la ge uni e si ies like he Uni e si y o Sydney [1] o in he Manches e Me opoli an
Uni e si y [2], i was necessa y se e al yea s ago o he Uni e si y o Po o o ind i s own s a egic pa h
o add ess i s speci ici ies in he de elopmen o e-lea ning ac i i ies [3], [4].
In he school yea 2003-2004, he U.PORTO began a pilo -p ojec o de elop pedagogical con en s o a
Lea ning Managemen Sys em (LMS) ha was inc easingly in eg a ed wi h he Uni e si y In o ma ion
Sys em - SiGARRA. This p ojec , he ea e “e-Lea ningUP”, has he ollowing gene al objec i es:
1. Imp o e he s uden s’ lea ning condi ions ou side he class oom;
2. Imp o e he s uden s’ au onomy and esea ch capaci y;
3. Wo k wi h he s uden s o ob ain om hem a mo e ene ge ic con ibu ion o he lea ning p ocess;
4. Gi e s uden s he oppo uni y o ha ing some suppo documen s in ad ance such ha hei
pa icipa ion in he class oom becomes mo e ac i e and consis en ;
P oceedings o INTED2010 Con e ence.
8-10 Ma ch 2010, Valencia, Spain.
ISBN:978-84-613-5538-9
001550
5. P i ilege s a egies ha use s uden ac i i ies’ elemen s, like eading and pe o ming small asks,
which con ibu e o he aining o hypo he ical-deduc i e hough [5], [6].
6. Quickly publish lea ning objec i es, syllabus, bibliog aphy and o he documen s o in e es ;
7. Inc ease he in e ac i i y be ween he s uden s and he acul y;
8. P omo e a democ a ic access o in o ma ion;
9. Inc ease he oppo uni y o access o he cou se con en s o disloca ed s uden s;
10. Acqui e expe ience in echniques o dis ance educa ion h ough he In e ne .
E-lea ningUP has pa icula cha ac e is ics, no only because bo h eache s and s uden s a e o ien ed
owa ds a common inal objec i e in a wide ange o a eas, bu also o he inno a i e pu pose o
in eg a ing esou ces ha a e al eady a ailable in an in o ma ion sys em wi h a lea ning managemen
sys em. Ano he pe inen ac is he policy o di ec ing his p ojec owa ds p o esso s wi h echnological
compe ences as so wa e use s. One o he main objec i es o his p ojec was he u u e expansion o
con en s p oduc ion o he en i e p o esso s’ communi y o he uni e si y. This was achie ed since he
pilo eam o eache s in ol ed, cons i u e a ep esen a i e sample o he academic communi y wi h low
echnological skills. Pe iodically du ing he p ojec , exis ing esul s we e used o mo i a e o he
p o esso s owa ds he p oduc ion o con en s ollowing p o ed me hodologies, and no o amaze hem
wi h o e de eloped mul imedia esou ces, which migh become un ealis ic and e en igh ening o hose
who do no ha e ad ance echnological knowledge.
We would like o poin ou he ac o he eache s’ e-con en de elopmen and ins uc ional design on he
LMS was, and s ill is, guided bu no modeled by GATIUP.
Th ee dis inc pla o ms we e al eady used – LUVIT, WebCT (bo h Campus Edi ion and VISTA) and
Moodle. The LUVIT pla o m was abandoned in 2006, and was eplaced by he Moodle pla o m, which
was al eady widely used by se e al p o esso s who had engaged in indi idual ini ia i es. Moodle
became he unique pla o m o U.PORTO in 2009. The LMS is now a common esou ce in he uni e si y
campus which s uden s, echnicians and eache s a e amilia wi h.
A he p esen ime all Facul ies a e in ol ed in he eLea ningUP p ojec , which comp ises mo e han
621 on-line cou ses on Moodle. Some Facul ies, like Sciences and Enginee ing, ha e hei own suppo
o ices, – Moodle@FCUP o e s 700 on-line cou ses o s uden s; Moodle@FEUP o e s abou 1000 on-
line cou ses. [7]
By online cou se i should be in ended a blended-lea ning suppo o p esen ial cou ses, gi en ha he
Uni e si y o Po o is a adi ional campus-based ins i u ion.
B. E-lea ningUP ou comes
As a p oduc o his p ojec , se e al pedagogic ma e ials and s a egies o hei use a ose. These
p oduc s a e analyzed by he p o esso s in ol ed and app aised by he en i e academic communi y,
allowing o he sus ainabili y o u u e decisions conce ning ac ions o he con inuous imp o emen o
he quali y o he uni e si ies’ cou ses. Fi e annual semina s we e held, om which books o
p oceedings, CDs and DVDs con aining he p esen ed wo k we e de eloped. Also, in 2005, U.PORTO
c ea ed an in e nal Excellence Awa d o e-lea ning. This awa d in ends o dis inguish hose ha use he
LMS wi h a p ope pedagogical s a egy in hei eaching ac i i ies du ing he school yea . The e-
con en s de eloped o suppo o he cou se as well as communica ion ools o o he inno a i e usage
o e-lea ning is aken unde conside a ion. [8]
F om he beginning o his p ocess, he GATIUP’s philosophy included he eache s’ in ol emen on
na ional and in e na ional con e ences, na ional unding p ojec s, as well as ying o include hem in
in e nal discussion g oups abou pedagogic and pa adigma ic ques ions ega ding new echnologies in
educa ion. Wi h ime, a conside able numbe o eache s began o eel he inc easing need o explo e a
di e si y o ai s, while main aining he belie ha hey would complemen and e en imp o e hei own
expe iences in e-lea ning wi h s uden s.
This is an in e es ing e olu ion o op-down p ocess wi h a s ong componen o in e es owa ds a
middle-ou p ocess.
2 MOTIVATION FOR COMPUTER BASED ASSESSMENT
Keeping in mind wha was p io said abou he use o elec onic ools in eaching and lea ning, we will
in oduce a his poin o he pape ano he ele an expe ience held in he Uni e si y.
Fo many decades mul iple choice ques ions (MCQ) o o he ypes o ques ions ha allow au oma ic
001551
g ading, we e used in exams h ough he Uni e si y. Some acul ies e en ha e a sys ema ic
digi aliza ion by op ic eading o all e alua ion es s. The ans o ma ion o he op ic eadings in o
ma ixes o answe s allows he applica ion o speci ic so wa e which, h ough an i e a i e calcula ion
p ocedu e, adjus s he da a o logis ic equa ions o he IRT (I em Response Theo y). The da a analysis
allows he measu emen o di icul y and disc imina ion pa ame e s o each i em, as well as amoun o
in o ma ion each i em complies. Consequen ly, a con inuously g owing se o calib a ed i ems has
been buil . Mo eo e , he de eloped wo k as been used o guide he cu en eaching ac i i y o he
cou ses.
CBA was a smoo h and na u al e olu ion o hese eache s: hey had accumula ed expe ience gi en
by pape -pencil exams, and hey ha e now he skills and knowledge o wo k wi h new echnologies in
he pedagogical p ocess. I a LMS was al eady being used o pedagogic ac i i ies such as deli e y o
documen s, assignmen s, discussions why no ex end ha o summa i e e alua ion?
Many eache s we e using quizzes as o ma i e e alua ion ool bu didn’ ake he leap in o he inal
exam.
GATIUP had he ole o suppo ing ha leap: making i simple.
3 INDIVIDUAL TEACHING EXPERIENCES USING COMPUTER BASED
ASSESSMENT
A. How online exams we e held
On he school yea o 2007-2008, one eache engaged on an expe ience o examining 409 s uden s
using he compu e and Moodle. This inc edibly high numbe o s uden s esul ed o he me ging o
wo di e en cou ses in one by he ligh o he Bologna p ocess o edesigning he s udy plan.
The cou se was on Moodle: s uden s could access he LMS o consul documen s o classes, ex s,
web pages, discussions and deli e assignmen s. E e y s uden had access o Moodle and all he
s uden s we e using i hea ily. The eache assembled on Moodle a ques ion bank wi h 252 MCQ.
The ini ial idea was o c ea e indi idual exams o each s uden wi h 60 andom ques ions. Since i
was a a he new expe imen in such a la ge scale i was decided no o engage in andom assembly
o exams bu o make 2 di e en exams o wo dis inc backg ound g oups o s uden s.
Immedia ely one could iden i y se e al issues on his p ocess ha had o be managed and sol ed:
- numbe o compu e s a ailable
- class ooms wi h compu e s connec ed o he in e ne
- secu i y issues
- use iden i ica ion issues
GATIUP and he eaching eam wo ked a ound his p oblem using he media a ailable:
- 3 ooms wi h compu e s connec ed o he in e ne on a o al o 45
- Technical acul y suppo in o de o p o ide secu i y o he online exam
On he day o he exam s uden s appea ed in g oups o 45, p e iously assembled and pos ed on
Moodle page o he cou se. One oom was booked o he s uden s lea e all hei belongings. One
membe o GATIUP accompanied he p ocess all day. G oups o 45 s uden s en e ed he oom e e y
60 minu es wi h 15 minu es b eak o he dis ibu ion by he 3 compu e ooms. On he echnical side,
he ne wo k was blocked only o access SIGARRA and Moodle (SIGARRA is whe e s uden s
au hen ica e hei p o iles and au oma ically en e he LMS). Also, on Moodle is possible o de ine he
IP add ess o he machines ha a e connec ed o ha pa icula exam. In addi ion o his e e y shi o
s uden s had a di e en passwo d o en e he exam.
The e we e no secu i y ails, on au hen ica ion o du ing he pe o mance o he s uden s on he
exam. The ne wo k se ices o he uni e si y we e also engaged in his p ocess in o de o p o ide a
s able in e ne connec ion ying o a oid any ailu es ha migh happen.
This exam day s a ed a 9:00 am and inished a 18:00pm wi h no lunch b eak. By he end 287
s uden s we e e alua ed and g aded.
001552
Each s uden could see hei pe sonal sco e by he ime hey submi ed hei exam. This allowed
s uden s o plan hei ecu ing pe iod o examina ions since hey knew hei sco e in ha pa icula
cou se.
The eedback om s uden s was posi i e.
The eache s we e deligh ed wi h he ac o ha ing he wo k done by he end o he day. I allowed
he eache s o ha e ime o spend in o he pedagogical ac i i ies. Moodle also p o ides an i em
analysis ool. Wi h a sho aining on how o in e p e hose da a eache s we e mo e awa e o he
quali y o he ques ions p oduced o ha exam.
This p ocess was epea ed on he ecu ing pe iod o examina ions, using 60 di e en ques ion o he
ques ion bank. In his pe iod 122 s uden s we e e alua ed and g aded.
Since his i s online examina ion expe imen o he eache s al eady did hei inal examina ions
online always wi h he suppo o GATIUP. Some numbe s:
On he school yea o 2008/2009, 1548 s uden s did hei inal exams online; in 2009/2010 only in he
i s semes e 869 s uden s we e e alua ed and g aded on Moodle.
B. Main easons ha igge ed he CBA
The la ge numbe o s uden s, ime spen on he p ocess (since de elopmen o classi ica ion), lack o
adjus ed es s, inadequa e me hods ela ing o new pa adigms o eaching and lea ning a e cu en
p oblems ha U.PORTO academics ace. These lead o he engagemen in he use o inno a i e
assessmen me hods.
Online assessmen wi h au oma ic eedback p esen s i sel as a possible solu ion o he p oblems
lis ed abo e and also allows o he scalabili y, p ocess liabili y, adjus ed es ing e en when a la ge
numbe o s uden s is being e alua ed, c ea ion o new oppo uni ies o s uden s (accessibili y) and
ins i u ions (new ma ke s) and de elopmen o me hodologies ha allow a secu e and liable
assessmen in online dis ance cou ses.
This las i em assumes i al impo ance o he Uni e si y hence i ’s engaging in he de elopmen o
online dis ance cou ses in li elong lea ning a ea.
4 INSTITUCIONAL SUPPORT FOR CBA IMPLEMENTATION
The us de eloped be ween he suppo ing uni o New Technologies in Educa ion and he academic
communi y is i al o he implemen a ion o such p ojec . Final assessmen is a se ious phase in he
pedagogic p ocess along a cou se and i has o be secu e and eliable.
Al hough he suppo ing eam had al eady a sus ained knowledge o educa ional echnology, a ha d
wo k on esea ching and sel - aining in he a ea o compu e -based assessmen (CBA) and compu e
adap i e es ing is being made.
The p ocess o ex end om e-lea ning o CBA looks a he as a na u al e olu ion and om he
concep ual poin o iew doesn’ seem o b ing special di icul ies. Howe e he ex ension o he en i e
campus aises scalabili y issues di icul o sol e, o which wo a e especially conce ning:
1. How o make his p ocess anspa en o acul y;
2. How o assu e some speci ic compe encies such as psychome ician expe ise and human and
physical esou ces a e p ope and eliable.
I is no easy o ind in he li e a u e a epo o a s a egy o implemen a ion o a compu e adap i e
sys em o a la ge communi y o academics. I is common o ind de ailed desc ip ions abou CAT
cons uc ion, deli e y and analysis wi h all he ma hema ical models explici ly shown and explained
wi hin he con ex .
The Uni e si y o Po o in ends o c ea e a s uc u e ha con ains in i sel he possibili y o he
academics o use CBA and/o CAT in a anspa en way. By his, he wo k will ocus in c ea ing
condi ions o a ailabili y o hese ools o e e y eache who wan s o use he compu e as a mean o
p oduce, deli e and analyze a es . I is impo an ha all he models and ma hema ical backg ound a e
assu ed by a psychome ician, who mus suppo he eache s who should no need o un h ough he
heo ies and p inciples o a CAT sys em.
001553
Keeping in mind he goal o implemen ing CATs in mo e han one discipline in he Uni e si y, he ini ial
app oach will be he use o CBA o e alua e s uden s.
The suppo o ice o new echnologies in educa ion aced se e al s eps in o de o pu sue his goal:
T aining Facul y Suppo S a - One o he unc ions o a acul y suppo o ice ha is o en o e looked,
e en hough i is expensi e and ime-consuming, is he aining o i s own s a . Facul y suppo
pe sonnel mus lea n (and buy) new e sions o so wa e in a imely manne , and mus equen ly add
adding new knowledge o hei epe oi e o skills and unde s anding.
T aining he academic communi y – building mul iple choice ques ions, assembling a es based on he
compu e , IRT, p inciples o CAT and o he subjec s ele an o he unde s anding and execu ion o his
p ojec . T aining o he academic s a in he de elopmen o mul iple choice ques ions is a demand o
he eache s ha wo k in his p ojec . GATIUP al eady minis e ed a 14 hou s aining cou se in he
de elopmen o MCQ. The second edi ion o his cou se ook place in Janua y 2010 wi h he
p esence o an in i ed expe in Psychome ics ha add essed 61 eache s on a wo kshop. The e en
was held unde he punch line: “Is in ui ion enough o build an exam?”. I was a success ul wo kshop
which p o ed o be an indica o ha his p ojec , h u aining is al eady inc easing he awa eness
a ound ques ions cons uc ion and hei quali y analysis a p io i and a pos e io i.
The eache s in ol ed in online es ing a e now on he calib a ion phase o hei ques ions. In addi ion
se e al o he eache s ha s ill use pape -pencil in hei exams a e con ac ing GATIUP in o de o
analyse h ough Classic Tes Theo y hei exams esul s in p e ious yea s.
Analyzing so wa e a ailable on he ma ke - he IMS Global Conso ium, an indus y and academic
conso ium, p oduced he IMS QTI (IMS Ques ion & Tes In e ope abili y Speci ica ion) o enable he
exchange o ques ion and es da a and also esul s epo s [9]. The QTI speci ica ion comp ises se e al
documen s which a e being aken unde conside a ion in he decision o choosing assessmen so wa e
wi h in e ope abili y o ques ion design.
Fe l Bec a [10], [11] lis s 21 so wa e ools o e-assessmen . F om ha lis only 4 p oduc s p o ide IMS
QTI and SCORM suppo o some ex en .
In ela ion wi h lea ning pla o ms he scena io o in e ope abili y doesn’ ge any be e . Moodle, which is
used a UP, allows o expo IMS QTI 2.0 o ma bu no o impo . [12],[13]
Among o he TDS analyzed [14] so wa e al e na i es we chose o use he Fas Tes P o [15] - Tes
De elopmen Sys em (TDS), Session Dis ibu ion Package (SDP) and I em and Tes Analysis
Package (ITAP).
Also, we a e ca e ully analyzing o line al e na i es ha can be synch onized wi h Moodle. This is a
e y impo an aspec because o he Moodle se e load du ing online exams. The numbe o
eache s and s uden s is aising e e y exam season.
Tes ing Rooms – o pe o m CBA i is necessa y o gua an ee ooms wi h a conside able numbe o
compu e s p epa ed o assessmen . These ooms wi h speci ic cha ac e is ics [16] o p o ide a
dis ac ion- ee, secu e es ing en i onmen wi h con inuous candida e su eillance can be al eady
ound in some o ou acul ies.
Wi h consecu i e expe iences we expec o iden i y ou comes ha allow us o make adjus men s o
ou s a egy in o de o succeed wi h he de elopmen o esou ces o compu e -based es ing in
campus-wide IT sys ems.
5 CONCLUSIONS
CAT and o he o ms o compu e based es ing a e ob ious ex ensions o e-lea ning ne wo ks in
academic se ings. Howe e , hese ools equi e complex analysis and expensi e equipmen . The
scien i ic backg ound o CAT is e y obus . Validi y and eliabili y issues a e also well cha ac e ized in
many s udies. Howe e he logis ics equi ed es ablishing CAT a a la ge scale in a uni e si y campus
can be daun ing. Ra he unexpec edly he e is lack o case epo s on well succeeded p og ams.
Howe e he lea ned expe ience o coping wi h he p oblems o es ablishing an e-lea ning p ojec in a
la ge, sp ead and adi ional uni e si y like Uni e si y o Po o may be use ul.
001554

ACKNOWLEDGMENTS
The au ho wish o hank P o esso Daniel Mou a, o his help and ad ice. The con ibu ion om ou
eache s, echnical pe sonnel and s uden s a e wa mly acknowledged.
Re e ences
[1] R.A. Ellis, N. Ja key, M.J. Mahony, M. Pea and S. Sheely, “Managing Quali y Imp o emen o
eLea ning in a La ge, Campus-Based Uni e si y,” Quali y Assu ance in Educa ion, ol. 15, pp.9-
23, 2007.
[2] Anonymous, “Manches e Me opoli an Uni e si y,” Technical Repo Mega ends in eLea ning
P o ision NO/05/C/F/RF-83250, No sk Kunnskaps-Ins i u , Bekkes ua, No way, a ailable a
h p://www.ne skolen.com/in_english/mega ends/MMU_a icle.pd , Ma . 2007
[3] A. Soei o, M.L. Pin o and E. Pe ei a, Pedagogia e No as Tecnologias: Es udos de Casos
Uni e si á ios, CDROM, Uni e si y o Po o Ed, 2000. (in Po uguese)
[4] R. Falcão, “E-lea ning a he Uni e si y o Po o om 1998-2005: Implemen ing E-lea ning a a
T adi ional Uni e si y,” a ailable a
h p:www.ai. uwienac.a /elea ning/inno a ion_lec u e, Dec. 2006.
[5] A.E. Lawson, Science Teaching and De elopmen o Thinking. Belmon , Cali .: Wadswo h, pp.
261-299, 1995.
[6] A.E. Lawson, The Neu ological Basis o Lea ning, De elopmen and Disco e y: Implica ions o
Teaching Science and Ma hema ics. Do d ech , The Ne he lands: Kluwe , pp. 225-260, 2003.
[7] E-lea [email protected] o, Ma . 2008;
h ps://siga a.up.p /up/web_base.ge a_pagina?P_pagina=18385 (in Po uguese)
[8] Senado da Uni e sidade do Po o, P émio Excelência e-Lea ningU.Po o, Ma . 2007;
h p://siga a.up.p / ei o ia/LEGISLACAO_GERAL.
e _legislacao?p_n =143 (in Po uguese)
[9] IMS Global Lea ning Conso ium. IMS Ques ion & Tes In e ope abili y Speci ica ion, Dec.2006 ;
h p://www.imsglobal.o g/ques ion/
[10] Fe l Bec a, “On-line Assessmen so wa e”, No . 2006;
h p:// e l.bec .o g.uk/display.c m?page=193
[11] S. Suleman, “E-assessmen ”, Fe l Fi s , BECTA, UK, 29 Ap 2003, No . 2006;
h p:// e l.bec .o g.uk/display.c m? esid=5367&p in able=1
[12] Moodle S a is ics, Dec. 2006;
h p://moodle.o g/s a s/
[13] Moodle. Release No es, Dec. 2006;
h p://docs.moodle.o g/en/Release_No es
[14] E.J. Gibson, P.W. B ewe , A. Dholakia, M.A. Vouk & D.L. Bi ze , “A compa a i e analysis o Web-
based es ing and e alua ion sys ems.”, P oc. 4 h WWW con e ence, Bos on, 1995;
h p:// enoi .csc.
ncsu.edu/MRA/Repo s/WebBasedTes ing.h ml
[15] Fas Tes P o. (2000). [Compu e p og am]. Assessmen Sys em Co po a ion, Ma . 2008;
h p://www.assess.com/Fas TEST.h ml
[16] Pea son Vue: Become a es cen e ; Jan. 2007; h p://www.pea son ue.com/p c/ acili y_ eqs.pd
001555