P ac ice Pape
Recommended ci a ion: Li, C., Almeida, C., Ma hews, S., & Eskelinen, H. (2025).
Pee Re iew Ve sus Manual Assessmen in a Mul icul u al 3D Modeling Cou se. In
Kangaslampi, R., Langie, G., Jä inen, H.-M., & Nagy, B. (Eds.), SEFI 53 d Annual
Con e ence. Eu opean Socie y o Enginee ing Educa ion (SEFI), Tampe e,
Finland. DOI: 10.5281/zenodo.17631325.
This Con e ence Pape is b ough o you o open access by he 53 d Annual Con e ence
o he Eu opean Socie y o Enginee ing Educa ion (SEFI) a Tampe e Uni e si y in
Tampe e, Finland. This wo k is licensed unde a C ea i e Commons
A ibu ion-NonComme cial-Sha e Alike 4.0 In e na ional License.
PEER REVIEW VERSUS MANUAL ASSESSMENT IN A
MULTICULTURAL 3D MODELING COURSE
C. Li a,
1
, C. N. Almeidab, S. Ma hews c, H. Eskelinen d
a LUT Uni e si y, Lappeen an a, Finland, ORCID 0000-0001-9400-3998
b LUT Uni e si y, Lappeen an a, Finland, ORCID 0000-0002-1832-5677
c LUT Uni e si y, Lappeen an a, Finland
d LUT Uni e si y, Lappeen an a, Finland
Con e ence Key A eas: Di e si y, equi y and inclusion in ou uni e si ies and in ou
eaching; Enginee ing e hics educa ion
Keywo ds: Pee e iew; la ge-scale cou ses; manual e alua ion; in e na ional
s uden s; enginee ing educa ion
ABSTRACT
This pape examines he implemen a ion and e ec i eness o pee e iew e alua ion
compa ed o manual assessmen in a la ge-scale 3D modeling cou se, o e ed o
in e na ional and local s uden s a a Eu opean uni e si y. The s udy compa es
s uden pe o mance da a ac oss wo academic yea s, wi h pee e alua ion in 2023
and manual assessmen in 2024 o he same cou se. Resul s demons a e ha pee
e iew e alua ion can be an e ec i e assessmen me hod o la ge-scale
enginee ing cou ses, educing ins uc o wo kload om app oxima ely 30 hou s o 1
hou pe exe cise while main aining educa ional e ec i eness. Addi ionally, an
in e es ing phenomenon was obse ed: in e na ional s uden s seemed mo e
mo i a ed o pu sue highe g ades compa ed o local s uden s. Thei pe o mance
ollowed a simila end, ega dless o he e alua ion me hod used. Finally, o be e
unde s and his di e ence, in o mal discussions we e held wi h s uden s and
eache s. These con e sa ions indica ed ha in e na ional s uden s ace mo e
challenges a e g adua ion, which may explain hei mo i a ion o achie e highe
academic esul s.
1
Co esponding Au ho
C Li
li.changyang@lu . i
1 INTRODUCTION
La ge-scale uni e si y cou ses and massi e open online cou ses (MOOCs), which
ypically ope a e wi hou a di ec ins uc o in ol emen , p esen signi ican
challenges o educa o s in selec ing app op ia e e alua ion me hods. Pee e iew
e alua ion, al eady well es ablished in academic pape publishing, o e s a po en ial
solu ion o managing assessmen s in high-en ollmen cou ses. In pape publishing
p ocess, e iewe s p o ide eedback o au ho s, who e ise hei wo k acco dingly.
Then e ised manusc ip is usually e-e alua ed and i he e iewe s a e sa is ied, i
is accep ed o publica ion. Simila ly, pee e iew has also been adop ed in uni e si y
cou ses o a ious easons, such as encou aging s uden s o hink c i ically, lea ning
om pee s, o o MOOC cou ses (B kić e al., 2024). In such cases, he eache
akes on a mo e obse a ional ole, o e seeing he p ocess a he han di ec ly
assessing each s uden . This can help educe he eache ’s wo kload, allowing mo e
ime o ocus on asks like de eloping he cou se con en .
In cou ses whe e s uden s need o use a so wa e o model some hing, i can be
echnically challenging o he eache o p o ide w i en eedback, especially in
la ge-scale cou ses (Van e al., 2023). Pee e iew e alua ion can suppo his o
some ex en , bu i equi es ha bo h eache s and s uden s a e amilia wi h he
necessa y echnology ools (Badea & Popescu, 2019). Fo example, eache s may
p o ide a de ailed e alua ion ma ix, and s uden s hen e iew each o he ’s wo k
based on ha ma ix o ensu e he quali y and consis ency o he e iew (Li e al.,
2010). S uden s can lea n no only om ecei ing pee eedback bu also om he
p ocess o e alua ing o he s’ wo k (Nicol e al., 2013). Howe e , pee e iew does
no always imp o e he o e all expe ience compa ed o adi ional assessmen
me hods, and i s eliabili y is ques ionable. S ill, i o e s ad an ages in o he a eas
(Chambe s e al., 2014). S uden s a e usually posi i e when hey s a p o iding pee
e iew bu may ha e di e en opinions when hey ecei e eedback om pee s, and
some imes hei a i udes migh become pola ized (Mulde e al., 2014).
S udies gene ally indica e ha s uden s wan o pa icipa e in he assessmen
p ocess and iew pee assessmen posi i ely (Wanne & Palme , 2018; Nieminen e
al., 2021). The e o e, i makes sense o include hem in he assessmen c i e ia.
Fu he mo e, pee e iew s ands ou as a aluable eaching me hodology ha no
only suppo s s uden lea ning bu also p omo es c i ical hinking. A s udy by Guel i
e al. (2021) ound mode a e ag eemen be ween pee -assigned and eache -
assigned g ades. By implemen ing pee e iew sys ems, ins uc o s can apply his
s a egy e ec i ely in la ge classes, educing hei wo kload while ac i ely engaging
s uden s in he assessmen p ocess.
Pa anen e al. (2023) highligh ha he as -e ol ing na u e o enginee ing equi es
g adua es o become li elong lea ne s, wi h an emphasis on deep lea ning s a egies
and sel - egula ion. To suppo his de elopmen , educa o s a e encou aged o
implemen eaching and assessmen me hods ha cul i a e hese compe encies,
including pee and sel -assessmen .
This pape p esen s a compa a i e case s udy examining di e ences obse ed when
pee e iew and manual eache assessmen s a e applied o he same cou se
exe cises ac oss consecu i e academic yea s. In 2023, pee e iew was used o
assess selec ed exe cises, while in 2024, he same exe cises we e e alua ed
manually by a single eache . The s udy aims o p o ide p ac ical insigh s o
enginee ing educa o s ega ding e alua ion me hod selec ion in la ge-scale cou ses.
The pape is s uc u ed as ollows: Chap e 2 in oduces he cou se implemen a ion;
Chap e 3 p esen s s uden pe o mance da a and he eache s’ insigh s; Chap e 4
p o ides concluding ema ks and implica ions o p ac ice.
2 COURSE IMPLEMENTATION
2.1 Cou se desc ip ion
The cou se “3D Modeling and Technical Documen a ion” is a manda o y 6-c edi ,
yea -long cou se designed o all i s -yea enginee ing s uden s, including bo h
in e na ional and local s uden s, a a Uni e si y in Eu ope. The cou se consis en ly
ecei es a high numbe o en ollmen s each yea , eaching 600 s uden s in 2023.
Upon comple ion, s uden s should be able o use 3D modeling so wa e o c ea e
complex 3D models and 2D d awings, as well as o p oduce he co esponding
echnical documen a ion. The lea ning objec i es include, o 3D modeling:
designing, modeling, assembling componen s using di e en ma es, and c ea ing
basic anima ions; and o 2D d awing: demons a ing an unde s anding o gene al
ole ances, applicable s anda ds, geome y ole ancing, enginee ing i , su ace
oughness, welding symbols, and he abili y o make assembly d awings.
The cou se, in i s cu en o m, was i s implemen ed in he academic yea 2021. As
s uden en ollmen inc eased signi ican ly, he eache s began o conside he use o
pee e iew e alua ion o ce ain exe cises. The goal was o educe he eache s’
wo kload wi hou comp omising he quali y o he cou se. The g ading scheme o he
cou se is p esen ed in Table 1, wi h pee e iew e alua ion applied o indi idual
exe cises 1-3.
Table 1. The g ading scheme o he cou se.
Assessmen ask
Desc ip ion
Sha e o he
inal g ade
Indi idual exe cises 1-3
S uden s inish he ask using
so wa e SolidWo ks
30%
Indi idual exe cise 4
S uden s inish he quiz- ype
ques ion
5%
G oup exe cise
1s phase: make 3D modeling
2nd phase: make 2D d awing
35%
O al examina ion
G oup o al discussion wi h a eache
a e g oup exe cise
30%
The aim o indi idual exe cises 1-3 is o encou age s uden s o c ea e 3D models
based on a gi en opic. De ailed s ep-by-s ep u o ials a e p o ided o guide hem,
and s uden s who demons a e inno a ion in hei wo k can ea n addi ional poin s.
These exe cises do no ha e a unique solu ion; he only s ic equi emen is ha
s uden s submi hei exe cises co ec ly. The le el o di icul y inc eases
p og essi ely om he i s o he hi d exe cise. As he second pa o he cou se
ocuses on echnical documen a ion in 2D d awing, which may be pe cei ed as less
in e es ing han 3D modeling, i is impo an o mo i a e s uden s a he beginning o
he cou se. The e o e, eache s adop a mo e lexible app oach when g ading
indi idual exe cises 1-3.
Due o he la ge numbe o s uden s, eaching assis an s, ec ui ed om he p e ious
yea ’s s uden s, we e hi ed o p o ide echnical suppo du ing he exe cise session.
Thei ole is limi ed o assis ing wi h modeling echniques and helping s uden s o
unde s and pee eedback. Since eaching assis an s a y sligh ly in hei echnical
expe ise, hey a e no in ol ed in e alua ing he exe cises.
In he pee e iew p ocess, andomized pee e iew g oups a e assigned o each
assignmen o educe bias and expose s uden s o di e en pe spec i es. Suppo ing
s uden well-being is also a p io i y: clea communica ion o deadlines and
expec a ions helps minimize s ess and p e en bu nou . Feedback is designed o
highligh s eng hs and p o ide cons uc i e sugges ions o imp o emen .
2.2 E alua ion me hods
In he academic yea 2023-2024, he pee - e iew assessmen was used o e alua e
indi idual exe cises 1 o 3. This was implemen ed using he in eg a ed “Wo kshop”
ool in Moodle. In p ac ice, once he submission deadline is closed, he sys em
au oma ically assigns each s uden wi h h ee andom exe cises o e alua ion.
S uden s hen assess hese h ee exe cises using he e alua ion ma ix in eg a ed
in o he sys em, which was designed by he eache . Because he assessmen is
subjec i e, signi ican a ia ion can occu i he e alua ion c i e ia a e oo gene al. To
add ess his, he e alua ion ques ions we e designed as p esen ed in column 2 in
Table 2.
Table 2. Assessmen C i e ia and Pe cen age Dis ibu ion.
Exe cise
Pee Re iew Assessmen (Moodle
Wo kshop)
Teache Assessmen
(Single Ins uc o )
1
•Co ec ile submission wi h basic
model: 57%
• D awing cleanliness: 14%
• Inno a ion: 29%
• Co ec basic model: 60%
• Inno a ion: 40%
2
• Co ec ile submission wi h basic
model and assembly unc ionali y: 29%
• D awing cla i y & dimensions: 57%
• Inno a ion: 14%
• Co ec basic model: 60%
• Inno a ion: 40%
3
• Co ec ile submission wi h basic
model and componen ma ing: 33%
• Technical documen a ion: 33%
• Inno a ion: 33%
• Co ec basic model: 67%
• Inno a ion: 33%
The assessmen c i e ia o e s uden s lexibili y and help educe he isk o blind
e alua ion. Since he inal sco e is based on he a e age o h ee assessmen s, he
po en ial o indi idual bias is expec ed o be ela i ely low. Du ing he e alua ion
p ocess, s uden s no only gain insigh by e iewing hei pee s’ wo k bu also bene i
om eading he eedback hey ecei e.
In he academic yea 2024-2025, he assessmen o indi idual exe cises 1 o 3 was
ealized manually by a eache . All he exe cises we e e alua ed by he same
eache o ensu e consis ency and ai ness h oughou he p ocess. The e alua ion
ma ix is p esen ed in column 3 o Table 2 and i was p o ided in he ins uc ions
be o e s uden s began hei exe cises, allowing hem o sel -assess hei wo k be o e
ecei ing a g ade.
Bo h assessmen me hods p io i ize co e enginee ing skills, alue inno a ion,
main ain consis en p o essional s anda ds, and a e s uc u ed o e lec inc easing
ask complexi y. To educe wo kload and simpli y communica ion, especially when
s uden s send emails o mul iple eache s, eache s limi esponses by email.
I s uden s ha e ques ions abou hei inal sco e, hey a e encou aged o con ac he
eache s du ing o ice hou s o a end he exe cise sessions. These in e ac ions
allow eache s o engage mo e di ec ly wi h s uden s, and when s uden s a end he
exe cise sessions, hey usually lea n mo e.
3 RESULTS AND INSIGHTS
3.1 S uden ’s pe o mance
The poin s o indi idual exe cises 1 o 3 we e expo ed om Moodle and analyzed.
Figu e 1(a) and 1(b) p esen ba cha s showing he a e age alues and s anda d
de ia ion dis ibu ions o each exe cise and he o al sco e ac oss all h ee
exe cises. The x-axis ep esen s he i em, and he y-axis shows he co esponding
poin alues. Di e en colo s indica e alues om di e en academic yea s.
Fig 1. (a) he a e age alue (b) s anda d de ia ion dis ibu ion o each exe cise and he sum
o hese exe cises.
Figu e 1 e eals wo key obse a ions: he i s obse a ion is ha he end o
pe o mance om indi idual exe cises 1 o 3 is he same, which is dec easing. This
can be e lec ed in he di icul ies o he exe cises, and s uden s’ mo i a ion may also
ha e sligh ly d opped du ing he cou se. The second obse a ion is ha he o al
poin s ob ained om hese h ee indi idual exe cises in 2024 a e lowe han in 2023,
wi h a d op o abou 33%. O e all, his esul was expec ed by eache s, because he
exe cises we e adjus ed a bi in 2024; i s uden s wan ed o ge highe poin s, hey
needed o be a li le bi mo e c ea i e, ins ead o blindly ollowing he u o ial.
F om a pedagogical pe spec i e, he obse ed esul s show di e ences be ween he
pee - e iewed yea 2023 and he manually assessed yea 2024. Da a indica es ha
manual assessmen in 2024 led o lowe a e age sco es bu a highe s anda d
de ia ion, pa ly because exe cises equi ed mo e c ea i i y om s uden s. I seems
(a)
(b)
ha manual assessmen by a single eache allows o ine g ading and cap u ing
de ails like c ea i i y, which he pee - e iew sys em migh a e age ou ac oss
di e en s uden e iewe s. While pee e iew in 2023 was unc ional, educed
eache wo kload, and showed less sco e a ia ion, i di e s om he manual
e alua ion which migh be mo e disc imina ing bu akes mo e ime. This p esen s a
pedagogical ade-o : he e iciency and s uden lea ning om pee e iew e sus
he po en ially mo e consis en and de ailed g ading om manual assessmen by one
expe . I is impo an o no e ha , when conside ing he o e all cou se sco es, he e
is li le di e ence be ween pee e iew e alua ion and eache s assessmen me hod.
Then o each exe cise in each academic yea , he poin s o each exe cise a e
di ided in o six in e als: 1) 0 poin s; 2) 1-2 poin s; 3) 3-4 poin s; 4) 5-6 poin s; 5) 7-8
poin s; 6) 9-10 poin s. All s uden s a e di ided in o wo g oups, local s uden s and
in e na ional s uden s. Thei pe o mances o exe cises 1, 2, and 3 in academic
yea s 2023 and 2024 a e illus a ed wi h line cha s and shown in Fig. 2 (a)-( ).
The esul s shown in Figu e 2 a e collec ed a e obse ing ha in e na ional
s uden s end o achie e highe g ades compa ed han local s uden s in he cou se.
F om he line cha s, i can be concluded ha mos local s uden s go poin s in he
ange o 5 o 8, while mos in e na ional s uden s go poin s in he ange o 9 o 10.
When ocusing on only one speci ic g oup, i can be concluded ha he in e na ional
s uden s’ pe o mance is in he same end no ma e which assessmen me hod is
used. Howe e , as o local s uden s, he e is an ob ious d op in he equency o
high poin s.
Based on he in o mal discussions wi h s uden s and eache s, h ee main easons
may explain he phenomenon. Fi s , in e na ional s uden s usually pay ui ion ees,
and schola ships a e o en based on pe o mance in cou ses while o he local
s uden s he educa ion is ee. Second, in e na ional s uden s o en seek high g ades
o applica ions o o he uni e si ies o coun ies, whe eas local s uden s usually
con inue hei mas e ’s s udies in he same uni e si y wi hou g ade equi emen s.
Thi d, in e na ional s uden s may ha e ewe social connec ions in a o eign coun y
compa ed o local s uden s; i in e na ional s uden s wan o in es in hemsel es o
he u u e, s udying and ge ing highe g ades is one o he op ions ha a e le . As
o local s uden s, due o cul u al di e ences, hey may ealize ha i hey a e going
o wo k in a company a e g adua ion, g ades a e no he only aspec ha a
company will conside . Some imes, a g ade o 3.5 migh be e en be e han a g ade
o 5, because i may imply ha s uden s ha e a good social li e and social skills, ha
s uden s a e c ea i e and do no always ollow s ic ins uc ions gi en by he
eache s, which a e seen as me i s by some companies. Howe e , in e na ional
s uden s o m a e y di e se g oup, including indi iduals om many di e en
backg ounds, and u he esea ch is needed o be e unde s and his beha io .
3.2 Teache ’s Insigh s
In he academic yea 2023-2024, pee e iew e alua ion was used o indi idual
exe cises 1 o 3. In con as , in he academic yea 2024-2025, hese exe cises we e
e alua ed manually by a single eache o ensu e ai ness and quali y. Du ing he
cou se, eache s wan ed o assess wo main aspec s: ime and quali y.
Fig 2. Dis ibu ion o s uden s’ pe o mances o exe cise 1 in 2023 (a) and 2024 (b), exe cise
2 in 2023 (c) and 2024 (d), and exe cise 3 in 2023 (e) and 2024 ( ).
Rega ding ime equi emen s, e alua ing each indi idual exe cise akes
app oxima ely h ee minu es pe s uden , he o al ime spen on each exe cise is
abou 30 hou s. When pee e iew is used, eache s only check a sample o
submissions, and he o al ime spen pe exe cise is educed o a ound 1 hou . Only
a small numbe o g ades, a ound 3%, we e adjus ed ac oss all indi idual exe cises,
when pee e iew was used. The mos common eason o hese adjus men s was
he absence o pee e alua ions, which equi ed he eache o e alua e he exe cise
and assign a g ade.
The obse a ion ha in e na ional s uden s a e mo e g ade-o ien ed and mo i a ed is
suppo ed by educa ional esea ch add essing whe he his s ems om be e
p epa a ion o highe mo i a ion. While p io p epa a ion is a ac o , s udies con i m a
"sel -selec ion e ec " whe e in e na ional s uden s o en exhibi highe sel -
de e mined mo i a ion and a s onge g ade-o ien a ion han domes ic pee s (Chue
(a)
(b)
(c)
(d)
(e)
( )
& Nie, 2016). In e ms o quali y, manual assessmen ends o be mo e bene icial o
bo h s uden s and eache s. Fo s uden s, g ades a e pe cei ed as mo e eliable
when gi en by eache s a he han pee s. Fo eache s, manually e iewing he
exe cises allows hem o iden i y he common mis akes, which can help in imp o ing
cou se ma e ials immedia ely o o he nex academic yea . Du ing discussions,
eache s eel mo e con iden , a e be e p epa ed o answe ques ions and can
p o ide mo e e ec i e guidance. On he o he hand, pee e iew enables s uden s o
lea n om e alua ing o he s’ wo k and ecognize hei own mis akes.
4 CONCLUSIONS
Analysis o quan i a i e da a and quali a i e eedback indica es ha pee e iew is a
iable assessmen me hod, demons a ing minimal impac on s uden s' inal g ades
despi e po en ial ade-o s in assessmen ideli y. Da a analysis con i ms ha pee
e iew signi ican ly educes eache ime equi emen s (app oxima ely 1h s s. 30h s
pe exe cise) compa ed o manual g ading. Wi h he esul s collec ed in he pas wo
academic yea s, eache s a e con iden ha pee - e iew assessmen can be used,
and hey a e hinking o o a ing di e en e alua ion me hods in di e en academic
yea s o ensu e cou se quali y main enance.
The compa ison showed di e ences be ween he me hods. Manual assessmen in
2024 enabled ine g ading and he cap u e o de ails like c ea i i y, leading o lowe
a e age sco es bu highe a ia ion. Pee e iew in 2023 was e icien and showed
less sco e a ia ion, p esen ing a ade-o be ween e iciency/s uden lea ning and
de ailed g ading consis ency. Manual g ading also p o ides di ec eedback o
ins uc o s abou common mis akes, helping o imp o e cou se ma e ials.
An in e es ing and pe sis en phenomenon was no iced: local s uden s a e less
mo i a ed o pu sue highe g ades compa ed o in e na ional s uden s, ega dless o
he assessmen me hod. F om he line cha s and discussions, i seems in e na ional
s uden s aim o op sco es possibly due o ac o s like ees, schola ships, o mobili y
goals, while local s uden s' pe o mance clus e s in he middle anges, pe haps
e lec ing di e en p io i ies. Howe e , u he esea ch is needed o be e
unde s and his beha io .
Since he exe cises in ol e 3D modelling a he han ma hema ics o physics
calcula ion, i is di icul o a sys em o e alua e hen au oma ically. The e a e some
s udies abou his opic by Sanna e al., (2012), bu implemen ing hem equi es
some speci ic skills om eache s. Howe e , he eache s a e ying o ind a eliable
solu ion o see i he e is any a i icial in elligence ool ha can be used in he
e alua ion; o example, a ce ain numbe o exe cises a e e alua ed in de ail by
eache s i s ; hen hese da a will be used in aining; and he ained model can be
used o e alua e he es o he submissions.
The indings o e p ac ical guidance o enginee ing educa o s managing la ge-scale
cou ses, while also highligh ing impo an conside a ions o di e se s uden
popula ions in in e na ional educa ional se ings. Fu u e esea ch should in es iga e
he op imal combina ion o pee e iew and manual assessmen o maximize bo h
e iciency and educa ional e ec i eness.
