Digital rubric-based assessment of oral presentation competence with technological resources for preservice teachers

21 ESTUDIOS SOBRE EDUCACIÓN / 2022 Digi al ub ic-based assessmen o o al p esen a ion compe ence wi h echnological esou ces o p ese ice eache s E aluación basada en úb icas digi ales de la compe encia de p esen ación o al con ecu sos ecnológicos pa a p o eso ado en o mación ANA BELÉN PÉREZ-TORREGROSA Depa amen o de Didác ica y O ganización Escola Facul ad de Ciencias de la Educación Uni e sidad de Málaga Campus de Tea inos, 29010, Málaga (España) [email p o ec ed] h ps://o cid.o g/0000-0002-3428-6348 MARÍA JESÚS GALLEGO-ARRUFAT Depa amen o de Didác ica y O ganización Escola Facul ad de Ciencias de la Educación Uni e sidad de G anada Campus Uni e si a io de Ca uja, 18071, G anada (España) mgallego@ug .es h ps://o cid.o g/0000-0002-2296-5431 MANUEL CEBRIÁN-DE-LA-SERNA Depa amen o de Didác ica y O ganización Escola Facul ad de Ciencias de la Educación Uni e sidad de Málaga Campus de Tea inos, 29010, Málaga (España) [email p o ec ed] h ps://o cid.o g/0000-0002-0246-7398 ISSN: 1578-7001 / DOI: 10.15581/004.43.009 Resumen: Es e es udio es á di igido a la e-e aluación de la compe encia de p esen ación o al u ilizando e- cu sos ecnológicos en un modelo ac i o que combi- na ap endizaje basado en p oyec os y aula in e ida. Es e es udio aplica el uso de una úb ica digi al pa a la e aluación de la compe encia en p esen ación o al en di e en es si uaciones de e aluación ac i a y p og esi- a de 99 u u os docen es en las que la pa icipación es opcional u obliga o ia. Los esul ados mues an que la úb ica digi al empleada en a ios momen os es una me odología y una ecnología que acili a el p oceso de e oalimen ación y diálogo en e docen- es y es udian es sob e los c i e ios de e aluación. Los esul ados apoyan u u as decisiones de diseño me odológico de e aluación o ma i a ap opiadas en en o nos de ap endizaje online. Palab as cla e: Rúb ica, Compe encia de P esen a- ción O al, E aluación, Fu u os Docen es. PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 22 ESTUDIOS SOBRE EDUCACIÓN / 2022 Abs ac : This s udy ocuses on e-assessmen o o al p esen a ion compe ence using echnology esou ces in a model ha combines p ojec -based lea ning and fl ipped lea ning. This s udy uses a digi al ub ic o assess o al p esen a ion compe ence in di e en si ua ions o p og essi e assessmen o 99 p ese ice eache s, si ua ions in which pa icipa ion was ei he op ional o compulso y. Findings show ha he digi al ub ic used a a ious imes is a me hodology and a echnology ha acili a es he eedback p ocess and dialogue be ween eache s and s uden s abou he assessmen c i e ia. The esul s suppo u u e deci- sions o me hodological design o o ma i e assess- men app op ia e o online lea ning en i onmen s. Keywo ds: Rub ic, O al p esen a ion compe ence, Assessmen , P ese ice eache s. INTRODUCTION D e eloping o al p esen a ion compe ence is an essen ial goal o highe edu- ca ion, as his compe ence is c ucial o u u e pe sonal and p o essional success (Dunba e al., 2006; Mo eale e al., 2017; an Ginkel e al., 2015). O al p esen a ion compe ence is ecognized as a cen al p o essional compe ence (Campbell e al., 2001). S udying o al p esen a ion compe ence is impo an be- cause i helps o de elop and imp o e skills ha a e necessa y o many aspec s o p o essional ca ee s. The concep o o al p esen a ion compe ence is g ounded in he in e ela ion o he cogni i e, beha io al, and a ec i e domains (Bowe e al., 2011; De G ez e al., 2009b; an Ginkel e al., 2017). These h ee domains a e b ough o bea on speaking in public in o de o in o m, exp ess onesel , and con- nec wi h o he s in a e o mula ion o “compe ence-based eache aining” (and i s goal o aining in specifi c eaching skills) (Mulde , 2014). O al p esen a ion compe ence includes app op ia e con en , design, and o - ganiza ion o he p esen a ion, as well as p esen a ion skills (Ri chie, 2016). These mus be de eloped in conjunc ion wi h digi al compe ence (Redecke , 2017). We canno sepa a e p og ess in acqui ing and imp o ing o al and digi al skills because he wo oge he o m he co e o he eaching p o ession. The eaching p o ession places g ea emphasis on he impo ance o o al p esen a ion skills, al hough ew s udies in educa ion esea ch ocus on imp o - ing p ese ice eache s’ o al p esen a ion compe ence by assessing i . Technology de elopmen has clea ly changed bo h he lea ning en i onmen and emphasis on use o suppo esou ces, linking o al p esen a ion and digi al compe ences mo e closely, while he Co id-19 pandemic has made i necessa y o use hese compe- ences in online en i onmen s. Pee and eache assessmen The expansion o lea ning o unde g adua es has gene a ed s udies o con inu- ous and p og essi e assessmen wi h digi al ub ics in enginee ing, medicine, DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 23 ESTUDIOS SOBRE EDUCACIÓN / 2022 an h opology, educa ion, and ma hema ics (among o he subjec s) (Ceb ián-de- la-Se na and Be gman 2014; C aw o d e al., 2020; Haans ad, 2020; Hous on and Thompson, 2017; Zheng e al., 2019). Such assessmen is especially impo an in he p ese ice aining o p ese ice eache s, due o he p o ession’s exe cise o he double ole o assesso and assessed and hei e olu ion (Pa hak and Le Vasan, 2015), as well as he impo ance o o al p esen a ion compe ence (Mulde , 2014; Pa hak and Le Vasan, 2015; an Ginkel e al., 2017). The digi al ub ic has a specifi c me hodology o compe ence-based assessmen and is a ool o such assessmen ha acili a es he eedback p ocess and dialogue on he assessmen c i e ia, con e s con inuous assessmen in o o ma i e (Hous on and Thompson, 2017; No d um e al., 2013). The ub ic enables clea o ien a ion o he academic asks, o he eaching and assessmen o o al p esen a ions on academic p ojec s and simula ed p o essional scena ios (De G ez e al., 2012; Gal án-Sánchez e al., 2017). This s udy ocuses on de eloping o al p esen a ion compe ence, which is key in he ea ly p ese ice eaching p o ession and h oughou li e. The de elopmen occu s h ough in e en ion using a s a egy o con inuous and p og essi e assess- men by eache s and pee s. A me a-analysis by Falchiko and Goldfi nch (2000) disco e ed ha he highes co ela ions be ween sco es gi en by classma es and by he eache occu ed when global judgemen s based on well-defi ned c i e ia a he han indi idual dimensions sco ing c i e ia we e used. Al hough pee sco es do no di e conside ably om eache sco es (Mulde , 2014). Gi en he wide use o pee -assessmen , especially in highe educa ion, he ela i e p ecision o sco ing by classma es s. he ins uc o is an impo an con- ce n o bo h educa o s and esea che s. This conce n has g own wi h he inc ease in pee -assessmen in online en i onmen s. Li e al. (2016) p o ide a syn hesis o esea ch fi ndings on pee -assessmen since 1999, when compu e -assis ed pee -as- sessmen began o be used. Adop ing a hie a chical linea modeling app oach, hey ound ha he a e age es ima ed Pea son co ela ion be ween sco es by classma es and ins uc o s was .63, a mode a ely s ong alue. This co ela ion is signifi can ly highe when pee -assessmen is op ional and no compulso y (among o he ac- o s), when i is no anonymous, when pee -assessmen p o ides bo h sco es and quali a i e commen s (no jus sco es), and when pee assesso s pa icipa e in de- elopmen o he sco ing c i e ia. Assessmen d i es he ac i i ies in which he s uden s pa icipa e. Because hese ac i i ies suppo s uden s’ lea ning, ca e ul design o a combined assessmen s a egy (including pee -assessmen ) gua an ees ha s uden s a e in ol ed in he associa ed lea ning esou ces (Meye s and Nul y 2009). PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 24 ESTUDIOS SOBRE EDUCACIÓN / 2022 Rub ic o assessmen The digi al ub ic p o ides he oppo uni y o o ma i e assessmen o s uden s’ o al p esen a ion, as i is an ins umen ha explains he c i e ia and expec a ions. I also acili a es eedback p ocesses ( an Ginkel e al., 2017). In aining e iewe s o use he ub ic, ime should be dedica ed o wa ching he p esen a ions and hen discussing how o assess hem o each compe ence. Such p epa a ion p omo es g ea e consis ency in indi idual sco ing among asses- so s (Dunba e al., 2006). The ub ics can hus be a ool ha infl uences e o s o con e con inuous assessmen in o o ma i e. Digi al ub ics a e used o assess o al p esen a ion compe ence in nume - ous fi elds –enginee ing, he sciences, conse a ion- es o a ion and design, among o he s (Delgado and Fonseca-Mo a, 2010; Menéndez-Va ela and G ego i-Gi al , 2016; Pa hak and Le Vasan, 2015). I is impo an ha s uden s p o ide eedback in o ma ion wi h he ub ic –no jus sco es bu also commen s. In his way, ub ics can be a ool o a icula ing eedback in o ma ion (No d um e al., 2013). P og essi e assessmen A single session o pee -assessmen wi h a ub ic is no enough o gene alize abou any imp o emen in his compe ence. The e iew by Zheng e al. (2019) showed ha mos s udies g ounded in “Ac i i y heo y” implemen only one ound o pee - assessmen and do no ewa d assesso s ( o example, poin s in he subjec a ings o ex a poin s). When a emp s ha e been made o in eg a e o ma i e assessmen o lea ning wi h summa i e assessmen o lea ning, o mulas such as epe i ion o he measu es ha e been p oposed (Hous on and Thompson, 2017). This s udy is based on h ee main cha ac e is ics o assessmen : – Con inui y, which gua an ees mo e objec i i y in he assessmen , since e- pea ing assessmen s enables con as o he esul s h oughou he o ma i e p ocess and moni o ing lea ning in o al compe ence. – Co-pa icipa ion, which is g ounded in in ol emen o he s uden s hem- sel es in sel -assessmen and assessmen o hei pee s using he same c i e- ia wi h which he eache will assess hem. This p ocedu e makes s uden s awa e o hei eal o ma i e le el, enabling hem o assume ull esponsibil- i y o di ec ing hei own lea ning. I also inc eases he le el o eliabili y (De G ez e al., 2009a; De G ez e al., 2012; Magin and Helmo e, 2001; Sko hol e al., 2019). – P og essi e assessmen , ueled by he g ades ha can be assigned, acco ding DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 25 ESTUDIOS SOBRE EDUCACIÓN / 2022 o he s uden him-o he sel and p o isionally, especially o ac i i ies whose pu pose is lea ning and assessmen . Du ing p og essi e assessmen , s uden s ha e he possibili y o modi ying hose g ades acco ding o he p og ess achie ed in a nego ia ion be ween s uden and eache (Mad uga e al., 2020). O al p esen a ion compe ence is a goal o uni e si y educa ion and p o essional success. I combines he impo ance o he eaching p o ession o o ali y, exp essi e na a ion, and dialogue among pee s. In his s udy, i is impo an o esea ch o al p esen a ion compe ence, oge he wi h p og essi e assessmen . O al compe ence is e y impo an o any eache in any eaching mode because i is hei ool o communica ion and dialogue o build knowledge wi h he s uden , being especially impo an in he domain o dis ance communica ion wi h he use o echnology. The s udy ocuses on assessing o al p esen a ion compe ence when s uden s de end p ojec s publicly and o ally ha p omo e pa icipa ion and he use o qual- i y echnological esou ces (digi al compe ence). S a ing om hese p emises, he goals o his s udy a e: Gene al goal. To unde s and he ela ionships be ween assessmen s by s u- den s and eache s h ough expe imen a ion using di e en assessmen s a egies ( wo ypes o pa icipa ion, “op ional/compulso y”, S a egies A and B) wi h u- b ics o encou age he lea ning o o al p esen a ion compe ence. Specifi c goals 1. To desc ibe and compa e he ends in sco es om pee and eache as- sessmen s du ing p og essi e assessmen o o al p esen a ion compe ence, depending on who pe o ms he assessmen a wha ime in he lea ning p ocess. 2. To explo e he p og essi e assessmen me hodology o o al p esen a ion compe ence, depending on who pe o ms he assessmen (pee and each- e ), he ime o he assessmen (ini ial, in e media e, and fi nal), and he pa - icipa ion s a egy (A and B). 3. To de e mine he ela ionship o he gene al g ade ob ained in he subjec o he ub ic sco e o o al p esen a ion compe ence depending on who pe o ms he assessmen and he ime o he assessmen , especially o use and quali y o he echnological esou ces used. 4. To e i y he alue o he di e en s a egies, pe sons, and imes o as- sessmen o o al p esen a ion compe ence o p edic ing fi nal g ades in he subjec . PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 26 ESTUDIOS SOBRE EDUCACIÓN / 2022 METHOD Pa icipan s The sample is a non-p obabili y con enience sample (Cohen e al., 2011) com- posed o n = 99 unde g adua es om he sou h o Spain. Du ing one semes e wo na u al g oups o subjec s in Educa ional Technology (EdTech) p epa ed o al p es- en a ions in s able wo k eams o 4-5 membe s (wi h an occasional excep ion, mini- mum 3 and maximum 6 membe s). The s uden s unde wen p og essi e e alua ion (ini ial, in e media e, and fi nal) by pee s and by he eache s using a digi al ub ic. Se ing, ools, and ub ic The g oup (n = 52) ha was equi ed o pa icipa e in he ull assessmen p ocess ollowed S a egy A (compulso y). The c i e ion o inclusion in he s udy was ha ing comple ed he h ee e- ub ic applica ions (elec onic assessmen by pee s and eache ) p og essi ely and in ull, as well as diligen ly a ending. The g oup (n = 47) no equi ed o pe o m p og essi e assessmen s excep o he las p es- en a ion, he sco e on which was decisi e o he g ade, ollowed S a egy B. Bo h g oups ollowed a combined model o p ojec -based lea ning (PBL) and fl ipped lea ning. The gene al me hodology o he subjec s consis ed mainly o an in e ed class oom model –Flipped Class oom–, whe e he ma e ials and esou ces we e p e iously uploaded o he in e ne wi h specifi c ac i i ies ha we e ca ied ou du ing class ime and a he end o he class an e alua ion ub ic was applied. The e was a single p ojec o be de eloped o he whole cou se by each eam, unde he P ojec Based Lea ning model, which se ed as he cen al axis o he wo k o each eam and s uden , whe e in h ee momen s he de elopmen o he wo k was p e- sen ed in public o moni o ing and e alua ion by he eache and pee s. The analy ic ub ic used o assess o al p esen a ion compe ence consis ed o ou assessmen c i e ia wi h indica o s and e idence assigned di e en poin al- ues and h ee o ou le els o achie emen (Table 1). The ub ic is a ailable on h ps://aco a .link/ud3JX5 DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 27 ESTUDIOS SOBRE EDUCACIÓN / 2022 Table 1. Rub ic o assess o al p esen a ion compe ence ASSESSMENT CRITERIA, INDICATORS AND EVIDENCE ACHIEVEMENT LEVELS 1. To p esen su fi cien , ele an , well-s uc u ed con en . 1.1. Quan i y and ele ance o in o ma ion. 1. Shi s in o a di e en opic, wi h concep ual e o s. 1.1.1. P esen s app op ia e, p ecise in o ma ion om a schola ly pe spec i e. 2. Focuses on seconda y issues, missing some impo an ones (mo e han 2 issues igno ed). 3. Focuses on he main issues bu o ge s 1-2 o hem. 4. T ea s all o he mos impo an elemen s on he opic well and comp ehensi ely. 1.2. O ganiza ion o message. 1. Diso ganized message, jumps om one opic o ano he . Canno be ollowed. 1.2.1. P esen s an o ganized message. 2. Lacks clea o ganiza ion, is di fi cul o ollow. 3. Is o ganized bu is no e y cohe en . 4. Is clea ly, cohe en ly o ganized (no con using). 2. To con ey a message confi den ly o an audience. 2.1. Fo mal quali ies o he p esen a ion. 1. Does no manage o fi nish he p esen a ion and does no allow ime o commen s and ques ions. Finishes e y ea ly. 2.1.1. Adap a ion o ime and o he equi emen s o he p esen a ion. 2. Adap s p esen a ion o he s ipula ed ime and commen s/ques ions bu does no allow enough ime o ques ions. 3. Adap s o he s ipula ed ime and lea es enough ime o commen s/ques ions. 2.2. Body language and audience appo . 1. Is s i , wi h ne ous uncon olled ges u es. Does no mo e a ound. 2.2.1. Mo es a ound in a confi den , elaxed manne . 2. Is s i bu con ols ne es. Does no mo e a ound. 3. Is some imes elaxed. Mo es a ound some. 4. Appea s elaxed and mo es a ound eely wi h g ace and composu e. 2.3. O al exp ession. 1. Unin elligible. Volume o oice is e y low, speaks in a mono one. Poo in ona ion. 2.3.1. Voice is always clea and has good in ona ion. 2. Is ha d o unde s and. Low olume o oice wi h poo in ona ion and une en one. 3. Fai ly in elligible. No mal olume o oice and/o good in ona ion. 4. Easy o unde s and. No mal olume o oice, speaks clea ly wi h good in ona ion. [CONTINÚA EN LA PÁGINA SIGUIENTE] PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 28 ESTUDIOS SOBRE EDUCACIÓN / 2022 ASSESSMENT CRITERIA, INDICATORS AND EVIDENCE ACHIEVEMENT LEVELS 3. To adap o he audience (gene al public o ibunal) by es ablishing fl uid in e ac ion. 3.1. Mo i a es and cap u es in e es o he audience o ibunal.1. Does no awaken in e es o encou age pa icipa ion. 3.1.1. Encou ages in e es and pa icipa ion o he audience o ibunal. 2. Awakens some in e es bu does no encou age pa icipa ion. 3. Awakens some in e es and some imes encou ages pa icipa ion. 4. Awakens in e es , encou ages pa icipa ion, and ge s audience o pa icipa e. 3.2. Managemen and con ol o audience o ibunal. 1. Answe s shi in o wha he speake knows, wi h dig essions. Does no know how o eac o unexpec ed si ua ions. 3.2.1. Handling o ques ions and answe s, commen s om he audience, and unexpec ed si ua ions. 2. Shows unce ain y in answe ing ques ions. Does no know how o eac o unexpec ed si ua ions. 3. Answe s ques ions well bu akes a long ime. Hesi a es and s u e s when aced wi h unexpec ed si ua ions. 4. Answe s ques ions co ec ly and c edibly. Handles unexpec ed si ua ions well. 4. To be able o use high-quali y echnology esou ces in eg a ed in o he message. 4.1. Deg ee o assu ance in handling esou ces. 1. Reads he whole alk om w i en no es and ha dly makes eye con ac wi h audience. 4.1.1. Uses echnological esou ces o p esen a ions o suppo speech and message. 2. O en eads om no es o slides and some imes makes eye con ac wi h audience. 3. Depends on slides o anspa encies as well as w i en no es, using he o me as a closed sc ip and es ablishing equen con ac wi h audience. 4. Slides o anspa encies a e in eg al suppo o he p esen a ion and complemen i . 4.2. Quali y o echnological esou ces. 1. Uses (many/ ew) slides wi h a lo o ex and o e loaded ou lines. 4.2.1. P epa es high-quali y echnological esou ces. 2. Uses (many/ ew) slides ha a e nei he a ac i e no clea . 3. Uses an app op ia e numbe o slides, which a e a ac i e bu unclea and inapp op ia e. 4. Uses a good numbe o slides, which a e a ac i e, clea , and app op ia e, as well as o he mul imedia esou ces. Table 1. Rub ic o assess o al p esen a ion compe ence DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 29 ESTUDIOS SOBRE EDUCACIÓN / 2022 The ub ic was hus fl exible. The assessmen c i e ia we e: C1. To p esen su fi cien , ele an , well-s uc u ed con en (he ea e Con en ). C2. To con ey a message confi den ly o an audience (he ea e Message). C3. To adap o he audience (gene al public o ibunal) by es ablishing fl uid in e ac ion (he ea e Adap ing). C4. To be able o use high-quali y echnology esou ces in eg a ed in o he message (he ea e Resou ces). Design and alida ion o ub ic To collec in o ma ion on he p ese ice eache s’ p og ess in de eloping o al p es- en a ion compe ence, an analy ic ub ic was c ea ed on he pla o m Co ub ic. com ha eache and s uden s access o assessmen and eedback. The ub ic was designed and alida ed du ing he 2017-2018 academic yea by expe eache s and esea che s in a ame p ojec on elec onic assessmen o compe ences om eigh uni e si ies in fi e coun ies. The alida ion p ocess has been de eloped in a p e ious s udy and he ub ic’s eliabili y was calcula ed using C onbach’s alpha (α), he alue 0.934 (Fe nández-Medina e al., 2021). We ha e ob ained a alue o α = 0.80 in his s udy. Measu emen s and p ocedu e The p ocedu e is an adap a ion o he 3P model de eloped by Biggs (2003) (Figu e 1), which s esses he p ocess o e-assessmen in p ojec -based lea ning (PBL) + fl ipped lea ning. Following he eam-based lea ning model, he pa icipan s we e dis ibu ed in o 29 eams o wo k on an educa ional echnology p ojec h oughou he semes e , wi h h ee assessmen imes (ini ial, in e media e, and fi nal p oduc ). The p ojec ep esen ed a a iable p opo ion o he o al g ade in he subjec ac- co ding o he uni e si ies in ol ed (10% UGR and 20% UMA). A e he p esen- a ion o he p ojec , s uden s and eache s had some ime o assess each eam using he pla o m. In he ini ial and in e media e e alua ions, s uden s could indica e which indica o s hey did o did no wish o include. PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 36 ESTUDIOS SOBRE EDUCACIÓN / 2022 Table 4. Coe fi cien s o he eg ession models (dependen a iable subjec a ing), aking o al assessmen s o pee and eache assessmen s as p edic o s (Compulso y G oup) Ini ial B ẞp 95% CI To al Pee -.011 -.068 .650 -.057-.036 To al Teache -.006 -.166 .270 -.017-.005 In e media e B ẞp 95% CI To al Pee .001 .006 .967 -.050-.052 To al Teache .016 .400 .013 .004-.029 Final B ẞp 95% CI To al Pee -.027 -.134 .271 -.075-.022 To al Teache .054 .621 <.001 .033-.075 B: Slope; ẞ: S anda dized Slope; p: C i ical le el o signifi cance; 95% CI: 95% Confi dence In e al o B. DISCUSSION AND CONCLUSIONS This s udy has confi med he alue o design, alida ion, and applica ion o a ub ic o o al p esen a ion compe ence (Jönsson and Panade o, 2017; Menéndez-Va ela and G ego i-Gi al , 2016; an Ginkel e al., 2015; an Ginkel e al., 2017). Mo e specifi cally, he ub ic in eg a es o al and digi al compe ences, as i assesses he ca- pabili y o using and ha ing good command o echnological esou ces ( an Ginkel e al., 2017, p. 486), capabili ies included in he assessmen c i e ion “ o be able o use high-quali y echnology esou ces in eg a ed in o he message” (Resou ces). Two indica o s (4.1. and 4.2.) we e used because i is he o m o echnological communica ion pa excellence ha suppo s o al discou se. Because e idence o s uden s’ in ol emen , commi men , and mo i a ion is impo an in he p ac ice o lea ning, we analyzed he impac o an inno a i e in e en ion ha s essed con inui y, co-pa icipa ion, and p og essi e assessmen in he acquisi ion o o al p esen a ion compe ence (De G ez e al., 2009a). The associa ion o his po en ial in ol emen wi h bo h op ional and compulso y pa - icipa ion in he assessmen made i a signifi can a iable in pee -assessmen . Al- hough he esea ch on eedback in he pee -assessmen o o al p esen a ion began as ela i ely limi ed (Topping, 1998), i has since inc eased (Li e al., 2016; Zheng e al., 2019) bu has no been esea ched by i sel . S udies a ely s a e whe he pa icipa ion in pee -assessmen was compulso y o op ional, al hough one could expec his a iable o impac accep abili y and eliabili y (Topping, 1998, p. 268). DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 37 ESTUDIOS SOBRE EDUCACIÓN / 2022 In ou s udy, he esul s o he o al o al p esen a ion compe ence (pee and eache assessmen ) show signifi can di e ences in he assessmen s based on ype o pa - icipa ion (compulso y o op ional) in he assessmen p ocess. When pa icipa ion is compulso y (S a egy A), mean sco es a e highe (Co>Op) in all assessmen s (ini- ial, in e media e and fi nal). The same end appea s when pa icipa ion is op ional (S a egy B), excep only he sco e ob ained o eache on he ini ial assessmen (Co>Op). The e o e, i seems ha he na u e o pa icipa ion, despi e being an impo an me hodological elemen , is no a de e mining ac o in he o al sco es o he ub ic o pee and eache assessmen . Resea ching he e ec o applying digi al ub ics wi h epea ed measu emen s (o en h ee measu emen s) suppo s decoupling assessmen wi h digi al ub ics om summa i e assessmen (Hous on and Thompson, 2017). We conclude ha he p og ession b ings successi e applica ions somewha close o o ma i e assess- men , al hough hese assessmen s a e mo e closely connec ed when eedback is p o ided (Mad uga e al., 2020). The ini ial, in e media e, and fi nal assessmen s wi h eedback by pee s and eache s a e an impo an s ep in he p og ess owa d o ma i e assessmen . In De G ez e al. (2009b) and Menéndez-Va ela and G ego- i-Gi al (2016), achie ing he lea ning ou comes and ob aining e idence o lea n- ing p og ess demons a e he alidi y o he in e ences d awn om he assessmen sys em. Pee -assessmen seems su fi cien ly eliable and alid in a wide a ie y o applica ions, al hough he li e a u e conside s sco e eliabili y a he han mo e de ailed o ma i e assessmen . The 3P model illus a es he infl uence o ac i e e-assessmen o o al p esen a- ion compe ence on he lea ning ou comes, exp essed in he o m o fi nal g ades in he Educa ional Technology subjec s in his s udy. We e i y ha he esul s o gene al g ade in he subjec a e ela ed o he sco e on he o al p esen a ion compe ence ub ic. When we analyze pee -assessmen , we confi m no signifi can ela ion be ween he sco e o he ub ic and subjec a ing, bu in he eache as- sessmen he e is. This sco e in he e- ub ic (pa icula ly a he in e media e and fi nal momen s) is s a is ically signifi can , so con ibu es o g ade in he subjec . Based on he assump ion o a solid ela ionship be ween o al and digi al com- pe ence when assessing o al p esen a ion compe ence, assessmen c i e ion Re- sou ces on he digi al ub ic ocuses on quali y o esou ces and mas e y o ech- nology o he o al p esen a ion. Ou s udy e eals a co ela ion no p e iously s udied in he li e a u e consul ed: he s uden s who ob ain be e sco es on use and quali y o echnology esou ces as suppo o speech and messages ob ain highe sco es on o al p esen a ion compe ence. This co ela ion is signifi can a all h ee e alua ion imes in eache assessmen . I is lowe in pee -assessmen and PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 38 ESTUDIOS SOBRE EDUCACIÓN / 2022 depends on he ime o he assessmen . These fi ndings a e impo an o p ese ice eache s’ p ac ice and will be a de e mining ac o in hei sco es on o al p esen a- ion compe ence, since i is essen ial ha hey use high-quali y esou ces o sup- po hei o al p esen a ion. Ou fi ndings a e simila o he esul s o o he s udies on p ecision o pee -as- sessmen g ades compa ed o eache assessmen g ades (De G ez e al., 2012; Li e al., 2016; Magin and Helmo e, 2001). Wi h he inc easing use o digi al pla o ms –e en g ea e now due o he Co id-19 pandemic– his is an impo an conce n o bo h educa o s and esea che s. Assessmen s by s uden s using ub ics di e om he eache s’ assessmen s, bu successi e applica ions o p og essi e assessmen p oduce close sco es as he numbe o assessmen s pe o med wi h he digi al ub ic inc eases. The me a-analysis by Li e al. (2016) shows ha his co ela ion is signifi can ly g ea e when pee -assessmen is op ional ins ead o compulso y, among o he ac o s. In ou s udy, he o al sco e o o al p esen a ion compe- ence in ini ial assessmen o ICC shows no ag eemen on he o al sco e o o al p esen a ion compe ence and signifi can ag eemen in he in e media e and fi nal assessmen . I is wo h s essing ha c i e ion Resou ces, which combines e idence om he ub ic on quali y and use o echnological esou ces, shows signifi can ag eemen be ween pee and eache assessmen s a ing wi h he fi s assessmen (ini ial assessmen ). In conclusion, his s udy ob ains signifi can in e ac ion e ec s be ween assesso s o all assessmen s, indica ing ha changes in assessmen a di - e en momen s in ime depend on he assesso . Finally, we encoun e some challenges in assessing o al p esen a ion compe- ence, such as challenges associa ed wi h pe o ming his ask in an in-pe son s. online en i onmen . On he one hand, c i e ia Message and Adap ing me i di - e en ia ed analysis as a whole and in each o hese en i onmen s. On he o he , i is necessa y o explo e o mulas o imp o e pa icipa ion in assessmen in blended lea ning en i onmen s wi h combina ions o i ual and in-pe son o al p esen a- ion. Fu u e s udies mus u he de elop assessmen o imp o emen in he qual- i y o he p ojec s as a whole (no only o he o al p esen a ion bu o he p ojec ), unde sco ing he need o combine o al and digi al compe ence h ough eedback, p omo ing assessmen o he p ojec s o imp o e lea ning. I is impo an o s akeholde s in highe educa ion o ecognize he need o gi e s uden s a oice in imp o ing compe encies ha a e especially use ul o employmen (Campbell e al., 2001; Rome o-Ga cía e al., 2020). The s udy confi ms an e ec i e me h- odology, wi h a pla o m and me hodology ha allows eache s o e alua e he impac o hei digi al ub ics as mo e fl exible, mo e obse able and ins an aneous, hus acili a ing be e decision making acco dingly han o he s udies p esen ed. DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 39 ESTUDIOS SOBRE EDUCACIÓN / 2022 Howe e , i emains o ex end he s udy in he u u e compa ed o con ol g oups whe e he ub ic me hodology is no used. Fecha de ecepción del o iginal: 8 de ma zo 2022 Fecha de acep ación de la e sión defi ni i a: 25 de mayo 2022 REFERENCES Biggs, J. B. (2003). Teaching o quali y lea ning a uni e si y: Wha he s uden does. Open Uni e si y P ess. Bowe , M., Ca anagh, M., Moloney, R., and Dao, M. (2011). De eloping com- munica ion compe ence using an online ideo efl ec ion sys em: P e-se ice eache s’ expe iences. Asia-Pacifi c Jou nal o Teache Educa ion, 39(4), 311-326. h ps://doi.o g/10.1080/1359866X.2011.614685 Campbell, K. S., Mo he sbaugh, D. L., B amme , C., and Taylo , T. (2001). Pee e sus sel assessmen o o al business p esen a ion pe o mance. Busi- ness Communica ion Qua e ly, 64(3), 23-40. h ps://doi.o g/10.1177/108056 990106400303 Ceb ián-de-la-Se na, M., and Be gman, M. E. (2014). Fo ma i e Assessmen wi h eRub ics: an App oach o he S a e o he A . REDU. Re is a de Docencia Uni- e si a ia, 12(1). 15-29. Cohen, L. M., Mannion, L. L., and Mo ison, K. (2011). Resea ch me hods in educa- ion. New Yo k. C aw o d, A. R., Johnson, E. S., Zheng, Y. Z., and Moylan, L. A. (2020). De elop- ing an unde s anding p ocedu es obse a ion ub ic o ma hema ics in e - en ion eache s. School Science and Ma hema ics, 120(3), 153-164. h ps://doi. o g/10.1111/ssm.12393 C eswell, J. W., and C eswell, J. D. (2018). Resea ch design: quali a i e, quan i a i e, and mixed me hods app oaches. SAGE. De G ez, L., Valcke, M., and Roozen, I. (2009a). The impac o an inno a i e in- s uc ional in e en ion on he acquisi ion o o al p esen a ion skills in highe educa ion. Compu e s and Educa ion, 53(1), 112-120. h ps://doi.o g/10.1016/j. compedu.2009.01.005 De G ez, L., Valcke, M., and Roozen, I. (2009b). The impac o goal o ien a ion, sel - efl ec ion and pe sonal cha ac e is ics on he acquisi ion o o al p esen a- ion skills. Eu opean Jou nal o Psychology o Educa ion, 24(3), 293-306. h ps:// doi.o g/10.1007/BF03174762 De G ez, L., Valcke, M., and Roozen, I. (2012). How e ec i e a e sel -and pee PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 40 ESTUDIOS SOBRE EDUCACIÓN / 2022 assessmen o o al p esen a ion skills compa ed wi h eache s’ assessmen s? Ac i e Lea ning in Highe Educa ion, 13(2), 129-142. h ps://doi.o g/10.1177/ 1469787412441284 Delgado, M. A., and Fonseca-Mo a, M. C. (2010). The use o co-ope a i e wo k and ub ics o de elop compe ences. Educa ion o Chemical Enginee s, 5(3), e33-e39. h ps://doi.o g/10.1016/j.ece.2010.05.002 Dunba , N. E., B ooks, C. F., and Kubicka-Mille , T. (2006). O al communica- ion skills in highe educa ion: Using a pe o mance-based e alua ion ub ic o assess communica ion skills. Inno a i e Highe Educa ion, 31(2), 115-128. h ps://doi.o g/10.1007/s10755-006-9012-x Falchiko , N., and Goldfi nch, J. (2000). S uden pee assessmen in highe educa- ion: A me a-analysis compa ing pee and eache ma ks. Re iew o Educa ional Resea ch, 70(3), 287-322. h ps://doi.o g/10.3102/00346543070003287 Fe nández-Medina, C., Luque-Gue e o, C., Ruiz-Rey, F., Ri e a-Rogel, D., And ade Va gas, L., and Ceb ián-de-la-Se na, M. (2021). Assessmen o al compe ence wi h digi al ub ics o he Ibe o-Ame ican Knowledge Space. Pixel-Bi . Re is a de Medios y Educación, 62, 71-106. h ps://doi.o g/10.12795/ pixelbi .83050. Gal án-Sánchez, I., Ve ano-Taco on e, D., González-Be anco , S. M., Fe nán- dez-Mon oy, M., and Bolí a -C uz, A. (2017). Assessing o al p esen a ion skills in Elec ical Enginee ing: De eloping a alid and eliable ub ic. In e - na ional Jou nal o Elec ical Enginee ing Educa ion, 54(1), 17-34. h ps://doi. o g/10.1177/0020720916659501 Haans ad, E. J. (2020). Collabo a i e E hnog aphic Assessmen : An An h opo- logical Rub ic o a Communi y Ecosys em. Teaching An h opology, 9(2), 1-8. h ps://doi.o g/10.22582/ a. 9i2.528 Hous on, D., and Thompson, J. N. (2017). Blending Fo ma i e and Summa i e Assessmen in a Caps one Subjec :‘I ’s no you ools, i ’s how you use hem’. Jou nal o Uni e si y Teaching and Lea ning P ac ice, 14(3), 2. Jönsson, A., and Panade o, E. (2017). The use and design o ub ics o suppo assessmen o lea ning. In D. Ca less, S.M. B idges, C.K.Y. Chan and R. Glo cheski R (Eds.), Scaling up assessmen o lea ning in highe educa ion (pp. 99-111). Sp inge . Li, H., Xiong, Y., Zang, X., L. Ko nhabe , M., Lyu, Y., Chung, K. S., and K. Suen, H. (2016). Pee assessmen in he digi al age: A me a-analysis compa ing pee and eache a ings. Assessmen and E alua ion in Highe Educa ion, 41(2), 245- 264. h ps://doi.o g/10.1080/02602938.2014.999746 DIGITAL RUBRIC-BASED ASSESSMENT OF ORAL PRESENTATION COMPETENCE 41 ESTUDIOS SOBRE EDUCACIÓN / 2022 Mad uga, T., Thiam, S., Vasquez, A., Ko ha i, R., and K auss, G. G. (2020). Re- iewe pe spec i e impac on design e iew eedback. The In e na ional Jou - nal o Enginee ing Educa ion, 36(2), 675-686. Magin, D., and Helmo e, P. (2001). Pee and eache assessmen s o o al p esen a- ion skills: how eliable a e hey? S udies in Highe Educa ion, 26(3), 287-298. h ps://doi.o g/10.1080/03075070120076264 Menéndez-Va ela, J. L., and G ego i-Gi al , E. (2016). The con ibu ion o u- b ics o he alidi y o pe o mance assessmen : a s udy o he conse a ion- es o a ion and design unde g adua e deg ees. Assessmen and E alua ion in Highe Educa ion, 41(2), 228-244. h ps://doi.o g/10.1080/02602938.2014.99 8169 Meye s, N. M., and Nul y, D. D. (2009). How o use (fi e) cu iculum design p inciples o align au hen ic lea ning en i onmen s, assessmen , s uden s’ ap- p oaches o hinking and lea ning ou comes. Assessmen and E alua ion in High- e Educa ion, 34(5), 565-577. h ps://doi.o g/10.1080/02602930802226502 Mo eale, S. P., Valenzano, J. M., and Baue , J. A. (2017). Why communica ion educa ion is impo an : A hi d s udy on he cen ali y o he discipline’s con- en and pedagogy. Communica ion Educa ion, 66(4), 402-422. h ps://doi.o g/ 10.1080/03634523.2016.1265136 Mon al ão, D., and Bake , T. (2015). Co ela ing pee and u o assessmen on a low- s akes enginee ing assignmen . In e na ional Jou nal o Mechanical Enginee ing Educa ion, 43(3), 168-179. h ps://doi.o g/10.1177/0306419015603008 Mulde , M. (2014). Concep ions o p o essional compe ence. In S. Bille , C. Ha - eis, and H. G ube (Eds.), In e na ional handbook o esea ch in p o essional and p ac ice-based lea ning (pp. 107-137). Sp inge . No d um, L., E ans, K., and Gus a sson, M. (2013). Compa ing s uden lea ning expe iences o in- ex commen a y and ub ic-a icula ed eedback: s a egies o o ma i e assessmen . Assessmen and E alua ion in Highe Educa ion, 38(8), 919-940. h ps://doi.o g/10.1080/02602938.2012.758229 Pa hak, A., and Le Vasan, M. (2015). De eloping O al P esen a ion Compe ence in P o essional Con ex s: A Design-Based Collabo a i e App oach. In e na- ional Jou nal o E alua ion and Resea ch in Educa ion, 4(4), 179-184. h p:// dx.doi.o g/10.11591/ije e. 4i4.4509 Redecke , C. (2017). Eu opean amewo k o he digi al compe ence o educa o s: Dig- CompEdu. Repo o he Eu opean Union no. JRC107466. Join Resea ch Cen e. Rome o-Ga cía, C., Buzón-Ga cía, O., Sac is án-San-C is óbal, M., and Na a o- Asencio, E. (2020). E alua ion o a P og am o he Imp o emen o Lea ning PÉREZ-TORREGROSA / GALLEGO-ARRUFAT / CEBRIÁN-DE-LA-SERNA 42 ESTUDIOS SOBRE EDUCACIÓN / 2022 and Digi al Compe ence in Fu u e Teache s U ilizing Ac i e Me hodologies. Es udios sob e Educación, 39, 179-205. h ps://doi.o g/10.15581/004.39.179-205 Sko hol , K., No dens öm, E., and S okoe, E. (2019). E alua i e conduc in each- e –s uden supe ision: When s uden s assess hei own pe o mance. Lin- guis ics and Educa ion, 50, 46-55. h ps://doi.o g/10.1016/j.linged.2019.03.001 an Ginkel, S., Gulike s, J., Biemans, H., and Mulde , M. (2015). Towa ds a se o design p inciples o de eloping o al p esen a ion compe ence: A syn hesis o esea ch in highe educa ion. Educa ional Resea ch Re iew, 14, 62-80. h ps:// doi.o g/10.1016/j.edu e .2015.02.002 an Ginkel, S., Gulike s, J., Biemans, H., and Mulde , M. (2017). The impac o he eedback sou ce on de eloping o al p esen a ion compe ence. S udies in Highe Educa ion, 42(9), 1671-1685. h ps://doi.o g/10.1080/03075079.2015. 1117064 an Ginkel, S., Lau en zen, R., Mulde , M., Mononen, A., Ky ä, J., and Ko elain- en, M. J. (2017). Assessing o al p esen a ion pe o mance: Designing a ub ic and es ing i s alidi y wi h an expe g oup. Jou nal o Applied Resea ch in High- e Educa ion, 9(3), 474-486. h ps://doi.o g/10.1108/JARHE-02-2016-0012 Zheng, L., Chen, N. S., Cui, P., and Zhang, X. (2019). A Sys ema ic Re iew o Technology-Suppo ed Pee Assessmen Resea ch: An Ac i i y Theo y Ap- p oach. In e na ional Re iew o Resea ch in Open and Dis ibu ed Lea ning, 20(5), 168-191. h ps://doi.o g/10.19173/i odl. 20i5.4333