ABSTRACT
A Scale o E alua ing he Me hodological Quali y
o S udies Based on Obse a ional Me hodology
Susana Sandu e e-Cha es1, Daniel López-A enas1, M. Te esa Angue a2 and Sal ado Chacón-Moscoso1,3
1 Uni e sidad de Se illa (Spain)
2 Uni e sidad de Ba celona (Spain)
3 Uni e sidad Au ónoma de Chile (Chile)
An eceden es: Has a donde conocemos, hay escasa e idencia de alidez de escalas pa a medi calidad me odológica
de es udios basados en me odología obse acional (EBMO). Se p esen an e idencias de alidez de la Escala de Calidad
Me odológica (MQSOM) pa a EBMO en base a su es uc u a in e na. Mé odo: Se aplicó MQSOM a 650 a ículos que
emplea on me odología obse acional. Se calculó el coe icien e de co elación in aclase (CCI) pa a la iabilidad in e
e in acodi icado es. Se ealizó un análisis pa alelo median e implemen ación óp ima pa a es udia su dimensionalidad.
Finalmen e, se ealizó un análisis ac o ial explo a o io con media mues a ob enida alea o iamen e, seguido de un
análisis ac o ial con i ma o io con la o a mi ad. Resul ados: La iabilidad in e e in a codi icado ue on adecuadas
(CCI > ,73). El análisis pa alelo sugi ió mul idimensionalidad (UniCo = .41, ECV = .31). Se ob u o una es uc u a
ac o ial de segundo o den ( ac o gene al: Calidad me odológica) con dos ac o es de p ime o den (F1 Diseño y F2
Medición y análisis), RMSEA = 0.000, NNFI = 1, GFI = .98, AGFI = .97, con adecuados esul ados en iabilidad y
disc iminación. Conclusiones: MQSOM es un ins umen o b e e (11 í ems), ú il pa a p o esionales de la in e ención,
in es igado es o comisiones de alo ación, pa a diseña , implemen a o e alua EBMO.
Keywo ds:
Me hodological quali y
Scale
S udies based on obse a ional
me hodology
Validi y e idence
Reliabili y
Palab as cla e:
Calidad me odológica
Escala
Es udios basados en me odología
obse acional
E idencias de alidez
Fiabilidad
Recei ed: May 16, 2024
Accep ed: July 19, 2024
ARTICLE INFO
Una Escala Pa a la E aluación de la Calidad Me odológica de Es udios Basados en
Me odología Obse acional
Ci e as: Sandu e e-Cha es, S., López-A enas, D., Angue a, M. T., & Chacón-Moscoso, S. (2025). A scale o e alua ing he me hodological quali y o s udies based on
obse a ional me hodology. Psico hema, 37(1), 1-10. h ps://doi.o g/10.70478/psico hema.2025.37.01
Co esponding au ho : Sal ado Chacón-Moscoso, [email p o ec ed]
This a icle is published unde C ea i e Commons License 4.0 CC-BY-NC
A icle
Psico hema (2025) 37(1) 1-10
Colegio O icial de Psicología del P incipado de As u ias
h ps://www.psico hema.com/es • ISSN 0214–9915 • eISSN 1886-144X
Psico hema
RESUMEN
Backg ound: To da e, no s udies ha e shown alidi y e idence o a scale ha measu es he me hodological quali y
o s udies based on obse a ional me hodology (SBOM). This s udy p esen s alidi y e idence o he Me hodological
Quali y Scale o S udies based on Obse a ional Me hodology (MQSOM) based on i s in e nal s uc u e. Me hod:
MQSOM was applied o 650 jou nal a icles ha used obse a ional me hodology. The In aclass Co ela ion
Coe icien (ICC) was calcula ed o ob ain e idence o in e - and in acode eliabili y. Pa allel analysis was done using
op imal implemen a ion o s udy he dimensionali y o he scale be o e conduc ing an explo a o y ac o analysis wi h
a andomly-selec ed hal o he sample, ollowed by a con i ma o y ac o analysis wi h he emaining hal . Resul s:
Bo h in e - and in a-code eliabili y we e adequa e, ICC > .73. Pa allel analysis sugges ed a lack o unidimensionali y,
UniCo = .41; ECV = .31. A second-o de ac o s uc u e (gene al ac o : Me hodological quali y) wi h wo i s -o de
ac o s (F1 Design, F2 Measu emen and Analysis) was ob ained, RMSEA = 0.000, NNFI = 1, GFI = .98, AGFI = .97,
wi h adequa e eliabili y and disc imina ion esul s. Conclusions: MQSOM is a sho (11 i ems), use ul ins umen o
p o essionals, esea che s o assessmen commissions when designing, implemen ing, o e alua ing SBOM.
2
Sandu e e-Cha es e al. / Psico hema (2025) 37(1) 1-10
Obse a ional me hodology allows he spon aneous beha io s
o pa icipan s in na u al si ua ions o be eco ded and hen
quan i ied (Angue a e al., 2020). This me hodology in ol es an
ini ial phase based on na u alis ic obse a ion and a second phase
in ol ing a quan i a i e analysis o pa icipan da a. The inal s age
o obse a ional me hodology includes quali a i e conclusions
based on he i s wo phases. This ype o me hodology is used no
only in psychology, bu also in social esea ch, educa ion, spo s,
and heal h. I s mul iple ad an ages include a low in e en ion
le el, independence om s anda dized measu emen ools, and i s
applicabili y in a ypical in e en ion con ex s (Angue a e al., 2018;
Chacón-Moscoso e al., 2014, 2018, 2021).
The e m obse a ional me hodology is used in his pape o
di e en ia e his me hodology om obse a ional s udies in heal h.
In ha ype o quan i a i e s udy, which can be coho , case-con ol,
o c oss sec ional (Coch an & Chambe s, 1965), esea che s
ack pa icipan s o iden i y cause-e ec ela ionships when
andomiza ion and expe imen al con ol canno be applied. Though
he publica ion o ools like he S eng hening he Repo ing o
Obse a ional S udies in Epidemiology (STROBE) s a emen ( on
Elm e al., 2007) has imp o ed he quali y o obse a ional s udies
in ecen decades, such ools a e no applicable o s udies ha ely on
obse a ional me hodology.
Few, i any, s udies ha e analyzed he e idence o alidi y o
me hodological quali y scales based on obse a ional me hodology.
Po ell e al. (2015) p oposed he Guidelines o Repo ing
E alua ions based on Obse a ional Me hodology (GREOM), which
o e simple s anda ds o s udies o his kind. Conside ing ha
obse a ional me hodology echnically quali ies as a mixed me hod
app oach (Angue a e al., 2012), he e a e se e al use ul ools o
assess me hodological quali y. These include he igo ous mixed
me hods amewo k (Ha ison e al., 2020), which b eaks down
epo s o mixed-me hods esea ch in o sequen ial componen s, and
he Guidelines o Conduc ing and Repo ing Mixed Resea ch (Leech
& Onwuegbuzie, 2010), which p o ides simple ules o o mula ing,
planning, and implemen ing mixed esea ch s udies. Howe e , hese
ools aim o measu e he main dimensions o epo quali y, and hey
p o ide no empi ical e idence o alidi y o eliabili y.
The absence o a me hodological quali y scale ep esen s a
p oblem o p ima y s udies based on obse a ional me hodology,
since esea che s a e unable o assess he me hodological quali y o
he s udies ha hey design. Addi ionally, i hampe s he in eg a ion
o high-quali y knowledge based on obse a ional me hodology
in he li e a u e (Chacón-Moscoso e al., 2013). This highligh s
he need o an ins umen wi h alidi y e idence ha speci ies
he minimum me hodological cha ac e is ics needed o e alua e
s udies elying on obse a ional me hodology. In o de o add ess
his, a Me hodological Quali y Checklis o S udies Based on
Obse a ional Me hodology (MQCOM) (Chacón-Moscoso e al.,
2019) based on he GREOM (Po ell e al., 2015) was d a ed.
MQCOM is comp ised o 16 Like scale i ems o assess he
me hodological quali y o obse a ional me hodology s udies. This
ins umen p esen ed e idence o con en alidi y and in e code
eliabili y.
The main objec i e o he s udy was o use he MQCOM o
e alua e he psychome ic p ope ies o he Me hodological Quali y
Scale o Obse a ional Me hodology s udies (MQSOM), and o es
alidi y e idence based on he MQSOM’s in e nal s uc u e. The
speci ic objec i es we e a) o s udy i s in a and in e code eliabili y;
b) o p o ide alidi y e idence based on i s in e nal s uc u e and
eliabili y based on i s in e nal consis ency; c) o ob ain empi ical
e idence o he disc imina ion and eliabili y o he scale ac o s;
and d) o apply he scale in obse a ional me hodology s udies and
in e p e he sco es.
Me hod
Pa icipan s (Uni s o Analysis)
Following PRISMA ecommenda ions (Page e al., 2021), he
inclusion c i e ia o he uni s o analysis (a icles) in his s udy we e
as ollows. All publica ions a) elied on obse a ional me hodology;
b) we e empi ical; c) p esen ed he usual sec ions o such s udies,
e.g., in oduc ion, me hod, esul s, and discussion; and d) we e
w i en in English o Spanish.
The pape selec ion was based on an exhaus i e sea ch in
PsycINFO, SCOPUS, Web o Science, Spo Discus, PSICODOC
and Google Schola o sea ch i le, abs ac , keywo ds and ull ex
o he e m “obse a ional me hodology,” wi h Decembe 1, 2022,
as he cu -o da e. The e e ence lis om he a icles collec ed in
his s ep we e also examined o iden i y addi ional s udies.
Ins umen s
P ima y pape s included we e coded using he MQCOM
(Chacón-Moscoso e al., 2019), which assesses he me hodological
quali y in obse a ional me hodology s udies. I p esen ed alidi y
e idence based on he con en o he i ems and adequa e in e code
eliabili y (> .75). The MQCOM is comp ised o 16 a ing scale
i ems. Each i em ecei es a sco e o me hodological quali y le els
o 0 (low), .5 (medium), o 1 (high). Excep ionally, i em 11 (Da a
analysis) had ou possible sco es ( om he lowes o he highes
me hodological quali y le el, 0, .33, .67 and 1). This i em had a
di e en esponse scale han he o he s because i was based on a
p e ious checklis wi h a igo ous con en alidi y and in e code
eliabili y s udy whe e expe s highly ecommended inc easing he
g adua ion o his i em (Chacón-Moscoso e al., 2019). None heless,
his i em main ains he mono onic inc emen al unc ion o all o dinal
i ems o he scale, wi h a ange o 0-1 p io o he s a is ical analysis.
The coding manual is a ailable a h ps://os .io/u 7cj.
P ocedu e
Mendeley Re e ence Manage was used o he sea ch o
pape s o o ganize and handle he in o ma ion ob ained h ough he
li e a u e sea ch. Du ing he i s sc eening, he inclusion c i e ia
we e applied o he i le, keywo ds, and abs ac . The esul ing
s udies we e assessed in a second s age in which he inclusion
c i e ia we e applied o he ull ex s. Two code s (DLA and IFM)
applied he c i e ia independen ly. In case o disag eemen s, a hi d
code (SCM) media ed un il an ag eemen was eached.
Fo he da a ex ac ion, IFM and DLA we e ained o apply
he MQCOM. Each i em and i s esponse op ions we e explained.
MTA media ed i he explana ions di e ed. Then bo h code s
independen ly applied he scale o wo obse a ional me hodology
pape s o compa e he coding. In case o disag eemen s, a hi d
3
Me hodological Quali y Scale o Obse a ion
code (SSC) media ed. A e he aining, he code s applied he
scale independen ly o a andomly selec ed 25% o he sample.
Finally, once a high le el o consensus was achie ed (> .7), DLA
applied he scale o he ull sample. The da a ex ac ion da abase is
a ailable a h ps://os .io/m6p h.
Da a Analysis
Using SPSS 27.0, he In aclass Co ela ion Coe icien (ICC)
was calcula ed o e alua e bo h in e and in acode conco dance.
Values highe han .7 we e conside ed indica o s o adequa e
eliabili y (Po ney & Wa kins, 2000). Addi ionally, desc ip i e
s a is ics we e p o ided.
A pa allel analysis was hen conduc ed using FACTOR 12
(Fe ando & Lo enzo-Se a, 2017) o ob ain empi ical e idence
o dimensionali y. Values highe han .95 o Unidimensional
Cong uence (UniCo) and highe han .85 o Explained Common
Va iance (ECV) sugges s ha da a can be ea ed as essen ially
unidimensional (Fe ando & Lo enzo-Se a, 2018).
A e he pa allel analysis, he da abase was andomly spli
in o wo subsamples o 325 pape s. Wi h one o he subsamples,
an explo a o y ac o analysis was ca ied ou using SPSS 27.0.
Fi s , a polycho ic co ela ion ma ix was c ea ed wi h all he
a iables included in he analysis. Following his s ep, he s a is ical
assump ions o he ma ix we e checked by unning Ba le ’s es
o sphe ici y, whe e signi ican esul s we e conside ed accep able,
and he Kaise -Meye -Olkin (KMO) es , whe e alues highe han
.5 we e conside ed adequa e (Ba le , 1954; Kaise & Rice, 1974).
Finally, he explo a o y ac o analysis was done using p incipal
axis me hod o ac o ex ac ion and Kaise ’s a imax c i e ion o
o hogonal o a ion (Field, 2018).
A e he explo a o y ac o analysis, a con i ma o y ac o
analysis was ca ied ou wi h he second subsample. SPSS 27.0
was used o calcula e he in e nal consis ency o he es and he
a e age disc imina ion index. The in e nal consis ency o he i ems
was measu ed using C onbach’s α, whe e alues highe han .7 we e
conside ed app op ia e. Fo he a e age disc imina ion index, alues
highe han .4 we e conside ed excellen , .3 - .39 good, and .2 - .29
accep able (Ba be o-Ga cía e al., 2015).
The bi a ia e no mali y assump ion was checked wi h PRELIS
12 o con i m he sui abili y o polycho ic co ela ions (Holgado-
Tello e al., 2010). The chi-squa e es (χ2) was un be ween each pai
o co ela ions assuming a 95% con idence in e al wi h Bon e oni
co ec ion α/c (95% con idence le el α = .05, and numbe o
con as s c = [numbe o i ems x numbe o i ems – 1]/2). The
pe cen age o accep ance o he bi a ia e no mali y assump ion was
hen calcula ed. Addi ionally, o o e come he la ge sample size bias
in he chi-squa e es , he Roo Mean Squa e E o o App oxima ion
(RMSEA) was calcula ed o each pai o co ela ions, along wi h
he pe cen age o occasions in which RMSEA was lowe han 0.1,
he adequa e alue o pa ame e es ima ion (Hoope e al., 2008).
To assess he ac o s uc u e o he scale, LISREL 12 was used
o es ima e he polycho ic co ela ions and he asymp o ic a iance-
co a iance ma ix. S anda dized ac o loadings we e hen calcula ed
(whe e lambdas o .3 o highe we e conside ed adequa e) along
wi h i indexes (Holgado-Tello e al., 2019; Sandu e e-Cha es e
al., 2018), i.e., χ2 (non-signi ican alues -p > .05- allows he null
hypo hesis o model i o be accep ed, hough la ge sample sizes
end o bias his index owa ds signi ican alues). O he i indexes
calcula ed included he Roo Mean Squa e Residual (RMR), whe e
alues below .05 we e conside ed adequa e); he Roo Mean Squa e
E o o App oxima ion (RMSEA), whe e alues o .08 o lowe
we e conside ed adequa e; he Expec ed C oss Valida ion Index
(ECVI), whe e alues close o he sa u a ed model as opposed
o he independen model we e conside ed adequa e; C i ical N
(CN), whe e alues highe han 200 we e conside ed adequa e; he
Pa simony Goodness-o -Fi Index (PGFI), whe e alues be ween
.5 - .9 we e conside ed adequa e. Finally, in he ollowing indexes,
alues highe han .9 we e conside ed adequa e: he No med Fi
Index (NFI), he Compa a i e Fi Index (CFI), he Non-No med
Fi Index (NNFI), he Inc emen al Fi Index (IFI), he Rela i e Fi
Index (RFI), he Goodness-o -Fi Index (GFI), and he Adjus ed
Goodness-o -Fi Index (AGFI).
Using JASP e sion 0.16, he eliabili y o each ac o ob ained
was s udied by calcula ing McDonald’s Omega (ω). Resul s highe
han .80 we e conside ed s ong eliabili y e idence and .65 – .80,
accep able (Kalkb enne , 2023). Fo i em disc imina ion, co ec ed
i em- o al co ela ion coe icien s we e compu ed. Resul s we e
in e p e ed as excellen when alues we e highe han .40, good o
alues be ween .30 – .40, adequa e o .20 – .30, and inadequa e o
< .20 (Ba be o-Ga cía, 1993).
Resul s
Selec ion o S udies
Figu e 1 summa izes he selec ion p ocess o he pape s o his
s udy. The inal sample o p ima y pape s included in he s udy was
650.
In e and In acode Reliabili y and Desc ip i e S a is ics o
he I ems
Table 1 p esen s bo h in e and in acode eliabili y. ICC
coe icien s we e adequa e, anging om .73 o 1.
Table 1 also p esen s desc ip i e s a is ics. The median was .5
o mos o he i ems, wi h 0 applying only o i em 8 (So wa e).
The means we e be ween .13 and .98, he s anda d de ia ions
anged be ween .11 and .46, and he e was no no mal dis ibu ion
o he i ems.
Fo i ems 11 (Da a analysis) and 13 (Theo e ical amewo k),
means we e o e .9, which implies a low disc imina ion capaci y.
I ems 11 (Da a analysis), 12 (Objec i es), 13 (Theo e ical
amewo k) and 14 (Uni s o s udy) showed s anda d de ia ion
o 0.25 o below, which implies a low a iabili y. I ems 11 (Da a
analysis) and 13 (Theo e ical amewo k) also showed an ex eme
nega i e skewness (o e 3 poin s). Finally, i em 11 (Da a analysis)
showed an ex eme posi i e ku osis.
To analyze he ela ionship be ween i ems, polycho ic co ela ions
we e calcula ed. Table 2 p esen s he bi a ia e polycho ic co ela ion
ma ix.
I ems 12 (Objec i es), 13 (Theo e ical amewo k), 14 (Uni s o
s udy), 15 (Sessions) and 16 (Discussion sec ion) s ood ou om
he o he i ems since hei co ela ions we e nega i e and/o low
4
Sandu e e-Cha es e al. / Psico hema (2025) 37(1) 1-10
Figu e 1
PRISMA 2020 Flow Diag am o Sys ema ic Re iews (Page e al., 2021)
Table 1
In e -In acode Reliabili y and Desc ip i e S a is ics o he I ems
I em In e code Reliabili y In acode Reliabili y Desc ip i e S a is ics
ICC LL UL ICC LL UL MMdn SD S K KS
1 Di ec /Indi ec .945 .926 .959 .725 .495 .855 .78 10.38 -1.33 0.09 .44
2 Obse a ion Uni .953 .936 .965 .922 .854 .958 .46 .5 0.45 0.15 -1.75 .29
3 Tempo ali y .947 .928 .961 .976 .955 .987 .46 .5 0.45 0.17 -1.73 .29
4 Dimensionali y .933 .910 .950 .845 .712 .917 .47 .5 0.46 0.14 -1.79 .30
5 Codi ica ion manual .967 .955 .975 .877 .771 .934 .77 10.35 -1.16 -0.02 .40
6 Da a ype .957 .942 .968 .835 .692 .912 .63 .5 0.33 -0.32 -0.75 .28
7 Obse a ion ins umen .958 .944 .969 .904 .822 .949 .72 10.42 -0.95 -0.90 .41
8 So wa e .797 .733 .846 .871 .759 .931 .13 00.33 2.14 2.70 .51
9 Type o pa ame e .961 .947 .971 .858 .736 .924 .65 .5 0.29 -0.14 -0.58 .34
10 Da a quali y con ol .975 .966 .982 .963 .930 .980 .82 10.38 -1.69 0.91 .50
11 Da a analysis .988 .984 .991 .920 .852 .957 .95 10.17 -3.84 15.34 .51
12 S udy objec i e .866 .821 .899 .951 .949 .953 .65 .5 0.25 0.38 -0.71 .41
13 Theo e ical amewo k .990 .986 .992 .756 .747 .765 .98 10.11 -6.19 41 .53
14 Uni s o s udy .855 .807 .891 111.64 .5 0.24 0.50 -0.72 .42
15 Sessions .926 .900 .945 .993 .987 .996 .53 .5 0.46 -0.13 -1.79 .30
16 Discussion .945 .926 .959 111.66 .5 0.27 0.02 -0.70 .36
No e. ICC = In aClass Co ela ion; LL = Lowe Limi ; UL = Uppe Limi ; S = skewness; K = ku osis; KS = Kolmogo o -Smi no no mali y es s a is ic. All ICC and KS
s a is ics yielded p < .05.
5
Me hodological Quali y Scale o Obse a ion
( = -.29 – .23). The highes posi i e bi a ia e co ela ions we e
be ween i ems 2 (Obse a ion uni c i e ia), 3 (Tempo al c i e ia),
and 4 (Dimensionali y c i e ia) ( = .98 – 1), bu his iad also
co ela ed wi h i em 1 (Di ec /indi ec obse a ion) ( = .82 – .84).
S udy o Dimensionali y
To ob ain empi ical e idence abou he numbe o ac o s ha he
scale p esen ed, a pa allel analysis was done. Table 3 p esen s he
esul o his analysis.
Table 3
Pa allel Analysis
FReal da a %
o a iance
Mean o andom %
o a iance
95 P o andom %
o a iance
142.97 13.28 15.61
210.32 11.38 12.95
38.87 10.44 11.73
47.21 9.68 10.85
56.15 8.91 9.86
66.08 8.21 9
74.41 7.5 8.19
83.4 6.82 7.47
93.06 6.11 6.78
10 2.03 5.39 6.07
11 1.85 4.63 5.51
12 1.58 3.87 5.02
13 1.44 3.06 4.55
14 0.6 2.26 3.93
15 0.02 1.26 2.87
No e. F = ac o s; P = pe cen ile
Bo h he pe cen age o a iance and he alue o McDonald’s
Omega (ω = .87) sugges only one ac o o be conside ed.
Ne e heless, he UniCo = .41 and he ECV = .31 lead da a o be
ea ed as essen ially mul idimensional (Fe ando & Lo enzo-Se a,
2018).
Explo a o y Fac o Analysis
To s udy he ac o s uc u e o he scale, a p incipal-axis ac o
analysis wi h oblimin o a ion was conduc ed on he 16 i ems.
Ba le ’s es o sphe ici y and he KMO measu e e i ied he
sampling adequacy o he analysis, Χ2 = 0.839, p < .001, KMO =
.84, and all he KMO alues o indi idual i ems we e abo e he
accep able limi o .5 (Field, 2018; Kaise & Rice, 1974). Acco ding
o he ini ial analysis o eigen alues o each ac o in he da a, i e
ac o s had eigen alues o e 1 and, in combina ion, hese explained
62.23% o he a iance. Based on he poin o maximum cu a u e
ob ained in he sc ee plo (see Figu e 2), wo ac o s ha join ly
explained 40.81% o he a iance we e e ained.
Figu e 2
Sc ee Plo Resul an o he Explo a o y Fac o Analysis
Table 4 p esen s he o a ed ac o loadings ma ix. As expec ed,
based on he examina ion o he polycho ic co ela ion ma ix
(Table 2), i ems 1 (Di ec /Indi ec obse a ion), 2 (Obse a ion
I em 12345678910 11 12 13 14 15 16
1 Di ec /Indi ec 1
2 Obse a ion Uni .84 1
3 Tempo ali y .82 11
4 Dimensionali y .83 .99 .98 1
5 Codi ica ion manual .36 .29 .26 .32 1
6 Da a ype .63 .66 .64 .63 .31 1
7 Obse a ion Ins umen .61 .60 .58 .61 .52 .51 1
8 So wa e .38 .58 .59 .53 .22 .52 .56 1
9 Type o pa ame e .42 .55 .53 .53 .32 .46 .45 .38 1
10 Da a quali y con ol .67 .44 .44 .39 .39 .54 .44 .42 .49 1
11 Da a analysis .40 .36 .36 .30 .27 .52 .24 .44 .74 .65 1
12 S udy objec i e .20 .10 .09 .09 .27 .17 .05 .07 .01 .10 .24 1
13 Theo e ical amewo k .04 -.05 -.01 .11 -.03 .09 .10 -.13 -.12 -.11 -.16 .18 1
14 Uni s o s udy .13 .23 .23 .23 -.01 .04 .10 .06 .06 .10 .04 .18 .09 1
15 Sessions .21 .08 .06 .06 .07 .08 .13 .16 .13 .21 .20 .16 .06 .03 1
16 Discussion .05 .04 .01 .05 .16 -.04 .01 -.29 .04 .23 .23 .14 -.25 .05 -.01 1
Table 2
Polycho ic Co ela ion Ma ix
6
Sandu e e-Cha es e al. / Psico hema (2025) 37(1) 1-10
uni c i e ia), 3 (Tempo al c i e ia), and 4 (Dimensional c i e ia)
p esen ed high co ela ions and showed high loading alues in he
same ac o . I ems 6 (Da a ype), 7 (Obse a ion ins umen ), and 9
(Type o pa ame e s) showed simila loadings in bo h ac o s. On
he o he hand, cu -o poin o ac o loading was se a .3. I ems
12 (S udy objec i e), 13 (Theo e ical amewo k), 14 (Obse a ion
uni s), 15 (Sessions), and 16 (Discussion), all o which had nega i e
and/o low co ela ions wi h all o he i ems, p esen ed ac o
loadings below .3 o bo h ac o s and we e excluded om he s udy.
Table 4
Ro a ed Fac o Loadings Ma ix Se o Two Fac o s
I em F1 F2
1 Di ec /Indi ec .54
2 Obse a ion Uni .97
3 Tempo ali y .97
4 Dimensionali y .94
5 Codi ica ion manual .53 .48
6 Da a ype .41
7 Obse a ion Ins umen .45
8 So wa e .52 .53
9 Type o pa ame e .54 .56
10 Da a quali y con ol .34 .62
11 Da a analysis .52
12 S udy objec i e
13 Theo e ical amewo k
14 Uni s o s udy
15 Sessions
16 Discussion
No e. F = ac o . We only p esen alues highe han he cu -o poin o ac o
loading, se a .3.
The i ems ha clus e on ac o 1 we e i ems 1 (di ec /indi ec
obse a ion), 2 (uni c i e ion o he obse a ional design), 3
( empo al c i e ion o he obse a ional design), 4 (dimensional
c i e ion o he obse a ional design), 5 (codi ica ion manual),
and 6 (da a ype), sugges ing ha ac o 1 ep esen s he quali y o
he s udy design. The i ems ha clus e on ac o 2 we e i ems 7
(obse a ional ins umen ), 8 (so wa e), 9 ( ype o pa ame e ), 10
(da a quali y con ol), and 11 (da a analysis), sugges ing ha ac o 2
ep esen s he quali y o he measu emen and analysis.
Con i ma o y Fac o Analysis
Based on he esul s o he pa allel analysis, i was de e mined
ha a single ac o should be e ained, and ha da a should be ea ed
as essen ially mul idimensional. This, in addi ion o he simila
ac o loadings ha se e al i ems displayed in bo h ac o s, led o
a decision o ca y ou a second-o de con i ma o y ac o analysis.
In e nal Consis ency
In e nal consis ency was good, ω = .87. Fac o s p oduced alues
highe han .6, which was conside ed app op ia e, ωF1 = .90; ωF2 =
.68.
A e age Disc imina ion Index
The global a e age disc imina ion index (D) was conside ed
app op ia e a .55. Fac o s also p oduced app op ia e alues, DF1 =
.46, DF2 = .67.
Bi a ia e No mali y Assump ion
Da a p oduced a ma ix o 55 pai s o i ems ([11 x 10]/2). The
bi a ia e no mali y assump ion was accep ed in 63.6% o occasions
(35 co ela ions), pco ec ed = .05/55 = .0009. Addi ionally, RMSEA
alues we e below 0.1 72.7% o he ime (40 co ela ions), so he
ac o analysis can be based on polycho ic ma ix.
Model Fi
The χ2 es was signi ican χ2(42) = 531.79, p < .001, p obably
due o he sensi i i y o his es o high sample sizes. RMR was
0.084, yielding a esul sligh ly highe han expec ed. The o he
indexes showed an app op ia e i on he second-o de ac o model,
RMSEA = 0.000, 90% CI [0.000, 0.000]; ECVI = 0.28 (sa u a ed
ECVI = 0.41; independen ECVI = 16.37); CN = 37964663941.58;
PGFI = .62; NFI = 1; CFI = 1; NNFI = 1.01; IFI = 1.01; RFI = 1;
GFI = .98; AGFI = .97.
The e o e, i seems logical o accep a ac o s uc u e comp ised
o a quali y design ac o and a quali y measu emen and analysis
ac o , bo h g ouped unde a second-o de global me hodological
quali y ac o . Figu e 3 shows he MQSOM s uc u e wi h he
s anda dized ac o loadings.
Figu e 3
S uc u e o he Me hodological Quali y Scale o Obse a ional Me hodology
S udies Wi h he S anda dized Fac o Loadings
7
Me hodological Quali y Scale o Obse a ion
(GF) o me hodological quali y encompasses he quali y o bo h he
design and o he measu emen and analysis.
I em sco es anged om 0 o 1, as did he a e ages o he i s
and second o de ac o s. Values below .5 we e conside ed low,
alues be ween .5 and .75 (bo h alues included) we e conside ed
medium, and alues o e .75 we e conside ed high. Table 6 shows
an example o he in e p e a ion o he me hodological quali y o a
se o s udies based on he sco es o each i em and he subsequen
mean pe ac o .
Table 7 p esen s he equency and pe cen age o s udies analyzed
ha ell in o each le el o quali y o F1, F2, and he GF. The quali y
le els o mos o he s udies included we e conside ed low in e ms
o he design (47.8%) and high in e ms o he measu emen and
analysis (61.8%). In e ms o o e all me hodological quali y, he
sample is dis ibu ed ai ly equally be ween he quali y le els, wi h
a sligh p edominance o he high le el o me hodological quali y
(36.8%).
Finally, Table 8 p esen s he esul ing MQSOM ool o measu e
he me hodological quali y in obse a ional me hodology s udies.
Discussion
This s udy ob ained a simple and use ul scale comp ised o 11
i ems o measu e me hodological quali y in many p o essional a eas
whe e obse a ional esea ch is conduc ed. The Me hodological
Quali y Scale o Obse a ional Me hodology s udies (MQSOM)
includes a second-o de me hodological quali y ac o ha con ains
wo i s -o de ac o s; hese se e as indica o s o quali y o bo h he
design and o he measu emen and analysis.
By c ea ing he i s scale o measu e me hodological quali y in
s udies based on obse a ional me hodology, his wo k esponds o
one o he mos impo an needs in a p omising and ui ul a ea o he
beha io al sciences. The main s eng h o his scale is ha i is based
on esul s ob ained o e he las 30 yea s by ou esea ch g oup and
elies on a b oad e iew o 650 obse a ional me hodology s udies.
Table 5 shows he desc ip i e s a is ics o each ac o .
Reliabili y yielded s ong-accep able e idence, and disc imina ion
was excellen .
Table 5
Desc ip i e S a is ics o he Fac o s
Fi s O de Fac o s Second O de Fac o
Desc ip i e s a is ic
Fac o 1
Design
quali y
Fac o 2
Measu emen and
analysis quali y
Gene al Fac o
(Me hodological
quali y)
Mean 0.51 0.76 0.62
S anda d de ia ion 0.32 0.21 0.24
McDonald’s Omega .9 .68 .87
A e age
disc imina ion .46 .67 .55
No e. Mean and s anda d de ia ion alues ange om 0 o 1.
In e p e a ion o he S udy Sco es
Fac o 1 (F1) assesses he quali y o he design and is o med by
i em 1 (obse a ional me hodology men ioned), i em 2 (obse a ion
uni c i e ia men ioned), i em 3 ( empo al c i e ia men ioned), i em
4 (dimensionali y c i e ia men ioned), i em 6 (codi ica ion manual
de ined), and i em 8 (da a ype speci ica ion men ioned). The
esea ch design o s udies wi h high F1 sco es would be based on he
exis ing li e a u e on obse a ional me hodology, wi h an explici
delimi a ion o i s obse a ional design, as well as he da a ype and
he manual ha suppo ed he subsequen obse a ion p ocess. Fac o
2 (F2) assesses he quali y o he measu emen and he analysis, and
i is o med by i em 5 (obse a ion ins umen adequacy), i em 7
(so wa e used), i em 9 ( ype o pa ame e conside ed), i em 10
( ype o da a quali y con ol), and i em 11 ( ype o da a analysis
pe o med). S udies wi h high F2 sco es would be cha ac e ized by
obus obse a ional me hodology esea ch p ocedu es, capable o
d awing well-de ined esul s based on a de ailed me hodology ha
leads o eliable conclusions. Finally, a global second-o de ac o
S udies Fac o 1 Fac o 2 A e age
I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 F1 F2 GF
Se na e al. (2017) 1111101.5 111.83 .90 .87
Lappi e al. (2017) 0000000.5 0000.10 .05
Lap esa e al. (2017) 11111111111111
Maciá e al. (2021) 1000101.5 .5 1 1 .33 .80 .57
A gudo-I u iaga e al. (2021) 100010.5 .5 .5 1 1 .33 .70 .52
No e. I = i em; F1 ( i s o de ac o 1) = Design; F2 ( i s o de ac o 2) = Measu emen and analysis; GF (second o de gene al ac o ) = Me hodological quali y.
In e p e a ion o each ac o : < 0.5 – Low quali y le el, [0.5 - 0.75] - Medium, > 0.75 – High quali y le el.
Table 6
Sco es o Each I em om Fi e S udies and A e age Values o Each Fac o
Table 7
Dis ibu ion o he Sample by Quali y Le el
Quali y le el F1 Quali y o he Design F2 Quali y o he Measu emen and Analysis GF Me hodological Quali y
Low 311 (47.8) 70 (10.8) 203 (31.2)
Medium 133 (20.5) 178 (27.4) 208 (32.0)
High 206 (31.7) 402 (61.8) 239 (36.8)
To al 650 (100) 650 (100) 650 (100)
No e. Pe cen ages a e p esen ed in b acke s.
In e p e a ion o each ac o : < 0.5 – Low quali y le el, [0.5 - 0.75] - Medium, > 0.75 – High quali y le el.
8
Sandu e e-Cha es e al. / Psico hema (2025) 37(1) 1-10
Table 8
Me hodological Quali y Scale o Obse a ional Me hodology S udies
Fac o 1. Quali y o he Design
I em 1
Di ec /indi ec obse a ion: Re e ence o obse a ional me hodology, speci ying whe he obse a ion is di ec o indi ec :
0: Me hodology is no e e enced.
0.5: Yes, jus i ied bu no documen ed.
1: Yes, jus i ied and documen ed.
I em 2
Obse a ion uni c i e ia (idiog aphic: s udy uni s a e o med by one o mo e pa icipan s i he e is a s able link be ween hem; nomo he ic: wo o mo e
s udy uni s):
0: No iden i ied.
0.5: Yes, obse a ion uni s a e iden i ied, bu wi hou jus i ica ion.
1: Yes, obse a ion uni s a e iden i ied, wi h jus i ica ion o he choice o an idiog aphic o nomo he ic app oach in acco dance wi h he s udy objec i es.
I em 3
Tempo al c i e ia (punc ual: one o wo obse a ion sessions; ollow-up: mo e han wo obse a ion sessions):
0: No iden i ied.
0.5: C i e ion o empo ali y iden i ied, bu wi hou di e en ia ing.
1: Tempo ali y c i e ion iden i ied, di e en ia ing be ween-session and wi hin-session ollow-up.
I em 4
Dimensionali y c i e ia (one-dimensional: one le el o esponse; mul idimensional: wo o mo e le els o esponse):
0: No iden i ied.
0.5: Dimensions iden i ied wi hou e e ence o any concep ual amewo k.
1: Dimensions iden i ied based on a concep ual amewo k.
I em 5
Codi ica ion manual wi h de ini ion o he ca ego ies/beha io s and speci ica ion o dimensions (in mul idimensional designs):
0: Manual no a ailable.
0.5: Pa ial in o ma ion (e.g., dimensions speci ied, bu wi hou de ini ion o he ca ego ies/codes o each dimension).
1: Codi ica ion manual wi h de ini ion o he ca ego ies/beha io s and speci ica ion o dimensions (in mul idimensional designs).
I em 6
Speci ica ion o da a ype (I, II, III, and IV [Bakeman, 1978]) as sequen ial/concu en (sequen ial da a: beha io s ha canno o e lap and belong o a single
dimension; concu en da a: beha io s ha can co-occu and belong o se e al dimensions) and e en -based/ ime-based (e en -based: he p ima y pa ame e
used in he eco d is o de o e en s; ime-based: he p ima y pa ame e is du a ion):
0: Da a ype no speci ied.
0.5: Da a ype speci ied bu no jus i ied.
1: Da a ype speci ied wi h jus i ica ion.
To al ac o 1 Quali y o he Design sco e:
Add he sco es ob ained in i ems 1-6 and di ide by he numbe o i ems.
Fac o 2. Quali y o he Measu emen and Analysis
I em 7
Adequacy o he obse a ion ins umen (combina ion o ield o ma wi h ca ego y sys em, ield o ma , ca ego y sys em, o scale o es ima ion [Angue a,
2003]):
0: Obse a ion ins umen no a ailable (e.g., only a lis o beha io s p o ided).
0.5: Obse a ion ins umen desc ibed bu no jus i ied based on he objec i es and obse a ional design.
1: Obse a ion ins umen jus i ied acco ding o he objec i es and obse a ional design.
I em 8
So wa e used o egis e da a (SDIS-GSEQ . 4.2.1./GSEQ 5, LINCE, MATCH VISION STUDIO, T ansana, o he : speci y), con ol da a quali y (SDIS-
GSEQ . 4.2.1./GSEQ 5, LINCE, HOISAN, GT, SAS, o he : speci y), and analyze da a (SDIS-GSEQ, HOISAN, THEME . 6, R, SAS, o he : speci y):
0: No used.
0.5: Used pa ially, only o some o he h ee aspec s.
1: Used o egis e da a, con ol da a quali y, and analyze da a.
I em 9
Type o pa ame e s acco ding o gi en use (Bakeman, 1978):
0: P ima y, o basic, egis a ion o a single ca ego y: equency, o de , and du a ion.
0.5: Seconda y, de i ed om a single ca ego y eco d ( a ios be ween p ima y indica o s): a e age equency, ela i e equency, a e, ela i e du a ion, a e age
du a ion, and o he : speci y.
1: Mixes, dynamic, o ansi ion ( wo ca ego ies conside ed o analyze he ansi ion om one ca ego y o ano he ): ansi ion equency, ela i e equency o
ansi ion, and ela i e du a ion o ansi ion.
I em 10
Be ween-obse e eliabili y (ag eemen be ween he eco ds o di e en obse e s)/wi hin-obse e eliabili y (ag eemen be ween he eco ds o he same
obse e a wo ime poin s):
0: No assessed.
0.5: Consensual ag eemen (quali a i e).
1: Ag eemen is global (based on p ima y indica o s, equency, and du a ion) sequen ial (based on sequen ial-o de indica o s: Pea son co ela ion, Be k
in a-class coe icien , e c.); o poin -by-poin (each eco d ha each obse e egis e s is compa ed): e.g., o al pe cen age o ag eemen , kappa coe icien ,
gene alizabili y heo y).
I em 11
Type o da a analysis pe o med (Blanco-Villaseño e al., 2003):
0: No da a analysis.
0.33: Quali a i e analysis only.
0.66: Desc ip i e analysis only.
1: In e en ial analysis: ela ionship be ween ca ego ical da a (compa ison o p opo ions); analysis o egula i ies (sequen ial analysis o delays, Ma ko chains,
T-pa e n de ec ion, analysis o pola coo dina e); mul i a ia e analysis (logis ic eg ession, log-linea , logi -p obi , co espondence analysis); analysis o he
empo al dimension (panel s udies, end analysis, ime se ies); nonpa ame ic es s; es s o ela ion (o dinal co ela ion, linea co ela ion, mul iple co ela ion);
mul idimensional scaling; o he : speci y.
To al ac o 2 Quali y o he Measu emen and Analysis sco e: Add he sco es ob ained in i ems 7-11 and di ide by he numbe o i ems.
Global ac o Me hodological quali y sco e: Add he sco es ob ained in ac o s 1 and 2 and di ide by 2.
No e. In e p e a ion o each ac o : < 0.5 – Low quali y le el, [0.5 - 0.75] - Medium, > 0.75 – High quali y le el.
9
Me hodological Quali y Scale o Obse a ion
One possible limi a ion o his s udy is he lack o unpublished
s udies on obse a ional me hodology. Al hough he pape selec ion
p ocess did no need o be exhaus i e, expe s could ha e been
consul ed o ob ain his kind o g ey li e a u e o s eng h he sample
ha o med he basis o he MQSOM (Sánchez-Meca, 2022). Fo
u he esea ch, alidi y e idence o MQSOM will be explo ed
based on i s con e gence and di e gence wi h o he ins umen s
applied in mixed me hod esea ch. Also, a guide will be d a ed o
in o m applied esea che s ac oss a wide ange o disciplines abou
he quali y o design, measu e and analysis, and me hodological
quali y as an o e all assessmen o in e en ion p og ams based on
obse a ion (Muñiz & Fonseca-Ped e o, 2019).
This o iginal con ibu ion can be conside ed a miles one in
he de elopmen o a me hodological cul u e based on sys ema ic
obse a ion. The MQSOM se es as bo h an assessmen ool and
as a guide. Thus, i should be dissemina ed among esea che s
in he yea s o come in o de o i s help au ho s o assess he
me hodological quali y o hei obse a ional me hodology s udies
and also guide applied esea che s in he design and implemen a ion
o in e en ion p og ams based on obse a ional me hodology.
Fu he mo e, his scale could become a e e ence o edi o ial boa ds
and o he decision-making commi ees. Reade s and po en ial use s
a e encou aged o sha e hei esul s when applying he scale, o
s eng hen i s po en ial.
Au ho Con ibu ions
Susana Sandu e e-Cha es: Da a cu a ion, Supe ision,
W i ing – Re iew & Edi ing. Daniel López-A enas: In es iga ion,
Da a Cu a ion, Fo mal Analysis, W i ing – O iginal D a . M.
Te esa Angue a: Concep ualiza ion, In es iga ion, W i ing –
Re iew & Edi ing. Sal ado Chacón-Moscoso: Concep ualiza ion,
Supe ision, Fo mal analysis, W i ing – Re iew & Edi ing.
Acknowledgemen s
Fi s , we would like o hank Isabella Fio a an e o he assis ance
applying he ins umen o p ima y s udies and F ancisco Pablo
Holgado-Tello o his aluable ad ice on imp o ing he dimensional
s uc u e o he scale. Thanks as well o Wendy Gosselin (Ame ican
T ansla ion Associa ion membe #275293) o e iewing he English
e sion o his ex o publica ion.
We g a e ully acknowledge he suppo o he Chilean na ional
p ojec s FONDECYT Regula 2019; Agencia Nacional de
In es igación y Desa ollo (ANID) o Chile (1190945); he Julio
Olea G an o Young Resea che s o he Spanish Associa ion o
Beha io al Science Me hodology (AEMCCO); he Gene ali a
de Ca alunya Resea ch G oup (G up de Rece ca e Inno ació en
Dissenys [GRID]) “Tecnología i aplicació mul imedia i digi al
als dissenys obse acionals” [g an numbe 2021 SGR 00718]
(2022-2024); and he suppo o he Spanish go e nmen p ojec
“In eg ación en e da os obse acionales y da os p o enien es
de senso es ex e nos: E olución del so wa e LINCE PLUS y
desa ollo de la aplicación mó il pa a la op imización del depo e y
la ac i idad ísica bene iciosa pa a la salud” [EXP_74847] (2023),
Minis e io de Cul u a y Depo e, Consejo Supe io de Depo e and
he Eu opean Union.
Funding
This wo k was suppo ed by he esea ch p ojec PID2020-
115486GB-I00 unded by he Minis e io de Ciencia, Inno ación y
Uni e sidades, MICIU/AEI/10.13039/501100011033, Go e nmen
o Spain. This unding sou ce had no ole in he design o his s udy,
da a collec ion, managemen , analysis, and in e p e a ion o da a,
w i ing o he manusc ip , and he decision o submi he manusc ip
o publica ion.
Decla a ion o In e es s
The au ho s decla e ha he e is no con lic o in e es .
Da a A ailabili y S a emen
The da ase s p esen ed in his s udy can be ound in he online
eposi o y Open Science F amewo k (h ps://os .io/m6p h).
Re e ences
Angue a, M. T. (2003). Obse a ional me hods (Gene al). In R. Fe nández-
Balles e os (Ed.), Encyclopedia o psychological assessmen , Vol. 2 (pp.
632-637). Sage.
Angue a, M. T., Blanco-Villaseño , A., Losada, J. L., & Po ell, M.
(2018). Guidelines o designing and conduc ing a s udy ha applies
obse a ional me hodology. Anua io de Psicología, 48(1), 9-17. h ps://
doi.o g/10.1016/j.anpsic.2018.02.001
Angue a, M. T., Blanco-Villaseño , A., Losada, J. L., & Sánchez-Alga a,
P. (2020). In eg ación de elemen os cuali a i os y cuan i a i os en
me odología obse acional [In eg a ion o quali a i e and quan i a i e
elemen s in obse a ional me hodology]. Ámbi os: Re is a In e nacional
de Comunicación, 49, 49-70. h ps://doi.o g/10.12795/Ambi os.2020.
i49.04
Angue a, M. T., Came ino, O., & Cas añe , M. (2012). Mixed me hods
p ocedu es and designs o esea ch on spo , physical educa ion and
dance. In O. Came ino, M. Cas añe , & M. T. Angue a (Ed.), Mixed
me hods esea ch in he mo emen sciences: Case s udies in spo ,
physical educa ion and dance (pp. 3-27). Rou ledge.
A gudo-I u iaga, F. M., Ga cí-Ma ín, P., Bo ges-He nández, P. J., &
Ruíz-La a, E. (2021). E ec s o ules changes on wa e polo shoo ing
pe o mance acco ding o he inal classi ica ion: High, medium, and
wo s le el. Jou nal o Physical Educa ion and Spo , 21(1), 188-200.
h ps://doi.o g/10.7752/jpes.2021.01026
Bakeman, R. (1978). Un angling s eams o beha io : Sequen ial analysis o
obse a ion da a. In G. P. Sacke (Ed.), Obse ing beha io : Vol. 2. Da a
collec ion and analysis me hods (pp. 63-78). Uni e si y o Pa k P ess.
Ba be o-Ga cía, M. I. (1993). Psicome ía II. Mé odos de elabo ación de
escalas [Psychome ics II. Me hods o de elop scales]. UNED.
Ba be o-Ga cía, M. I., Vila-Abad, E., & Holgado-Tello, F. P. (2015).
Psicome ía [Psychome ics]. Sanz y To es.
Ba le , M. S. (1954). A no e on mul iplying ac o s o a ious chi-squa ed
app oxima ions. Jou nal o he Royal S a is ical Socie y, Se ies B, 16,
296-298. h ps://www.js o .o g/s able/2984057
Blanco-Villaseño , A., Losada, J. L., & Angue a, M. T. (2003). Da a analysis
echniques in obse a ional designs applied o he en i onmen -beha io
ela ion. Medio Ambien e y Compo amien o Humano, 4(2), 111-126.
h p://mach.webs.ull.es/PDFS/VOL4_2/VOL_4_2_d.pd
