This is a sel -a chi ed e sion o an o iginal a icle. This e sion
may di e om he o iginal in pagina ion and ypog aphic de ails.
Au ho (s):
Ti le:
Yea :
Ve sion:
Copy igh :
Righ s:
Righ s u l:
Please ci e he o iginal e sion:
CC BY 4.0
h ps://c ea i ecommons.o g/licenses/by/4.0/
Using deep neu al ne wo ks o kinema ic analysis : challenges and oppo uni ies
© 2021 The Au ho (s). Published by Else ie L d.
Published e sion
C onin, Neil J.
C onin, N. J. (2021). Using deep neu al ne wo ks o kinema ic analysis : challenges and
oppo uni ies. Jou nal o Biomechanics, 123, A icle 110460.
h ps://doi.o g/10.1016/j.jbiomech.2021.110460
2021
Using deep neu al ne wo ks o kinema ic analysis: Challenges
and oppo uni ies
Neil J. C onin
⇑
Neu omuscula Resea ch Cen e, Facul y o Spo and Heal h Sciences, Uni e si y o Jy askyla, Finland
School o Spo and Exe cise, Uni e si y o Glouces e shi e, UK
a icle in o
A icle his o y:
Accep ed 20 Ma ch 2021
Keywo ds:
Mo ion analysis
Kinema ics
Deep neu al ne wo k
Ma ke less acking
AI
abs ac
Kinema ic analysis is o en pe o med in a lab using op ical came as combined wi h e lec i e ma ke s.
Wi h he ad en o a i icial in elligence echniques such as deep neu al ne wo ks, i is now possible
o pe o m such analyses wi hou ma ke s, making ou doo applica ions easible. In his pape I sum-
ma ise 2D ma ke less app oaches o es ima ing join angles, highligh ing hei s eng hs and limi a ions.
In compu e science, so-called ‘‘pose es ima ion” algo i hms ha e exis ed o many yea s. These me hods
in ol e aining a neu al ne wo k o de ec ea u es (e.g. ana omical landma ks) using a p ocess called
supe ised lea ning, which equi es ‘‘ aining” images o be manually anno a ed. Manual labelling has
se e al limi a ions, including labelle subjec i i y, he equi emen o ana omical knowledge, and issues
ela ed o aining da a quali y and quan i y. Neu al ne wo ks ypically equi e housands o aining
examples be o e hey can make accu a e p edic ions, so aining da ase s a e usually labelled by mul iple
people, each o whom has hei own biases, which ul ima ely a ec s neu al ne wo k pe o mance. A
ecen app oach, called ans e lea ning, in ol es modi ying a model ained o pe o m a ce ain ask
so ha i e ains some lea ned ea u es and is hen e- ained o pe o m a new ask. This can d as ically
educe he equi ed numbe o aining images. Al hough de elopmen is ongoing, exis ing ma ke less
sys ems may al eady be accu a e enough o some applica ions, e.g. coaching o ehabili a ion.
Accu acy may be u he imp o ed by le e aging no el app oaches and inco po a ing ealis ic physiolog-
ical cons ain s, ul ima ely esul ing in low-cos ma ke less sys ems ha could be deployed bo h in and
ou side o he lab.
Ó2021 The Au ho (s). Published by Else ie L d. This is an open access a icle unde he CC BY license
(h p://c ea i ecommons.o g/licenses/by/4.0/).
1. In oduc ion
In ecen yea s, he long-held d eam o aking biomechanical
analyses ou o he labo a o y has edged close o eali y wi h
he ad en o new echnology. Fo example, wea able de ices can
now be used o ack indi idual s ide cha ac e is ics du ing gai
(e.g. Da idson e al., 2019; Liu e al., 2010), including join angles
(Mund e al., 2020; Zimme mann e al., 2018). The compu a ion
o join angles is challenging wi h wea able de ices bu can be
achie ed ela i ely easily using a se o came as. Un il ecen ly,
kinema ic analysis was gene ally pe o med in a lab using op ical
came as in combina ion wi h e lec i e ma ke s, bu his se up is
no p ima ily designed o ou doo use (see Colye e al., 2018 o
a e iew o he me hodological de elopmen ). Wi h he ad en o
deep neu al ne wo ks (deep lea ning; see Table 1 o a glossa y
o key e ms), i is now possible o es ima e join angles wi hou
he need o e lec i e ma ke s. This equi es combining one o
mo e came as wi h an app oach e e ed o in compu e science
as ‘‘pose es ima ion” o de ec body landma ks, and hen using
simple geome y o es ima e join angles (i.e. he angle be ween
wo ec o s ha each ep esen a body segmen ). Thus, a leas
in heo y, kinema ic analysis can be pe o med ou side o a labo a-
o y, including in clinical and spo ing en i onmen s (C onin e al.,
2019; Kidzin
´ski e al., 2020). The pu pose o his pape is o sum-
ma ise popula ma ke less app oaches o es ima ing join angles,
highligh ing hei s eng hs and limi a ions. I ocus mainly on 2D
applica ions, since he use o pose es ima ion o ma ke less 3D
join angle p edic ion is s ill in i s in ancy (see Nakano e al.,
2020; Na h e al., 2019).
2. Pose es ima ion
In he pas ew yea s, use - iendly neu al ne wo k-based me h-
ods o pose es ima ion ha e eme ged ha allow ma ke less de ec-
ion o ana omical landma ks (A ac e al., 2019; Cao e al., 2017;
h ps://doi.o g/10.1016/j.jbiomech.2021.110460
0021-9290/Ó2021 The Au ho (s). Published by Else ie L d.
This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/).
⇑
Add ess: Vi eca 234, Rau pohjanka u 8, 40700 Jy äskylä, Finland.
E-mail add ess: [email p o ec ed]
Jou nal o Biomechanics 123 (2021) 110460
Con en s lis s a ailable a ScienceDi ec
Jou nal o Biomechanics
jou nal homepage: www.else ie .com/loca e/jbiomech
www.JBiomech.com
G a ing e al., 2019; Ma his e al., 2018). Some o hese me hods
e en allow ideos o be p ocessed in eal- ime (Cao e al., 2017;
Kane e al., 2020). One algo i hm ha has ecei ed pa icula a en-
ion is DeepLabCu (Ma his e al., 2018), which was ini ially
designed o acking animal beha iou , bu can also be used o
ack human mo emen in 2D o 3D (C onin e al., 2019; Na h
e al., 2019). These and many o he ecen s udies ha e demon-
s a ed he po en ial alue o ma ke less neu al ne wo k
app oaches in he ield o human mo emen science (see also
Tome e al., 2018). Could hese me hods lead o a e olu ion in
human mo ion analysis?
To add ess his ques ion, i is i s impo an o examine whe e
hese new app oaches came om. In compu e science, he ield o
pose es ima ion (Table 1) has exis ed o many yea s, and he cu -
en s a e o he a is qui e ad anced, wi h se e al ongoing com-
pe i ions in his a ea ensu ing con inuous de elopmen o new
me hods (e.g. h ps://pose ack.ne /). Fo example, an open sou ce
me hod called OpenPose enables key body landma ks o be acked
om mul iple humans in a ideo in eal- ime (Cao e al., 2019,
2017), and has been used as pa o a 3D ma ke less sys em o cal-
cula e join angles du ing gai wi h p omising esul s (Nakano
e al., 2020). Howe e , he e a e some c i ical dis inc ions be ween
pose es ima ion and kinema ic analysis. Fi s ly, s ic ly speaking
pose es ima ion only in ol es he de ec ion o body landma ks,
which a e hen used in combina ion wi h geome y o compu e
he angle be ween any wo body segmen s. Secondly, he accu acy
equi emen s o pose es ima ion a e less s ic han hose o kine-
ma ic analysis. Common applica ions o pose es ima ion include
gaming, obo ics and anima ion, and hese algo i hms a e also use-
ul o help au oma ed ehicles de ec pedes ians. Fo hese appli-
ca ions, i is usually su icien o p edic he loca ion o a body
landma k o wi hin abou 5–10 cm. Howe e , when calcula ing
join angles o kinema ic analysis, his magni ude o e o is unac-
cep able. Thus, pose es ima ion algo i hms canno simply be used
ou o he box o accu a e kinema ic analysis (see See hapa hi
e al., 2019 o u he limi a ions).
None heless, he eme gence o new, mo e ad anced app oaches
ha allow a use o ain hei own models (such as DeepLabCu )
may gi e us he aw ing edien s needed o de elop ma ke less
deep lea ning app oaches ha could con end wi h exis ing gold
s anda d me hods such as op ical mo ion analysis (and manual
digi isa ion). Howe e , o de elop a ma ke less deep lea ning
me hod o es ima ing join angles, i is i s necessa y o ain a
model o de ec he desi ed ea u es, which in his case a e
ana omical landma ks. Exis ing pose es ima ion me hods achie e
his ia a p ocess called supe ised lea ning (see Ma his e al.,
2020 o discussion o indi idual algo i hms).
3. Supe ised lea ning
In he con ex o his pape , supe ised lea ning in ol es ain-
ing an algo i hm o iden i y pa e ns be ween images and hei
co esponding labels, which a e p o ided by a human ‘supe iso ’
(Cunningham e al., 2008). These labels indica e whe e in he
image a pa icula body pa o objec is loca ed. The p emise is
ha a e seeing a su icien numbe o examples o a body pa ’s
appea ance, he ne wo k can obus ly lea n o iden i y his body
pa in o he images ha i has no p e iously seen (Fig. 1).
Un o una ely, he labelling p ocess is augh wi h di icul ies.
Fi s ly, i is ine i ably subjec i e. Each labelle has hei own con-
cep o ana omical landma ks, and whe e exac ly he label should
be placed. I all o he da a ha a e used o ain he model (i.e.
aining da a) a e labelled by he same pe son, he ne wo k may
lea n o iden i y he body pa s consis en ly acco ding o ha
labelle ’s logic, bu a di e en labelle may s ill a gue ha he neu-
al ne wo k labels images inco ec ly (Nowak and Rüge , 2010).
The e is no easy solu ion o his p oblem because we o en do
no know he g ound u h (i.e. he objec i ely co ec loca ion),
bu one app oach is o 2 o mo e people o label he da a and hen
con i m ag eemen be ween hei es ima es based on some p e-
de ined eliabili y c i e ion.
Ano he di icul y o he labelling p ocess ela es o he quali y
and a ie y o he aining da a. When selec ing hese da a, i is
common p ac ice o i s collec ideos ha a e ele an o he ask
a hand, o example, ideos o people walking and unning. We
hen ex ac indi idual images om hose ideos and label he
ex ac ed images. The goal is o p oduce a aining se ha includes
lo s o a iabili y, so ha he neu al ne wo k lea ns o label images
obus ly. In ou example, we would wan o include images om
di e en pa s o he s ep cycle, people wea ing di e en clo hes,
wi h di e en skin colou s, di e en ligh ing, and om di e en
angles and scales. By exposing ou ne wo k o all hese sou ces
o a iabili y, he e is a be e chance ha a e aining i will
ecognise wide a ia ions in new images. Na u ally, hese equi e-
men s mean ha la ge and a ied aining se s need o be colla ed
and labelled, bo h o which can be ime consuming. Mo eo e , i
we wan a obus ma ke less app oach, we canno ain he model
using images ha con ain e lec i e ma ke s (which could ac as
g ound u h). I we did, he model could use he appea ance o
he ma ke s o help iden i y each body pa , so when ying o
analyse a new image whe e he ma ke s we e no p esen , he
model may no make accu a e p edic ions.
Came a se ings a e ano he issue ele an o he collec ion o
aining da a, pa icula ly he ame a e and shu e speed wi h
Table 1
Glossa y o key e ms.
Neu al ne wo k An i e a i e compu a ional me hod ha uses a ne wo k o
unc ions o lea n ea u es om da a. Con olu ional
neu al ne wo ks a e a a ian commonly used in image
p ocessing because o hei use o ma hema ical
con olu ions o de ec spa ial pa e ns. G aph neu al
ne wo ks a e ano he a ian ha allow connec ions
be ween di e en s uc u es o be encoded in he model,
e.g. he spa ial ela ionships be ween body pa s
Deep lea ning Using neu al ne wo ks wi h mul iple laye s (hence
‘‘deep”) o de ec pa e ns in da ase s
G ound u h Known ‘‘co ec ” answe s agains which a neu al
ne wo k’s p edic ions a e compa ed o de e mine i s
accu acy, e.g. join cen e loca ions in an image
de e mined manually
O e i ing A p ocess whe e a ained model lea ns he ea u es o he
aining da a so well (‘‘o e i s” he da a) ha i does no
gene alise well (makes poo p edic ions when exposed o
new da a)
Pose es ima ion A compu e ision me hod ha uses some kind o neu al
ne wo k model o de ec body landma ks in an image
P obabili y hea
map
Con olu ional neu al ne wo ks assign p obabili ies o
each pixel o an image depending on he lea ned
likelihood ha a ea u e is p esen in ha pa o he
image. Fo example, a model ained o de ec hands will
assign high p obabili ies o pixels whe e he hands a e
isible and low p obabili ies o o he pixels
Sel -supe ised
lea ning
A me hod o aining a neu al ne wo k ha does no
equi e he use o p o ide manually labelled da a as
inpu . Ins ead he labels a e ex ac ed au oma ically om
he da a. Fo example, we could cu an image in o 9
equally-sized squa es and jumble hem up. By lea ning o
ea ange he squa es in he co ec o de , he model can
‘‘lea n” use ul ea u es om he image
Supe ised
lea ning
The p ocess o using labelled da a o ain a neu al
ne wo k o lea n desi ed ea u es. A e aining, he
ne wo k can de ec he p esence o lea ned ea u es in
new, p e iously unseen images
T ans e
lea ning
Using a model ained o pe o m one ask as he basis o a
model o a new ask
N.J. C onin Jou nal o Biomechanics 123 (2021) 110460
2
which he ideos a e sampled (Ma his e al., 2020). As a gene al
ule, shu e speed should be a leas double he sampling a e.
Many mode n came as, such as hose in mobile phones and web-
cams, will by de aul sample da a a a ound 30 ames pe second,
necessi a ing a shu e speed o a leas 1/60 h o a second. Wi h
hese se ings, he indi idual ames o a ideo can be e y blu y
in dynamic scenes (Fig. 2A), and his makes labelling challenging.
Depending on he came a, his can usually be o e come by manu-
ally inc easing shu e speed. Image esolu ion is ano he impo -
an ac o : e y low- esolu ion images esul in pixella ed close-
up iews ha can make i di icul o accu a ely label a body pa
(Fig. 2B). Fu he mo e, body pa occlusion is common in a 2D
came a iew, e.g. he hand blocking he hip (Fig. 2C). The labelle
mus hen decide whe he o label he loca ion whe e he blocked
pa is belie ed o be, o o a oid labelling he pa o ha image
(see Table 2 o some ecommenda ions).
As well as da a quali y, da a olume is a key elemen o supe -
ised lea ning. Gene ally, neu al ne wo ks equi e lo s o aining
examples o eliably iden i y an objec o body pa in new images.
E en a ask as mundane as iden i ying ca s in images is su p is-
ingly di icul o neu al ne wo ks, equi ing ens o housands o
examples (e.g. K izhe sky e al., 2012). Thus, o ain obus mod-
els, we need willing indi iduals o label he aining images. In
la ge, open sou ce da ase s, his ask is achie ed using c owd-
sou cing whe eby a la ge numbe o indi iduals a e ec ui ed
and paid o each label a subse o images ( he la ges da ase cu -
en ly in exis ence, ImageNe , cu en ly con ains o e 14 million
labelled images; Deng e al., 2009). All he labelled da a a e hen
combined and used o ain one la ge model ha includes hou-
sands o labelled images. A good example is OpenPose (Cao e al.,
2019, 2017), which was ained using ens o housands o images
ha we e labelled by a la ge numbe o people. This is p oblema ic
because we canno ensu e ha each o he people labelling he
da a used he same logic (o indeed whe he hey possess he nec-
essa y ana omical knowledge). In some cases, hose who publish
he esul ing model openly acknowledge ha hey we e (unde -
s andably) no able o manually check he labelling esul s o all
images, due o he olume o da a. This can esul in a con lic :
we inpu many di e en images o a body pa o he model, bu
his body pa has been labelled in di e en ways by di e en peo-
ple, making i di icul o he model o lea n a eliable cons uc o
wha ha body pa looks like (see Table 2 o labelling
ecommenda ions).
In addi ion o OpenPose, many o he open sou ce pose es ima-
ion models ha e been ained using open da ase s and c owd-
sou ced labels. Fo example, in Fig. 3, sample images om he
commonly used MPII da ase (And iluka e al., 2014) a e shown
along wi h he accompanying c owdsou ced labels. In he majo i y
o cases, he labels clea ly do no co espond wi h ana omical land-
ma ks (e.g. he knees in Fig. 3B and he hips in Fig. 3C), which
would likely in luence he esul ing join angles. Mo eo e , body
pa s a e labelled e en when hey a e occluded, which in a 2D
image usually esul s in he label being placed on he w ong side
o he body (se e al examples in Fig. 3A). Ma ke s a e also placed
on di e en aspec s o he same body pa (e.g. he ankles in
Fig. 3B), which likely makes i mo e di icul o a neu al ne wo k
o lea n app op ia e ea u es.
Clea ly, c owdsou cing he label p ocess is no app op ia e
when he goal is o ain a model o accu a ely de ec human
ana omical landma ks o kinema ic analysis. I models a e ained
using labels ha do no e lec he ac ual body pa s ha a
biomechanis is in e es ed in, he neu al ne wo k will no ‘lea n’
o label new images co ec ly. As an example, Fig. 4 shows he ou -
pu o p ocessing a single unning ial wi h OpenPose ( o esul -
ing ideo see supplemen a y ma e ial), one o he bes -known pose
es ima ion algo i hms, as well as wi h manual analysis pe o med
by he au ho . The OpenPose ma ke placemen s o a gi en body
pa o en exhibi so-called ‘‘ji e ” be ween ames (e.g. he hip
ma ke s), which is p obably a leas pa ly due o he con lic ing
labelling logic and/o use o mul iple labelle s men ioned abo e.
The algo i hm also some imes mislabels he igh and le limbs.
Bo h o hese issues a e cha ac e is ic o an algo i hm ha has
no obus ly lea n o iden i y speci ic body pa s, and hey ha e
impo an consequences o calcula ing a iables such as body seg-
Fig. 1. In supe ised lea ning we i s ain a con olu ional neu al ne wo k (A, o de ini ion see Table 1) by eeding in image-label pai s (only one pai shown o simplici y).
Once he model is ained, we pe o m in e ence, i.e. p ocess new images wi h he ained model. The model labels he images using he logic ha was ‘lea ned’ du ing
aining (B). All images a e om he MPII da ase (And iluka e al., 2014).
N.J. C onin Jou nal o Biomechanics 123 (2021) 110460
3
men leng hs and join angles (Fig. 4). Based on he abo e-
men ioned limi a ions and my own expe ience, some ecommen-
da ions o labelling a e gi en in Table 2 (see also Ma his e al.,
2020).
The issues ou lined abo e lead us o a quanda y: we need lo s o
aining examples o p oduce obus deep neu al ne wo ks. On he
o he hand, such la ge olumes o da a canno be easibly labelled
by a single pe son, and ye using mul iple labelle s can a ec
model accu acy. One possible solu ion lies in he use o an
app oach called ans e lea ning.
4. T ans e lea ning
T ans e lea ning in ol es modi ying a model ha has been
ained o pe o m a ce ain ask, say, iden i ying ehicles in
images, in such a way ha i e ains some o he lea ned ea u es
and is hen e- ained o iden i y ea u es in a new se o images,
namely hose ela ed o ou ask (Donahue e al., 2013). Fo exam-
ple, we migh wish o use ans e lea ning o adap a model ha
has been ained o de ec di e en ca ego ies o objec s in an
image (e.g. ables and chai s) so ha i can be used o de ec
human body pa s. To do his, we would e- ain he exis ing
model by modi ying i s s uc u e and hen showing i a se o
labelled images o he ana omical landma ks o in e es . Thus,
al hough he de ec ion o ables and chai s is no a all ela ed o
ou ask, he e a e common ea u es o bo h asks ha allow o
‘‘knowledge” o be ans e ed be ween hem, hence he name
ans e lea ning (see Johnson e al., 2019 o an example ela ed
o human mo ion analysis).
Because ans e lea ning in ol es modi ying a model ha has
al eady been ained on a la ge numbe o images, when we ain
i o pe o m a new ask, we no longe need huge da ase s. Dee-
pLabCu , o example, exploi s his concep and can yield models
ha make accu a e p edic ions despi e being ained wi h jus a
ew hund ed aining examples (Ma his e al., 2018; C onin e al.,
2019; Papic e al., 2020). I should be no ed howe e ha i aining
images a e o low quali y (e.g. low esolu ion, blu ing, occlusion),
o include mo emen s such as gymnas ics o pole aul whe e he
a hle e is o en upside down, mo e aining images may be
equi ed. None heless, i is easonable o conclude ha ecen
ad ances ha e b ough us close o he d eam o a uly ma ke less
(and open sou ce) app oach ha can be used ou side o a lab en i-
onmen , including in na u al spo s and aining se ings.
5. Wha nex ?
A i icial in elligence has ecei ed huge media in e es in ecen
yea s, and pe haps as a esul , people o en o e -es ima e he cu -
en s a e o he a and wha can ealis ically be achie ed (Siegel,
2019). I is impo an o emembe ha neu al ne wo ks do no
pe o m magic icks; hey iden i y ma hema ical pa e ns in da a.
I he da ase used o ain a model is small and homogeneous (e.g.
only includes da a om 2 o 3 indi iduals), he ained model is
e y likely o make poo p edic ions when es ed on images o p e-
iously unseen people, o e en he same people om he aining
da ase imaged in di e en se ings. In o he wo ds, neu al ne -
wo ks pe o m well on he speci ic ask o which hey we e
ained. To p oduce obus models, he aining da ase should
include examples o a wide ange o human poses, en i onmen s,
clo hing, ligh ing e c. As wi h any scien i ic ool, neu al ne wo ks
ha e hei limi a ions, bu when used app op ia ely hey ha e
he po en ial o e olu ionise he way kinema ic analyses a e pe -
Fig. 2. Challenges associa ed wi h labelling 2D images. A: Image blu . The op image is blu y because o a slow shu e speed (1/15). In his case, i would be di icul o
accu a ely label he ee and ankles. In he bo om image, an indi idual is shown unning on a eadmill. This image was ex ac ed om a ideo sampled a 60 Hz wi h a
shu e speed o 1/250, and he blu ing e ec is much less e iden . B: Image esolu ion. In he uppe image, which is low esolu ion (177 201 pixels; om he Leeds Spo s
da ase ; Johnson and E e ingham, 2010), zooming in on a smalle egion can make i di icul o accu a ely iden i y ana omical landma ks because o pixella ion (inse ). In he
lowe image (1920 1080 pixels; om he MPII da ase ; And iluka e al., 2014), pixella ion is much less e iden . Howe e , in bo h cases he challenge o accu a ely
iden i ying he join cen e wi hou being able o physically palpa e emains a challenge. C: Occlusion. When using a single came a, body pa s a e o en no isible in he
esul ing 2D images. Occlusion can be in he o m o one body pa blocking ano he ( op; hand blocking knee), o simply because he a ge body pa is blocked by he body
i sel (middle; igh side no isible) o by an implemen (bo om; igh knee blocked by ball). All images in C a e om MPII (And iluka e al., 2014).
N.J. C onin Jou nal o Biomechanics 123 (2021) 110460
4

o med. Al hough he accu acy o hese me hods compa ed o
g ound u h is s ill somewha unclea (D’An onio e al., 2020;
Nakano e al., 2020), his is a e y ac i e esea ch a ea. In ac , i
is likely ha in he bes case, he accu acy o such sys ems is
al eady su icien o hem o be use ul in ce ain se ings, such
as o gi ing apid eedback o a hle es abou aining pe o mance,
o o assis in clinical moni o ing o gai dis u bances. Ul ima ely,
he biomechanics communi y will also need o decide how accu-
a e is accu a e enough.
Exis ing echniques ely on supe ised lea ning o de ec body
landma ks, bu he e a e se e al exci ing new a enues ha may
e en ually help o imp o e accu acy u he . Fo example, sel -
supe ised lea ning, as he name sugges s, emo es he need o
manually label aining da a and ins ead elies on cues (o ‘‘labels”)
ha can be ex ac ed au oma ically om he da a i sel (Table 1).
This could heo e ically allow e y la ge, di e se da ase s o be
used o ain accu a e models capable o 2D o e en 3D analysis.
P omising ad ances in his a ea ha e al eady been made (e.g.
Kocabas e al., 2019; Kundu e al., 2020), al hough hese me hods
ha e no ye been applied o he speci ic ask o kinema ic analysis,
which equi es no jus he accu a e de ec ion o body landma ks,
bu addi ional pos -p ocessing o yield join angles. O he neu al
ne wo k app oaches such as g aph-based me hods allow in o ma-
ion (o ‘knowledge’) o be encoded in o an algo i hm (e.g. Ge e al.,
2019). I hese echniques could be success ully le e aged, neu al
ne wo ks migh lea n o make ewe mis akes. Mo eo e , hey
would be be e equipped o deal wi h some o he cons ain s o
human mo emen , such as he ixed connec ions be ween body
pa s, and ealis ic ame- o- ame changes in join ange o
mo ion o mo emen eloci y.
To da e, ew compa isons ha e been pe o med be ween ma ke -
lessand ma ke -based me hods, pa lybecause i is challenging o do
eliably. Howe e , OpenPose can p edic ma ke loca ions ha a e
o en wi hin 1–3 cm o he ac ual ana omical landma ks acco ding
o op ical sys ems (Nakano e al., 2020). I belie e ha his magni ude
o e o can be imp o ed upon using models ailo ed o he equi e-
Table 2
Recommenda ions o wo king wi h deep lea ning pose es ima ion me hods.
P oblem Possible solu ion(s) Commen s
Blu y aining images Use highe shu e
speed (and po en ially
ame a e)
Image blu is ele an
du ing bo h he aining
and es ing phases and
may need o be
conside ed when
choosing came a ype
and se ings
Pixella ed aining
images
When aiming o ack
as mo emen s (e.g.
unning), use a leas
FHD esolu ion
(1920 1080) when
collec ing aining da a
Highe image esolu ion
makes accu a e labelling
easie . Images (and
labels) can be
downscaled p io o
neu al ne wo k aining,
helping o dec ease he
memo y equi emen s
o aining
Inconsis en model
pe o mance due o
mul iple labelle s
o aining images
Ag ee upon ana omical
landma ks in ad ance;
check g oup ag eemen
on (a leas ) a subse o
images o ensu e
aining da a a e eliable
Wi h ans e lea ning
app oaches (see ex ), i
is usually easible o
mul iple indi iduals o
label all aining da a
Occluded body pa Only label body
landma ks ha a e
clea ly isible (Papic
e al., 2020)
A smalle numbe o
accu a e labels is
supe io o a la ge
numbe o inaccu a e
labels. Poo o
inconsis en labelling
dis o s he model’s
‘‘unde s anding” o
wha a pa icula pa
should look like
Inconsis en labelling Label a gi en body pa
consis en ly, e.g. always
he la e al side o he
pa i isible. I bo h
sides o a join need o
be labelled, label hem
sepa a ely
Gene ic labelling o a
body pa (e.g. labelling
any isible pa o he
ankle) necessi a es a
la ge olume o
aining da a o he
model o de ec ha
pa consis en ly
Gaps in da a labelled
by a ained model
Fill gaps using spa io-
empo al il e ing
(Ka ashchuk, 2020;
Papic e al., 2020)
Simple il e s (e.g.
median) may su ice
when labels a e only
missing o 1–2
consecu i e ames
Fig. 3. Examples o mislabelled body pa s in he MPII da ase (c owdsou ced labels
a e shown wi h ed ci cles). A: Le -side body pa labels a e placed on he igh side
because he le side o he body is no isible. B: Ma ke s o he le and igh limb
a e placed inconsis en ly, e.g. la e al e sus medial side (knees, elbows, ankles).
No e also ha he igh hip ma ke is placed whe e he hip is p esumed o be, bu
he label i sel is on he le o ea m. C: Simila issues o A and B, especially a he
shoulde s and w is s. No e ha he hip ma ke s do no co espond wi h he g ea e
ochan e loca ion commonly used in op ical mo ion analysis. Inse : he labelling
scheme used o he MPII da ase . (Fo in e p e a ion o he e e ences o colou in
his igu e legend, he eade is e e ed o he web e sion o his a icle.)
N.J. C onin Jou nal o Biomechanics 123 (2021) 110460
5
men s o biomechanical analysis. Gi en ha he cu en gold s an-
da d op ical sys ems and manual digi isa ion also include inhe en
limi a ions (e.g. mo emen o skin and ma ke s ela i e o he unde -
lying ana omical landma k), i we each a s a e whe e ma ke -based
and ma ke less me hods yield esul s wi hin a ew mm o each o he ,
ma ke less mo ionanalysiscould ulybea easibleop ion o human
mo emen scien is s, bo h in and ou side o he lab.
Decla a ion o Compe ing In e es
The au ho s decla e ha hey ha e no known compe ing inan-
cial in e es s o pe sonal ela ionships ha could ha e appea ed
o in luence he wo k epo ed in his pape .
Acknowledgemen s
I am g a e ul o he Academy o Finland o unding (decision
numbe : 323473), and o Roope Ui o and Leonoo Vlaande en
o hei assis ance. I also hank he g oups o I ene Da is (Ha a d
Medical School) and Benedic e Vanwanseele (KU Leu en) o alu-
able discussions abou his opic.
Appendix A. Supplemen a y da a
Supplemen a y da a o his a icle can be ound online a
h ps://doi.o g/10.1016/j.jbiomech.2021.110460.
Re e ences
And iluka, M., Pishchulin, L., Gehle , P., Schiele, B., 2014. 2D Human Pose
Es ima ion: New Benchma k and S a e o he A Analysis. CVPR, 3686–3693.
h ps://doi.o g/10.1109/CVPR.2014.471.
A ac, A., Zhao, P., Dobkin, B.H., Ca michael, S.T., Golshani, P., 2019. DeepBeha io : A
Deep Lea ning Toolbox o Au oma ed Analysis o Animal and Human Beha io
Imaging Da a. F on . Sys . Neu osci. 13, 20. h ps://doi.o g/10.3389/
nsys.2019.00020.
Cao, Z., Hidalgo Ma inez, G., Simon, T., Wei, S.-E., Sheikh, Y.A., 2019. OpenPose:
Real ime Mul i-Pe son 2D Pose Es ima ion using Pa A ini y Fields. IEEE T ans.
Pa e n Anal. Mach. In ell. 1–1. h ps://doi.o g/10.1109/ pami.2019.2929257.
Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y., 2017. Real ime Mul i-Pe son 2D Pose
Es ima ion using Pa A ini y Fields. CVPR, 1302–1310. h ps://doi.o g/
10.1109/CVPR.2017.143.
Colye , S.L., E ans, M., Coske , D.P., Salo, A.I.T., 2018. A Re iew o he E olu ion o
Vision-Based Mo ion Analysis and he In eg a ion o Ad anced Compu e Vision
Me hods Towa ds De eloping a Ma ke less Sys em. Spo . Med. - Open 4, 24.
h ps://doi.o g/10.1186/s40798-018-0139-y.
C onin, N.J., Ran alainen, T., Ah iainen, J.P., Hynynen, E., Walle , B., 2019. Ma ke less
2D kinema ic analysis o unde wa e unning: A deep lea ning app oach. J.
Biomech. 87, 75–82. h ps://doi.o g/10.1016/j.jbiomech.2019.02.021.
Cunningham, P., Co d, M., Delany, S.J., 2008. Supe ised lea ning. In: Cogni i e
Technologies. Sp inge Ve lag, pp. 21–49.
D’An onio, E., Tabo i, J., Pale mo, E., Rossi, S., Pa ane, F., 2020. A ma ke less sys em
o gai analysis based on OpenPose lib a y. I2MTC 2020 - In e na ional
Ins umen a ion and Measu emen Technology Con e ence, P oceedings.
Ins i u e o Elec ical and Elec onics Enginee s Inc..
Da idson, P., Vi ekunnas, H., Sha ma, D., Piché, R., C onin, N., 2019. Con inuous
Analysis o Running Mechanics by Means o an In eg a ed INS/GPS De ice.
Senso s 19, 1480. h ps://doi.o g/10.3390/s19061480.
Deng, J., Dong, W., Soche , R., Li, L.-J., Li, K., Fei-Fei, L., 2009. ImageNe : A La ge-Scale
Hie a chical Image Da abase. CVPR, 248–255. h ps://doi.o g/10.1109/
CVPR.2009.5206848.
Donahue, J., Jia, Y., Vinyals, O., Ho man, J., Zhang, N., Tzeng, E., Da ell, T., 2013.
DeCAF: A Deep Con olu ional Ac i a ion Fea u e o Gene ic Visual Recogni ion.
31s In . Con . Mach. Lea n. ICML 2014, 2, pp. 988–996.
Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., Yuan, J., 2019. 3D Hand Shape and Pose
Es ima ion om a Single RGB Image. CVPR, 10833–10842.
G a ing, J.M., Chae, D., Naik, H., Li, L., Koge , B., Cos elloe, B.R., Couzin, I.D., 2019.
Deepposeki , a so wa e oolki o as and obus animal pose es ima ion using
deep lea ning. Eli e 8. h ps://doi.o g/10.7554/eLi e.47994.
Johnson, S, E e ingham, M, 2010. Clus e ed pose and nonlinea appea ance models
o human pose es ima ion. P oceedings o he B i ish Machine Vision
Con e ence. h ps://doi.o g/10.5244/C.24.12.
Johnson, W.R., Mian, A., Lloyd, D.G., Alde son, J.A., 2019. On- ield playe wo kload
exposu e and knee inju y isk moni o ing ia deep lea ning. J. Biomech. 93,
185–193. h ps://doi.o g/10.1016/j.jbiomech.2019.07.002.
Kane, G., Lopes, G., Saunde s, J.L., Ma his, A., Ma his, M.W., 2020. Real- ime, low-
la ency closed-loop eedback using ma ke less pos u e acking. bioRxi
2020.08.04.236422. h ps://doi.o g/10.1101/2020.08.04.236422
Ka ashchuk, P, e al., 2020. Anipose: a oolki o obus ma ke less 3D pose
es ima ion. bioRxi . h ps://doi.o g/10.1101/2020.05.26.117325.
Kidzin
´ski, Ł., Yang, B., Hicks, J.L., Rajagopal, A., Delp, S.L., Schwa z, M.H., 2020. Deep
neu al ne wo ks enable quan i a i e mo emen analysis using single-came a
ideos. Na . Commun. 11, 1–10. h ps://doi.o g/10.1038/s41467-020-17807-z.
Kocabas, M., Ka agoz, S., Akbas, E., 2019. Sel -Supe ised Lea ning o 3D Human
Pose using Mul i- iew Geome y. CVPR, 1077–1086.
K izhe sky, A., Su ske e , I., Hin on, G.E., 2012. ImageNe classi ica ion wi h deep
con olu ional neu al ne wo ks. In . Con . Neu al In . P ocess. Sys ., pp 1097–
1105. h ps://doi.o g/10.1145/3065386.
Kundu, J.N., Se h, S., Jampani, V., Rakesh, M., Babu, R.V., Chak abo y, A., 2020. Sel -
Supe ised 3D Human Pose Es ima ion ia Pa Guided No el Image Syn hesis.
A Xi , 6151–6161.
Liu, T., Inoue, Y., Shiba a, K., 2010. A Wea able G ound Reac ion Fo ce Senso Sys em
and I s Applica ion o he Measu emen o Ex insic Gai Va iabili y. Senso s 10,
10240–10255. h ps://doi.o g/10.3390/s101110240.
Fig. 4. Sagi al plane analysis o a segmen o eadmill unning, sampled a 60 Hz
wi h a GoP o HERO8 came a. No e ha he x,y coo dina es (A and B) o en show
la ge, apid changes in loca ion be ween ames (e.g. ankle and knee x-coo dina es).
Segmen leng hs compu ed using he dis ance o mula (C and D) also show la ge,
non-physiological de ia ions o e ime, which a e possible because OpenPose does
no cons ain segmen leng hs. Join angles (E and F) we e compu ed using x,y
coo dina es p edic ed by OpenPose (solid black aces). Fo segmen leng hs and
join angles, esul s o he au ho ’s manual analysis o he same ideo a e also
shown (do ed ed aces). This igu e clea ly shows ha inaccu a e de ec ion o
ana omical landma ks leads o inaccu a e join angle es ima es. (Fo in e p e a ion
o he e e ences o colou in his igu e legend, he eade is e e ed o he web
e sion o his a icle.)
N.J. C onin Jou nal o Biomechanics 123 (2021) 110460
6
Ma his, A., Mamidanna, P., Cu y, K.M., Abe, T., Mu hy, V.N., Ma his, M.W., Be hge,
M., 2018. DeepLabCu : ma ke less pose es ima ion o use -de ined body pa s
wi h deep lea ning. Na . Neu osci. 21, 1281–1289. h ps://doi.o g/10.1038/
s41593-018-0209-y.
Ma his, A., Schneide , S., Laue , J., Ma his, M.W., 2020. A P ime on Mo ion Cap u e
wi h Deep Lea ning: P inciples, Pi alls, and Pe spec i es. Neu on 108, 44–65.
h ps://doi.o g/10.1016/j.neu on.2020.09.017.
Mund , M., Thomsen, W., Wi e , T., Koeppe, A., Da id, S., Bame , F., Po has , W.,
Ma ke , B., 2020. P edic ion o lowe limb join angles and momen s du ing
gai using a i icial neu al ne wo ks. Med. Biol. Eng. Compu . 58, 211–225.
h ps://doi.o g/10.1007/s11517-019-02061-3.
Nakano, N., Saku a, T., Ueda, K., Omu a, L., Kimu a, A., Iino, Y., Fukashi o, S.,
Yoshioka, S., 2020. E alua ion o 3D Ma ke less Mo ion Cap u e Accu acy Using
OpenPose Wi h Mul iple Video Came as. F on . Spo . Ac . Li ing 2, 50. h ps://
doi.o g/10.3389/ spo .2020.00050.
Na h, T., Ma his, A., Chen, A.C., Pa el, A., Be hge, M., Ma his, M.W., 2019. Using
DeepLabCu o 3D ma ke less pose es ima ion ac oss species and beha io s.
Na . P o oc. 14, 2152–2176. h ps://doi.o g/10.1101/476531.
Nowak, S., Rüge , S., 2010. How eliable a e anno a ions ia c owdsou cing? A s udy
abou in e -anno a o ag eemen o mul i-label image anno a ion. In:
P oceedings o he In e na ional Con e ence on Mul imedia In o ma ion
Re ie al, p. 557. h ps://doi.o g/10.1145/1743384.1743478
Papic, C., Sande s, R.H., Naemi, R., Elipo , M., Ande sen, J., 2020. Imp o ing da a
acquisi ion speed and accu acy in spo using neu al ne wo ks. J. Spo s Sci.
h ps://doi.o g/10.1080/02640414.2020.1832735.
See hapa hi, N., Wang, S., Saluja, R., Blohm, G., Ko ding, K.P., 2019. Mo emen
Science Needs Di e en Pose T acking Algo i hms. A Xi , p. a Xi :1907.10226.
Siegel, E., 2019. The Media’s Co e age o AI is Bogus [WWW Documen ]. Sci. Am.
URL h ps://blogs.scien i icame ican.com/obse a ions/ he-medias-co e age-
o -ai-is-bogus/
Tome, D., Toso, M., Agapi o, L., Russell, C., 2018. Re hinking Pose in 3D: Mul i-s age
Re inemen and Reco e y o Ma ke less Mo ion Cap u e. P oc. - 2018 In . Con .
3D Vision, 3DV, 2018, pp. 474–483.
Zimme mann, T., Tae z, B., Blese , G., 2018. IMU- o-Segmen Assignmen and
O ien a ion Alignmen o he Lowe Body Using Deep Lea ning. Senso s 18,
302. h ps://doi.o g/10.3390/s18010302.
N.J. C onin Jou nal o Biomechanics 123 (2021) 110460
7