Resea ch A icle
Con ol o Magne ic Manipula o Using Rein o cemen Lea ning
Based on Inc emen ally Adap ed Local Linea Models
Ma in B ablc ,
1
Jan ˇ
Zegkli z ,
2
Robe G epl ,
1
and Robe Babuˇ
ska
2
,
3
1
Ins i u e o Solid Mechanics, Mecha onics and Biomechanics, Facul y o Mechanical Enginee ing,
B no Uni e si y o Technology, B no 616 69, Czech Republic
2
Czech Ins i u e o In o ma ics, Robo ics and Cybe ne ics, Czech Technical Uni e si y, P ague, P ague 16 636, Czech Republic
3
Cogni i e Robo ics, Del Uni e si y o Technology, Del 2628 CD, Ne he lands
Co espondence should be add essed o Ma in B ablc; [email p o ec ed]
Recei ed 11 Decembe 2020; Re ised 11 No embe 2021; Accep ed 16 No embe 2021; Published 20 Decembe 2021
Academic Edi o : Aydin Azizi
Copy igh ©2021 Ma in B ablc e al. This is an open access a icle dis ibu ed unde he C ea i e Commons A ibu ion License,
which pe mi s un es ic ed use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed.
Rein o cemen lea ning (RL) agen s can lea n o con ol a nonlinea sys em wi hou using a model o he sys em. Howe e , ha ing
a model b ings benefi s, mainly in e ms o a educed numbe o unsuccess ul ials be o e achie ing accep able con ol pe -
o mance. Se e al modelling app oaches ha e been used in he RL domain, such as neu al ne wo ks, local linea eg ession, o
Gaussian p ocesses. In his a icle, we ocus on echniques ha ha e no been used much so a : symbolic eg ession (SR), based on
gene ic p og amming and local modelling. Using measu ed da a, symbolic eg ession yields a nonlinea , con inuous- ime analy ic
model. We benchma k wo s a e-o - he-a me hods, SNGP (single-node gene ic p og amming) and MGGP (mul igene gene ic
p og amming), agains a s anda d inc emen al local eg ession me hod called RFWR ( ecep i e field weigh ed eg ession). We
ha e in oduced modifica ions o he RFWR algo i hm o be e sui he low-dimensional con inuous- ime sys ems we a e mos ly
dealing wi h. The benchma k is a nonlinea , dynamic magne ic manipula ion sys em. The esul s show ha using he RL
amewo k and a sui able app oxima ion me hod, i is possible o design a s able con olle o such a complex sys em wi hou he
necessi y o any haphaza d lea ning. While all o he app oxima ion me hods we e success ul, MGGP achie ed he bes esul s a
he cos o highe compu a ional complexi y. Index Te ms–AI-based me hods, local linea eg ession, nonlinea sys ems,
magne ic manipula ion, model lea ning o con ol, op imal con ol, ein o cemen lea ning, symbolic eg ession.
1. In oduc ion
A ein o cemen lea ning (RL) agen in e ac s wi h he
sys em o be con olled by measu ing i s s a es and applying
ac ions acco ding o a policy so ha a gi en goal s a e is
a ained. The policy is i e a i ely adap ed in such a way ha
he agen ecei es he highes possible cumula i e ewa d,
which is a scala alue accumula ed o e ajec o ies in he
sys em’s s a e space. The ewa d associa ed wi h each
ansi ion in he s a e space is desc ibed by a p edefined
alue unc ion.
Exis ing RL algo i hms can be di ided in o c i ic-only,
ac o -only, and ac o -c i ic a ian s. The c i ic-only a ian s
op imize he alue unc ion (V- unc ion) ha is hen used o
de i e he policy; he ac o -only a ian s wo k di ec ly on
he policy op imiza ion wi hou any need o a alue
unc ion; and ac o -c i ic a ian s op imize bo h unc ions
simul aneously. An example o he ac o -only RL a ian ,
o en called Q-lea ning, can be ound in [1] and ha o he
ac o -c i ic a ian in [2].
F om a diffe en poin o iew, RL algo i hms can be also
di ided in o model-based and model- ee a ian s. Exam-
ples o bo h app oaches can be ound in [3, 4]. The model-
based a ian s include a model ep esen a ion o he sys em
o be con olled and can be p e ained in simula ion (offline)
and hen upda ed when con olling he ac ual sys em
(online). Model- ee me hods lea n online exclusi ely
h ough ial and e o . Bo h a ian s ha e hei specific
ad an ages and disad an ages. We can o en find ema ks
abou he model- ee app oach equi ing much mo e da a,
Hindawi
Complexi y
Volume 2021, A icle ID 6617309, 12 pages
h ps://doi.o g/10.1155/2021/6617309
especially in high-dimensional cases [1]. In his pape , we
employ he model-based, c i ic-only a ian wi hou any
online aining so ha we can compa e diffe en modelling
app oaches.
We ocus on wo p omising ca ego ies o app oxima ion
algo i hms: gene ic p og amming and local linea eg ession.
Ou aim is o con ibu e o he me hodology o choosing he
op imal ou o dozens o exis ing modelling algo i hms when
p esen ed wi h a specific RL ask. This p oblem a ises no only
in connec ion wi h he RL amewo k (see [5]) bu in
modelling o a dynamical sys em in gene al [6, 7].
Gene ic algo i hms (GA) and hei many a ia ions a e
well es ablished as a ool o modelling o pa ame e es i-
ma ion o dynamical sys ems [8, 9]. Howe e , gene ic
p og amming as a modelling app oach used wi hin RL is
ela i ely new and p omises good esul s wi h high-di-
mensional sys ems whe e o he app oaches ail. I c ea es a
con inuous- ime, globally nonlinea model desc ibed by an
analy ical equa ion buil o combina ions o p edefined
unc ions [10]. As i is common wi h gene ic op imiza ion
algo i hms, hese me hods end o be compu a ionally de-
manding. On he o he hand, local eg ession is a well-
es ablished modelling app oach o model-based RF agen s
whe e he model is composed o local linea models, offe ing
as and compu a ionally cheap app oxima ion. The e a e
se e al a ian s o local modelling me hods; comp ehensi e
examples o g id-based local linea model s uc u e and
da a-based local linea eg ession (LLR) a e desc ibed in
[11, 12], espec i ely. E en hough he use o local eg ession
echniques wi hin RL has been esea ched in he pas , i was
mainly based on simple, memo y-based app oxima ion
me hods such as he LLR, which is ho oughly desc ibed and
examined in [13, 14], and mo e complex inc emen al
me hods such as he ecep i e field weigh ed eg ession
(RFWR) [15, 16] o locally weigh ed p ojec ion eg ession
(LWPR) [17] we e omi ed, wi h he excep ion o [18], whe e
he RFWR algo i hm was used as a c i ic app oxima o . The
RFWR and LWPR me hods p o ide significan benefi s in
lowe memo y use and highe s abili y by employing op-
imiza ion-based (RFWR) o s a is ical (LWPR) me hods o
disco e he op imal dis ibu ion o he local models’ a eas o
alidi y, ha is, he ecep i e fields.
I is impo an o benchma k he modelling me hods
because o he la ge numbe o exis ing app oaches, which
aim a simila asks, while he e a e no simple guidelines on
he me hod choice. Also, he p esen ed algo i hms a e no
ye well es ablished wi hin he RL domain. Finally, s udying
con ol algo i hms o magne ic manipula ion sys ems has
impo ance on i s own because o i s applica ion in many
indus ial fields (medical applica ions, magne ic le i a ion
sys ems, e c.), hus leading o he wo sepa a e aims o his
pape : explo ing con ol algo i hms sui able o con ol o
p ecise magne ic manipula o sys ems and benchma king
diffe en modelling app oaches.
When dealing wi h eal magne ic manipula o sys ems,
we also need o add ess p ac ical issues ha a e o en
neglec ed in simula ions, ha is, nonlinea i ies such as ac-
ua o dead zones, sa u a ions, Coulomb ic ion, signal
delays, and so on. These p esen significan obs acles hen
implemen ing he con ol algo i hm on a eal sys em. In
some cases, he dead zone and sa u a ion p oblem can be
add essed by nonlinea o adap i e con ol laws. Fo ex-
ample, [19] shows an app oach using uzzy con ol wi h
Gaussian membe ship unc ions, which is in p ac ice simila
o he RFWR me hod, and [20] desc ibes a gain-scheduling
adap i e app oach o deal wi h in e nal sys em bounds.
Using RL o find a con ol law o a nonlinea sys em also has
he ad an age ha i can o en deal wi h such dis u bances
on i s own h ough he op imiza ion p ocess; o example,
only a limi ed ange o he ac o ou pu s may be limi ed,
which is he app oach u ilized in his pape .
In his pape , we also p esen mino adjus men s o he
RFWR algo i hm in Sec ion 3, p oposed o lowe he
compu a ional complexi y while p ese ing s abili y when
wo king wi h low-dimensional p oblems.
2. Me hods
2.1. Magne ic Manipula o . Gene ic p og amming was al-
eady applied o nonlinea sys ems like an in e ed pen-
dulum o a collabo a i e obo [2, 10, 21]. To u he
in es iga e he app oxima ion capabili ies o hese me hods,
we use a diffe en sys em–a magne ic manipula o (Mag-
man). This sys em consis s o ou coils ha a e indepen-
den ly ope a ed by sepa a e cu en con olle s and a s eel
ball ha can mo e eely o e he coils; see Figu e 1. To
ensu e ha he ball mo es only in he measu ed di ec ion
wi h limi s on he edges, i is placed in a g oo e wi h 10 mm
in size. In his case, we decided o limi he sys em o he fi s
wo coils only, as a sys em wi h ou inpu s is much mo e
complex in e ms o he RL compu a ional complexi y, while
i does no en ich he sys em wi h diffe en nonlinea i ies as
i only spa ially epea s he same kind o nonlinea
beha iou .
The s eel ball can be posi ioned by p ope ly con olling
he cu en and he eby he magne ic o ce o he coils. The
magne ic o ce a coil exe s on he ball is highly dependen
on he dis ance o he ball om he coil’s cen e, which
in oduces a significan nonmono onic nonlinea i y
[22–24].
All expe imen s and simula ion we e sc ip ed in
MATLAB. The coil cu en s a e con olled by s abilized
cu en sou ce modules, which communica e ha MATLAB
h ough a USB/RS232 anscei e using he i ual COM
po (VCP) p o ocol on Windows OS. As he ball posi ion is
measu ed wi h a lase senso wi h analog ( ol age) ou pu ,
he Humuso MF634 IO ca d was used o measu e he signal
in eal ime om he MATLAB en i onmen wi h a sam-
pling pe iod o 5 ms. E en hough he Window OS is no an
RTOS, wi h his sampling equency, he pe iod ji e is
negligible (below 0.1%), and hus, he sys em can be con-
side ed eal ime.
Table 1 lis s he pa ame e s o he magne ic manipula o
we use in ou expe imen s. Wi h he ask being a p ecise
posi ioning o an objec in a magne ic field, simila concep s
can be ound in many eal-wo ld applica ions, o example,
magle , mic o obo s, con ac less s i ing o chemicals, and
so on.
2Complexi y
App oxima e equa ions o mo ion in e ed using he fi s
p inciple me hod can be ound in [24]. The sys em pa-
ame e s we e ei he measu ed di ec ly o es ima ed using
MATLAB pa ame e es ima ion oolbox based on measu ed
da a.
Gene ally, he sys em can be desc ibed by a con inuous-
ime, nonlinea s a e-space model as ollows:
x
.� (x, u),(1)
whe e x� [x, _
x]⊤is he s a e ec o composed o he po-
si ion xand eloci y _
xo he ball, o ming he con inuous
sys em s a e space x∈X⊂R2;x
.in _
X⊂R2is he s a e
ec o de i a i e; and u� [u1, u2]⊤is he inpu (ac ion)
ec o composed o he coil cu en s. u1, u2 o m he sys em
inpu space u∈U⊂R2. The nonlinea ec o unc ion
:X×U⟶_
X hus desc ibes he sys em dynamics.
In his pape , by modelling he sys em, we mean ap-
p oxima ing he unde lying eal unc ion using a ious
me hods, which all build upon expe imen ally measu ed
inpu -ou pu da a. Each da a poin is o med by co e-
sponding assumed inpu s and ou pu s o he unc ion
–(x
.
k,xk,uk). In p ac ice, hese da a poin s a e co up ed by
noise and o he dis u bances ha a e assumed o be wi h
ze o means.
2.2. SNGP. Single-node gene ic p og amming (SNGP) is a
g aph-based gene ic p og amming algo i hm e ol ing a
popula ion o ganized as an o de ed linea a ay o in e -
linked indi iduals, each ep esen ing a single p og am node
[2, 10, 21]. Gene ally, symbolic eg ession algo i hms y o
find a model in he o m o an analy ical exp ession o a
gi en da a se by o ming and e ol ing he exp ession ou o
elemen al unc ions and ope a ions. In ou case, he algo-
i hm is based on he assump ion ha he nonlinea unc ion
in (1) can be efficien ly app oxima ed by he ollowing
equa ion:
x
.� (x, u) � 􏽘
n ,ns
i,j
βi ixj, uj
􏼐 􏼑,(2)
whe e he nonlinea unc ion i, called he ea u e, is de-
eloped by means o gene ic p og amming wi h n being he
maximum numbe o ea u es, nsnumbe o s a es, and he
coefficien s βies ima ed by he leas -squa es me hod. The
ea u es a e cons uc ed om a lis o elemen a y unc ions
ha a e assumed o be able o p oduce he equi ed fi ing
app oxima ion o he p esen ed da a. The ea u es can be
combined by common ope a o s o nes ed, bu he maximal
dep h o he exp ession is limi ed o a oid o e fi ing. The
symbolic model is e ol ed so ha he mean-squa ed e o
o e he aining da a is minimized.
2.3. MGGP. The second GP algo i hm we used is called
mul igene gene ic p og amming (MGGP). As opposed o
SNGP, i combines he ea u es defined also by 2 in o ee-
like s uc u ed exp essions called genes. The final exp ession
is o med by a linea combina ion o hese genes, which ac
as he indi idual ea u es in equa ion (2). The pa ame e s o
his op-le el linea combina ion a e again es ima ed
h ough leas squa es. Fu he de ails abou he algo i hm
can be ound in [25]. The ac ual MGGP implemen a ion we
used is ex ended wi h linea combina ions o ea u es [26]
ha enable he algo i hm o find affine ans o ma ions o
he ea u e space ia a backp opaga ion-like echnique, hus
making i easie o he d i ing gene ic p og amming al-
go i hm o app oxima e he nonlinea i ies.
2.4. Recep i e Field Weigh ed Reg ession. Recep i e field
weigh ed eg ession (RFWR) is an inc emen al app oxi-
ma ion me hod ha c ea es a se o local linea models and
he co esponding Gaussian basis unc ions called he e-
cep i e fields and g adually upda es hem o fi he inpu -
ou pu da a. The se o local linea models is upda ed wi h
new da a poin s (called he que y poin s) using a weigh ed
a ian o he ecu si e leas squa es (RLS) me hod and he
basis unc ions a e upda ed h ough a g adien sea ch wi h
he help o heu is ic decision ules. I can con inually im-
p o e he se o models while s ill p o iding he bes es i-
ma ion o he app oxima ed unc ion a each que y poin
based on he p e iously p o ided da a. The o iginal algo-
i hm, fi s p esen ed in [15, 16], which is he basis we build
upon, can be bes desc ibed by he ollowing pseudocode:
(1) Fo each new que y poin (x
.
k,xk,uk)
(2) Fo each exis ing local model
(3) Calcula e model weigh wacco ding o (4)
(4) I w>ac i a ion limi wac
(5) Upda e model pa ame e s using RLS acco ding o
(6) and (7)
(6) Upda e he co esponding ecep i e field using (12)
and (14)
(7) End
(8) End
Figu e 1: A schema ic d awing o he magne ic manipula o
sys em.
Table 1: Magne ic manipula o pa ame e s.
Pa ame e Value Uni
Ball mass 53 g
Ball diame e 20 mm
Dis ance be ween edge coils and he ball posi ion
limi 20 mm
Dis ance be ween coils 25 mm
Maximal coil cu en 0.6 A
Sampling pe iod 0.005 s
Complexi y 3
(9) I no model was ac i a ed
(10) Place a new model a he que y poin using (15)
(11) Else i wo o mo e models we e ac i a ed wi h
weigh w>p uning limi wp un
(12) P une he model wi h he smalle ecep i e field
(13) End
(14) Calcula e he model ou pu as a weigh ed a e age o
he ac i a ed local models
(15) End
Usually, he ecep i e field ac i a ion limi is se as
wac �0.001. This pa ame e ep esen s he weigh limi
o a local model o be upda ed acco ding o he new da a
and o be included in he ou pu es ima ion h ough a
weigh ed a e age wi h ano he ac i a ed model. The
p uning limi is usually se as wp un �0.7, which ep e-
sen s he highes accep able o e lap o neighbou ing e-
cep i e fields.
The RFWR a ian desc ibed in his pape ollows he
main ou line o he o iginal algo i hm [15] wi h se e al
adjus men s and imp o emen s o he sake o s abili y
and compu a ional complexi y o low-dimensional
p oblems. This mainly conce ns he ules o adding new
local models, adjus ing hei ecep i e fields, and gen-
e alizing he algo i hm in a way ha he ecep i e fields
a e placed and op imized in a lowe numbe o di-
mensions han he o de o he models. This is especially
use ul in cases when he nonlinea i ies a e significan
mainly in one o wo dimensions o he s a e space o he
sys em. This algo i hm, in i s o iginal implemen a ion, is
success ully being used o app oxima e in e se models o
nonlinea sys ems o be used as a eed o wa d com-
pensa o [27, 28]. Figu e 2 shows an example app oxi-
ma ion o a complex uni a ia e nonlinea unc ion by he
RFWR algo i hm.
Each o he local models is ep esen ed by a pa ame e
ec o b� [b1, b2,. . . , bn]⊤. Wi h he inpu ec o (a que y
poin ) Xq� [x1, x2,. . . , xn]⊤, he ou pu yqis calcula ed by
yq�X⊤
qb. (3)
The weigh wo a local model a a que y poin Xqis
de e mined by i s Gaussian ecep i e field as ollows:
w(X) � e−1/2 Xq−c
 􏼁⊤C−1Xq−c
 􏼁,(4)
wi h c� [c1, c2,. . . , cn] he ec o o model cen e coo di-
na es and C−1 he dis ance inducing ma ix o he basis
unc ion ( ecep i e field). The o e all ou pu is hen cal-
cula ed as a weigh ed a e age o he ou pu s o he ac i a ed
local models.
The ou pu es ima e o he se o local models and hei
ecep i e fields is calcula ed by he ollowing equa ion:
􏽢
y�1
􏽐n
i�1wi􏽘
n
i�1
wiyi.(5)
We modified he o iginal RFWR algo i hm desc ibed in
[15] o be used o low-dimensional p oblems. These
modifica ions consis o he ollowing:
(1) P ecise placemen o new ecep i e fields ha akes
in o accoun he loca ion and dimensions o he
exis ing su ounding ecep i e fields (see sec ion 3.3)
(2) Heu is ic ules o s able upda ing o he ecep i e
fields (see sec ion 3.1)
(3) Recep i e fields can be dis ibu ed along a lowe
numbe o dimensions han he dimension o he
da a space (see sec ion 3.2)
2.4.1. Upda ing pa ame e s o Local Models. When a new
que y poin is acqui ed, he ac i a ed local models a e
upda ed using he ecu si e leas -squa es algo i hm (RLS)
acco ding o
Pn+1�1
λPn−PnXT
qXqPn
λ/w+XqPnXT
q
⎛
⎝⎞
⎠,
bn+1�bn+wPn+1XT
qe,
e�yq−Xqb,
(6)
x
y
Nonlinea unc ion app oxima ion by RFWR
Nonlinea unc ion
Local models
x
Recep i e ield weigh
Recep i e ields
Figu e 2: Example o RFWR app oxima ion o a complex non-
linea unc ion and he dis ibu ion o ecep i e fields.
4Complexi y
whe e Pis he co a iance ma ix o he es ima e, λis a
o ge ing pa ame e , and yqis he acqui ed ou pu o he
ac ual sys em s a e Xqcalled he que y poin . The co a iance
ma ix Pneeds is usually ini ialized as a diagonal ma ix.
2.4.2. Upda ing dimensions o Basis Func ions. To a oid
calcula ing he ma ix in e sion in (4) o e e y local model,
an uppe iangula ma ix Mis used ins ead o C. Because
o symme y and posi i e defini eness, hese ma ices ela e
acco ding o
C−1�MTM. (7)
To upda e he ecep i e field, we upda e Musing a
g adien -descen op imiza ion
Mi+1�Mi−αzJ(M)
zM,(8)
o he cos unc ion Jas ollows:
J�1
􏽐n
i�1wi􏽘
n
i�1
wiyq−yi
􏼐 􏼑2,(9)
whe e wiis he ac i a ed ecep i e field weigh , yiis he
es ima ed ou pu o he espec i e model a he que y poin
(yq;Xq), and nis he numbe o local models. The pa ame e
αis he g adien op imiza ion s ep size. As he calcula ion o
he cos unc ion Jacco ding o (9) is compu a ionally e y
complex, we simplified he op imiza ion algo i hm h ough
a se o heu is ic decision ules and implemen ed he op-
imiza ion as ollows:
Mi+1�Mi−αpzw(M)
zM
zw(M)
zM�
ze−1/2 Xq−c
 􏼁⊤MTM Xq−c
 􏼁
􏼒 􏼓
zM�
� − Xq−c
􏼐 􏼑⊤M Xq−c
􏼐 􏼑e−1
2Xq−c
􏼐 􏼑⊤MTM Xq−c
􏼐 􏼑.
(10)
This implemen a ion in oduces a pa ame e p, which is
an exp ession o a simple heu is ic o decide whe he he
alue o a basis unc ion (weigh ) a he ac ual que y poin
should be inc eased o dec eased. This enables o s op
upda ing he dis ance inducing ma ix when a p ecision
c i e ion is me and o limi he maximal numbe o local
models o a oid o e fi ing.
Pa ame e pcan be de e mined by a ious decision
ules. A simple ye effec i e se , which was used in his
esea ch, can be c ea ed by using a long- e m (cumula ed
o e ime) MSE o a pa icula model acco ding o he da a
poin s, which can be desc ibed by
p�−1,i MSE >MSElim
1,i MSE <MSElim
􏼨(11)
2.4.3. Adding New Local Models. Du ing he op imiza ion
p ocess, i is possible ha no model exceeds he ac i a ion
limi wac . In such a case, a new local model wi h a ecep i e
field is added o he app oxima ion se . The cen e o he
ecep i e field is au oma ically placed a he ac ual que y
poin , and he model pa ame e s a e ini ialized o fi he
measu ed ou pu o he app oxima ed sys em. Wha needs o
be de e mined is he a ea in he s a e space ha should be
co e ed by he newly c ea ed ecep i e field. The o iginal
algo i hm uses a de aul diagonal dis ance inducing ma ix o
e e y local model. Howe e , an op imal dis ance inducing
ma ix can be de e mined. In ui i ely, he new ecep i e field
should co e he gap be ween he al eady exis ing models. The
dis ance inducing ma ix should be ini ialized as a diagonal
ma ix wi h pa ame e s ha ensu e ha he new ecep i e
field does no o e lap wi h any exis ing one mo e han a p ese
limi . In ou case, he limi was se o 0.5wp un. Since his
would be a complex op imiza ion ask no sui able o eal-
ime calcula ion, we simplified he c i e ion so ha he
maximal o e lapping weigh o wo models is analyzed only
o e he line segmen connec ing hei cen es. In ha case,
he dis ance pa ame e o ini ializing he dis ance inducing
ma ix can be de e mined by he ollowing equa ion.whe e
i�cn−ciis a ec o be ween he new cen e cnand he
cen e o a neighbou ing ecep i e field ci. A wo-dimensional
example is shown in Figu e 3. This me hod yields a be e
es ima e o he dis ance inducing ma ix o he new ecep i e
field han he fixed ini ial dimension ma ix in he o iginal
algo i hm as i equi es ewe i e a ions o s abilize and o
co e he gap be ween neighbou ing ecep i e fields.
di� − 2 log wp un/2
􏼐 􏼑
i
􏼌􏼌􏼌􏼌􏼌􏼌􏼌􏼌−����������������������
−2 log wp un/2
􏼐 􏼑/ iMT
iMi i
􏽱,(12)
The dis ance pa ame e dihas o be calcula ed o e e y
exis ing local model, and he minimal dis ance dmin is used
o ini ialize he dis ance inducing ma ix acco ding o
M0
n�I����
dmin
􏽱,(13)
whe e Iis a uni y ma ix o he co esponding o de .
In he specific case o he magne ic manipula o , he
inpu s o he local models would co espond o (x, u)and
he ou pu o x
..
2.5.Rein o cemen Lea ning. Conside he ollowing disc e e
de e minis ic s a e-space model o a sys em o be con olled:
xk+1� xk, uk
 􏼁,(14)
whe e k∈Zdeno es disc e e ime ins an s,
xk, xk+1∈X⊂Rnis he s a e ec o , and uk∈U⊂Rmis he
inpu ec o . An RL agen lea ns o con ol he sys em so ha
i achie es he maximal cumula ed ewa d on a ajec o y
om he ini ial s a e o he desi ed s a e [10]. A each s a e
ansi ion, as desc ibed by (14), he agen ecei es a scala
ewa d acco ding o
Complexi y 5

�ρxk, uk, xk−1
 􏼁.(15)
The ewa d unc ion ρis usually based on he dis ance o
he cu en s a e o he goal s a e. The op imal con ol law,
called he policy, π:X⟶Uis de e mined as ollows such
ha i maximizes he cumula i e ewa d, called he e u n:
Rπ�E􏽘
∞
k�0
ckρxk,πxk
 􏼁, xk+1
 􏼁
⎧
⎨
⎩⎫
⎬
⎭,(16)
whe e c∈(0,1)is he discoun ac o and he ini ial s a e x0
is selec ed om he s a e space domain X. The e u n o any
pe missible ini ial s a e xis cap u ed by he alue unc ion
V:X⟶Rdefined as ollows:
V(x) � E􏽘
∞
k�0
ckρxk,πxk
 􏼁, xk+1
 􏼁
⎧
⎨
⎩⎫
⎬
⎭, x0�x. (17)
An app oxima ion o he op imal V- unc ion 􏽢
V(x)can
be ound by sol ing he Bellman equa ion as ollows:
􏽢
V(x) � maxu∈U[ρ(x, π(x), (x, u)) + c􏽢
V( (x, u))].
(18)
The op imal ac ion can be ound as he ac ion ha s ee s
he sys em o a s a e wi h maximal alue [21]. This co e-
sponds o maximiza ion o he igh -hand side o (18):
u�a g max
u′∈U
ρx, u′, x, u′
 􏼁 􏼁+cV x, u′
 􏼁 􏼁􏼂 􏼃.(19)
3. Expe imen al Resul s
We p epa ed aining and alida ion I/O da a se s measu ed
on he magne ic manipula o wi h andom inpu signals as a
lis o da a poin s in he o m (x
.
k, xk, uk). The andom inpu
signals (coil cu en s) we e gene a ed in he way ha only
one coil was ac i e a a ime ha elimina ed possible elec-
omagne ic in e ac ions be ween hem ( he coil cu en was
con olled by an HW-based cu en eedback con olle
module ende ing he ansien imes negligible). Figu e 4
shows an example o a aining da a se .
E en hough he ball’s posi ion measu emen is e y
p ecise, i s ill con ains significan noise. Fo ha eason, he
ime-domain de i a i es o he posi ion ( eloci y and ac-
cele a ion) needed o he dynamic model app oxima ion
we e de e mined using he Sa i zky–Golay fil e , which is an
FIR fil e based on leas -squa es polynomial app oxima ion
able o pe o m nume ical diffe en ia ion while fil e ing he
noise simul aneously [29, 30].
Especially o he RFWR implemen a ion, i is impo an
o no e ha he sys em’s nonlinea i y is mainly significan
along he posi ion o he ball and he sys em can be seen as
linea in pa ame e s along he o he dimensions (accele -
a ion and eloci y). In his case, he gene al model (1) can be
ea anged as ollows:
€
x� (x, u) + b0sign(_
x) + b1_
x+b2_
x2,(20)
whe e he unc ion (x, u) ep esen s he significan non-
linea i y sui able o local app oxima ion, he e m
b0sign(_
x) ep esen s a simple model o d y ic ion, he e m
b1_
x ep esen s he iscous ic ion, and he las e m b2_
x2
models nonlinea damping caused by elec omagne ic in-
duc ion influencing he s eel ball while mo ing apidly
h ough a magne ic field. Despi e being nonlinea , all o he
e ms a e linea in hei pa ame e s and can be modelled
globally, which means ha he local models sha e pa am-
e e s b0 h ough b2.
The e m b0sign(_
x)in he (20) is qui e impo an in
p ac ical si ua ions whe e Coulomb ic ion is no negligible.
The sign unc ion is o en being used o app oxima e he
effec s o Coulomb ic ion whe e e he e is no significan
s ic ion (diffe ence be ween s a ic and dynamic ic ion
effec s). The e a e be e app oxima ions o simula ion
pu poses, o example, he sigmoid unc ion; howe e , mos
o hem a e no linea in pa ame e s and hus no applicable
o RLS pa ame e es ima ion.
The same da a se was p esen ed o all o he app oxi-
ma ion me hods (RFWR, SNGP, and MGGP). Due o he
s ochas ic na u e o he wo algo i hms based on gene ic
p og amming, he same p ocess was epea ed wi h diffe en
pseudo andom seeds. O e all, 30 uns o SNGP and MGGP
and 1 un o RFWR we e made. Table 2 shows he summa y
o he MSE esul s.
Since he MSE o he models wi h espec o he aining
se is no sufficien o decide which models a e be e , wo
Recep i e ields - 2D
Recep i e ields - 1D sec ion
Old ecep i e ield
New ecep i e ield
X2
X1
ci
ci
wmax
w
cn
X
cn
SECTION PLANE
Figu e 3: Demons a ion o adding a new e e ence field o he se
o RFWR local models and ma ching i s dimensions o fi he
su ounding e e ence fields.
6Complexi y
sepa a e da a se s we e measu ed: one was used o he
aining o he models and one o alida ion. Howe e ,
since he magne ic manipula o is no open-loop s able, he
common open-loop alida ion is no sui able, as e e y model
di e ges quickly e en hough he pa ame e s may be close o
ideal due o e o s in oduced by nume ical in eg a ion.
The e o e, he models we e alida ed in se e al s eps-ahead
p edic ion mode. As we also suspec ed ha only one-
sample-ahead alida ion could be influenced by he
emaining noise in he measu ed signals, we alida ed he
model o 1, 3, 5, 10, 50, 100, and 250 samples ahead. We
used he fi e-s ep-ahead p edic ion (5-SAP) as a baseline o
selec ing he bes models o u he expe imen s. The eason
o choose fi e samples is based on an expe imen ally
0 0.5
(s)
1 1.5 2
0 0.5
(s)
1 1.5 2
0
20
40
60
80
x (mm)
U (A)
Measu ed
0
0.2
0.4
0.6
Coil 1
Coil 2
Figu e 4: Random inpu signal (gene alized bina y noise wi h andom coil swi ching) wi h only one coil ac i e a a ime o aining and
alida ion se s gene a ion.
Table 2: S a is ical esul s o he bes models.
Model T aining MSE 5-SAP alida ion MSE Con ol MSE
SNGP 3 6.53 ·10−25.71 ·10−12 4.02 ·10−5
SNGP 7 6.78 ·10−25.77 ·10−12 2.46 ·10−4
SNGP 10 5.89 ·10−25.36 ·10−12 4.23 ·10−5
SNGP 13 7.28 ·10−26.37 ·10−12 4.28 ·10−5
SNGP 17 7.48 ·10−25.84 ·10−12 1.28 ·10−4
SNGP 18 7.23 ·10−25.79 ·10−12 2.06 ·10−4
SNGP 21 7.65 ·10−26.41 ·10−12 3.40 ·10−3
SNGP 27 5.32 ·10−26.57 ·10−12 1.16 ·10−4
SNGP 29 5.85 ·10−26.19 ·10−12 9.60 ·10−5
SNGP 30 6.93 ·10−25.99 ·10−12 3.75 ·10−5
MGGP 5 5.94 ·10−25.49 ·10−12 6.34 ·10−5
MGGP 8 4.87 ·10−25.13 ·10−12 4.14 ·10−5
MGGP 9 5.69 ·10−26.18 ·10−12 3.80 ·10−5
MGGP 13 6.59 ·10−25.67 ·10−12 4.13 ·10−5
MGGP 16 5.96 ·10−25.29 ·10−12 6.59 ·10−5
MGGP 17 5.99 ·10−25.08 ·10−12 5.46 ·10−5
MGGP 18 7.35 ·10−26.28 ·10−12 1.50 ·10−4
MGGP 21 5.60 ·10−25.92 ·10−12 6.74 ·10−5
MGGP 22 5.55 ·10−25.74 ·10−12 8.84 ·10−5
MGGP 23 6.00 ·10−25.82 ·10−12 3.56 ·10−5
RFWR 6.25 ·10−21.24 ·10−11 6.57 ·10−5
Complexi y 7
alida ed assump ion ha sho e in e als do no show he
model’s imp ecision and longe in e als cause e en e y
p ecise models o di e ge andomly.
The n-s ep-ahead p edic ion alida ion is based on a
mo ing ame o nconsecu i e da a poin s, whe e he fi s
da a poin is applied as he ini ial condi ion o nume ical
in eg a ion (using he ode45 sol e ) o he dynamical model
being es ed. When he simula ion eaches he n- h s ep, an
MSE esidual is calcula ed be ween he co esponding da a
poin and he model p edic ion. The esul ing model ali-
da ion me ic is hen calcula ed as he sum o esiduals o e
each o he p edic ion ames.
Figu e 5 shows examples o he models cons uc ed by
each algo i hm. As he ansi ion model o he sys em is
ou -dimensional, o isualiza ion pu poses, he figu es
show a wo-dimensional si ua ion o he inpu ec o se o
u� [0,0.6]T. This co esponds o he si ua ion when he fi s
coil is u ned off and he cu en h ough he second coil is
0.2
0
-4
SNGP model
-3
-2
0
-1
0.02
x (m)
0
-0.2
1
0.04
2
0.06
3
0.08
00.02
x (m)
x (m)
0.04 0.06 0.08
0.2
0
MGGP model
-0.2
0.2
0
RFWR model
00.02 -0.2
0.04 0.06 0.08
x
·· (m·s-2)
-4
-3
-2
-1
0
1
2
3
x
·· (m·s-2)
-4
-3
-2
-1
0
1
2
3
x
·· (m·s-2)
x
· (m·s-1)
x
· (m·s-1)
x
· (m·s-1)
Figu e 5: Compa ison o app oxima ed dynamic models o he SNGP, MGGP, and RFWR me hod, bes -fi ing models om each ca ego y.
The 3D isualiza ion is shown o cons an inpu u� [0,0.6]T.
x (m)
0.1
-25
-20
-15
V (–)
V (–)
V (–)
SNGP model
0
-10
0.02
-5
-0.1
0
0.04
x (m)
x (m)
0.02
0.04
0.1
-25
-20
-15
MGGP model
0
-10
-5
-0.1
0
0.1
-25
-20
-15
RFWR model
0
-10
0.02
-5
-0.1
0
0.04
x
· (m·s
-1
)
x
· (m·s
-1
)
x
· (m·s
-1
)
Figu e 6: Compa ison o V- unc ions based on he bes fi ing models om each o he app oxima ion me hods: SNGP, MGGP, and RFWR.
8Complexi y
0.02 0.04
-0.1
0
0.1
SNGP - Coil 1 SNGP - Coil 2
MGGP - Coil 1 MGGP - Coil 2
RFWR - Coil 1 RFWR - Coil 2
x
· (m·s-1)
-0.1
0
0.1
x
· (m·s-1)
-0.1
0
0.1
x
· (m·s-1)
-0.1
0
0.1
x
· (m·s-1)
-0.1
0
0.1
x
· (m·s-1)
-0.1
0
0.1
x
· (m·s-1)
x (m)
0.02 0.04
x (m)
0.02 0.04
x (m)
0.02 0.04
x (m)
0.02 0.04
x (m)
0.02 0.04
x (m)
Figu e 7: Compa ison o policies based on he bes fi ing models o he magne ic manipula o om each o he app oxima ion me hods.
05
(s)
x (mm)
U (A)
10 15
05
(s)
10 15
0
10
20
30
40
0
0.1
0.2
0.3
0.4
Coil 1
Coil 2
Measu ed
Desi ed
Figu e 8: Example o he con ol pe o mance o a con olle based on he SNGP 10 model.
Complexi y 9