Quantum reinforcement learning: foundations, algorithms, applications

Uni e sidade do Minho
Escola de Engenha ia
And é Sequei a
Quan um Rein o cemen Lea ning:
Founda ions, algo i hms, applica ions
Janua y, 2025
Quan um Rein o cemen Lea ning:
Founda ions, algo i hms, applica ions
And é Sequei a
UMinho |2025
ii
Uni e sidade do Minho
Escola de Engenha ia
And é Sequei a
Quan um Rein o cemen Lea ning:
Founda ions, algo i hms, applica ions
Doc o a e Thesis
Doc o a e in In o ma ics (PDIn )
Wo k de eloped unde he supe ision o :
Luis Paulo San os
Janua y, 2025
COPYRIGHT AND TERMS OF USE OF THIS WORK BY A THIRD PARTY
This is academic wo k ha can be used by hi d pa ies as long as in e na ionally accep ed ules and good
p ac ices ega ding copy igh and ela ed igh s a e espec ed.
Acco dingly, his wo k may be used unde he license p o ided below.
I he use needs pe mission o make use o he wo k unde condi ions no p o ided o in he indica ed
licensing, hey should con ac he au ho h ough he Reposi o iUM o Uni e sidade do Minho.
License g an ed o he use s o his wo k
C ea i e Commons A ibu ion-NonComme cial-Sha eAlike 4.0 In e na ional
CC BY-NC-SA 4.0
h ps://c ea i ecommons.o g/licenses/by-nc-sa/4.0/deed.en
This documen was c ea ed wi h he (pd /Xe/Lua)L
A
T
EX p ocesso and he NOVA hesis empla e ( 7.1.18) [120].
ii
Acknowledgemen s
Comple ing he PhD has been an inc edible jou ney, one ha I could no ha e na iga ed alone. I am
deeply indeb ed o many indi iduals whose suppo and encou agemen ha e been essen ial h oughou
his p ocess.
Fi s and o emos , I would like o exp ess my p o ound g a i ude o my supe iso s, P o esso Luis Paulo
San os and P o esso Luis Soa es Ba bosa. Luis Paulo, you ha e been mo e han jus a supe iso ; you
ha e been a ue iend. You guidance, wisdom, and unwa e ing suppo ha e been in aluable and
changed me o e e . Luis Ba bosa, you insigh ul eedback and men o ship ha e g ea ly en iched my
esea ch, and I am deeply hank ul o you suppo and belie in my po en ial.
To my amily, you lo e and suppo ha e been he ounda ion upon which his jou ney has been buil . To
my pa en s, hank you o you unwa e ing belie in me and o he coun less sac i ices you’ e made o
help me achie e my d eams. To my b o he , you humo and he coun less laughs you p o ided we e a
b ea h o esh ai du ing mos challenging imes.
A hea el hank you o Ri a, my lo e and bes iend. You kindness, pa ience, and cons an suppo ha e
been my ancho h ough he ups and downs o many yea s. You kep eminded me o he impo ance o
balance in li e, pulling me ou o so many momen s o in e nal u moil.
I also wan o hank all my iends who walked his jou ney wi h me. Sha ing his expe ience wi h you el
like being cellma es in he PhD p ison—se ing ime oge he , plo ing ou escape, and making he ha d
days jus a bi mo e bea able wi h humo and cama ade ie.
Las ly, I would like also o acknowledge my ins i u ions, he Uni e si y o Minho and he High-Assu ance
So wa e Labo a o y - HASLAB INESCTEC. Thank you o p o iding me wi h all he necessa y condi ions o
sucess ully comple e his hesis. This wo k is pa ially inanced by Na ional Funds h ough he Po uguese
unding agency, FCT - Fundação pa a a Ciência e a Tecnologia, wi hin p ojec UIDB/50014/2020 (DOI
10.54499/UIDB/50014/2020), and inanced by Na ional Funds h ough FCT - Fundação pa a a Ciência
e a Tecnologia, I.P. (Po uguese Founda ion o Science and Technology) wi hin he p ojec IBEX, wi h
e e ence PTDC/CCI-COM/4280/2021 (DOI 10.54499/PTDC/CCI-COM/4280/2021).
iii

STATEMENT OF INTEGRITY
I he eby decla e ha ing conduc ed his academic wo k wi h in eg i y. I con i m ha I ha e no used pla-
gia ism o any o m o undue use o in o ma ion o alsi ica ion o esul s along he p ocess leading o i s
elabo a ion.
I u he decla e ha I ha e ully acknowledged he Code o E hical Conduc o he Uni e sidade do Minho.
(And é Sequei a)
i
Resumo
Os ápidos a anços na compu ação quân ica ab i am no as possibilidades pa a o ap imo amen o da
ap endizagem po e o ço (RL), especialmen e a a és de ci cui os quân icos pa ame izados (PQCs) como
ap oximado es de unções em algo i mos híb idos quân ico-clássicos. Es a disse ação abo da desa ios
e opo unidades no uso de PQCs pa a RL, explo ando o seu design, eino e po encial pa a alcança
an agem quân ica. A p imei a pa e in es iga a exp essi idade e capacidade de eino de polí icas base-
adas em PQCs. Técnicas como ein odução de dados e escalamen o de en adas/saídas demons am
que os PQCs podem e desempenho equi alen e ou supe io ao de edes neu ais clássicas, equen e-
men e com menos pa âme os. No en an o, a capacidade de eino é limi ada pelo enómeno de Ba en
Pla eau (BP), onde g adien es nulos di icul am a o imização. Es a disse ação iden i ica condições pa a
mi iga BPs, ga an indo eino em ci cui os de p o undidade loga í mica com medições locais. Com base
nisso, a segunda pa e explo a écnicas de o imização pa a RL baseado em PQCs. Uma compa ação
en e g adien es na u ais quân icos (QNG), com ma iz de Fishe quân ica (QFIM), e mé odos com ma iz
de Fishe clássica (CFIM) e ela comp omissos en e o imizações no espaço de es ados e de polí icas.
Embo a QNGs o e eçam maio es abilidade, seus bene ícios ace à CFIM dependem do con ex o. Pa a
equilib a eino e icien e e in a abilidade clássica, a e cei a pa e p opõe polí icas de PQCs baseadas
em ci cui os com ge ado es comu a i os. Es es e i am o enómeno de BP enquan o pe manecem di íceis
de simula classicamen e, ep esen ando um caminho p omisso pa a alcança an agem quân ica. A
pa e inal in eg a écnicas ole an es a alhas com mé odos baseados em PQCs, p opondo uma es u u a
pa a alcança an agem quân ica p o á el em ambien es pa cialmen e obse á eis, com demons ação
de acele ação quad á ica na complexidade amos al pa a a ualizações de c enças ia in e ência Baye-
siana quân ica. Es a disse ação con ibui pa a a comp eensão do RL baseado em PQCs, o e ecendo
pe spe i as sob e o seu design, eino e o imização, des acando o po encial da compu ação quân ica
pa a e oluciona o RL e iabiliza agen es quân ico-ap imo ados escalá eis.
Pala as-cha e: Ap endizagem po Re o ço Quân ica, A ualização Quân ica de Con icções
Ba en Pla-
eaus
, G adien es Na u ais Quân icos, Ins an âneos Polinomiais Quân icos
Abs ac
Quan um Rein o cemen Lea ning: Founda ions, algo i hms, ap-
plica ions
The apid ad ancemen s in quan um compu ing ha e opened new a enues o enhancing ein o cemen
lea ning (RL), pa icula ly h ough he use o pa ame e ized quan um ci cui s (PQCs) as unc ion app oxi-
ma o s in hyb id quan um-classical algo i hms. This disse a ion add esses c i ical challenges and oppo -
uni ies in le e aging PQCs o RL, explo ing hei design, ainabili y, and po en ial o achie ing quan um
ad an age. The i s pa o his wo k in es iga es he exp essi i y and ainabili y o PQC-based policies.
By in oducing echniques such as da a euploading, inpu scaling, and ou pu scaling, we demons a e
ha PQCs can achie e pe o mance on pa wi h o supe io o classical neu al ne wo ks, o en wi h ewe
ainable pa ame e s. Howe e , PQC ainabili y is hinde ed by he Ba en Pla eau (BP) phenomenon,
whe e anishing g adien s impede op imiza ion. This disse a ion iden i ies condi ions unde which BPs
can be mi iga ed, ensu ing ainabili y in loga i hmic-dep h ci cui s wi h local measu emen s. Building
on hese indings, he second pa explo es op imiza ion echniques o PQC-based RL agen s. A c i i-
cal compa ison o quan um na u al g adien s (QNG), le e aging he quan um Fishe in o ma ion ma ix
(QFIM), and classical Fishe in o ma ion ma ix (CFIM)-based upda es e eals adeo s in s a e-space
e sus policy-space op imiza ions. While QNG p o ides s abili y and in o med upda es, i s bene i s o e
CFIM-based me hods a e con ex -dependen . To add ess he balance be ween ainabili y and classical in-
ac abili y, he hi d pa p oposes PQC-based policies de i ed om commu ing-gene a o ci cui s. These
ci cui s a e designed o be e icien ly ainable, a oiding he BP phenomenon, while emaining classically
ha d o simula e. These p esen a p omising ou e owa d achie ing quan um ad an age in RL. Finally,
a aul - ole an quan um amewo k was p oposed o achie e p o able quan um ad an age in pa ially
obse able en i onmen s, suppo ed by a demons a ed quad a ic speedup in belie upda es using quan-
um Bayesian in e ence. This disse a ion con ibu es o he ounda ional unde s anding o PQC-based
RL, o e ing insigh s in o hei design, ainabili y, and op imiza ion. The esul s highligh he po en ial o
quan um compu ing o e olu ionize RL, pa ing he way o scalable and ad an ageous quan um-enhanced
agen s.
Keywo ds: Ba en Pla eaus, Ins an aneous Quan um Polynomial, Quan um Na u al G adien s, Quan um
Policy G adien s, Quan um Rein o cemen Lea ning
i
Con en s
Lis o Figu es xi
Lis o Tables x iii
Lis o Algo i hms xx
Ac onyms xxii
1 In oduc ion 1
1.1 Mo i a ion..................................... 2
1.2 Rela edwo k.................................... 5
1.3 Thesis s uc u e and synopsis . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Lis o publica ions................................. 9
I Backg ound 10
2 Quan um In o ma ion and Compu a ion 11
2.1 S a espace .................................... 11
2.2 Timee olu ion................................... 13
2.3 Composi e sys ems and en anglemen . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 Measu emen s and expec a ion alues . . . . . . . . . . . . . . . . . . . . . . . 16
2.5 Pa ame e ized quan um ci cui s . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6 Pa ame e es ima ion and Fishe in o ma ion . . . . . . . . . . . . . . . . . . . . 21
2.7 Uni e sali y and classical simula ion . . . . . . . . . . . . . . . . . . . . . . . . 24
3 Va ia ional quan um algo i hms 27
3.1 Classical da a encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2 Exp essi i y o quan um machine lea ning models . . . . . . . . . . . . . . . . . 32
3.3 Op imiza ion o pa ame e ized quan um ci cui s . . . . . . . . . . . . . . . . . . 35
3.4 The ba en pla eau phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4 Rein o cemen lea ning 44
ii
45 Cumula i e ewa ds ob ained by he Bo n policies in he Ac obo en i onmen : (a) UQC wi h
a numbe o qubi s in {1,2,4}and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is
labeled
global
in helegend............................... 108
46 En anglemen du ing aining o he Bo n policies in he Ac obo en i onmen : (a) UQC wi h
a numbe o qubi s in {1,2,4}and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is
labeled
global
in helegend............................... 108
47 Cumula i e ewa ds ob ained by he Bo n policies wi h so max ac i a ion in he Ca pole
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -
like policy is labeled
global
in helegend......................... 109
48 En anglemen du ing aining o he Bo n policies wi h so max ac i a ion in he Ca pole
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -
like policy is labeled
global
in helegend......................... 110
49 Cumula i e ewa ds ob ained by he Bo n policies wi h so max ac i a ion in he Ac obo
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -
like policy is labeled
global
in helegend......................... 110
50 En anglemen du ing aining o he Bo n policies wi h so max ac i a ion in he Ac obo
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -
like policy is labeled
global
in helegend......................... 111
51 Local so max policy in he Ca pole en i onmen : (a) cumula i e ewa d and (b) en angle-
men du ing aining. The UQC wi h {1,2,4}qubi s and Je bi a chi ec u e is conside ed. 112
52 Local so max policy in he Ac obo en i onmen : (a) cumula i e ewa d and (b) en anglemen
du ing aining. The UQC wi h {1,2,4}qubi s and Je bi a chi ec u e is conside ed. . . . 112
53 (a) Va iance o he p obabili y o measu ing he all-ze o s a e unde a global p ojec o . (b)
log plo o he a iance o T [𝜌𝜃P], whe e Pis ei he he global o local p ojec o as
desc ibed abo e. The a iance is plo ed e sus he numbe o qubi s o 1000 andomly
sampled pa ame e s 𝜃∈𝑈(−𝜋, 𝜋)........................... 120
54 En angled PQC composed o wo Bell s a es in a ou qubi sys em. . . . . . . . . . . . 122
55 Va iance o he log policy g adien o h ee dis inc en angled s a es. (a) Simpli ied wo de-
sign. (b) S ongly en angling laye s. (c) Random s a es composed o Pauli o a ions sampled
uni o mly a andom ollowed by andomly selec ed CZ ga es. (d) Va iance as a unc ion o
he numbe o qubi s o 𝑁laye s o building blocks o each o he ci cui s (a)-(c). . . . . 124
56 The simpli ied wo-design ansa z wi h an addi ional o a ion laye (in pu ple) o s a e encoding
𝑆(𝑠). Each o he 𝑁qubi s encodes a ea u e o 𝑠.................... 129
57 Va iance o he log policy g adien o a con iguous-like Bo n policy: (a) and (b) s. |𝐴|, and
(c) a semi-log plo s. numbe o qubi s. Unclipped p obabili ies. . . . . . . . . . . . . 130
58 Va iance o he log policy g adien o a con iguous-like Bo n policy unde polynomial clipping:
(a) s. |𝐴|, and (b) a semi-log plo s. 𝑁......................... 130
xi

59 Va iance o he log policy g adien o a pa i y-like Bo n policy: (a) and (b) s. |𝐴|, and (c) a
semi-log plo s. 𝑁. Unclipped p obabili ies. . . . . . . . . . . . . . . . . . . . . . . 131
60 Va iance o he log policy g adien o a pa i y-like Bo n policy unde polynomial clipping: (a)
s. |𝐴|, and (b) a semi-log plo s. 𝑁. ......................... 131
61 Eigen alue dis ibu ion o he FIM o he pa i y-like Bo n policy, compa ing |𝐴|=2(a) and
|𝐴|=2𝑁(b) as 𝑁g ows................................ 132
62 Eigen alue dis ibu ion o he FIM o he con iguous-like Bo n policy, compa ing |𝐴|=2(a)
and |𝐴|=2𝑁(b) as 𝑁g ows.............................. 132
63 Resul s wi h |𝐴|=𝑁: (a,b) P obabili y o picking he bes a m o con iguous-like s. pa i y-
like Bo n policies; (c) a iance o he log policy g adien . . . . . . . . . . . . . . . . . 133
64 Resul s wi h |𝐴|=2𝑁−4: (a,b) P obabili y o picking he bes a m o con iguous-like s.
pa i y-like Bo n policies; (c) a iance o he log policy g adien . . . . . . . . . . . . . . 133
65 Pa ame e ized Quan um Ci cui s (PQCs) used: (a) simpli ied 2-design, (b) SEL (gene al
single-qubi ga es 𝐺), (c) andom ansa z ( andom single-qubi o a ions 𝑃). The pu ple
(shaded) boxes illus a e one laye . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
66 Log-scale a iance o he log-policy g adien o local and global so max policies in (a) sim-
pli ied 2-design, (b) SEL, (c) andom ansa z. . . . . . . . . . . . . . . . . . . . . . . 138
67 Log-scale a iance o he pa ial de i a i e o he log policy o global, local, and pa ial ob-
se ables in each o he h ee PQCs. . . . . . . . . . . . . . . . . . . . . . . . . . . 138
68 Log-scale a iance o 𝜕𝜃log 𝜋(𝑎|𝜃)unde global, local, and pa ial obse ables, o di e -
en |𝐴| ∈ {2,4,8,16}. ................................ 139
69 P obabili y o picking he bes a m o e 100 episodes o global, local, and pa ial so max
policies......................................... 139
70 Agen -en i onmen in e ace o he Quan um Na u al Policy G adien (QNPG) algo i hm. The
agen in e ac s wi h he en i onmen o gene a e a ajec o y 𝑇o s a es and ac ions. An
in o ma ion ma ix ( he CFIM o Quan um Fishe In o ma ion Ma ix (QFIM)) is es ima ed
om hese da a and used o upda e he policy pa ame e s. . . . . . . . . . . . . . . . 145
71 Pa ame e ized quan um ci cui used in he nume ical expe imen s. Da a euploading ol-
lows [93], bu inpu scaling is omi ed o acili a e mo e accu a e QFIM ma ix es ima ion. 153
72 Pe o mance o NPG (and i s gene alized quan um coun e pa ) in he Ca pole en i onmen .
Sub igu e (a) uses a Bo n policy, while sub igu e (b) uses a So max policy. The cumula i e
ewa d is shown on he y-axis, o e aining episodes on he x-axis. . . . . . . . . . . . 154
73 Pe o mance o NPG (and i s gene alized quan um coun e pa ) in he Ac obo en i onmen .
Sub igu e (a) uses a Bo n policy, while sub igu e (b) uses a So max policy. . . . . . . 154
74 The SWAP es ci cui o es ima ing he dis ance be ween policies. . . . . . . . . . . . 158
75 IQP ci cui . (a) Z-p og am. (b) X-p og am. . . . . . . . . . . . . . . . . . . . . . . . 160
76 IQP ci cui s wi h wo laye s. (a) Z-p og am. (b) X-p og am. . . . . . . . . . . . . . . . 160
x
77 Diagonal ga e pa e n o IQP ci cui s. (a) B ick-like. (b) Py amid-like. (c) Nex nea es neigh-
bo . (d) All o all connec i i y. (e) Single-qubi ga es added. . . . . . . . . . . . . . . . 162
78 IQP ci cui wi h wo laye s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
79 Ligh -cone o he O(1)-local e m in he con ibu ion o nex -nea es neighbo IQP ci cui s. 169
80 IQP ci cui s wi h single and wo-qubi diagonal ga es conside ed in he nume ical simula-
ions. (a) Nex -nea es neighbo connec i i y wi h a single laye . (b) Nex -nea es neighbo
connec i i y wi h wo laye s. (c) All- o-all connec i i y wi h a single laye . . . . . . . . . . 173
81 a iance o he cos unc ion o IQP ci cui s wi h (a) nex -nea es neighbo connec i i y single-
laye , (b) All o all connec i i y. Va iance is es ima ed as a unc ion o he numbe o qubi s
o |𝐴|=𝑁2and |𝐴|=2𝑁ac ions........................... 173
82 a iance o he cos unc ion o IQP ci cui s wi h (a) nex -nea es neighbo connec i i y wo-
laye s, (b) nex -nea es neighbo connec i i y h ee-laye s. Va iance is es ima ed as a unc ion
o he numbe o qubi s o |𝐴|=𝑁2and |𝐴|=2𝑁ac ions. . . . . . . . . . . . . . . 174
83 pe o mance o he IQP-based con iguous-like Bo n policy wi h and wi hou So max ac i a ion
in he con ex ual bandi se ing. The a e age ewa d is plo ed wi h he numbe o episodes
as a unc ion o he numbe o qubi s in he policy. . . . . . . . . . . . . . . . . . . . 176
84 pe o mance o he IQP-based con iguous-like Bo n policy wi h So max ac i a ion in he con-
ex ual bandi se ing and espec i e compa ison wi h classical neu al ne wo k policies. Clas-
sical models a e iden i ied by he lag ”hs”indica ing he hidden size laye sℎ𝑠 ={4,8,16,32}
p esen in he ully connec ed neu al ne wo ks. The a e age ewa d is plo ed wi h he num-
be o episodes as a unc ion o he numbe o qubi s in he policy. (a) IQP-based models wi h
𝐿={1,2,3}laye s. (b) Classical neu al ne wo k s IQP-based models wi h 𝐿=2laye s.
(c) Classical neu al ne wo k s IQP-based models wi h 𝐿=3laye s. . . . . . . . . . . 177
85 Modi ied Agen -en i onmen in e ace o quan um Q-lea ning. . . . . . . . . . . . . . 181
86 Ha dwa e-e icien ci cui s conside ed in he nume ical expe imen s. (a)
Skolik
a chi ec u e
inspi ed by Skolik e .al [185]. (b)
Uni e sal Quan um Classi ie (UQC)
a chi ec u e inspi ed
by he single-qubi uni e sal app oxima o p oposed by Salinas e .al [152]. . . . . . . . 192
87 Pe o mance o Baseline Models (on he le ) and Da a Re-Uploading models (on he igh )
in he (a) Ca Pole- 0 en i onmen and (b) Ac obo - 1 en i onmen . Resul s plo he aining
wi h and wi hou ainable inpu and/o ou pu scaling. The e u ns a e a e aged o e 10
agen s. The ull se o hype pa ame e s can be seen in Table 13 . . . . . . . . . . . . 193
88 T ainabili y o Baseline Models (on he le ) and Da a Re-Uploading models (on he igh ) in
he (a) Ca Pole- 0 en i onmen and (b) Ac obo - 1 en i onmen . In bo h Sub igu es, he le
g aph ep esen s he g adien ’s no m h oughou aining and he igh g aph he a iance o
heno m........................................ 194
89 Cumula i e ewa d ( op g aph) and he espec i e loss unc ion e olu ion as a unc ion o 𝐶
o
Skolik euploading
model in (a) Ca pole- 0 and (b) Ac obo - 1, en i onmen s. The ull
se o hype pa ame e s can be seen in Table 14. . . . . . . . . . . . . . . . . . . . . 196
x i
90 G adien no m and a iance o he
Skolik euploading
model wi h inc easing alues o 𝐶in
(a) Ca pole- 0 and (b) Ac obo - 1 en i onmen s. . . . . . . . . . . . . . . . . . . . . 197
91 Pe o mance o he
UQC
a chi ec u e wi h Full and Pa ial Encoding conside ing in he (a)
Ca Pole- 0 and (b) Ac obo - 1 en i onmen s. The esul s a e plo ed o 2 and 4 qubi s, wi h
and wi hou en anglemen . The ull se o hype pa ame e s can be seen in Table 15. . . 199
92 Ci cui o he
UQC
a chi ec u e wi h Full Encoding and linea en anglemen o 4 qubi s. 199
93 T ainabili y o he
UQC
a chi ec u e wi h Full Encoding and linea en anglemen o (a) Ca pole-
0 and (b) Ac obo - 1 en i onmen s. The g adien no m and a iance a e plo ed as a unc ion
o he numbe o qubi s and he numbe o aining s eps. The ull se o hype - pa ame e s
canbeseeninTable16................................. 201
94 Va iance o he g adien o he
UQC
a chi ec u e wi h Full Encoding and linea en anglemen
o he Ca pole- 0 en i onmen conside ing local and global cos unc ions. The esul s a e
plo ed as a unc ion o he numbe o qubi s. ..................... 202
95 T aining and alida ion accu acies and MSE loss o (a)
Skolik Da a Re-uploading
and (b)
Mul i-Qubi
UQC
models in he bina y classi ica ion p oblem. The esul s a e plo ed as a
unc ion o he numbe o qubi s. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
96 G adien no m and a iance o he (a)
Skolik Da a Re-uploading
and (b) Mul i-Qubi
UQC
models in he bina y classi ica ion p oblem. The esul s a e plo ed as a unc ion o he
numbe o qubi s.................................... 204
97 A simple Bayesian ne wo k wi h h ee nodes. . . . . . . . . . . . . . . . . . . . . . . 207
98 Gene ic ep esen a ion o a Dynamic Bayesian Ne wo k (DBN). . . . . . . . . . . . . . 208
99 Gene ic ep esen a ion o a Dynamic Decision Ne wo k (DDN) wi h ac ion and ewa d nodes
clea ly illus a ed o wo ime s eps. . . . . . . . . . . . . . . . . . . . . . . . . . . 209
100 Quan um ci cui o he Quan um Bayesian Ne wo k (QBN) o he example in Figu e 97. . 210
101 A simple one- ime-s ep DDN o a Pa ially Obse able Ma ko Decision P ocess (POMDP). 212
102 Quan um ci cui o he single ime s ep DDN o a POMDP. . . . . . . . . . . . . . . 214
103 Lookahead sea ch ee o ho izon wo. Belie nodes a e pen agonally shaped and obse a ion
nodes a e ci cle shaped. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
104 Illus a ion o he POMDPs conside ed in he nume ical expe imen s. (a) The ige p oblem.
(b) The obo explo a ion p oblem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
105 Cumula i e ewa d di e ence o he one-s ep lookahead in (a) ige p oblem and (b) obo
explo a ionp oblem................................... 219
106 Cumula i e ewa d di e ence o he wo-s ep lookahead in (a) ige p oblem and (b) obo
explo a ionp oblem................................... 220
x ii
Lis o Tables
1 Cha ac e is ics o di e en ypes o Bo n policies. . . . . . . . . . . . . . . . . . . . . 78
2 Applicabili y o inpu and ou pu scaling in PQC-based policies. . . . . . . . . . . . . . 87
3 G adien ecipes o each policy and hei espec i e pa ame e s, including he Gaussian
policy.......................................... 92
4 Policy g adien anges and espec i e g adien es ima ion sample complexi y o PQC-based
policies......................................... 96
5 Numbe o pa ame e s ained o bo h en i onmen s. En : en i onmen ; I: Inpu laye ; O: Ou pu
laye ; #N: neu ons; #R: o a ions pe qubi ; 𝑤: ou pu -scaling; #P: o al pa ame e s. ......... 101
6 Numbe o pa ame e s o he UQC and Je bi ci cui s, whe e |𝑠|is he numbe o ea u es,
𝐿is he numbe o laye s, and 𝑁is he numbe o qubi s. . . . . . . . . . . . . . . . 105
7 Obse ables o he So max policy in di e en en i onmen s . . . . . . . . . . . . . . 111
8 Summa y o esul s. The i s column indica es he ype o in o ma ion ma ix conside ed.
The second and hi d columns indica e whe he he no m and app oxima ion e o inequali ies
a e gua an eed, espec i ely. The ou h column indica es i he eg e is imp o ed. . . . 150
9 Simulabili y o con iguous-like Bo n policies composed o commu ing-gene a o ci cui s, as a
unc ion o he numbe o ac ions |𝐴|. ......................... 163
10 Complexi y compa ison be ween classical and quan um ejec ion sampling algo i hms. 𝑁
is he numbe o a iables, 𝑀is he numbe o pa en s o any a iable and 𝑃(𝑒)is he
p obabili y o he e idence aking alue 𝑒. ....................... 211
11 Cha ac e iza ion o he en i onmen s conside ed in he nume ical expe imen s. . . . . 248
12 Cha ac e iza ion o he PQC’s conside ed in he nume ical expe imen s. 𝑃𝑖indica es he
p ojec o in he compu a ional basis in decimal. Fo he Ca pole en i onmen a single-qubi
was measu ed and he p obabili y o each basis s a e associa ed o an ac ion. In he Ac obo
en i onmen , he ac ion assignmmen was made using 𝑖𝑛𝑡 (𝑏)mod 3=𝑎 o a pa icula
basis s a e 𝑏. ..................................... 249
13 PQC-based DQN hype pa ame e s o he nume ical expe imen s o Sec ion 9.5.1. . . . . 251
x iii
14 Complexi y compa ison be ween classical and quan um ejec ion sampling algo i hms. 𝑁
is he numbe o a iables, 𝑀is he numbe o pa en s o any a iable, and 𝑃(𝑒)is he
p obabili y o he e idence aking alue 𝑒. ....................... 252
15 Hype pa ame e s o Models o Figu e 91 . . . . . . . . . . . . . . . . . . . . . . . 253
16 Hype pa ame e s o Models o Figu e 93 . . . . . . . . . . . . . . . . . . . . . . . 254
xix

Lis o Algo i hms
1 Es ima ing he exp essi i y o a PQC ensemble . . . . . . . . . . . . . . . . . . . 21
2 Q-Lea ning..................................... 54
3 DeepQ-Lea ning.................................. 57
4 REINFORCE .................................... 60
5 Na u al Policy G adien . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6 Quan um Na u al Policy G adien (QNPG) . . . . . . . . . . . . . . . . . . . . . . 146
7 PQC-based Deep Q-Lea ning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
8 Quan um ejec ion sampling algo i hm [121] whe e he ope a o 𝐺2𝑘comes om he
exponen ial p og ession o he ampli ude ampli ica ion algo i hm, p o ided he op imal
numbe o i e a ions is no known [28]. . . . . . . . . . . . . . . . . . . . . . . . 211
xx
Lis ings
5.1 Example Py hon Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
xxi
Ac onyms
BN Bayesian Ne wo k
(pp. 206, 207, 210, 211, 213)
BP Ba en Pla eau
(pp. 3, 4, 6–9, 29, 32, 39–43, 69, 116, 117, 121, 126–128, 130, 131, 133, 134,
163, 166–170, 172–174, 178, 222, 223)
CFIM Classical Fishe In o ma ion Ma ix
(pp. xiii, x , 4, 8, 22, 23, 34, 37, 61, 101, 102, 117, 127–129,
131, 140, 141, 143–148, 150–153, 155, 157, 178, 223)
DBN Dynamic Bayesian Ne wo k
(pp. x ii, 208, 209, 211)
DDN Dynamic Decision Ne wo k
(pp. x ii, 209, 212–215)
DLA Dynamical Lie Algeb a
(pp. 34, 41–43, 164–167, 174)
DQN Deep Q-Ne wo k
(pp. 5, 6)
IQP Ins an aneous Quan um Polynomial
(pp. xi, x i, 8, 25, 26, 159, 160, 162–179, 223, 224)
MDP Ma ko Decision P ocess
(pp. 46, 47, 79, 206, 213)
ML Machine Lea ning
(pp. 1, 41, 43, 120, 157)
NISQ Noisy In e media e-Scale Quan um
(pp. 1, 25, 27)
NPG Na u al Policy G adien
(pp. 3, 61, 62, 64, 65, 143, 144, 147, 223)
POMDP Pa ially Obse able Ma ko Decision P ocess
(pp. x ii, 206, 207, 212–218, 220, 221)
PQC Pa ame e ized Quan um Ci cui
(pp. x, xi, xiii, xi , x iii, xx, 1–9, 11, 18–23, 27–35, 37–39, 42,
43, 67–69, 72, 74, 78–83, 86–89, 91, 94–96, 98–100, 102–106, 108, 113–119, 121–123, 125,
127–129, 132, 134, 139–149, 152–157, 159, 164, 166, 167, 170, 178, 180–187, 189–195, 197,
198, 200–205, 221–225, 250–252, 254)
QBN Quan um Bayesian Ne wo k
(pp. x ii, 209–211)
QFIM Quan um Fishe In o ma ion Ma ix
(pp. x , 3, 4, 6, 8, 23, 34, 38, 143–150, 152, 153, 155–157,
178, 223)
QML Quan um Machine Lea ning
(pp. xi, 1, 2, 19, 29, 30)
xxii
ACRONYMS
xxiii
QNG Quan um Na u al G adien
(pp. 38, 178)
QNPG Quan um Na u al Policy G adien
(pp. x , 3, 143, 145–147)
QRAM Quan um Random Access Memo y
(pp. xi, 1, 2)
QRL Quan um Rein o cemen Lea ning
(pp. xi, 2–5, 9, 29, 157)
RL Rein o cemen Lea ning
(pp. xii, 2–8, 22, 23, 29, 44, 46, 48, 50, 51, 55, 56, 60, 62–64, 67,
69–72, 75, 79, 84, 96, 114, 118, 121, 125, 127, 129, 134, 136, 142, 143, 151, 158, 180, 191, 192,
202, 203, 206, 208, 209, 211, 212, 222–225)
VQA Va ia ional Quan um Algo i hm
(pp. xi, 18, 27–29, 35, 71)
1.3. THESIS STRUCTURE AND SYNOPSIS
Pa I p o ides he ounda ional knowledge equi ed o unde s and he disse a ion’s co e con ibu ions.
This sec ion comp ises h ee chap e s:
1. Chap e 2: Quan um In o ma ion and Compu a ion P o ides an in oduc ion o he p inciples
o quan um in o ma ion and compu a ion, co e ing ounda ional opics such as quan um s a es,
uni a y ope a ions, and measu emen pos ula es. I also del es in o essen ial concep s like he
expec a ion alue o obse ables and quan um en anglemen , se ing he s age o he applica ion
o PQCs in machine lea ning and RL.
2. Chap e 3: Va ia ional Quan um Algo i hms The chap e examines he ma hema ical s uc-
u e o PQCs, hei capaci y as uni e sal unc ion app oxima o s, and echniques such as da a
encoding and ansa z design. I also add esses challenges like BPs, g adien op imiza ion, and
he adeo be ween exp essi i y and scalabili y, p o iding a comp ehensi e unde s anding o how
PQCs a e employed in hyb id quan um-classical algo i hms.
3. Chap e 4: Rein o cemen Lea ning P o ides a comp ehensi e o e iew o RL, s a ing wi h
ounda ional concep s such as he agen -en i onmen in e ac ion loop, ewa d unc ions, and alue
unc ions. I co e s classical abula me hods, including policy and alue i e a ion, as well as model-
ee app oaches like Q-lea ning. The chap e hen p og esses o ad anced opics, such as policy
op imiza ion echniques, unc ion app oxima ion, and he in eg a ion o neu al ne wo ks in deep
RL. Special a en ion is gi en o challenges such as he explo a ion-exploi a ion adeo , sample
e iciency, and scalabili y, se ing he s age o he applica ion o quan um me hods in RL.
Pa II p esen s he disse a ion’s p ima y con ibu ions, each add essing speci ic esea ch ques ions
posed in Chap e 1. Each chap e adap s and ex ends indings om a collec ion o au ho ed esea ch
a icles:
1. Chap e 5: Quan um Policy G adien s This chap e add esses he i s pa o Resea ch Ques-
ion RQ1, ocusing on he design and exp essi i y o PQC-based policies. Key s a egies such as
da a euploading, inpu scaling, and ou pu scaling a e in oduced o enhance pe o mance. These
con ibu ions a e based on:
•
Policy G adien s using Va ia ional Quan um Ci cui s
, Quan um Machine In elligence, Sp inge ,
DOI: 10.1007/s42484-023-00101-8, 2023.
The chap e demons a es ha PQC-based policies wi h singly encoded s a es can achie e compa-
able o supe io pe o mance o classical neu al ne wo ks while using ewe ainable pa ame e s.
Enhanced exp essi i y h ough da a euploading u he imp o es agen pe o mance.
7

CHAPTER 1. INTRODUCTION
2. Chap e 6: T ainabili y Issues in Quan um Policy G adien s This chap e explo es he sec-
ond pa o Resea ch Ques ion RQ1, ocusing on he ainabili y challenges posed by he BP phe-
nomenon in PQC-based policies. The esul s a e published in:
•
T ainabili y Issues in Quan um Policy G adien s
, IOP Machine Lea ning: Science and Technol-
ogy, DOI: 10.1088/2632-2153/ad6830, 2024.
•
T ainabili y Issues in Quan um Policy G adien s wi h So max Ac i a ions
, IEEE In e na ional
Con e ence on Quan um Compu ing and Enginee ing (QCE), 2024.
Key con ibu ions include condi ions unde which anishing g adien s can be expec ed and mi -
iga ed. I is shown ha BPs a e p esen in e e y PQC-based ins ance unde uni o mly andom
ini ializa ion in models comp ised o local 2-designs. Al hough, such policies p o ide a ainable
egion a loga i hmic dep h, p o ided local measu emen s a e pe o med, hus a oiding anishing
g adien s.
3. Chap e 7: Quan um Na u al Policy G adien s This chap e add esses Resea ch Ques ion
RQ2 by explo ing second-o de op imiza ion echniques o enhance he aining o PQC-based poli-
cies using he QFIM. The esul s a e based on:
•
Quan um Na u al Policy G adien s
, IEEE T ansac ions on Quan um Enginee ing, DOI: 10.1109
/ TQE.2024.3418094, 2024.
Key indings demons a e ha quan um na u al g adien s yield mo e s able and in o med upda es
compa ed o Euclidean g adien s, enhancing he con e gence and pe o mance o quan um poli-
cies. Howe e , h ough an analysis o Löwne inequali ies be ween he CFIM and QFIM, i is shown
ha PQC-based policies do no , in gene al, consis en ly bene i om upda es p econdi ioned by he
QFIM o e he CFIM. This chap e p o ides a c i ical compa ison be ween s a e-space and policy-
space upda es, shedding ligh on he p ac ical adeo s o employing QFIM in RL con ex s wi h
classical da a.
4. Chap e 8: E icien ly T ainable Quan um Ci cui s o Classically In ac able Policy
G adien s This chap e add esses Resea ch Ques ion RQ4, de eloping PQC-based policies de i ed
om he class o Ins an aneous Quan um Polynomial (IQP) ci cui s. The key inding is ha he e
a e ci cui s om his class ha enable a ainable and classically ha d- o-simula e window, p o ided
he policy is ob ained om measu ing mo e han log(𝑁)qubi s and O(poly(𝑁)) ac ions. This
app oach balances he adeo be ween e icien ainabili y and classical in ac abili y, ad ancing
he design o PQC-based agen s o achie ing quan um ad an age. A manusc ip is cu en ly in
p epa a ion.
5. Chap e 9: T adeo Be ween T ainabili y and Exp essi i y This chap e e isi s Resea ch
Ques ion RQ1, examining he in e play be ween ainabili y and exp essi i y in PQC-based alue
unc ion app oxima o s. I is based on:
8
1.4. LIST OF PUBLICATIONS
•
VQC-Based Rein o cemen Lea ning wi h Da a Re-uploading: Pe o mance and T ainabili y
,
Quan um Machine In elligence, Sp inge , DOI: 10.48550/a Xi .2401.11555, 2024.
The chap e iden i ies a new phenomenon whe e g adien s end o inc ease wi h aining, depending
on a ge ne wo k upda es and mo ing a ge s in deep Q-lea ning. E en hough g adien s inc ease
wi h aining, BPs can indeed occu o andom ini ializa ion in local 2-design models. Howe e , i
is igo ously demons a ed ha anishing g adien s can be a oided p o ided loga i hmic dep h and
local measu emen s.
6. Chap e 10: Quan um Bayesian Rein o cemen Lea ning This chap e add esses Resea ch
Ques ion RQ4, p oposing a no el amewo k o planning in pa ially obse able en i onmen s us-
ing quan um Bayesian in e ence. The esul s build on quan um ejec ion sampling echniques o
e icien belie upda es. I was demons a ed a quad a ic speedup in sample complexi y o belie
upda es in a nea -op imal ho izon-based planning algo i hm, o pa ially obse able en i onmen s.
A manusc ip is cu en ly in p epa a ion.
Finally, Chap e 11 p o ides a comp ehensi e summa y o he disse a ion’s main con ibu ions and in-
eg a es he insigh s gained om add essing he esea ch ques ions. While each chap e concludes wi h
i s own indings and p ospec s o u u e wo k, his chap e uni ies hese indi idual conclusions o p esen
a cohesi e pe spec i e on he b oade implica ions o he esea ch. Addi ionally, i highligh s po en ial
di ec ions o u u e explo a ion in he ield o QRL, o e ing a oadmap o ad ancing he applica ion and
unde s anding o PQC-based amewo ks.
1.4 Lis o publica ions
•
Policy G adien s using Va ia ional Quan um Ci cui s
- Quan um Machine In elligence, Sp inge , DOI:
10.1007/s42484-023-00101-8, 2023. Analyzed and ex ended in Chap e 5.
•
T ainabili y Issues in Quan um Policy G adien s
- IOP Machine Lea ning: Science and Technology,
DOI: 10.1088/2632-2153/ad6830, 2024. Analyzed in Chap e 6.
•
T ainabili y Issues in Quan um Policy G adien s wi h so max ac i a ions
- IEEE In e na ional Con-
e ence on Quan um Compu ing and Enginee ing (QCE), 2024. Analyzed in Chap e 6.
•
Quan um Na u al Policy G adien s
- IEEE T ansac ions on Quan um Enginee ing, DOI: 10.1109/
TQE.2024.3418094, 2024. Analyzed in Chap e 7.
•
VQC-Based Rein o cemen Lea ning wi h Da a Re-uploading: Pe o mance and T ainabili y
- Quan-
um Machine In elligence, Sp inge , DOI: 10.48550/a Xi .2401.11555, 2024. Analyzed and ex-
ended in Chap e 9.
9
Pa
I
Backg ound
10
C h a p e
2
Quan um In o ma ion and Compu a ion
The ield o quan um in o ma ion and compu a ion lies a he c oss oads o quan um mechanics and
compu e science, o e ing ans o ma i e pe spec i es on how in o ma ion can be p ocessed, s o ed,
and manipula ed. This chap e p o ides a comp ehensi e o e iew o he undamen al concep s, no a ion,
and p inciples ha unde lie quan um in o ma ion and compu a ion, se ing as he heo e ical basis o
he discussions h oughou his disse a ion. Sec ions 2.1 o ?? in oduce he ounda ional pos ula es
o quan um mechanics and hei ma hema ical ep esen a ion. We begin by examining how quan um
s a es a e desc ibed using Di ac no a ion, s a e ec o s, and densi y ma ices, ollowed by an explo a ion
o quan um measu emen p ocesses and he es ima ion o expec a ion alues o physical obse ables.
Building upon hese ounda ional elemen s, Sec ion 2.5 u ns o he concep o PQCs and he a ia ional
p inciple. These ci cui s o m he co e compu a ional model conside ed in his disse a ion, enabling he
in eg a ion o classical op imiza ion echniques wi h quan um ope a ions. We hen del e deepe in o he
connec ion be ween PQCs and pa ame e es ima ion in Sec ion 2.6, which p o ides insigh s in o how
Fishe in o ma ion can quan i y sensi i i y and ainabili y in pa ame e ized quan um models.
2.1 S a e space
Quan um s a es a e ep esen ed as ec o s in a Hilbe space H ⊆ ℂ𝐾. A s a e is ypically deno ed by
he
ke
|𝜓i=(𝛼0, . . . , 𝛼𝐾−1)𝑇. The no m o his ec o is exp essed h ough he inne p oduc , w i en
in Di ac no a ion as h·|·i, whe e h·| is he conjuga e anspose (b a) o he ec o ke . Fo wo ec o s |𝜓i
and |𝜙iin H, he ollowing iden i y holds:
h𝜓|𝜙i∗=h𝜙|𝜓i,(2.1)
and he no m o |𝜓iis
||𝜓|| =ph𝜓|𝜓i.
11
CHAPTER 2. QUANTUM INFORMATION AND COMPUTATION
Le {|𝑒𝑖i}𝑖∈ℕbe an o hono mal basis o he Hilbe space H. A gene al quan um s a e can hen be
expanded in his basis as
|𝜓i=Õ
𝑖|𝑒𝑖i h𝑒𝑖|𝜓i
|{z}
𝛼𝑖
=Õ
𝑖
𝛼𝑖|𝑒𝑖i,(2.2)
whe e each 𝛼𝑖is a
p obabili y ampli ude
, and |𝑒𝑖ih𝑒𝑖|ac s as he
p ojec ion ope a o
on o he basis s a e
|𝑒𝑖i. The
comple eness
o
esolu ion o he iden i y
condi ion s a es ha
Õ
𝑖|𝑒𝑖ih𝑒𝑖|=𝐼,
whe e 𝐼is he iden i y ope a o . This decomposi ion is undamen al o analyzing measu emen ou comes
on quan um s a es.
A canonical example o an o hono mal basis in he wo-dimensional case is he
compu a ional basis
{|0i,|1i}, which de ines a wo-le el quan um sys em known as a
qubi
. A gene ic qubi s a e can be
w i en as a linea combina ion o hese basis s a es:
|𝜓i=𝛼0|0i + 𝛼1|1i.(2.3)
When |𝜓iis a
pu e
s a e, i can be isualized as a poin on he su ace o he
Bloch sphe e
, as depic ed
in Figu e 3.
Figu e 3: Bloch sphe e ep esen a ion o a single-qubi s a e. Image sou ce:
Machine Lea ning wi h
Quan um Compu e s
by Schuld e al. [168]
In p ac ice, one o en encoun e s unce ain y abou he exac s a e due o en i onmen al noise o impe ec
isola ion o he sys em. In such scena ios, he s a e is no pu ely desc ibed by a single ke bu may exis
in a
s a is ical mix u e
o pu e s a es, e e ed o as a
mixed s a e
. Geome ically, mixed s a es lie in he
12

2.2. TIME EVOLUTION
in e io o he Bloch sphe e ( o one qubi ), a he han on i s su ace. These mix u es a e desc ibed by a
densi y ma ix
(o
densi y ope a o
)𝜌, o mally w i en as
𝜌=Õ
𝑗
𝑝𝑗|𝜓𝑗ih𝜓𝑗|,(2.4)
whe e each |𝜓𝑗iis a pu e s a e and 𝑝𝑗is he p obabili y o p epa ing ha pu e s a e. A pu e s a e |𝜓iis
i sel a special case, in which
𝜌=|𝜓ih𝜓|.
Thus, whe he a s a e is pu e o mixed, densi y ma ices o e a powe ul and gene al o malism o
desc ibing quan um s a es in ealis ic se ings.
2.2 Time e olu ion
The ime e olu ion o a closed quan um sys em is go e ned by he Sch ödinge equa ion. I |𝜓(𝑡)i is he
s a e o he sys em a ime 𝑡and 𝐻is he Hamil onian (i.e., he o al ene gy ope a o ), hen
𝑖ℏ𝑑
𝑑𝑡 |𝜓(𝑡)i =𝐻|𝜓(𝑡)i,(2.5)
whe e ℏis he educed Planck cons an . The o mal solu ion o his equa ion is gi en by he uni a y
ope a o 𝑈(𝑡),
|𝜓(𝑡)i =𝑈(𝑡) |𝜓(0)i,(2.6)
wi h
𝑈(𝑡)=𝑒−𝑖
ℏ𝐻 𝑡 .
The uni a i y o 𝑈(𝑡)ensu es ha quan um e olu ion is e e sible and p ese es he no maliza ion o
quan um s a es, e lec ing he de e minis ic na u e o quan um mechanics o closed sys ems. When he
sys em’s ini ial s a e is desc ibed by a densi y ma ix 𝜌(0)a 𝑡=0, he co esponding ime-e ol ed s a e
is
𝜌(𝑡)=𝑈(𝑡)𝜌(0)𝑈†(𝑡),(2.7)
a o malism known as he
Sch ödinge pic u e
[144]. Al e na i ely, one may mo e all ime dependence o
he obse ables hemsel es in he
Heisenbe g pic u e
, a concep e isi ed in Sec ion 3.
In he con ex o quan um compu ing, pa icula ly in he ci cui -based model, he ime e olu ion unde
a Hamil onian 𝐻can be iewed as he applica ion o a
uni a y ga e
on qubi s. These ga e ope a ions,
ep esen ed by uni a y ma ices, cap u e he allowed e olu ions o quan um s a es in a compu a ional
se ing. Consequen ly, he Hamil onian 𝐻e ec i ely se es as a gene a o o uni a y ans o ma ions
[144]. I is c ucial o no e ha eal-wo ld quan um sys ems o en in e ac wi h ex e nal en i onmen s,
leading o
open quan um sys ems
, whe e he ules o e olu ion di e . In such cases, one mus inco po a e
en i onmen al e ec s in o he Hamil onian. Howe e , h oughou his disse a ion, only
closed
sys ems
a e conside ed. Fo de ails on open quan um sys ems, see [33].
13
CHAPTER 2. QUANTUM INFORMATION AND COMPUTATION
One o he undamen al consequences o uni a y e olu ion in quan um mechanics is he
no-cloning he-
o em
, which p ohibi s he pe ec copying o a bi a y unknown quan um s a es [208]. A di ec co olla y
o his heo em is ha non-o hogonal quan um s a es canno be pe ec ly dis inguished [15].
2.3 Composi e sys ems and en anglemen
Composi e quan um sys ems a e desc ibed ia he enso p oduc o he indi idual Hilbe spaces o hei
cons i uen subsys ems. Fo example, i wo quan um subsys ems ha e Hilbe spaces H𝐴⊆ℂ𝑁and
H𝐵⊆ℂ𝑁, he combined sys em li es in he enso p oduc space
H=H𝐴⊗ H𝐵.
A pu e s a e |𝜓i ∈ H is called a
p oduc s a e
(o
sepa able
s a e) i and only i he e exis indi idual
s a es |𝜙i ∈ H𝐴and |𝜑i ∈ H𝐵such ha
|𝜓i=|𝜙i ⊗ |𝜑i.(2.8)
I no choice o p oduc s a es exis s in any basis, he s a e is said o exhibi co ela ions o , mo e o mally,
i is
en angled
. In o he wo ds, he educed s a e o any one subsys em canno be speci ied wi hou
e e encing he o he subsys em.
Conside he uni o m supe posi ion o all basis s a es in an 𝑁-qubi sys em,
|𝜓i=1
2𝑁|00 ···00i + |00 ···01i + ··· + |11 ···11i.(2.9)
I can be ac o ed in o a p oduc o single-qubi s a es as
|𝜓i=
2𝑁−1
Ì
𝑖=0
1
√2|0i+|1i,(2.10)
indica ing ha his pa icula s a e is
no
en angled.
A cen al example o en anglemen occu s in wo-qubi
Bell s a es
:
|Φ+i=1
√2|00i+|11i,|Φ−i=1
√2|00i−|11i,
|Ψ+i=1
√2|01i+|10i,|Ψ−i=1
√2|01i−|10i.(2.11)
Collec i ely, hese ou s a es o m he
Bell basis
. They a e maximally en angled in he sense ha a
p ojec i e measu emen on one qubi yields comple ely andom ou comes. Bell-basis measu emen s a e
in eg al o quan um p o ocols such as
quan um elepo a ion
[144].
14
2.3. COMPOSITE SYSTEMS AND ENTANGLEMENT
Measu ing he deg ee o en anglemen in a quan um s a e is c ucial o unde s anding and exploi ing
quan um e ec s in compu a ion and communica ion asks. The
Schmid decomposi ion
[122] p o ides
one way o cha ac e ize bipa i e pu e s a es. Any bipa i e s a e |𝜓ican be w i en as
|𝜓i=
𝑑
Õ
𝑖=0
𝑎𝑖𝜙𝑖i⊗|𝜑𝑖,(2.12)
whe e 𝑑=min(𝑑𝐴, 𝑑𝐵)is he smalle dimension o he wo subsys em Hilbe spaces, and {|𝜙𝑖i},
{|𝜑𝑖i} a e o hono mal se s. The Schmid numbe 𝑆is he coun o nonze o coe icien s 𝑎𝑖. A pu e
bipa i e s a e is en angled i and only i 𝑆≥2. La ge Schmid numbe indica es a g ea e le el o
en anglemen .
Closely ied o he Schmid decomposi ion is he
on Neumann en opy
, commonly used o quan i y
en anglemen in bipa i e pu e s a es. Fo a bipa i e densi y ma ix 𝜌𝐴𝐵, he educed densi y ma ix o
subsys em 𝐴is de ined as
𝜌𝐴=T 𝐵𝜌𝐴𝐵.
The on Neumann en opy o 𝜌𝐴is
𝑆(𝜌𝐴)=−T 𝜌𝐴log 𝜌𝐴.(2.13)
Fo a pu e bipa i e s a e, 𝑆(𝜌𝐴)=𝑆(𝜌𝐵), highligh ing ha he en anglemen is in a ian unde which
subsys em is aced ou . A maximally en angled s a e in a 𝑑-dimensional subsys em has on Neumann
en opy log 𝑑. In con as , o
mixed
global s a es, he on Neumann en opy no longe p o ides a
s aigh o wa d measu e o en anglemen .
In mul ipa icle sys ems wi h 𝑁qubi s, unde s anding
global en anglemen
(i.e., how en anglemen is
dis ibu ed among many qubi s) can be mo e nuanced. One app oach is he
Meye –Wallach measu e
(MWM) [129], which cap u es he ex en o en anglemen ac oss he en i e 𝑁-qubi s a e a he han
ocusing on pai wise en anglemen . Fo a pu e s a e |𝜓io 𝑁qubi s, he MWM is de ined as
𝑄|𝜓i=4
𝑁
𝑁
Õ
𝑘=1
𝐷𝜌𝑘,(2.14)
whe e 𝜌𝑘=T 𝑘|𝜓ih𝜓|is he educed densi y ma ix o he 𝑘- h qubi , and 𝐷(𝜌𝑘)=1−T 𝜌2
𝑘is
he
linea en opy
. The measu e 𝑄is in a ian unde local uni a ies and no malized o lie wi hin [0,1],
wi h 𝑄=0 o p oduc s a es and 𝑄=1 o maximally en angled s a es.
One limi a ion o MWM is i s insensi i i y o he speci ic
ype
o en anglemen in he s a e. Fo ins ance,
he p oduc o wo Bell pai s can gi e he same MWM as a ou -qubi G eenbe ge –Ho ne–Zeilinge (GHZ)
s a e [32], e en hough hese s a es exhibi quali a i ely di e en mul ipa i e en anglemen s uc u es.
15
CHAPTER 2. QUANTUM INFORMATION AND COMPUTATION
2.4 Measu emen s and expec a ion alues
Once a quan um s a e has e ol ed in ime, a
measu emen
mus be pe o med o ex ac classical in-
o ma ion om i . Du ing his p ocess, he wa e unc ion
collapses
, i e e sibly des oying he quan um
in o ma ion in he s a e. Concep ually, his collapse can be in e p e ed as a classical obse e in e ac ing
wi h (o “looking a ”) he quan um sys em, hus o cing he sys em o p oduce ou comes in he classical
domain.
A measu emen is ypically associa ed wi h a physical obse able, ep esen ed by a He mi ian ope a o
Mwi h a disc e e spec um:
M=Õ
𝑚
𝜇𝑚𝑃𝑚,(2.15)
whe e each 𝜇𝑚is an eigen alue associa ed wi h eigens a e |𝑚i, and 𝑃𝑚=|𝑚i h𝑚|is he co esponding
p ojec o . The simples measu emen scheme is a
p ojec i e measu emen
, whe e he ope a o s {𝑃𝑚}
p ojec he sys em on o he eigens a es |𝑚io M.
Suppose he quan um s a e |𝜓iis exp essed as a supe posi ion o he eigens a es o M:
|𝜓i=Õ
𝑚
𝛼𝑚|𝑚i.(2.16)
When Mis measu ed on |𝜓i, he p obabili y o obse ing a pa icula eigens a e |𝑚iis gi en by he
Bo n
ule
:
𝑝(𝑚)=h𝜓|𝑃𝑚|𝜓i=|h𝑚|𝜓i|2=|𝛼𝑚|2,(2.17)
o , mo e gene ally o a densi y ma ix 𝜌,
𝑝(𝑚)=T 𝑃𝑚𝜌.(2.18)
These ela ions highligh he p obabilis ic na u e o quan um mechanics and he no ion o
wa e unc ion
collapse
upon measu emen . By p epa ing and measu ing he same (o iden ically p epa ed) s a es
epea edly, a dis ibu ion o ou comes consis en wi h 𝑝(𝑚)is ob ained.
In he single-qubi case, wi h compu a ional basis {|0i,|1i}, he measu emen ope a o s a e 𝑃0=|0ih0|
and 𝑃1=|1ih1|. I
|𝜓i=𝛼0|0i+𝛼1|1i,
hen measu ing in his basis p oduces ou come |𝑖iwi h p obabili y |𝛼𝑖|2. A e wa d, he s a e becomes
|𝜓i←− 𝑃𝑚|𝜓i
ph𝜓|𝑃𝑚|𝜓i,(2.19)
in acco d wi h he Bo n ule. Mo e gene al measu emen schemes a e desc ibed by
posi i e ope a o -
alued measu es
(POVMs) {𝐴𝑖} ha sa is y Í𝑖𝐴†
𝑖𝐴𝑖=𝐼[168].
16
2.6. PARAMETER ESTIMATION AND FISHER INFORMATION
a e di ec ly obse able, so he measu emen ope a o mus be conside ed. The measu emen can be
decomposed in o p ojec o s Π𝑙=|𝑙ih𝑙|, yielding p obabili ies
𝑝(𝑙|𝜃)=T 𝜌(𝜃)Π𝑙.(2.37)
Hence, 𝑝M(𝜃) o ms a
pa ame e ized p obabili y dis ibu ion
o e ou comes. A na u al way o quan i y
he dis ibu ion’s sensi i i y o 𝜃is by in oducing a dis ance measu e, such as he
Kullback–Leible (KL)
di e gence
o ela i e en opy [113],
𝐷KL 𝑝M(𝜃) k𝑝M(𝜃+𝛿)=Õ
𝑙
𝑝(𝑙|𝜃)logh𝑝(𝑙|𝜃)
𝑝(𝑙|𝜃+𝛿)i.(2.38)
A second-o de Taylo expansion o his di e gence a ound 𝛿=0 eco e s he CFIM [130]:
𝐼(𝜃)𝑖𝑗 =Õ
𝑙∈M
1
𝑝(𝑙|𝜃)
𝜕 𝑝(𝑙|𝜃)
𝜕𝜃𝑖
𝜕 𝑝(𝑙|𝜃)
𝜕𝜃𝑗,(2.39)
which can be in e p e ed as he Hessian o he ela i e en opy. La ge en ies imply s ong pa ame e
sensi i i y. I should also be no ed ha al e na i e di e en iable dis ance measu es would yield a CFIM
di e ing only by a mul iplica i e cons an .
Classical p obabili y dis ibu ions can be iewed as special cases o quan um s a es wi h diagonal densi y
ma ices [130]. A mo e gene al amewo k conside s dis ances be ween quan um s a es hemsel es,
a he han he measu emen -induced dis ibu ions. In his scena io, a dis ance measu e such as he
ideli y
is o en chosen, as i p o ides a na u al mono one me ic o pu e s a es [155]:
𝑓|𝜓(𝜃)i,|𝜓(𝜃+𝛿)i=h𝜓(𝜃)|𝜓(𝜃+𝛿)i2.
A co esponding dis ance is hen
𝑑|𝜓(𝜃)i,|𝜓(𝜃+𝛿)i=1−𝑓|𝜓(𝜃)i,|𝜓(𝜃+𝛿)i.
Expanding his ideli y-based dis ance o second o de in 𝛿p oduces he QFIM o pu e s a es:
F𝑖 𝑗 =4 Re[𝜕𝑖𝜓(𝜽)|𝜕𝑗𝜓(𝜽)−h𝜕𝑖𝜓(𝜽)|𝜓(𝜽)i𝜓(𝜽)|𝜕𝑗𝜓(𝜽) (2.40)
exac ly as desc ibed in [130]. C ucially, he e is a ma ix inequali y ela ing CFIM and QFIM:
𝐼(𝜃) ≤ F(𝜃),
which is de i ed using he Löwne o de o posi i e semide ini e ma ices [23], s a ing ha F(𝜃)−𝐼(𝜃) ≥
0is posi i e semi-de ini e. I implies ha he QFIM p o ides an uppe bound on he CFIM gene a ed by any
measu emen scheme pe o med on he quan um s a e. As will be discussed in Chap e 7, hese Fishe
in o ma ion ma ices and hei ela ionships a e in eg al o unde s anding he op imiza ion o PQC-based
RL agen s.
23

CHAPTER 2. QUANTUM INFORMATION AND COMPUTATION
2.7 Uni e sali y and classical simula ion
One o he co e insigh s o quan um compu ing is he p omise o uni e sal quan um compu a ion: he
abili y o implemen any uni a y ope a ion (o app oxima e i a bi a ily well) using a ini e se o ga es.
This no ion o uni e sali y unde pins he compu a ional powe o quan um de ices, enabling hem o, in
p inciple, sol e ce ain p oblems mo e e icien ly han classical compu e s. Howe e , no all quan um ga e
se s a e equal in exp essi e powe ; some can be e icien ly simula ed by classical means, while o he s
a e belie ed o anscend classical capabili ies. A quan um ga e se is called
uni e sal
i any uni a y
ope a ion on 𝑛qubi s can be ealized (exac ly o o a bi a y p ecision) using a ini e combina ion o ga es
om his se . In pa icula ,
uni e sal ga e se s
can densely gene a e he g oup 𝑆𝑈 (2𝑛). A well-known
example o a uni e sal se is composed o a bi a y single qubi o a ions and con olled-NOT ope a ions,
{𝑅𝑥(𝜃), 𝑅𝑦(𝜃), 𝑅𝑧(𝜃),CNOT}[144]. The wo-qubi con olled-NOT en angling ga e in combina ion wi h
a bi a y single-qubi ga es, can app oxima e any 𝑛-qubi ope a ion. A ounda ional esul conce ning
uni e sali y is he
Solo ay-Ki ae heo em
[58] s a ing ha any uni a y in 𝑆𝑈 (2𝑛)can be app oxima ed
o wi hin 𝜖by a ci cui o size O(log𝑐(1/𝜖)), whe e 𝑐is a cons an ypically nea 3 o 4. The Solo ay-
Ki ae heo em gua an ees ha uni e sal ga e se s a e no me ely heo e ical cons uc s bu also
e icien ly
implemen able
, as i ules ou exponen ial o e head in ga e decomposi ions.
Ano he commonly used uni e sal ga e se is he Cli o d+T se , which includes he Cli o d ga es and he
T-ga e o 𝜋/8ga e - {𝐻, 𝑆, CNOT,𝑇 }The T-ga e is a non-Cli o d ga e ha , when added o he Cli o d
se , makes he ga e se uni e sal. I u ns ou ha ci cui s consis ing solely o Cli o d ga es, along wi h
s abilize -s a e inpu s and measu emen in he compu a ional basis, a e classically simulable in polynomial
ime by i ue o he
Go esman-Knill heo em
[144]. When discussing whe he a quan um ci cui can be
classically simula ed
, one ypically dis inguishes wo egimes [142]:
•Weak simula ion: The ask is o
sample
om he same ou pu dis ibu ion p oduced by he
quan um ci cui . A weak simula o p oduces bi s ings wi h p obabili ies iden ical (o a bi a ily
close) o hose o he quan um de ice. This amoun s o gene a ing samples in he same manne
as he quan um ci cui wi hou necessa ily compu ing exac p obabili ies.
•S ong simula ion: The ask is o compu e
exac
p obabili ies (o app oxima e hem up o a ce ain
p ecision) o he quan um ci cui ’s measu emen ou comes. S ong simula ion is ypically mo e
demanding han weak simula ion, as i equi es knowledge o he comple e p obabili y dis ibu ion
a he han jus he abili y o gene a e samples.
Weak simula ion
asks whe he we can simply
sample
om he same dis ibu ion as he quan um ci cui ,
while
s ong simula ion
equi es compu ing exac p obabili ies o p obabili y ampli udes o a ce ain p e-
cision. The e o e, he Go esman-Knill heo em s a es ha he inal measu emen ou comes o Cli o d
ci cui s can be
s ongly simula ed
classically in polynomial ime. This demons a es ha en anglemen
24
2.7. UNIVERSALITY AND CLASSICAL SIMULATION
alone—such as he one gene a ed by cno ga es—does no necessa ily yield uni e sal quan um compu a-
ion no su pass classical me hods. Mo eo e , no ice ha a leas in he NISQ egime, quan um de ices
su e om noise ha hinde s he abili y o a quan um de ice exac ly compu e he ou pu p obabili ies.
The e o e, in p ac ice, quan um ad an age p oposals e ol e a ound he weak simula ion se ing demon-
s a ing ha
no e icien weak simula ion
is likely, o ce ain amilies o ci cui s, highligh ing a undamen al
gap be ween classical and quan um compu a ional powe .
De ini ion 2.7.1 (IQP ci cui s).Quan um ci cui s in which all ga es
commu e
and can hus be conside ed
o ac “ins an aneously.” Typically, he ga es a e diagonal in he compu a ional (𝜎𝑧) basis. Fo mally, an
IQP ci cui on 𝑁qubi s gene a e dis ibu ions
𝑝𝑖=|h𝑖|𝐻⊗𝑁𝑈𝑧𝐻⊗𝑁|𝑖i|2(2.41)
whe e 𝑈𝑧is composed o O(poly(𝑁)) commu ing ga es in he Z basis, o ins ance, {𝑇, 𝑍,𝐶𝑍,𝐶𝐶𝑍 ...}.
A p ominen example o a ci cui amily ha is conjec u ed o be ha d o simula e a e IQP ci cui s, as
de ined in De ini ion 2.7.1. These ci cui s ha e been employed in se e al quan um sup emacy p oposals,
as hey can p oduce highly non i ial sampling dis ibu ions despi e hei seemingly simple s uc u e o
commu ing ga es. Gi en N qubi s, IQP ci cui s s a in he |+i⊗𝑁s a e and a e measu ed in he Hadama d
basis. In be ween, hey apply a uni a y 𝑈𝑧composed o O(poly(𝑁)) commu ing ga es, o ins ance,
{𝑇, 𝑍,𝐶𝑍,𝐶𝐶𝑍 ...}[178] , as illus a ed in Figu e 6. They a e called ins an aneous because all ga es
commu e and he e is no empo al o de .
Figu e 6: Example o a s anda d IQP ci cui . Figu e aken om [63].
The IQP ci cui s a e conjec u ed o be ha d o weakly simula e classically unde compu a ional complexi y
assump ions such as he collapse o he polynomial hie a chy o i s second le el [125]. IQP ci cui s se e
as an impo an example o a subuni e sal model o quan um compu a ion - hey a e nei he uni e sal
in he s anda d sense no i ial o simula e. Despi e hei appa en simplici y (commu ing ga es, low
dep h), i is widely conjec u ed ha no e icien classical algo i hm can sample om he same dis ibu-
ion p oduced by a su icien ly la ge IQP ci cui unless he polynomial hie a chy in classical complexi y
25
CHAPTER 2. QUANTUM INFORMATION AND COMPUTATION
heo y collapses. Consequen ly, IQP ci cui s ha e ea u ed p ominen ly in p oposals o demons a ing
quan um enhancemen s o e classical me hods [66] sugges ing ha e en low-dep h QAOA could achie e
an exponen ial quan um speedup o op imiza ion p oblems [138].
No all es ic ed ci cui amilies, howe e , a e conjec u ed o be classically in ac able. A no able
opposi e
example is p o ided by
ma chga e ci cui s
[99], which can indeed be e icien ly simula ed on a classical
compu e . Ma chga es a e s uc u ed 𝐺(𝐴, 𝐵)ga es such ha ,
𝐺(𝐴, 𝐵)=©«
𝑝0 0 𝑞
0𝑤 𝑥 0
0𝑦 𝑧 0
𝑟0 0 𝑠
ª®®®®®¬
𝐴= 𝑝 𝑞
𝑟 𝑠 !𝐵= 𝑤 𝑥
𝑦 𝑧 !(2.42)
whe e 𝐴and 𝐵a e bo h in 𝑆𝑈 (2)wi h de (𝐴)=de (𝐵). Ma chga e ci cui s we e shown o be weakly
simulable o a bi a y p oduc s a e inpu s and/o he measu emen s a e o e a bi a ily many ou pu
qubi s, al hough, p o ided ha ma chga es ac only on nea es neighbo qubi s [35]. In p ac ice, ma ch-
ga e ci cui s o m a subuni e sal model analogous o IQP ci cui s in hei es ic ed s uc u e. Howe e ,
ma chga es become uni e sal when nex nea es neighbo connec i i y is allowed.
26
C h a p e
3
Va ia ional quan um algo i hms
VQAs ha e eme ged as powe ul ools o ackling compu a ionally demanding p oblems in he NISQ e a,
wi h he an icipa ion o achie ing quan um ad an ages [22]. These algo i hms make use o PQCs, as
desc ibed in Sec ion 2.5, and ope a e in a manne ha pa allels he s uc u e o
classical neu al ne wo ks
[77]. In essence, a VQA includes:
• A
cos unc ion
based on he ou pu s o measu emen s pe o med on a PQC.
• A se o unable
ee pa ame e s
in he ci cui , adjus ed by a classical op imiza ion p ocedu e un il
a desi ed a ge (encoded in he cos unc ion) is app oached.
Due o he analogy wi h neu al ne wo ks, such models a e some imes e e ed o as
quan um neu al
ne wo ks
[168, 5].
Figu e 7: Schema ic depic ion o a VQA in a hyb id quan um–classical amewo k. Adap ed om [45].
27
CHAPTER 3. VARIATIONAL QUANTUM ALGORITHMS
A gene ic VQA, illus a ed in Figu e 7, ypically ollows an i e a i e hyb id quan um–classical loop wi h wo
p incipal s ages:
1.
S a e p epa a ion and measu emen
: A PQC p epa es an ini ial hypo hesis, also known as
Ansa z
,
ha aims o sol e he a ge p oblem, and a cos unc ion is e alua ed using measu emen s o an
obse able on he esul ing quan um s a e.
2.
Classical op imiza ion
: A classical op imize upda es he
ee pa ame e s
in he PQC, s ee ing he
quan um s a e close o he solu ion in an i e a i e manne .
In
quan um-enhanced machine lea ning
[168], whe e a quan um de ice is employed o sol e a classical
machine lea ning ask, i becomes necessa y o embed classical da a in o he quan um sys em. Va ious
encoding echniques exis , each wi h dis inc ade-o s and complexi ies [112]. These will be add essed in
Subsec ion 3.1. Fo mally, i da a poin s a e d awn om a da ase 𝑥∼𝐷, and he pa ame e ized quan um
ci cui p oduces a densi y ma ix 𝜌(𝑥, 𝜃), a cos unc ion o he o m
𝑓𝜃(𝑥)=𝔼𝑥∼𝐷h𝑓T 𝜌(𝑥,𝜃)𝑂i(3.1)
can be de ined, whe e 𝑂is he measu emen ope a o , and 𝑓deno es a classical pos -p ocessing unc ion
o he expec a ion alue. He e,
𝜌(𝑥, 𝜃)=|𝜓(𝑥, 𝜃)ih𝜓(𝑥,𝜃)|=𝑈(𝜃)𝑆(𝑥)|0ih0|𝑆(𝑥)†𝑈(𝜃)†,
wi h 𝑆(𝑥)injec ing he da a 𝑥in o he ci cui , and 𝑈(𝜃)comp ising he lea nable ci cui pa ame e s.
Al e na i ely, in he
Heisenbe g pic u e
[122], he da a encoding is cap u ed by 𝜌(𝑥)alone, while he
pa ame e s a e encapsula ed in a ans o med obse able 𝑂𝜃=𝑈(𝜃)†𝑂 𝑈 (𝜃):
𝑓𝜃(𝑥)=𝔼𝑥∼𝐷h𝑓T 𝜌(𝑥)𝑂𝜃i,(3.2)
whe e 𝜌(𝑥)=𝑆(𝑥)|0i⊗𝑁h0|⊗𝑁𝑆(𝑥)†. In ei he pic u e, he cos unc ion is minimized by choosing
𝜃∗=a g min𝜃𝐶(𝜃),(3.3)
whe e 𝜃∗a e he op imal pa ame e s o he ci cui . Depending on he speci ic obse ables and asks,
s anda d loss unc ions such as mean squa ed e o o log-likelihood can be adap ed o PQC-based sce-
na ios [197]. The ou pu s o PQC-based models can be in e p e ed in wo p ima y ways, as shown in
Figu e 8 [168]:
1. De e minis ic models: The obse able is measu ed o ob ain a single scala alue (o mul iple
scala alues i mul iple obse ables a e measu ed). This app oach is commonplace in
supe ised
lea ning
se ings, leading o
a ia ional quan um classi ie s
[69, 171]. Fo example, conside an
28

obse able 𝑂=Ë𝑁−1
𝑖=0𝜎(𝑖)
𝑧in an 𝑁-qubi sys em, whe e 𝑓𝜃(𝑥) ∈ [−1,1]ac s analogously o a
logis ic o linea classi ie . Bina y classi ica ion can hen be pe o med by h esholding:
ˆ𝑦=sign𝑓𝜃(𝑥).
Such ci cui s a e o en ega ded as
explici quan um linea models
, because hey e ec i ely imple-
men linea unc ions in he induced
ea u e space
F[166, 167], i.e.,
𝑓𝜃(𝑥)=h𝜙(𝑥),𝑤𝜃iF,
wi h 𝜙(𝑥) ep esen ing he da a embedding and 𝑤𝜃 he lea nable pa ame e s.
2. P obabilis ic (gene a i e) models: Ra he han e u ning a single scala , he ull p obabili y dis-
ibu ion o e compu a ional basis s a es is conside ed. Fo ins ance, i Π𝑦=|𝑦ih𝑦|a e p ojec o s
on o basis s a es, hen he model ou pu s
𝑝(𝑦|𝑥)=T 𝜌(𝑥)Π𝑦.
Such models a e ele an o
unsupe ised lea ning
, se ing as
gene a i e
quan um models, also
e e ed o as
Bo n machines
[50]. In ce ain a chi ec u es based on Ising- ype Hamil onians, i has
been shown ha he dis ibu ions ealized by hese PQCs canno be e icien ly sampled classically,
he eby indica ing
quan um lea ning sup emacy
[55].
Figu e 8: PQC-based models depic ed as (a) de e minis ic (single scala obse able) and (b) p obabilis ic/-
gene a i e (dis ibu ion o e basis s a es). Adap ed om [168]. 𝑆(𝑥)deno es he da a encoding uni a y,
and 𝑊(𝜃) he lea nable ci cui .
Bo h de e minis ic and p obabilis ic o mula ions play essen ial oles in QML and QRL se ings. This
hesis explo es bo h pe spec i es in Chap e 5, whe e PQC-based agen s o RL a e analyzed unde a ious
modeling assump ions. The emainde o his chap e is di ided in o ou main sec ions, each add essing a
co e aspec o VQAs. Sec ion 3.1 ocuses on he me hods by which classical in o ma ion can be embedded
in o a quan um ci cui . I examines a ious encoding s a egies, discussing hei adeo s in e ms o
ci cui dep h, esou ce o e head, and exp essi i y. Sec ion 3.2 co e s s a egies o inc ease he exp essi e
powe o PQC-based models. Sec ion 3.3 co e s he me hods used o ain hese ci cui s in p ac ice. I
e iews classical op imiza ion algo i hms and g adien -based echniques, including how g adien s can be
es ima ed on quan um ha dwa e. Las ly, Sec ion 3.4 p esen one o he mos signi ican obs acles in
aining PQCs - he BP p oblem. This sec ion de ines he phenomenon, cha ac e izing how g adien s can
anish along wi h s a egies o mi iga e i s e ec s.
29
CHAPTER 3. VARIATIONAL QUANTUM ALGORITHMS
3.1 Classical da a encoding
Classical da a has his o ically been encoded in o quan um s a es ia
ampli ude encoding
[168], whe e a
ea u e ec o 𝑥=(𝑥0, . . . , 𝑥𝑀−1)is mapped o he ampli udes o a quan um s a e:
|𝑥i=1
k𝑥kÕ
𝑖
𝑥𝑖|𝑖i,(3.4)
wi h k𝑥kas he no m o 𝑥, and |𝑖i he 𝑖 h basis s a e. I 𝑀is no a powe o wo, he ec o is padded wi h
ze os. Ampli ude encoding is pa icula ly use ul in
aul - ole an
QML algo i hms, such as he quan um
suppo ec o machine [160], whe e ampli ude-based manipula ions a e bene icial. Howe e , in PQC-
based models, ampli ude encoding ypically becomes imp ac ical due o he la ge ci cui dep h in ol ed:
al hough he numbe o qubi s scales only
loga i hmically
wi h 𝑀, an
exponen ial
numbe o ga es is
usually equi ed [168].
An al e na i e, called
una y encoding
, was p oposed in [98] o educe ci cui dep h. Each ea u e 𝑥𝑖is
encoded in he ampli ude o a
one-ho
basis s a e |00 ···1𝑖···0i, i.e., only hose s a es o Hamming
weigh one. By es ic ing he encoding o a smalle subse o basis s a es, una y encoding can s ill
p oduce complex, en angled ea u e maps o ce ain quan um classi ica ion schemes, as shown in a
QML-based nea es cen oid classi ie [98]. I also unde lies he design o o hogonal quan um neu al
ne wo ks p ese ing he Hamming weigh [103].
A mo e di ec app oach is
basis encoding
, which ans o ms da a ea u es in o bina y s ings and encodes
hem in o o hogonal basis s a es. Consequen ly, all da a poin s become pai wise o hogonal in he Hilbe
space, i ially gua an eeing linea sepa abili y. Howe e , because e e y da a poin is o hogonal o e e y
o he , dis ance-based o inne -p oduc -based algo i hms (e.g., SVMs) may no ully exploi his encoding
[168].
Ano he widely employed me hod in QML is
angle encoding
, whe ein each da a componen 𝑥𝑖is con e ed
in o a single-qubi Pauli o a ion, o en wi h a numbe o qubi s scaling linea ly in 𝑀[168]. Fo ins ance,
|𝑥i=
𝑀−1
Ì
𝑖=0
𝑅𝜎𝑖𝑥𝑖|0i,(3.5)
whe e 𝑅𝜎𝑖(𝑥𝑖)=exp𝑖 𝑥𝑖
2𝜎𝑖is he Pauli o a ion ope a o , and 𝜎𝑖is he chosen Pauli ma ix (e.g., 𝑋,
𝑌, o 𝑍). One immedia e conside a ion is he
pe iodici y
: he model canno dis inguish 𝑥𝑖 om 𝑥𝑖+2𝜋,
necessi a ing no maliza ion o p ep ocessing in many asks. Despi e his, angle encoding is powe ul
in gene a ing a ied da a embeddings, since an exponen ial numbe o Pauli s ing combina ions (up o
O(4𝑁)in he 𝑁-qubi case) migh be used. In his se ing, he model’s exp essi i y can be inc eased by
epea ing he da a encoding s eps (o en called
da a euploading
[170, 151]), in e lea ed wi h pa ame e -
ized blocks, as depic ed in Figu e 9. Pe ez-Salinas e al. [151] p oposed a one-qubi ansa z composed o
epe i ions o a undamen al ga e 𝑈𝑈𝐴𝑇 , named o i s ela ion o he
uni e sal app oxima ion heo em
30
3.1. CLASSICAL DATA ENCODING
[90]. Speci ically,
𝑈𝑈𝐴𝑇 ®𝑥, ®𝑤, 𝛼, 𝜑=𝑅𝑧2®𝑤· ®𝑥+2𝛼𝑅𝑦2𝜑,(3.6)
𝑈𝑥, Θ=
𝐿−1
Ö
𝑘=0
𝑈𝑈𝐴𝑇 ®𝑥, ®𝑤(𝑘), 𝛼(𝑘), 𝜑(𝑘),(3.7)
whe e Θ={®𝑤, 𝛼, 𝜑}𝑘is he collec ion o ainable pa ame e s and 𝐿deno es he numbe o epe i-
ions. In he limi 𝐿→ ∞, he ci cui can app oxima e a b oad class o unc ions, hus se ing as a
uni e sal unc ion app oxima o
in he quan um se ing. Mul iqubi gene aliza ions inco po a e en angling
ga es be ween hese single-qubi blocks, hus esembling ully connec ed classical neu al ne wo ks (see
Figu e 10).
Figu e 9: A “se ies” da a euploading a chi ec u e o a single qubi , epea ing encoding and pa ame e ized
blocks mul iple imes.
Figu e 10: A mul i-qubi da a euploading scheme. Each epe i ion in e lea es da a encoding wi h pa ame-
e ized ope a ions, akin o ully connec ed laye s in classical neu al ne wo ks. Figu e sou ce: Pennylane’s
u o ial on da a euploading.
Al hough he uni e sal unc ion-app oxima ing p ope y is well-unde s ood o single-qubi a chi ec u es,
i s ull gene aliza ion o mul iqubi sys ems emains an a ea o ongoing esea ch [170]. Howe e , he
p e ailing iewpoin is ha a Fou ie -based analysis o PQC-based models suppo s po en ial uni e sali y in
mul iqubi ci cui s as well. Da a euploading models gene ally canno be exp essed as a s anda d quan um
linea model since 𝑓𝜃(𝑥)=T [𝜌(𝑥, 𝜃)𝑂𝜃]and da a ga es appea a mul iple laye s. In p inciple, such
laye ing p ecludes collec ing all pa ame e ized ga es a he end o he ci cui (i.e., commu ing e e y hing
in o a single block). Howe e , Je bi e al. [95] showed ha bo h app oxima e and exac
linea
ealiza ions
o da a euploading ci cui s can be cons uc ed in highe -dimensional Hilbe spaces. Fo ins ance, an
app oxima e ealiza ion may be achie ed by combining
basis-encoded
ea u es in sepa a e egis e s, hen
applying con olled ga es o emula e he o iginal da a euploading blocks [Figu e 11(a)]. A ully
exac
linea mapping can also be done using ga e elepo a ion, mo ing all da a-dependen ga es in o ancillas
[Figu e 11(b)]. These esul s highligh ha da a euploading ci cui s, while no i ially linea in he o iginal
space, can none heless exhibi explici linea i y in a sui ably enla ged o modi ied con igu a ion.
31
CHAPTER 3. VARIATIONAL QUANTUM ALGORITHMS
Figu e 11: App oxima e (le ) and exac ( igh ) linea ealiza ions o da a euploading models, using (a) ba-
sis encoding and (b) ga e elepo a ion. Adap ed om [95].
O e all, he choice o da a encoding signi ican ly in luences he ep esen a ional powe and ainabili y o
quan um models. The nex sec ions explo e he
exp essi i y
o PQC-based models and he challenges in
op imizing
hem, including phenomena such as BPs.
3.2 Exp essi i y o quan um machine lea ning models
In Subsec ion 2.5, he exp essi i y o PQCs was discussed in e ms o
Hilbe space co e age
: ci cui s ha
mo e closely app oxima e a Haa - andom uni a y a e deemed mo e exp essi e. Howe e , in he con ex
o machine lea ning, a da a-dependen no ion o exp essi i y is equi ed, one ha explici ly cha ac e izes
he amily o unc ions a PQC-based model can gene a e o i gi en da a. Indeed, any PQC-based model
can be w i en as a
unca ed
Fou ie se ies [170]. Fo he simples case o one-dimensional da a 𝑥, a
PQC-based unc ion can be exp essed as
𝑓𝜃(𝑥)=Õ
𝑤∈Ω
𝑐𝑤(𝜃)exp𝑖 𝑤 𝑥,(3.8)
whe e he equency se Ωis ini e and 𝑐𝑤(𝜃)=𝑐𝑤(𝜃)∗. As a conc e e example, conside a
da a
euploading
model o he o m
𝑈(𝑥, 𝜃)=𝑊𝜃𝐿𝐿−1
Ö
𝑘=0
𝑆(𝑥)𝑊𝜃𝑘(3.9)
ac ing on 𝑁qubi s. He e, 𝑆(𝑥)is an encoding uni a y, which can o en be iewed as 𝑆(𝑥)=exp𝑖 𝑥 𝐻 ,
wi h 𝐻an 𝑁-qubi He mi ian ma ix o dimension 𝑑≤2𝑁. Wi hou loss o gene ali y, 𝐻may be aken o
be diagonal (o uni a ily ans o med in o a diagonal o m). As illus a ed in Figu e 12, epea ed applica ion
o such diagonal encodings, ollowed by pa ame e ized uni a ies 𝑊(𝜃𝑘), yields e ms ha commu e in
hei exponen ial expansions, gi ing ise o sums o exponen ials o eigen alue sums.
When 𝑆(𝑥)is diagonal, he ampli udes o he inal quan um s a e amoun o p oduc s o exponen ials
o he o m 𝑒𝑖 𝜆𝑗𝑥, and hese exponen ials commu e. Fo ins ance, he 𝑖 h en y o 𝑈(𝑥, 𝜃)|0i𝑖can be
w i en as a sum o e p oduc s o exponen ials,
𝑈(𝑥, 𝜃)|0i𝑖=
𝑑
Õ
𝑗1,...,𝑗𝐿=1
exph−𝑖𝜆𝑗1+···+𝜆𝑗𝐿𝑥i×𝑊(𝐿+1)
𝑖 𝑗𝐿··· 𝑊(2)
𝑗2𝑗1𝑊(1)
𝑗11,(3.10)
32
3.4. THE BARREN PLATEAU PHENOMENON
3.4 The ba en pla eau phenomenon
To es ima e g adien s while eusing he quan um de ice ia
pa ame e -shi ules
(Subsec ion 3.3) is
indeed a aluable ea u e o PQC-based models. Howe e , compa ed wi h classical backp opaga ion,
pa ame e -shi ypically does no achie e equally a o able scaling, pa icula ly as he numbe o pa am-
e e s g ows. Mo eo e , e en i g adien s can be ob ained e icien ly, a u he challenge a ises in he o m
o BPs, whe e he aining landscape becomes exponen ially la wi h espec o he numbe o qubi s,
he eby hinde ing op imiza ion a scale. Figu e 14 illus a es he la ening e ec .
Figu e 14: Ba en pla eaus and he la ening o he aining landscape o andom pa ame e ized quan um
ci cui s as a unc ion o he numbe o qubi s. Image sou ce:
Machine Lea ning wi h Quan um Compu e s
by Schuld e al. [168].
The in es iga ion o BPs in he aining landscape is o en e e ed o as he
ainabili y analysis
o PQC
models. In his se ing, g adien s a e a e aged o e a andom sampling o pa ame e s 𝜃. In a BP he a -
e age g adien is ze o. Howe e , examining he a e age g adien alone is insu icien ; he second momen
( a iance) o hese g adien s mus also be conside ed o asce ain how quickly hese luc ua e, and how
a e hey concen a ing nea hei a e age alue o ze o. BPs end o a ise whene e he ci cui e ec i ely
“sc ambles” in o ma ion, o example by being su icien ly deep o using global measu emen s [127], by
ope a ing unde noisy condi ions [201], o by gene a ing s a es wi h high en anglemen en opy in ela ion
o ano he sys em [124]. The unde lying causes o his phenomenon will be explo ed, beginning wi h he
seminal wo k o McClean e ,al. [127] and ex ending o mo e ecen ad ances.
Fo a PQC-based cos unc ion
𝐶(𝜃)=h0|𝑉(𝜃)†𝑂 𝑉 (𝜃)|0i,(3.27)
he a iance o he pa ial de i a i e wi h espec o a pa ame e 𝜇is exp essed as
𝕍(𝜕𝜇𝐶(𝜃)) =h𝜕𝜇𝐶(𝜃)2i𝑉− h𝜕𝜇𝐶(𝜃)i2
𝑉,(3.28)
whe e he expec a ion is aken o e he uni a ies 𝑉(𝜃)in he ansa z. In a BP, h𝜕𝜇𝐶(𝜃)i =0and he
a iance decays exponen ially wi h he sys em size 𝑁, which means ha he pa ial de i a i e concen a es
39

CHAPTER 3. VARIATIONAL QUANTUM ALGORITHMS
exponen ially a ound ze o as he numbe o qubi s g ows. Because he a iance anishes exponen ially
in 𝑁, an exponen ially g owing numbe o sho s is necessa y o dis inguish di ec ions in he landscape,
causing he op imize o pe o m a andom walk [127]. A asmi h e al. [12] showed ha BPs a e equi alen
o he cos unc ion i sel “concen a ing,” a he han me ely he pa ial de i a i e, i.e.,
𝕍(𝜕𝜇𝐶(𝜃)) ∈ O1
𝛼𝑁=⇒𝕍𝐶(𝜃+𝜎) −𝐶(𝜃)∈ O1
𝛼𝑁.(3.29)
Hence, BPs a ec no only g adien -based, bu also g adien - ee me hods [11], and e o mi iga ion alone
does no esol e hem [200].
The ques ion hen a ises as o which uni a ies 𝑉(𝜃)yield anishing g adien s. In he seminal wo k by Mc-
Clean e al. [127], he assump ion was ha 𝑉(𝜃)is sampled om he
Haa dis ibu ion
(Subsec ion 2.5).
Because he a iance is a second momen , he esul is alid o uni a y 2-designs (i.e., expec a ions
appea Haa -like up o second o de , see De ini ion 3.4.1).
De ini ion 3.4.1 (Uni a y 𝑡-design).A p obabili y dis ibu ion Do e he uni a y g oup U(𝑑)(whe e 𝑑
is he dimension o he Hilbe space) is called a
uni a y
𝑡
-design
i i ep oduces he momen s o he Haa
measu e up o o de 𝑡. Conc e ely, o any polynomial 𝑝o deg ee up o 𝑡in he ma ix elemen s o 𝑈
and 𝑈†, he a e age o 𝑝unde Dma ches ha unde he Haa dis ibu ion up o he 𝑡- h momen .
By spli ing 𝑉(𝜃)in o le / igh uni a ies 𝑈𝐿,𝑈𝑅a ound a single-pa ame e ga e 𝐺(𝜇), McClean e al.
showed ha
𝕍(𝜕𝜇𝐶(𝜃)) ∈ O(2−𝑁),(3.30)
p o ided ha a leas one o 𝑈𝐿o 𝑈𝑅is a 2-design. This analysis did no cons ain he measu emen
obse able no he inpu s a e, conside ing a
global
obse able ac ing on all qubi s. In p ac ice, uni a ies
in mos ansa z cons uc ions ollow a
pe iodic s uc u e
:
𝑉(𝜃)=
𝐿−1
Ö
𝑙=0
𝑈(𝜃𝑙)𝑊𝑙,(3.31)
whe e each laye 𝑈(𝜃𝑙)is pa ame e ized, and 𝑊𝑙is a 𝜃-independen uni a y epea ed 𝐿 imes. Ce ezo
e al. [42] la e showed ha p io esul s can di e unde
local
measu emen s, so BPs also depend on
ci cui dep h: local measu emen s on su icien ly deep en angled ci cui s e ec i ely beha e as global. In
pa icula , when local 2-designs (i.e., each laye is a 2-design) a e combined wi h local measu emen s, a
a o able ainable egion is obse ed a loga i hmic dep h, as illus a ed in Figu e 15.
When log(𝑁)qubi s a e measu ed opologically, he pa ial-de i a i e a iance anishes polynomially in
𝑁,
𝕍(𝜕𝜇𝐶(𝜃)) ∈ Ω1
poly(𝑁),(3.32)
unde log(N)-dep h condi ions, hus enabling a
ainable
egion whe e only a polynomial numbe o sho s
is equi ed. A dep h O(poly(log(𝑁))), he decay ou paces any polynomial bu emains subexponen ial,
𝕍(𝜕𝜇𝐶(𝜃)) ∈ Ω1
2poly(log(𝑁)).(3.33)
40
3.4. THE BARREN PLATEAU PHENOMENON
Figu e 15: T ainabili y egion as a unc ion o ci cui dep h. Image sou ce:
Cos unc ion dependen ba en
pla eaus in shallow quan um neu al ne wo ks
by Ce ezo e al. [42].
Rudolph e al. [162] ex ended his esul o algeb aically local (o
low-bodied
) obse ables. In deepe
ci cui s o ming a 2-design, local measu emen s e ec i ely ac as global ones.
In many ML asks, da a is i s
encoded
in o a (possibly en angled) quan um s a e, which can in oduce
BPs e en i he subsequen ansa z is simple o e en o m a p oduc s a e [197]. Leone e al. [114] posi ed
ha ha dwa e-e icien ci cui s wi h a ea-law en angled da a s a es can be p omising o demons a ing
quan um ad an age, bu emain subjec o a “deadly iad” o ci cui exp essi i y, global measu emen s,
and en angled encoding s a es. Ragone e al. [158] p oposed a uni ying Lie-algeb aic amewo k, shown
in Figu e 16, ha accoun s o hese majo BP sou ces.
In ha wo k, i was shown ha i 𝑂∈𝑖𝔤o 𝜌∈𝑖𝔤(wi h 𝔤as he DLA om De ini ion 3.2.1), hen
he mean o he loss unc ion anishes o e semisimple componen s 𝔤1⊕ ··· ⊕𝔤𝑘−1, and he a iance
beha es as:
𝕍𝜽𝐶(𝜃)=
𝑘−1
Õ
𝑗=1
P𝔤𝑗(𝜌)P𝔤𝑗(𝑂)
dim(𝔤𝑗),(3.34)
whe e P𝔤𝑗(𝜌)and P𝔤𝑗(𝑂)deno e he pu i y o 𝜌and 𝑂 es ic ed o subalgeb a 𝔤𝑗. This esul excludes
41
CHAPTER 3. VARIATIONAL QUANTUM ALGORITHMS
Figu e 16: Uni ied heo y o ba en pla eaus connec ing mul iple p io esul s. Image sou ce:
A Uni ied
Theo y o Ba en Pla eaus o Deep Pa ame ized Quan um Ci cui s
by Ragone e al. [158].
s a es o measu emen s ou side he DLA, bu an exponen ially la ge dim(𝔤)can s ill induce
exp essi i y-
based BPs
[43] i.e.
dim(𝔤)=𝛼𝑛, 𝛼 >1=⇒𝕍𝜽ℓ𝜽(𝜌,𝑂)∈ O1
𝛼𝑛.(3.35)
By con as , polynomially la ge DLAs alone do no necessa ily p oduce BPs; he inpu s a e and measu e-
men de e mine whe he hose a ise. Equa ion 3.34 u he p o ides
exac
exp essions o he a iance in
he assump ion ha he ci cui o ms a 2-design a su icien dep h. 𝜖-
app oxima e
2-designs a e gene a ed
when he numbe o pa ame e ized blocks 𝐿sa is ies
𝐿≥log(1/𝜖)
log(1/k𝐴k2),(3.36)
whe e k𝐴k2is he Hilbe –Schmid no m cha ac e izing how much he second momen s o one ci cui
laye de ia e om Haa (c . Algo i hm 1). A single laye al eady o ms a 2-design i k𝐴k2=0.
Such 𝑡-design assump ions do no usually hold in p ac ice. Le che e al. [115] de i ed igh loss and g a-
dien bounds o b oad classes o PQCs and a bi a y obse ables, wi hou elying on 𝑡-design a gumen s.
Fo a ci cui obeying Equa ion (3.31) and any 𝐻=Í𝑖𝛼𝑖𝑃𝑖wi h 𝑃𝑖∈ {𝐼, 𝑋, 𝑌, 𝑍 }𝑁, e e y Pauli e m
con ibu es independen ly o he a iance:
𝕍𝜃[𝐶(𝜃)] =Õ
𝑖
𝛼2
𝑖𝕍𝜃[𝐶(𝜃)𝑖]
whe e each con ibu ion is igh ly bounded by
Ω(𝜌)𝔼𝜃h1
4Δ𝜃
𝑖i≤𝕍𝜃𝐶(𝜃)𝑖≤𝔼𝜃h1
2Δ𝜃
𝑖i,(3.37)
whe e Δ𝜃
𝑖is he backwa ds
ligh -cone
o 𝑃𝑖, i.e. he numbe o qubi s on which 𝑈†(𝜃)𝑃𝑖𝑈(𝜃)ac s
non i ially, and
Ω(𝜌)=Õ
𝑖
T 𝑃𝑖𝜌2
ep esen s a measu e o o hogonali y ha quan i ies he po ion o 𝜌o hogonal o he i s laye o
o a ions.
42
3.4. THE BARREN PLATEAU PHENOMENON
Se e al s a egies ha e been explo ed o mi iga e BPs, including
laye wise lea ning
[187], specialized pa-
ame e ini ializa ions [78, 210, 164], and ca e ully designed shallow-dep h a chi ec u es [154]. None he-
less, Ce ezo e al. [44] p o ed ha while ce ain es ic i e app oaches may a oid BPs, hey o en ende
he model classically simulable, complica ing e o s o achie e quan um ad an age. Ye , a con i ed ex-
ample was also p o ided, indica ing ha “sma e ini ializa ions” migh enable powe ul, nonclassically
simulable PQC-based models.
C ucially, all hese BP esul s hold independen ly o a ML objec i e, as mos bounds a e de i ed unde a
linea expec a ion- alue cos wi hou explici da a. Thanasilp e al. [197] ound ha s anda d ML losses,
such as
mean squa ed e o
and
log-likelihood
, do no signi ican ly al e BP beha io , al hough da a encod-
ing can in oduce addi ional complexi y. Mo eo e , a less is known abou how da a euploading a ec s
he ainabili y o models. On one hand, i may enable as e en y in o he o e pa ame e ized egime and
mi iga e spu ious minima [111], bu on he o he , i also inc eases he model’s o e all exp essi i y and
dep h, which is known o exace ba e BPs. Fu he mo e, ci cui s wi h exponen ially la ge DLAs gene ally
equi e an exponen ial numbe o pa ame e s o each o e pa ame e iza ion, hus necessi a ing signi ican
dep h ha leads o a 2-design, la ening he aining landscape. How o balance he exp essi e powe
in oduced by da a euploading wi h gua an eed ainabili y s ill emains an open ques ion.
43
Chap e
4
Rein o cemen lea ning
The supe ised lea ning amewo k o machine lea ning can be in e p e ed as ha ing an agen lea n
om a eache ha knows he co ec answe o e e y ques ion—
labels
. The e o e, he agen is usually
limi ed o he amoun o knowledge imposed by he eache and canno easily su pass o p ope ly answe
ques ions ou side he men o ’s expe ise.
Rein o cemen Lea ning
(RL), on he o he hand, emo es he
eache and le s he s uden — e e ed o as he
agen
—lea n by in e ac ing wi h he en i onmen , which
ul ima ely cap u es he consequences o he agen ’s ac ions. This di e en lea ning pa adigm b ings us
close o
gene al a i icial in elligen agen s
[182]. C ucially, i closely e lec s how humans ac ually lea n,
being s ongly inspi ed by biological models o lea ning [194]. RL is esponsible o majo b eak h oughs
in a i icial in elligence, such as he amous AlphaZe o [181], which bea he wo ld champion o Go, o
MuZe o
[165], which gene alized he algo i hm o o he complex en i onmen s. Mo e ecen ly, RL has
been used o sol e complex p oblems ou side o games and p o ide solu ions o eal-wo ld challenges,
such as as e ma ix mul iplica ion and so ing algo i hms [70, 123], quan um eedback con ol [73],
quan um ci cui op imiza ion [72], and mo e.
In his chap e , we in oduce he basic concep s o RL and he main algo i hms used in he ield and
co e ed in his wo k. We begin wi h he ma hema ical ounda ions o RL in Subsec ion 4.1. Then, we
p esen he concep s o policies, alue unc ions, and how he agen can achie e op imal beha io in
Subsec ion 4.2. A e wa ds, we explo e app oxima ion algo i hms o es ima ing alue unc ions and
policies in Subsec ions 4.3 and 4.4, espec i ely.
4.1 Founda ions
The RL pa adigm consis s o wo en i ies: he
Agen
and he
En i onmen
, o ming he well-known
Agen -
En i onmen in e ace
[194], as illus a ed in Figu e 17. In his se ing, he agen lea ns di ec ly om
44

4.1. FOUNDATIONS
in e ac ions wi h i s su ounding en i onmen , wi hou equi ing supe ision o comple e models o he
en i onmen .
Figu e 17: Agen -En i onmen in e ace. Image adap ed om
Rein o cemen Lea ning: An in oduc ion
by
Su on e al. [194]. 𝑂𝑡,𝐴𝑡, and 𝑅𝑡a e he obse a ion, ac ion, and ewa d o he agen a ime s ep 𝑡.
Le 𝑆and 𝐴be he space o all possible s a es and ac ions de ined o a gi en en i onmen , espec i ely.
A ime s ep 𝑡, he agen obse es he s a e o he en i onmen 𝑂𝑡, which can be a
pa ial obse a ion
o he ue s a e 𝑠𝑡∈𝑆(e.g., he cu en s a e in a game o poke , whe e he agen obse es only he
ca ds on he able wi hou in o ma ion abou he emaining ca ds). Gi en such a s a e, he ac ion se
can be s a e-dependen , so he agen selec s an ac ion 𝑎𝑡 om he se o a ailable ac ions o ha s a e,
𝐴𝑜𝑡. This ac ion al e s he en i onmen ’s s a e. Indeed, such ac ions can ha e
de e minis ic
o
s ochas ic
ou comes. Fo ins ance, in chess, an ac ion always leads o he same esul ing s a e, whe eas a cleaning
obo ha pe o ms a o wa d mo ion ac ion migh slip and end up in a di e en s a e han in ended.
Thus, in gene al, he en i onmen is desc ibed by a
ansi ion unc ion
ha cap u es he dynamics o he
en i onmen as a p obabili y dis ibu ion o e he possible nex s a es, gi en he cu en ac ion and s a e o
he en i onmen , 𝑝(𝑠𝑡+1|𝑠𝑡, 𝑎𝑡), such ha Í𝑠𝑡+1𝑝(𝑠𝑡+1|𝑠𝑡, 𝑎𝑡)=1. Since his in o ma ion is encoded
in he en i onmen , he agen does no ha e access o i .
E e y ime he agen pe o ms an ac ion, i obse es he new s a e 𝑠𝑡+1o he en i onmen and ecei es a
ewa d 𝑟𝑡+1. The ewa d is a scala alue ha quan i ies he immedia e bene i o he ac ion aken by he
agen . The way he ewa d is deli e ed a ies wi h he en i onmen , as he ewa d can be s a e-dependen
and/o s a e-ac ion-dependen . None heless, since he ewa d is he eedback o he ac ion aken, he
main goal o he agen is o
maximize he expec ed ewa d
. The
mul i-a med bandi
(MAB) en i onmen is
pe haps he simples en i onmen . In his se ing, he e a e 𝑘slo machines ha he agen may choose
om. We can in e p e he MAB as a
s a eless
en i onmen since he agen pulls one o he 𝑘a ms,
ecei es a ewa d, and mo es o he nex ime s ep, co esponding e ec i ely o he same s a e. The
agen ’s goal is o lea n he bes a m o pull in o de o maximize he expec ed ewa d.
Howe e , in mos p ac ical lea ning scena ios, asks a e mo e complica ed since he agen will indeed
be pe o ming ac ions in a
con inuing ask
wi h mul iple s a es. In ha case, he agen is aced wi h
he concep o
delayed ewa d
: i mus lea n how o balance immedia e and long- e m ewa ds. Indeed,
he agen should look ahead be o e making a decision, bu no oo a in o he u u e. Rewa ds can be
penalized o discoun ed by a
discoun ac o
𝛾∈ [0,1] ha weighs u u e ewa ds such ha immedia e
ewa d weighs mo e han ewa ds a he in o he u u e:
45
CHAPTER 4. REINFORCEMENT LEARNING
𝐺𝑡=∞
Õ
𝑘=0
𝛾𝑘𝑟𝑡+𝑘+1.(4.1)
𝐺𝑡is also known as he
e u n
o he agen — he cumula i e discoun ed ewa d he agen ge s s a ing
a ime s ep 𝑡. Thus, conside ing 0≤𝛾<1ensu es ha he e u n is ini e e en o in ini ely long
ajec o ies. In p ac ice, a ask is usually unca ed wi h a ini e
ho izon
𝑇,
𝐺𝑡=
𝑇−𝑡−1
Õ
𝑘=0
𝛾𝑘𝑟𝑡+𝑘+1,(4.2)
whe e he
e ec i e ho izon
depends en i ely on he discoun ac o 𝛾used o he ask. Typically,𝑇e ec i e =
O( 1
1−𝛾)[6]. The discoun ac o in luences he policy, which in u n in luences he en i onmen , which
in luences he da a he agen ac ually sees du ing aining. This is a challenging p oblem since he agen
mus lea n how o balance explo a ion and exploi a ion. The agen mus explo e he en i onmen o lea n
i s dynamics and exploi he knowledge i has o maximize he expec ed ewa d. This is also known as he
explo a ion-exploi a ion dilemma
, as illus a ed in Figu e 18.
Figu e 18: Explo a ion-Exploi a ion dilemma. Image om he UC Be keley AI cou se.
The agen mus no explo e a all imes, since ha would e ec i ely educe o a b u e- o ce sea ch. On he
o he hand, he agen mus no exploi a all imes, o i would no lea n he en i onmen ’s dynamics, po-
en ially s icking o a subop imal policy. The e o e, a p ope balance be ween explo a ion and exploi a ion
is essen ial, and so is he design o he policy unde which ac ions a e pe o med in he en i onmen .
The ma hema ical amewo k behind an RL p oblem is he
Ma ko Decision P ocess
(Ma ko Decision
P ocess (MDP)), which encapsula es such sequen ial decision-making p oblems in he en i onmen ’s o -
mula ion. I can be iewed as a di ec ed g aph whe e nodes a e possible s a es o he en i onmen , and
edges ep esen he ansi ion unc ion o a gi en s a e-ac ion pai . This g aph is ex ended wi h he ewa d
unc ion associa ed wi h s a e o s a e-ac ion pai s, as illus a ed in Figu e 19.
46
4.1. FOUNDATIONS
Figu e 19: Racing ca MDP. Image om he UC Be keley AI cou se. The MDP is ep esen ed wi h a se o
s a es 𝑆={Cool,Wa m,O e hea ed}and a se o ac ions 𝐴={Slow,Fas }. The ewa d unc ion in his
en i onmen depends on s a e-ac ion pai s.
The en i onmen is hus assumed o be Ma ko ian, espec ing he
Ma ko p ope y
—
he u u e is inde-
penden o he pas gi en he p esen
:
𝑝(𝑠𝑡+1|𝑠0, . . . ,𝑠𝑡)=𝑝(𝑠𝑡+1|𝑠𝑡).(4.3)
An MDP is a uple (𝑆, 𝐴, 𝑃, 𝑅,𝛾)whe e 𝑆is he se o s a es, 𝐴is he se o ac ions, 𝑃is he ansi ion
unc ion, 𝑅is he ewa d unc ion, and 𝛾is he discoun ac o . The MDP se es as a model o he
en i onmen , which he agen uses o lea n he op imal
policy
𝜋∗ ha maximizes he expec ed e u n.
The policy 𝜋:𝑆→𝐴maps s a es o ac ions. Typically, he policy is in e p e ed as
de e minis ic
once he
agen knows which ac ion is op imal in a gi en s a e. In gene al, howe e , he policy will be a p obabili y
dis ibu ion o e he se o a ailable ac ions o each s a e, 𝜋(𝑎|𝑠), also deno ed as
𝜋(𝑎|𝑠)=𝑝(𝑎𝑡=𝑎|𝑠𝑡=𝑠).(4.4)
The op imal policy 𝜋∗can indeed be
s ochas ic
, o ins ance, in se ings whe e he en i onmen can
only be pa ially obse ed. Ne e heless, in a ully obse ed en i onmen , he e always exis s an op imal
de e minis ic policy [193]. As he agen ollows a policy by in e ac ing wi h he en i onmen , i sequen ially
collec s (o gene a es) ajec o ies 𝜏o s a es, ac ions, and ewa ds,
𝜏=(𝑠0, 𝑎0,𝑟1,𝑠1, 𝑎1, 𝑟2, . . . , 𝑠𝑇).(4.5)
The e o e, he agen expe iences a ajec o y wi h p obabili y
𝑝(𝜏)=
𝑇−1
Ö
𝑡=0
𝜋(𝑎𝑡|𝑠𝑡)𝑝(𝑠𝑡+1|𝑠𝑡, 𝑎𝑡).(4.6)
47
CHAPTER 4. REINFORCEMENT LEARNING
I is also c ucial o no e wo main pa adigms in RL:
model-based
and
model- ee
RL. In he o me , he
agen has access o o lea ns he en i onmen ’s dynamics. He e, RL educes o planning in la ge s a e-
ac ion spaces whe e dynamic p og amming is exploi ed [194]. In he la e , which is he mos common
scena io in RL, he agen does no ha e access o he en i onmen ’s dynamics. I mus lea n he op imal
policy by in e ac ing wi h he en i onmen —usually om he ewa d p o ided as eedback. Many ad anced
RL algo i hms combine bo h pa adigms, o en in simula ion-based se ings, whe e expe ience can be
ga he ed ia s anda d ial-and-e o while also lea ning a model o he wo ld o pe o m op imal planning
[165]. In p ac ice, howe e , we may no wish o lea n he en i onmen ’s model (e.g., because i migh be
unnecessa y o oo complica ed). Model- ee RL is gene ally he mos e sa ile pa adigm, enabling lea ning
o op imal beha io s o di e se en i onmen s wi hou needing o lea n he en i onmen ’s dynamics, as
long as he cos o en i onmen sampling is no oo high. In his wo k, we ocus on model- ee RL algo i hms
and hus will no co e model-based RL algo i hms, e e ing he eade o [194] o a comp ehensi e
in oduc ion o ha opic.
4.2 Value unc ions and op imal beha io
To lea n wi hou knowing he en i onmen ’s dynamics, unde s anding he ole o
alue unc ions
is c ucial.
These unc ions es ima e he expec ed e u n o he agen , s a ing om a gi en s a e and ollowing a gi en
policy, and hus quan i y how bene icial i is o he agen o be in a ce ain s a e o o pe o m a ce ain
ac ion in ha s a e.
The
s a e- alue unc ion
𝑉𝜋(𝑠) o a s a e 𝑠∈𝑆is he expec ed e u n, s a ing om s a e 𝑠and ollowing
policy 𝜋o e a ho izon 𝑇:
𝑉𝜋(𝑠)=𝔼𝜋[𝐺𝑡|𝑠𝑡=𝑠]=𝔼𝜋h𝑇−𝑡−1
Õ
𝑘=0
𝛾𝑘𝑟𝑡+𝑘+1𝑠𝑡=𝑠i.(4.7)
Simila ly, he
ac ion- alue unc ion
𝑄𝜋(𝑠, 𝑎) o a s a e-ac ion pai (𝑠, 𝑎) ∈ 𝑆×𝐴is he expec ed e u n
s a ing om s a e 𝑠, aking ac ion 𝑎, and ollowing policy 𝜋 o a ho izon 𝑇:
𝑄𝜋(𝑠, 𝑎)=𝔼𝜋[𝐺𝑡|𝑠𝑡=𝑠, 𝑎𝑡=𝑎]=𝔼𝜋h𝑇−𝑡−1
Õ
𝑘=0
𝛾𝑘𝑟𝑡+𝑘+1𝑠𝑡=𝑠, 𝑎𝑡=𝑎i.(4.8)
These alue unc ions a e i al o de e mining op imal beha io . They can be de ined ecu si ely, o ming
he well-known
Bellman equa ions
:
48
4.3. VALUE-BASED METHODS
Figu e 22: Ca pole en i onmen . Image adap ed om
G okking Deep Rein o cemen Lea ning
by Mo ales
e al. [139]. The agen mus balance he pole by mo ing he ca le o igh .
Ca pole is a well-known en i onmen [19] in which he agen applies le o igh impulses o keep he
pole balanced. Thus, he ac ion space is small, bu he s a e space— ep esen ed by he ca ’s posi ion,
eloci y, he pole’s angle, and angula eloci y—is con inuous. A lookup able o all s a e-ac ion pai s is
hus in easible.
Mul iple solu ions ha e been s udied in RL o add ess his
cu se o dimensionali y
— o example, abs ac ion
[6] and dimensionali y educ ion [189]. One widely success ul app oach is o use
unc ion app oxima o s
o he ac ion- alue unc ion, commonly ia
Neu al Ne wo ks
since hey a e uni e sal unc ion app oxima-
o s [77]. This combina ion o
Deep Lea ning
and RL is known as
Deep Rein o cemen Lea ning
[139].
In his se ing, he ac ion- alue unc ion is pa ame e ized by a neu al ne wo k 𝑄(𝑠, 𝑎;𝜃). We hen seek o
lea n he pa ame e ec o 𝜃∗ ha app oxima es he op imal ac ion- alue unc ion 𝑄∗(𝑠, 𝑎). This ne wo k
is called a
Deep Q-Ne wo k
(DQN) [136]. Figu e 23 shows an abs ac DQN o Ca pole.
Figu e 23: Deep Q-Ne wo k o he Ca pole en i onmen . Image adap ed om
G okking Deep Rein o ce-
men Lea ning
by Mo ales e al. [139]. The agen encodes he s a e in o he ne wo k, which ou pu s he
ac ion- alue o each ac ion.
Pa ame e ized models allow a single o wa d pass o es ima e he ac ion- alue unc ion o all ac ions, as
55

CHAPTER 4. REINFORCEMENT LEARNING
in Figu e 23. Impo an ly, o 𝜃∈ℝ𝑘, we desi e 𝑘 |𝑆||𝐴|, so he model gene alizes e ec i ely o
unseen s a es. O he wise, we e e o some hing close o he abula egime.
In his con ex , he TD upda e ule (Equa ion (4.22)) is modi ied o
g adien -based
upda es. The loss
unc ion becomes he mean squa ed e o be ween he TD a ge and he ne wo k p edic ion:
L(𝜃)=𝔼h𝑟+𝛾max
𝑎0𝑄(𝑠0, 𝑎0;𝜃) −𝑄(𝑠, 𝑎;𝜃)2i,(4.23)
whe e he expec a ion is o e ansi ions (𝑠, 𝑎, 𝑟, 𝑠0)sampled om he en i onmen . The g adien o L
wi h espec o 𝜃upda es he ne wo k ia
𝜃←𝜃−𝜂𝔼h𝑟+𝛾max
𝑎0𝑄(𝑠0, 𝑎0;𝜃) −𝑄(𝑠, 𝑎;𝜃)∇𝜃𝑄(𝑠, 𝑎;𝜃)i,(4.24)
whe e he a ge emains cons an o he g adien s ep.
Whe eas abula Q-lea ning upda es a single en y in isola ion, pa ame e ized models p opaga e he upda e
h ough sha ed weigh s, allowing hem o lea n co ela ions be ween s a es and ac ions and disco e
complex, nonlinea ela ionships.
Howe e , con e gence is no gua an eed wi h pa ame e ized models—indeed, he algo i hm can be uns a-
ble o e en di e ge. One sou ce o ins abili y is ha g adien -descen equi es independen and iden ically
dis ibu ed (i.i.d.) da a and s a iona y a ge s, while in RL he da a is non-i.i.d. (as i comes om a chang-
ing policy) and he a ge s a e e e -shi ing ( he es ima es hemsel es imp o e o e ime). One popula
app oach o mi iga e hese issues is
expe ience eplay
combined wi h a
a ge ne wo k
[136]. Expe ience
eplay s o es ansi ions (𝑠, 𝑎, 𝑟, 𝑠0)in a eplay bu e and samples mini-ba ches o upda e he ne wo k,
b eaking co ela ion be ween samples. A a ge ne wo k is a copy o he o iginal ne wo k used o compu e
he a ge max𝑎0𝑄(𝑠0, 𝑎0;𝜃−)wi h pa ame e s 𝜃− ha a e pe iodically synced wi h 𝜃. This s abilizes
aining by educing he non-s a iona i y o he a ge . Pseudocode is p esen ed in Algo i hm 3.
Va ious e inemen s ha e been p oposed o u he s abilize aining o educe dependence on he a ge
ne wo k [14]. None heless, he ul ima e goal o an RL agen is o lea n he op imal policy di ec ly. We now
u n o
policy-based me hods
, which enable he agen o lea n a pa ame e ized policy wi hou needing o
es ima e ac ion alues i s .
56
4.4. POLICY GRADIENT METHODS
Algo i hm 3: Deep Q-Lea ning
Inpu : Beha io policy 𝜋, lea ning a e 𝜂, ho izon 𝑇, en i onmen en .
Ini ialize 𝑄(𝑠, 𝑎;𝜃); ini ialize a ge ne wo k 𝑄(𝑠, 𝑎;𝜃−);
ini ialize eplay bu e D; a ge upda e equency 𝐶.
Ou pu : App oxima ion o op imal ac ion- alue unc ion 𝑄∗(𝑠, 𝑎).
1while
no con e ged
do
2𝑠=𝑠0
3 o 𝑡=0. . .𝑇 −1do
4𝑎∼𝜋(· | 𝑠,𝜃)
5𝑠0,𝑟 =en (𝑠, 𝑎)
6D ← D ∪ (𝑠, 𝑎, 𝑟, 𝑠0)
7B ← sample(D)
8 o (𝑠, 𝑎, 𝑟, 𝑠0) ∈ B do // Upda e ne wo k
9𝜃←𝜃−𝜂𝔼Bh𝑟+𝛾max𝑎0𝑄(𝑠0, 𝑎0;𝜃−) −𝑄(𝑠, 𝑎;𝜃)∇𝜃𝑄(𝑠, 𝑎;𝜃)i
10 i 𝑡mod 𝐶=0 hen
11 𝜃−←𝜃// Upda e a ge ne wo k
12 𝑠←𝑠0
4.4 Policy g adien me hods
Le us conside he
long co ido en i onmen
in Figu e 24, wi h se e al s a es in a long chain. The agen
s a s an episode in a andomly chosen s a e along he chain and aims o each one o he goal s a es lo-
ca ed a he ends o he chain as quickly as possible. Each ime s ep spen in a non-goal s a e has a ewa d
o −1, while he goal s a es ha e ze o ewa d. The agen has wo ac ions, 𝐴={mo e igh ,mo e le }.
Figu e 24: Long co ido en i onmen . The agen s a s in one o he middle s a es. The op imal policy
in he exac middle s a e is s ochas ic, wi h a 50/50 chance o going le o igh , while s a es o he
immedia e le / igh o he middle ha e de e minis ic p e e ed di ec ions.
Because he chain is symme ic, he op imal policy in he middle s a e o he chain is s ochas ic. Indeed,
mo ing le o igh yields he same ewa d. Also obse e ha he en i onmen ’s s uc u e makes i
ques ionable whe he a alue-based app oach is e icien , since o he middle s a e, one simply needs o
pick andomly. Mo e impo an , many pa ially obse ed en i onmen s na u ally bene i om a s ochas ic
policy.
Finally, some en i onmen s ha e con inuous ac ion spaces. Fo example, a con inuous e sion o Ca -
pole allows a con inuum o o ques o be applied o he ca . Es ima ing alues o all possible ac ions is
daun ing o impossible. E en o la ge bu disc e e ac ion se s, he necessa y max𝑎0ope a ion can scale
57
CHAPTER 4. REINFORCEMENT LEARNING
poo ly. Di ec ly lea ning a policy ha emo es he maximiza ion s ep migh be p e e able.
Policy-based me hods
op imize a pa ame e ized policy di ec ly. Among hem,
policy g adien s
[195] a e
widely used. He e, he policy 𝜋(𝑎|𝑠, 𝜃)is di e en iable in 𝜃, and we use
g adien -based
op imiza-
ion o lea n 𝜃∗, maximizing expec ed e u n and yielding he op imal policy. Fo example, conside a
pa ame e ized so max policy like in Subsec ion 4.1. A abula o m migh be:
𝜋(𝑎|𝑠, 𝜃)=exp𝜃𝑠,𝑎
Í𝑎0exp𝜃𝑠,𝑎0,(4.25)
whe e 𝜃∈ℝ|𝑆||𝐴|. Howe e , his is no e y exp essi e. Mo e commonly, one uses neu al ne wo ks, e.g.,
𝜋(𝑎|𝑠, 𝜃)=expℎ(𝑠, 𝑎, 𝜃)
Í𝑎0expℎ(𝑠, 𝑎0,𝜃),(4.26)
whe e ℎ(𝑠, 𝑎, 𝜃)is he ne wo k’s “p e e ence” o (𝑠, 𝑎). No e ha 𝜋is di e en iable. Fo de e minis ic
policies, he g adien can anish. Hence, a su icien ly exp essi e pa ame e iza ion ha yields a well-
de ined g adien o all ac ions is p e e ed.
He e, we shi he objec i e om alue-based me hods o maximizing he expec ed e u n di ec ly wi h
espec o 𝜃. This is
on-policy
, as he agen ’s policy ully de e mines he da a i collec s. To compu e
∇𝜃𝐽(𝜃)o he policy’s pe o mance, we use he
policy g adien heo em
[194]. Fo a simpli ied de i a ion
[57], le 𝑝𝜃(𝜏)be he ajec o y p obabili y unde 𝜋(· | 𝑠, 𝜃):
𝑝𝜃(𝜏)=
𝑇−1
Ö
𝑡=0
𝑝(𝑠𝑡+1|𝑠𝑡, 𝑎𝑡)𝜋(𝑎𝑡|𝑠𝑡, 𝜃).(4.27)
Then he expec ed e u n is
𝐽(𝜃)=Õ
𝜏
𝑝𝜃(𝜏)𝐺(𝜏),(4.28)
whe e 𝐺(𝜏)is he ajec o y’s e u n (cumula i e discoun ed ewa d). I s g adien is:
58
4.4. POLICY GRADIENT METHODS
∇𝜃𝐽(𝜃)=Õ
𝜏∇𝜃𝑝𝜃(𝜏)𝐺(𝜏)
=Õ
𝜏
𝑝𝜃(𝜏)∇𝜃log 𝑝𝜃(𝜏)𝐺(𝜏),(4.29)
using he
log-likelihood ick
∇𝜃𝑝𝜃(𝜏)=𝑝𝜃(𝜏)∇𝜃log 𝑝𝜃(𝜏). Nex ,
∇𝜃log 𝑝𝜃(𝜏)=∇𝜃
𝑇−1
Õ
𝑡=0
log 𝑝(𝑠𝑡+1|𝑠𝑡, 𝑎𝑡)𝜋(𝑎𝑡|𝑠𝑡, 𝜃)
=
𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑡|𝑠𝑡, 𝜃),(4.30)
since he ansi ion p obabili ies do no depend on 𝜃. Subs i u ing in o Equa ion (4.29), we ge :
∇𝜃𝐽(𝜃)=Õ
𝜏
𝑝𝜃(𝜏)
𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑡|𝑠𝑡, 𝜃)𝐺(𝜏)=𝔼𝜏∼𝑝𝜃(𝜏)h𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑡|𝑠𝑡, 𝜃)𝐺(𝜏)i.
(4.31)
Hence, he g adien o he expec ed e u n is he expec ed alue (unde he policy) o he log-policy g adien
mul iplied by e u n. This allows an
empi ical
g adien es ima e om sampled ajec o ies:
∇𝜃𝐽(𝜃) ≈ 1
𝑁
𝑁
Õ
𝑖=1
𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑖
𝑡|𝑠𝑖
𝑡,𝜃)𝐺(𝜏𝑖),(4.32)
whe e (𝑎𝑖
𝑡,𝑠𝑖
𝑡)is he ac ion and s a e in ajec o y 𝑖. Equa ion (4.32) is he ounda ion o he
REINFORCE
algo i hm [207]. G adien ascen on 𝜃 hen ollows:
𝜃←𝜃+𝜂∇𝜃𝐽(𝜃),(4.33)
whe e 𝜂is he lea ning a e. In p ac ice, a iance is o en educed by sub ac ing om 𝐺(𝜏𝑖)a s a e-
dependen
baseline
𝑏(𝑠𝑡):
∇𝜃𝐽(𝜃) ≈ 1
𝑁
𝑁
Õ
𝑖=1
𝑇−1
Õ
𝑡=0∇𝜃log 𝜋𝑎𝑖
𝑡|𝑠𝑖
𝑡,𝜃𝐺(𝜏𝑖) −𝑏(𝑠𝑖
𝑡).(4.34)
59
CHAPTER 4. REINFORCEMENT LEARNING
The baseline is o en an es ima e o he s a e- alue unc ion o he a e age discoun ed ewa d o ha s a e.
Algo i hm 4 summa izes REINFORCE.
Algo i hm 4: REINFORCE
Inpu : Policy 𝜋(· | 𝑠, 𝜃), lea ning a e 𝜂, numbe o ajec o ies 𝑁, ho izon 𝑇, en i onmen
en . Ini ialize policy pa ame e s 𝜃.
Ou pu : App oxima ion o he op imal policy 𝜋∗.
1while
no con e ged
do
2 o 𝑖=1. . . 𝑁 do
3𝑠=𝑠0
4 o 𝑡=0. . .𝑇 −1do
5𝑎∼𝜋(· | 𝑠,𝜃)
6𝑠0,𝑟 =en (𝑠, 𝑎)
7𝜏𝑖←𝜏𝑖∪ (𝑠, 𝑎, 𝑟, 𝑠0)
8𝑠←𝑠0
9 o 𝑖=1. . . 𝑁 do
10 o 𝑡=0. . .𝑇 −1do
11 𝜃←𝜃+𝜂1
𝑁Í𝑁
𝑖=1Í𝑇−1
𝑡=0∇𝜃log 𝜋𝑎𝑖
𝑡|𝑠𝑖
𝑡,𝜃𝐺(𝜏𝑖) −𝑏(𝑠𝑖
𝑡)
REINFORCE is a ounda ional algo i hm in RL and can be implemen ed in only a ew lines. Howe e , i is
sample-ine icien , equi ing many ajec o ies o o m a low- a iance g adien es ima e. Mo eo e , pe o -
mance is sensi i e o he baseline choice. O en he baseline is lea ned wi h a neu al ne wo k, gi ing ise
o
Ac o -C i ic
me hods [107], which combine policy g adien wi h a lea ned alue unc ion. Ac o -C i ic
me hods unde lie many o oday’s bes -pe o ming RL algo i hms, such as
P oximal Policy Op imiza ion
(PPO) [173], widely used in indus y (e.g., aining o GPT [36]).
Recall he e a e many ways o pa ame e ize 𝜋. In classical RL, a pa ame e ized so max policy is mos
common. I s g adien exp ession can be expanded:
∇𝜃log 𝜋(𝑎|𝑠,𝜃)=∇𝜃log expℎ(𝑠, 𝑎, 𝜃)
Í𝑎0expℎ(𝑠, 𝑎0,𝜃)
=∇𝜃ℎ(𝑠, 𝑎, 𝜃) − Õ
𝑎0
𝜋(𝑎0|𝑠, 𝜃)∇𝜃ℎ(𝑠, 𝑎0,𝜃),(4.35)
i.e., he g adien is he g adien o he ac ion’s p e e ence minus he a e age g adien weigh ed by 𝜋.
No ice ha one could inco po a e an “in e se empe a u e” o une explo a ion, bu ypically, one allows
he policy o lea n s ochas ic o nea -de e minis ic beha io di ec ly. S ill, neu al ne wo ks migh collapse
o a de e minis ic policy p ema u ely, hampe ing explo a ion.
En opy egula iza ion
[135] is o en added
o he objec i e,
60

4.4. POLICY GRADIENT METHODS
𝐻𝜃(𝜋)=−Õ
𝑎
𝜋(𝑎|𝑠, 𝜃)log 𝜋(𝑎|𝑠, 𝜃),(4.36)
and he policy op imiza ion objec i e becomes
𝐽(𝜃)=𝔼𝜏∼𝑝𝜃(𝜏)h𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑡|𝑠𝑡, 𝜃)𝐺(𝜏) −𝑏(𝑠𝑡)+𝛽 𝐻𝜃(𝜋)i,(4.37)
whe e 𝛽is an en opy coe icien encou aging s ochas ici y. This can yield as e aining and imp o ed
s abili y [8].
4.4.1 Na u al Policy G adien s and T us Regions
Se e al s a egies imp o e policy op imiza ion con e gence. One heo e ically g ounded app oach is o
use
na u al g adien s
[9] wi hin he policy op imiza ion amewo k, leading o
Na u al Policy G adien s
(NPG) [100]. The idea is o p econdi ion ∇𝜃𝐽(𝜃)by he (classical) CFIM, cap u ing he sensi i i y o he
dis ibu ion 𝜋 o pa ame e changes. Fo mally,
𝐼(𝜃)=𝔼𝑠∼𝑑𝜋𝜃, 𝑎∼𝜋(·|𝑠,𝜃)h∇𝜃log 𝜋(𝑎|𝑠, 𝜃)∇𝜃log 𝜋(𝑎|𝑠, 𝜃)𝑇i,(4.38)
whe e 𝑑𝜋𝜃is he s a e- isi a ion dis ibu ion unde 𝜋𝜃. Then ∇𝜃𝐽(𝜃)can be adap ed o
𝜃←𝜃+𝜂 𝐼−1(𝜃)∇𝜃𝐽(𝜃),(4.39)
which is he
na u al policy g adien
. While ∇𝜃𝐽(𝜃)is in Euclidean space, na u al g adien s measu e
changes in he “in o ma ion geome y” o he pa ame e space, o en imp o ing con e gence [100]. How-
e e , compu ing and in e ing 𝐼(𝜃)a each s ep is expensi e when 𝜃is high-dimensional.
Hence, a ious app oxima ions and heu is ics ha e been de eloped. One concep is o ensu e he new
policy emains close (in KL-di e gence) o he old one, o ming a
us egion
[174]. In s anda d policy
g adien , a single g adien s ep can d as ically al e 𝜋. A
T us Region Policy Op imiza ion
(TRPO) app oach
es ic s he change in 𝜋by bounding he KL dis ance om he old policy:
max
𝜃𝐽(𝜃)subjec o 𝐷𝐾𝐿𝜋𝜃old 𝜋𝜃≤𝛿. (4.40)
61
CHAPTER 4. REINFORCEMENT LEARNING
App oxima ing he KL cons ain quad a ically yields an analy ic solu ion akin o NPG, wi h an adap i e
s ep size dependen on 𝛿. The policy upda e unde NPG can be summa ized as
𝜃←𝜃+s2𝛿
∇𝜃𝐽𝑇
𝜃old 𝐼(𝜃)∇𝜃𝐽𝜃old
𝐼−1(𝜃)∇𝜃𝐽(𝜃).(4.41)
This gua an ees he policy imp o emen is mono onic. Algo i hm 5 shows a high-le el ou line o NPG.
Algo i hm 5: Na u al Policy G adien
Inpu : Policy 𝜋, policy di e gence 𝛿, lea ning a e 𝜂, numbe o ajec o ies 𝑁, ho izon 𝑇,
en i onmen en .
Ini ialize policy pa ame e s 𝜃.
Ou pu : App oxima ion o he op imal policy 𝜋∗.
1while
no con e ged
do
2 o 𝑖=1. . . 𝑁 do
3𝑠=𝑠0
4 o 𝑡=0. . .𝑇 −1do
5𝑎∼𝜋(· | 𝑠,𝜃)
6𝑠0,𝑟 =en (𝑠, 𝑎)
7𝜏𝑖←𝜏𝑖∪ (𝑠, 𝑎, 𝑟, 𝑠0)
8𝑠←𝑠0
// Policy upda e wi h NPG s ep
9 o 𝑖=1. . . 𝑁 do
10 o 𝑡=0. . .𝑇 −1do
11 𝜃←𝜃+ 2𝛿
∇𝜃𝐽𝑇
𝜃old 𝐼(𝜃)∇𝜃𝐽𝜃old
𝐼−1(𝜃)∇𝜃𝐽(𝜃)
In p ac ice, he TRPO op imiza ion p oblem can be ackled ia an uncons ained e sion wi h a KL penal y:
𝜃𝑡+1=2 max
𝜃h𝐽(𝜃) − 𝛽 𝐷𝐾𝐿𝜋𝜃old 𝜋𝜃i,(4.42)
whe e 𝛽is a KL penal y ac o . Howe e , picking 𝛽is non i ial.
P oximal Policy Op imiza ion
(PPO) [173]
ackles his by de ining a clipped su oga e objec i e ha main ains a mono onic imp o emen gua an ee.
PPO is he backbone o many s a e-o - he-a RL agen s [139, 173], hough i is ou o scope he e.
62
4.5. EVALUATION AND PERFORMANCE OF REINFORCEMENT LEARNING AGENTS
4.5 E alua ion and pe o mance o Rein o cemen Lea ning
agen s
Se e al s a egies a e used o assess and compa e RL agen s. In p ac ice, he mos common me ic is
he
a e age ewa d
unde he agen ’s policy. Thus, we o en empi ically e alua e pe o mance by plo ing
mo ing a e ages o cumula i e ewa ds o e episodes o ime s eps, as shown in Figu e 25.
Figu e 25: Pe o mance o a ious RL agen s using di e en policy op imiza ion algo i hms, plo ed agains
he numbe o policy i e a ions. Image om
T us Region Policy Op imiza ion
by Schulman e al. [174].
In ini e-ho izon en i onmen s, he e is o en a clea “sol ed” c i e ion. Fo ins ance, in he Ca pole
en i onmen (Subsec ion 4.2), he agen ecei es +1 pe ime s ep, wi h a maximum o 200 (o 500).
The en i onmen is conside ed sol ed i he agen achie es he maximum ewa d o 100 consecu i e
episodes. Hence, a mo ing a e age o ewa ds can e eal how quickly each agen con e ges. In mo e
open-ended asks, we s ill use such plo s o see which agen ob ains highe ewa d wi hin a ce ain aining
budge .
F om a heo e ical pe spec i e, pe o mance is usually analyzed ia
sample complexi y
[101], also e e ed
o in RL as he numbe o s a e-ac ion isi s equi ed o achie e a nea -op imal policy, o en measu ed
h ough
eg e
. The eg e is he di e ence be ween he ewa d he agen collec s and he ewa d an
o acle op imal agen would collec :
R𝑇=𝔼h𝑁−1
Õ
𝑛=0𝑉∗(𝑠𝑘
0) − 𝑉𝜋(𝑠𝑘
0)i,(4.43)
63
CHAPTER 4. REINFORCEMENT LEARNING
whe e 𝑇=𝑁𝐻 o ho izon 𝐻, and he expec a ion is o e he en i onmen and he agen ’s sampling.
RL algo i hms can be highly da a-hung y. Unde gene al assump ions, sample e iciency is di icul ; many
algo i hms ha e complexi y exponen ial in he ho izon |𝑆||𝐴|𝐻[101].
NPG (Algo i hm 5) is a co e RL me hod wi h solid con e gence p ope ies. Aga wal e al. [7] show i has
loga i hmic eg e in e ms o he o al numbe o en i onmen ac ions o sui able pa ame e ized policies.
Reg e Lemma
(Lemma 6.2) om [7] will be used la e in he con ex o PQC-based policies, so we es a e
i o comple eness.
Fi s , a concep called
compa ible unc ion app oxima ion
[195] is needed. Le 𝑓𝑤(𝑠, 𝑎)app oxima e he
ad an age unc ion 𝐴(𝑠, 𝑎). Then:
𝜓(𝑠, 𝑎)=∇𝜃log 𝜋(𝑎|𝑠, 𝜃), 𝑓𝑤(𝑠, 𝑎)=𝑤𝑇𝜓(𝑠, 𝑎).(4.44)
𝑓𝑤is said o be “compa ible” wi h 𝜋because he co esponding policy g adien is s ill exac [195]. Tha
is, he linea model 𝑓𝑤uses ∇𝜃log 𝜋(𝑎|𝑠,𝜃)as ea u es, which can be bene icial in ac o -c i ic se ups.
Also, le 𝑤∗minimize he squa ed e o
𝑤∗=2 min
𝑤
𝔼𝑠∼˜
𝑑, 𝑎∼e𝜋(·|𝑠)𝐴(𝑠, 𝑎) −𝑤𝑇∇𝜃log 𝜋(𝑎|𝑠,𝜃)2,(4.45)
whe e ˜
𝑑and e𝜋a e e e ence s a e dis ibu ion and policy. Kakade [100] showed he op imum sa is ies
𝑤∗=𝐹−1∇𝜃log 𝜋(𝑎|𝑠,𝜃),(4.46)
whe e 𝐹is he Fishe in o ma ion ma ix. The
Reg e Lemma
4.5.1 le e ages his.
Lemma 4.5.1 (NPG Reg e Lemma [7]).
Fix a compa ison policy
˜𝜋
and a s a e dis ibu ion
𝜌
. Assume
o all
𝑠∈ S
and
𝑎∈ A
ha
log 𝜋(𝑎|𝑠, 𝜃)
is
𝛽
-smoo h in
𝜃
. Conside
𝜋(0)
as he uni o m dis ibu ion
o e ac ions a each s a e, and le
𝑤(0), . . . ,𝑤(𝑇)
be he sequence o weigh s wi h
k𝑤(𝑡)k2≤𝑊
. De ine
he app oxima ion e o a ime
𝑡
:
𝜖𝑡=𝔼𝑠∼˜
𝑑, 𝑎∼e𝜋(·|𝑠)𝐴(𝑡)(𝑠, 𝑎) −𝑤(𝑡)·∇𝜃log 𝜋(𝑡)(𝑎|𝑠).
64
5.1. PARAMETERIZED QUANTUM POLICIES
whe e models wi h da a euploading a e con e ed o
𝑈(𝑠, 𝜃) eup =𝑊(𝜃0)
𝐿
Ö
𝑙=1
𝑆(𝑠)𝑊(𝜃𝑙)(5.8)
He e, 𝜃𝑙∈ℝ𝑘is he ec o o pa ame e s wi hin laye 𝑙.𝑊(𝜃𝑙)is usually decomposed in o a sequence o
single-qubi and wo-qubi pa ame e ized ga es o con ol exp essi i y and educe he numbe o ainable
pa ame e s. These models allow us o conside
ha dwa e-e icien ansä ze
(HEA) ha a e sui able o
nea - e m VQAs due o hei low-dep h s uc u e, esul ing in lowe -noise ci cui s. Addi ionally, 𝑤(𝜃𝑙)is
usually ollowed by a se ies o unpa ame e ized ga es (CNOT/CZ ga es) ac ing on neighbo ing qubi s o
include en anglemen in o he sys em. The neighbo ing condi ion is usually conside ed o accommoda e
qubi connec i i y wi hin he ha dwa e, hus sa ing hea ie swap ope a ions, as illus a ed in Figu e 28.
Figu e 28: Long ange CNOT ga e decomposed wi h swap ga es in a de ice suppo ing nea es neighbo
connec i i y.
The neighbo ing condi ion can be li ed, once conside ing all- o-all qubi connec i i y o accommoda e
mo e complex en anglemen pa e ns. Addi ionally, he en anglemen i sel should also be uned o he
p oblem a hand by conside ing pa ame e ized ga es ins ead o ixing he en anglemen pa e n ia un-
pa ame e ized CNOT/CZ ga es. Howe e , in his wo k, we conside single-qubi pa ame e ized ga es o
a oid he bu den o decomposing wo-qubi ga es ha lead o ci cui s wi h inc eased dep h. Mo eo e ,
wi h single-qubi pa ame e ized ga es we allow less exp essi e ci cui s ha a e also easie o ain ia
g adien op imiza ion (See Subsec ion 3.4).
In he con ex o model- ee RL, we do no ha e access o he en i onmen ’s model and do no possess
ea u e enginee ing ools. Tha is, he agen can only see he cu en s a e i is in bu does no know
how he ea u es a e co ela ed. The e o e, one app oach o designing an ansa z is o conside a bi a y
pa ame e ized single-qubi ga es wi h unpa ame e ized ga es explo ing bo h sho - and long- ange co e-
la ions in he inpu da a. As an example, conside h ee laye s o he
S ongly En angling Ci cui
[171], as
illus a ed in Figu e 29.
71

CHAPTER 5. QUANTUM POLICY GRADIENTS
Figu e 29: S ongly En angling Ci cui p oposed in [171], composed o h ee laye s.
The S ongly En angling Ci cui is composed o a bi a y pa ame e ized single-qubi ga es 𝐺(𝜃𝑙
𝑖)whe e
𝜃𝑙
𝑖∈ℝ3is he pa ame e ec o o he decomposi ion o he ga e ac ing on qubi 𝑖in laye 𝑙. Fo ins ance,
he decomposi ion can be 𝐺(𝜃𝑙
𝑖)=𝑅𝑧(𝜃𝑙
𝑖,0)𝑅𝑦(𝜃𝑙
𝑖,1)𝑅𝑧(𝜃𝑙
𝑖,2). Qubi s a e hen en angled wi h he con ols
o he CNOT ga es ac ing ch onologically on he 𝑁qubi s, 𝑗={0, . . . , 𝑁 −1}, and he a ge qubi
de i ed h ough (𝑖+𝑟)mod 𝑁, whe e 𝑟is he ange o he con ol. I has been shown ha his way,
all qubi s wi h numbe s ha a e a mul iple o gcd(𝑁,𝑟)can be en angled wi h a con ollable numbe
o ga es. Fu he mo e, such an ansa z uses signi ican ly ewe CNOT/CZ ga es (O(𝑁𝐿)) compa ed o a
s anda d all- o-all en anglemen pa e n ha uses 𝑁
𝑘𝐿=𝑁!
𝑘!(𝑁−𝑘)!𝐿ga es and ies o en angle all qubi s
al eady a he i s laye .
Any PQC conside ed in o he o ms o lea ning can, in heo y, also be conside ed in RL. The c ucial aspec
in he design o a PQC-based policy is he measu emen scheme. Recall ha we in end o use he PQC as
a policy gene a o o he agen . Thus, we need o be able o gene a e a classical p obabili y dis ibu ion
o e he ac ion space.
5.1.1 Disc e e ac ion spaces
Le us assume ha he ac ion space is disc e e and composed o |𝐴|ac ions. The simples app oach is
o eso o he
Bo n ule
o quan um mechanics (see Subsec ion ??) and use he Pauli-Z measu emen
o ob ain a p obabili y dis ibu ion o e he compu a ional basis s a es.
Suppose we ha e a PQC 𝜌𝑠𝑡,𝜃 =|𝜓(𝑠𝑡,𝜃)ih𝜓(𝑠𝑡, 𝜃)| encoding he s a e o he agen a ime s ep 𝑡. Le
|𝐴|=2𝑁. In his se ing, he 𝑎 h basis s a e, whe e 𝑎∈ {0,1,2, . . . , 2𝑁−1}, can be associa ed wi h
ac ion 𝑎∈𝐴. The e o e, he policy can be es ima ed di ec ly om he expec a ion alue,
𝜋(𝑎|𝑠𝑡,𝜃)=T 𝜌𝑠𝑡,𝜃𝑂𝑎(5.9)
72
5.1. PARAMETERIZED QUANTUM POLICIES
whe e 𝑂𝑎=|𝑎ih𝑎|is he p ojec o on o he 𝑎 h basis s a e. The policy can hen be used o sample
ac ions 𝑎∼𝜋(·|𝑠𝑡,𝜃)and in e ac wi h he classical en i onmen . No ice, howe e , ha his app oach
wo ks only o he case |𝐴|=2𝑁, which is no always he case. Indeed, he numbe o qubi s p esen in
he ci cui depends no only on he numbe o ac ions bu also on he numbe o ea u es in he encoding
s a e. I we conside single-qubi angle encoding o ea u es, hen he numbe o qubi s 𝑁is equal o he
numbe o ea u es 𝑁=|𝑠𝑡|. Howe e , in his se ing, we ha e h ee dis inc cases depending on he
numbe o ac ions:
1. |𝐴|=2𝑁— The numbe o ac ions is equal o he numbe o basis s a es. In his case, he policy
can be di ec ly es ima ed om he expec a ion alue as in Equa ion (5.9).
2. |𝐴|>2𝑁— The numbe o ac ions is g ea e han he numbe o basis s a es. In his case, he
numbe o qubi s p esen in he sys em is no su icien . The numbe o qubi s should be inc eased,
besides he numbe o ea u es, o accommoda e he numbe o ac ions.
3. |𝐴|<2𝑁— The numbe o ac ions is less han he numbe o basis s a es. In his case, we ha e
o conside a pa i ion o he basis s a es in o |𝐴|g oups.
No ice ha he measu emen i sel does no need o be es ic ed o he compu a ional basis. Indeed, we
can conside any se o eigens a es o a bi a y He mi ian obse ables. The e o e, in he mos gene al
o m, le us deno e he
Bo n policy
, ob ained om he p obabili y o measu ing a pa i ion o he eigens a es
o an obse able, as in De ini ion 5.1.1.
De ini ion 5.1.1. (Bo n policy) Le 𝑠∈ S be a s a e embedded in an 𝑛-qubi pa ame e ized quan um
s a e, 𝜌𝑠,𝜃 =|𝜓(𝑠, 𝜃)ih𝜓(𝑠, 𝜃)|, whe e 𝜃∈ℝ𝑘. The p obabili y associa ed wi h a gi en ac ion 𝑎∈𝐴in
he Bo n amewo k is gi en by:
𝜋(𝑎|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎(5.10)
whe e 𝑃𝑎=Í𝑣∈𝑉𝑎|𝑣ih𝑣|is he p ojec o in o a pa i ion 𝑉𝑎⊆𝑉o |𝑉𝑎|eigens a es o an obse able
𝑂=
2𝑁−1
Õ
𝑖=0
𝜆𝑖|𝑖ih𝑖|.(5.11)
Mo eo e , Ð𝑎∈𝐴𝑉𝑎=𝑉and 𝑉𝑎∩𝑉𝑎0=∅.
De ini ion 5.1.1 accommoda es he mos gene al e sion o he Bo n policy. The main di icul y eso s
o he pa i ion unc ion. Indeed, inding he op imal pa i ioning o basis s a es in o g oups o ac ions is
ex emely challenging. Fu he mo e, in p ac ice, we need s a egies o dis inguish ac ions wi hou eso ing
o he ac ual p obabili y ec o in o de o make he algo i hm e icien , since we would need o s o e a
numbe o elemen s ha inc eases exponen ially wi h he numbe o qubi s. One way is o conside
he bi s ings ha esul om he measu emen o he quan um s a e. Then we need a
pos -p ocessing
73
CHAPTER 5. QUANTUM POLICY GRADIENTS
unc ion
ha maps he bi s ing o he co ec ac ion g oup encoding he pa i ion unc ion. Tha way, he
policy can be es ima ed wi h sho -based lea ning, as illus a ed in Figu e 30.
Figu e 30: Agen -en i onmen in e ace wi h PQC-based policy using sho -based lea ning. The policy is
es ima ed a each ime s ep om he measu emen ou comes using a pos -p ocessing unc ion 𝑓 ha
maps a bi s ing o he ac ion g oup.
Le us de ine he pos -p ocessing unc ion 𝑓:{0,1}𝑁→ {0,1, . . . , |𝐴| −1} ha maps he bi s ing o
he ac ion g oup. Fo a numbe o measu emen s 𝐶, he sho -based Bo n policy is es ima ed as
𝜋(𝑎|𝑠𝑡,𝜃)=1
𝐶
𝐶−1
Õ
𝑐=0
𝛿𝑓(𝑏𝑐)=𝑎(5.12)
whe e 𝑏𝑐is he bi s ing ob ained om he 𝑐 h measu emen . In e es ingly, no e ha , conside ing he
pos -p ocessing unc ion 𝑓, he ac ion a ime-s ep 𝑡can be ob ained wi h a single sho . Howe e , ecall
ha o he policy op imiza ion s ep (i.e., o op imize 𝜃), one needs o es ima e 𝜕𝜃log 𝜋(𝑎|𝑠, 𝜃), which
does equi e knowledge o he ac ual p obabili y ec o 𝜋(𝑎|𝑠, 𝜃). Hence, mo e sho s would gene ally be
needed o es ima e he g adien accu a ely.
Despi e g adien op imiza ion, he e is a mul i ude o pos -p ocessing unc ions, each leading o a di e en
policy. I is c ucial o no e ha he pa i ioning unc ion is ul ima ely linked wi h he amoun o in o ma ion
ex ac ed om he policy, which is in u n linked o he numbe o qubi s we need o measu e. Le us
igno e, o now, he ex eme case o |𝐴|=2𝑁, since in ha scena io he policy is a one- o-one mapping.
Ins ead, ocus on he case |𝐴|<2𝑁. In his se ing, we need cle e assignmen s o he measu ed
bi s ings, so he pos -p ocessing unc ion indeed plays a signi ican ole.
Recall ha in o ma ion heo y p o ides a undamen al way o de e mine he lowe bound on he numbe o
bi s equi ed o encode in o ma ion h ough he concep o en opy, which quan i ies he a e age amoun
o in o ma ion p oduced by a s ochas ic sou ce o da a. Since we need o dis inguish be ween |𝐴|<2𝑁
74
5.1. PARAMETERIZED QUANTUM POLICIES
ac ions, he lowe bound on he numbe o bi s necessa y is log |𝐴|. I is no possible o wo k wi h ewe
bi s. The e o e, he heo e ical minimum leads o log |𝐴|qubi s being measu ed. None heless, he e a e
s ill 𝑁
log |𝐴|possible pa i ions wi h he same amoun o ex ac ed in o ma ion.
As an example, le us conside he RL base case whe e he numbe o ac ions |𝐴|=2. In his case, a
single bi is necessa y o disce n be ween he wo ac ions. Le he numbe o qubi s be 𝑁=3again.
Figu e 31 illus a es h ee al e na i e pa i ion unc ions.
Figu e 31: Th ee possible pa i ion unc ions ha a ain he lowe bound o 1 bi o |𝐴|=2and 𝑁=3,
illus a ed as a uni o m dis ibu ion o e all 23basis s a es. Figu es (a),(b) and (c) ep esen he pa i ion
unc ions ob ained om measu ing qubi s 𝑖,𝑗, and 𝑘 espec i ely, highligh ed in ed in he igu e.
In gene al, o an 𝑁-qubi sys em, a
con iguous-like
pa i ioning o he basis s a es can be gene a ed
ollowed by he measu emen o log |𝐴|
adjacen
qubi s, as de ined in De ini ion 5.1.2.
De ini ion 5.1.2. (Con iguous-like Bo n policy) Le 𝑠∈ S be a s a e embedded in an 𝑁-qubi
pa ame e ized quan um s a e, 𝜌𝑠,𝜃 =|𝜓(𝑠,𝜃)ih𝜓(𝑠, 𝜃)|, whe e 𝜃∈ℝ𝑘. Le w.l.g |𝐴|<2𝑁be he
numbe o ac ions. The con iguous-like Bo n policy is gi en by
𝜋(𝑎|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎(5.13)
whe e 𝑃𝑎=Í𝑣∈𝑉𝑎|𝑣ih𝑣|is he p ojec o on o a pa i ion 𝑉𝑎⊆𝑉o |𝑉𝑎|gene a ed om he measu emen
o log |𝐴|adjacen qubi s—whe e adjacency he e means nume ical adjacency in he bina y ep esen a ion
o he basis s a es.
The Con iguous-like Bo n policy o ms he lowe bound on he globali y o he measu emen ope a o .
Indeed, he e a e o he pa i ions wi h mo e ex ac ed in o ma ion. In heo y, he uppe bound is 𝑁
qubi s. Fo |𝐴|=2ac ions, one could, o ins ance, conside wo o he 2𝑁basis s a es and no malize
hei p obabili ies o o m he policy. Fo ins ance,
75
CHAPTER 5. QUANTUM POLICY GRADIENTS
𝜋(𝑎0|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎0
T 𝜌𝑠,𝜃 𝑃𝑎0+T 𝜌𝑠,𝜃 𝑃𝑎1(5.14)
𝜋(𝑎1|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎1
T 𝜌𝑠,𝜃 𝑃𝑎0+T 𝜌𝑠,𝜃 𝑃𝑎1(5.15)
whe e 𝑃𝑎𝑖=|𝑎𝑖ih𝑎𝑖|is he p ojec o on o he 𝑎 h
𝑖basis s a e, as in De ini ion 5.1.3.
De ini ion 5.1.3. (Ac ion-p ojec o -like Bo n policy) Le 𝑠∈ S be a s a e embedded in an 𝑁-qubi
pa ame e ized quan um s a e, 𝜌𝑠,𝜃 =|𝜓(𝑠, 𝜃)ih𝜓(𝑠, 𝜃)|, whe e 𝜃∈ℝ𝑘. Le |𝐴|<2𝑁be he numbe
o ac ions. The ac ion-p ojec o -like Bo n policy is gi en by
𝜋(𝑎|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎
Í𝑎0∈𝐴T 𝜌𝑠,𝜃 𝑃𝑎0(5.16)
whe e 𝑃𝑎=|𝑎ih𝑎|is he p ojec o on o he 𝑎 h basis s a e.
The ac ion-p ojec o -like Bo n policy is cha ac e ized by an 𝑁-local measu emen . Howe e , he policy
is highly ine icien , pa icula ly when he numbe o qubi s is signi ican ly la ge han he numbe o ac-
ions (𝑁 |𝐴|). In his case, despi e ha ing a global measu emen (which a ains he uppe bound
on he in o ma ion), i is i sel de imen al in e ms o policy op imiza ion. This is because he p obabili y
o measu ing one o he basis s a es is anishing exponen ially wi h he numbe o qubi s, equi ing an
exponen ial numbe o sho s o es ima e he policy ai h ully. In essence, o a la ge numbe o qubi s, we
would likely ne e wi ness he eigens a e o in e es .
We need pa i ion unc ions ha balance he in o ma ion ex ac ed and do no disca d he as majo i y
o he basis s a es. Fo ins ance, one can conside he Hamming dis ance be ween he bi s ing and he
ac ion g oup usually used in c yp og aphic p o ocols. This way, we would be conside ing a la ge numbe
o basis s a es compa ed wi h he ac ion-p ojec o -like policy. An immedia e p oblem in his case would
be a limi a ion o a maximum o |𝐴|=𝑁+1possible ac ions. Conside he case 𝑁=3qubi s once
mo e. In his scena io, we could gene a e he ollowing pa i ions:
𝑉0={000}(5.17)
𝑉1={001,010,100}(5.18)
𝑉2={011,101,110}(5.19)
𝑉3={111}(5.20)
The Hamming dis ance policy is de ined in De ini ion 5.1.4.
76

5.1. PARAMETERIZED QUANTUM POLICIES
De ini ion 5.1.4. (Hamming-like Bo n policy) Le 𝑠∈ S be a s a e embedded in an 𝑁-qubi pa am-
e e ized quan um s a e, 𝜌𝑠,𝜃 =|𝜓(𝑠,𝜃)ih𝜓(𝑠, 𝜃)|, whe e 𝜃∈ℝ𝑘. Le w.l.g |𝐴| ≤ 𝑁+1be he numbe
o ac ions. The Hamming-like Bo n policy is gi en by
𝜋(𝑎|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎(5.21)
whe e 𝑃𝑎=Í𝑣∈𝑉𝑎|𝑣ih𝑣|is he p ojec o on o a pa i ion 𝑉𝑎⊆𝑉o |𝑉𝑎|gene a ed om eigens a es 𝑣
wi h Hamming weigh 𝑎.
Besides he limi a ion on he numbe o ac ions, he Hamming-like Bo n policy gene a es une en dis i-
bu ions, i.e., gi ing di e en p io i ies o di e en basis s a es. Fo ins ance, Hamming weigh 0 would
always conside jus he all-ze o basis s a e. This is no ideal since he policy would be highly biased
owa d s a es wi h la ge Hamming weigh s, making he ac ion 𝑎0less explo ed in he en i onmen while
also being mo e di icul o op imize, as i s p obabili y becomes exponen ially small wi h he numbe o
qubi s.
While he Hamming-like policy p oduces une en dis ibu ions and limi s he size o he ac ion space,
one can s ill le e age he Hamming weigh idea o o m a di e en 𝑁-local policy. Fo he base case
|𝐴|=2, we can conside a pa i y pos -p ocessing unc ion, which is simply a Hamming weigh mod2 o
he bi s ing. Thus, he policy is ep esen ed as:
𝜋(𝑎|𝑠, 𝜃)=⊕𝑏=𝑎
Õ
𝑏∈{0,1}𝑛h𝜓(𝑠, 𝜃)|𝑏ih𝑏|𝜓(𝑠, 𝜃)i (5.22)
whe e 𝑎∈ {0,1}. Such an assignmen cons i u es a global measu emen , and he au ho s o [132]
showed ha i co esponds o he assignmen ha maximizes he ex ac ed in o ma ion. No ice ha
ins ead o he Pauli-Z measu emen on e e y qubi , one could ins ead measu e ei he a single-qubi o an
ancilla, as illus a ed in Figu e 32.
Figu e 32: Decomposi ion o a global measu emen using a single-qubi measu emen .
77
CHAPTER 5. QUANTUM POLICY GRADIENTS
Fo an a bi a y numbe o ac ions |𝐴| ≤ 2𝑁, p o ided ha |𝐴|is a powe o wo, a ecu si e pa i y
unc ion can be applied o he bi s ing o disce n among ac ions, as p oposed by Meye e al. [132]. Le
𝑚=log |𝐴|be he numbe o ecu si e calls and 𝒃be an 𝑛-bi bi s ing measu ed h ough sampling
om he PQC. Then, he pa i ion can be de ined ecu si ely as
C(𝑚)
[𝑎]2=(𝒃|
𝑛−1
Ê
𝑖=𝑚
𝑏𝑖=𝑎0∧𝒃∈ C(𝑚−1)
𝑎𝑚···𝑎2(𝑎1⊕𝑎0))(5.23)
whe e [𝑎]2=𝑎𝑚. . . 𝑎0is he bina y expansion o ac ion 𝑎. Since we equi e each o he 𝑛bi s o
compu ing he pa i y, a pa i y-based policy will be composed o a global measu emen (o 𝑛-local) o
|𝐴|=2as he base case. Thus, i will always be global ega dless o he numbe o ac ions. The
pa i y-like Bo n policy is de ined in De ini ion 5.1.5.
De ini ion 5.1.5. (Pa i y-like Bo n policy) Le 𝑠∈ S be a s a e embedded in an 𝑁-qubi pa ame e -
ized quan um s a e, 𝜌𝑠,𝜃 =|𝜓(𝑠, 𝜃)ih𝜓(𝑠, 𝜃)|, whe e 𝜃∈ℝ𝑘. Le |𝐴| ≤ 2𝑁be he numbe o ac ions
and a powe o wo. Le 𝑚=log |𝐴|be he numbe o ecu si e calls. The Pa i y-like Bo n policy is gi en
by:
𝜋(𝑎|𝑠, 𝜃)=T 𝜌𝑠,𝜃 𝑃𝑎(5.24)
whe e 𝑃𝑎=Í𝑣∈𝑉𝑎|𝑣ih𝑣|is he p ojec o on o a pa i ion 𝑉𝑎⊆𝑉o |𝑉𝑎|gene a ed om eigens a es
espec ing he ecu si e pa i ion
C(𝑚)
[𝑎]2=(𝒃|
𝑛−1
Ê
𝑖=𝑚
𝑏𝑖=𝑎0∧𝒃∈ C(𝑚−1)
𝑎𝑚···𝑎2(𝑎1⊕𝑎0)).(5.25)
A wide ange o pos -p ocessing unc ions can be applied o a Bo n policy. In his wo k, we ocus on he
policy o mula ions discussed abo e, which a e summa ized in Table 1.
Bo n policy Measu emen ope a o Ou pu dis ibu ion
Con iguous-like log |𝐴|-local (adjacen qubi s) E en dis ibu ion bu lowe
bound on in o ma ion.
Ac ion-p ojec o -like 𝑁-local E en dis ibu ion and uppe
bound on in o ma ion, bu
exponen ially ha d o es ima e.
Hamming-like 𝑁-local Uppe bound on in o ma ion,
bu une en dis ibu ion.
Pa i y-like 𝑁-local E en dis ibu ion and uppe
bound on in o ma ion o |𝐴|a
powe o wo.
Table 1: Cha ac e is ics o di e en ypes o Bo n policies.
78
5.1. PARAMETERIZED QUANTUM POLICIES
Each Bo n policy has i s own ad an ages and disad an ages, as summa ized in Table 1. Howe e , e e y
Bo n policy sha es a common limi a ion: none can p ope ly adjus i s g eediness. Recall ha in RL, he
agen needs o balance explo a ion and exploi a ion. S ochas ic policies a e desi ed o hei explo a o y
beha io . Howe e , o he as majo i y o he en i onmen s designed as MDPs, he agen needs a
some poin o con e ge o a de e minis ic op imal policy in which he agen knows he bes s a egy o
exploi he en i onmen and maximize he ewa d. In classical RL, we in oduced he So max policy wi h
a g eediness con ol hype pa ame e (see Subsec ion 4.2). In he quan um se ing, one can apply he
non-linea So max ac i a ion o he ou pu dis ibu ion o any o he Bo n policies o add a con ol o e i s
g eediness. Le h𝑃𝑎i𝑠,𝜃 be he expec a ion alue o he p ojec o 𝑃𝑎 esul ing om any Bo n policy. The
So max policy can be de ined as
𝜋(𝑎|𝑠, 𝜃)=𝑒𝛽h𝑃𝑎i𝑠,𝜃
Í𝑎0∈𝐴𝑒𝛽h𝑃𝑎0i𝑠,𝜃
(5.26)
whe e 𝛽=1
𝜏is he in e se empe a u e pa ame e ha con ols he g eediness o he policy. Ne e he-
less, ecall ha i is an ex emely challenging ask o ind an op imal annealing schedule since his is o en
p oblem-dependen . The e o e, in p ac ice, he g eediness should be con olled o lea ned au oma ically
by he policy’s pa ame e iza ion using expe ience om he en i onmen . Fu he mo e, no ice ha he
so max unc ion no malizes he ec o i ecei es as inpu o o m a p obabili y dis ibu ion. As a conse-
quence, we do no need o conside s ic ly non-nega i e inpu s de i ed om he Bo n ule o quan um
mechanics bu can gene alize i o he expec a ion alue o a bi a y He mi ian ope a o s. Le h𝑂𝑎i𝑠,𝜃 be
he expec a ion alue o a He mi ian obse able 𝑂𝑎 ha encodes he p e e ence o ac ion 𝑎. A PQC-based
So max policy can be de ined as in De ini ion 5.1.6.
De ini ion 5.1.6. (So max policy) Le 𝑠∈ S be a s a e embedded in an 𝑁-qubi pa ame e ized
quan um s a e, 𝜌𝑠,𝜃 =|𝜓(𝑠, 𝜃)ih𝜓(𝑠, 𝜃)|, whe e 𝜃∈ℝ𝑘. Le 𝑂𝑎be an a bi a y He mi ian obse able
and he expec a ion alue
h𝑂𝑎i𝑠,𝜃 =T 𝜌𝑠,𝜃𝑂𝑎(5.27)
ep esen he nume ical p e e ence o ac ion 𝑎∈𝐴. The p obabili y associa ed wi h he ac ion is gi en
by:
𝜋(𝑎|𝑠, 𝜃)=𝑒h𝑂𝑎i𝑠,𝜃
Í𝑎0𝑒h𝑂𝑎0i𝑠,𝜃 .(5.28)
The So max policy allows one o conside O(|𝐴|) di e en obse ables. Thus, i a ies signi ican ly
om any o he Bo n policies de ined p e iously, allowing, in heo y, g ea e exp essi e powe . The e a e,
79
CHAPTER 5. QUANTUM POLICY GRADIENTS
howe e , se e al componen s ha play a ole in he exp essi i y o he PQC-based policy. This is co e ed
in g ea e de ail in Sec ion 5.2.
5.1.2 Con inuous ac ion spaces
Le us now conside con inuous ac ion spaces, i.e., he ac ion space is a subse o ℝ𝑑. Nei he he
Bo n no he So max policies p oposed in Subsec ion 5.1.1 can be di ec ly applied o con inuous ac ion
spaces. One can, howe e , disc e ize he ac ion space and apply he same policies as be o e, bu his
is no p ac ical since he numbe o ac ions would be oo la ge, hus en o cing an exponen ial numbe o
measu emen s o ai h ully es ima e he policy. The e o e, we need o conside a di e en app oach. One
simple s a egy is o use he PQC o lea n he op imal pa ame e s o a
Gaussian dis ibu ion
—namely he
mean
and
a iance
. Gaussian policies p o ide a na u al and lexible way o ep esen con inuous ac ions
because hey can model a wide ange o beha io s h ough he manipula ion o hei pa ame e s (mean
and a iance). The mean (𝜇) shi s he cen e o he dis ibu ion, di ec ing he likely ac ions, while he
a iance (𝜎2) adjus s he explo a ion le el by con olling he dis ibu ion’s sp ead a ound he mean. Figu e
33 illus a es he e ec o changing he mean and a iance o a Gaussian dis ibu ion, p oducing di e en
policies.
Figu e 33: E ec o changing he mean and a iance o a Gaussian dis ibu ion.
The e o e, he p obabili y densi y unc ion o a PQC-based Gaussian policy is gi en by
𝜋(𝑎|𝑠, 𝜃)=1
p2𝜋𝜎 (𝑠, 𝜃)2exp−(𝑎−𝜇(𝑠, 𝜃))2
2𝜎(𝑠, 𝜃)2,(5.29)
whe e he p obabili y is gi en by he in eg al o he p obabili y densi y unc ion o e he ac ion space.
80
5.2. EXPRESSIVITY
Figu e 36: So max policy wi h (a) a single obse able o e e y ac ion wi h one ou pu scaling pa ame e
pe ac ion, and (b) one ou pu scaling pa ame e o e e y ac ion wi h di e en obse ables.
No ice ha he o al numbe o ainable pa ame e s has inc eased signi ican ly. The ou pu scaling alone
can depend on he o al numbe o ac ions. Recall ha unc ion app oxima ion is only wo hwhile p o ided
ha he numbe o pa ame e s does no exceed |𝑆||𝐴|. In in e media e domains— hose no con aining oo
la ge ac ion spaces—i could be easible o conside a single ou pu scaling pa ame e o e e y ac ion. As
an example, conside he Ca pole en i onmen once mo e. In his se ing |𝐴|=2. The e is an e iden
co ela ion be ween he wo ac ions; he e o e, conside ing a single-ou pu scaling is su icien , as ha
pa ame e changes bo h ac ions simul aneously. Howe e , o en i onmen s wi h la ge ac ion spaces,
he beha io is no en i ely clea . Fo ha eason, we need o inspec he g adien beha io o p ope ly
add ess his ques ion. This is done in Sec ion 5.3.
Despi e hese hu dles, ecall ha so a we ha e explo ed he ole o he ou pu scaling jus o he PQC-
based So max policy. Indeed, adap i e g eediness con ol in he o m o ou pu scaling o he Bo n policy
is no possible. This helps us conclude ha he So max policy can be mo e exp essi e and malleable o
a wide ange o en i onmen s. The applicabili y o inpu and ou pu scaling in bo h PQC-based policies
is summa ized in Table 2.
Policy Inpu scaling Ou pu scaling
Bo n 3 7
So max 3𝑊∈ℝo 𝑊∈ℝ|𝐴|
Table 2: Applicabili y o inpu and ou pu scaling in PQC-based policies.
87

CHAPTER 5. QUANTUM POLICY GRADIENTS
5.3 G adien es ima ion
The policy imp o emen s ep in he
policy g adien
o malism is based on g adien -based op imiza ion.
Indeed, g adien ascen is pe o med on he expec ed e u n wi h espec o he policy pa ame e s. Re-
call ha he ounda ional REINFORCE algo i hm (see Algo i hm 4) pe o ms g adien ascen on he log
likelihood weigh ed by he cumula i e ewa d as
𝜃←𝜃+𝜂∇𝜃𝐽(𝜃)whe e ∇𝜃𝐽(𝜃)=1
𝑁
𝑁−1
Õ
𝑖=0
𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑖
𝑡|𝑠𝑖
𝑡,𝜃)𝐺(𝜏𝑖)(5.44)
o 𝑁episodes o ho izon 𝑇, whe e 𝜂is he lea ning a e. The e o e, o use he PQC-based policies
p oposed in Sec ion 5.1, we need o be able o e icien ly es ima e he g adien o he log policy. I
he model i sel is being simula ed on a classical de ice, i can be conside ed like any o he classical
pa ame e ized model, and he g adien can be es ima ed using au oma ic di e en ia ion. Howe e , wi h a
eal quan um de ice, g adien s mus be es ima ed on he de ice i sel , and hus he policy g adien mus
be exp essed in a o m whe e
pa ame e -shi ules
(see Subsec ion 3.3) can be applied. Fo una ely,
pa ame e -shi s can s ill be used h ough he chain ule o calculus.
5.3.1 G adien ecipes
Le us s a wi h Bo n policies (see De ini ion 5.1.1). In gene al, he Bo n policy is ep esen ed wi h a PQC
𝜌𝑠,𝜆,𝜃 =|𝜓(𝑠, 𝜆, 𝜃)ih𝜓(𝑠, 𝜆,𝜃)| and a pa i ion unc ion 𝑃𝑎 o ac ion 𝑎. G adien s a e equi ed o be
es ima ed o inpu scaling pa ame e s 𝜆and weigh s 𝜃. The g adien o he log policy is exp essed by
∇𝜃log 𝜋(𝑎|𝑠,𝜃)=1
𝜋(𝑎|𝑠, 𝜃)∇𝜃𝜋(𝑎|𝑠, 𝜃)
=1
2𝜋(𝑎|𝑠, 𝜃)T 𝜌𝑠,𝜆,𝜃+𝜋
2𝑃𝑎−T 𝜌𝑠,𝜆,𝜃−𝜋
2𝑃𝑎(5.45)
wi h he pa ame e -shi applied o he da a-encoding-independen ga es. Simila ly, o inpu scaling pa-
ame e s,
∇𝜆log 𝜋(𝑎|𝑠, 𝜃)=1
2𝜋(𝑎|𝑠, 𝜃)T 𝜌𝑠,𝜆+𝜋
2,𝜃 𝑃𝑎−T 𝜌𝑠,𝜆−𝜋
2,𝜃 𝑃𝑎.(5.46)
The e o e, he g adien ec o o he log policy o a Bo n policy can be es ima ed using wo mo e expec-
a ion alue es ima ions o each pa ame e . Recall ha he denomina o 𝜋(𝑎|𝑠, 𝜃)is al eady es ima ed
o policy e alua ions and sampling, so i does no con ibu e o he compu a ional cos o g adien es-
ima ion. In essence, he pa ial de i a i e can be es ima ed up o e o 𝜖using O(𝜖−2)sho s. No ice
88
5.3. GRADIENT ESTIMATION
ha 𝜋(𝑎|𝑠, 𝜃)canno be ze o since in he con ex o policy g adien s we a e es ima ing he g adien o
he policy en y ha was selec ed based on sampling. I he ac ion was sampled om he dis ibu ion,
he p obabili y is g ea e han ze o. Howe e , he p obabili y can be a bi a ily close o ze o. The e o e,
we see ha o Bo n policies, he g adien is
unbounded
, which can lead o nume ical ins abili y, as we
discuss in Chap e 6. Le us now u n ou a en ion o he PQC-based So max policy.
So max policies (see De ini ion 5.1.6), in hei gene al o m, can depend on a se o h ee pa ame e s
{𝜃, 𝜆,𝑤}, whe e 𝜃a e he weigh s o he pa ame e ized ga es, 𝜆a e he inpu scaling pa ame e s, and
𝑤is he ou pu scaling pa ame e . Fo simplici y o analysis, le us igno e he in e se empe a u e hy-
pe pa ame e since i is un ainable and p o ides only a cons an scaling ac o o he g adien . Le he
so max policy be ep esen ed as
𝜋(𝑎|𝑠, 𝜃, 𝜆,𝑤)=𝑒𝑤h𝑂𝑎i𝑠,𝜃,𝜆
Í𝑎0𝑒𝑤h𝑂𝑎0i𝑠,𝜃,𝜆 .(5.47)
Then, he g adien o he log policy can be ob ained as a unc ion o he expec a ion alues es ima ed wi h
he quan um de ice, by expanding he log policy ope a o (as we did in Equa ion 4.35):
∇𝜃log 𝜋(𝑎|𝑠,𝜃, 𝜆,𝑤)=∇𝜃log 𝑒𝑤h𝑂𝑎i𝑠,𝜃,𝜆
Í𝑎0𝑒𝑤h𝑂𝑎0i𝑠,𝜃,𝜆
=∇𝜃𝑤h𝑂𝑎i𝑠,𝜃,𝜆 −log Õ
𝑎0
𝑒𝑤h𝑂𝑎0i𝑠,𝜃,𝜆 
=𝑤∇𝜃h𝑂𝑎i𝑠,𝜃,𝜆 −Í𝑎0𝑒𝑤h𝑂𝑎0i𝑠,𝜃,𝜆 ∇𝜃h𝑂𝑎0i𝑠,𝜃,𝜆
Í𝑎0𝑒𝑤h𝑂𝑎0i𝑠,𝜃,𝜆
=𝑤∇𝜃h𝑂𝑎i𝑠,𝜃,𝜆 −Õ
𝑎0
𝑤∇𝜃h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠, 𝜃, 𝜆,𝑤).(5.48)
As opposed o he Bo n policy, he g adien depends on e e y ac ion’s expec a ion alue and on he policy
i sel . Simila ly, o he inpu scaling pa ame e s,
∇𝜆log 𝜋(𝑎|𝑠, 𝜃, 𝜆,𝑤)=𝑤∇𝜆h𝑂𝑎i𝑠,𝜃,𝜆 −Õ
𝑎0
𝑤∇𝜆h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠,𝜃, 𝜆, 𝑤),(5.49)
whe e o pa ame e s {𝜃, 𝜆} he g adien o he expec a ion alue can be es ima ed h ough pa ame e -
shi ules:
89
CHAPTER 5. QUANTUM POLICY GRADIENTS
∇𝜃h𝑂𝑎i𝑠,𝜃,𝜆 =1
2h𝑂𝑎i𝑠,𝜃+𝜋
2,𝜆 − h𝑂𝑎i𝑠,𝜃−𝜋
2,𝜆,
∇𝜆h𝑂𝑎i𝑠,𝜃,𝜆 =1
2h𝑂𝑎i𝑠,𝜃,𝜆+𝜋
2− h𝑂𝑎i𝑠,𝜃,𝜆−𝜋
2.(5.50)
The g adien o he ou pu scaling depends on he numbe o ou pu pa ame e s we ha e. I we conside
a single ou pu scaling pa ame e o all ac ions (𝑤∈ℝ), he g adien is gi en by
∇𝑤log 𝜋(𝑎|𝑠, 𝜃, 𝜆,𝑤)=h𝑂𝑎i𝑠,𝜃,𝜆 −Õ
𝑎0h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠, 𝜃, 𝜆,𝑤).(5.51)
I we conside a dis inc ou pu scaling pa ame e o each ac ion (𝑤∈ℝ|𝐴|), he g adien is
∇𝑤log 𝜋(𝑎|𝑠, 𝜃, 𝜆,𝑤)=h𝑂𝑎i𝑠,𝜃,𝜆 − h𝑂𝑎i𝑠,𝜃,𝜆 𝜋(𝑎|𝑠,𝜃, 𝜆,𝑤).(5.52)
The g adien exp essions in Equa ions (5.51) and (5.52) help cla i y he beha io o he ou pu scaling
pa ame e s. No ice ha i mul iple ou pu scaling pa ame e s a e conside ed, a single pa ame e 𝑤𝑎is
upda ed while only aking in o accoun he expec a ion alue o ha ac ion. Howe e , i a single ou pu scal-
ing pa ame e is conside ed, he g adien wi h espec o ha pa ame e is ob ained om he di e ence
be ween he nume ical p e e ence o ac ion 𝑎and he expec a ion o e all ac ion nume ical p e e ences.
Thus, he la e uses an a e age o e he ac ion space o upda e he pa ame e s, as opposed o he o -
me , which upda es he pa ame e s indi idually. The wo di e en app oaches lead o di e en beha io s
and associa ed complexi ies.
A single pa ame e :
• Simpli ies he model, educing he numbe o pa ame e s ha need o be lea ned. This can be
pa icula ly ad an ageous in en i onmen s whe e da a a e spa se o he lea ning a es need o be
e y ca e ully managed o a oid o e i ing.
• Applies he same le el o explo a ion o exploi a ion ac oss all ac ions. This uni o mi y ensu es ha
no single ac ion is inhe en ly mo e explo a i e o exploi a i e pu ely due o he pa ame e se ing.
• Does no allow o ac ion-speci ic adjus men s in explo a ion endencies. Fo example, i ce ain
ac ions equi e ine con ol o mo e cau ious explo a ion due o hei consequences in he en i on-
men , a single pa ame e canno accommoda e his.
• In complex en i onmen s whe e di e en ac ions ha e as ly di e en scales o ewa ds o u ili ies,
a single scaling ac o migh no be op imal o lea ning he bes policy ac oss all ac ions.
90
5.3. GRADIENT ESTIMATION
Mul iple pa ame e s:
• Each ac ion can ha e i s own scaling ac o , allowing he policy o adap mo e inely o di e en pa s
o he ac ion space. This can be especially use ul in he e ogeneous en i onmen s whe e ac ions
a y signi ican ly in hei e ec s, isks, o ewa ds.
• Di e en ac ions may equi e di e en le els o explo a ion. Fo ins ance, some ac ions migh be
sa e and well-unde s ood and hus can be exploi ed mo e, whe eas o he s migh be isky o less
unde s ood and hus equi e mo e explo a ion.
• Mo e pa ame e s mean a highe isk o o e i ing, especially wi h limi ed da a. I also compli-
ca es he lea ning p ocess, po en ially equi ing mo e sophis ica ed algo i hms o egula iza ion
echniques. Addi ionally, mo e pa ame e s can mean slowe con e gence and highe compu a-
ional cos s. I may also equi e mo e in e ac ions wi h he en i onmen o accu a ely es ima e he
bes alues o each pa ame e , a ec ing sample e iciency.
In gene al, he choice be ween a single o mul iple ou pu scaling pa ame e s depends on he complexi y
o he en i onmen , he na u e o he ac ions, and he a ailable da a.
On a di e en no e, le us now conside he PQC-based Gaussian policy as in De ini ion 5.1.7. Fo com-
ple eness, le wo dis inc PQCs encode he pa ame e ized mean and a iance wi h pa ame e s {𝜃𝜇,𝜃𝜎}.
The Gaussian policy is ep esen ed as
𝜋(𝑎|𝑠, 𝜃𝜇,𝜃𝜎)=1
p2𝜋𝜎 (𝑠, 𝜃𝜎)exp−(𝑎−𝜇(𝑠,𝜃𝜇))2
2𝜎(𝑠,𝜃𝜎)2.(5.53)
The g adien o he log policy can be exp essed as a unc ion o he expec a ion alues o bo h se s o
pa ame e s. Fo he mean and a iance pa ame e s, he g adien ecipe is
∇𝜃𝜇log 𝜋(𝑎|𝑠,𝜃𝜇, 𝜃𝜎)=𝑎−𝜇(𝑠,𝜃𝜇)
𝜎(𝑠, 𝜃𝜎)2∇𝜃𝜇𝜇(𝑠, 𝜃𝜇),
∇𝜃𝜎log 𝜋(𝑎|𝑠,𝜃𝜇, 𝜃𝜎)=(𝑎−𝜇(𝑠,𝜃𝜇))2
𝜎(𝑠, 𝜃𝜎)3−1
𝜎(𝑠, 𝜃𝜎)∇𝜃𝜎𝜎(𝑠, 𝜃𝜎),(5.54)
The g adien ecipes o each policy and hei espec i e pa ame e s a e summa ized in Table 3.
91
CHAPTER 5. QUANTUM POLICY GRADIENTS
Policy Pa ame e G adien ecipe
Bo n 𝜃1
2h𝑂𝑎i𝑠,𝜃+𝜋
2− h𝑂𝑎i𝑠,𝜃 −𝜋
2
Bo n 𝜆1
2h𝑂𝑎i𝑠,𝜆+𝜋
2,𝜃 − h𝑂𝑎i𝑠,𝜆−𝜋
2,𝜃 
So max 𝜃 𝑤∇𝜃h𝑂𝑎i𝑠,𝜃,𝜆 −
Í𝑎0𝑤∇𝜃h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠, 𝜃, 𝜆,𝑤)
So max 𝜆 𝑤∇𝜆h𝑂𝑎i𝑠,𝜃,𝜆 −
Í𝑎0𝑤∇𝜆h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠, 𝜃, 𝜆,𝑤)
So max 𝑤∈ℝh𝑂𝑎i𝑠,𝜃,𝜆 −Í𝑎0h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠, 𝜃, 𝜆, 𝑤)
So max 𝑤∈ℝ|𝐴|h𝑂𝑎i𝑠,𝜃,𝜆 − h𝑂𝑎i𝑠,𝜃,𝜆 𝜋(𝑎|𝑠,𝜃, 𝜆,𝑤)
Gaussian 𝜃𝜇𝑎−𝜇(𝑠,𝜃𝜇)
𝜎(𝑠,𝜃𝜎)2∇𝜃𝜇𝜇(𝑠,𝜃𝜇)
Gaussian 𝜃𝜎(𝑎−𝜇(𝑠,𝜃𝜇))2
𝜎(𝑠,𝜃𝜎)3−1
𝜎(𝑠,𝜃𝜎)∇𝜃𝜎𝜎(𝑠,𝜃𝜎)
Table 3: G adien ecipes o each policy and hei espec i e pa ame e s, including he Gaussian policy.
5.3.2 Sample complexi y
The sample complexi y o he g adien es ima ion p ocedu e is a c ucial aspec o conside in policy
g adien algo i hms. Recall ha he policy g adien is being empi ically es ima ed h ough he
loglikelihood
ick
(see Subsec ion 4.4). The e o e, sample complexi y he e e e s o he numbe o aining examples
equi ed o ha e a ai h ul es ima ion o he policy g adien . The numbe o samples is de ined as he
numbe o isi ed s a es. Since he e a e 𝑁 ajec o ies 𝜏𝑖, each isi ing 𝑇s a es, he o al numbe o
samples is O(𝑁𝑇). We wan a igh e bound on his quan i y. Lemma 5.3.1 es ablishes an uppe
bound on he numbe o samples equi ed o 𝜖-es ima e he policy g adien ˆ
∇𝜃𝐽(𝜃)assuming w.l.g ha
∇𝜃log 𝜋(𝑎|𝑠,𝜃) ≤ G. Le us analyze he lemma and only hen speci y he ype o policy and he
implica ions o g adien es ima ion. The mos ele an insigh om he lemma is ha i cla i ies ha
he numbe o samples equi ed o es ima e he g adien g ows only loga i hmically wi h he numbe o
ainable pa ame e s, which is a o able o he scalabili y o he algo i hm.
Lemma 5.3.1.
Le
𝜃∈ℝ𝑘
and
∇𝜃𝐽(𝜃)
be he expec ed policy g adien empi ically es ima ed h ough
𝑁
ajec o ies o ho izon
𝑇
wi h
𝑅max
being he maximum possible ewa d in any ime s ep. Le
𝛾∈ [0,1]
be he discoun ac o . Assume
∇𝜃log 𝜋(𝑎|𝑠,𝜃) ≤ G
. An
𝜖
-app oxima ion o he policy g adien
ˆ
∇𝜃𝐽(𝜃)
,
|ˆ
∇𝜃𝐽(𝜃) −∇𝜃𝐽(𝜃)| ≤ 𝜖(5.55)
can be ob ained wi h p obabili y
1−𝛿
, using a numbe o samples gi en by
𝑁𝑇 ≈ OG2𝑅2
max𝑇3
𝜖2(𝛾−1)4log2𝑘
𝛿.(5.56)
92

5.3. GRADIENT ESTIMATION
P oo .
Recall ha he policy g adien o 𝑁 ajec o ies wi h ho izon 𝑇is
∇𝜃𝐽(𝜃)=1
𝑁
𝑁−1
Õ
𝑖=0
𝑇−1
Õ
𝑡=0∇𝜃log 𝜋(𝑎𝑖
𝑡|𝑠𝑖
𝑡,𝜃)𝐺(𝜏𝑖),
whe e we eplace he s a ic e u n 𝐺(𝜏𝑖)by a e u n pe ime s ep 𝐺𝑡(𝜏𝑖) o dis inguish e e y ac ion. Le
us s a by de ining a i ial uppe bound on he e u n pe ajec o y, conside ing a maximum ewa d pe
ime s ep 𝑅max:
𝐺(𝜏)=
𝑇−1
Õ
𝑡=0
𝛾𝑡𝑟𝑡+1≤𝑅max
𝑇−1
Õ
𝑡=0
𝛾𝑡=𝑅max
𝛾𝑇−1
𝛾−1.(5.57)
Using he exp ession o he sum o 𝑇 e ms o a geome ic p og ession, we ge :
𝑇−1
Õ
𝑡=0
𝐺𝑡(𝜏) ≤ 𝑅max
𝑇−1
Õ
𝑡=0
𝛾𝑇−𝑡−1
(𝛾−1)≤𝑅max
𝑇
(𝛾−1)2.(5.58)
Assuming ha ∇𝜃log 𝜋(𝑎|𝑠, 𝜃) ≤ G o all 𝑎and 𝑠, and conside ing he abo e bound on he e u n, we
can bound each s ep 𝑡≤𝑇o he policy g adien as ollows:
∇𝜃log 𝜋(𝑎|𝑠,𝜃)𝐺𝑡(𝜏𝑖) ≤ G𝑅max
𝑇
(𝛾−1)2.(5.59)
Le us assume ha 𝑋𝑛=Í𝑇−1
𝑡=0∇𝜃log 𝜋(𝑎|𝑠,𝜃)𝐺𝑡(𝜏𝑖)is he sum o 𝑇bounded andom a iables
𝑋𝑛∈ [0,G𝑅max 𝑇
(𝛾−1)2]. Then, Hoe ding’s inequali y can be used o bound he p obabili y ha he sum
o he policy g adien is 𝜖-inaccu a e:
ℙh1
𝑁
𝑁
Õ
𝑖=1𝑋𝑖−𝔼[𝑋𝑖]≥𝜖i≤2 exp−2𝑁𝜖2
(𝑏−𝑎)2,(5.60)
whe e 𝑋𝑖∈ [𝑎,𝑏]. Replacing he a iables,
ℙh|∇∗
𝜃𝐽(𝜃) −∇𝜃𝐽(𝜃)| ≥ 𝜖i≤2 exp−2𝑁𝜖2(𝛾−1)4
G2𝑅2
max𝑇2.(5.61)
Using he union bound o all 𝜃∈ℝ𝑘,
93
CHAPTER 5. QUANTUM POLICY GRADIENTS
ℙhØ
𝑘
2 exp−2𝑁𝜖2(𝛾−1)4
G2𝑅2
max𝑇2i ≤2𝑘exp−2𝑁𝜖2(𝛾−1)4
G2𝑅2
max𝑇2.(5.62)
Le 𝛿=ℙ[|∇∗
𝜃𝐽(𝜃) −∇𝜃𝐽(𝜃)| ≥ 𝜖]. Then,
1−𝛿=ℙh|∇∗
𝜃𝐽(𝜃) −∇𝜃𝐽(𝜃)| ≤ 𝜖i
≥1−2𝑘exp−2𝑁𝜖2(𝛾−1)4
G2𝑅2
max𝑇2,
𝛿≤2𝑘exp−2𝑁𝜖2(𝛾−1)4
G2𝑅2
max𝑇2.
(5.63)
Thus, an uppe bound on 𝑁is
𝑁≤G2𝑅2
max𝑇2
𝜖2(𝛾−1)4log2𝑘
𝛿.(5.64)
Conside ing 𝑁𝑇 samples comple es he p oo .
□
The lemma p o ides an uppe bound on he numbe o samples equi ed o es ima e he policy g adien .
The bound g ows loga i hmically wi h he numbe o ainable pa ame e s. Howe e , we need o cla i y he
bound on he log policy g adien o speci ic PQC-based policies. Le us s a wi h he PQC-based So max
policy. The g adien wi h espec o 𝜃 o he log policy is
∇𝜃log 𝜋(𝑎|𝑠,𝜃, 𝜆,𝑤)=𝑤∇𝜃h𝑂𝑎i𝑠,𝜃,𝜆 −Õ
𝑎0
𝑤∇𝜃h𝑂𝑎0i𝑠,𝜃,𝜆 𝜋(𝑎0|𝑠, 𝜃, 𝜆,𝑤).(5.65)
The e o e, wi hou loss o gene ali y, we can assume ha he obse able whose expec a ion alue ep e-
sen s he nume ical p e e ence o ac ion 𝑎is a sum o 𝑀 e ms:
𝑂𝑎=
𝑀−1
Õ
𝑚=0
𝑐𝑖𝑃𝑖,(5.66)
whe e 𝑃𝑖∈ {𝕀, 𝜎𝑥, 𝜎𝑦, 𝜎𝑧}⊗𝑁is a Pauli s ing ac ing on he 𝑁qubi s and 𝑐𝑖∈ℝi s eal coe icien . Le
𝑐𝑖∈ [−𝐶,𝐶] o some 𝐶∈ℝ. Then he expec a ion alue o he obse able is bounded as h𝑂𝑎i𝑠,𝜃,𝜆 ∈
[−𝐶𝑀,𝐶𝑀]. The e o e, he g adien o he log policy, using pa ame e -shi ules, is bounded as
94
5.3. GRADIENT ESTIMATION
∇𝜃log 𝜋(𝑎|𝑠,𝜃, 𝜆,𝑤) ∈ [−2𝑤𝐶𝑀, 2𝑤𝐶𝑀].(5.67)
Hence,
𝑁𝑇 ≈ O4𝑤2𝐶2𝑀2𝑅2
max𝑇3
𝜖2(𝛾−1)4log2𝑘
𝛿.(5.68)
The bound on he numbe o samples equi ed o es ima e he policy g adien o he PQC-based So max
policy depends hea ily on he ou pu scaling pa ame e 𝑤, he numbe o e ms in he obse able 𝑀,
and hei espec i e eal coe icien s 𝐶. None heless, he g adien exp ession is s ill bounded, and hus
he numbe o samples can be inc eased a bi a ily o ensu e a ai h ul es ima ion o he policy g adien .
Le us now conside he PQC-based Bo n policy.
Recall ha o an a bi a y Bo n policy (see De ini ion 5.1.1) wi h a pa i ion unc ion 𝑃𝑎associa ed o
ac ion 𝑎, he g adien o he log policy wi h espec o 𝜃is
∇𝜃log 𝜋(𝑎|𝑠,𝜃)=1
2𝜋(𝑎|𝑠, 𝜃)T 𝜌𝑠,𝜆,𝜃+𝜋
2𝑃𝑎−T 𝜌𝑠,𝜆,𝜃−𝜋
2𝑃𝑎.(5.69)
Since we main ain he pa i ion unc ion o he shi ing, i p oduces a new p obabili y dis ibu ion. The e-
o e, he shi ing ope a ion emains bounded, as T 𝜌𝑠,𝜆,𝜃±𝜋
2𝑃𝑎∈ [0,1] o all 𝑎. The denomina o i sel
is also bounded 𝜋(𝑎|𝑠, 𝜃) ∈ [𝑏, 1]. I is no bounded in he ull ange [0,1]because we a e es ima ing
he g adien o he selec ed ac ion a a gi en ime s ep. The e o e, he p obabili y i sel canno be s ic ly
ze o in he g adien es ima ion phase. Howe e , i can become a bi a ily close o ze o. Indeed, he
p obabili y o selec ing he ac ion may dec ease exponen ially wi h he numbe o qubi s, which makes he
g adien i sel exponen ially la ge. Thus, he log policy g adien is bounded abo e by
∇𝜃log 𝜋(𝑎|𝑠,𝜃) ∈ h−1
2,1
2𝑏i,(5.70)
indica ing ha he numbe o samples equi ed o ai h ully es ima e he policy g adien o he PQC-
based Bo n policy can inc ease exponen ially wi h he numbe o qubi s. This is a clea indica ion o
he ainabili y issues ha can a ise when using he Bo n policy. These aining ins abili ies a e u he
discussed in Chap e 6. The sample complexi y o he PQC-based Bo n policy is
𝑁𝑇 ≈ O𝑅2
max𝑇3
𝑏2𝜖2(𝛾−1)4log2𝑘
𝛿.(5.71)
95
CHAPTER 5. QUANTUM POLICY GRADIENTS
The policy g adien anging condi ions and espec i e g adien es ima ion sample complexi y o PQC-
based policies a e summa ized in Table 4.
Policy Policy g adien ange Sample complexi y
Bo n ∇𝜃log 𝜋(𝑎|𝑠, 𝜃) ∈ −1
2,1
2𝑏O𝑅2
max𝑇3
𝑏2𝜖2(𝛾−1)4log2𝑘
𝛿
So max ∇𝜃log 𝜋(𝑎|𝑠, 𝜃, 𝜆,𝑤) ∈
[−2𝑤𝐶𝑀, 2𝑤𝐶𝑀]O4𝑤2𝐶2𝑀2𝑅2
max𝑇3
𝜖2(𝛾−1)4log2𝑘
𝛿
Table 4: Policy g adien anges and espec i e g adien es ima ion sample complexi y o PQC-based
policies.
5.4 Nume ical expe imen s
This sec ion del es in o he p ac ical applica ion and empi ical analysis o a ious quan um policy ne wo ks
p oposed in Sec ion 5.1, h ough a se ies o de ailed nume ical expe imen s. These expe imen s p o ide
insigh s no only in o he ope a ional dynamics o hese policies bu also in o empi ical pe o mance on
s anda d RL benchma k en i onmen s, compa ed wi h classical pa ame e ized models ypically used o
sol e hese asks. Subsec ion 5.4.1 s a s wi h a simple explo a ion o a basic so max policy amewo k
wi hou conside ing da a euploading. This model, e alua ed in ou p elimina y esea ch a icle [175], se s
a ounda ional baseline model o subsequen expe imen al inqui ies. Subsec ion 5.4.2 hen in es iga es
he e ec o da a euploading on he pe o mance o he quan um policy ne wo ks. The expe imen s a e
conduc ed on bo h Bo n and So max policies p oposed in Sec ion 5.1.
5.4.1 A single- equency so max policy
This subsec ion summa izes he empi ical esul s ob ained in ou p elimina y esea ch a icle [175]. The
main objec i e o his expe imen is o e alua e he pe o mance o a simple so max policy wi hou da a
euploading in a se o s anda d classical con ol RL benchma king en i onmen s [194]. In [93], he
au ho s sol ed hese en i onmen s using a pa ame e ized o m wi h mul iple laye s o da a eupload-
ing ga es. Mo eo e , hey showed ha he PQC-based so max policy (see De ini ion 5.1.6) has be e
sample complexi y gua an ees compa ed wi h he Bo n policy (see De ini ion 5.1.1). Howe e , eupload-
ing inc eases bo h he ci cui dep h and he numbe o ainable pa ame e s (conside ing inpu scaling),
leading o he well-known exp essi i y- ainabili y adeo [183]. Thus, in his expe imen , we aimed o in-
es iga e whe he a simple so max model wi hou da a euploading could also sol e hese en i onmen s,
wi h he goal o e ec i ely educing he o al numbe o ainable pa ame e s and he ci cui dep h. Indeed,
o such en i onmen s, da a euploading was no necessa y o sol ing hem. Howe e , ega ding sample
complexi y, i was ul ima ely shown ha he da a euploading model equi ed ewe samples o achie e
he same pe o mance—highligh ing he impo ance o model exp essi i y.
96
5.4. NUMERICAL EXPERIMENTS
5.4.2 Da a euploading e ec on pe o mance
In he p e ious subsec ion, a simple so max policy was e alua ed h ough classical benchma king en i-
onmen s. None heless, he e a e se e al PQC-based Bo n policies, as p oposed in Sec ion 5.1, whose
pe o mance should also be in es iga ed. Fu he mo e, hese should be compa ed agains he PQC-based
so max policy. In ha ega d, we compa e policies in h ee di e en scena ios:
1. Bo n policies — We conside he
con iguous-like
(see De ini ion 5.1.2),
pa i y-like
(see De ini ion
5.1.5), and
ac ion-p ojec o -like
(see De ini ion 5.1.3).
2. Bo n policies w/ so max ac i a ion — The same Bo n policies as in 1) bu wi h a so max
pos -p ocessing ac i a ion o add g eediness con ol.
3. So max policy — A gene al PQC-based so max policy (see De ini ion 5.1.6) wi h He mi ian ob-
se ables.
Fu he mo e, we choose a small se o classical benchma king en i onmen s and di e en PQC a chi ec-
u es. In so doing, we can p ope ly assess and compa e di e en policies. We conside he same se o
classical en i onmen s as in he p e ious subsec ion wi h he same mino modi ica ions, desc ibed below
o comple eness:
•Ca pole: This simula ion in ol es a ca ha mo es along a ic ionless ack wi h an in e ed
pendulum a ached. The sys em is desc ibed by ou ea u es: he ca ’s posi ion and eloci y, and
he angle and angula eloci y o he pole. The agen has wo possible ac ions: mo ing he ca
le o igh . The p ima y objec i e is o keep he pole balanced up igh , wi h he agen ecei ing
a ewa d o +1 o each ime s ep ha he pole emains up igh , wi h a maximum o 200 s eps.
The e o e, he maximum possible ewa d is 200.
•Ac obo : Consis ing o a wo-link obo ic a m, he Ac obo en i onmen o iginally ea u es six s a e
a iables: he cosine and sine o he wo join angles and hei espec i e eloci ies. Howe e , o
simpli y and uni y he ea u e space wi h he Ca pole en i onmen , we conside only he angles
di ec ly, educing he ea u e coun o ou . This adap a ion ocuses on he essen ial dynamic cha -
ac e is ics o he sys em. The agen con ols he o que a he second join and has h ee ac ion
choices: le wa d o que, no o que, o igh wa d o que. The goal is o swing he lowe link o a
speci ic heigh as quickly as possible. The agen ecei es a ewa d o -1 o each imes ep un il he
a ge is eached. The maximum numbe o s eps is 500, bu he e is no ixed maximum ewa d
since he agen can each he a ge in ewe s eps. Fo his en i onmen only, he
ac ion-p ojec o -
like
and
con iguous-like
policies a e conside ed because he ac ion space is no a powe o wo;
hus, he
pa i y-like
policy is no applicable.
103

CHAPTER 5. QUANTUM POLICY GRADIENTS
The en i onmen s’ ull desc ip ion can be ound in Table 11. Fo simplici y and scalabili y, he numbe o
ea u es o he Ac obo en i onmen was educed om six o ou , such ha we can conside he same
PQC in bo h en i onmen s and e ec i ely educe signi ican ly he numbe o ainable pa ame e s. This
is c ucial o imp o e he algo i hm’s ime complexi y.
Rega ding he PQC a chi ec u e, wo ci cui s we e selec ed ha use only single-qubi pa ame e ized ga es.
Two-qubi ga es a e applied in a non-pa ame e ized ashion o gene a e en anglemen in he ci cui wi hou
inc easing he numbe o ainable pa ame e s. The ci cui s a e de ined as ollows:
1.
Je bi
— Ansa z composed o single-qubi pa ame e ized o a ions abou wo o hogonal axes {𝑅𝑧, 𝑅𝑦},
ollowed by an
all- o-all
en anglemen pa e n o CZ ga es, as p oposed by Je bi e al. [93]. The
da a-encoding p ocedu e is done ia s anda d angle-encoding, using wo o a ion axes as well bu in
he opposi e o de {𝑅𝑦, 𝑅𝑧}, and is applied a e he pa ame e ized block. The ci cui has as many
qubi s as he numbe o ea u es in he inpu s a e. A laye o he PQC is illus a ed (shaded pu ple)
in Figu e 42(a).
2.
Uni e sal Quan um Classi ie (UQC)
— A single-qubi a chi ec u e composed o wo o hogonal o a-
ion axes {𝑅𝑦, 𝑅𝑧}, wi h a se o ainable pa ame e s Θ={𝜑,𝑤, 𝛼}. The angle o he 𝑧- o a ion
is exp essed simila ly o a classical neu on: h𝑠,𝑤i+𝛼, whe e h𝑠, 𝑤iis he inne p oduc be ween
he s a e and ainable pa ame e s 𝑤.𝛼plays he ole o a bias e m as in a classical linea model.
The axis o o a ion can be lipped; o hogonali y is he necessa y condi ion. Salinas e al. [152]
p o ed ha in he limi o in ini e epe i ions, he UQC ci cui is a uni e sal app oxima o . In his
wo k, we conside he UQC ci cui as i enables ine con ol o e he numbe o qubi s, as i is
independen o he numbe o ea u es. The e o e, we es he pe o mance o he UQC o a ini e
numbe o qubi s {1,2,4}, wi h ou being he maximum allowed o ma ch he numbe o qubi s
used in he
Je bi
a chi ec u e. Fo mo e han a single qubi , a nea es -neighbo en angling block o
CZ ga es is applied. The ci cui is illus a ed in Figu e 42(b).
104
5.4. NUMERICAL EXPERIMENTS
Figu e 42: Pa ame e ized quan um ci cui s conside ed o he expe imen . (a) Je bi a chi ec u e, and (b)
UQC ci cui o a ou -qubi sys em. The undamen al laye is shaded pu ple o bo h ci cui s.
The ange o qubi s o he
UQC
ci cui was chosen o be {1,2,4}, enabling a ai compa ison wi h he
Je bi
ci cui , which also uses ou qubi s when ma ching he numbe o ea u es. The numbe o ainable
pa ame e s o he UQC ci cui inc eases wi h he numbe o qubi s, e en ually ma ching he o al numbe
o pa ame e s o he
Je bi
ci cui a 𝑁=2. The o al numbe o pa ame e s o bo h ci cui s is exp essed
in Table 6.
Ci cui Numbe o pa ame e s
Je bi 4|𝑠|𝐿
UQC 𝑁(|𝑠| +2)𝐿
Table 6: Numbe o pa ame e s o he UQC and Je bi ci cui s, whe e |𝑠|is he numbe o ea u es, 𝐿is
he numbe o laye s, and 𝑁is he numbe o qubi s.
PQC-based policies a e ained using he REINFORCE algo i hm (see Algo i hm 4) by maximizing he
a e age ewa d collec ed h oughou ajec o ies, wi h a lea ning a e o 0.01 and a discoun ac o o
𝛾=0.99. To e alua e he pe o mance o di e en policies, we conside he me hodology desc ibed in
Subsec ion 5.3.2: he policy is ained o 500 episodes, and he ewa d is collec ed and plo ed as a
unc ion o he numbe o episodes. The ewa d is a e aged o e 10 uns wi h di e en se s o andomly
ini ialized pa ame e s om a Gaussian dis ibu ion N(0,1). In he ollowing subsec ions, we analyze
he pe o mance o he model in bo h en i onmen s ega ding he policies in ques ion. Fu he mo e,
105
CHAPTER 5. QUANTUM POLICY GRADIENTS
o analyze he quan um s a e a he end o aining, we moni o he en anglemen du ing aining, o
be e unde s and he quan umness o he esul ing s a e. We conside he Meye -Wallach measu e o
en anglemen , due o i s scalabili y and ease o compu a ion (see Sec ion 2.3). C ucially, i he PQC has
low en anglemen a he end o aining, he quan um model can e ec i ely be eplaced wi h a classical
model, indica ing ha he quan um de ice ha nessed en anglemen o ain, and indeed a es ing phase
he model can be eadily deployed wi h a classical de ice, sa ing mul iple esou ces.
5.4.2.1 Cumula i e ewa d
To keep he analysis as s aigh o wa d as possible, we b eak i in o h ee pa s, co esponding o he h ee
dis inc policy se s in oduced a he s a o his subsec ion. In each pa , we analyze bo h he ewa d
and he en anglemen o each en i onmen .
1) Bo n policies
The Bo n policies conside ed in his expe imen a e he
con iguous-like
,
pa i y-like
, and
ac ion-p ojec o -
like
policies. Conside he ollowing measu emen appa a us o bo h en i onmen s using an 𝑁-qubi
PQC-based policy:
1. Ca pole — |𝐴|=2
a) Con iguous-like — Recall om De ini ion 5.1.2 ha he con iguous pa i ion is ob ained om a
log |𝐴|-local measu emen . The e o e, in his se ing, we measu e only a single qubi .
b) Pa i y-like — The pa i y-like policy is ob ained om an 𝑁-local measu emen . In his se ing,
he s anda d pa i y unc ion wi hou ecu sion is applied (see De ini ion 5.1.5).
c) Ac ion-p ojec o -like — The ac ion-p ojec o -like policy is also ob ained om an 𝑁-local mea-
su emen . In his se ing, we conside he p ojec o s 𝑃0=|0ih0|and 𝑃1=|1ih1|.
2. Ac obo — |𝐴|=3
a) Con iguous-like — We spli he 2𝑁basis s a es in a con iguous ashion o he h ee a ailable
ac ions.
b) Ac ion-p ojec o -like — We conside he p ojec o s 𝑃0=|0ih0|,𝑃1=|1ih1|, and 𝑃2=|2ih2|.
The cumula i e ewa d ob ained du ing aining by he di e en Bo n policies, using he
UQC
and
Je bi
PQCs, is depic ed in Figu e 43.
106
5.4. NUMERICAL EXPERIMENTS
Figu e 43: Cumula i e ewa ds ob ained by he Bo n policies in he Ca pole en i onmen : (a) UQC wi h
a numbe o qubi s in {1,2,4}and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is labeled
global
in he legend o conciseness.
Figu es 43 (a) and (b) show he cumula i e ewa ds ob ained by he Bo n policies in he Ca pole en i on-
men using he UQC and Je bi a chi ec u es, espec i ely. The esul s eadily indica e an o e all supe io
pe o mance o he Je bi-based policies compa ed o he UQC-based policies. Howe e , su p isingly, a
single-qubi UQC is able o main ain a sa is ac o y pe o mance. Ne e heless, he pa i y-based policy is
he bes -pe o ming policy in bo h ci cui s, highligh ing he exp essi i y o he pa i y unc ion. In e es ingly,
he bes -pe o ming UQC model is composed o wo qubi s, indica ing ha ha ing he same numbe o
ainable pa ame e s alone does no gua an ee be e pe o mance. Indeed, a majo challenge wi h he
UQC is encoding e e y ea u e in o he same qubi , which can sc amble he in o ma ion. Meanwhile,
he Je bi ci cui possesses an
all- o-all
en anglemen pa e n, enabling mo e co ela ions be ween pai s o
ea u es and inc easing he exp essi i y. Figu e 44 shows he en anglemen du ing aining.
Figu e 44: En anglemen du ing aining o he Bo n policies in he Ca pole en i onmen : (a) UQC wi h
a numbe o qubi s in {1,2,4}and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is labeled
global
in he legend.
Figu es 44 (a) and (b) show ha he Je bi ci cui gene ally displays highe en anglemen du ing aining
compa ed wi h he UQC ci cui , pa icula ly when compa ing he con iguous-like policies. This is expec ed
because he Je bi ci cui has an
all- o-all
en anglemen pa e n, whe eas he UQC ci cui has a nea es -
neighbo en anglemen pa e n. No e also ha bo h pa i y-like policies success ully main ain en anglemen
107
CHAPTER 5. QUANTUM POLICY GRADIENTS
h oughou aining. By con as , he ac ion-p ojec o -like and con iguous-like policies in bo h ci cui s wi h
ou qubi s exhibi a endency o hei en anglemen o dec ease. Thei pe o mance is no be e han he
pa i y-like policy, sugges ing ha en anglemen could be a key ac o in he powe o PQC-based policies.
Rega ding he Ac obo en i onmen , he esul s a e depic ed in Figu e 45.
Figu e 45: Cumula i e ewa ds ob ained by he Bo n policies in he Ac obo en i onmen : (a) UQC wi h a
numbe o qubi s in {1,2,4}and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is labeled
global
in he legend.
Figu es 45 (a) and (b) show he cumula i e ewa ds ob ained by he Bo n policies in he Ac obo en i on-
men using he UQC and Je bi a chi ec u es, espec i ely. Again, he Je bi ci cui demons a es supe io
pe o mance compa ed wi h he UQC ci cui . Mo eo e , he inc eased complexi y o his en i onmen has
also inc eased he a iance in he esul s. The UQC ci cui emains no ably mo e uns able han he Je bi
ci cui . In e es ingly, o bo h ci cui s, he con iguous policy is he bes -pe o ming one (pa i y-like is no
applicable in a h ee-ac ion se ing). Figu e 46 illus a es he en anglemen o e he cou se o aining.
Figu e 46: En anglemen du ing aining o he Bo n policies in he Ac obo en i onmen : (a) UQC wi h
a numbe o qubi s in {1,2,4}and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is labeled
global
in he legend.
The esul s indica e ha he en anglemen pa e n du ing aining o he Bo n policies emains oughly
he same o bo h en i onmen s when using he UQC. The Je bi ci cui shows highe en anglemen ea ly
108

5.4. NUMERICAL EXPERIMENTS
in aining, bu he e is a clea downwa d end as aining p og esses. I aining con inued o mo e
episodes, he en anglemen migh dec ease e en u he , pa icula ly wi h he con iguous policy. I is
impo an o no e ha he a iance in he esul s is also in luenced by he small numbe o agen s (10)
used in he a e aging p ocess. Gi en he en i onmen ’s complexi y and he lack o high-pe o mance
simula o s, i was no easible o conside mo e agen s wi hin a easonable ime. This limi a ion will also
apply o subsequen esul s, so i will no be ei e a ed below.
2) Bo n policies w/ so max ac i a ion
Because he so max unc ion gene ally has mo e con olled g adien s and, c ucially, adds g eediness
con ol o he policy, his pa analyzes he pe o mance o he same Bo n policies conside ed p e iously,
bu augmen ed wi h a so max ac i a ion and pa ame e 𝛽=1
𝜏 o con olling g eediness, whe e 𝜏is he
empe a u e. Since we a e conside ing a small se o classical con ol en i onmen s, we used a linea
annealing schedule o he empe a u e. The cumula i e ewa ds ob ained du ing aining by he di e en
Bo n policies using he
UQC
and
Je bi
a chi ec u es a e depic ed in Figu e 47.
Figu e 47: Cumula i e ewa ds ob ained by he Bo n policies wi h so max ac i a ion in he Ca pole
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is
labeled
global
in he legend.
As expec ed, he so max ac i a ion imp o es he pe o mance o he Bo n policies unde bo h ci cui s.
S ill, he Je bi a chi ec u e ou pe o ms he UQC a chi ec u e, u he s abilizing aining. Figu e 48 shows
he en anglemen du ing aining.
109
CHAPTER 5. QUANTUM POLICY GRADIENTS
Figu e 48: En anglemen du ing aining o he Bo n policies wi h so max ac i a ion in he Ca pole
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is
labeled
global
in he legend.
The esul s indica e ha despi e he so max ac i a ion, he en anglemen pa e n du ing aining o he
Bo n policies emains oughly he same.
Fo he Ac obo en i onmen , he esul s a e depic ed in Figu e 49.
Figu e 49: Cumula i e ewa ds ob ained by he Bo n policies wi h so max ac i a ion in he Ac obo en-
i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is
labeled
global
in he legend.
Figu es 49 (a) and (b) show he cumula i e ewa ds ob ained by he Bo n policies wi h so max ac i a ion
in he Ac obo en i onmen using he UQC and Je bi a chi ec u es, espec i ely. Unlike in he Ca pole
en i onmen , he so max ac i a ion does no imp o e he pe o mance o he Bo n policies in he Ac obo
en i onmen . This could be explained by he en i onmen ’s highe a iance o by he linea annealing
schedule o 𝜏. The en i onmen ’s mo e complex dynamics equi e mo e explo a ion. The e o e, a linea
schedule may in ac be subop imal. Figu e 50 shows he en anglemen du ing aining.
110
5.4. NUMERICAL EXPERIMENTS
Figu e 50: En anglemen du ing aining o he Bo n policies wi h so max ac i a ion in he Ac obo
en i onmen : (a) UQC wi h {1,2,4}qubi s, and (b) Je bi a chi ec u e. The ac ion-p ojec o -like policy is
labeled
global
in he legend.
These esul s indica e ha he en anglemen pa e n du ing aining o he Bo n policies wi h so max
ac i a ion emains oughly he same as o he o iginal Bo n policies in bo h en i onmen s. Howe e ,
o he Je bi ci cui shown in Figu e 50(b), he en anglemen is e en lowe o he con iguous-like policy,
which migh be due simply o s a is ical noise om ha ing ewe agen s.
3) So max policy
He e, we conside he So max policy. In his se ing, he choice o he obse able is c ucial o he
policy’s pe o mance. Le us conside he ollowing local and global obse ables o bo h en i onmen s,
as desc ibed in Table 7.
Table 7: Obse ables o he So max policy in di e en en i onmen s
En i onmen Ci cui Qubi s Obse ables
Ca pole
UQC
1[𝑍0,−𝑍0]
2𝐿=[𝑍1,−𝑍1]𝐺=[𝑍0𝑍1,−𝑍0𝑍1]
4𝐿=[𝑍3,−𝑍3]𝐺=[𝑍0𝑍1𝑍2𝑍3,−𝑍0𝑍1𝑍2𝑍3]
Je bi 4𝐿=[𝑍3,−𝑍3]𝐺=[𝑍0𝑍1𝑍2𝑍3,−𝑍0𝑍1𝑍2𝑍3]
Ac obo
UQC
1[𝑍0, 𝑋0,−𝑍0]
2[𝑍0, 𝑍0𝑍1, 𝑍1]
4[𝑍0, 𝑍1𝑍2, 𝑍3]
Je bi 4[𝑍0, 𝑍1𝑍2, 𝑍3]
The choice o local and global obse ables impac s he pe o mance, en anglemen , and ainabili y o he
model. He e, ainabili y is no pa icula ly es ic i e because he model is small (up o ou qubi s and
ou laye s o single-qubi pa ame e ized ga es). The cumula i e ewa d and en anglemen ob ained du ing
111
CHAPTER 5. QUANTUM POLICY GRADIENTS
aining by he local So max policy using he
UQC
and
Je bi
a chi ec u es in he Ca pole en i onmen
a e depic ed in Figu e 51.
Figu e 51: Local so max policy in he Ca pole en i onmen : (a) cumula i e ewa d and (b) en anglemen
du ing aining. The UQC wi h {1,2,4}qubi s and Je bi a chi ec u e is conside ed.
As seen in Figu e 51(a), he local So max policy om he Je bi a chi ec u e, e en hough i measu es only
a single qubi , achie es be e pe o mance in he Ca pole en i onmen han he UQC, which pe o ms
signi ican ly wo se ela i e o he (1) Bo n and (2) Bo n w/ so max ac i a ion policies. We expec mo e
complex obse ables o imp o e he agen ’s pe o mance. Figu e 51(b) shows a sligh dec easing end in
en anglemen o he Je bi a chi ec u e in he So max o mula ion, al hough i migh be due o s a is ical
noise.
Fo he Ac obo en i onmen , he esul s a e shown in Figu e 52.
Figu e 52: Local so max policy in he Ac obo en i onmen : (a) cumula i e ewa d and (b) en anglemen
du ing aining. The UQC wi h {1,2,4}qubi s and Je bi a chi ec u e is conside ed.
Figu e 52(a) e eals a clea di e ence in pe o mance be ween he UQC and Je bi a chi ec u es in he
Ac obo en i onmen . The Je bi a chi ec u e wi h he obse ables indica ed in Table 7 is he only policy
ha achie es sa is ac o y pe o mance. None o he UQC con igu a ions come close o he Je bi a chi ec-
u e. Gi en ha o he So max policy, we also used he same linea annealing schedule, hese esul s
112
6.2. BORN POLICIES
obse ables (see Table 1). Subsec ion 6.2.1 begins wi h a case s udy o p oduc s a es, ollowed by he
gene al beha io o en angled s a es in Subsec ion 6.2.2. The a iance as a unc ion o he numbe o
ac ions is in es iga ed in Subsec ion 6.2.3. Subsec ion 6.2.4 hen examines he Fishe In o ma ion spec-
um associa ed wi h he policy. Finally, Subsec ion 6.2.5 p o ides nume ical expe imen s ha alida e
he heo e ical esul s om Subsec ions 6.2.3 and 6.2.4.
6.2.1 The ins uc i e case o p oduc s a es
Conside a scena io whe e a PQC-based Bo n policy is composed o a p oduc s a e PQC wi hou inco -
po a ing he agen ’s s a e. Le he 𝑁-qubi PQC be
|𝜓𝜃i=
𝑁−1
Ì
𝑖=0
𝑒−𝑖𝜃𝑖𝑃𝑖|0i, 𝜌𝜃=|𝜓𝜃ih𝜓𝜃|,(6.4)
Assume 𝑃𝑖=𝑌 o all 𝑖∈ {0,1, . . . , 𝑁 −1}and conside he ask o lea ning he all-ze o s a e. The cos
unc ion o be minimized is
𝐶(𝜃)=1−T 𝜌𝜃𝑃0,(6.5)
whe e 𝑃0=|0ih0|is he p ojec o on o he all-ze o s a e. The minimum cos co esponds o a p obabili y
o one o measu ing he all-ze o s a e, indica ing ha he en i e s a e is concen a ed in |0i⊗𝑁. Recall
ha
|𝜓𝜃i=
𝑁−1
Ì
𝑖=0©«
cos𝜃𝑖
2
sin𝜃𝑖
2ª®¬.(6.6)
Then he p obabili y o measu ing he all-ze o s a e is
𝑝0(𝜃)=
𝑁−1
Ö
𝑖=0
cos2(𝜃𝑖).(6.7)
The unc ion 𝑝0(𝜃)is a p oduc o 𝑁 ac o s, each wi hin he in e al [0,1]. I each 𝜃𝑖is andomly d awn
(e.g., om a uni o m dis ibu ion 𝜃𝑖∼𝑈[−𝜋, 𝜋]), hen he a e age alue o cos2(𝜃𝑖)is s ic ly less han
1 (in ac , i is 1/2i 𝜃𝑖is uni o m on [−𝜋, 𝜋]). The e o e, uni o m ini ializa ion ypically leads o an
exponen ial
dec ease in 𝑝0(𝜃)as 𝑁g ows (see Figu e 53(a)). Fo ins ance, i hcos2(𝜃𝑖)i =𝑐<1, hen
h𝑝0(𝜃)i =𝑁−1
Ö
𝑖=0
cos2(𝜃𝑖)≈𝑐𝑁,
Howe e , i 𝜃𝑖≈0(o 𝜃𝑖≈2𝑘𝜋 o in ege 𝑘), hen cos2(𝜃𝑖) ≈ 1, so 𝑝0(𝜃)
may no
decay as 𝑁
inc eases. Likewise, ca e ully chosen angles o ci cui designs ha keep cos2(𝜃𝑖)nea 1 o all 𝑖may
esul in 𝑝0(𝜃) emaining la ge. This unde sco es he impo ance o pa ame e ini ializa ion in PQC-based
policies. In [42], i was shown ha his e ec a ises no only because o andom ini ializa ion bu also
because a global p ojec o is measu ed. The au ho s p oposed modi ying he cos unc ion by including
local con ibu ions pe qubi :
𝑂𝐿=1
𝑁
𝑁−1
Õ
𝑗=0|0ih0|𝑗⊗𝕀¯
𝑗,
119

CHAPTER 6. TRAINABILITY ISSUES IN QUANTUM POLICY GRADIENTS
whe e 𝕀¯
𝑗is he iden i y on he o he qubi s. The cos unc ion hen becomes
𝐶(𝜃)=1−T 𝜌𝜃𝑂𝐿=1−1
𝑁
𝑁−1
Õ
𝑖=0
cos2(𝜃𝑖),(6.8)
which ai h ully es ima es he all-ze o s a e while yielding polynomially anishing quan i ies wi h 𝑁, as
illus a ed in Figu e 53(b). This emphasizes he key in luence o a sui ably chosen cos unc ion.
Figu e 53: (a) Va iance o he p obabili y o measu ing he all-ze o s a e unde a global p ojec o . (b) log
plo o he a iance o T [𝜌𝜃P], whe e Pis ei he he global o local p ojec o as desc ibed abo e. The
a iance is plo ed e sus he numbe o qubi s o 1000 andomly sampled pa ame e s 𝜃∈𝑈(−𝜋, 𝜋).
In a b oade ML con ex , pa icula ly o policy g adien s, he log-likelihood is op imized a he han he
di ec p obabili y conside ed abo e. Tha cos unc ion beha es di e en ly. Fo ins ance, i
𝐽(𝜃)=log T 𝜌𝜃𝑃0,(6.9)
he loga i hm decomposes he p oduc o cosines in o a sum:
𝐽(𝜃)=
𝑁−1
Õ
𝑖=0
log cos2(𝜃𝑖).(6.10)
Thus, he pa ial de i a i e wi h espec o 𝜃𝑖is
𝜕𝜃𝑖𝐽(𝜃)=
𝑁−1
Õ
𝑖=0
𝜕𝜃𝑖log cos2(𝜃𝑖)
=
𝑁−1
Õ
𝑖=0−2 sin(𝜃𝑖)
cos(𝜃𝑖)=
𝑁−1
Õ
𝑖=0−2 an(𝜃𝑖).(6.11)
which is no longe a p oduc o 𝑁 e ms in [0,1]. I a pa ame e is sha ed ac oss all ga es (e.g., in he
QAOA laye s [66]), he pa ial de i a i e does depend on 𝑁. Howe e , o he case 𝜃∈ℝ𝑁(i.e., unsha ed
pa ame e s), he pa ial de i a i e decouples:
𝜕𝜃𝑖𝐽(𝜃)=−2 an(𝜃𝑖),
120
6.2. BORN POLICIES
so he a iance o his pa ial de i a i e is independen o 𝑁. No ably, wi h uni o mly sampled 𝜃∼
𝑈[−𝜋, 𝜋], he expec ed alue o he pa ial de i a i e may s ill be unde ined (due o 𝜃=±𝜋/2). When
measu ing he all-ones basis s a e 𝑝2𝑁(𝜃)ins ead, a simila exp ession in ol ing a c an(𝜃𝑖)a ises, which
can be ze o-mean in he same andom ini ializa ion in e al.
Hence, pa ame e ini ializa ion is c ucial; o example, 𝜃sampled om [𝜋
4,𝜋
4]migh ensu e bounded
pa ial de i a i es. Gene ally, when 𝜌𝜃is a p oduc s a e, he p obabili y o measu ing |𝑎i ac o s in o
indi idual qubi con ibu ions,
T 𝜌𝜃𝑃𝑎=
𝑁−1
Ö
𝑖=0
T 𝜌𝜃𝑖𝑃𝑎𝑖.
The log-likelihood hus sepa a es in o a sum,
𝐽(𝜃)=
𝑁−1
Õ
𝑖=0
log T 𝜌𝜃𝑖𝑃𝑎𝑖,
po en ially a oiding a BP since i u ns i in o a sum o e 𝑁 e ms, and make anishing quan i ies inde-
penden od he numbe o qubi s and depeden o he ini ializa ion only.
Conside nex an a bi a y p oduc s a e composed o 𝐿laye s o single-qubi o a ions,
|𝜓(𝜃)i =
𝐿−1
Ö
𝑙=0
𝑁−1
Ì
𝑖=0
𝑒−𝑖 𝜃𝑖,𝑙 𝑃𝑖,𝑙 |0i, 𝜌𝜃=|𝜓(𝜃)ih𝜓(𝜃)|,(6.12)
whe e 𝑃𝑖,𝑙 ∈ {𝑌}and 𝜃𝑖,𝑙 ∈ℝ. The p obabili y o measu ing a basis s a e 𝑃𝑎 ac o s in o indi idual
qubi s, bu he pa ial de i a i e o he log p obabili y wi h espec o 𝜃𝑖,𝑙 may depend on he numbe o
qubi s and laye s, because each qubi ’s p obabili y is now a sum o 𝐿 e ms:
T 𝜌𝜃𝑖,𝑙 𝑃𝑎𝑖=
𝐿−1
Õ
𝑙=0
𝑎(𝜃𝑖,𝑙 ).
Then
𝜕𝜃𝑖,𝑙 𝐽(𝜃)=
𝑁−1
Õ
𝑙=0
𝜕𝜃𝑖,𝑙 log𝐿−1
Õ
𝑙=0
𝑎(𝜃𝑖,𝑙 ).
I pa ame e s a e no sha ed ac oss ga es, he pa ial de i a i e o each qubi decouples om 𝑁, bu i
can s ill scale wi h he numbe o laye s. Speci ically,
𝜕𝜃𝑖,𝑙 𝐽(𝜃)=𝜕𝜃𝑖,𝑙 Í𝐿−1
𝑙=0𝑎(𝜃𝑖,𝑙 )
Í𝐿−1
𝑙=0𝑎(𝜃𝑖,𝑙 )=𝜕𝜃𝑖,𝑙 𝑎(𝜃𝑖,𝑙 )
Í𝐿−1
𝑙=0𝑎(𝜃𝑖,𝑙 ).
Hence, ci cui dep h ma e s alongside qubi coun . In PQC-based RL, no e ha hese examples i a Bo n
policy in he egime |𝐴|=2𝑁, implying no pa i ion o e he ac ion space. The beha io changes i he
ac ion space is pa i ioned, as explo ed in subsequen sec ions.
121
CHAPTER 6. TRAINABILITY ISSUES IN QUANTUM POLICY GRADIENTS
6.2.2 Gene alized beha io o en angled s a es
In his subsec ion, we analyze he a iance o he log-p obabili y o en angled s a es. In pa icula , le
us s ill conside he ex eme scena io o a numbe o ac ions scaling exponen ially wi h sys em size as
|𝐴|=2𝑛, same as Subsec ion 6.2.1. The e o e, he measu emen s a e s ill composed o global p ojec o s
in o one o he 2𝑁possible basis s a es.
Le 𝜌(𝜃)be an a bi a ily en angled PQC. Le 𝑃𝑎=|𝑎ih𝑎|be he p ojec o in o he 𝑎 h basis s a e. In
his scena io, he p obabili y o measu ing a basis s a e 𝑎is, in gene al, no decomposed by he p oduc
o indi idual qubi subsys ems,
T 𝜌𝜃𝑃𝑎≠
𝑁−1
Ö
𝑖=0
T 𝜌𝜃𝑖𝑃𝑎𝑖(6.13)
since he sys em is no sepa able. Indeed, he p obabili y will be ac o ed as he p oduc o he p obabili y
associa ed wi h un angled subsys ems 𝑆(𝑁). Le us conside he en angled quan um s a e illus a ed in
Figu e 54, composed o a 𝑁=4PQC. In his se ing he PQC e en hough composed o a ou qubi
Figu e 54: En angled PQC composed o wo Bell s a es in a ou qubi sys em.
sys em i can be analyzed sepa a ely as wo qubi subsys ems 𝑆1(𝑁)and 𝑆2(𝑁)each one ac ing on
wo qubi s. Thus, he p obabili y o measu ing a global p ojec o 𝑃𝑎on all ou qubi s is ac o ed as he
p oduc ,
T 𝜌𝜃𝑃𝑎=T 𝜌𝜃𝑃𝑆1
𝑎T 𝜌𝜃𝑃𝑆2
𝑎(6.14)
whe e 𝑃𝑆1
𝑎and 𝑃𝑆2
𝑎a e he p ojec o s in o he 𝑎 h basis s a e decomposed in o he subsys ems 𝑆1(𝑁)
and 𝑆2(𝑁), espec i ely. The e o e, in en angled s a es he loga i hm can s ill sepa a e he p oduc o
p obabili ies bu now ins ead o indi idual qubi con ibu ions, he p oduc is sepa a ed in o he p oduc
o he p obabili ies o he se o unen angled subsys ems {𝑆(𝑁)},
𝐽(𝜃)=log T 𝜌𝜃𝑃𝑎=Õ
𝑖∈{𝑆(𝑁)}
log T 𝜌𝜃𝑃𝑆𝑖
𝑎 (6.15)
122
6.2. BORN POLICIES
The ainabili y o he en angled s a e is hen analyzed by he a iance o he log p obabili y pa ial de i a i e
and in his se ing i boils down essen ially o he size o he subsys em whe e he pa ame e is con ained.
The e o e, no ice ha , since we a e measu ing e e y qubi wi hin he subsy em, i is s ill a global measu e-
men . Thus, he p obabili y o measu ing a s a e wi hin each subsys em will be exponen ially small wi h
he size o he subsys em. This d as ically changes he beha io o he pa ial de i a i e. We ha e he
loga i hm o a p obabili y ha is anishing exponen ially wi h he numbe o qubi s. The e o e, he pa ial
de i a i e will go in he opposi e di ec ion. Tha is, i will inc ease wi h he numbe o qubi s p esen in
he subsys em. Thus, i seems ha an exploding g adien is bound o happen ins ead o a anishing
g adien . Ne e heless, i s ill u ns o be ha d o ain he model. No ice ha he p obabili y is ge ing
exponen ially small wi h he numbe o qubi s. The e o e, e en hough he g adien inc eases, we s ill
need e en ually an exponen ial numbe o sho s (o quan um ci cui execu ions) o p ope ly es ima e he
p obabili y. The e o e, ainabili y will be gua an eed p o ided e icien es ima ion o p obabili ies.
Le us s ess ha such beha io is expec ed because we a e allowing global measu emen s wi hin each
subsys em. Mo e gene ally, i is going o depend on he s uc u e o ga es and measu emen s. To gene -
alize, le us conside he esul s om Ce ezo e .al [42] in which he au ho s show ha O(log 𝑁)dep h
p esen s a ainable egion, esul ing as well om he measu emen o O(log 𝑁)qubi s. This is gua an-
eed o ci cui s able o p oducing local 2-designs (See De ini ion 3.4.1). Meaning ha e icien p obabili y
es ima ion will depend on bo h he numbe o qubi s being measu ed as well as he dep h o he ci cui o
e icien g adien signal p opaga ion. Indeed, o PQC-based policies, his in u n depend on he numbe
o ac ions |𝐴|o he en i onmen we a e ying o sol e since hese impac he locali y o he measu emen
(see Table 1).
To demons a e he e ec o |𝐴|=2𝑁in he con ex o gene alized en angled s a es, le us conside
h ee ypes o ci cui s:
1. Simpli ied 2-design ansa z illus a ed in Figu e 55(a).
2. S ongly en angling laye s, depic ed in Figu e 55(b).
3. S a e gene a ed om Pauli o a ions sampled uni o mly a andom ollowed by andomly selec ed
CZ ga es, as illus a ed in 55(c).
Figu e 55(d) illus a es he a iance o he g adien 2-no m o he log p obabili y o a se o andomly
selec ed global p ojec o s. The a iance is illus a ed as a unc ion o he numbe o qubi s o 𝑁laye s
o he blocks shown in hei espec i e igu es. Mo eo e , p ojec o s we e sampled uni o mly a andom
om he se o 2𝑁a ailable ones and he a iance illus a ed o an a e age o a housand expe imen s.
F om Figu e 55(d), i is e iden ha in each expe imen , he a iance o he log-p obabili y inc eases wi h
he numbe o qubi s when global p ojec o s a e conside ed. This beha io is akin o wha was desc ibed
be o e. P obabili ies a e ge ing exponen ially supp essed wi h he numbe o qubi s 𝑁making he pa ial
123
CHAPTER 6. TRAINABILITY ISSUES IN QUANTUM POLICY GRADIENTS
Figu e 55: Va iance o he log policy g adien o h ee dis inc en angled s a es. (a) Simpli ied wo de-
sign. (b) S ongly en angling laye s. (c) Random s a es composed o Pauli o a ions sampled uni o mly a
andom ollowed by andomly selec ed CZ ga es. (d) Va iance as a unc ion o he numbe o qubi s o
𝑁laye s o building blocks o each o he ci cui s (a)-(c).
de i a i e o he log p obabili y inc ease. The a iance eaches ex emely high le els as a unc ion o 𝑁,
indica ing ha al hough hese ci cui s a e p one o he exploding g adien phenomenon. This u he leads
us o conclude ha an exponen ially la ge numbe o quan um ci cui execu ions is equi ed o accu a ely
es ima e bo h he p obabili y and i s g adien . Howe e , ecall ha in he con ex o RL, we will need o do
a pa i ioning o possibly all 2𝑁basis s a es in o he se o a ailable ac ions |𝐴|. In such cases, a ainable
egion could be c ea ed depending on he locali y o he p ojec o , which in u n is hea ily in luenced by
he ype o Bo n policy implemen ed. In he ollowing subsec ion, we examine he a iance o he cos
unc ion o di e en Bo n policies as a unc ion o he numbe o ac ions |𝐴|.
Le us now p oceed o he analysis o he a iance o he log likelihood cos unc ion o Bo n policies
as a unc ion o he numbe o ac ions |𝐴|.
6.2.3 Va iance as a unc ion o he numbe o ac ions
This subsec ion analyzes he a iance o he log-p obabili y cos unc ion o Bo n policies as a unc ion
o he en i onmen ’s a ailable ac ions. In Subsec ion 6.2.2, i was obse ed ha es ima ing policy p ob-
abili ies can become exponen ially ha d when
global
measu emen s a e pe o med on bo h p oduc and
en angled s a es wi h inc easing qubi coun . Consequen ly, he pa ial de i a i e o he log-p obabili y
would po en ially “explode” because he measu ed p obabili ies a e exponen ially small. In p ac ice, his
beha io equi es cla i ica ion, since di e en beha io a ise when he ac ion space is pa i ioned. Recall
ha Con iguous (De ini ion 5.1.2) and Pa i y-like (De ini ion 5.1.5) Bo n policies di e in how hey pa i ion
measu emen ou comes and hus induce di e en obse able locali y. The ype o Bo n policy he e o e
has a s ong impac on ainabili y, pa icula ly as a unc ion o he en i onmen ’s complexi y (i.e., he
numbe o ac ions |𝐴|). To es ablish heo e ical bounds o he pa ial de i a i e a iance o gene al
classes o ci cui s, known esul s o local 2-design ci cui s a e le e aged [42], since i is known ha deep
124

6.2. BORN POLICIES
PQCs o m 2-designs [la occaTheo yO e pa ame iza ionQuan um2023.]
We begin wi h an analy ical uppe bound on he log-p obabili y’s pa ial-de i a i e a iance, s a ed in
Lemma 6.2.1. Th oughou his discussion, le 𝑓(𝜋𝜃)=log 𝜋(𝑎|𝑠, 𝜃) o simplici y.
Lemma 6.2.1.
Conside a gene al
𝑁
-qubi Bo n policy
𝜋(𝑎|𝑠, 𝜃)
(De ini ion 5.1.1) wi h
|𝐴|
ac ions.
Then an uppe bound o he a iance o he log policy g adien is
𝕍𝜃𝜕𝜃log 𝜋(𝑎|𝑠, 𝜃)≤2𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞h𝕍𝜃𝜕𝜃𝜋𝜃+𝔼𝜃𝜕𝜃𝜋𝜃2i.(6.16)
P oo .
𝕍𝜃𝜕𝜃log 𝜋(𝑎|𝑠, 𝜃)=𝕍𝜃𝜕𝜃𝑓(𝜋𝜃)
=𝕍𝜃𝜕𝜋𝜃𝑓(𝜋𝜃)𝜕𝜃𝜋𝜃(A)
≤2𝕍𝜃𝜕𝜃𝜋𝜃𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞+2𝔼𝜃𝜕𝜃𝜋𝜃2𝕍𝜃𝜕𝜋𝜃𝑓(𝜋𝜃)(B)
≤2𝕍𝜃𝜕𝜃𝜋𝜃𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞+2𝔼𝜃𝜕𝜃𝜋𝜃2𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞(C)
=2𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞h𝕍𝜃𝜕𝜃𝜋𝜃+𝔼𝜃𝜕𝜃𝜋𝜃2i,(D)
whe e (A) applies he chain ule, (B) uses a a iance-o -p oduc bound 𝕍[𝑋𝑌] ≤ 2𝕍[𝑋]|𝑌|2
∞+2𝔼[𝑋]2𝕍[𝑌][197],
(C) bounds he a iance o 𝜕𝜋𝜃𝑓(𝜋𝜃)by i s sup emum no m, and (D) collec s e ms. □
The uppe bound depends c ucially on 𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞, which in u n depends on he o al numbe o ac ions
|𝐴|and he obse able used o es ima e he policy. Assuming 1-design pa ame e blocks be o e and a e
he pa ame e 𝜃, one ob ains 𝔼𝜃𝜕𝜃𝜋𝜃=0[42].
In RL, he log policy g adien is compu ed only o
sampled
ac ions, so 𝜋(𝑎|𝑠,𝜃)canno be s ic ly ze o.
Howe e , i can be ex emely small. In p ac ice, a minimal clipping pa ame e 𝑏is o en adop ed such
ha 𝜋min ∈ [𝑏, 1]. I 𝑏is exponen ially small wi h espec o 𝑁, hen log 𝜋(𝑎|𝑠,𝜃)may lead o la ge
de i a i es (see Subsec ion 6.2.2). The ques ion becomes how small 𝑏can be while s ill ensu ing e ec i e
p obabili y es ima ion and a oiding exploding g adien s. As shown nex , he Bo n policy a ian and he
numbe o ac ions |𝐴|a e c i ical ac o s. Lemma 6.2.2 p o ides an uppe bound on he log-g adien
a iance o a
pa i y-like
Bo n policy.
Lemma 6.2.2.
(Va iance o pa i y-like Bo n policy) Le
𝜋(𝑎|𝑠, 𝜃)
be an
𝑁
-qubi pa i y-like Bo n
policy (De ini ion 5.1.5) wi h
|𝐴|
ac ions. I each block in he pa ame e ized quan um ci cui o ms a local
2-design, hen he policy g adien a iance anishes
exponen ially
wi h he numbe o qubi s,
𝕍𝜃𝜕𝜃log 𝜋(𝑎|𝑠, 𝜃)∈ O1
𝛼𝑛, 𝛼 >1,
p o ided
|𝐴| ∈ O(poly(𝑁))
. Con e sely, when
|𝐴|
exceeds polynomial g ow h in
𝑁
, he policy g adien
a iance scales as
𝕍𝜃𝜕𝜃log 𝜋(𝑎|𝑠, 𝜃)∈ O𝛽
𝛼𝑛, 𝛼, 𝛽 >1.
125
CHAPTER 6. TRAINABILITY ISSUES IN QUANTUM POLICY GRADIENTS
and he uppe bound can become loose. Pa icula ly, i
𝛽>𝛼
, implying he a iance
inc eases
wi h
𝑛
.
P oo .
A pa i y-like Bo n policy pa i ions 2𝑁basis s a es by measu ing all 𝑁qubi s (see De ini ion 5.1.5).
I |𝐴| ∈ O(poly(𝑁)), we can assume a minimum p obabili y 𝑏∈Ω1
poly(𝑁)[197], ensu ing ha each
ac ion p obabili y is a leas polynomially small. This is a easonable assump ion in RL o |𝐴|  2𝑁.
The o al numbe o ea u es in an RL agen ’s s a e, 𝑠𝑓, is ypically la ge and 𝑠𝑓 |𝐴| o a disc e e
ac ion space and se e al qubi s a e o en equi ed o encode he s a e o he agen . T adi ionally, s anda d
angle encoding schemes a e conside ed in mos li e a u e [93, 48, 175, 96]. The e o e, 𝑁∼𝑠𝑓, which
implies ha |𝐴|  2𝑁and alida es he poly(𝑁)clipping assump ion. Unde hese condi ions, he no m
𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞∈ O(poly(𝑁)), and i each ci cui block is a local 2-design, hen Va 𝜕𝜃𝜋𝜃∈ O1
𝛼𝑛 o
some 𝛼>1[42]. Thus he a iance anishes exponen ially in 𝑁, inducing a BP.
Ou side poly(𝑁)ac ions, 𝜋min can be exponen ially small, Ω1
𝛽𝑛 o 𝛽>1, causing 𝜕𝜋𝜃𝑓(𝜋𝜃)2
∞ o
g ow as O(𝛽𝑛). Consequen ly,
𝕍𝜃𝜕𝜃log 𝜋(𝑎|𝑠, 𝜃)∈ O𝛽
𝛼𝑛,
which inc eases wi h 𝑁i 𝛽>𝛼. In his egime, he policy g adien exhibi s exploding g adien s, bu
also equi es an exponen ially la ge numbe o sho s o es ima e p obabili ies, making he policy ha d o
ain. □
In con as , he base case |𝐴|=2unde a con iguous-like Bo n policy in ol es
single-qubi
measu emen s,
yielding a e y di e en ainabili y p o ile han he pa i y-like policy. In ui i ely, con iguous-like policies
become ha de o ain as |𝐴|g ows, because la ge |𝐴| ypically implies a mo e global measu emen .
Hence, a
ainabili y window
exis s should |𝐴| emain su icien ly small. In gene al, con iguous-like policy
employs up o log(|𝐴|)-local measu emen s (see De ini ion 5.1.2); o example, i |𝐴|=𝑁, hen a mos
log(𝑁)adjacen qubi s a e measu ed. Such log(𝑁)-local measu emen s a e known o a oid BPs unde
ce ain condi ions [162]. Lemma 6.2.3 p o ides a lowe bound on he a iance o he log-p obabili y
g adien o con iguous-like Bo n policies, as a unc ion o he numbe o ac ions |𝐴|.
Lemma 6.2.3.
(Va iance o Con iguous-like Bo n policy) Conside an
𝑁
-qubi con iguous-like
Bo n policy
𝜋(𝑎|𝑠, 𝜃)
wi h
|𝐴|
ac ions (De ini ion 5.1.1). I each ci cui block o ms a local 2-design, hen
𝕍𝜃𝜕𝜃𝜋(𝑎|𝑠, 𝜃)∈Ω1
poly(𝑛)
o
|𝐴| ∈ O(𝑁)
and ci cui dep h
O(log(𝑁))
. Con e sely, o
|𝐴| ∈ O(𝑁)
and dep h
O(poly(log(𝑁)))
,
he a iance scales as
𝕍𝜃𝜕𝜃𝜋(𝑎|𝑠, 𝜃)∈Ω2−poly(log(𝑛)).
The pa ial de i a i e o he log-p obabili y likewise emains bounded, since p obabili ies do no anish
exponen ially. Lemma 6.2.3 hus p o ides a lowe bound on he policy g adien a iance unde local 2-
design assump ions. As long as |𝐴| ∈ O(𝑁)and he ci cui dep h is a mos O(log(𝑁)), he a iance
126
6.2. BORN POLICIES
dec eases a wo s polynomially in 𝑁, and he equi ed quan um measu emen s emain polynomially
la ge. I he dep h ex ends o O(poly(log(𝑁))), he a iance decays as e han polynomially bu no
ully exponen ially, e lec ing pa ially global obse ables. A de ailed de i a ion is de e ed o Appendix A.
The nex sec ion examines an al e na i e iew o ainabili y by s udying he Fishe in o ma ion spec um.
6.2.4 Analysis o he Fishe in o ma ion spec um
In compu a ional lea ning heo y, he CFIM is used o assess how a ia ions in model pa ame e s a ec he
model’s ou pu . In RL, he CFIM mus accoun o s a es d awn om he policy-induced s a e dis ibu ion
𝑑𝜋
𝑠. Fo a pa ame e ized policy 𝜋(𝑎|𝑠, 𝜃), he ma ix is exp essed as he expec a ion o he ou e p oduc
o he log-likelihood g adien (see Sec ion 4.4.1):
I(𝜃)=𝔼𝑠∼𝑑𝜋
𝑠𝔼𝑎∼𝜋(·|𝑠,𝜃)h∇𝜃log 𝜋(𝑎|𝑠, 𝜃)∇𝜃log 𝜋(𝑎|𝑠, 𝜃)𝑇i.(6.17)
The CFIM indica es how pa ame e changes in luence he policy’s ou pu dis ibu ion. No ably, he CFIM’s
spec um undamen ally cha ac e izes BPs in PQC-based s a is ical models ained ia log-likelihood ob-
jec i es [5]. Al hough RL objec i es also inco po a e cumula i e ewa ds (which he CFIM does no di ec ly
cap u e), he CFIM spec um s ill helps o iden i y BP signa u es—p o ided ewa ds a e non-ze o.
In a BP, he CFIM eigen alues concen a e exponen ially nea ze o wi h he numbe o qubi s 𝑁[5]. The
expec ed alue o a diagonal en y 𝑘in he CFIM can be w i en as
𝔼𝜃I𝑘𝑘 (𝜃)=𝔼𝜃h𝜕𝜃𝑘log 𝜋(𝑎|𝑠,𝜃)2i
=𝕍𝜃h𝜕𝜃𝑘log 𝜋(𝑎|𝑠, 𝜃)i+𝔼𝜃𝜕𝜃𝑘log 𝜋(𝑎|𝑠, 𝜃)2
,(6.18)
which ollows om he de ini ion o a iance. Hence, each diagonal componen is bounded below by he
a iance o he log-likelihood g adien :
𝔼𝜃I𝑘𝑘 (𝜃)≥𝕍𝜃h𝜕𝜃𝑘log 𝜋(𝑎|𝑠,𝜃)i.(6.19)
bu i can also be assumed 1-design pa ame e ized blocks o ensu e 𝔼𝜃𝜕𝜃𝑘log 𝜋(𝑎|𝑠, 𝜃)2=0. Sum-
ming o e all pa ame e s 𝜃∈ℝ𝐾 hen implies
𝔼𝜃T I(𝜃)≥
𝐾−1
Õ
𝑘=0
𝕍𝜃h𝜕𝜃𝑘log 𝜋(𝑎|𝑠, 𝜃)i.(6.20)
Thus, any lowe bound on he pa ial-de i a i e a iance (e.g. om Lemma 6.2.3) ansla es in o a lowe
bound on he CFIM ace. In a BP each CFIM diagonal en y anishes exponen ially wi h 𝑁, also equi ing
an exponen ial numbe o measu emen sho s o es ima e i accu a ely.
By Lemma 6.2.3, a Con iguous-like Bo n policy wi h |𝐴| ∈ O(poly(𝑁)) has pa ial de i a i es whose
a iance decays a mos
polyloga i hmically
in 𝑁. Consequen ly, he CFIM eigen alues do no all anish
exponen ially, and he CFIM spec um does
no
e eal a BP.
127
CHAPTER 6. TRAINABILITY ISSUES IN QUANTUM POLICY GRADIENTS
Fo a Pa i y-like Bo n policy wi h |𝐴| ∈ O(poly(𝑁)), he a iance o he log-likelihood g adien sh inks
exponen ially wi h 𝑁. In u n, he CFIM eigen alues also collapse exponen ially, signaling a BP.
In scena ios whe e he numbe o ac ions exceeds Poly(𝑁), no only do he equi ed measu emen s o
accu a e policy es ima ion become p ohibi i ely la ge, bu he p obabili ies associa ed wi h ac ions emain
exponen ially small. This implies ha , despi e a oiding BPs, hese scena ios a e mo e likely o encoun e
exploding g adien s a he han BPs, e lec ed in inc easing CFIM en ies and a less concen a ed spec um
a ound ze o. Hence, a non- anishing CFIM spec um in his la ge-ac ion egime does
no
necessa ily imply
good ainabili y. Indeed, while he spec um is less concen a ed nea ze o, es ima ing he policy (and
hus he g adien ) demands exponen ially mo e measu emen s.
In summa y, he CFIM spec um can e ec i ely cha ac e ize BPs o PQC-based policies when |𝐴| ∈
O(poly(𝑁)). Ou side ha ange, he CFIM spec um ends o be la ge (i.e. no concen a ed nea ze o)
bu does no gua an ee s aigh o wa d ainabili y, because exponen ially many measu emen s a e o en
equi ed. The nex sec ion del es u he in o hese ainabili y issues by examining nume ical expe imen s.
6.2.5 Nume ical expe imen s
This subsec ion empi ically in es iga es he ainabili y issues o Con iguous and Pa i y-like Bo n policies
(as discussed in Lemma 6.2.3). Two p ima y asks a e conside ed:
•
T ainabili y wi h a simpli ied 2-design:
Empi ical alida ion o heo e ical esul s on he a iance o
he log-likelihood g adien o Con iguous and Pa i y-like Bo n policies, p esen ed in Lemmas 6.2.3
and 6.2.2. We conside a “simpli ied wo-design” ansa z [42], see Figu e 55(a) o explo e how he
a iance o he log-likelihood g adien and he CFIM spec um a y wi h bo h he policy ype and he
numbe o ac ions |𝐴|.
•
Mul i-a med bandi s:
A syn he ic mul i-a med bandi en i onmen is in oduced o compa e how
hese Bo n policies (Con iguous o Pa i y-like) dis inguish he bes a m h ough sampling and
g adien -based upda es as a unc ion o he numbe o ac ions |𝐴|and ha ing only access o a
polynomial numbe o measu emen s.
In he i s ask, al hough he selec ed ansa z does no p ecisely o m a wo-design, i is known o exhibi
cos - unc ion BPs [42], making i well-sui ed o simula ion a la ge qubi coun s and dep hs. We choose
a dep h o O(𝑁2)in ou expe imen s. Since la ge ac ion-space RL benchma ks o PQC-based policies
a e sca ce, we adop he mul i-a med bandi en i onmen in he second ask. This choice allows us o
keep a consis en objec i e unc ion while scaling he numbe o ac ions and he qubi coun , he eby
ocusing on ainabili y. All simula ions use Pennylane’s quan um simula o [21], wi h pa ame e -shi
g adien es ima ion [172] and a polynomial numbe o measu emen s O(poly(𝑁)). Ou code is a ailable
on Gi Hub a T ainabili y-issues-in-QPGs.
128
BIBLIOGRAPHY
[59] M. de Oli ei a and L. S. Ba bosa.
Quan um Bayesian Decision-Making*
. 2020-10. doi: 10.4855
0/a Xi .2010.02088. a Xi : 2010.02088 [quan -ph]. (Visi ed on 2024-06-04) (ci . on
p. 206).
[60] F. Di Ma can onio e al. “Quan um Ad an age Seeke wi h Ke nels (QuASK): A So wa e F amewo k
o Speed up he Resea ch in Quan um Machine Lea ning”. In:
Quan um Machine In elligence
5.1
(2023-05), p. 20. issn: 2524-4914. doi: 10.1007/s42484-023-00107-2. (Visi ed on 2024-
04-04) (ci . on p. 35).
[61] C. Dimi akakis and R. O ne .
Decision Making Unde Unce ain y and Rein o cemen Lea ning:
Theo y and Algo i hms
. Vol. 223. In elligen Sys ems Re e ence Lib a y. Cham: Sp inge In e na-
ional Publishing, 2022. isbn: 978-3-031-07612-1 978-3-031-07614-5. doi: 10.1007/978-3-0
31-07614-5. (Visi ed on 2024-05-31) (ci . on p. 206).
[62] D. Dong e al. “Quan um Rein o cemen Lea ning”. In:
IEEE T ansac ions on Sys ems, Man, and
Cybe ne ics, Pa B (Cybe ne ics)
38.5 (2008-10), pp. 1207–1220. issn: 1083-4419. doi: 10.11
09/TSMCB.2008.925743. a Xi : 0810.3828 [quan -ph]. (Visi ed on 2024-06-07) (ci . on
p. 5).
[63] Y. Du e al. “Exp essi e Powe o Pa ame ized Quan um Ci cui s”. In:
Physical Re iew Resea ch
2.3 (2020-07), p. 033125. issn: 2643-1564. doi: 10.1103/PhysRe Resea ch.2.033125.
(Visi ed on 2024-03-26) (ci . on pp. 3, 25).
[64] V. Dunjko and H. J. B iegel. “Machine Lea ning & A i icial In elligence in he Quan um Domain:
A Re iew o Recen P og ess”. In:
Repo s on P og ess in Physics
81.7 (2018), p. 074001. (Visi ed
on 2024-06-07) (ci . on pp. 1, 2, 5).
[65] V. Dunjko, J. M. Taylo , and H. J. B iegel. “Quan um-Enhanced Machine Lea ning”. In:
Physi-
cal Re iew Le e s
117.13 (2016-09), p. 130501. issn: 0031-9007, 1079-7114. doi: 10 .1103
/PhysRe Le .117.130501. a Xi : 1610.08251 [quan -ph]. (Visi ed on 2024-06-05)
(ci . on pp. 2, 5, 213, 222).
[66] E. Fa hi, J. Golds one, and S. Gu mann.
A Quan um App oxima e Op imiza ion Algo i hm
. 2014-11.
doi: 10.48550/a Xi .1411.4028. a Xi : 1411.4028 [quan -ph]. (Visi ed on 2024-03-
28) (ci . on pp. 3, 21, 26, 120, 222).
[67] E. Fa hi and A. W. Ha ow.
Quan um Sup emacy h ough he Quan um App oxima e Op imiza ion
Algo i hm
. 2019-10. doi: 10.48550/a Xi .1602.07674. a Xi : 1602.07674 [quan -ph].
(Visi ed on 2024-03-28) (ci . on pp. 3, 21, 222).
[68] E. Fa hi and A. W. Ha ow.
Quan um Sup emacy h ough he Quan um App oxima e Op imiza ion
Algo i hm
. 2019-10. a Xi : 1602.07674. (Visi ed on 2024-11-05) (ci . on p. 166).
[69] E. Fa hi and H. Ne en.
Classi ica ion wi h Quan um Neu al Ne wo ks on Nea Te m P ocesso s
.
2018-08. doi: 10.48550/a Xi .1802.06002. a Xi : 1802.06002 [quan -ph]. (Visi ed
on 2024-03-28) (ci . on p. 28).
231

BIBLIOGRAPHY
[70] A. Fawzi e al. “Disco e ing Fas e Ma ix Mul iplica ion Algo i hms wi h Rein o cemen Lea ning”.
In:
Na u e
610.7930 (2022-10), pp. 47–53. issn: 1476-4687. doi: 10.1038/s41586-022-05
172-4. (Visi ed on 2024-04-22) (ci . on p. 44).
[71] R. P. Feynman. “Simula ing Physics wi h Compu e s”. In:
In e na ional Jou nal o Theo e ical
Physics
21.6 (1982-06), pp. 467–488. issn: 1572-9575. doi: 10.1007/BF02650179. (Visi ed
on 2024-06-11) (ci . on p. 1).
[72] T. Fösel e al.
Quan um Ci cui Op imiza ion wi h Deep Rein o cemen Lea ning
. 2021-03. doi:
10.48550/a Xi .2103.07585. a Xi : 2103.07585 [quan -ph]. (Visi ed on 2024-04-
22) (ci . on p. 44).
[73] T. Fösel e al. “Rein o cemen Lea ning wi h Neu al Ne wo ks o Quan um Feedback”. In:
Physical
Re iew X
8.3 (2018-09), p. 031084. doi: 10.1103/PhysRe X.8.031084. (Visi ed on 2024-
04-22) (ci . on p. 44).
[74] K. Fujii and T. Mo imae. “Quan um Commu ing Ci cui s and Complexi y o Ising Pa i ion Func-
ions”. In:
New Jou nal o Physics
19.3 (2017-03), p. 033003. issn: 1367-2630. doi: 10.108
8/1367-2630/aa5 db. a Xi : 1311.2128 [quan -ph]. (Visi ed on 2025-01-06) (ci . on
p. 163).
[75] J. Gacon e al. “Simul aneous Pe u ba ion S ochas ic App oxima ion o he Quan um Fishe In-
o ma ion”. In:
Quan um
5 (2021-10), p. 567. issn: 2521-327X. doi: 10.22331/q-2021-10-
20-567. a Xi : 2103.09232 [quan -ph]. (Visi ed on 2024-04-04) (ci . on p. 38).
[76] B. T. Ga d e al. “E icien Symme y-P ese ing S a e P epa a ion Ci cui s o he Va ia ional
Quan um Eigensol e Algo i hm”. In:
npj Quan um In o ma ion
6.1 (2020-01), pp. 1–9. issn:
2056-6387. doi: 10.1038/s41534-019-0240-1. (Visi ed on 2024-04-04) (ci . on p. 35).
[77] I. Good ellow, Y. Bengio, and A. Cou ille.
Deep Lea ning
. Adap i e Compu a ion and Machine
Lea ning. Camb idge, Mass: The MIT p ess, 2016. isbn: 978-0-262-03561-3 (ci . on pp. 27, 36,
55, 100).
[78] E. G an e al. “An Ini ializa ion S a egy o Add essing Ba en Pla eaus in Pa ame ized Quan um
Ci cui s”. In:
Quan um
3 (2019-12), p. 214. doi: 10.22331/q-2019-12-09-214. (Visi ed on
2024-04-12) (ci . on p. 43).
[79] A. G e on e al.
A Ke nel Me hod o he Two-Sample P oblem
. 2008-05. doi: 10.48550/a Xi .
0805.2368. a Xi : 0805.2368 [cs]. (Visi ed on 2024-05-24) (ci . on p. 142).
[80] L. K. G o e .
A Fas Quan um Mechanical Algo i hm o Da abase Sea ch
. 1996-11. doi: 10.4
8550/a Xi .quan -ph/9605043. a Xi : quan -ph/9605043. (Visi ed on 2024-06-04)
(ci . on pp. 1, 211).
[81] A. Hannun.
An In oduc ion o Fishe In o ma ion
. h ps://awni.gi hub.io/in o- ishe -in o ma ion/.
(Visi ed on 2024-04-03) (ci . on p. 22).
232
BIBLIOGRAPHY
[82] A. Ha ow and J. Napp. “Low-Dep h G adien Measu emen s Can Imp o e Con e gence in Va i-
a ional Hyb id Quan um-Classical Algo i hms”. In:
Physical Re iew Le e s
126.14 (2021-04),
p. 140502. issn: 0031-9007, 1079-7114. doi: 10.1103/PhysRe Le .126.140502. a Xi :
1901.05374 [quan -ph]. (Visi ed on 2024-04-08) (ci . on p. 38).
[83] A. W. Ha ow, A. Hassidim, and S. Lloyd. “Quan um Algo i hm o Sol ing Linea Sys ems o
Equa ions”. In:
Physical Re iew Le e s
103.15 (2009-10), p. 150502. issn: 0031-9007, 1079-
7114. doi: 10.1103/PhysRe Le .103.150502. a Xi : 0811.3171 [quan -ph]. (Visi ed
on 2024-06-11) (ci . on p. 1).
[84] T. Haug, K. Bha i, and M. S. Kim. “Capaci y and Quan um Geome y o Pa ame ized Quan-
um Ci cui s”. In:
PRX Quan um
2.4 (2021-10), p. 040309. issn: 2691-3399. doi: 10 . 1103
/PRXQuan um.2.040309. a Xi : 2102.01659 [quan -ph, s a ]. (Visi ed on 2024-04-
03) (ci . on p. 34).
[85] T. Haug and M. S. Kim.
Gene aliza ion wi h Quan um Geome y o Lea ning Uni a ies
. 2023-03.
a Xi : 2303.13462 [quan -ph, s a ]. (Visi ed on 2023-03-27) (ci . on p. 147).
[86] T. Haug and M. S. Kim.
Op imal T aining o Va ia ional Quan um Algo i hms wi hou Ba en
Pla eaus
. 2021-06. a Xi : 2104 . 14543 [quan -ph, s a ]. (Visi ed on 2023-09-11) (ci .
on pp. 149, 156).
[87] P. Hayden, D. W. Leung, and A. Win e . “Aspec s o Gene ic En anglemen ”. In:
Communica ions
in Ma hema ical Physics
265.1 (2006-07), pp. 95–117. issn: 0010-3616, 1432-0916. doi: 10.1
007/s00220-006-1535-6. a Xi : quan -ph/0407049. (Visi ed on 2024-03-26) (ci . on
p. 20).
[88] C. W. Hels om. “Quan um De ec ion and Es ima ion Theo y”. In:
Jou nal o S a is ical Physics
1.2 (1969-06), pp. 231–252. issn: 1572-9613. doi: 10.1007/BF01007479. (Visi ed on 2024-
04-02) (ci . on p. 21).
[89] Z. Holmes e al. “Connec ing Ansa z Exp essibili y o G adien Magni udes and Ba en Pla eaus”.
In:
PRX Quan um
3.1 (2022-01), p. 010313. issn: 2691-3399. doi: 10.1103/PRXQuan um.3
.010313. a Xi : 2101.02138 [quan -ph, s a ]. (Visi ed on 2024-03-26) (ci . on pp. 20,
195, 197).
[90] K. Ho nik. “App oxima ion Capabili ies o Mul ilaye Feed o wa d Ne wo ks”. In:
Neu al Ne wo ks
4.2 (1991-01), pp. 251–257. issn: 0893-6080. doi: 10.1016/0893-6080(91)90009-T.
(Visi ed on 2024-03-29) (ci . on p. 31).
[91] J.-Y. Hsiao e al.
Unen angled Quan um Rein o cemen Lea ning Agen s in he OpenAI Gym
. 2022-
03. doi: 10.48550/a Xi .2203.14348. a Xi : 2203.14348 [cond-ma , physics:quan -ph].
(Visi ed on 2024-06-11) (ci . on pp. 2, 6).
233
BIBLIOGRAPHY
[92] M. Incudini e al.
Au oma ic and E ec i e Disco e y o Quan um Ke nels
. 2023-12. doi: 10.485
50/a Xi .2209.11144. a Xi : 2209.11144 [quan -ph]. (Visi ed on 2024-04-04) (ci . on
p. 35).
[93] S. Je bi e al. “Pa ame ized Quan um Policies o Rein o cemen Lea ning”. In:
Ad ances in Neu-
al In o ma ion P ocessing Sys ems
. Vol. 34. Cu an Associa es, Inc., 2021, pp. 28362–28375.
(Visi ed on 2023-03-20) (ci . on pp. 2, 6, 33, 96, 104, 113, 126, 152, 153, 155, 193).
[94] S. Je bi e al. “Quan um Enhancemen s o Deep Rein o cemen Lea ning in La ge Spaces”. In:
PRX Quan um
2.1 (2021-02), p. 010328. issn: 2691-3399. doi: 10.1103/PRXQuan um.2.0
10328. (Visi ed on 2024-04-26) (ci . on pp. 6, 51).
[95] S. Je bi e al. “Quan um Machine Lea ning beyond Ke nel Me hods”. In:
Na u e Communica ions
14.1 (2023-01), p. 517. issn: 2041-1723. doi: 10 . 1038 / s41467 - 023 - 36159 - y. a Xi :
2110.13162 [quan -ph, s a ]. (Visi ed on 2024-03-28) (ci . on pp. 31, 32).
[96] S. Je bi e al.
Quan um Policy G adien Algo i hms
. 2022-12. doi: 10.4230/LIPIcs.TQC.20
23.13. a Xi : 2212.09328 [quan -ph, s a ]. (Visi ed on 2023-10-23) (ci . on pp. 6, 126,
213).
[97] S. Je bi e al.
Shadows o Quan um Machine Lea ning
. 2023-05. a Xi : 2306.00061 [quan -ph,
s a ]. (Visi ed on 2024-03-28) (ci . on p. 115).
[98] S. Joh i e al.
Nea es Cen oid Classi ica ion on a T apped Ion Quan um Compu e
. 2020-12.
a Xi : 2012.04145 [quan -ph]. (Visi ed on 2024-03-28) (ci . on p. 30).
[99] R. Jozsa and A. Miyake.
Ma chga es and Classical Simula ion o Quan um Ci cui s
. 2008-11. doi:
10.48550/a Xi .0804.4050. a Xi : 0804.4050. (Visi ed on 2024-11-04) (ci . on pp. 26,
162).
[100] S. M. Kakade. “A Na u al Policy G adien ”. In:
Ad ances in Neu al In o ma ion P ocessing Sys-
ems
. Vol. 14. MIT P ess, 2001. (Visi ed on 2023-09-08) (ci . on pp. 3, 61, 64, 102, 144, 146).
[101] S. M. Kakade. “On he Sample Complexi y o Rein o cemen Lea ning”. PhD hesis. 2003 (ci . on
pp. 63, 64).
[102] S. Kazi e al.
Analyzing he Quan um App oxima e Op imiza ion Algo i hm: Ansä ze, Symme ies,
and Lie Algeb as
. 2024-10. doi: 10 . 48550 / a Xi . 2410 . 05187. a Xi : 2410 . 05187.
(Visi ed on 2024-11-10) (ci . on p. 166).
[103] I. Ke enidis, J. Landman, and N. Ma hu .
Classical and Quan um Algo i hms o O hogonal Neu al
Ne wo ks
. 2022-12. doi: 10.48550/a Xi .2106.07198. a Xi : 2106.07198 [quan -ph].
(Visi ed on 2024-03-28) (ci . on p. 30).
[104] S. Kim e al. “Deepmellow: Remo ing he Need o a Ta ge Ne wo k in Deep Q-lea ning”. In:
P oceedings o he 28 h In e na ional Join Con e ence on A i icial In elligence
. IJCAI’19. Macao,
China: AAAI P ess, 2019-08, pp. 2733–2739. isbn: 978-0-9992411-4-1. (Visi ed on 2024-05-29)
(ci . on p. 205).
234
BIBLIOGRAPHY
[105] D. P. Kingma and J. Ba.
Adam: A Me hod o S ochas ic Op imiza ion
. 2017-01. doi: 10.48550
/a Xi .1412.6980. a Xi : 1412.6980 [cs]. (Visi ed on 2024-04-05) (ci . on p. 35).
[106] M. J. Kochende e and T. A. Wheele .
Algo i hms o Op imiza ion
. Camb idge: The MIT p ess,
2019. isbn: 978-0-262-03942-0 (ci . on p. 35).
[107] V. Konda and J. Tsi siklis. “Ac o -C i ic Algo i hms”. In:
Ad ances in Neu al In o ma ion P ocessing
Sys ems
. Vol. 12. MIT P ess, 1999. (Visi ed on 2024-05-03) (ci . on p. 60).
[108] G. K use e al. “Va ia ional Quan um Ci cui Design o Quan um Rein o cemen Lea ning on
Con inuous En i onmen s”. In:
P oceedings o he 16 h In e na ional Con e ence on Agen s and
A i icial In elligence
. 2024, pp. 393–400. doi: 10.5220/0012353100003636. a Xi : 2312
.13798 [quan -ph]. (Visi ed on 2024-06-10) (ci . on p. 114).
[109] J. M. Küble e al. “An Adap i e Op imize o Measu emen -F ugal Va ia ional Algo i hms”. In:
Quan um
4 (2020-05), p. 263. doi: 10.22331/q-2020-05-11-263. (Visi ed on 2024-04-04)
(ci . on p. 35).
[110] M. La occa e al. “Diagnosing Ba en Pla eaus wi h Tools om Quan um Op imal Con ol”. In:
Quan um
6 (2022-09), p. 824. issn: 2521-327X. doi: 10.22331/q-2022-09-29-824. a Xi :
2105.14377 [quan -ph]. (Visi ed on 2022-09-30) (ci . on pp. 113, 140).
[111] M. La occa e al. “Theo y o O e pa ame iza ion in Quan um Neu al Ne wo ks”. In:
Na u e Com-
pu a ional Science
3.6 (2023-06), pp. 542–551. issn: 2662-8457. doi: 10.1038/s43588-02
3-00467-6. (Visi ed on 2024-03-27) (ci . on pp. 35, 37, 43, 156, 157).
[112] R. LaRose and B. Coyle. “Robus Da a Encodings o Quan um Classi ie s”. In:
Physical Re iew A
102.3 (2020-09), p. 032420. issn: 2469-9926, 2469-9934. doi: 10.1103/PhysRe A.102.0
32420. a Xi : 2003.01695 [quan -ph]. (Visi ed on 2024-03-28) (ci . on p. 28).
[113] E. L. Lehmann and G. Casella.
Theo y o Poin Es ima ion
. 2nd ed. Sp inge Tex s in S a is ics.
New Yo k: Sp inge , 1998. isbn: 978-0-387-98502-2 (ci . on pp. 22, 23).
[114] L. Leone e al.
On he P ac ical Use ulness o he Ha dwa e E icien Ansa z
. 2022-11. doi: 10
.48550/a Xi .2211.01477. a Xi : 2211.01477 [quan -ph]. (Visi ed on 2023-05-11)
(ci . on p. 41).
[115] A. Le che , S. Woe ne , and C. Zou al. “Tigh and E icien G adien Bounds o Pa ame e ized
Quan um Ci cui s”. In:
Quan um
8 (2024-09), p. 1484. doi: 10.22331/q-2024-09-25-148
4. (Visi ed on 2025-01-02) (ci . on pp. 42, 140, 167, 169).
[116] T. Liang e al.
Fishe -Rao Me ic, Geome y, and Complexi y o Neu al Ne wo ks
. 2019-02. doi:
10.48550/a Xi .1711.01530. a Xi : 1711.01530 [cs, s a ]. (Visi ed on 2024-04-
08) (ci . on p. 38).
235
BIBLIOGRAPHY
[117] J. Liu e al. “Quan um Fishe In o ma ion Ma ix and Mul ipa ame e Es ima ion”. In:
Jou nal
o Physics A: Ma hema ical and Theo e ical
53.2 (2020-01), p. 023001. issn: 1751-8113, 1751-
8121. doi: 10.1088/1751-8121/ab5d4d. a Xi : 1907.08037 [quan -ph]. (Visi ed on
2024-04-02) (ci . on p. 22).
[118] S. Lloyd, M. Mohseni, and P. Reben os . “Quan um P incipal Componen Analysis”. In:
Na u e
Physics
10.9 (2014-09), pp. 631–633. issn: 1745-2481. doi: 10.1038/nphys3029. (Visi ed on
2024-06-11) (ci . on p. 1).
[119] O. Lockwood and M. Si.
Rein o cemen Lea ning wi h Quan um Va ia ional Ci cui s
. 2020-08. doi:
10.48550/a Xi .2008.07524. a Xi : 2008.07524 [quan -ph, s a ]. (Visi ed on
2024-06-07) (ci . on p. 5).
[120] J. M. Lou enço.
The NOVA hesis L
A
T
EX Templa e Use ’s Manual
. NOVA Uni e si y Lisbon. 2021. u l:
h ps://gi hub.com/joaomlou enco/no a hesis/ aw/main/ empla e.pd
(ci . on p. ii).
[121] G. H. Low, T. J. Yode , and I. L. Chuang. “Quan um In e ence on Bayesian Ne wo ks”. In:
Physical
Re iew A
89.6 (2014-06), p. 062315. issn: 1050-2947, 1094-1622. doi: 10.1103/PhysRe A.8
9.062315. a Xi : 1402.7359 [quan -ph]. (Visi ed on 2024-05-31) (ci . on pp. 206, 208,
210, 211, 224).
[122] R. Manen i and M. Mo a.
Quan um In o ma ion Science
. Ox o d, New Yo k: Ox o d Uni e si y
P ess, 2023-11. isbn: 978-0-19-878748-8 (ci . on pp. 15, 28).
[123] D. J. Mankowi z e al. “Fas e So ing Algo i hms Disco e ed Using Deep Rein o cemen Lea ning”.
In:
Na u e
618.7964 (2023-06), pp. 257–263. issn: 1476-4687. doi: 10.1038/s41586-023-
06004-9. (Visi ed on 2024-04-22) (ci . on p. 44).
[124] C. O. Ma e o, M. Kie e o á, and N. Wiebe.
En anglemen Induced Ba en Pla eaus
. 2021-03. doi:
10.48550/a Xi .2010.15968. a Xi : 2010.15968 [quan -ph]. (Visi ed on 2023-05-11)
(ci . on p. 39).
[125] S. C. Ma shall, S. Aa onson, and V. Dunjko.
Imp o ed Sepa a ion be ween Quan um and Classical
Compu e s o Sampling and Func ional Tasks
. 2024-10. a Xi : 2410.20935. (Visi ed on 2024-
11-05) (ci . on pp. 25, 159, 160).
[126] G. B. Mbeng, R. Fazio, and G. San o o.
Quan um Annealing: A Jou ney h ough Digi aliza ion,
Con ol, and Hyb id Quan um Va ia ional Schemes
. 2019-12. a Xi : 1906.08948 [quan -ph].
(Visi ed on 2024-04-04) (ci . on p. 35).
[127] J. R. McClean e al. “Ba en Pla eaus in Quan um Neu al Ne wo k T aining Landscapes”. In:
Na u e Communica ions
9.1 (2018-11), p. 4812. issn: 2041-1723. doi: 10.1038/s41467-01
8-07090-4. (Visi ed on 2024-04-09) (ci . on pp. 3, 39, 40, 117).
[128] E. Meckes.
The Random Ma ix Theo y o he Classical Compac G oups
. 2019-08. isbn: 978-1-
108-41952-9. doi: 10.1017/9781108303453 (ci . on pp. 19, 20).
236

BIBLIOGRAPHY
[129] D. A. Meye and N. R. Wallach. “Global En anglemen in Mul ipa icle Sys ems”. In:
Jou nal o
Ma hema ical Physics
43.9 (2002-09), pp. 4273–4278. issn: 0022-2488, 1089-7658. doi: 10.1
063/1.1497700. a Xi : quan -ph/0108104. (Visi ed on 2024-03-23) (ci . on p. 15).
[130] J. J. Meye . “Fishe In o ma ion in Noisy In e media e-Scale Quan um Applica ions”. In:
Quan um
5 (2021-09), p. 539. doi: 10.22331/q-2021-09-09-539. (Visi ed on 2023-03-27) (ci . on
pp. 22, 23, 143–145, 147, 155).
[131] N. Meye e al.
Quan um Na u al Policy G adien s: Towa ds Sample-E icien Rein o cemen Lea n-
ing
. 2023-08. doi: 10.48550/a Xi .2304.13571. a Xi : 2304.13571 [quan -ph].
(Visi ed on 2023-10-23) (ci . on pp. 3, 6, 223).
[132] N. Meye e al.
Quan um Policy G adien Algo i hm wi h Op imized Ac ion Decoding
. 2023-05. doi:
10.48550/a Xi .2212.06663. a Xi : 2212.06663 [quan -ph]. (Visi ed on 2023-05-29)
(ci . on pp. 6, 77, 78).
[133] F. Mezzad i.
How o Gene a e Random Ma ices om he Classical Compac G oups
. 2007-02. doi:
10.48550/a Xi .ma h-ph/0609050. a Xi : ma h-ph/0609050. (Visi ed on 2024-03-26)
(ci . on p. 19).
[134] K. Mi a ai e al. “Quan um Ci cui Lea ning”. In:
Physical Re iew A
98.3 (2018-09), p. 032309.
issn: 2469-9926, 2469-9934. doi: 10.1103/PhysRe A.98.032309. a Xi : 1803.00745
[quan -ph]. (Visi ed on 2024-04-05) (ci . on pp. 36, 141).
[135] V. Mnih e al.
Asynch onous Me hods o Deep Rein o cemen Lea ning
. 2016-06. doi: 10.4855
0/a Xi .1602.01783. a Xi : 1602.01783 [cs]. (Visi ed on 2024-05-03) (ci . on p. 60).
[136] V. Mnih e al. “Human-Le el Con ol h ough Deep Rein o cemen Lea ning”. In:
Na u e
518.7540
(2015-02), pp. 529–533. issn: 1476-4687. doi: 10.1038/na u e14236. (Visi ed on 2024-04-
30) (ci . on pp. 55, 56, 181, 195).
[137] M. Moh i, A. Ros amizadeh, and A. Talwalka .
Founda ions o Machine Lea ning
. Second edi ion.
Adap i e Compu a ion and Machine Lea ning. Camb idge, Massachuse s: The MIT P ess, 2018.
isbn: 978-0-262-03940-6 (ci . on p. 33).
[138] A. Mon ana o and L. Zhou.
Quan um Speedups in Sol ing Nea -Symme ic Op imiza ion P oblems
by Low-Dep h QAOA
. 2024-11. doi: 10.48550/a Xi .2411.04979. a Xi : 2411.04979
[quan -ph]. (Visi ed on 2024-12-30) (ci . on p. 26).
[139] M. Mo ales and C. Isbell.
G okking Deep Rein o cemen Lea ning
. Shel e Island: Manning, 2020.
isbn: 978-1-61729-545-4 (ci . on pp. 55, 62).
[140] T. Mülle e al.
Towa ds Mul i-Agen Rein o cemen Lea ning Using Quan um Bol zmann Machines
.
2021-11. doi: 10.48550/a Xi .2109.10900. a Xi : 2109.10900 [cs]. (Visi ed on 2024-
06-10) (ci . on p. 6).
237
BIBLIOGRAPHY
[141] Y. Naka a and M. Mu ao. “Diagonal-Uni a y 2-Designs and Thei Implemen a ions by Quan um Ci -
cui s”. In:
In e na ional Jou nal o Quan um In o ma ion
11.07 (2013-10), p. 1350062. issn: 0219-
7499, 1793-6918. doi: 10.1142/S0219749913500627. a Xi : 1206.4451 [quan -ph].
(Visi ed on 2025-01-07) (ci . on p. 167).
[142] M. V. den Nes .
Classical Simula ion o Quan um Compu a ion, he Go esman-Knill Theo em,
and Sligh ly Beyond
. 2009-10. doi: 10 . 48550 / a Xi . 0811 . 0898. a Xi : 0811 . 0898
[quan -ph]. (Visi ed on 2024-12-30) (ci . on p. 24).
[143] B. Neyshabu , R. R. Salakhu dino , and N. S eb o. “Pa h-SGD: Pa h-No malized Op imiza ion in
Deep Neu al Ne wo ks”. In:
Ad ances in Neu al In o ma ion P ocessing Sys ems
. Vol. 28. Cu an
Associa es, Inc., 2015. (Visi ed on 2024-04-08) (ci . on p. 37).
[144] M. A. Nielsen and I. L. Chuang.
Quan um Compu a ion and Quan um In o ma ion: 10 h An-
ni e sa y Edi ion
. 10 h. USA: Camb idge Uni e si y P ess, 2011. isbn: 978-1-107-00217-3 (ci .
on pp. 13, 14, 24).
[145] M. Y. Niu e al. “Uni e sal Quan um Con ol h ough Deep Rein o cemen Lea ning”. In:
npj Quan-
um In o ma ion
5.1 (2019-04), pp. 1–8. issn: 2056-6387. doi: 10.1038/s41534-019-0141
-3. (Visi ed on 2023-10-24) (ci . on pp. 97, 143).
[146] R. O ús. “Tenso Ne wo ks o Complex Quan um Sys ems”. In:
Na u e Re iews Physics
1.9 (2019-
09), pp. 538–550. issn: 2522-5820. doi: 10 . 1038 / s42254 - 019 - 0086 - 7. (Visi ed on
2024-04-05) (ci . on p. 36).
[147] G. D. Papa o e al. “Quan um Speedup o Ac i e Lea ning Agen s”. In:
Physical Re iew X
4.3
(2014-07), p. 031002. issn: 2160-3308. doi: 10.1103/PhysRe X.4.031002. (Visi ed on
2024-06-07) (ci . on p. 5).
[148] A. Paszke e al.
PyTo ch: An Impe a i e S yle, High-Pe o mance Deep Lea ning Lib a y
. 2019-
12. doi: 10.48550/a Xi .1912.01703. a Xi : 1912.01703 [cs, s a ]. (Visi ed on
2024-04-05) (ci . on p. 114).
[149] A. Pellow-Ja man e al. “A Compa ison o Va ious Classical Op imize s o a Va ia ional Quan um
Linea Sol e ”. In:
Quan um In o ma ion P ocessing
20.6 (2021-06), p. 202. issn: 1570-0755,
1573-1332. doi: 10.1007/s11128- 021- 03140- x. a Xi : 2106.08682 [quan -ph].
(Visi ed on 2024-04-04) (ci . on p. 35).
[150] A. Pé ez-Salinas e al.
Da a Re-Uploading o a Uni e sal Quan um Classi ie
. 2020-06. doi: 10.4
8550/a Xi .1907.02085. a Xi : 1907.02085. (Visi ed on 2024-11-12) (ci . on p. 165).
[151] A. Pé ez-Salinas e al. “Da a Re-Uploading o a Uni e sal Quan um Classi ie ”. In:
Quan um
4
(2020-02), p. 226. issn: 2521-327X. doi: 10.22331/q-2020-02-06-226. a Xi : 1907.02
085 [quan -ph]. (Visi ed on 2024-03-28) (ci . on p. 30).
238
BIBLIOGRAPHY
[152] A. Pé ez-Salinas e al. “One Qubi as a Uni e sal App oximan ”. In:
Physical Re iew A
104.1 (2021-
07), p. 012405. doi: 10.1103/PhysRe A.104.012405. (Visi ed on 2024-04-01) (ci . on
pp. 33, 104, 113, 192, 198).
[153] A. Pe uzzo e al. “A Va ia ional Eigen alue Sol e on a Pho onic Quan um P ocesso ”. In:
Na u e
Communica ions
5.1 (2014-07), p. 4213. issn: 2041-1723. doi: 10.1038/ncomms5213. (Visi ed
on 2024-03-27) (ci . on p. 21).
[154] A. Pesah e al. “Absence o Ba en Pla eaus in Quan um Con olu ional Neu al Ne wo ks”. In:
Physical Re iew X
11.4 (2021-10), p. 041011. doi: 10.1103/PhysRe X.11.041011. (Visi ed
on 2023-05-11) (ci . on p. 43).
[155] D. Pe z and C. Ghinea. “In oduc ion o Quan um Fishe In o ma ion”. In:
Quan um P obabili y and
Rela ed Topics
. Vol. Volume 27. QP-PQ: Quan um P obabili y and Whi e Noise Analysis. WORLD
SCIENTIFIC, 2011-01, pp. 261–281. isbn: 978-981-4338-73-8. doi: 10.1142/978981433874
5_0015. (Visi ed on 2024-04-02) (ci . on pp. 22, 23).
[156] J. P eskill. “Quan um Compu ing in he NISQ E a and Beyond”. In:
Quan um
2 (2018-08), p. 79.
doi: 10.22331/q-2018-08-06-79. (Visi ed on 2024-06-11) (ci . on p. 1).
[157] R. Puig-i-Valls e al.
Va ia ional Quan um Simula ion: A Case S udy o Unde s anding Wa m S a s
.
2024-04. doi: 10.48550/a Xi .2404.10044. a Xi : 2404.10044 [quan -ph, s a ].
(Visi ed on 2024-05-24) (ci . on pp. 141, 156).
[158] M. Ragone e al.
A Uni ied Theo y o Ba en Pla eaus o Deep Pa ame ized Quan um Ci cui s
.
2023-09. a Xi : 2309.09342 [quan -ph]. (Visi ed on 2024-01-12) (ci . on pp. 41, 42).
[159] J. Rajakuma , J. D. Wa son, and Y.-K. Liu.
Polynomial-Time Classical Simula ion o Noisy IQP
Ci cui s wi h Cons an Dep h
. 2024-10. doi: 10.48550/a Xi .2403.14607. a Xi : 2403.1
4607 [quan -ph]. (Visi ed on 2025-01-06) (ci . on p. 164).
[160] P. Reben os , M. Mohseni, and S. Lloyd. “Quan um Suppo Vec o Machine o Big Da a Classi-
ica ion”. In:
Physical Re iew Le e s
113.13 (2014-09), p. 130503. issn: 0031-9007, 1079-7114.
doi: 10.1103/PhysRe Le .113.130503. a Xi : 1307.0471 [quan -ph]. (Visi ed on
2024-03-28) (ci . on pp. 1, 30).
[161] P. Ronagh.
The P oblem o Dynamic P og amming on a Quan um Compu e
. 2021-07. doi: 10
.48550/a Xi .1906.02229. a Xi : 1906.02229 [quan -ph]. (Visi ed on 2024-06-07)
(ci . on p. 5).
[162] M. S. Rudolph e al.
T ainabili y Ba ie s and Oppo uni ies in Quan um Gene a i e Modeling
.
2023-05. a Xi : 2305.02881 [hep-ex, physics:quan -ph, s a ]. (Visi ed on 2023-
08-10) (ci . on pp. 41, 126, 142).
[163] S. Russell and P. No ig.
A i icial In elligence: A Mode n App oach (4 h Edi ion)
. Pea son, 2020.
isbn: 978-0-13-461099-3. (Visi ed on 2023-10-23) (ci . on pp. 208, 209).
239
BIBLIOGRAPHY
[164] S. H. Sack e al. “A oiding Ba en Pla eaus Using Classical Shadows”. In:
PRX Quan um
3.2
(2022-06), p. 020365. issn: 2691-3399. doi: 10 . 1103 / PRXQuan um . 3 . 020365. a Xi :
2201.08194 [quan -ph]. (Visi ed on 2024-04-12) (ci . on p. 43).
[165] J. Sch i wiese e al. “Mas e ing A a i, Go, Chess and Shogi by Planning wi h a Lea ned Model”.
In:
Na u e
588.7839 (2020-12), pp. 604–609. issn: 1476-4687. doi: 10.1038/s41586-020-
03051-4. (Visi ed on 2024-04-22) (ci . on pp. 44, 48).
[166] M. Schuld.
Supe ised Quan um Machine Lea ning Models A e Ke nel Me hods
. 2021-04. doi:
10.48550/a Xi .2101.11020. a Xi : 2101.11020 [quan -ph, s a ]. (Visi ed on
2024-03-28) (ci . on pp. 1, 29).
[167] M. Schuld and N. Killo an. “Quan um Machine Lea ning in Fea u e Hilbe Spaces”. In:
Physical
Re iew Le e s
122.4 (2019-02), p. 040504. doi: 10.1103/PhysRe Le .122.040504.
(Visi ed on 2024-03-28) (ci . on p. 29).
[168] M. Schuld and F. Pe uccione.
Machine Lea ning wi h Quan um Compu e s
. Second edi ion.
Cham, Swi ze land: Sp inge , 2021. isbn: 978-3-030-83098-4 (ci . on pp. 12, 16, 18, 27–30,
33, 38, 39).
[169] M. Schuld, R. Sweke, and J. J. Meye .
The E ec o Da a Encoding on he Exp essi e Powe o
Va ia ional Quan um Machine Lea ning Models
. 2021-03. doi: 10.48550/a Xi .2008.0860
5. a Xi : 2008.08605. (Visi ed on 2024-11-12) (ci . on p. 165).
[170] M. Schuld, R. Sweke, and J. J. Meye . “The E ec o Da a Encoding on he Exp essi e Powe o Va i-
a ional Quan um Machine Lea ning Models”. In:
Physical Re iew A
103.3 (2021-03), p. 032430.
issn: 2469-9926, 2469-9934. doi: 10.1103/PhysRe A.103.032430. a Xi : 2008.08605
[quan -ph, s a ]. (Visi ed on 2023-05-12) (ci . on pp. 1, 30–33).
[171] M. Schuld e al. “Ci cui -Cen ic Quan um Classi ie s”. In:
Physical Re iew A
101.3 (2020-03),
p. 032308. issn: 2469-9926, 2469-9934. doi: 10.1103/PhysRe A.101.032308. a Xi :
1804.00633 [quan -ph]. (Visi ed on 2024-03-28) (ci . on pp. 28, 71, 72, 137).
[172] M. Schuld e al. “E alua ing Analy ic G adien s on Quan um Ha dwa e”. In:
Physical Re iew A
99.3
(2019-03), p. 032331. issn: 2469-9926, 2469-9934. doi: 10.1103/PhysRe A.99.032331.
a Xi : 1811.11184 [quan -ph]. (Visi ed on 2023-10-24) (ci . on pp. 36, 128).
[173] J. Schulman e al.
P oximal Policy Op imiza ion Algo i hms
. 2017-08. doi: 10.48550/a Xi .1
707.06347. a Xi : 1707.06347 [cs]. (Visi ed on 2023-10-23) (ci . on pp. 60, 62, 114, 142).
[174] J. Schulman e al.
T us Region Policy Op imiza ion
. 2017-04. doi: 10.48550/a Xi .1502.0
5477. a Xi : 1502.05477 [cs]. (Visi ed on 2023-10-23) (ci . on pp. 61, 63, 157).
[175] A. Sequei a, L. P. San os, and L. S. Ba bosa. “Policy G adien s Using Va ia ional Quan um Ci -
cui s”. In:
Quan um Machine In elligence
5.1 (2023-04), p. 18. issn: 2524-4914. doi: 10.1007
/s42484-023-00101-8. (Visi ed on 2023-05-11) (ci . on pp. 2, 96, 100, 126).
240
The lowe bound on he a iance o he policy g adien can hus be u he simpli ied o:
𝕍𝜃𝜕𝜃log 𝜋𝜃≥𝔼𝜃(𝜕𝜃𝜋𝜃)2𝔼𝜃1
𝜋2
𝜃−3
2𝕍𝜃𝜕𝜃𝜋𝜃1
𝜋2
𝜃max
≥𝔼𝜃𝜕𝜃𝜋𝜃2
−𝕍𝜃𝜕𝜃𝜋𝜃2𝔼𝜃1
𝜋𝜃2
−𝕍𝜃1
𝜋𝜃2−3
2𝕍𝜃𝜕𝜃𝜋𝜃1
𝜋𝜃2
max
(A)
=𝕍𝜃𝜕𝜃𝜋𝜃2
𝕍𝜃1
𝜋𝜃2
−𝕍𝜃𝜕𝜃𝜋𝜃2
𝔼𝜃1
𝜋𝜃2
+3
2𝕍𝜃𝜕𝜃𝜋𝜃1
𝜋𝜃2
max(B)
≥𝕍𝜃𝜕𝜃𝜋𝜃2
𝕍𝜃1
𝜋𝜃2
−3𝕍𝜃𝜕𝜃𝜋𝜃2
𝔼𝜃1
𝜋𝜃2
(C)
=𝕍𝜃𝜕𝜃𝜋𝜃2𝕍𝜃1
𝜋𝜃2
−3𝔼𝜃1
𝜋𝜃2
| {z }
(𝑎)
(D)
whe e (A) is ob ained om he lowe bound o he expec a ion alue o he p oduc o wo non-nega i e
andom a iables, (B) om he assump ion ha ei he pa ame e ized block be o e/a e 𝜃 o ms a 1-
design and hus 𝔼𝜃[𝜕𝜃𝜋𝜃]=0and eo ganizing e ms and (C) om he uppe bound on he expec a ion
alue and joining e ms.
Since he a iance is non-nega i e i implies ha (𝑎) ≥ 0. The e o e he a iance will be lowe bounded
depending on he numbe o ac ions and co esponding globali y o he obse able. Fo |𝐴| ∈ O(𝑛),
𝕍𝜃𝜕𝜃𝜋𝜃2
∈Ω(1
poly(𝑛)) o O(log(𝑛))dep h. I decays polynomially wi h he numbe o qubi s since
we a e measu ing log(𝑛)(adjacen ) qubi s [42]. Mo eo e , (𝑎) ≤ poly(𝑛). Thus, he o e all a iance
deacay a mos polynomially wi h he numbe o qubi s. When he numbe o ac ions |𝐴| ∈ O(poly(𝑛)),
𝕍𝜃𝜕𝜃𝜋𝜃2
∈Ω(2−poly(log(𝑛))). I decays as e han polynomially bu slowe han exponen ially since we
a e measu ing log(poly(𝑛))qubi s [42]. In his case (𝑎) ≤ 2poly(log(𝑛))since we ha e |𝐴| ∈ poly(𝑛).
The e o e he o e all a iance decay a mos polyloga i hmically wi h he numbe o qubi s. Thus, com-
ple ing he p oo . □
247

Appendix
B
En i onmen cha ac e is ics
En i onmen S a e Ac ion Rewa d
unc ion
Ho izon Te mina ion
c i e ia
Ca pole 4 ea u es 2 ac ions
𝐴={0,1}
+1 pe ime
s ep
200 ime
s eps
Reach ho izon
o ou o
bounds
Ac obo 4 ea u es 3 ac ions
𝐴={0,1,2}
-1 + heigh 500 ime
s eps
Reach goal o
ho izon
Table 11: Cha ac e iza ion o he en i onmen s conside ed in he nume ical expe imen s.
248
Appendix
C
Na u al policy g adien s - hype pa ame e s
En i onmen Policy Laye s Obse ables Ba ch Size
Ca Pole Bo n 4 {𝑃0, 𝑃1}10
So max 4 {𝑃0, 𝑃1}10
Ac obo Bo n 5 {𝑃0,3, 𝑃1, 𝑃2}10
So max 5 {𝑃0,3, 𝑃1, 𝑃2}10
Table 12: Cha ac e iza ion o he PQC’s conside ed in he nume ical expe imen s. 𝑃𝑖indica es he p o-
jec o in he compu a ional basis in decimal. Fo he Ca pole en i onmen a single-qubi was measu ed
and he p obabili y o each basis s a e associa ed o an ac ion. In he Ac obo en i onmen , he ac ion
assignmmen was made using 𝑖𝑛𝑡 (𝑏)mod 3=𝑎 o a pa icula basis s a e 𝑏.
249
Appendix
D
PQC-based DQN - hype pa ame e s
250
Hype pa ame e Ca Pole- 0 Ac obo - 1
Qubi s (n) 4 4
Laye s 5 5
𝛾0.99 0.99
T ainable Inpu Scaling Yes, No Yes, No
T ainable Ou pu Scaling Yes, No Yes, No
Lea ning Ra e o Pa ame e s 𝜃0.001 0.001
Lea ning Ra e o Inpu Scaling
Pa ame e s
0.1 0.1
Lea ning Ra e o Ou pu Scaling
Pa ame e s
0.1 0.1
Ba ch Size 16 32
Decaying Schedule o 𝜖-G eedy
Policy
Exponen ial Exponen ial
𝜖ini 1 1
𝜖dec 0.99 0.99
𝜖min 0.01 0.01
Upda e Model 1 5
Upda e Ta ge Model 1 250
Size o Replay Bu e 10000 50000
Da a Re-uploading Yes, No Yes, No
Inpu Scaling Ini ializa ion Ini ialized as 1s Ini ialized as 1s
Ou pu Scaling Ini ializa ion Ini ialized as 1s Ini ialized as 1s
Ro a ional Pa ame e s
Ini ializa ion
Uni o mly sampled be ween 0
and 𝜋
Uni o mly sampled be ween 0
and 𝜋
˜𝑤Ini ializa ion - -
˜
𝑏Ini ializa ion - -
Obse ables (𝑍0𝑍1, 𝑍2𝑍3) (𝑍0, 𝑍1𝑍2, 𝑍3)
Table 13: PQC-based DQN hype pa ame e s o he nume ical expe imen s o Sec ion 9.5.1.
251
APPENDIX D. PQC-BASED DQN - HYPERPARAMETERS
Pa ame e Ca Pole- 0 Ac obo - 1
Qubi s (n) 4 4
Laye s 5 5
𝛾0.99 0.99
T ainable Inpu Scaling Yes Yes
T ainable Ou pu Scaling Yes Yes
Lea ning Ra e o Pa ame e s 𝜃0.001 0.001
Lea ning Ra e o Inpu Scaling
Pa ame e s
0.1 0.1
Lea ning Ra e o Ou pu Scaling
Pa ame e s
0.1 0.1
Ba ch Size 16 32
Decaying Schedule o 𝜖-G eedy
Policy
Exponen ial Exponen ial
𝜖ini 1 1
𝜖dec 0.99 0.99
𝜖min 0.01 0.01
Upda e Model 1 5
Upda e Ta ge Model 1, 500, 1000, 2500 100, 1000, 2500, 5000
Size o Replay Bu e 10000 50000
Da a Re-uploading Yes Yes
Inpu Scaling Ini ializa ion Ini ialized as 1s Ini ialized as 1s
Ou pu Scaling Ini ializa ion Ini ialized as 1s Ini ialized as 1s
𝜃Ini ializa ion Uni o mly sampled be ween 0
and 𝜋
Uni o mly sampled be ween 0
and 𝜋
˜𝑤Ini ializa ion - -
˜
𝑏Ini ializa ion - -
Obse ables (𝑍0𝑍1, 𝑍2𝑍3) (𝑍0, 𝑍1𝑍2, 𝑍3)
Table 14: Complexi y compa ison be ween classical and quan um ejec ion sampling algo i hms. 𝑁is
he numbe o a iables, 𝑀is he numbe o pa en s o any a iable, and 𝑃(𝑒)is he p obabili y o he
e idence aking alue 𝑒.
252

Pa ame e Ca Pole- 0 Ac obo - 1
Qubi s (n) 1, 2, 4 1, 2, 4
Laye s 5 5
𝛾0.99 0.99
T ainable Inpu Scaling Yes Yes
T ainable Ou pu Scaling Yes Yes
Lea ning Ra e o Pa ame e s 𝜃0.001 0.001
Lea ning Ra e o Inpu Scaling
Pa ame e s
0.001 0.001
Lea ning Ra e o Ou pu Scaling
Pa ame e s
0.1 0.1
Ba ch Size 16 32
Decaying Schedule o 𝜖-G eedy
Policy
Exponen ial Exponen ial
𝜖ini 1 1
𝜖dec 0.99 0.99
𝜖min 0.01 0.01
Upda e Model 1 5
Upda e Ta ge Model 1, 500, 1000, 2500 100, 1000, 2500, 5000
Size o Replay Bu e 10000 50000
Da a Re-uploading Yes Yes
Inpu Scaling Ini ializa ion - -
Ou pu Scaling Ini ializa ion Ini ialized as 1s Ini ialized as 1s
𝜃Ini ializa ion - -
˜𝑤Ini ializa ion Gaussian Dis ibu ion (mean=0,
s d=0.01)
Gaussian Dis ibu ion (mean=0,
s d=0.01)
˜
𝑏Ini ializa ion Ini ialized as 0s Ini ialized as 0s
Obse ables (𝑍0𝑍1, 𝑍2𝑍3) (𝑍0, 𝑍1𝑍2, 𝑍3)
Table 15: Hype pa ame e s o Models o Figu e 91
253
APPENDIX D. PQC-BASED DQN - HYPERPARAMETERS
Pa ame e Ca Pole- 0 Ac obo - 1
Qubi s (n) 2, 4, 6, 8, 10, 12 2, 4, 6, 8, 10, 12
Laye s 5 5
𝛾0.99 0.99
T ainable Inpu Scaling Yes Yes
T ainable Ou pu Scaling Yes Yes
Lea ning Ra e o Pa ame e s 𝜃0.001 0.001
Lea ning Ra e o Inpu Scaling
Pa ame e s
0.001 0.001
Lea ning Ra e o Ou pu Scaling
Pa ame e s
0.1 0.1
Ba ch Size 16 32
Decaying Schedule o 𝜖-G eedy
Policy
Exponen ial Exponen ial
𝜖ini 1 1
𝜖dec 0.99 0.99
𝜖min 0.01 0.01
Upda e Model 1 5
Upda e Ta ge Model 1, 500, 1000, 2500 100, 1000, 2500, 5000
Size o Replay Bu e 10000 50000
Da a Re-uploading Yes Yes
Inpu Scaling Ini ializa ion - -
Ou pu Scaling Ini ializa ion Ini ialized as 1s Ini ialized as 1s
𝜃Ini ializa ion - -
˜𝑤Ini ializa ion Gaussian Dis ibu ion (mean=0,
s d=0.01)
Gaussian Dis ibu ion (mean=0,
s d=0.01)
˜
𝑏Ini ializa ion Ini ialized as 0s Ini ialized as 0s
Obse ables (𝑍0. . . 𝑍𝑛/2−1, 𝑍𝑛/2. . . 𝑍𝑛) (𝑍0, 𝑍1. . . 𝑍𝑛−1, 𝑍𝑛)
Table 16: Hype pa ame e s o Models o Figu e 93
254
Appendix
E
Quan um belie upda e
Lemma E.0.1.
Le
𝜌=|𝜓
inal
ih𝜓
inal
|
be he quan um s a e a e he ampli ude ampli ica ion ope a o .
The p obabili y o measu ing he s a e
𝑠0
is gi en by,
h𝑠0|𝜌|𝑠0i=1
𝜂𝑃(𝑜|𝑠0, 𝑎)Õ
𝑠
𝑏(𝑠)𝑃(𝑠0|𝑠, 𝑎)(E.1)
which is an equi alen belie upda e ule.
P oo .
𝜌=|𝜓 inal ih𝜓 inal |
=1
𝜂Õ
𝑠,𝑎0,𝑜0,𝑟 Õ
𝑠★∈S p𝑏(𝑠★)Õ
𝑠0∈S p𝑃(𝑠0|𝑠★, 𝑎)p𝑃(𝑜|𝑠0, 𝑎)Õ
𝑟0∈R p𝑃(𝑟0|𝑠★, 𝑎) h𝑠𝑎0𝑜0𝑟|𝑠★𝑎𝑠0𝑜𝑟0i!
Õ
𝑠★∈S p𝑏(𝑠★)Õ
𝑠0∈S p𝑃(𝑠0|𝑠★, 𝑎)p𝑃(𝑜|𝑠0, 𝑎)Õ
𝑟0∈R p𝑃(𝑟0|𝑠★, 𝑎) h𝑠★𝑎𝑠0𝑜𝑟0|𝑠𝑎0𝑜0𝑟i!
=1
𝜂Õ
𝑠,𝑎0,𝑜0,𝑟 Õ
𝑠★∈S p𝑏(𝑠★)Õ
𝑠0∈S p𝑃(𝑠0|𝑠★, 𝑎)p𝑃(𝑜|𝑠0, 𝑎)Õ
𝑟0∈R p𝑃(𝑟0|𝑠★, 𝑎)𝛿𝑠𝑠★𝛿𝑎𝑎0𝛿𝑜𝑜0𝛿𝑟𝑟0|𝑠0i!
Õ
𝑠★∈S p𝑏(𝑠★)Õ
𝑠0∈S p𝑃(𝑠0|𝑠★, 𝑎)p𝑃(𝑜|𝑠0, 𝑎)Õ
𝑟0∈R p𝑃(𝑟0|𝑠★, 𝑎)𝛿𝑠𝑠★𝛿𝑎𝑎0𝛿𝑜𝑜0𝛿𝑟𝑟0h𝑠0|!
=1
𝜂Õ
𝑠∈S
𝑏(𝑠)Õ
𝑟∈R Õ
𝑠0∈S p𝑃(𝑠0|𝑠, 𝑎)p𝑃(𝑜|𝑠0, 𝑎)p𝑃(𝑟|𝑠, 𝑎)|𝑠0i!
Õ
𝑠0∈S p𝑃(𝑠0|𝑠, 𝑎)p𝑃(𝑜|𝑠0, 𝑎)p𝑃(𝑟|𝑠, 𝑎)h𝑠0|!
255
APPENDIX E. QUANTUM BELIEF UPDATE
Then, he p obabili y o measu ing s a e 𝑆𝑡+1wi h alue 𝑠0is compu ed as ollows:
h𝑠0|𝜌|𝑠0i=1
𝜂Õ
𝑠∈S
𝑏(𝑠)Õ
𝑟∈R Õ
𝑠★∈S p𝑃(𝑠★|𝑠, 𝑎)p𝑃(𝑜|𝑠★, 𝑎)p𝑃(𝑟|𝑠, 𝑎)𝑠0|𝑠★!
Õ
𝑠★∈S p𝑃(𝑠★|𝑠, 𝑎)p𝑃(𝑜|𝑠★, 𝑎)p𝑃(𝑟|𝑠, 𝑎)𝑠★|𝑠0!
=1
𝜂Õ
𝑠∈S
𝑏(𝑠)Õ
𝑟∈R Õ
𝑠★∈S p𝑃(𝑠★|𝑠, 𝑎)p𝑃(𝑜|𝑠★, 𝑎)p𝑃(𝑟|𝑠, 𝑎)𝛿𝑠0𝑠★!
Õ
𝑠★∈S p𝑃(𝑠★|𝑠, 𝑎)p𝑃(𝑜|𝑠★, 𝑎)p𝑃(𝑟|𝑠, 𝑎)𝛿𝑠★𝑠0!
=1
𝜂Õ
𝑠∈S
𝑏(𝑠)Õ
𝑟∈R
𝑃(𝑠0|𝑠, 𝑎)𝑃(𝑜|𝑠0, 𝑎)𝑃(𝑟|𝑠, 𝑎)
=1
𝜂𝑃(𝑜|𝑠0, 𝑎)Õ
𝑠∈S
𝑃(𝑠0|𝑠, 𝑎)𝑏(𝑠)Õ
𝑟∈R
𝑃(𝑟|𝑠, 𝑎)
=1
𝜂𝑃(𝑜|𝑠0, 𝑎)Õ
𝑠∈S
𝑃(𝑠0|𝑠, 𝑎)𝑏(𝑠)
□
256