Diane i, Jodi; Dumi escu, Roxana; Fe a i, Gio gio; Xu, Renyuan
Wo king Pape
En opy egula iza ion in mean- ield games o op imal
s opping
Cen e o Ma hema ical Economics Wo king Pape s, No. 755
P o ided in Coope a ion wi h:
Cen e o Ma hema ical Economics (IMW), Biele eld Uni e si y
Sugges ed Ci a ion: Diane i, Jodi; Dumi escu, Roxana; Fe a i, Gio gio; Xu, Renyuan (2025) : En opy
egula iza ion in mean- ield games o op imal s opping, Cen e o Ma hema ical Economics
Wo king Pape s, No. 755, Biele eld Uni e si y, Cen e o Ma hema ical Economics (IMW), Biele eld,
h ps://nbn- esol ing.de/u n:nbn:de:0070-pub-30076673
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/333540
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
755
Oc obe 2025
En opy Regula iza ion in Mean-Field
Games o Op imal S opping
Jodi Diane i, Roxana Dumi escu, Gio gio Fe a i and Renyuan Xu
Cen e o Ma hema ical Economics (IMW)
Biele eld Uni e si y
Uni e si ¨a ss aße 25
D-33615 Biele eld ·Ge many
e-mail: [email p o ec ed]
uni-biele eld.de/zwe/imw/ esea ch/wo king-pape s
ISSN: 0931-6558
Unless o he wise no ed, his wo k is licensed unde a C ea i e Commons
A ibu ion 4.0 In e na ional (CC BY) license. Fu he in o ma ion:
h ps://c ea i ecommons.o g/licenses/by/4.0/deed.en
h ps://c ea i ecommons.o g/licenses/by/4.0/legalcode.en
ENTROPY REGULARIZATION IN MEAN-FIELD GAMES OF OPTIMAL
STOPPING
JODI DIANETTI, ROXANA DUMITRESCU, GIORGIO FERRARI, AND RENYUAN XU
Abs ac . We s udy mean- ield games o op imal s opping (OS-MFGs) and in oduce an en opy-
egula ized amewo k o enable lea ning-based solu ion me hods. By u ilizing andomized s opping
imes, we e o mula e he OS-MFG as a mean- ield game o singula s ochas ic con ols (SC-MFG)
wi h en opy egula iza ion. We es ablish he exis ence o equilib ia and p o e hei s abili y as
he en opy pa ame e anishes. Fic i ious play algo i hms ailo ed o he egula ized se ing a e
in oduced, and we show hei con e gence unde bo h Las y–Lions mono onici y and supe modula
assump ions on he ewa d unc ional. Ou wo k lays he heo e ical ounda ion o model- ee
lea ning app oaches o OS-MFGs.
Keywo ds: Mean- ield game o op imal s opping, singula s ochas ic con ol, en opy egula -
iza ion, andomized s opping imes, ic i ious play algo i hm.
AMS subjec classi ica ion: 91A16, 60G40, 93E20, 68T01
1. In oduc ion
The s udy o mean- ield games (MFGs) has become cen al o he analysis o la ge-popula ion
s ochas ic con ol sys ems, whe e indi idual agen s in e ac h ough he empi ical dis ibu ion o
s a es and/o con ols. Fo me hodologies, echniques, and applica ions, we e e o he wo- olume
book [12]. In his pape , we ocus on a speci ic class o such p oblems: mean- ield games o op imal
s opping (OS-MFGs). Op imal s opping has a wide ange o applica ions in Economics, Finance,
Ope a ions Resea ch, and o he applied ields. Rele an examples include p icing Ame ican de i a-
i es, en y-exi p oblems ( eal op ions models), and op imal iming o buying o selling an asse ,
among o he s. On he o he hand, OS-MFGs ha e ecei ed a en ion only e y ecen ly. Among he
undamen al con ibu ions on his opic (wi hou lea ning conside ed), we i s men ion [52], whe e
he au ho s conside a speci ic game in which he in e ac ion occu s h ough he numbe o playe s
who ha e al eady s opped. In [13], he au ho s s udy an op imal s opping game wi h B ownian
common noise, inspi ed by a model o bank uns. They p o e he s ong exis ence o mean- ield
equilib ia in a se ing wi h s a egic complemen a i y, and weak exis ence in a se ing wi h con i-
nui y. In [7], a pu ely analy ical app oach is adop ed o sol e he p oblem – namely, he s udy o
he sys em o coupled Hamil on-Jacobi-Bellman and Fokke -Planck equa ions – and he exis ence
o mixed solu ions is p o ed. Mo e ecen ly, a compac i ica ion app oach based on he linea p o-
g amming o mula ion is de eloped in a se ies o pape s [8], [33] and [34], in which he exis ence
o elaxed solu ions, in e p e ed ia occupa ional measu es, is es ablished. These echniques ha e
been hen applied o sol e en y-exi op imal s opping games in elec ici y ma ke s in [1] and [5],
and in he case wi h common noise in [35]. In [32], a igo ous connec ion be ween he occupa ion
measu es a ising in he linea p og amming app oach and andomized s opping is es ablished. In
Da e: Oc obe 20, 2025.
J. Diane i: Depa men o Economics and Finance, Uni e si y o Rome To Ve ga a. Email:
jo[email p o ec ed].
R. Dumi escu: ENSAE-CREST, Ins i u Poly echnique de Pa is. Email: o[email p o ec ed].
G. Fe a i: Cen e o Ma hema ical Economics, Biele eld Uni e si y. Email: [email p o ec ed].
R. Xu: Depa men o Finance and Risk Enginee ing, NYU. Email: [email protected].
1
addi ion, [53] s udies an op imal s opping game h ough he lens o he mas e equa ion. Ano he
ecen con ibu ion is [46], whe e he au ho s de elop a new app oach o sol ing a ious ypes o
MFGs (including op imal s opping) based on a mean- ield e sion o he Bank-El Ka oui ep e-
sen a ion heo em o s ochas ic p ocesses. In he con ex o op imal s opping MFGs wi h pa ial
in o ma ion, [9,10,59] s udy he exis ence o he mean- ield solu ion o a one-dimensional il e ing
p oblem unde di e en model se -ups. Finally, ela ed wo ks on McKean-Vlaso op imal s opping
a e [61] and [21].
Despi e hei impo ance, OS-MFGs pose signi ican heo e ical and compu a ional challenges.
Fi s , hey equi e mo e sub le echniques o be sol ed han MFGs wi h s anda d con ol because
o he i egula i y o he low measu e gene a ed by he simul aneous exi o a signi ican numbe o
playe s. This class o games becomes pa icula ly challenging when he goal is o de elop model- ee,
lea ning-based solu ion me hods.
The p ima y mo i a ion o ou wo k is o es ablish he ounda ions o a sui able OS-MFG ame-
wo k ha suppo s model- ee solu ion me hods. A cen al challenge in his di ec ion is o igo ously
o malize and analyze andomized and explo a i e s a egies in games, which play a c ucial ole in
he design o ein o cemen lea ning (RL) algo i hms. Es ablishing he exis ence and uniqueness o
(mean- ield) equilib ia ha accommoda e such s a egies is he e o e essen ial—no only o heo-
e ical comple eness bu also o enabling p ac ical lea ning algo i hms wi h p o able gua an ees. In
he absence o such a ounda ion, igo ous analysis o mul i-agen lea ning emains elusi e. Mo e-
o e , he con e gence o RL me hods o en depends on he p ope ies o he unde lying i e a i e
schemes (e.g., ic i ious play), whose well-posedness simila ly hinges on he exis ence and uniqueness
o equilib ia in games ha allow o andomized beha io (see, e.g., [40,41]). This need becomes
e en mo e p onounced in en i onmen s cha ac e ized by i egula o spa se decision-making, such
as op imal s opping p oblems, whe e agen s make a single, i e e sible decision based on limi ed
in o ma ion. Unlike s anda d Ma ko Decision P ocesses (MDPs) [54,60], whe e equen ac ions
and ewa ds can educe he eliance on explici explo a ion, op imal s opping games equi e mo e
delibe a e and ca e ully designed explo a ion mechanisms o acili a e in o ma ion acquisi ion [26].
On he echnical side, howe e , a majo challenge a ises in applying RL o OS-MFGs. S anda d
equilib ium- inding algo i hms in MFGs ypically ely on he s abili y o op imal con ols wi h e-
spec o a ia ions in he dis ibu ion o agen s. This s abili y, howe e , is pa icula ly di icul o
ensu e in he con ex o op imal s opping, whe e unde s anding how equilib ium s opping policies
espond o small pe u ba ions in he measu e low is highly non i ial. To illus a e his, conside
ha in a Ma ko ian se ing, equilib ium s opping ules a e o en gi en by hi ing imes o he
unde lying s a e p ocess a a ee bounda y. De e mining he s abili y p ope ies (e.g., Lipschi z
con inui y) o hese ee bounda ies is al eady a echnically in ica e p oblem, e en in single-agen
op imal s opping se ings (see, e.g., [23] and he discussion he ein).
Ou con ibu ions. To add ess hese challenges, we de elop a new heo e ical amewo k ha
enables he use o model- ee algo i hms o OS-MFGs. The co e idea is o e o mula e he game
as a MFG o singula s ochas ic con ols (SC-MFG) by in oducing andomized s opping imes (see
[30,62]). A andomized s opping ime can be in e p e ed as he (condi ional) p obabili y o s opping
be o e a ce ain ime . This in e p e a ion leads na u ally o an adap ed, nonnega i e, nondec easing
p ocess wi h igh -con inuous sample pa hs, bounded by 1 – an objec ha , ollowing e minology
om con ol heo y, we e e o as a singula con ol (see Chap e VIII in [37] o an in oduc ion
o singula s ochas ic con ols). To p omo e explo a ion, we in oduce an en opy egula iza ion
e m in o he s opping unc ional, as p e iously es ablished in he single-agen se ing o [30]. This
egula iza ion e m is go e ned by a empe a u e pa ame e λ≥0and induces s ic conca i y
in he pe o mance c i e ion. This change enables op imize s o de ia e om pu e 0-1s a egies
(o no s opping o s opping) and ins ead a o e lec ing- ype con ols. In he con ex o MFG,
he equilib ium is cha ac e ized ia a wo-s ep p ocedu e. Fi s , gi en a ixed low o measu es
2
and a join dis ibu ion, we de e mine he singula con ol ha maximizes he egula ized ewa d
unc ional. Second, we impose he so-called consis ency condi ion, which equi es ha he ixed
inpu measu es coincide wi h he dis ibu ion o he s a e p ocess be o e s opping and he join
dis ibu ion o he ( andomized) s opping ime and he s opped s a e. Na u ally, in his amewo k,
he consis ency condi ions mus be e o mula ed in e ms o andomized s opping imes – namely, in
e ms o he andom Bo el measu e on he ime axis induced by he singula con ol. The pa icula
s uc u e o hese consis ency condi ions p e en s us om applying exis ing exis ence esul s o
mean- ield equilib ia in singula con ol games (see, e.g., [11,25,38,39]). We he e o e es ablish a
no el exis ence esul o equilib ia in ou SC-MFG amewo k, which we belie e is o independen
heo e ical in e es (see Theo em 3.6).
Ha ing p o ed he exis ence o equilib ia in andomized s opping imes, we hen mo e on o
es ablishing λ-s abili y as he empe a u e pa ame e λ↓0. We show ha any equilib ium o he
SC-MFG app oxima es, as λ↓0(up o a subsequence), an equilib ium o he o iginal MFG o
op imal s opping wi hou en opy egula iza ion. Fu he mo e, unde a Las y-Lions mono onici y
condi ion, we can p o e a s onge esul : he en i e sequence o mean- ield equilib ia o he singula
con ol game (pa ame ized by λ) con e ges o an equilib ium o he o iginal op imal s opping game
(see Theo em 3.10). This esul ensu es ha he solu ions o he egula ized p oblems con e ge o
equilib ia o he o iginal OS-MFG, p o iding a c ucial heo e ical jus i ica ion o using en opy-
based app oxima ions in RL me hods.
We hen design no el ic i ious-play algo i hms ailo ed o he en opy egula ized SC-MFG ame-
wo k. We es ablish hei con e gence unde bo h he Las y–Lions mono onici y condi ion (see Theo-
em 4.5) and in he case o supe modula ewa d s uc u es (see Theo em 4.9). I is impo an o no e
ha he p esence o he en opy egula iza ion e m in oduces nonlinea i y in o he pe o mance
c i e ion, placing ou model ou side he scope o ecen app oaches based on linea p og amming
o MFGs [8,33,34]. This nonlinea i y necessi a es he de elopmen o new analy ical and nume i-
cal ools. The ic i ious-play algo i hm unde he Las y–Lions condi ion he e o e ex ends p e ious
algo i hms de eloped unde his mono onici y assump ion o cases whe e he payo unc ional is
non-linea wi h espec o he con ol. Unde he supe modula i y condi ion, he ic i ious-play al-
go i hm is new in he li e a u e. The heo e ical ounda ion p esen ed he e will be complemen ed
by a companion pape ocused on he algo i hmic design, ela ed heo e ical con e gence analysis,
and nume ical alida ion o he p oposed amewo k, which is cu en ly in p og ess [28].
Rela ed li e a u e. Rein o cemen lea ning applied o op imal s opping p oblems is closely ied
o he b oade challenge o lea ning in en i onmen s wi h spa se ewa ds [26,45]. In hese se ings,
a ewa d is only g an ed a he s opping ime, esul ing in ex eme spa si y and signi ican lea ning
challenges compa ed o mo e con en ional con ol asks, whe he o mula ed in con inuous ime o
as classical MDPs.
When he model is ully known, ecen s udies such as [56] and [58] ha e in oduced deep lea ning
echniques o app oxima e op imal s opping bounda ies. In a ela ed ein, [6] in es iga ed andom-
ized o mula ions o op imal s opping and p o ided con e gence esul s o bo h o wa d and back-
wa d Mon e Ca lo-based op imiza ion schemes. Simila ly, [3] showcased he e ec i eness o deep
lea ning me hods o sol ing high-dimensional singula con ol p oblems.
In he ealm o con inuous- ime RL, a no able ad ancemen comes om [24], who p oposed
a gene al policy g adien amewo k applicable o a wide a ay o s ochas ic con ol p oblems,
including op imal s opping, impulse con ol, and swi ching. Thei app oach le e ages connec ions
be ween s ochas ic con ol and i s andomized coun e pa s. Howe e , hei amewo k cu en ly
lacks heo e ical gua an ees o con e gence. The wo k [31] s udies a egula ized e sion o he one-
dimensional Ame ican Pu op ion unde a Shannon en opy amewo k. By employing an in ensi y-
based con ol o mula ion, hey in oduce explo a ion and p o e con e gence o he policy i e a ion
algo i hm (PIA) o ixed empe a u e pa ame e s. Ne e heless, he con e gence o he op imal
3
policy as he empe a u e pa ame e λ↓0 emains an open ques ion. An al e na i e pe spec i e was
ecen ly o e ed by [22], who e o mula ed he op imal s opping p oblem as a egula con ol p oblem
wi h bina y (0o 1) ac ions and applied en opy egula iza ion. This ans o med he p oblem in o
a classical en opy- egula ized con ol se ing, enabling he use o s anda d RL algo i hms. The
e y ecen pape [51] p oposes a con inuous- ime ein o cemen lea ning amewo k o a class o
singula s ochas ic con ol p oblems wi hou en opy egula iza ion and gene alizes he exis ing
policy e alua ion heo ies wi h egula con ols o lea n op imal singula con ol law and de elop a
policy imp o emen heo em.
Ano he impo an esea ch di ec ion le e ages non-pa ame ic s a is ical me hods o s ochas ic
p ocesses o design lea ning-based con ol algo i hms [16,17,18,19]. In pa icula , [17] and [18]
add essed one-dimensional singula and impulse con ol p oblems by lea ning c i ical h esholds
using a non-pa ame ic di usion amewo k. This app oach was ex ended o mul i a ia e e lec ion
p oblems in [19], while [16] con ibu ed heo e ical gua an ees, including eg e bounds and non-
asymp o ic PAC es ima es.
En opy egula ized MFGs and hei ein o cemen lea ning (RL) o mula ions ha e ecei ed
inc easing a en ion o e he pas yea . We do no a emp o p o ide an exhaus i e li e a u e
e iew he e, bu ins ead highligh a ew key e e ences: [15], which p esen s an RL app oach o
model- ee MFGs o MDPs; [2], which discusses Q-lea ning in egula ized MFGs o MDPs; [14],
which p o ides a con e gence analysis o machine lea ning algo i hms o mean- ield con ol and
games o e a ini e ime ho izon in con inuous ime and space; and [36,42], which add esses en opy
egula ized MFGs in di usion se ings.
As a as i conce ns he mo e speci ic class o en opy- egula ized MFGs o op imal s opping
– as hose conside ed in his pape – he only pape b ough o ou a en ion is [65]. This pape
in es iga es a disc e e- ime majo -mino MFG whe e he majo playe can choose o ei he con ol
o s op. The goal is o ind a elaxed ( andomized) s opping equilib ium, o mula ed as a ixed poin
o a se - alued map, whose di ec analysis is di icul due o he majo playe ’s in luence. To add ess
his, he au ho s in oduce en opy egula iza ion o he majo playe ’s p oblem and e o mula e
he mino playe s’ s opping p oblems using linea p og amming o e occupa ion measu es, in he
spi i o [8]. They p o e he exis ence o egula ized equilib ia ia a ixed-poin heo em and show
ha , as egula iza ion anishes, hese equilib ia con e ge o a solu ion o he o iginal p oblem, hus
es ablishing he exis ence o a elaxed equilib ium. Al hough he e a e clea connec ions be ween
[65] and he p esen pape , he ma hema ical analysis is en i ely di e en , as he p oblem in [65]
in ol es a S ackelbe g game (which is no p esen in ou case) on op o he mean- ield in e ac ion,
wi hin a disc e e- ime and disc e e-space se ing – unlike ou con inuous- ime and con inuous-space
amewo k.
O ganiza ion o he pape . The es o he pape is o ganized as ollows. In Sec ion 2we
p esen he OS-MFG p oblem and in oduce i s en opy- egula ized o mula ion h ough andomized
s opping imes. In Sec ion 3we p o e exis ence and s abili y o equilib ia, while in Sec ion 4we
p esen he ic i ious play algo i hms and hei con e gence. Finally, Appendix Acollec s esul s on
a connec ion be ween he s udied MFG o singula con ol and (ano he ) auxilia y MFG o op imal
s opping.
1.1. Gene al no a ion. Le p≥1and (E, d)a non-emp y Polish space. De ine he se Psub
p(E)o
subp obabili y measu es νon he Bo el subse s o Esuch ha REd(z, x0)pν(dz)<∞, o some (and
hus all) x0∈E. The space Pp(E) ep esen s he se o elemen s o Psub
p(E)which a e p obabili y
measu es, and i is endowed wi h he Wasse s ein dis ance dpde ined as
dp(¯ν, ν) := in
π∈Π(¯ν,ν)ZE×E
d(¯z, z)pπ(d¯z, dz),(1.1)
4
whe e Π(¯ν, ν)is he se o π∈ Pp(E×E)which has ¯νas i s ma ginal and νas second ma ginal.
Following [20, Sec ion B], o de ine he Wasse s ein dis ance d′
pon Psub
p(E), we in oduce a ceme e y
poin ∂and ob ain he enla ged space ¯
E:= E∪∂. By de ining d(x, ∂) := d(x, x0)+1, we ex end
he de ini ion o don ¯
E, in such a way ha (¯
E, d)is s ill Polish. We de ine he classical Wasse s ein
dis ance dpon ¯
Eusing he same exp ession (1.1). The Wasse s ein ype dis ance d′
pon Psub
p(¯
E)is
gi en by
d′
p(µ, ν) := dp(¯µ, ¯ν),
whe e
¯µ(·) := µ(· ∩ E) + (1 −µ(E))δ∂,¯ν(·) := ν(· ∩ E) + (1 −ν(E))δ∂.
Fix T > 0. We in oduce he se Mp(E)o measu able unc ions m: [0, T ]→ Psub
p(E)iden i ied
d -a.e. on [0.T], endowed wi h he con e gence in measu e which is induced by he me ic
dM
p(m, ν) := ZT
0
1∧d′
p(m , ν )d , m, ν ∈Mp(E).(1.2)
A sequence (mn)ncon e ges o m(mn→m, in sho ) i o any ε > 0we ha e
ZT
0
1{ :d′
p(mn
,m )>ε}d →0.
Recall ha , o any con e gen sequence mn→m, he e exis s a subsequence (mnk)ksuch ha
(1.3) d′
p(mn
, m )→0, d -a.e.
We e e o Appendix B in [34] o u he de ails.
Th oughou he pape , C > 0will deno e a gene ic cons an ha may change om line o line.
2. MFGs o op imal s opping and en opy egula iza ion
This sec ion o mally in oduces he OS-MFG p oblem wi h andomiza ion and en opy egu-
la iza ion, leading o a singula con ol o mula ion. Mo e speci ically, Sec ion 2.1 in oduces p e-
limina ies and model assump ions; Sec ion 2.2 desc ibes he OS-MFG p oblem; and Sec ion 2.3
de elops he amewo k o andomized s opping imes, en opy egula iza ion, and he esul ing
singula con ol p oblem.
2.1. P elimina ies and assump ions. Le T∈(0,∞)be gi en and ixed h oughou he es
o he pape . Fo d, d1∈N {0}, conside he con inuous unc ions b: [0, T]×Rd→Rd, σ :
[0, T]×Rd→Rd×d1, : [0, T ]×Rd× Psub(Rd)→R,g:Rd× P(Rd)→R.
De ine
Mp:= Mp(Rd)and M:= Mp× Pp([0, T]×Rd),
endowed wi h he p oduc opology. Elemen s o Mwill ypically be deno ed by couples (m, µ).
Mo eo e , gi en a sequence (mn, µn)n⊂ M and a limi poin (m, µ)⊂ M, we will w i e
(mn, µn)n→(m, µ)
o deno e he con e gence in he p oduc opology.
On a comple e p obabili y space (Ω,F,P), conside a d1-dimensional B ownian mo ion W:=
(W ) and a Rd- alued squa e in eg able F0-measu able andom a iable x0independen o W.
Deno e by Fx0,W := (Fx0,W
) he igh -con inuous ex ension o he il a ion gene a ed by Wand
x0. Fo a ime-ho izon T∈(0,∞), le Tbe he se o Fx0,W -s opping imes τsuch ha τ≤Ta.s..
Assump ion 2.1. The da a o he p oblem (x0, b, σ, , g) e i y:
(1) E[|x0|p]<∞.
5
(2) The e exis s a cons an L > 0such ha
|b( , x)|+|σ( , x)| ≤ L(1 + |x|),∀ ∈[0, T], x ∈Rd,
|b( , ¯x)−b( , x)|+|σ( , ¯x)−σ( , x)| ≤ L|¯x−x|,∀ ∈[0, T ], x, ¯x∈Rd.
(3) The e exis s a cons an K > 0such ha
| ( , x, m)| ≤ K1 + |x|p+ZRd
|z|pm(dz),∀( , x, m)∈[0, T ]×Rd× Psub
p(Rd),
|g(x, µ)| ≤ K1 + |x|p+Z[0,T ]×Rd
|z|pµ(ds, dz),∀(x, µ)∈Rd× Pp([0, T ]×Rd).
(4) The unc ions and ga e con inuous, and gis con inuous in µ, locally uni o mly wi h
espec o x. Tha is, he e exis a cons an K > 0and a unc ion wg:Pp([0, T]×Rd)×
Pp([0, T]×Rd)→[0,∞)wi h limdp(µ,¯µ)→0wg(¯µ, µ)=0such ha |g(x, ¯µ)−g(x, µ)| ≤
K(1 + |x|p)wg(¯µ, µ) o any x∈Rd, µ, ¯µ∈ Pp([0, T]×Rd).
2.2. The MFG o op imal s opping. Conside he mean- ield game o op imal s opping (OS-
MFG, in sho ) in which, o a gi en beha io (m, µ)∈ M o he popula ion o playe s, he ep e-
sen a i e playe maximizes, o e he choice o s opping imes τ∈ T , he p o i unc ional
(2.1) J(τ, m, µ) := EZτ
0
( , X , m )d +g(Xτ, µ),
subjec o dX =b( , X )d +σ( , X )dW , ∈[0, T ], X0=x0.
Thanks o Assump ion 2.1, he e exis s a unique s ong solu ion o he s ochas ic di e en ial equa-
ion (SDE) o Xwi h con inuous pa hs a.s., and such ha
(2.2) Ehsup
∈[0,T ]
|X |pi<∞.
Mo eo e , he p o i unc ional J(τ, m, µ)is well de ined o any τ∈ T , he maximiza ion p ob-
lem has ini e alue (i.e., supτJ(τ, m, µ)<∞), and he e exis s an op imal s opping (i.e., τ∗∈
a g maxτJ(τ, m, µ)). We e e he in e es ed eade o Appendix D in [49] o u he de ails.
We conside he ollowing no ion o equilib ium.
De ini ion 2.2 (OS-MFG equilib ium wi h s ic s opping).An op imal s opping MFG equilib ium
(wi h s ic s opping) is a iple (ˆτ, ˆm, ˆµ)∈ T × M such ha
(2.3)
ˆτ∈a g maxτ∈T J(τ, ˆm, ˆµ),
ˆm (A) = P(X ∈A, ≤ˆτ),∀A∈ B Rd, ∈[0, T ),
ˆµ(B×[0, ]) = P(Xˆτ∈B, ˆτ≤ ),∀B∈ B Rd, ∈[0, T ].
The i s condi ion is he op imali y condi ion, while he second and he hi d condi ion co espond
o he so-called consis ency condi ions.
Unde he equilib ium no ion in De ini ion 2.2, P oblem 2.1 in oduces a MFG whe e agen s
in e ac h ough he unning ewa d unc ion wi h ins an aneous mean- ield measu e m and
h ough he e minal cos gwi h measu e µquan i ying he s opping ime and s opping posi ion.
2.3. Singula con ol o mula ion and en opy egula iza ion. In his subsec ion, we e o -
mula e he OS-MFG p oblem wi h he aim o p o iding a heo e ical amewo k ha enables agen s
o lea n equilib ia using RL algo i hms. To his end, we modi y he o iginal OS-MFG amewo k
by inco po a ing he explo a ion–exploi a ion ade-o , which is empi ically known o enhance he
con e gence o lea ning me hods. Fic i ious play algo i hms will be discussed la e in Sec ion 4.
6
2.3.1. Randomized s opping imes and singula con ols. We i s in oduce he concep o andom-
ized s opping imes. We bo ow ideas om he li e a u e in game heo y (see, e.g., [62]) and om
[30], whe e he en opy egula ized e sion o an op imal s opping p oblem is conside ed.
A andomized s opping ime can be in insically connec ed o singula con ol when in e p e ed
as he condi ional p obabili y o s opping be o e a gi en ime . To build his connec ion o mally,
de ine he se o s ochas ic p ocesses
A:= ξ: Ω ×[0, T]→[0,1],Fx0,W -adap ed, nondec easing, càdlàg, wi h ξ0−= 0 and ξT= 1.
In he es o his pape , wi h e e ence o he e minology o s ochas ic con ol heo y, we shall
e e o an elemen o Aas o a singula con ol (see, e.g., Chap e VIII in [37]).
Conside a andom a iable U: Ω →[0,1] which is uni o mly dis ibu ed and independen om
Wand x0. Unless o enla ge he o iginal p obabili y space, we can assume U o be de ined on
(Ω,F,P)as well, and o be F-measu able. Gi en ξ∈ A, a andomized s opping ime is de ined as
τξ:= in { ∈[0, T]|ξ > U},
wi h he con en ion in ∅=T.
No ice ha τξis a s opping ime wi h espec o he enla ged il a ion Fx0,W,U , gene a ed by
x0, W and U. Hence τξis no necessa ily an Fx0,W -s opping ime. Howe e , i τ∈ T , he p o-
cess ξτ∈ A de ined by ξτ:= (1{ ≥τ}) is such ha τ=τξτ. This gi es a na u al inclusion o
(ξτ
) , ξτ
=1{ ≥τ}in o A. We inally obse e ha , hanks o he de ini ion o τξand he inde-
pendence o Uwi h espec o Wand x0, we ha e
(2.4) Pτξ≤ | Fx0,W
=PU≤ξ | Fx0,W
=Zξ
0
du =ξ ,
so ha ξ can be in e p e ed as he (condi ional) p obabili y o s opping be o e ime .
Wi h sligh abuse o no a ion, we will e alua e he payo Jas in (2.1) on andomized s opping
imes as well. In pa icula , using (2.4), an applica ion o he owe p ope y and o he independence
o Uwi h espec o (W, x0)allows o ew i e he payo unc ional o any (m, µ)∈ M as
(2.5) J(τξ, m, µ) = E"ZT
0
(s, Xs, ms) (1 −ξs)ds +Z[0,T ]
g(Xs,, µ)dξs#.
He e, ξ∈ A is he singula con ol associa ed o he andomized s opping ime τξ, and o a gene ic
con inuous p ocess Y:= (Y ) we ha e se R[0,T ]Ysdξs:= Y0ξ0+RT
0Ysdξs.
2.3.2. Consis ency condi ions o andomized s a egies. Equa ion (2.5) p o ides he exp ession o
he ep esen a i e playe ’s payo unc ional when she is allowed o play a andomized s opping ule.
We now de i e a con enien o m o he consis ency condi ions in (2.3) o andomized s opping
imes τξin e ms o he ela ed singula con ol ξ∈ A.
Fo any ξ∈ A, by using (2.4), we ew i e he i s consis ency condi ion in (2.3) as
(2.6)
PX ∈A, < τξ=P(X ∈A, ξ ≤U)
=EhEh1{X ∈A,U⩾ξ }| Fx0,W
ii
=EZ1
0
1{X ∈A,ξ ≤u}du
=EZ1
ξ
1{X ∈A}dq
=E1{X ∈A}(1 −ξ ),∀A∈ B(Rd), ∈[0, T],
7
(2) Jλ(ξ, mξ, µξ)−Jλ(¯
ξ, mξ, µξ)−(Jλ(ξ, m¯
ξ, µ¯
ξ)−Jλ(¯
ξ, m¯
ξ, µ¯
ξ)) ≤0.
No ice ha Condi ion 1in Assump ion 3.7 is always sa is ied when λ > 0.
Rema k 3.8. We p o ide he e an example in which Condi ion (2)in Assump ion (3.7)is sa is ied.
Fo example, one could conside and go he o m
( , x, m) = ¯
k(x)¯
, ZRd
¯
k(x)m (dx), g(x, µ) = ¯
ℓ(x)¯
h Z[0,T ]×Rd
¯
ℓ(x)µ(d , dx)!,
wi h ¯
non-inc easing in he second a gumen and ¯
hnon-dec easing.
Theo em 3.9. Unde Assump ions 2.1 and 3.7, o any λ≥0 he e exis s a unique equilib ium o
λ-SC-MFG (2.10).
P oo . The p oo o his esul his heo em is s anda d and sligh ly adap ed om [8], we include i
o comple eness.
A guing by con adic ion, assume ha he e exis s wo dis inc mean- ield equilib ia (ξ, m, µ),
(¯
ξ, ¯m, ¯µ). By uniqueness o he op imal con ol and he de ini ion o equilib ium, we ha e
Jλ(ξ, mξ, µξ)−Jλ(¯
ξ, mξ, µξ)>0and Jλ(¯
ξ, m¯
ξ, µ¯
ξ)−Jλ(ξ, m¯
ξ, µ¯
ξ)>0.
Summing up hese wo inequali ies, we ob ain
Jλ(ξ, mξ, µξ)−Jλ(¯
ξ, mξ, µξ)−(Jλ(ξ, m¯
ξ, µ¯
ξ)−Jλ(¯
ξ, m¯
ξ, µ¯
ξ)) >0,
which con adic s he second condi ion in Assump ion 3.7. The e o e, he equilib ium is unique. □
En opy egula iza ion and he co esponding empe a u e pa ame e λa e in oduced o encou -
age andomiza ion in a lea ning en i onmen . I is impo an o unde s and he closeness be ween
he equilib ia o he en opy- egula ized p oblem (2.10) – which we aim o lea n – and hose o he
o iginal p oblem (2.3). We discuss his con e gence in he ollowing heo em.
Theo em 3.10. Unde Assump ion 2.1, o any λ > 0, le (ˆ
ξλ,ˆmλ,ˆµλ)be an equilib ium o he
λ-SC-MFG. Then
(1) (ˆ
ξλ,ˆmλ,ˆµλ)→(ˆ
ξ0,ˆm0,ˆµ0)up o subsequence as λ→0, whe e (ˆ
ξ0,ˆm0,ˆµ0)is an equilib ium
o he 0-SC-MFG;
(2) Unde he addi ional Assump ion 3.7, we ha e (ˆ
ξλ,ˆmλ,ˆµλ)→(ˆ
ξ0,ˆm0,ˆµ0)as λ→0.
P oo . To simpli y he no a ion, we simply w i e (ξλ, mλ, µλ)ins ead o (ˆ
ξλ,ˆmλ,ˆµλ), o any λ≥0.
Fix any sequence (λn)n⊂(0,∞)wi h λn→0as n→ ∞ and se
(ξn, mn, µn) := (ξλn, mλn, µλn).
Since (ξn, mn, µn)a e assumed o be equilib ia, we ha e (mn, µn) = Γ(Rλ(ξn)), so ha (mn, µn)n⊂
Γ(A). By compac ness o A×Γ(A)(see Lemma 3.3), we can ex ac a subsequence (no elabeled) o
(ξn, mn, µn)nand a limi poin (ξ0, m0, µ0)such ha (ξn, mn, µn)→(ξ0, m0, µ0). Thanks o Lemma
2, we ha e ha ξ0∈R0(m0, µ0). Mo eo e , by epea ing he a gumen s in he p oo o Theo em 3.6,
we ha e ha Γ(ξ0) = (m0, µ0). This, in u n implies ha ξ0∈R0(m0, µ0) = R0(Γ(ξ0)) = R0(ξ0),
so ha (ξ0, m0, µ0)is an equilib ium o he 0-SC-MFG. This comple es he p oo o Claim (1).
When he addi ional Assump ion 3.7 holds, by Theo em 3.9 we ha e he uniqueness o he equi-
lib ium o he 0-SC-MFG. Hence, by he p e ious a gumen , any sequence (ξλn, mλn, µλn)con e ges
o (ξ0, m0, µ0), hus p o ing Claim (2). □
14
4. Fic i ious play algo i hms
In his sec ion, we in oduce wo no el ic i ious play algo i hms o compu ing he unique mean-
ield equilib ium λ-SC-MFG in he case when λ > 0, and we es ablish hei con e gence unde
di e en da a assump ions.
No e ha he con e gence o i e a i e nume ical schemes – such as ic i ious play algo i hms
unde known model pa ame e s – se es as a undamen al ounda ion o analyzing he con e gence
o RL algo i hms in en i onmen s wi h unknown pa ame e s. In many popula RL me hods, such
as Q-lea ning and ac o -c i ic algo i hms [60], he known quan i ies in hese schemes (e.g., he alue
unc ion) a e simply eplaced by hei co esponding es ima es, compu ed om a ailable obse a-
ions.
Algo i hm 1: Fic i ious play algo i hm
Da a: A numbe o s eps n o he equilib ium app oxima ion; a con ol ¯
ξ0∈ A and
(¯µ0,¯m0) := Γ(¯
ξ0)∈ Rλ;
Resul : App oxima e MFG Nash equilib ium
1 o k= 0,1, . . . , n −1do
2Se ξk+1 := a g maxξ∈A Jλ(ξ, ¯mk,¯µk);
3Se ¯
ξk+1 := k
k+1 ¯
ξk+1
k+1ξk+1 =1
k+1 Pk+1
ℓ=1 ξℓ;
4Se (¯µk+1,¯mk+1) := Γ(¯
ξk+1).
5end
Obse e ha , due o he linea dependence o he measu e (m, µ)wi h espec o ξ, we ha e ha
(¯µn,¯mn) = 1
nPn
k=1(µξk, mξk).
In he nex wo subsec ions, we es ablish he con e gence o he ic i ious play algo i hm ( ¯mn,¯µn)n
o he equilib ia o he λ-SC-MFG, unde sui able s uc u al condi ions on he da a, and o sui able
choice o he ini ializa ion.
4.1. Fic i ious play unde he Las y-Lions condi ion. We s a wi h he ic i ious play algo-
i hm unde he Las y-Lions mono onici y condi ion.
This ic i ious-play algo i hm is no el in he con ex o SC-MFGs and also b ings new con i-
bu ions compa ed o he ic i ious-play algo i hm de eloped in [34] o OS-MFGs. In pa icula , i
ex ends o cases whe e he p o i - unc ional is non-linea wi h espec o he andomized s opping, a
se ing in which he linea p og amming app oach de eloped in [34] ails.
To simpli y he p esen a ion, we se up some no a ion. Fi s , no ice ha o a gi en (m, µ)∈Γ(A)
and ξ∈ A, by using he iden i ica ion be ween con ols and associa ed measu es gi en by (3.1), we
can ew i e Jλas ollows:
Jλ(ξ, m, µ) = ZT
0ZRd ( , x, m )mξ
(dx)d +g( , x, µ)µξ(d , dx)+λEZT
0
E(ξ )d .
We now in oduce he bilinea o m no a ion
⟨ (m), m′⟩:= ZT
0ZRd
( , x, m )m′
(dx)d , ⟨g(µ), µ′⟩:= Z[0,T ]×Rd
g( , x, µ)µ′(d , dx),
whe e (µ, m),(µ′, m′)∈ Pp([0, T]×R)×Mp. The e o e, Jλw i es as ollows:
Jλ(ξ, m, µ) = ⟨ (m), mξ⟩+⟨g(µ), µξ⟩+λEZT
0
E(ξ )d .
We assume he ollowing condi ions hold:
15
Assump ion 4.1. The e exis cons an s c >0and cg>0such ha o all ∈[0, T ],x, x′∈Rd,
m, m′∈ Psub
p(Rd),µ, µ′∈ Pp([0, T]×Rd),
( , x, m)− ( , x, m′)≤c (1 + |x|)ZRd
(1 + |z|p)|m−m′|(dz),
| ( , x, m)− ( , x, m′)− ( , x′, m) + ( , x′, m′)| ≤ c |x−x′|ZRd
(1 + |z|p)|m−m′|(dz),
|g( , x, µ)−g( , x, µ′)| ≤ cg(1 + |x|)Z[0,T ]×Rd
(1 + |z|p)|µ−µ′|(ds, dz),
|g( , x, µ)−g( , x, µ′)−g( ′, x′, µ)+g( ′, x′, µ′)| ≤ cg(| − ′|+|x−x′|)Z[0,T ]×Rd
(1+|z|p)|µ−µ′|(ds, dz),
whe e |m|deno es he o al a ia ion measu e o m.
Be o e p o ing he main esul on he con e gence o he ic i ious play algo i hm, we i s ecall
om [34] some es ima es (on he linea pa o he payo unc ional, as well as on he admissible
measu es), which will be needed in he p oo o Theo em 4.5 below.
Lemma 4.2. (Es ima es on dis ances) Fo all n≥1, we ha e he ollowing es ima es:
(i) The e exis s a cons an C1>0such ha
|dM
1( ¯mn+1,¯mn)| ≤ C1
n,|d1(¯µn+1,¯µn)| ≤ C1
n.
(ii) The e exis s a cons an C2>0(independen o n) such ha o all ∈[0, T ]:
Z[0,T ]×R
(1 + |x|p)|¯µn+1 −¯µn|(d , dx)≤C2
n,Z[0,T ]×R
(1 + |x|p)|¯mn+1
−¯mn
|(d , dx)≤C2
n.(4.1)
Lemma 4.3 (Es ima es on he ewa ds).The e exis cons an s C >0and Cg>0such ha o
all n≥1
⟨ ( ¯mn+1)− ( ¯mn), mn+2 −mn+1⟩ ≤ C
ndM
1(mn+1, mn+2),
⟨g(¯µn+1)−g(¯µn), µn+2 −µn+1⟩ ≤ Cg
nd1(µn+1, µn+2).
Lemma 4.4. (P oximi y be ween wo successi e bes esponses) The ollowing con e gence esul s
hold:
lim
n→∞ d1(µn, µn+1)=0,lim
n→∞ dM
1(mn, mn+1)=0.(4.2)
P oo . We ollow he same a gumen s as in [34] by using he con inuous map Γ◦Rλ om (M, dM
1⊗d1)
o (M, dM
1⊗d1).□
We now gi e he main con e gence esul o his sec ion.
Theo em 4.5. Unde Assump ions 2.1,3.7 and 4.1, o any ini ializa ion (¯
ξ0,¯m0,¯µ0), he sequence
(¯
ξn,¯mn,¯µn)nde ined in Algo i hm 1 con e ges o he unique equilib ium dis ibu ion (ˆ
ξλ,ˆmλ,ˆµλ).
P oo . The p oo is o ganized in wo s eps.
S ep 1. We i s in oduce he exploi abili y e o s (εn)n≥1which a e de ined as ollows:
εn:= Jλ(ξn+1,¯mn,¯µn)−Jλ(¯
ξn,¯mn,¯µn)
=⟨ ( ¯mn), mn+1 −¯mn⟩+⟨g(¯µn), µn+1 −¯µn⟩+λEZT
0
(E(ξn+1
)− E(¯
ξn
))d .(4.3)
By he op imali y o ξn+1, i is easy o obse e ha εn≥0, o all n≥1.
16
We w i e εn+1 −εn=ε(1)
n+ε(2)
n, whe e
ε(1)
n:= ⟨ ( ¯mn),¯mn⟩+⟨g(¯µn),¯µn⟩−⟨ ( ¯mn+1),¯mn+1⟩−⟨g(¯µn+1),¯µn+1⟩
+λEZT
0
(E(¯
ξ n)− E(¯
ξ
n+1))d ,(4.4)
ε(2)
n:= ⟨ ( ¯mn+1), mn+2⟩+⟨g(¯µn+1), µn+2⟩−⟨ ( ¯mn), mn+1⟩−⟨g(¯µn), µn+1⟩
+λEZT
0
(E(ξn+2
)− E(ξn+1
)))d .(4.5)
In he ollowing C > 0deno es a cons an (independen o n) ha may a y om line o line. We
i s es ima e ε(1)
n. Fi s , by he conca i y o he en opy unc ional E(·), we ge
EZT
0E(¯
ξn
)− E(¯
ξn+1
)d =EZT
0E(¯
ξn
)− E n
n+ 1 ¯
ξn
+1
n+ 1ξn+1
)d
≤1
n+ 1EZT
0E(¯
ξn
)− E(ξn+1
)d .(4.6)
Then, by using he es ima es om Lemma 4.3, we ha e
−1
n+ 1⟨ ( ¯mn+1)− ( ¯mn), mn+1 −¯mn⟩ ≤ 1
n+ 1⟨| ( ¯mn+1)− ( ¯mn)|, mn+1 + ¯mn⟩
≤C
n2.
We deduce ha
⟨ ( ¯mn),¯mn⟩−⟨ ( ¯mn+1),¯mn+1⟩=⟨ ( ¯mn),¯mn⟩
− ⟨ ( ¯mn+1),¯mn+1
n+ 1(mn+1 −¯mn)⟩
=⟨ ( ¯mn)− ( ¯mn+1),¯mn⟩ − 1
n+ 1⟨ ( ¯mn+1), mn+1 −¯mn⟩
≤ ⟨ ( ¯mn)− ( ¯mn+1),¯mn⟩ − 1
n+ 1⟨ ( ¯mn), mn+1 −¯mn⟩+C
n2.
Simila o he abo e inequali y, we also de i e
⟨g(¯µn),¯µn⟩−⟨g(¯µn+1),¯µn+1⟩
≤ ⟨g(¯µn)−g(¯µn+1),¯µn⟩ − 1
n+ 1⟨g(¯µn), µn+1 −¯µn⟩+C
n2.
The e o e, by combining he las h ee inequali ies, we de i e
ε(1)
n=⟨ ( ¯mn),¯mn⟩+⟨g(¯µn),¯µn⟩−⟨ ( ¯mn+1),¯mn+1⟩−⟨g(¯µn+1),¯µn+1⟩
+λEZT
0
(E(¯
ξ n)− E(¯
ξ
n+1))d
≤ ⟨ ( ¯mn)− ( ¯mn+1),¯mn⟩+⟨g(¯µn)−g(¯µn+1),¯µn⟩ − εn
n+ 1 +C
n2.
We shall now p oceed wi h he es ima ion o ε(2)
n. Fi s no ice ha , since
Jλ(ξn+1,¯mn,¯µn)≥Jλ(ξ, ¯mn,¯µn),∀ξ∈ A,(4.7)
17
i ollows ha
⟨ ( ¯mn), mn+1⟩+⟨g(¯µn), µn+1⟩+λEZT
0
E(ξn+1
)d
≥ ⟨ ( ¯mn), mn+2⟩+⟨g(¯µn), µn+2⟩+λEZT
0
E(ξn+2
)d .(4.8)
We he e o e ha e
ε(2)
n=⟨ ( ¯mn+1), mn+2⟩+⟨g(¯µn+1), µn+2⟩+λEZT
0
(E(ξn+2
)
− ⟨ ( ¯mn), mn+1⟩−⟨g(¯µn), µn+1⟩ − λEZT
0
E(ξn+1
))d
≤ ⟨ ( ¯mn+1), mn+2⟩+⟨g(¯µn+1), µn+2⟩−⟨ ( ¯mn), mn+2⟩−⟨g(¯µn), µn+2⟩
=⟨ ( ¯mn+1)− ( ¯mn), mn+2⟩+⟨g(¯µn+1)−g(¯µn), µn+2⟩
=⟨ ( ¯mn+1)− ( ¯mn), mn+1⟩+⟨g(¯µn+1)−g(¯µn), µn+1⟩
+⟨ ( ¯mn+1)− ( ¯mn), mn+2 −mn+1⟩+⟨g(¯µn+1)−g(¯µn), µn+2 −µn+1⟩.
By using he es ima es o Lemma 4.3, we ob ain
⟨ ( ¯mn+1)− ( ¯mn), mn+2 −mn+1⟩ ≤ C
ndM
1(mn+1, mn+2),
⟨g(¯µn+1)−g(¯µn), µn+2 −µn+1⟩ ≤ Cg
nd1(µn+1, µn+2).
The e o e
ε(2)
n≤ ⟨ ( ¯mn+1)− ( ¯mn), mn+1⟩+⟨g(¯µn+1)−g(¯µn), µn+1⟩
+C
n(dM
1(mn+1, mn+2) + d1(µn+1, µn+2)).
We se
δn:= CdM
1(mn+1, mn+2) + d1(µn+1, µn+2) + 1
n,
and we de i e
εn+1 −εn=ε(1)
n+ε(2)
n≤ ⟨ ( ¯mn)− ( ¯mn+1),¯mn⟩+⟨g(¯µn)−g(¯µn+1),¯µn⟩ − εn
n+ 1
+⟨ ( ¯mn+1)− ( ¯mn), mn+1⟩+⟨g(¯µn+1)−g(¯µn), µn+1⟩+δn
n
=⟨ ( ¯mn+1)− ( ¯mn), mn+1 −¯mn⟩+⟨g(¯µn+1)−g(¯µn), µn+1 −¯µn⟩ − εn
n+ 1 +δn
n
= (n+ 1) ⟨ ( ¯mn+1)− ( ¯mn),¯mn+1 −¯mn⟩+⟨g(¯µn+1)−g(¯µn),¯µn+1 −¯µn⟩
−εn
n+ 1 +δn
n
≤ − εn
n+ 1 +δn
n,
whe e he las inequali y comes om he Las y–Lions mono onici y condi ion. Obse e ha δn→0
by Lemma 4.4. By Lemma 3.1 in [43], we conclude ha εn→0as n→ ∞. The esul ollows.
S ep 2. We show now ha he sequence (¯
ξn,¯mn,¯µn)con e ges o he unique λ-SC-MFG equi-
lib ium (ˆ
ξλ,ˆmλ,ˆµλ).By compac ness o he se A × Γ(A), om any subsequence (¯
ξkn,¯mkn,¯µkn)
we can sub ac a u he subsequence (no elabeled) such ha (¯
ξkn,¯mkn,¯µkn)con e ges in he
18
opology τw
2⊗dp⊗Wp o some (¯
ξ, ¯m, ¯µ), whe e τw
2 ep esen s he opology associa ed o he weak
con e gence in L2(Ω ×[0, T]). Fi s , by using simila a gumen s as o he p oo o (3.9), we can
es ablish ha
( ¯m, ¯µ) = Γ(¯
ξ).(4.9)
Then, by op imali y o ξkn+1
·, we ge
Jλ(ξkn+1 ,¯mkn,¯µkn)≥Jλ(ξ, ¯mkn,¯µkn), o all ξ∈ A.(4.10)
By de ini ion o εkn, we de i e
Jλ(¯
ξkn,¯mkn,¯µkn)≥Jλ(ξ, ¯mkn,¯µkn)−εkn, o all ξ∈ A.(4.11)
By using simila a gumen s as he ones used o es ablish (3.6), (3.7) and (3.8) and by using he
con e gence o (εn) o 0when n→ ∞, we ge
Jλ(¯
ξ, ¯m, ¯µ)≥Jλ(ξ, ¯m, ¯µ), o all ξ∈ A.(4.12)
F om he abo e inequali y and ela ion (4.9), we conclude ha (¯
ξ, ¯m, ¯µ)is a λ-SC-MFG equilib ium.
By uniqueness o he equilib ium, i coincides wi h (ˆ
ξλ,ˆmλ,ˆµλ). Since om any subsequence o
(¯
ξn,¯mn,¯µn)we can sub ac a u he subsequence which con e ges o he unique λ-SC-equilib ium,
we conclude ha he whole sequence con e ges o he unique MFG equilib ium. □
4.2. Fic i ious play in he supe modula case. We now p o ide con e gence o he ic i ious
play algo i hm when he so-called supe modula i y condi ion holds ue (see, e.g., [27,29,64]). Such
a p ope y na u ally appea s in many inancial and economic applica ions, as o example in models
o bank uns (see [13] and Example 4.7 below). The in e es ed eade is e e ed o he ex book
[63] o u he de ails on supe modula games.
We hus en o ce he ollowing s uc u al condi ion:
Assump ion 4.6 (Supe modula i y).The ollowing hold ue:
(1) Fo any (m, µ)∈ M,Rλ(m, µ)is a single on.
(2) Fo each ( , x)∈[0, T ]×Rd, he unc ion ( , x, ·)is nondec easing wi h espec o he
measu e a gumen in he ollowing sense: o any wo measu es m, ¯m∈ Psub
p(Rd), i m(A)≤
¯m(A) o all A∈ B(Rd), hen
( , x, m)≤ ( , x, ¯m).
(3) We ha e g(x, µ) = ˜g(x, ⟨ψ, µ⟩), o some unc ions ˜g∈ C2(Rd×R)and ψ∈ C1,2([0, T]×Rd)
such ha (∂ +L)ψ≥0and y7→ L˜g(x, y)is nondec easing o any x∈Rd.
Fo he sake o illus a ion, we discuss some key examples.
Example 4.7. An example in which he supe modula i y condi ion is sa is ied is when he unc ion
gdoes no depend on µ(in his case, Condi ion (3)abo e is no needed) and ( , x, m) = 0( , x, 1−
m(Rd)), wi h 0( , x, ·)noninc easing o any , x. This se ing ep esen s supe modula MFGs wi h
equilib ium condi ion
ˆτ∈a g max
τ
EZτ
0
0( , X ,ˆm )d +g(Xτ)and ˆm =P[ˆτ≤ ],
which a e also e e ed o as MFGs o iming. The supe modula i y condi ion is ypical o models o
bank uns [13]. Na u al examples a e o addi i e ype 0( , x, y) = 1(x)− 2(y)o o mul iplica i e
ype 0( , x, y) = − 1(x) 2(y)(wi h 1≥0), o nondec easing unc ions 1, 2.
19
Example 4.8. The unde lying p ocess e ol es as a one dimensional geome ic B ownian mo ion
dZ =Z (b0d +σ0dW ), Z0=z0>0,P-a.s.,
wi h b0∈R,σ0>0, and he payo s a e
( , z, m) = ZR
(z+y)m(dy)and g( , z, µ) = ZR×[0,T ]
( +s)µ(ds, dy).
Wi h espec o he p e ious no a ion, he s a e p ocess becomes X := ( , Z )so ha ime-dependence
in gis allowed.
Be o e discussing he main esul o his subsec ion, we ecall ha he con ols ξa e in e p e ed
as p obabili y measu es; in pa icula , ξ is he p obabili y o s opping be o e ime . In his sense,
i ξ ≥¯
ξ o any ∈[0, T ],P-a.s., hen τξ≤τ¯
ξP-a.s., and we say ha ξis ea lie han ¯
ξ(o ha
¯
ξis la e han ξ). Gi en wo equilib ia (ξ, m, µ)and (¯
ξ, ¯m, ¯µ), we say ha he i s equilib ium is
ea lie ( esp. la e ) ha he second one i ξis ea lie ( esp. la e ) ha ¯
ξ. Gi en λ≥0, he ea lies
equilib ium (ξλ, mλ, µλ)and he la es equilib ium (ξλ, mλ, µλ)a e such ha ξλ
≥ξ ≥ξλ
o any
∈[0, T],P-a.s., o any o he equilib ium (ξ, m, µ).
The ollowing heo em discusses he exis ence and app oxima ion o he ea lies and la es equi-
lib ia.
Theo em 4.9. Unde Assump ions 2.1 and 4.6, he ea lies equilib ium (ξλ, mλ, µλ)and he la es
equilib ium (ξλ, mλ, µλ)exis . Mo eo e , he ollowing s a emen s hold ue:
(1) Wi h ini ializa ion ¯
ξ0
= 1 o any ∈[0, T ], he sequence ¯
ξnis noninc easing, ¯
ξn+1 ≤¯
ξn,
and (¯
ξn,¯mn,¯µn)con e ges o he ea lies equilib ium (ξλ, mλ, µλ).
(2) Wi h ini ializa ion ¯
ξ0
= 0 o any ∈[0, T ], he sequence ¯
ξnis nondec easing, ¯
ξn+1 ≥¯
ξn,
and (¯
ξn,¯mn,¯µn)con e ges o he la es equilib ium (ξλ, mλ, µλ).
P oo . We limi ou sel o show he claim on he exis ence and app oxima ion o he ea lies equi-
lib ium, as he p oo o he la es equilib ium ollows by he same a gumen . The p oo is di ided
in o h ee s eps.
S ep 1. In his s ep we show he some mono onici y p ope ies o he maps Γand Rλ.
We i s show an i-mono onici y p ope y o he consis ency map Γ. Obse e ha , i ξ ≤ξ′
,∀ ∈
[0, T],P-a.s., om he de ini ion o mξ, mξ′ha e
mξ
(A) = E[1A(X )(1 −ξ )] ≥E[1A(X )(1 −ξ′
)] = mξ′
(A),∀A∈ B(Rd), ∈[0, T].
Mo eo e , o ψas in Assump ion 4.6, by using in eg a ion by pa s and ha ξT= 1, we ha e
⟨ψ, µξ⟩:= EZ[0,T ]
ψ( , X )dξ =Eψ(T, XT)−ZT
0
ξ (∂ +L)ψ( , X )d ,
and he analogous exp ession can be ob ained o ξ′. Thus, since (∂ +L)ψ≥0, we ind
⟨ψ, µξ⟩=Eψ(T, XT)−ZT
0
ξ (∂ +L)ψ( , X )d
≥Eψ(T, XT)−ZT
0
ξ′
(∂ +L)ψ( , X )d =⟨ψ, µξ′⟩.
To summa ize, we ha e ound ha
(4.13) i ξ ≤ξ′
,∀ , P-a.s., hen mξ
(A)≥mξ′
(A),∀A, , and ⟨ψ, µξ⟩ ≥ ⟨ψ, µξ′⟩,
which is he desi ed an i-mono onici y o Γ.
20
We nex show he an i-mono onici y o he bes eply map Rλ; ha is, o any m, m′,
(4.14) i m (A)≥m′
(A),∀A, , and ⟨ψ, µ⟩ ≥ ⟨ψ, µ′⟩, hen Rλ(m, µ) ≤Rλ(m′, µ′) ,∀ , P-a.s.
To see his, se ξ:= Rλ(m, µ), ξ′:= Rλ(m′, µ′), and de ine he p ocesses
ξ∧ξ′:= min{ξ , ξ′
} and ξ∨ξ′:= max{ξ , ξ′
} .
Fo a gene ic p ocess ζ∈ A, using in eg a ion by pa s o he in eg al in dζ, we ind
Jλ(ζ, m, µ) = E−ZT
0 ( , X , m ) + Lg(X , µ)ζ d +ZT
0 ( , X , m ) + λE(ζ )d +g(XT, µ),
and, de ining ˆ
( , x, m, µ) = ( , x, m) + Lg(x, µ), we ew i e he la e exp ession as
Jλ(ζ, m, µ) = E−ZT
0
ˆ
( , X , m , µ)ζ d +ZT
0 ( , X , m ) + λE(ζ )d +g(XT, µ).
Now, no icing ha ξ′
−ξ ∧ξ′
=ξ ∨ξ′
−ξ and ha E(ξ ∨ξ′
)− E(ξ′
) = E(ξ )− E(ξ ∧ξ′
), we i s
ind
Jλ(ξ∨ξ′, m′, µ′)−Jλ(ξ′, m′, µ′)
=EZT
0ˆ
( , X , m′
, µ′)ξ′
−ξ ∨ξ′
+λ(E(ξ ∨ξ′
)− E(ξ′
))d
=EZT
0ˆ
( , X , m′
, µ′)ξ ∧ξ′
−ξ +λ(E(ξ )− E(ξ ∧ξ′
))d .
Thus, i m (A)≥m′
(A),∀A, , and ⟨ψ, µ⟩≥⟨ψ, µ′⟩, om Condi ions 2and 3in Assump ion 4.6 we
ha e
Jλ(ξ∨ξ′, m′, µ′)−Jλ(ξ′, m′, µ′)
≥EZT
0ˆ
( , X , m , µ )ξ′
∧ξ −ξ +λ((E(ξ )− E(ξ ∧ξ′
))d
=Jλ(ξ, m, µ)−Jλ(ξ∧ξ′, m, µ).
Hence, om he op imali y o ξ′ o (m′, µ′)and he ac ha ξ∨ξ′∈ A, we deduce ha
0≥Jλ(ξ∨ξ′, m′, µ′)−Jλ(ξ′, m′, µ′)≥Jλ(ξ, m, µ)−Jλ(ξ∧ξ′, m, µ).
This in u n implies ha ξ∧ξ′=Rλ(m, µ), and, by uniqueness o he op imize as in Condi ion 1
in Assump oin 4.6, we conclude ha ξ∧ξ′=ξ, hus p o ing (4.14).
S ep 2. In his s ep we show he mono onici y o ¯
ξnby an induc ion a gumen . To simpli y he
no a ion, we will w i e m≥m′ins ead o m (A)≥m′
(A),∀A, . Mo eo e , (mk, µk) := Γ(ξk), so
ha (due o he linea dependence o he measu e (m, µ)wi h espec o ξ) we ha e he ela ion
(¯µn,¯mn) = 1
n
n
X
k=1
(µξk, mξk) = 1
n
n
X
k=1
(µk, mk),
ha will be used se e al imes in he sequel.
We p oceed wi h an induc ion a gumen . Since ¯
ξ0≡1and ( ¯m0,¯µ0)=(mξ0, µξ0), we ha e
ξ1=Rλ( ¯m0,¯µ0)≤¯
ξ0,
(m1, µ1)=(mξ1, µξ1),
¯m1=m1=mξ1≥m¯
ξ0= ¯m0,
⟨ψ, ¯µ1⟩=⟨ψ, µ1⟩=⟨ψ, µξ1⟩≥⟨ψ, µ¯
ξ0⟩=⟨ψ, ¯µ0⟩.
21
Hence, by he mono onici y o Rλin (4.14) and o Γin (4.13), we ha e
ξ2=Rλ( ¯m1,¯µ1)≤Rλ( ¯m0,¯µ0) = ξ1,
m2=mξ2≥mξ1=m1,
⟨ψ, µ2⟩=⟨ψ, µξ2⟩≥⟨ψ, µξ1⟩=⟨ψ, µ1⟩,
¯m2=1
2(m2+m1)≥m1= ¯m1,
⟨ψ, ¯µ2⟩=⟨ψ, 1
2(µ2+µ1)⟩≥⟨ψ, µ1⟩≥⟨ψ, ¯µ0⟩.
Assume ha o a gene ic nwe ha e
ξn≤ξn−1≤ · · · ≤ ξ1,
mn=mξn≥mn−1≥ · · · ≥ m1,
⟨ψ, µn⟩=⟨ψ, µξn⟩≥⟨ψ, µn−1⟩ ≥ · · · ≥ ⟨ψ, µ1⟩,
¯mn=1
n
n
X
k=1
mk≥¯mn−1≥ · · · ≥ ¯m1,
⟨ψ, ¯µn⟩=⟨ψ, 1
n
n
X
k=1
µk⟩≥⟨ψ, ¯µn−1⟩ ≥ · · · ≥ ⟨ψ, ¯µ1⟩.
Then, he mono onici y o Rλin (4.14) and o Γin (4.13), again gi es
ξn+1 =Rλ( ¯mn,¯µn)≤Rλ( ¯mn−1,¯µn−1) = ξn,
mn+1 =mξn+1 ≥mξn=mn≥ · · · ≥ m1,
⟨ψ, µn+1⟩=⟨ψ, µξn+1 ⟩≥⟨ψ, µξn⟩=⟨ψ, µn⟩ ≥ · · · ≥ ⟨ψ, µ1⟩.
Mo eo e , om he las wo inequali ies we deduce ha
mn+1 ≥¯mn=1
n
n
X
k=1
mkand ⟨ψ, µn+1⟩ ≥ ⟨ψ, ¯µn⟩=⟨ψ, 1
n
n
X
k=1
µk⟩,
which allows o conclude ha
¯mn+1 =1
n+ 1(mn+1 +n¯mn)≥¯mn,
⟨ψ, ¯µn+1⟩=⟨ψ, 1
n+ 1(µn+1 +n¯µn)⟩ ≥ ⟨ψ, ¯µn⟩,
hus comple ing he induc ion a gumen .
Finally, he mono onici y o he sequence ¯
ξn ollows om he mono onici y o ξn.
S ep 3. We nex de ine ( hanks o he mono onici y o ξnand ¯
ξn) he limi poin s
ξ:= in
nξn= in
n
¯
ξn, m := mξ,and µ:= µξ,
and show ha hese coincide wi h he minimal equilib ia (ξλ, mλ, µλ).
No ice ha he sequence (¯
ξn)ncon e ges s ongly. Mo eo e , by mono onici y, i is easy o see
ha he consis ency map Γis con inuous along his sequence. The e o e, we ha e
(m, µ) = (mξ, µξ) = Γ(ξ) = lim
nΓ(¯
ξn) = lim
n( ¯mn,¯µn),
22
so ha he sequence ( ¯mn,¯µn)ncon e ges o (m, µ)as well. Hence, he con inui y o Rλin u n
implies ha
ξ= lim
nRλ( ¯mn,¯µn) = Rλ(m, µ),
which p o es he op imali y. This show ha (ξ, m, µ)is a mean- ield equilib ium.
Finally, we show he minimali y o (ξ, m, µ). Le (ˆ
ξ, ˆm, ˆµ)be ano he mean- ield equilib ium. By
de ini ion o ¯
ξ0, we ha e ˆ
ξ≤¯
ξ0. Hence, by mono onici y o Γin (4.13), we ha e mˆ
ξ= ˆm≥m1=
mξ1= ¯m1and ⟨ψ, µˆ
ξ⟩=⟨ψ, ˆµ⟩≥⟨ψ, µ1⟩=⟨ψ, µξ1⟩=⟨ψ, ¯µ1⟩. Hence, using he mono onici y o
Rλso ha ˆ
ξ=Rλ( ˆm, ˆµ)≤Rλ( ¯m1,¯µ1) = ξ2by mono onici y o Rλ. P oceeding by induc ion, we
conclude ha ˆ
ξ≤¯
ξn,∀n,
so ha ˆ
ξ≤ξ= in n¯
ξn, hus gi ing he desi ed minimali y. □
Appendix A. Connec ion be ween he λ-singula con ol MFG p oblem and a
λ-op imal s opping MFG p oblem
In his pa , we p esen he connec ion be ween he singula mean- ield p oblem λ-SC-MFG and a
new ype o OS-MFG p oblems, ha we in oduce below and we call λ-OS-MFG (λ-op imal s opping
MFG). Le ¯
Tdeno e he se o Fx0,W,U -s opping imes wi h alues in [0, T ], whe e he il a ion
Fx0,W,U has been in oduced in Sec ion 2.3. Fo ease o exposi ion, we es ic ou sel es o he
ela i e en opy E(z) = −zlog z; he analysis o a gene al en opy unc ion sa is ying Assump ion
2.3 is analogous.
De ini ion A.1. A iple ( ˆmλ,ˆµλ,ˆ
θλ)is said o be an equilib ium o he λ-OS-MFG p oblem i :
ˆ
θλ∈a g maxθ∈¯
TEhRθ
0 ( , X ,ˆmλ
)−λ(1 + log U)d +g(Xθ,ˆµλ)i
ˆmλ
(A) = PhX ∈A, ≤ˆ
θλi, A ∈ B(Rd)
ˆµλ(B×[0, ]) = PhXˆ
θ∈B, ˆ
θλ≤ iB∈ B(Rd).
Obse e ha , o any s opping ime θ∈¯
T, since θis a Fx0,W,U -s opping ime, hen he e exis s
a measu able unc ion, s ill deno ed by θ, such ha θ=θ(x0, W, U).
We now es ablish he i s ela ion be ween he wo game p oblems.
Theo em A.2. Le ( ˆmλ,ˆµλ,ˆ
ξλ)be an equilib ium o he λ-SC-MFG. De ine
ˆ
θλ:= θλ(·,1−U),(A.1)
whe e
θλ(·, u) := in { ≥0|ˆ
ξλ
≥u}.(A.2)
Then ( ˆmλ,ˆµλ,ˆ
θλ)is an equilib ium o he λ-OS-MFG .
P oo . Le ( ˆmλ,ˆµλ,ˆ
ξλ)be an λ-SC-MFG equilib ium and le ˆ
θλbe gi en by (A.1). Obse e ha
{ < ˆ
θλ}={ˆ
ξλ
<1−U}.(A.3)
In iew o his obse a ion, we ob ain:
E"Zˆ
θλ
0
( , X ,ˆmλ
)d #=EZT
0
( , X ,ˆmλ
)1{ <ˆ
θλ}d =EZT
0
( , X ,ˆmλ
)P(U < 1−ˆ
ξλ
|Fx0,W
)d
=EZT
0
( , X ,ˆmλ
)(1 −ˆ
ξλ
)d 23
