Inverse Optimization: Theory, Methods, and Practical Applications

FINAL DEGREE PROJECT
In e se Op imiza ion: Theo y,
Me hods, and P ac ical Applica ions
P esen ed by:
Jai o Escánez Ga cía
Supe ised by:
DR. EMILIO CARRIZOSA PRIEGO
FACULTY OF MATHEMATICS
S a is ics and Ope a ional Resea ch Depa men
Se illa, June 2025
Con en s
Abs ac 11
Resumen 13
1 In oduc ion 15
1.1 Con en s basic desc ip ion . . . . . . . . . . . . . . . . . . . . . . . 15
1.2 No a ion................................. 15
1.3 Gene alconcep s ............................ 16
1.3.1 The Fo wa d P oblem . . . . . . . . . . . . . . . . . . . . . 16
1.3.2 Op imal Solu ion Se . . . . . . . . . . . . . . . . . . . . . . 17
1.3.3 The In e se P oblem . . . . . . . . . . . . . . . . . . . . . . 17
2 Classical In e se Op imiza ion 19
2.1 Classical In e se Op imiza ion Pa adigm . . . . . . . . . . . . . . . . 20
2.2 Linea Models.............................. 20
2.2.1 Es ima ing he Objec i e Vec o . . . . . . . . . . . . . . . . 21
2.2.2 Es ima ing he Cons ain s Pa ame e s . . . . . . . . . . . . . 22
2.2.2.1 Es ima ing he Cons ain Ma ix . . . . . . . . . . 22
2.2.2.2 Es ima ing Bo h he Cons ain Ma ix and he Righ -
Hand-Side Vec o . . . . . . . . . . . . . . . . . . 23
2.2.3 Join ly Es ima ing he Objec i e Vec o and he Cons ain Pa-
ame e s............................. 23
2.3 Conic and Con ex Models . . . . . . . . . . . . . . . . . . . . . . . 24
2.4 Pa ially Cons ained In e se P oblems . . . . . . . . . . . . . . . . . 25
2.5 In e se Op imal Value . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3 Da a-D i en In e se Op imiza ion 27
3.1 Dis ance F om he Op imal Solu ion Se . . . . . . . . . . . . . . . . 28
3.2 Subop imali y o he Da ase . . . . . . . . . . . . . . . . . . . . . . 32
3.2.1 Absolu e Subop imali y Loss Func ion . . . . . . . . . . . . . 32
3.2.1.1 Es ima ing he Objec i e Vec o o a Linea Model . 32
3
4 Con en s
3.2.1.2 Es ima ing he Cons ain s Pa ame e s o a Linea
Model ........................ 33
3.2.2 Rela i e Subop imali y Loss Func ion . . . . . . . . . . . . . 34
3.2.2.1 Es ima ing he Objec i e Vec o o a Linea Model . 34
3.2.2.2 Join ly Es ima ing he Fo wa d Model Pa ame e s o
aLinea Model ................... 35
3.3 Viola ing he KKT Condi ions . . . . . . . . . . . . . . . . . . . . . 36
4 Deep In e se Op imiza ion 37
4.1 In oduc ion............................... 37
4.2 Deep Lea ning F amewo k o IO . . . . . . . . . . . . . . . . . . . 38
4.3 Discussion................................ 40
5 Implemen a ion and compu a ional expe imen s 41
5.1 Neu al Ne wo k Model . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 Deep In e se Op imiza ion . . . . . . . . . . . . . . . . . . . . . . . 42
5.3 E alua ionMe ics ........................... 43
5.4 Linea P og am wi h Dynamic Feasible Regions . . . . . . . . . . . . 43
5.4.1 Fo wa d Op imiza ion P oblem . . . . . . . . . . . . . . . . 43
5.4.2 Model Desc ip ion . . . . . . . . . . . . . . . . . . . . . . . 45
5.4.2.1 Neu al Ne wo k Model . . . . . . . . . . . . . . . 45
5.4.2.2 Deep In e se Op imiza ion Algo i hm . . . . . . . 46
5.4.3 Resul s ............................. 46
5.5 Linea P og am wi h Fixed Feasible Regions . . . . . . . . . . . . . . 49
5.5.1 Fo wa d Op imiza ion P oblem . . . . . . . . . . . . . . . . 49
5.5.2 Model Desc ip ion . . . . . . . . . . . . . . . . . . . . . . . 50
5.5.2.1 Neu al Ne wo k Model . . . . . . . . . . . . . . . 50
5.5.2.2 Deep In e se Op imiza ion Algo i hm . . . . . . . 51
5.5.3 Resul s ............................. 51
5.6 Conclusions............................... 52
A Linea P og am wi h Dynamic Feasible Regions 55
A.1 Pa 1: da a ob aining and p ep ocessing . . . . . . . . . . . . . . . . 55
A.2 Pa 2: model de ini ion and aining . . . . . . . . . . . . . . . . . . 58
A.2.1 E alua ion o he model on he es se . . . . . . . . . . . . . 68
A.3 Pa 3:DeepIOmodel ......................... 71
B Linea P og am wi h Fixed Feasible Regions 79
B.1 Pa 1: da a ob aining and p ep ocessing . . . . . . . . . . . . . . . . 79
B.2 Pa 2: model de ini ion and aining . . . . . . . . . . . . . . . . . . 84
B.2.1 E alua ion o he model on he es se . . . . . . . . . . . . . 94
B.3 Pa 3:DeepIOmodel ......................... 97
Con en s 5
Bibliog aphy 102

6 Con en s
Lis o Figu es
1.1 Taxonomy o In e se Op imiza ion Models . . . . . . . . . . . . . . 16
3.1 Example o discon inui y on he objec i e unc ion. . . . . . . . . . . 30
5.1 Neu al Ne wo k Model . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.2 Linea P og am wi h Dynamic Feasible Regions . . . . . . . . . . . . 44
5.3 Neu al Ne wo k Model 1 . . . . . . . . . . . . . . . . . . . . . . . . 46
5.4 DeepIOsolu ions............................ 48
5.5 Neu al Ne wo k solu ions . . . . . . . . . . . . . . . . . . . . . . . . 48
5.6 Compa ison o solu ions gene a ed by bo h models. . . . . . . . . . . 48
5.7 Linea P og am wi h Fixed Feasible Regions . . . . . . . . . . . . . . 50
5.8 DeepIOsolu ions............................ 52
5.9 Neu al Ne wo k solu ions . . . . . . . . . . . . . . . . . . . . . . . . 52
5.10 Compa ison o solu ions gene a ed by bo h models. . . . . . . . . . . 52
7
8 Lis o Figu es
Lis o Tables
5.1 Me ics o bo h models in expe imen 1 . . . . . . . . . . . . . . . . 47
5.2 Me ics o bo h models in expe imen 2 . . . . . . . . . . . . . . . . 51
9
16 1.3. Gene al concep s
We use lowe case le e s o deno e scala s. Fo example, kis a scala .
Le in (C)deno e he in e io , and ex (C)deno e he se o ex eme poin s o he
con ex hull o a se C.
The Di ac del a unc ion is deno ed by δx(·).
P obabili y dis ibu ions a e deno ed by P, and he expec a ion ope a o is deno ed by
EP[·].
The empi ical dis ibu ion co esponding o a da ase {ˆxi}N
i=1 is deno ed by PN:=
1
NPN
i=1 δˆxi.
The p-no m o a ec o xis deno ed by ∥x∥p.
1.3 Gene al concep s
In In e se Op imiza ion, we dis inguish be ween wo key o elemen s, he Fo wa d
P oblem and he In e se P oblem, which we will de ine in he ollowing sec ions.
Figu e 1.1: Taxonomy o In e se Op imiza ion Models
1.3.1 The Fo wa d P oblem
The Fo wa d P oblem e e s o he op imiza ion p oblem ha he decision-make
seeks o sol e. As an op imiza ion p oblem, i is de ined by he objec i e unc ion
(x,u,θ)and he easible se X(u,θ), whe e u∈Uand θ∈Θa e gi en pa ame e s
ha con ol he objec i e. The goal is o ind he op imal solu ion x∗ ha minimizes he
objec i e unc ion. The pa ame e u ep esen s he inpu o con ex , cha ac e izing he
o wa d op imiza ion model, while he pa ame e θis he pa ame e o be es ima ed,
e lec ing he decision-make ’s p e e ences.
De ini ion 1.3.1. Fo wa d Op imiza ion P oblem
Le he Fo wa d Op imiza ion P oblem be de ined as
FOP(u,θ) := min
x{ (x,u,θ)|x∈X(u,θ)}.
No a ion 1.3.1. Fo a gi en ui, we deno e he Fo wa d Op imiza ion P oblem as
FOPi(θ) := FOP(ui,θ).

Chap e 1. In oduc ion 17
1.3.2 Op imal Solu ion Se
De ini ion 1.3.2. Op imal Solu ion Se
Fo ixed u∈Uand θ∈Θ, we de ine he Op imal Solu ion Se as
Xop (u,θ) := a g min
x{ (x,u,θ)|x∈X(u,θ)}
This is he se o op imal solu ions o he Fo wa d Op imiza ion P oblem FOP(u,θ)
o ixed uand θ.
No a ion 1.3.2. Fo a gi en ui, we deno e he Op imal Solu ion Se as Xop
i(θ) :=
Xop (ui,θ).
1.3.3 The In e se P oblem
The In e se P oblem e e s o he ask o in e ing he pa ame e s θo he o wa d
op imiza ion model om a da ase o op imal decisions {(ˆxi,ˆui)}N
i=1.
The goal is o ind he pa ame e ec o θ∗ ha bes i s he obse ed decisions, whe e
he deg ee o i ing will be measu ed depending on he app oach used o sol e he
In e se P oblem.
The In e se P oblem can be o mula ed using wo di e en app oaches based on he
obse ed decisions: he Classical In e se Op imiza ion P oblem and he Da a-D i en
In e se Op imiza ion P oblem.
• In he Classical In e se Op imiza ion P oblem, we assume ha he obse ed
decisions a e op imal, meaning ha ˆxiis he op imal solu ion o he Fo wa d
Op imiza ion P oblem FOP(ˆui,θ) o he en i e da ase . This is called Pe ec
Model-Da a Fi .
• In he Da a-D i en In e se Op imiza ion P oblem, we do no equi e he ob-
se ed decisions o be op imal, so we do no assume Pe ec Model-Da a Fi .
Wi h mo e de ail, gi en a da ase , In e se Op imiza ion es ima es a pa ame e ec o
θ∗such ha he agg ega e i o he co esponding o wa d model FOPi(θ∗) o ˆxi
is op imized. An es ima e θ∗is conside ed a pe ec i o ˆxii ˆxi∈Xop
i(θ∗). To
o malize his, we in oduce he concep o he In e se-Feasible-Se .
De ini ion 1.3.3. In e se-Feasible-Se
Le he In e se-Feasible-Se be de ined as
Θin (ˆui,ˆxi) := {θ∈Θ|ˆxi∈Xop (ˆui,θ)}
This is he se o pa ame e s θ ha make he obse ed decision ˆxiop imal unde he
Fo wa d Op imiza ion P oblem FOP(ˆui,θ).
18 1.3. Gene al concep s
Wi h his concep , we can e o mula e he wo app oaches o he In e se P oblem as
ollows.
In he Classical In e se Op imiza ion P oblem, we no only assume ha he obse ed
decisions a e op imal, bu also ha he pa ame e θ ha we a e ying o es ima e is
in e se- easible, meaning ha θ∈Θin (ˆui,ˆxi) o he en i e da ase and minimizes
a speci ic objec i e unc ion h(θ). The Classical In e se Op imiza ion P oblem is
o mula ed as ollows:
min
θ{h(θ)|θ∈Θin (ˆui,ˆxi)∀i= 1,...,N}(1.1)
In ui i ely, we sea ch o he bes pa ame e θ ha minimizes he objec i e unc ion
h(θ)and makes all he obse ed decisions op imal wi hin he se TN
i=1 Θin (ˆui,ˆxi).
On he o he hand, in he Da a-D i en In e se Op imiza ion P oblem, in e se easi-
bili y is no en o ced. We use he concep o a loss unc ion ℓ(xi,Xop
i(θ)) o measu e
he e o o he Fo wa d Op imiza ion Model wi h espec o he obse ed decisions.
The Da a-D i en In e se Op imiza ion P oblem is o mula ed as ollows:
min
θ{k h(θ) + 1
N
N
X
i=1
ℓ(ˆxi,Xop (ˆui,θ))|θ∈Θ}(1.2)
He e, kis a posi i e cons an ha weigh s he impo ance o he objec i e unc ion
h(θ)and he loss unc ion ℓ(ˆxi,Xop (ˆui,θ)).
No a ion 1.3.3. Conside a da ase {(ˆxi,ˆui)}N
i=1 o N≥1decisions. We can al e na-
i ely de ine he ollowing objec s:
i(x,θ) := (x,ˆui,θ)
Xi(θ) := X(ˆui,θ)
Using his no a ion, we can dis inguish be ween di e en ypes o inpu da a:
1. We obse e a da ase o decisions and inpu s {(ˆxi,ˆui)}N
i=1.
2. We obse e a da ase o decisions and o wa d models {(ˆxi,FOPi(θ))}N
i=1.
3. We obse e a da ase o decisions and op imal solu ion se s {(ˆxi,Xop
i(θ))}N
i=1.
Chap e 2
Classical In e se Op imiza ion
In his chap e , we explo e he Classical In e se Op imiza ion Model (1.1), in o-
duced in chap e 1, i s compu a ional challenges, and me hods o sol e speci ic in-
s ances o he p oblem. In de ail, we p esen echniques o linea , conic, con ex, and
pa ially cons ained models, ha each e o mula e he in e se op imiza ion model in o
a ac able op imiza ion p oblem as discussed in [4].
Al hough he e o mula ion echniques discussed in his chap e can sol e any in-
s ances o P oblem (1.1), we ocus on he mos common objec i e unc ion, namely
h(θ) = ∥θ−ˆ
θ∥p, whe e ˆ
θis a ixed ec o known as he e e ence alue. This e e -
ence alue can be de i ed om expe knowledge, his o ical da a, o o he sou ces o
in o ma ion.
Fo ease o exposi ion, we assume a single decision inpu , i.e., n= 1. The echniques
p esen ed in his chap e can be easily ex ended o cases whe e n>1. A single de-
cision inpu implies a single o wa d model, allowing us o omi he con ex ual inpu .
The e o e, we exclude uand he index iin his chap e .
Now, we can ede ine he Classical In e se Op imiza ion Model o his speci ic objec-
i e unc ion.
De ini ion 2.0.1. Classical In e se Op imiza ion Model
Gi en a o wa d op imiza ion model FOP(θ)and a e e ence alue ˆ
θ, he Classical
In e se Op imiza ion Model is de ined as ollows:
IOP −C(ˆx,ˆ
θ) := min
θ{∥θ−ˆ
θ∥p|θ∈Θin (ˆx)}
I is impo an o no e ha he pa ame e o es ima e, θ, is in e se- easible wi h espec
19
20 2.1. Classical In e se Op imiza ion Pa adigm
o he obse ed decision ˆx, as θ∈Θin (ˆx). As men ioned in he in oduc ion, his is a
cha ac e is ic o Classical In e se Op imiza ion.
Nex , we pa icula ize he model o speci ic o wa d op imiza ion p oblems and p o-
ide cha ac e iza ions o he in e se easible se o e o mula e he in e se op imiza ion
p oblem in o a ac able op imiza ion p oblem.
2.1 Classical In e se Op imiza ion Pa adigm
P oposi ion 2.1.1. Gi en a o wa d op imiza ion model FOP(θ)wi h objec i e unc-
ion . Then, ˆx is an op imal solu ion o FOP(θ)i and only i
(ˆx,θ)≤ (x,θ),∀x∈X(θ)
Using 2.1.1, he in e se easible se Θin (ˆx)can be ew i en as:
Θin (ˆx) := {θ∈Θ| (ˆx, θ)≤ (x,θ),∀x∈X(θ)}(2.1)
The ea lies pa adigms in Classical In e se Op imiza ion elied on his cha ac e iza ion
o sol e he in e se op imiza ion p oblem. Howe e , he compu a ional complexi y
o his app oach is high because i equi es enume a ing all easible solu ions o he
o wa d op imiza ion model, which is gene ally imp ac ical. As a esul , al e na i e
and ac able cha ac e iza ions o he in e se easible se ha e been de eloped.
2.2 Linea Models
Conside he linea o wa d op imiza ion model:
FOP −L(θ,Φ,Ψ) := min
x{θTx|Φx ≥Ψ}
whe e he objec i e ec o θ∈Rn, he cons ain ma ix Φ∈Rm×n, and he igh -
hand-side cons ain ec o Ψ∈Rm.
The li e a u e on linea in e se op imiza ion models gene ally ocuses on he es ima-
ion o ei he he objec i e ec o θo he cons ain pa ame e s Φand Ψindepen-
den ly. Tha is, assuming ha he o he pa ame e s a e known:
• Es ima ing he objec i e ec o θ: Le Φ=A,Ψ=b, be he known cons ain
pa ame e s. The in e se op imiza ion p oblem seeks o es ima e he objec i e
ec o θwi h minimum dis ance om he e e ence alue ˆ
θ.
• Es ima ing he cons ain s Φand Ψ: Le θ=cbe he known objec i e ec o ,
whe e c∈Rn. The in e se op imiza ion p oblem sea ches o es ima e he
Chap e 2. Classical In e se Op imiza ion 21
cons ain pa ame e s (Φ,Ψ)o a minimum dis ance om he e e ence alues
(ˆ
Φ,ˆ
Ψ). No e ha his p oblem gene alizes he cases o es ima ing only he
cons an ma ix Φ(i.e., Θ:={(Φ,Ψ)|Ψ=b}) o only he igh -hand-side
ec o Ψ(i.e., Θ := {(Φ,Ψ)|Φ=A}).
No a ion 2.2.1. When i is ob ious om he con ex , we simpli y he no a ion by omi -
ing om FOP −L(θ,Φ,Ψ) he pa ame e s ha a e known.
Fo example, FOP −L(θ):=FOP −L(θ,A,b) = minx{θTx|Ax≥b}.
2.2.1 Es ima ing he Objec i e Vec o
In his sec ion, we show ha he in e se op imiza ion p oblem o a linea p og am can
be e o mula ed as an op imiza ion p oblem wi h linea cons ain s, by cha ac e izing
he in e se easible se Θin (ˆx)using he complemen a y slackness condi ions in linea
p og amming.
P oposi ion 2.2.1. Complemen a y Slackness
I ˆx is an op imal solu ion o FOP −L(θ), hen he e exis s a dual ec o λ∈Rm
sa is ying:
1. Dual Feasibili y: ATλ=θ
2. λ≥0
3. Complemen a y Slackness: (Aˆx −b)Tλ= 0
Using 2.2.1, we can ew i e he in e se easible se as:
Θin (ˆx) := {θ∈Rn| ∃λ∈Rm,λ≥0,ATλ=θ,(Aˆx −b)Tλ= 0}
Thus, he in e se op imiza ion p oblem can be e o mula ed as a linea op imiza ion
p oblem:
IOP −L(ˆx) := min
θ∈Θ∥θ−ˆ
θ∥p
s. .ATλ=θ
(Aˆx −b)Tλ=0
λ≥0
(2.2)
We can also e o mula e he linea in e se op imiza ion p oblem by using he S ong
Duali y p ope y om linea p og amming.
P oposi ion 2.2.2. S ong Duali y
I ˆx is an op imal solu ion o FOP −L(θ), hen he e exis s a dual ec o λ∈Rm
sa is ying:

22 2.2. Linea Models
1. Dual Feasibili y: ATλ=θ
2. λ≥0
3. S ong Duali y: θTˆx =λTb
P oposi ion 2.2.2 allows us o ew i e he second cons ain (Aˆx −b)Tλ=0. And he
new in e se op imiza ion p oblem is:
IOP −L(ˆx) := min
θ∈Θ∥θ−ˆ
θ∥p
s. .ATλ=θ
θTˆx =λTb
λ≥0
(2.3)
A gene aliza ion o his p oblem is p esen ed in chap e 3, whe e he op imali y o he
da ase is no assumed.
2.2.2 Es ima ing he Cons ain s Pa ame e s
Recall ha when es ima ing he cons ain pa ame e s, we assume he objec i e ec o
θis known, i.e., θ=c.
2.2.2.1 Es ima ing he Cons ain Ma ix
Fi s , we assume he igh -hand-side ec o Ψis known, i.e., Ψ=b. The in e se op i-
miza ion p oblem aims o es ima e he cons ain ma ix Φsuch ha i is o minimum
dis ance om he e e ence alue ˆ
Φ.
Using P oposi ion 2.2.2, he in e se easible se can be ew i en as:
Θin (ˆx) := {Φ∈Rm×n| ∃λ∈Rm,λ≥0,ΦTλ=c,cTˆx =λTb}
The in e se op imiza ion p oblem can be e o mula ed as he ollowing op imiza ion
p oblem:
IOP −L(ˆx) := min
Φ∈Rm×n∥Φ−ˆ
Φ∥p
s. .ΦTλ=c
cTˆx =λTb
λ≥0
(2.4)
No e ha p oblem (2.4) con ains a bilinea cons ain , ΦTλ=c, which makes he
p oblem non-con ex, non-linea , and compu a ionally challenging o sol e. Chan and
Kaw [2] p opose a solu ion o his p oblem by obse ing ha o ˆx o be op imal, a
Chap e 2. Classical In e se Op imiza ion 23
leas one cons ain mus sa is y he equali y. Speci ically, an op imal solu ion o he
in e se p oblem can be ob ained by pe u bing he nea es ace {x|ˆ
ΦT
jx≥bj} om
ˆx un il i sa is ies he equali y cons ain .
2.2.2.2 Es ima ing Bo h he Cons ain Ma ix and he Righ -Hand-Side Vec o
We now conside he gene al case whe e bo h he cons ain ma ix Φand he igh -
hand-side ec o Ψa e unknown. The in e se op imiza ion p oblem is o es ima e
he cons ain pa ame e s (Φ,Ψ)wi h minimum dis ance om he e e ence alues
(ˆ
Φ,ˆ
Ψ).
Using P oposi ion 2.2.2, we can ew i e he in e se easible se as:
Θin (ˆx) := {(Φ,Ψ)∈Rm×n×Rm| ∃λ∈Rm,λ≥0,ΦTλ=c,cTˆx =λTΨ}
Obse e ha now he e a e wo bilinea cons ain s ha make he p oblem non-con ex,
non-linea , and compu a ionally di icul o sol e. Because o his, we use he simples
cons ain o o mula e he p oblem:
min
(Φ,Ψ)∈Rm×n×Rm∥(Φ,Ψ)−(ˆ
Φ,ˆ
Ψ)∥p
s. .Φˆx ≥Ψ
2.2.3 Join ly Es ima ing he Objec i e Vec o and he Cons ain
Pa ame e s
In his sec ion, we conside he case whe e bo h he objec i e ec o θand he con-
s ain pa ame e s a e unknown, excep o he cons ain ma ix Φ, which we assume
is known Φ=A. The in e se op imiza ion p oblem is o es ima e he objec i e ec o
θand he igh -hand-side ec o Ψ"such ha hey a e o minimum dis ance om he
e e ence alues (ˆ
θ,ˆ
Ψ).
Conside he linea o wa d op imiza ion model FOP −L(θ,Ψ)wi h Φ=A, and
he objec i e ec o θand he igh -hand-side ec o Ψa e unknown.
Using P oposi ion 2.2.1, we can le e age he complemen a y slackness o mula ion:
min
θ∈Θ∥(θ,Ψ)−(ˆ
θ,ˆ
Ψ)∥p
s. .ATλ=θ
(Aˆx −Ψ)Tλ=0
λ≥0
(2.5)
Again, he complemen a y slackness cons ain is bilinea . To linea ize i , we in oduce
an addi ional a iable z∈ {0,1}m. Thus, P oblem (2.5) can be o mula ed as a mixed-
24 2.3. Conic and Con ex Models
in ege linea op imiza ion p oblem (MILP).
min
θ∈Θ,Ψ∈Rm∥(θ,Ψ)−(ˆ
θ,ˆ
Ψ)∥p
s. .Aˆx −Ψ≥0
Aˆx −Ψ≤Mz
λ≤M(1−z)
ATλ=θ
λ≥0
z∈ {0,1}m
(2.6)
whe e Mis a su icien ly la ge posi i e cons an .
2.3 Conic and Con ex Models
In his sec ion, we conside m≥2,C⊆Rnas a closed con ex cone wi h nonemp y
in e io , such ha {x,−x} ⊆ Cimplies x=0.
The conic o wa d op imiza ion model is:
FOP −CON(θ) := min
x{ (x,θ)| − g(x,θ)∈C}(2.7)
whe e (x,θ)is a con ex unc ion in x o ixed θ, and g(x,θ) = (g1(x,θ), . . . , gm(x,θ))
is a ec o - alued con ex unc ion whose componen s a e each di e en iable and con-
ex.
No e ha se ing C=Rn
+ educes he o wa d p oblem o a classical con ex op imiza-
ion p oblem whe e, −g(x,θ)∈Ci and only i g(x,θ)≥0.
Iyenga and Kang [7] used he KKT condi ions o cha ac e ize he in e se easible
se o he conic o wa d op imiza ion model and de i e a ac able e o mula ion o
IOP −C(ˆx,ˆ
θ).
P oposi ion 2.3.1. KKT Condi ions
A decision ˆx is an op imal solu ion o FOP −CON(θ)i :
1. The e exis s x∈Csuch ha −g(x,θ)∈in (C)
2. The e exis s λ∈Csuch ha ∇x (ˆx, θ)+Pm
j=1 λj∇xgj(ˆx, θ)=0and λjgj(ˆx, θ) =
0,∀j= 1, . . . , m
Using P oposi ion 2.3.1, we can ew i e he in e se easible se as:
Θin (ˆx) := {θ∈Θ| ∃x∈C,−g(x,θ)∈in (C)
∃λ∈C,∇x (ˆx, θ) +
m
X
j=1
λj∇xgj(ˆx, θ)=0
λjgj(ˆx, θ)=0,∀j= 1, . . . , m}
Chap e 2. Classical In e se Op imiza ion 25
Fo a gene al FOP −CON(θ), his se includes bilinea cons ain s in λjand gj(ˆx, θ),
making he co esponding in e se op imiza ion p oblem no easily sol able.
Lemma 2.3.1. Conside he conic o wa d op imiza ion model FOP −CON(θ). As-
sume ha g(x,θ)=g(x)and ha he e exis s xsuch ha −g(x)∈in (C). Then, he
IOP −C(ˆx,ˆ
θ)is equi alen o he ollowing op imiza ion p oblem:
min
θ∈Θ∥θ−ˆ
θ∥p
s. .∇x (ˆx, θ) +
m
X
j=1
λj∇xgj(ˆx) = 0
λjgj(ˆx) = 0,∀j= 1, . . . , m
λ∈C
Fu he mo e, i ∇x (x,θ)is an a ine unc ion o θ, hen he p oblem is con ex.
2.4 Pa ially Cons ained In e se P oblems
In his sec ion, we conside he case whe e he o wa d op imiza ion model is pa ially
cons ained, meaning only a pa o a decision is obse ed.
The inpu o hese in e se p oblems is no a single decision ˆx ∈X, bu a he a se o
decisions ˆ
X:= {x|x∈X,xi=ˆxi,∀i∈M} ⊆ X, whe e M⊆ {1,...,n}is he se
o obse ed indices.
The pa ially cons ained in e se op imiza ion p oblem aims o es ima e a θsuch ha
a leas one decision x∈ˆ
Xis op imal o he o wa d op imiza ion model:
min
θ,x{∥θ−ˆ
θ∥p|x∈ˆ
X,θ∈Θin (x)}(2.8)
Fo example, o he linea o wa d op imiza ion model FOP −L(θ), using P oposi-
ion 2.2.1, P oblem (2.8) can be e o mula ed as:
min
θ∈Θ∥θ−ˆ
θ∥p
s. .Ax≥b
ATλ=θ
(Ax−b)Tλ=0
λ≥0
x∈ˆ
X
32 3.2. Subop imali y o he Da ase
3.2 Subop imali y o he Da ase
In his sec ion, we in oduce wo popula loss unc ions ha measu e he deg ee o sub-
op imali y o he obse ed decisions ˆx unde ou es ima ed o wa d model FOPi(θ).
The li e a u e has ocused on he ollowing wo loss unc ions:
•Absolu e subop imali y loss unc ion: This unc ion measu es he absolu e di -
e ence be ween he objec i e unc ion alue o he obse ed decision and he
op imal decision unde he es ima ed o wa d model.
ℓASO(ˆx,X(θ)) := | (ˆx,θ)−min
x∈X(θ) (x,θ)|
•Rela i e Subop imali y loss unc ion: This unc ion measu es he compe i i e
a io o he objec i e unc ion alue o he obse ed decision and he op imal
decision unde he es ima ed o wa d model.
ℓRSO(ˆx,X(θ)) := 
(ˆx,θ)
minx∈X(θ) (x,θ)−1
Le IOP −DD(ℓASO,PN)and IOP −DD(ℓRSO,PN)be he in e se op imiza ion
da a-d i en p oblems using he Absolu e Subop imali y and Rela i e Subop imali y
loss unc ions, espec i ely.
3.2.1 Absolu e Subop imali y Loss Func ion
Fi s le us conside he special case whe e he obse ed decisions o he da ase a e
known o be easible, his is, ˆxi∈Xi(θ) o all (ˆxi,ˆui)in Dand θin Θ. In his case,
we can emo e he absolu e alue om he loss unc ion and e o mula ing he In e se
Op imiza ion P oblem Da a-D i en as:
IOP −DD(ℓASO,PN) = min
θ∈Θ
1
N
N
X
i=1
( i(ˆxi,θ)− i(xi,θ))
s. .xi∈Xop
i(θ)∀i∈ {1, . . . , N}
3.2.1.1 Es ima ing he Objec i e Vec o o a Linea Model
Now we pa icula ize he linea o wa d model FOP −L(θ)wi hou assuming ea-
sible obse a ions. In e se Absolu e Subop imali y p oblem IOP −DD(ℓASO,PN)
becomes an e icien linea p og am by e o mula ing i using S ong Duali y.

Chap e 3. Da a-D i en In e se Op imiza ion 33
Theo em 3.2.1. Conside FOP −Li(θ). Then he In e se Absolu e Subop imali y
p oblem IOP −DD(ℓASO,PN)is equi alen o he ollowing linea p og am:
min
θ∈Θ
1
N
N
X
i=1
|θTˆxi−bT
iλi|
s. .AT
iλi=θ∀i∈ {1, . . . , N}
λi≥0∀i∈ {1, . . . , N}
(3.4)
The p oo o his heo em is based on applying S ong Duali y o he linea p og am
FOP −Li(θ)as we did in (3.3).
P oblem (3.4) is a gene aliza ion o he In e se Linea Op imiza ion p oblem in o-
duced in sec ion 2.2. I ˆxia e all op imal solu ions, hen he objec i e unc ion alue
will be ze o, and he objec i e can be eplaced wi h cons ain s θTˆxi=bT
iλ o all i,
and he p oblem becomes he o iginal p oblem Equa ion 2.2.
As in P oblem (3.3), he numbe o cons ain s and a iables inc eases wi h he numbe
o obse a ions, making he in e se op imiza ion p oblem compu a ionally expensi e
and ha d o sol e.
Howe e , as in P oblem (3.3), i we assume ha all o he poin s ˆxia e solu ions o
he same o wa d model, i.e., FOP −Li(θ) = FOP −L(θ) o all i, hen he dual
a iable λiis he same o all i, i.e., λi=λ o all i. In his case, we only need o
con ol a single dual a iable ins ead o N, and he p oblem becomes:
min
θ∈Θ
1
N
N
X
i=1
|θTˆxi−bTλ|
s. .ATλ=θ
λ≥0
3.2.1.2 Es ima ing he Cons ain s Pa ame e s o a Linea Model
Now, we ocus on es ima ing pa ame e s in he cons ain s o he o wa d models
FOP −Li(Φ). Recall ha when es ima ing hese pa ame e s, we assume ha he
cos ec o , ci, and he igh -hand side ec o , bi, a e known.
The in e se op imiza ion p oblem is essen ially he same as p e iously men ioned
p oblem, bu wi h he cos ec o and he igh -hand side ec o known and he pa-
ame e s in he cons ain s unknown and in oducing he e m Φxi≥bi o make su e
ha he obse a ions a e easible.
34 3.2. Subop imali y o he Da ase
The p oblem is o mula ed as:
min
Φ∈Rm×n
1
N
N
X
i=1
|cT
iˆxi−bT
iλi|
s. .Φλi=ci∀i∈ {1,...,N}
Φxi≥bi∀i∈ {1,...,N}
λi≥0∀i∈ {1, . . . , N}
This p oblem is a bilinea p og am, due o he i s cons ain , whe e wo a iables a e
mul iplied.
3.2.2 Rela i e Subop imali y Loss Func ion
In gene al, minimizing he Rela i e Subop imali y loss unc ion is di icul o sol e
due o he ac ional componen . Howe e , unde he assump ion ha he o wa d
model sa is ies a scaling in a iance p ope y, i.e., θcan be scaled main aining he same
op imal alue and op imal solu ion se , his ac ional componen can be emo ed.
De ini ion 3.2.1. Scaling In a iance P ope y.
A o wa d model FOP(u,θ)sa is ies he scaling in a iance p ope y i , o all θ∈Θ
and α > 0,x∗∈Xop (θ)implies:
•x∗∈Xop (αθ)
• (x∗, αθ) = α (x∗,θ)
I he o wa d model sa is ies he scaling in a iance p ope y and has a nonnega i e op-
imal alue, hen he pa ame e θcan be scaled o make he op imal alue equal o one
while p ese ing he op imal solu ion se . Thus, he IOP −DD(ℓRSO,PN)p oblem
can be e o mula ed as:
min
θ∈Θ
1
N
N
X
i=1
| i(ˆxi,θ)−1|
s. . i(xi,θ)=1∀i∈ {1,...,N}
xi∈Xop
i(θ)∀i∈ {1, . . . , N}
3.2.2.1 Es ima ing he Objec i e Vec o o a Linea Model
Now, we pa icula ize he linea o wa d model FOP −L(θ)using s ong duali y and
he scaling in a iance p ope y.
Chap e 3. Da a-D i en In e se Op imiza ion 35
Theo em 3.2.2. Conside FOP −Li(θ). I i sa is ies ha bi≥0componen wise o
all i∈ {1, . . . , N}, hen he In e se Rela i e Subop imali y p oblem IOP −DD(ℓRSO,PN)
is equi alen o he ollowing linea p og am:
min
θ∈Θ
1
N
N
X
i=1
|θTˆxi−1|
s. .bT
iλi=1∀i∈ {1, . . . , N}
AT
iλi=θ∀i∈ {1, . . . , N}
λi≥0∀i∈ {1, . . . , N}
This linea p og am su e s he same p oblem as he one in (3.4), he numbe o con-
s ain s and a iables inc eases wi h he numbe o obse a ions, making he p ob-
lem compu a ionally expensi e and ha d o sol e when he numbe o obse a ions is
la ge. Bu when all o he poin s ˆxia e solu ions o he same o wa d model, i.e.,
FOP −Li(θ) = FOP −L(θ) o all i, he in e se p oblem can be simpli ied o:
min
θ∈Θ
1
N
N
X
i=1
|θTˆxi−1|
s. .bTλ= 1
ATλ=θ
λ≥0
3.2.2.2 Join ly Es ima ing he Fo wa d Model Pa ame e s o a Linea Model
Finally, we ocus on join ly es ima ing he cons ain ma ix and he cos ec o o he
o wa d models FOP −L((θ,Φ)) := minx{θTx|Φx ≥b}.
The in e se op imiza ion p oblem is o mula ed simila ly o he p e ious one, excep
eplacing Aiwi h Φi:
min
θ∈Θ,Φi∈Rm×n
1
N
N
X
i=1
|θTˆxi−1|
s. .bT
iλi= 1 ∀i∈ {1, . . . , N}
ΦT
iλi=θ∀i∈ {1, . . . , N}
λi≥0∀i∈ {1, . . . , N}
This p oblem su e s he same issue as he p e ious ones, he numbe o cons ain s
and a iables inc eases wi h he numbe o obse a ions, making he p oblem com-
pu a ionally expensi e and ha d o sol e when he numbe o obse a ions is la ge.
Howe e , i all o he poin s ˆxia e solu ions o he same o wa d model, he in e se
p oblem can be simpli ied.
36 3.3. Viola ing he KKT Condi ions
3.3 Viola ing he KKT Condi ions
In his sec ion, we in oduce ano he popula loss unc ion ha measu es he deg ee o
iola ion o he KKT condi ions (P oposi ion 2.3.1) o he obse ed decisions ˆx unde
ou es ima ed o wa d model FOP −CVX(θ) := minx{ (x,θ)| − g(x,θ)∈C}.
Recall ha , KKT condi ions s a e ha i is necessa y o a poin o be op imal ha
he e exis s a dual a iable and ha bo h, he decision and he dual a iable, sa is y a
s a iona i y and a complemen a y slackness condi ion.
Le (x,λ)be a decision and a dual a iable. We de ine he ollowing loss unc ions
ha measu es he deg ee o iola ion o each KKT condi ion:
•S a iona i y loss unc ion:
ℓs (x,Xop (θ),λ) := 
∇x (x,θ) +
m
X
j=1
λj∇xgj(x,θ)
•Complemen a y Slackness loss unc ion:
ℓcs(x,Xop (θ),λ) := ∥(λ1g1(x,θ), . . . , λmgm(x,θ))∥p
These unc ions a e no s ic ly loss unc ions, as hey a e de ined in e ms o he dual
a iable λ, which is unknown. Howe e , using hem we can de ine he ollowing loss
unc ion wi h which we will wo k wi h in his sec ion:
ℓKKT(ˆx,Xop (θ)) := min
λ≥0ℓs (ˆx,Xop (θ),λ)+ℓcs(ˆx,Xop (θ),λ)
In mo e de ail, he KKT loss unc ion, gi en (ˆx,Xop (θ)), measu es he minimum
deg ee o iola ion o he KKT condi ions o he obse ed decisions ˆx unde ou es i-
ma ed o wa d model.
Obse e ha he KKT loss unc ion is a noncon ex unc ion, as i is de ined as he min-
imum o a noncon ex unc ion, he Complemen a y Slackness loss unc ion. The e-
o e, in p ac ical e ms, o a gi en ˆx, compu ing he KKT loss unc ion equi es
sol ing a noncon ex op imiza ion p oblem. Ne e heless, i we ocus on es ima ing
only he pa ame e s o he objec i e unc ion, hen he in e se op imiza ion p oblem
IOP −DD(ℓKKT,PN)becomes a con ex op imiza ion p oblem due o he ollowing
p oposi ion.
P oposi ion 3.3.1. Conside FOP −CVX(θ). I gdoes no depend on θ, his is
gi(x,θ)=gi(x)∀i∈ {1,...,N}, and (x,θ)is con ex in θ, hen he in e se op i-
miza ion p oblem IOP −DD(ℓKKT,PN)is a con ex op imiza ion p oblem.
Chap e 4
Deep In e se Op imiza ion
Unlike he o he wo chap e s, his chap e does no ocus on e o mula ing he in e se
op imiza ion p oblem, i p esen s an algo i hm ha uses deep lea ning echniques o
sol e i . The algo i hm is based on he wo k o [12]. The main idea is o un oll an
i e a i e op imiza ion p ocess and hen use backp opaga ion o lea n he pa ame e s o
he o wa d model ha gene a e he obse a ions.
4.1 In oduc ion
Again, we ha e a da ase o decisions and inpu s {(ˆxi,ˆui)}N
i=1 wi h N≥1. The
ec o s ˆxia e he decisions, and he inpu ec o ˆui ep esen s condi ions unde which
we may wan o ins an ia e and, wi h ou app oxima ion, sol e he p og am o an inpu
ˆu.
To illus a e his chap e , we ecall he o wa d op imiza ion p oblem:
FOP(u,θ) := min
x{ (x,u,θ)|x∈X(u,θ)}.
In his chap e , we assume ha he decision a iables ˆxia e op imal solu ions o he
FOP, and he pa ame e o es ima e, θ, is pa o he cos and cons ain s o he p o-
g am.
Thus, he goal is, gi en he da ase , o lea n he pa ame e θsuch ha he e exis s
an op imal solu ion o FOP(ˆui,θ) ha is consis en wi h he co esponding obse ed
decision ˆxi. The consis ency is de ined by a loss unc ion, as we will see la e .
As is men ioned be o e, he pu pose o his chap e is o explain a deep lea ning algo-
i hm o make he lea ning p ocess o his pa ame e . This pa ame e can appea on he
objec i e unc ion and cons ain s, indi idually o bo h. This is some hing ha has no
been ex ensi ely add essed in p e ious chap e s because he gene al case o lea ning
37

38 4.2. Deep Lea ning F amewo k o IO
he pa ame e s o a pa ame ic linea op imiza ion p oblem whe e he cos ec o , con-
s ain ma ix, and igh -hand side cons ain ec o depend on ucan be sol ed by he
algo i hm p esen ed in his chap e [12]. Addi ionally, his app oach can be applicable
in online se ings, whe e he da ase is upda ed o e ime [1].
4.2 Deep Lea ning F amewo k o IO
We conside Deep Lea ning as a se o echniques o op imizing unc ions whe e he
global op imum is no gua an eed o be ound [6]. These echniques gene ally do no
gua an ee inding he global op imum, bu , in p ac ice, hey a e e y e ec i e o many
p oblems.
The algo i hm sea ches o he ea u es o he model h ough au oma ic di e en ia ion,
a di e en ia ion echnique ha compu es he g adien o he loss unc ion wi h espec
o he model’s pa ame e s and hen upda es he pa ame e s in he opposi e di ec ion
o he g adien o minimize he loss unc ion. This echnique is e y accessible o use
because many cu en machine lea ning lib a ies p o ide buil -in au oma ic di e en i-
a ion capabili ies, such as he one used o he expe imen s, PyTo ch [10].
Chap e 4. Deep In e se Op imiza ion 39
The algo i hm cycles h ough h ee s eps: (1) ins an ia e a o wa d op imiza ion model
wi h he cu en pa ame e s θ, (2) sol e he op imiza ion p oblem wi h a s anda d
algo i hm ( he amewo k used o illus a e he expe imen s implemen s he IPM al-
go i hm), and (3) compu e and upda e he pa ame e s θwi h he g adien o he loss
unc ion.
As is common in deep lea ning, he pa ame e s o lea n a e called weigh s.
Algo i hm 1 Deep in e se op imiza ion amewo k.
Requi e: Ini ial weigh s θini, aining da ase {(ˆxi,ˆui)}N
i=1.
Ensu e: Lea ned weigh s θl n.
1: θ←θini
2: o sin 1 o max_s eps do
3: ∆θ←0
4: o iin 1 o Ndo
5: x←FOP(ˆui,θ)▷Run o wa d op imize o comple ion
6: ℓ←ℓ(x,ˆxi)▷Compu e loss w. . . a ge
7: ∂ℓ
∂θ←backp op(ℓ)▷Backp opaga e g adien o weigh s
8: ∆θ←∆θ+1
N
∂ℓ
∂θ▷Accumula e a e age g adien
9: end o
10: ∆θ←α⊙∆θ▷Scale g adien componen -wise
11: β←line_sea ch(θ,∆θ)▷Find sa e s ep size
12: θ←θ−β∆θ▷Upda e weigh s
13: end o
14: e u n θ
This algo i hm akes he da ase {(ˆxi,ˆui)}N
i=1 as inpu and ini ializes he weigh s θ o
θini. G adien -based algo i hms a e e y sensi i e o he ini ial weigh s, as hey can
signi ican ly a ec he con e gence o he algo i hm. The e o e, he choice o ini ial
weigh s is c ucial: a good choice can speed up he con e gence o he algo i hm o an
op imal solu ion, al hough his is no gua an eed o be he global op imum.
The algo i hm hen i e a es o e a ixed numbe o s eps, max_s eps, o de e mine.
Each s ep consis s o i e a ing o e he da ase , calcula ing he loss unc ion, compu -
ing he g adien , scaling he g adien , pe o ming a line-sea ching, and upda ing he
weigh s. This p ocess is called an epoch.
In each epoch, all he da a poin s a e p ocessed, and o each da a poin (ˆxi,ˆui), an
ins ance o he o wa d op imiza ion p oblem is sol ed wi h he cu en weigh s θ(line
5), and he loss is compu ed be ween his op imal solu ion and he obse ed decision
ˆxi(line 6). The g adien o he loss unc ion wi h espec o θis compu ed using
au oma ic di e en ia ion h ough he ope a ions o he o wa d p ocess (line 7). Nex ,
he g adien is scaled by a ac o αusing a componen -wise p oduc (⊙) o con ol he
40 4.3. Discussion
lea ning a e (line 10). Line sea ch [9] is a echnique used o ind a sa e s ep size β ha
minimizes he loss unc ion (line 11). Finally, he weigh s a e upda ed by sub ac ing
he scaled g adien (line 12).
This algo i hm is a gene al amewo k ha can be applied o any di e en iable o wa d
op imiza ion model. G adien s can be compu ed e en wi h non-linea cons ain s o
objec i e unc ions. The o wa d op imize can be any op imiza ion algo i hm, such
as he simplex me hod, he in e io -poin me hod, o any o he op imiza ion algo i hm.
The hype pa ame e s o une include he ini ial weigh s θini, he lea ning a e α, he
numbe o s eps max_s eps, and he sol e used o sol e he o wa d op imiza ion
p oblem.
4.3 Discussion
In his chap e , we ha e p esen ed a deep lea ning algo i hm o sol e he in e se op-
imiza ion p oblem h ough he lens o deep lea ning. This allows us o apply all he
echniques and ools o deep lea ning o sol e he in e se op imiza ion p oblem [8].
A key s eng h o his algo i hm is i s gene ali y and applicabili y o any o wa d op i-
miza ion model o which au oma ic di e en ia ion can be applied. While i may no
be he bes app oach o sol e some speci ic in e se op imiza ion p oblems, such as
he ones p esen ed in he p e ious chap e s o ou con ibu ion p esen ed in chap e 5,
bu i p o ides a new s a egy. Ano he ad an age is ha he algo i hm can be applied
in online se ings, whe e he da ase is upda ed o e ime [1], and o di e en loss
unc ions.
On he o he hand, he algo i hm inhe i s he limi a ions o deep lea ning. Speci ically,
he g adien descen algo i hm can ge s uck in local minima, and he algo i hm can be
slow o con e ge. Also, he algo i hm can be sensi i e o he ini ial weigh s and he
lea ning a e. Ano he limi a ion is we need o p o ide a o wa d op imiza ion model,
which may no be known be o ehand.
Chap e 5
Implemen a ion and compu a ional
expe imen s
This sec ion p esen s he implemen a ion o a new model o sol e he In e se Op imiza-
ion P oblem using neu al ne wo ks and a compa ison wi h he Deep In e se Op imiza-
ion algo i hm o e wo di e en In e se Op imiza ion p oblems: Linea P og am wi h
Dynamic Feasible Regions and Linea P og am wi h ixed Feasible Regions.
5.1 Neu al Ne wo k Model
One o he main con ibu ions o his p ojec is he de elopmen o a new model o
sol e he In e se Op imiza ion P oblem using neu al ne wo ks.
The main idea is o use a neu al ne wo k o lea n he pa ame e s o he o wa d model
ha gene a e he obse a ions, his is, he unc ion o op imize (x,u), and he se
X(u). Obse e ha now, (x,ui)and X(ui)do no depend on he con ex pa ame e
θ, since now, he ne wo k is lea ning he en i e o wa d model.
The model is ained using a da ase o decisions and inpu s {(ˆxi,ˆui)}N
i=1 wi h N≥1,
o a gi en ui he model ou pu s ˜xi ha ep esen s he decision ha he o wa d model
would make unde he p edic ed o wa d model.
To p o ide knowledge o he model, he penul ima e laye o he neu al ne wo k ou pu s
he pa eme e s o he es ima ed Fo wa d Op imiza ion P oblem, his is, he unc ion o
op imize (x,ui), and he se X(ui), and he las laye o he neu al ne wo k ecei es
hese wo ou pu s and sol es he op imiza ion p oblem o ou pu he decision ˜xi. Figu e
5.1 shows he a chi ec u e o he neu al ne wo k model.
41
48 5.4. Linea P og am wi h Dynamic Feasible Regions
Figu e 5.4: Deep IO solu ions Figu e 5.5: Neu al Ne wo k solu ions
Figu e 5.6: Compa ison o solu ions gene a ed by bo h models.

Chap e 5. Implemen a ion and compu a ional expe imen s 49
5.5 Linea P og am wi h Fixed Feasible Regions
The goal o his expe imen is o e alua e and compa e he pe o mance o wo ap-
p oaches o sol ing he In e se Op imiza ion P oblem in he con ex o a Linea P o-
g am wi h Fixed Feasible Regions: he p oposed neu al ne wo k model and he Deep
In e se Op imiza ion algo i hm. This p oblem appea s o be simple han he p e ious
one because he easible egion emains ixed, while he objec i e unc ion a ies wi h
he inpu pa ame e u. We expec bo h models o pe o m be e in his scena io; how-
e e , we aim o e alua e and compa e hei pe o mance o iden i y he s eng hs and
limi a ions o each app oach.
To p o ide a comp ehensi e compa ison, we will analyze me ics such as compu a-
ional e iciency, accu acy in he solu ions, and mean squa ed e o . This analysis will
help us unde s and which app oach is mo e sui able o p ac ical applica ions whe e
he easible egion is ixed bu he objec i e unc ion depends on a ying inpu pa am-
e e s.
5.5.1 Fo wa d Op imiza ion P oblem
In his expe imen , we conside a Linea P og am wi h Fixed Feasible Regions as he
Fo wa d Op imiza ion P oblem o gene a e he da ase {(ˆxi,ˆui)}N
i=1.
The da ase {(ˆxi,ˆui)}N
i=1 was gene a ed by sol ing his Fo wa d Op imiza ion P ob-
lem o a ange o inpu alues u∈[0,6]. Fo each u, he op imal solu ion ˆxiwas
compu ed, and he pai (ˆxi,ˆui)was eco ded. This da ase was hen used o ain bo h
models.
The Fo wa d Op imiza ion P oblem is de ined as ollows:
FOP(u,θ) := min
x{c(u,θ)Tx|Ax≤b},(5.2)
wi h he ollowing pa ame e s:
•c(u,θ) = cos(θ+u2
2)
sin(θ+u2
2)
•A(u,θ) = 





0 1
0−1
1 0
−1 0
1 1






•b(u,θ) = 





60
0
60
0
100






50 5.5. Linea P og am wi h Fixed Feasible Regions
•u∈[0,6]
•θ=4
5
Figu e 5.7 illus a es how he objec i e ec o change as u a ies o his alue o
θ. The he objec i e ec o is colo -coded based on he alue o u. Da ke shades
co espond o he Fo wa d Op imiza ion P oblem wi h u= 0, while ligh e shades
co espond o u= 6. As uinc eases om 0 o 6, he colo ansi ions smoo hly om
da k o ligh . Addi ionally, he blue poin s in he igu e ep esen he op imal solu ions
o each Fo wa d Op imiza ion P oblem co esponding o he di e en alues o u.
Figu e 5.7: Linea P og am wi h Fixed Feasible Regions
5.5.2 Model Desc ip ion
In his expe imen , we compa e wo app oaches o sol ing he In e se Op imiza ion
P oblem: he p oposed neu al ne wo k model and he Deep In e se Op imiza ion algo-
i hm. Below, we desc ibe each model in de ail.
5.5.2.1 Neu al Ne wo k Model
The p oposed neu al ne wo k model is designed o lea n he pa ame e s o he Fo wa d
Op imiza ion P oblem, including he objec i e ec o c(u)and he pa ame e s de ining
Chap e 5. Implemen a ion and compu a ional expe imen s 51
he easible egion, Aand b. The model e ains he same a chi ec u e and hype pa am-
e e s as he one used in he p e ious expe imen , allowing o a di ec compa ison o
i s pe o mance ac oss he wo di e en scena ios.
5.5.2.2 Deep In e se Op imiza ion Algo i hm
Fo his expe imen , we de ined Deep In e se Op imiza ion model wi h he same Fo -
wa d Op imiza ion P oblem as he p obem we a e ying o es ima e (5.2) bu wi h
θ= 4. The goal o his model is o es ima e he eal pa ame e θo he Fo wa d
Op imiza ion P oblem om which he da ase was gene a ed.
5.5.3 Resul s
The Deep In e se Op imiza ion algo i hm achie es pe ec accu acy (100%) and ze o
mean squa ed e o (MSE), indica ing ha i pe ec ly eco e s he pa ame e s o he
Fo wa d Op imiza ion P oblem in his scena io. Addi ionally, i is signi ican ly as e ,
equi ing only 9 minu es compa ed o he 66 minu es needed by he neu al ne wo k
model. This sugges s ha he Deep In e se Op imiza ion algo i hm is well-sui ed o
p oblems wi h ixed easible egions and a ying objec i e unc ions, whe e he easi-
ble egion is known in ad ance.
On he o he hand, he neu al ne wo k model achie es an MSE o 29.238162 and 0%
accu acy, sugges ing ha i s uggles o eco e he exac pa ame e s o he Fo wa d
Op imiza ion P oblem in his se ing. Despi e i s lowe pe o mance in e ms o accu-
acy and MSE, he neu al ne wo k model may s ill be ad an ageous in scena ios whe e
he Fo wa d Op imiza ion P oblem is unknown o di icul o model explici ly.
Me ic Neu al Ne wo k Model Deep In e se Op imiza ion
MSE 29.238162 0
Accu acy (%) 0 100
Time (min) 66 9
Table 5.2: Me ics o bo h models in expe imen 2
To u he illus a e he pe o mance o bo h models, Figu e 5.10 shows he solu ions
gene a ed by he neu al ne wo k model and he Deep In e se Op imiza ion algo i hm
compa ed o he ue solu ions. The le image co esponds o he Deep In e se Op i-
miza ion algo i hm, while he igh image co esponds o he neu al ne wo k model.
We can obse e ha he Deep In e se Op imiza ion algo i hm p o ides highly accu a e
and easible solu ions, aligning pe ec ly wi h he ue solu ions. In con as , he neu al
52 5.6. Conclusions
ne wo k model s uggles o cap u e he exac s uc u e o he easible egion, esul ing
in less accu a e p edic ions.
Figu e 5.8: Deep IO solu ions Figu e 5.9: Neu al Ne wo k solu ions
Figu e 5.10: Compa ison o solu ions gene a ed by bo h models.
5.6 Conclusions
In his chap e , we p esen ed he implemen a ion o a new model o sol e he In e se
Op imiza ion P oblem using neu al ne wo ks and compa ed i s pe o mance wi h he
Deep In e se Op imiza ion algo i hm o e wo di e en scena ios: a Linea P og am
wi h Dynamic Feasible Regions and a Linea P og am wi h Fixed Feasible Regions.
The esul s highligh he s eng hs and limi a ions o each app oach, p o iding alu-
able insigh s o hei applica ion in p ac ical se ings.
The esul s o he expe imen s show ha he neu al ne wo k model ou pe o ms he
Deep In e se Op imiza ion algo i hm in e ms o accu acy and p ecision o he Lin-
ea P og am wi h Dynamic Feasible Regions. The neu al ne wo k model achie ed a
signi ican ly lowe Mean Squa ed E o (MSE) o 0.0000431 compa ed o he Deep In-
e se Op imiza ion algo i hm, which had an MSE o 0.2473. Addi ionally, he neu al
ne wo k model achie ed an accu acy o 65.9%, while he Deep In e se Op imiza ion
algo i hm achie ed only 0.81%.
In con as , he Deep In e se Op imiza ion algo i hm demons a ed supe io pe o -
mance in e ms o accu acy and compu a ional e iciency o he Linea P og am wi h
Chap e 5. Implemen a ion and compu a ional expe imen s 53
Fixed Feasible Regions. The algo i hm achie ed pe ec accu acy (100%) and ze o
mean squa ed e o (MSE), indica ing ha i pe ec ly eco e s he pa ame e s o he
Fo wa d Op imiza ion P oblem in his scena io. Addi ionally, i is signi ican ly as e ,
equi ing only 9 minu es compa ed o he 66 minu es needed by he neu al ne wo k
model.
The esul s o he expe imen s p o ide aluable insigh s in o he s eng hs and limi-
a ions o bo h app oaches and highligh he impo ance o selec ing he app op ia e
model o a gi en p oblem. By compa ing he pe o mance o he neu al ne wo k
model and he Deep In e se Op imiza ion algo i hm, we can de e mine which ap-
p oach is be e sui ed o di e en scena ios and iden i y a eas o u u e esea ch
and imp o emen , aking in o accoun ha bo h models a e based on g adien -based
op imiza ion me hods which come wi h inhe en limi a ions ha mus be conside ed.
The expe imen s show ha he choice o he model depends hea ily on he na u e o he
p oblem. Fo dynamic en i onmen s o cases whe e he Fo wa d Op imiza ion P ob-
lem is unknown o di icul o model explici ly, he neu al ne wo k model is a be e
choice due o i s lexibili y and adap abili y. In con as , o p oblems wi h ixed easi-
ble egions o known Fo wa d Op imiza ion P oblems, he Deep In e se Op imiza ion
algo i hm is mo e sui able, as i can e icien ly and accu a ely eco e he pa ame e s
o he Fo wa d Op imiza ion P oblem.
Fu u e wo k could explo e hyb id app oaches ha combine he s eng hs o bo h mod-
els, le e aging he lexibili y o neu al ne wo ks o dynamic componen s and he e i-
ciency o he Deep In e se Op imiza ion algo i hm o ixed s uc u es. Addi ionally,
u he esea ch could in es iga e he scalabili y o hese models o la ge and mo e
complex op imiza ion p oblems, as well as hei obus ness o noise and unce ain y in
he obse ed decisions. These ad ancemen s could u he enhance he applicabili y o
In e se Op imiza ion echniques in eal-wo ld scena ios.

54 5.6. Conclusions
Appendix A
Linea P og am wi h Dynamic
Feasible Regions
A.1 Pa 1: da a ob aining and p ep ocessing
[1]: impo numpy as np
impo deep_in _op as io
impo deep_in _op .plo as iop
impo o ch
impo o ch.nn as nn
om o ch.u ils.da a impo Da ase , Da aLoade
om sklea n.model_selec ion impo
,→ ain_ es _spli
om di cp impo Sol e E o
impo o ch
impo c xpy as cp
om c xpylaye s. o ch impo C xpyLaye
impo con ex lib
impo io as io2
impo ma plo lib
impo ma plo lib.pyplo as pl
ma plo lib. cPa ams[’ igu e.max_open_wa ning’]
,→= 0 # Le he plo s low!
%ma plo lib inline
55
56 A.1. Pa 1: da a ob aining and p ep ocessing
[2]: u_ ain =io. enso (np.linspace(-1.5,1.5,
,→1024). eshape((-1,1)))
[3]: u_ ain, u_ al = ain_ es _spli (u_ ain,
,→ es _size=0.2, andom_s a e=42)
u_ ain, u_ es = ain_ es _spli (u_ ain,
,→ es _size=0.15, andom_s a e=42)
Now we gene a e he da a om he eal p oblem.
[4]: class ExamplePLP(io.Pa ame icLP):
# Gene a e an LP om a gi en ea u e ec o u
,→and weigh ec o w.
de gene a e(sel , u, w):
c=[[ o ch.cos(w[0]+w[1]*u)],
[ o ch.sin(w[0]+w[1]*u)]]
A_ub =[[-1.0,0.0],
[0.0,-1.0],
[ w[0], 1+w[1]*u/3]]
b_ub =[[ 0.2*w[0]*u],
[-0.2*w[1]*u],
[ w[0]+ 0.1*u]]
e u n c, A_ub, b_ub, None,None
[5]: plp_ ue =ExamplePLP(weigh s=[1.0,1.0])
# Gene a e aining a ge s by sol e he ue
,→PLP.
x_ ain = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_ ain])
# o ch.sa e(x_ ain, "x_ ain.p ")
x_ al = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_ al])
# o ch.sa e(x_ al, "x_ al.p ")
x_ es = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_ es ])
Appendix A. Linea P og am wi h Dynamic Feasible Regions 57
# o ch.sa e(x_ es , "x_ es .p ")
[34]: u_show =io. enso (np.linspace(-1.5,1.5,10).
,→ eshape((-1,1)))
x_show = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_show])
iop.plo _pa ame ic_linp og(plp_ ue, u_show,
,→colo =[0,0,1], xylim=((-0.5,2.5),(-0.5,2.5)),
,→cxy=(1.2,1.2), show_solu ions=T ue)
[6]: # x_ al = o ch.load("x_ al.p ")
x_ al
[6]: enso ([[-0.0079, 0.0079],
[ 0.0906, -0.0906],
[ 0.0396, -0.0396],
64 A.2. Pa 2: model de ini ion and aining
nn.ini .kaiming_no mal_(m.weigh ,
,→nonlinea i y=’leaky_ elu’)# inicializamos los pesos
,→de la ed con Kaiming
nn.ini .ze os_(m.bias) # inicializamos los bias
,→a 0
# ini ialize_weigh s(model)
# We load he ained model weigh s
model.load_s a e_dic ( o ch.load("model_weigh s.
,→p h"))
[19]: <All keys ma ched success ully>
[20]: de nea ( s, x_ba ch, ol=1e-3):
’’’
s: Tenso con aining he solu ions o he
,→LPs
x_ba ch: Tenso con aining he desi ed
,→solu ions o he LPs
ol: Tole ance o conside wo ec o s as
,→equal
This unc ion calcula es he numbe o LP
,→solu ions ha a e wi hin a dis ance
less han ol om he desi ed solu ions.
’’’
d= o ch.sum(( s -x_ba ch) ** 2, dim=1)
e u n o ch.sum(d < ol).i em()
[21]: ol = 0.0001
de ain_loop(da aloade , model, loss_ n,
,→op imize ):
model. ain()
size =len(da aloade .da ase )
num_ba ches =len(da aloade )
loss_media = 0.0
accu acy = 0.0
ba ch_size =da aloade .ba ch_size

Appendix A. Linea P og am wi h Dynamic Feasible Regions 65
o ba ch, (u_ba ch, x_ba ch) in
,→enume a e(da aloade ):
s =model(u_ba ch)
loss =loss_ n( s, x_ba ch)
# Backp opaga ion and op imiza ion
loss.backwa d()
op imize .s ep()
op imize .ze o_g ad()
i ba ch %8==0:
loss =loss.i em()
cu en =ba ch *ba ch_size +ba ch_size
p in ( "loss: {loss:>7 }[{cu en :>5d}/{size:
,→>5d}]")
loss_media += loss
accu acy += nea ( s, x_ba ch, ol)
e u n loss_media /num_ba ches, accu acy /size
[22]: de al_loop(da aloade , model, loss_ n):
model.e al()
size=len(da aloade .da ase )
num_ba ches =len(da aloade )
al_loss, co ec = 0,0
o u_ba ch, x_ba ch in da aloade :
s =model(u_ba ch)
al_loss += loss_ n( s, x_ba ch).i em()
co ec += nea ( s, x_ba ch, ol) # co ec is
,→ he numbe o LP solu ions ha a e wi hin a dis ance
,→less han ol om he desi ed solu ions.
al_loss /= num_ba ches
co ec /= size
p in ( "Valida ion E o : n Accu acy:
,→{(100*co ec ):>0.1 }%, A g loss: { al_loss:>8 }
,→ n n")
e u n al_loss, co ec
66 A.2. Pa 2: model de ini ion and aining
[23]: al_acc =[]
al_loss =[]
ain_acc =[]
ain_loss =[]
ain_acc = o ch.load(" ain_acc.p h")
al_acc = o ch.load(" al_acc.p h")
ain_loss = o ch.load(" ain_loss.p h")
al_loss = o ch.load(" al_loss.p h")
[101]: # En enamien o
epochs = 1
o in ange(epochs):
p in ( "Epoch
,→{ +1} n-------------------------------")
loss, accu acy = ain_loop(da aloade , model,
,→loss_ n, op imize )
ain_acc.append(accu acy)
ain_loss.append(loss.clone().de ach())
p in ( " nT aining E o : n Accu acy:
,→{(100*accu acy):>0.1 }%, A g loss: {loss:>8 } n")
loss_ al, accu acy_ al =
,→ al_loop( al_da aloade , model, loss_ n)
al_acc.append(accu acy_ al)
al_loss.append(loss_ al)
p in ("Done!")
Epoch 1
-------------------------------
loss: 0.000139 [ 16/ 696]
loss: 0.000127 [ 144/ 696]
loss: 0.008852 [ 272/ 696]
loss: 0.000221 [ 400/ 696]
loss: 0.000206 [ 528/ 696]
loss: 0.000207 [ 656/ 696]
T aining E o : Accu acy: 30.7%, A g loss: 0.002772
Valida ion E o : Accu acy: 69.3%, A g loss: 0.002392
Appendix A. Linea P og am wi h Dynamic Feasible Regions 67
Done!
[24]: ig, axes =pl .subplo s(1,2, igsize=(12,5))
# (Accu acy s Epoch)
axes[0].plo ( ain_acc, label=" ain")
axes[0].plo ( al_acc, label=" al")
axes[0].legend()
axes[0].se _xlabel("Epoch")
axes[0].se _ylabel("Accu acy")
axes[0].se _ i le("Accu acy s Epoch")
axes[0].g id()
axes[1].se _ylim(0,1)
# (Loss s Epoch)
axes[1].plo ( ain_loss, label=" ain")
axes[1].plo ( al_loss, label=" al")
axes[1].legend()
axes[1].se _xlabel("Epoch")
axes[1].se _ylabel("Loss")
axes[1].se _ i le("Loss s Epoch")
axes[1].g id()
axes[1].se _ylim(0,0.04)
pl . igh _layou ()
pl .show()
68 A.2. Pa 2: model de ini ion and aining
[103]: o ch.sa e(model.s a e_dic (), "model_weigh s.
,→p h")
[104]: # sa e he accu acy
o ch.sa e( ain_acc, " ain_acc.p h")
o ch.sa e( al_acc, " al_acc.p h")
o ch.sa e( ain_loss, " ain_loss.p h")
o ch.sa e( al_loss, " al_loss.p h")
A.2.1 E alua ion o he model on he es se
[23]: model.e al()
[23]: Pa ame icLPNe (
( c): Sequen ial(
(0): Linea (in_ ea u es=1, ou _ ea u es=12,
,→bias=T ue)
(1): ReLU()
(2): Linea (in_ ea u es=12, ou _ ea u es=40,
,→bias=T ue)
(3): ReLU()
(4): Linea (in_ ea u es=40, ou _ ea u es=150,
,→bias=T ue)
(5): ReLU()
(6): Linea (in_ ea u es=150, ou _ ea u es=300,
,→bias=T ue)
(7): ReLU()
(8): Linea (in_ ea u es=300, ou _ ea u es=150,
,→bias=T ue)
(9): ReLU()
(10): Linea (in_ ea u es=150, ou _ ea u es=90,
,→bias=T ue)
(11): ReLU()
(12): Linea (in_ ea u es=90, ou _ ea u es=40,
,→bias=T ue)
(13): Sigmoid()
(14): Linea (in_ ea u es=40, ou _ ea u es=14,
,→bias=T ue)
)
)
Appendix A. Linea P og am wi h Dynamic Feasible Regions 69
[24]: u= o ch. enso ([[0.25]], d ype= o ch. loa 64)
x= o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u])
u,x
[24]: ( enso ([[0.2500]], d ype= o ch. loa 64),
enso ([[-0.0500, 0.0500]], d ype= o ch.
,→ loa 64))
[25]: x_p ed =model( o ch. enso ([[0.25]]))
x_p ed
[25]: enso ([[-0.0495, 0.0562]],
,→g ad_ n=<S ackBackwa d0>)
[26]: x, x_p ed, o ch.no m(x -x_p ed)
[26]: ( enso ([[-0.0500, 0.0500]], d ype= o ch.
,→ loa 64),
enso ([[-0.0495, 0.0562]],
,→g ad_ n=<S ackBackwa d0>),
enso (0.0063, d ype= o ch. loa 64,
,→g ad_ n=<LinalgVec o No mBackwa d0>))
[27]: al_loop( es _da aloade , model, loss_ n)
Valida ion E o :
Accu acy: 65.9%, A g loss: 0.000431
[27]: (0.00043140000343555585, 0.6585365853658537)
Now we plo he esul s o he model on he es se .
[28]: de p edic (model, da aloade ):
model.e al()
size=len(da aloade .da ase )
num_ba ches =len(da aloade )
x_p ed = o ch.emp y(0,2)
o u_ba ch, x_ba ch in da aloade :
s =model(u_ba ch)
x_p ed = o ch.ca ((x_p ed, s))
e u n x_p ed.de ach()

70 A.2. Pa 2: model de ini ion and aining
[29]: X_p ed =p edic (model, es _da aloade )
X_p ed
[29]: enso ([[ 7.6126e-01, 2.5680e-01],
[ 8.1555e-01, 1.2352e-01],
[ 9.9044e-02, -1.0426e-01],
...,
[-8.9096e-03, 1.2083e-02],
[-7.1218e-02, 8.6232e-02],
[ 6.3070e-03, -4.5897e-03]])
[85]: pl . igu e( igsize=(10,10))
pl .plo (x_ es [:20,0], x_ es [:20,1], ’o’,
,→label=’T ue solu ions’, ma ke size=10)
pl .plo (X_p ed[:, 0], X_p ed[:, 1], ’x’,
,→label=’P edic ed solu ions’, ma ke size=15)
pl .xlabel(’x’)
pl .ylabel(’y’)
pl . i le(’T ue s P edic ed solu ions’)
pl .legend()
pl .g id()
# ma co los limi es de la x y la y
pl .xlim(-0.2,1.1)
pl .ylim(-0.3,1.2)
pl .gca().se _aspec (’equal’, adjus able=’box’)
,→# es o es pa a que los ejes engan la misma escala
pl . igh _layou ()
pl .show()
pl .sa e ig(" ue_ s_p edic ed_nn.png")
Appendix A. Linea P og am wi h Dynamic Feasible Regions 71
<Figu e size 640x480 wi h 0 Axes>
A.3 Pa 3: Deep IO model
[30]: plp_lea n =ExamplePLP([1.5,1.2])
[ ]: u_show =io. enso (np.linspace(-1.5,1.5,10).
,→ eshape((-1,1)))
72 A.3. Pa 3: Deep IO model
x_show = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_show])
# Da a se . Black poin s a e he eal
,→solu ions, blue poin s a e he solu ions o he
,→plp_lea n
iop.plo _ a ge s(x_show, ma ke size=13)
iop.plo _pa ame ic_linp og(plp_lea n, u_show,
,→colo =[0,0,1], xylim=((-0.5,2.5),(-0.5,2.5)),
,→cxy=(1.2,1.2), show_solu ions=T ue)
[78]: u_show =io. enso (np.linspace(1,1.5,10).
,→ eshape((-1,1)))
x_show = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_show])
Appendix A. Linea P og am wi h Dynamic Feasible Regions 73
# Da a se . Black poin s a e he eal
,→solu ions, blue poin s a e he solu ions o he
,→plp_lea n
iop.plo _ a ge s(x_show, ma ke size=13)
iop.plo _pa ame ic_linp og(plp_lea n, u_show,
,→colo =[0,0,1], xylim=((-0.5,2.5),(-0.5,2.5)),
,→cxy=(1.2,1.2), show_solu ions=T ue)
[58]: io.in e se_pa ame ic_linp og(u_ ain, x_ ain,
,→plp_lea n, max_s eps=10,
callback=io.
,→in e se_pa ame ic_linp og_s ep_p in e ())
in e se_pa ame ic_linp og[0001]: loss=0.360737
,→weigh s=[1.5000 1.2000]
in e se_pa ame ic_linp og[0002]: loss=0.350147
,→weigh s=[1.3332 0.6538]
80 B.1. Pa 1: da a ob aining and p ep ocessing
[2]: u_ ain =io. enso (np.linspace(0,6,1024).
,→ eshape((-1,1)))
u_ ain
[2]: enso ([[0.0000e+00],
[5.8651e-03],
[1.1730e-02],
...,
[5.9883e+00],
[5.9941e+00],
[6.0000e+00]], d ype= o ch. loa 64)
[3]: u_ ain, u_ al = ain_ es _spli (u_ ain,
,→ es _size=0.2, andom_s a e=42)
u_ ain, u_ es = ain_ es _spli (u_ ain,
,→ es _size=0.15, andom_s a e=42)
[4]: u_ ain
[4]: enso ([[3.3255e+00],
[4.7918e+00],
[5.9062e+00],
...
[4.0000e+00],
[1.0029e+00],
[3.5425e+00]], d ype= o ch. loa 64)
[5]: u_ al
[5]: enso ([[3.0792],
[2.0938],
[2.6041],
...
[3.0968],
[1.7243],
[4.3695]], d ype= o ch. loa 64)
[6]: u_ es
[6]: enso ([[5.4721],
[5.4428],

Appendix B. Linea P og am wi h Fixed Feasible Regions 81
[0.7625],
...
[3.0616],
[2.9384],
[5.3959]], d ype= o ch. loa 64)
Now we gene a e he da a om he eal p oblem.
[7]: class ExamplePLP(io.Pa ame icLP):
# Gene a e an LP om a gi en ea u e ec o u
,→and weigh ec o w.
de gene a e(sel , u, w):
c=[[ o ch.cos(w +u**2/2)],
[ o ch.sin(w +u**2/2)]]
A_ub =[[ 0.0,1.0],
[0.0,-1.0],
[1.0,0.0],
[-1.0,0.0],
[1.0,1.0]
]
b_ub =[[ 60.0 ],
[0.0 ],
[60.0 ],
[0.0 ],
[100.0 ]
]
e u n c, A_ub, b_ub, None,None
[8]: plp_ ue =ExamplePLP(weigh s=[0.8])
[10]: u_show =io. enso (np.linspace(0,6,8).
,→ eshape((-1,1)))
xylim =((-4,65), (-4,65))
cxy =(55,55)
iop.plo _pa ame ic_linp og(plp_ ue, u_show,
,→colo =[0,0,1], xylim=xylim, cxy=cxy,
,→show_solu ions=T ue)
82 B.1. Pa 1: da a ob aining and p ep ocessing
[11]: u_show =io. enso (np.linspace(0,6,8).
,→ eshape((-1,1)))
xylim =((53,57), (53,57))
cxy =(55,55)
iop.plo _pa ame ic_linp og(plp_ ue, u_show,
,→colo =[0,0,1], xylim=xylim, cxy=cxy,
,→show_solu ions=T ue)
Appendix B. Linea P og am wi h Fixed Feasible Regions 83
[12]: # Gene a e aining a ge s by sol e he ue
,→PLP.
x_ ain = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_ ain])
# o ch.sa e(x_ ain, "x_ ain.p ")
x_ al = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_ al])
# o ch.sa e(x_ al, "x_ al.p ")
x_ es = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_ es ])
# o ch.sa e(x_ es , "x_ es .p ")
[13]: # x_ al = o ch.load("x_ al.p ")
x_ al
84 B.2. Pa 2: model de ini ion and aining
[13]: enso ([[3.1065e-05, 6.0000e+01],
[6.0000e+01, 1.3445e-04],
[4.0000e+01, 6.0000e+01],
...,
[3.9506e-05, 6.0000e+01],
[6.0000e+01, 4.0443e-05],
[4.0000e+01, 6.0000e+01]], d ype= o ch.
,→ loa 64)
[14]: # x_ ain = o ch.load("x_ ain.p ")
x_ ain
[14]: enso ([[1.5130e-05, 3.2630e-04],
[2.3855e-05, 6.0000e+01],
[3.7182e-05, 6.0000e+01],
...,
[6.0000e+01, 2.5838e-05],
[5.2023e-05, 1.4278e-05],
[1.9593e-05, 1.9354e-05]], d ype= o ch.
,→ loa 64)
[15]: # x_ es = o ch.load("x_ es .p ")
x_ es
[15]: enso ([[6.0000e+01, 4.0000e+01],
[6.0000e+01, 2.0927e-04],
[2.9810e-05, 1.5524e-05],
...
[2.8643e-05, 6.0000e+01],
[3.8379e-05, 6.0000e+01],
[6.0000e+01, 4.0141e-05]], d ype= o ch.
,→ loa 64)
B.2 Pa 2: model de ini ion and aining
[16]: # Check i we a e using a GPU o CPU
de ice = o ch.accele a o .
,→cu en _accele a o (). ype i o ch.accele a o .
,→is_a ailable() else "cpu"
p in ( "Using {de ice}de ice")
Using cuda de ice
Appendix B. Linea P og am wi h Fixed Feasible Regions 85
[17]: lea ning_ a e = 1e-3
ba ch_size = 16
[18]: # Da ase and Da aLoade de ini ion
class UDa ase (Da ase ):
de __ini __(sel , da a, a ge s):
sel .da a =da a.clone(). o(d ype= o ch. loa 32)
sel . a ge s = a ge s.clone(). o(d ype= o ch.
,→ loa 32)
de __len__(sel ):
e u n len(sel .da a)
de __ge i em__(sel , idx):
e u n sel .da a[idx], sel . a ge s[idx]
# Da ase (u, x)
da ase =UDa ase (u_ ain, x_ ain)
da aloade =Da aLoade (da ase ,
,→ba ch_size=ba ch_size, shu le=T ue)
[19]: al_da ase =UDa ase (u_ al, x_ al)
al_da aloade =Da aLoade ( al_da ase ,
,→ba ch_size=ba ch_size, shu le=T ue)
es _da ase =UDa ase (u_ es , x_ es )
es _da aloade =Da aLoade ( es _da ase ,
,→ba ch_size=ba ch_size, shu le=T ue)
[20]: de g adien _descen _sol e (c, A, b, l =0.01,
,→max_i e =1000, ol=1e-6, M=1000.0):
"""
Sol e a linea op imiza ion p oblem wi h
,→cons ain s using g adien descen .
Pa ame e s:
- c: Cos ec o (n,)
- A: Cons ain ma ix (m, n)
- b: Cons ain ec o (m,)
- l : Lea ning a e
- max_i e : Maximum numbe o i e a ions
- ol: Con e gence ole ance

86 B.2. Pa 2: model de ini ion and aining
Re u ns:
- x: Op imal solu ion ound
"""
n=c.shape[0]
x= o ch.ze os(n, d ype=c.d ype,
,→ equi es_g ad=T ue)
op imize = o ch.op im.SGD([x], l =l )
o _in ange(max_i e ):
op imize .ze o_g ad()
objec i e = o ch.do (c, x) +M* o ch.
,→sum( o ch. elu(A @x-b)) # elu(x) = max(0, x)
objec i e.backwa d( e ain_g aph=T ue)
op imize .s ep()
i o ch.no m(x.g ad) < ol:
b eak
e u n x
de new on_sol e (c, A, b, max_i e =100,
,→ ol=1e-6, M=1000.0):
"""
Sol e a linea op imiza ion p oblem wi h
,→cons ain s using he New on me hod.
Pa ame e s:
- c: Cos ec o (n,)
- A: Cons ain ma ix (m, n)
- b: Cons ain ec o (m,)
- max_i e : Maximum numbe o i e a ions
- ol: Con e gence ole ance
Re u ns:
- x: Op imal solu ion ound
"""
n=c.shape[0]
x= o ch.ze os(n, d ype=c.d ype,
,→ equi es_g ad=T ue)
o _in ange(max_i e ):
Appendix B. Linea P og am wi h Fixed Feasible Regions 87
loss = o ch.do (c, x) +M* o ch.sum( o ch.
,→ elu(A @x-b))
# G adien and Hessian
g ad = o ch.au og ad.g ad(loss, x,
,→c ea e_g aph=T ue)[0]
hess = o ch.au og ad. unc ional.hessian(lambda
,→y: o ch.do (c, y) +M* o ch.sum( o ch. elu(A @y-
,→b)), x)
# New on s ep: Δx = -H^-1 *g ad
y:
del a_x = o ch.linalg.sol e(hess, -g ad)
excep Run imeE o :
b eak # B eak i he Hessian is singula
# Ac ualize x
x=x+del a_x
# B eak i con e ged
i o ch.no m(del a_x) < ol:
b eak
e u n x
[21]: de sol e (c_, A_, b_):
y:
M= 1000.0
m,n=A_.shape
A=cp.Pa ame e ((m, n))
b=cp.Pa ame e (m)
c=cp.Pa ame e (n)
x=cp.Va iable(n)
obj =cp.Minimize(c.T@x+M*cp.sum(cp.pos(A
,→@x-b))) # cp.pos = max(0, x)
cons =[0*x>= 0]
p ob =cp.P oblem(obj, cons)
88 B.2. Pa 2: model de ini ion and aining
laye =C xpyLaye (p ob, pa ame e s=[c, A, b],
,→ a iables=[x])
wi h io2.S ingIO() as bu , con ex lib.
,→ edi ec _s dou (bu ):
solu ion, =laye (c_, A_, b_)
e u n solu ion
excep Sol e E o :
e u n new on_sol e (c_, A_, b_)
[22]: # De ine he model
class Pa ame icLPNe (nn.Module):
de __ini __(sel ):
supe (Pa ame icLPNe , sel ).__ini __()
sel . c =nn.Sequen ial(
nn.Linea (1,12),
nn.ReLU(),
nn.Linea (12,40),
nn.ReLU(),
nn.Linea (40,150),
nn.ReLU(),
nn.Linea (150,300),
nn.ReLU(),
nn.Linea (300,150),
nn.ReLU(),
nn.Linea (150,90),
nn.ReLU(),
nn.Linea (90,40),
nn.Sigmoid(),
nn.Linea (40,14)
)
de o wa d(sel , u):
ou pu =sel . c(u)
c=ou pu [:, 0:2]
A=ou pu [:, 2:10]. eshape(-1,4,2)
b=ou pu [:, 10:14]
e u n o ch.s ack(lis (map(sol e , c, A, b)))
[23]: loss_ n =nn.MSELoss()
Appendix B. Linea P og am wi h Fixed Feasible Regions 89
[24]: model =Pa ame icLPNe ()
op imize = o ch.op im.Adam(model.
,→pa ame e s(), l =lea ning_ a e)
# op imize = o ch.op im.SGD(model.
,→pa ame e s(), l =lea ning_ a e) # elegi una de las
,→dos
[25]: model
[25]: Pa ame icLPNe (
( c): Sequen ial(
(0): Linea (in_ ea u es=1, ou _ ea u es=12,
,→bias=T ue)
(1): ReLU()
(2): Linea (in_ ea u es=12, ou _ ea u es=40,
,→bias=T ue)
(3): ReLU()
(4): Linea (in_ ea u es=40, ou _ ea u es=150,
,→bias=T ue)
(5): ReLU()
(6): Linea (in_ ea u es=150, ou _ ea u es=300,
,→bias=T ue)
(7): ReLU()
(8): Linea (in_ ea u es=300, ou _ ea u es=150,
,→bias=T ue)
(9): ReLU()
(10): Linea (in_ ea u es=150, ou _ ea u es=90,
,→bias=T ue)
(11): ReLU()
(12): Linea (in_ ea u es=90, ou _ ea u es=40,
,→bias=T ue)
(13): Sigmoid()
(14): Linea (in_ ea u es=40, ou _ ea u es=14,
,→bias=T ue)
)
)
[26]: # Ini ialize he weigh s o he model
de ini ialize_weigh s(m):
i isins ance(m, nn.Linea ):
96 B.2. Pa 2: model de ini ion and aining
[36]: X_p ed =p edic (model, es _da aloade )
X_p ed
[36]: enso ([[ 5.8926e+01, 1.8113e+01],
[ 5.9214e+01, 5.4558e-01],
[ 1.5126e+01, 5.9356e+01],
...,
[ 2.5615e+00, 5.9404e+01],
[ 4.9921e+01, -6.5277e-01]])
[44]: pl . igu e( igsize=(10,10))
pl .plo (x_ es [:20,0], x_ es [:20,1], ’o’,
,→label=’T ue solu ions’, ma ke size=10)
pl .plo (X_p ed[:, 0], X_p ed[:, 1], ’x’,
,→label=’P edic ed solu ions’, ma ke size=15)
pl .plo ((0,0), (0,60), ’k--’, linewid h=2)
pl .plo ((0,40), (60,60), ’k--’, linewid h=2)
pl .plo ((40,60), (60,40), ’k--’, linewid h=2)
pl .plo ((60,60), (40,0), ’k--’, linewid h=2)
pl .plo ((60,0), (0,0), ’k--’, linewid h=2)
pl .xlabel(’x’)
pl .ylabel(’y’)
pl . i le(’T ue s P edic ed solu ions’)
pl .legend()
pl .g id()
# ma co los limi es de la x y la y
pl .xlim(-10,70)
pl .ylim(-10,70)
pl .gca().se _aspec (’equal’, adjus able=’box’)
,→# es o es pa a que los ejes engan la misma escala
pl . igh _layou ()
pl .show()
pl .sa e ig(" ue_ s_p edic ed_nn.png")

Appendix B. Linea P og am wi h Fixed Feasible Regions 97
<Figu e size 640x480 wi h 0 Axes>
B.3 Pa 3: Deep IO model
[38]: plp_lea n =ExamplePLP([4.5])
[39]: u_show =io. enso (np.linspace(0,6,10).
,→ eshape((-1,1)))
x_show = o ch.ca ([io.linp og(*plp_ ue(ui)).
,→de ach(). () o ui in u_show])
98 B.3. Pa 3: Deep IO model
[40]: # Da a se . Black poin s a e he eal
,→solu ions, blue poin s a e he solu ions o he
,→plp_lea n
iop.plo _ a ge s(x_show, ma ke size=13)
iop.plo _pa ame ic_linp og(plp_lea n, u_show,
,→colo =[0,0,1], xylim=((-5,65),(-5,65)),
,→cxy=(55,55), show_solu ions=T ue)
[85]: io.in e se_pa ame ic_linp og(u_ ain, x_ ain,
,→plp_lea n, max_s eps=10,
callback=io.
,→in e se_pa ame ic_linp og_s ep_p in e ())
in e se_pa ame ic_linp og[0001]: loss=5217.176260
,→weigh s=[4.5000]
in e se_pa ame ic_linp og[0002]: loss=5121.833691
,→weigh s=[116.2986]
Appendix B. Linea P og am wi h Fixed Feasible Regions 99
in e se_pa ame ic_linp og[0003]: loss=2309.924151
,→weigh s=[112.7349]
in e se_pa ame ic_linp og[0004]: loss=20.648341
,→weigh s=[16004.0603]
in e se_pa ame ic_linp og[0005]: loss=2.253935
,→weigh s=[16004.0728]
in e se_pa ame ic_linp og[0006]: loss=0.000091
,→weigh s=[16004.0748]
in e se_pa ame ic_linp og[0007]: loss=0.000091
,→weigh s=[16004.0748]
in e se_pa ame ic_linp og[0008]: loss=0.000091
,→weigh s=[16004.0748]
in e se_pa ame ic_linp og[0009]: loss=0.000091
,→weigh s=[16004.0748]
in e se_pa ame ic_linp og[0010]: loss=0.000091
,→weigh s=[16004.0748]
in e se_pa ame ic_linp og[done]: loss=0.000091
,→weigh s=[16004.0748]
[85]: (<__main__.ExamplePLP a 0x751b21bec470>,
enso (9.0747e-05, d ype= o ch. loa 64,
,→g ad_ n=<MeanBackwa d0>),
enso ([[1.5131e-05, 3.1417e-04],
[2.3842e-05, 6.0000e+01],
[3.7136e-05, 6.0000e+01],
...,
[6.0000e+01, 2.3598e-05],
[5.2365e-05, 1.4271e-05],
[1.9629e-05, 1.9319e-05]], d ype= o ch.
,→ loa 64,
g ad_ n=<Ca Backwa d0>))
[41]: # Plo he new plp_lea n, o show ha he
,→solu ions (blue ci cles) now line up wi h he
,→obse ed a ge s (black ci cles).
iop.plo _ a ge s(x_show, ma ke size=13)
iop.plo _pa ame ic_linp og(plp_lea n, u_show,
,→colo =[0,0,1], xylim=((-5,65),(-5,65)),
,→cxy=(55,55), show_solu ions=T ue)
100 B.3. Pa 3: Deep IO model
[42]: w= 16004.0748
es ima ed_plp =ExamplePLP([w])
accy =[ o ch.no m(x -io.
,→linp og(*es ima ed_plp(u)). ())<0.001 o u,xin
,→zip(u_ es , x_ es )]
lss =[ o ch.no m(x -io.
,→linp og(*es ima ed_plp(u)). ()) o u,xin
,→zip(u_ es , x_ es )]
p in ( "T ain accu acy: {100*sum(accy)/
,→len(accy):.2 }%")
p in ( "T ain a g loss: {sum(lss)/len(lss):.
,→2 }")
T ain accu acy: 100.00%
T ain a g loss: 0.00
Appendix B. Linea P og am wi h Fixed Feasible Regions 101
[43]: X_p ed_io = o ch.ca ([io.
,→linp og(*es ima ed_plp(u)). ().de ach() o uin
,→u_ es ])
X_p ed_io
[43]: enso ([[6.0000e+01, 4.0000e+01],
[6.0000e+01, 2.1331e-04],
[2.9915e-05, 1.5509e-05],
...,
[2.8590e-05, 6.0000e+01],
[3.8216e-05, 6.0000e+01],
[6.0000e+01, 4.0342e-05]], d ype= o ch.
,→ loa 64)
[46]: pl . igu e( igsize=(10,10))
pl .plo (x_ es [:20,0], x_ es [:20,1], ’o’,
,→label=’T ue solu ions’, ma ke size=10)
pl .plo (X_p ed_io[:, 0], X_p ed_io[:, 1], ’x’,
,→label=’P edic ed solu ions’, ma ke size=15)
pl .plo ((0,0), (0,60), ’k--’, linewid h=2)
pl .plo ((0,40), (60,60), ’k--’, linewid h=2)
pl .plo ((40,60), (60,40), ’k--’, linewid h=2)
pl .plo ((60,60), (40,0), ’k--’, linewid h=2)
pl .plo ((60,0), (0,0), ’k--’, linewid h=2)
pl .xlabel(’x’)
pl .ylabel(’y’)
pl . i le(’T ue s P edic ed solu ions’)
pl .legend()
pl .g id()
# ma co los limi es de la x y la y
pl .xlim(-10,70)
pl .ylim(-10,70)
pl .gca().se _aspec (’equal’, adjus able=’box’)
,→# es o es pa a que los ejes engan la misma escala
pl . igh _layou ()
pl .show()
pl .sa e ig(" ue_ s_p edic ed_io.png")

102 B.3. Pa 3: Deep IO model
<Figu e size 640x480 wi h 0 Axes>
Bibliog aphy
[1] Oma Besbes, Yu i Fonseca, and Ilan Lobel. Con ex ual in e se op imiza ion:
O line and online lea ning. Ope a ions Resea ch, 2023.
[2] Timo hy CY Chan and Neal Kaw. In e se op imiza ion o he eco e y o con-
s ain pa ame e s. Eu opean Jou nal o Ope a ional Resea ch, 282(2):415–427,
2020.
[3] Timo hy CY Chan, Taewoo Lee, and Da ia Te ekho . In e se op imiza ion:
Closed- o m solu ions, geome y, and goodness o i . Managemen Science,
65(3):1115–1135, 2019.
[4] Timo hy CY Chan, Ra id Mahmood, and Ian Yihang Zhu. In e se op imiza ion:
Theo y and applica ions. Ope a ions Resea ch, 2023.
[5] Jai o Escánez. Fdp expe imen s eposi o y. h ps://gi hub.com/
jai oescanez/ g_expe imen s, 2024. Gi Hub eposi o y con aining
he implemen a ion and expe imen s o he FDP.
[6] Ian Good ellow, Yoshua Bengio, and Aa on Cou ille. Deep Lea ning. MIT
P ess, 2016. h p://www.deeplea ningbook.o g.
[7] Ga ud Iyenga and Wanmo Kang. In e se conic p og amming wi h applica ions.
Ope a ions Resea ch Le e s, 33(3):319–330, 2005.
[8] Ami ha Ma hew, P. Amudha, and S. Si akuma i. Deep lea ning echniques: An
o e iew. In Aboul Ella Hassanien, Rohee Bha naga , and Ash a Da wish,
edi o s, Ad anced Machine Lea ning Technologies and Applica ions, pages 599–
608, Singapo e, 2021. Sp inge Singapo e.
[9] Jo ge J Mo é and Da id J Thuen e. Line sea ch algo i hms wi h gua an eed
su icien dec ease. ACM T ansac ions on Ma hema ical So wa e (TOMS),
20(3):286–307, 1994.
[10] Adam Paszke, Sam G oss, Soumi h Chin ala, G ego y Chanan, Edwa d Yang,
Zacha y DeVi o, Zeming Lin, Alban Desmaison, Luca An iga, and Adam Le e .
Au oma ic di e en ia ion in py o ch. In NIPS 2017 Wo kshop on Au odi , 2017.
103
104 Bibliog aphy
[11] Adam Paszke, Sam G oss, F ancisco Massa, Adam Le e , James B adbu y, G e-
go y Chanan, T e o Killeen, Zeming Lin, Na alia Gimelshein, Luca An iga,
e al. Py o ch: An impe a i e s yle, high-pe o mance deep lea ning lib a y. Ad-
ances in neu al in o ma ion p ocessing sys ems, 32, 2019.
[12] Da ia Te ekho Yingcong Tan, And ew Delong. Deep In e se Op imiza ion,
pages 540–556. Sp inge , Depa men o Mechanical, Indus ial and Ae ospace
Enginee ing, Conco dia Uni e si y, Mon eal, Canada, 2019.