scieee Science in your language
[en] (orig)

Hybrid Machine Learning/Simulation Approaches for Logistic Systems Optimization

Francisco Alexandre Lourenço Maia · 2020

Abstract

Hoje em dia, tem-se testemunhado um abrupto crescimento e desenvolvimento da indústria, refletido no elevado grau de complexidade e inteligência que os sistemas de produção correntes apresentam, onde se destacam os sistemas logísticos. Esta incessante procura pela inovação e melhoramento contínuo são muito recorrentes na época atual, traduzindo-se em constantes transformações no conceito da qualidade de um produto. Deste modo, emerge a necessidade em otimizar os layouts fabris conduzindo a um aumento da flexibilidade face aos seus comportamentos dinâmicos. Neste seguimento surge a imprescindibilidade de aprimoramento do comportamento do veículo autónomo associado, com vista a finalidades comuns como o aumento da produtividade e minimização de custos e lead times. Neste âmbito, esta dissertação, para além da implementação do modelo de simulação do sistema logístico, desenvolve numa fase inicial comportamentos elementares a aplicar ao veículo, implementadas no próprio ambiente de simulação. Posteriormente, dado que a área de Machine Learning tem obtido tanto sucesso noutras áreas tecnológicas, surgiu o desafio da introdução do conceito de rede neuronal, através da criação de uma nova entidade designada Agente e caraterizada pela técnica de aprendizagem baseada em Reinforcement Learning. Por fim, nesta dissertação, para além de se concluir que a abordagem baseada em Reinforcement Learning proporcionou os melhores resultados de produtividade, retiraram-se ainda conclusões no que à robustez destes modelos diz respeito, a fim de avaliar a sua flexibilidade quando sujeitos a diferentes contextos, simulando um ambiente real.

Read accessible full text

Hybrid Machine Learning/Simulation Approaches for Logistic Systems Optimization

Author: Francisco Alexandre Lourenço Maia
Year: 2020
DOI: 10.34626/cjng-n931
Source: https://repositorio-aberto.up.pt/bitstream/10216/132965/2/411607.pdf
FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO
Hyb id Machine Lea ning/Simula ion
App oaches o Logis ics Sys ems
Op imiza ion
F ancisco Alexand e Lou enço Maia
FINAL VERSION
Mes ado In eg ado em Engenha ia Ele o écnica e de Compu ado es
Supe iso : Amé ico Lopes de Aze edo
Second Supe iso : João Ped o Ta a es Viei a Bas o
July 22, 2020
c
F ancisco Alexand e Lou enço Maia, 2020
Resumo
Hoje em dia, em-se es emunhado um ab up o c escimen o e desen ol imen o da indús ia, e-
le ido no ele ado g au de complexidade e in eligência que os sis emas de p odução co en es
ap esen am, onde se des acam os sis emas logís icos. Es a incessan e p ocu a pela ino ação e
melho amen o con ínuo são mui o eco en es na época a ual, aduzindo-se em cons an es ans-
o mações no concei o da qualidade de um p odu o.
Des e modo, eme ge a necessidade em o imiza os layou s ab is conduzindo a um aumen o da
lexibilidade ace aos seus compo amen os dinâmicos. Nes e seguimen o su ge a imp escindibili-
dade de ap imo amen o do compo amen o do eículo au ónomo associado, com is a a inalidades
comuns como o aumen o da p odu i idade e minimização de cus os e lead imes.
Nes e âmbi o, o obje i o des a disse ação é a combinação de écnicas de Rein o cemen
Lea ning com abo dagens de simulação pa a a o imização de um sis ema logís ico job-shop, no
que à p odu i idade diz espei o.
Pa a além da implemen ação do modelo de simulação do sis ema logís ico, es a disse ação
desen ol e ambém numa ase inicial compo amen os elemen a es a aplica ao eículo, imple-
men adas no p óp io ambien e de simulação.
Pos e io men e, dado que a á ea de Machine Lea ning em ob ido an o sucesso nou as á eas
ecnológicas, su giu o desa io da in odução do concei o de ede neu onal, a a és da c iação de
uma no a en idade designada Agen e e ca a e izada pela écnica de ap endizagem baseada em
Rein o cemen Lea ning.
Po im, nes a disse ação, pa a além de se conclui que a abo dagem baseada em Rein o ce-
men Lea ning p opo cionou os melho es esul ados de p odu i idade, e i a am-se ainda con-
clusões no que à obus ez des es modelos diz espei o, a im de a alia a sua lexibilidade quando
sujei os a di e en es con ex os, simulando um ambien e eal.
i
ii
Abs ac
Nowadays, we ha e been wi nessing an ab up g ow h and de elopmen o he indus y, e lec ed in
he high le el o complexi y and in elligence ha he cu en p oduc ion sys ems p esen , in which
he logis ics sys ems s and ou . This incessan sea ch o inno a ion and con inuous imp o emen
a e e y common oday, ep oducing in o cons an changes in he p oduc quali y concep .
In his sense, he need o op imize he ac o y layou s eme ges, leading o an inc ease in lexi-
bili y because o hei dynamic beha iou s. In his segmen , he e is an essen ial need o imp o e
he beha iou o he associa ed au onomous ehicle, o each common objec i es such as inc eas-
ing he p oduc i i y and minimizing cos s and lead imes.
In his con ex , he objec i e o his disse a ion is he combina ion o Rein o cemen Lea n-
ing echniques wi h simula ion app oaches o he op imiza ion o a job-shop logis ics sys em,
ega ding p oduc i i y.
Beyond he implemen a ion o he simula ion model o he logis ics sys em, his disse a ion
de elops, in an ini ial phase, elemen a y beha iou s o be applied o he ehicle, implemen ed in
he simula ion en i onmen i sel .
Subsequen ly, gi en ha he Machine Lea ning a ea has been so success ul in o he echno-
logical a eas, he challenge o in oducing he concep o he neu al ne wo k appea s, h ough he
c ea ion o a new en i y called Agen and cha ac e ized by he Rein o cemen Lea ning echnique.
Finally, in his disse a ion, in addi ion o concluding ha he Rein o cemen Lea ning-based
app oach p o ided he bes p oduc i i y esul s, conclusions we e also d awn ega ding he o-
bus ness o hese models, in o de o assess hei lexibili y when subjec o di e en con ex s,
simula ing a eal en i onmen .
iii

i
Acknowledgemen s
My i s wo ds go o Enginee João Bas o and P o esso Amé ico Aze edo o all hei a ailabili y,
comp ehension and suppo du ing his mas e ’s disse a ion. Thei help was undamen al in his
whole p ocess and I am e y g a e ul o hem o ha .
Secondly, I would also like o acknowledge he suppo o he Enginee s Romão San os and
Na ciso Caldas o all he a ailabili y and incen i e du ing he disse a ion.
In his segmen , I also ha e a wo d add essed o INESC-TEC, in pa icula o he Cen e
o En e p ise Sys ems Enginee ing and all i s elemen s o he acili ies o e ed h oughou he
disse a ion and o he excellen en i onmen p o ided.
I would like o hank he Facul y o Enginee ing o he Uni e si y o Po o and i s p o esso s
o e e y hing ha has been ansmi ed o me o e hese 5 yea s, no only academically bu also
pe sonally.
I ha e o highligh all he iendly ela ionships I ha e c ea ed in hese 5 yea s, namely Ca los
Ca alho, wi h whom I sha ed his expe ience in he INESC-TEC du ing his inal semes e , as
well as Fábio Quei ós, Edua do Caldas, F ancisco Pi es, And é Oli ei a, And é Cip iano, Lídio
Ribei o, Ma ia Pe ei a, Alexand a San os and A u Almeida, among many o he s ha I ake wi h
me o he es o my li e.
I also hank my amily o all he suppo and mo i a ion ha helped me o go h ough his
jou ney, namely my pa en s F ancisco Maia, Ma ia Isilda and sis e Ca olina Maia.
Finally, I add ess a wo d o app ecia ion o all my iends o all he encou agemen and
s imula ion du ing his pe iod o my li e, no only in he bad momen s bu also in he good ones.
F ancisco Maia
i
A momen o pain is wo h a li e ime o glo y.
Louis Zampe ini
ii
xi LIST OF TABLES
5.19 P oduc i i y (in pa s) analysis e e ing o di e en p oduc ion mixes, o a ime
ho izono 52hou s................................. 71
5.20 P oduc i i y (in pa s) in s ochas ic en i onmen s, o a ime ho izon o 36 hou s . 72
5.21 P oduc i i y (in pa s) in s ochas ic en i onmen s, o a ime ho izon o 52 hou s . 72

Abb e ia ions and Symbols
AGV Au oma ed Guided Vehicle
AI A i icial In elligence
FIFO Fi s In, Fi s Ou
IB Inpu Bu e
IBWSx Inpu Bu e o Wo ks a ion x
JIT Jus -in-Time
MHS Ma e ial Handling Sys em
OB Ou pu Bu e
OBWSx Ou pu Bu e o Wo ks a ion x
PPO P oximal Policy Op imiza ion
RL Rein o cemen Lea ning
VRP Vehicle Rou ing P oblem
WIP Wo k-in-P og ess
WS Wo ks a ion
x
Chap e 1
In oduc o y Analysis
1.1 Con ex ualiza ion
La ely, he e has been an ab up g ow h and de elopmen o he indus y, in which he concep “In-
dus y 4.0” appea ed, which allowed he exchange o in o ma ion be ween a a ie y o equipmen s
in a ac o y. Namely ega ding he op imiza ion o in e nal p ocesses, as well as a company’s p od-
uc s and e en se ices, his pa adigm is e olu ionizing he indus y wo ldwide. Consequen ly, he
cu en need and demand o inno a ion and imp o emen ollow he model based on con inuous
imp o emen . [A yea wi hou imp o ing is a yea won by compe i o s - J. M. Ju an]. [8]
In his sense, he concep o Lean Thinking, ounded by Taiichi Ohno and Eiji Toyoda, which
combines he elimina ion o was e (Jus -in- ime - "Any ac i i y ha he cus ome is no willing
o pay" - Taiichi Ohno) wi h he immedia e eac ion o any p oblem ha could a ise in du ing a
p ocess (Jidoka - Japanese e m), has eme ged in his con ex . The main objec i e o his idea is
o inc ease cus ome sa is ac ion, c ea ing signi ican changes in he manu ac u ing p ocesses ha
con ibu e o hei be e unc ioning (Kaizen), inc easing p oduc i i y and e iciency. [9]
And wha is he eason o his incessan sea ch o inno a ion and op imiza ion o indus ial
p ocesses? The answe is p e y simple, cus ome s a e he main eason. Thei demands and needs
ha e also been inc easing o e he pas ew yea s, bo h in e ms o a ie y and quali y.
A he beginning o he s udy o hese subjec s, i was conside ed ha quali y would only be e-
la ed o he p oduc speci ica ions ("Quali y is con o mance o he speci ica ions" - Philip C osby).
Howe e , his concep has been cons an ly upda ed, conside ing ha he p ima y ac o is he sa -
is ac ion o he cus ome ’s needs ("Quali y is i ness o use" - Joseph Ju an). Fo his, i is i s ly
necessa y o in e he p oduc speci ica ions, ollowed by he iden i ica ion o he p obable e o s
ha may a ise du ing he p ocesses and hei causes, be o e p oceeding o hei elimina ion. [10]
The need o inc ease quali y, dec ease cos s and educe deli e y imes, led o he c ea ion o
se e al ypes o ac o y layou s, namely he unc ional (p ocess-o ien ed, job-shop), line (linea
lows, low-shop) and ixed ( he p oduc canno be mo ed).
In his sense, he indus ies s a ed o adop di e en modes o p oduc ion, namely he job-
shop, cha ac e ized by he exis ence o specialized a eas by unc ion, and low-shop, de ined by he
1
2In oduc o y Analysis
p oduc ion lines. O e all, he layou s a e designed o minimize Ma e ial Handling cos s, elimina e
bo lenecks, educe cycle imes, elimina e was e and inc ease p ocess lexibili y.
Cu en ly, many indus ies make use o he job-shop p oduc ion mode, because i is ela ed o a
high di e si y o p oduc s p oduced in low olume (p oduc ion o o de ). I also allows he inc ease
o he lexibili y o p oduc ion p ocesses, howe e , hey p esen high WIP and queues. [11]
In o de o op imize hese logis ics sys ems, Ma e ial Handling is in g ea ocus a he p esen .
In o de o ob ain sho e cycle imes and lowe cos s in anspo ing aw ma e ials be ween wo k-
s a ions (WSs), he e is a need o de elop new op imiza ion algo i hms. Fo his, i is necessa y
o ake in o accoun he ime and space (o he wa ehouse, o example). In o he wo ds, i is
necessa y o coo dina e all he asks o each wo ks a ion in o de , o example, o sa is y all o de s
wi h he sho es possible lead ime. [12]
Di ec ly associa ed wi h hese ma e ial managemen sys ems a e he AGVs (Au oma ed Guided
Vehicle) ha allow he ma e ials o be anspo ed be ween s a ions, and a e gene ally unmanned.
Cu en ly, ehicle ou ing p oblems a e in he spo ligh and hei objec i e is o a el he sho es
dis ance possible, minimizing cos s. This ype o p oblem has some common cha ac e is ics o he
"T a eling Salesman" p oblem, in which i is supposed o isi a ce ain numbe o ci ies co e ing
he sho es possible dis ance. In he case o indus ies, each ehicle has a maximum capaci y.
Mos o hese p oblems can be sol ed using he Milk-Run sys em, which is a deli e y sys em
ha allows us o educe s ocks, educe was e and op imize ou es. I is a deli e y sys em in which
one p oduc is deposi ed and ano he one is collec ed igh a e , in o de o sa e ime. I should
also be no ed ha he AGV’s ou e is ixed. The collec ion o p oduc s om supplie s is ca ied
ou on a scheduled basis, in s ipula ed quan i ies, making cycle imes mo e p edic able. [13]
Besides heu is ics such as Milk-Run, o he app oaches, based on me aheu is ics, ha e been
also de eloped o sol e his kind o p oblems. [14]
Allied o hese app oaches, he concep o Machine Lea ning eme ges. Since i has been qui e
success ul in o he a eas o echnology, why no make use o his ool and apply i o p oduc ion
sys ems? I is p ecisely hese issues ha a e cu en ly being s udied and de eloped.
1.2 Mo i a ion
The g owing e olu ion o he indus y due o he con inued inc ease in compe i i eness has led
logis ics managemen o be in ogue hese days. In o he wo ds, i s p ocesses ha e unde gone
signi ican changes, bo h inancially and empo a ily, in addi ion o he objec i e o making he
sys em mo e obus and secu e.
The impo ance o educing human-made ailu es has led o he use o AGVs, which, besides
he sa e y and p ecision issues, also con ibu e o he au oma iza ion o p oduc ion sys ems, wi h
ega d o he sequence o ope a ions.
Fo his pu pose, ce ain heu is ics we e c ea ed and de eloped ha allowed, o example,
o minimize he cycle imes o a p oduc ion line as well as he cos s associa ed wi h Ma e ial
1.3 Objec i es and Resea ch Ques ions 3
Handling. I should also be no ed ha a la ge pa o he lead ime is spen on anspo , wi h only
a mino i y ocusing on p oduc ion p ocesses.
One o he mos impo an o mula ions ha eme ged, aking in o accoun he need o accel-
e a e he low o ma e ials be ween loca ions, was he concep o Milk-Run sys ems. These ones
allow ha , in a deli e y sys em, when a ce ain p oduc is deli e ed o a s ipula ed loca ion, ano he
one is also collec ed, sa ing ime in anspo a ion. In sho , hese ypes o sys ems con ibu e o
he in eg a ion be ween he logis ics sys ems and supply chains. [15]
Tha said, he Milk-Run concep was la e anspo ed o a ac o y layou con ex , in which
each loca ion co esponds o a wo ks a ion. As men ioned, hese heu is ics ensu e he esolu ion
o p oblems such as Job-Shop Scheduling and Ma e ial Handling.
In o de o sol e hese op imiza ion p oblems, some ools like Machine Lea ning (lea ning
algo i hms such as neu al ne wo ks) can be combined o sol e ma e ial mo emen p oblems. This
ool has a g ea p ominence nowadays and allows o make p edic ions aking in o accoun da a
collec ed p e iously, e en in highly complex en i onmen s. In his segmen , one o he pa adigms
o Machine Lea ning ha eme ges is Rein o cemen Lea ning (RL). Based on he en i onmen in
ques ion and he decisions aken by he Agen , his lea ning me hod allows ecei ing eedback ha
indica es he bes decision made so a . [16]
In summa y, he s udy o he in luence ha hese me hods ha e on sol ing dynamic ehicle
ou ing p oblems associa ed wi h he anspo o ma e ials be ween wo ks a ions in a ac o y is
e y in e es ing and allows o c ea e e y e ec i e and adap able algo i hms.
These algo i hms based on Rein o cemen Lea ning can be applied ega dless o he complex-
i y o he sys em, and he e is no need o change he ac o y layou . In addi ion o all o his, he
in es men is low and he esul s achie ed a e sa is ac o y.
In conclusion, he cons an need o op imize bo h p oduc ion imes and anspo , imp o ing
he ou es, leads us o ques ion he ac ha , i Machine Lea ning has been so success ul in o he
a eas, why no adap i o he p oduc i e sys ems and combine i wi h simula ion app oaches o
he op imiza ion o logis ics sys ems. [17]
1.3 Objec i es and Resea ch Ques ions
The main objec i e o his disse a ion is he combina ion o echniques based on Machine Lea n-
ing wi h simula ion app oaches o logis ics sys ems op imiza ion.
Fi s ly, i is necessa y o model he p oblem, unde s and he ac o y’s ope a ion (layou ), c ea e
he simula ion model ( h ough he Flexsim so wa e ool) and, inally, de ine simple decision ules
o command he AGV.
Subsequen ly, i is possible o in eg a e aining algo i hms based on Rein o cemen Lea ning
echniques, which de ine a comple ely dynamic beha iou o he logis ics sys em, o inc ease he
p oduc i i y and minimize he makespans ( he o al ime o comple e a sequence o asks). These
algo i hms, made a ailable by OpenAI Baselines, a e conside ed he s a e-o - he-a o Rein o ce-
men Lea ning, nowadays.

4In oduc o y Analysis
Finally, i is essen ial o es whe he he algo i hm adap s o changes in he ac o y, wi h
espec o changes in he p oduc ion mix o in he s ochas ici y o p ocessing imes.
In o de o be able o accomplish hese objec i es, we need o answe he ollowing esea ch
ques ions (RQ):
•RQ 1: How can we model a ac o y’s ope a ion in a simula ion model wi h simple decision
ules o command he AGV?
•RQ 2: How can we in eg a e aining algo i hms based on RL echniques wi h a simula ion
model o s udy he ac o y’s p oduc i i y and makespan?
•RQ 3: How can we e alua e he obus ness o he p oposed app oaches?
1.4 Me hodological App oach
This disse a ion ollows he s udy o a se o hyb id app oaches o simula ion and Machine Lea n-
ing, whose consequen pu pose is he op imiza ion o a logis ics sys em, leading o an inc ease in
i s p oduc i i y and minimiza ion o makespan.
The ini ial phase o he disse a ion is alloca ed o he cons uc ion o he simula ion model o
he job-shop layou , using he Flexsim so wa e, in which an au oma ed guided ehicle will also
be inco po a ed.
Pos e io ly, he e a e wo elemen a y decision ules ha will be implemen ed, which p o ide
he indica ion o he wo ks a ion (WS) whe e a load o an en i y will be pe o med. The i s ule,
e e ing o he inaugu al simula ion model, is based on he Fi s -in Fi s -ou (FIFO) algo i hm,
while he second model is a Milk-Run op imized anspo sys em.
Then, he concep o communica ion be ween he simula ion en i onmen and an ex e nal p o-
g am will be in oduced, which will allow he c ea ion o a dis ibu ed sys em, because he las wo
ules add essed we e implemen ed in he simula ion model. This ex e nal p og am, called Se e ,
uses he Py hon language and i will communica e wi h he simula ion en i onmen , becoming e-
sponsible o AGV decision making. This new s age also equi es ha he communica ion be ween
he wo en i ies has o be p elimina ily es ablished, using he TCP communica ion p o ocol.
A he beginning o he inal s age, he concep s o Machine Lea ning will be in oduced and
discussed, namely he in oduc ion o a hi d en i y called Agen , which implemen s a neu al
ne wo k o decision making. This ne wo k will be esponsible o de ining he WS whe e he
AGV will load he espec i e en i ies. Fo ha , he Agen ecei es om Flexsim, h ough he
models subsequen ly de eloped, a se o ele an in o ma ion, namely obse a ions o he cu en
s a e o he plan and he cu en loca ion o he AGV. Howe e , i should be no ed ha he Se e
is s ill p esen , assuming he ole o an in e media y be ween he Agen and Flexsim.
Consequen ly, h ough a Rein o cemen Lea ning algo i hm, called P oximal Policy Op imiza-
ion (PPO), he neu al ne wo k will de e mine he espec i e WS whe e he AGV has o load an
en i y. A p e- aining p ocess o he Agen will also be s udied and applied, which will allow he
1.5 Disse a ion O ganiza ion 5
imp o emen o i s lea ning phase and, consequen ly, lead o a possible maximiza ion o p oduc-
i i y ("Base Model") and minimiza ion o makespan ("Makespan Model").
Finally, he p oduc ion mixes will be changed and he concep o p obabili y dis ibu ion will
be in oduced. In his sense, he models p e iously discussed will be applied o a s ochas ic en i-
onmen , in o de o e i y i hey p esen a high le el o obus ness so ha hey can be applied in
a eal con ex , whe e he p ocessing imes a e subjec o cons an changes.
All hese phases a e ep esen ed in Figu e 1.1.
Figu e 1.1: Disse a ion app oach me hod
1.5 Disse a ion O ganiza ion
Wi h ega d o he o ganiza ion and s uc u e o his documen , i is di ided in o six chap e s.
The i s chap e p e ends o in oduce he heme o he disse a ion, including i s con ex ual-
iza ion, mo i a ion, objec i es and esea ch ques ions, as well as he schema ic o he me hodolog-
ical app oach ollowed.
Chap e 2p esen s he s a e-o - he-a o he subjec s co e ed by he disse a ion, like he in-
dus ial p oduc ion sys ems, in pa icula he job-shop ype, Ma e ial Handling Sys ems, Machine
Lea ning concep s and disc e e-e en simula ion.
The desc ip ion and cha ac e is ics o he p oblem, as well as he desc ip ion o i s me hod-
ology, a e co e ed in chap e 3, e en as he implemen a ion o he i s h ee anspo sys ems
conside ed h oughou he disse a ion.
Chap e 4 e lec s wo addi ional anspo sys ems, in oducing Rein o cemen Lea ning ech-
niques in combina ion wi h simula ion app oaches. The obus ness o he sys ems is ela ed o he
p oduc ion mixes and p ocessing imes, which a e also add essed.
6In oduc o y Analysis
The esul s ob ained a e exposed and ea ed in chap e 5wi h ega d o he i e anspo
sys ems de eloped and hei obus ness.
Finally, chap e 6p esen s he conclusions o he disse a ion p ojec and he sugges ions o a
u u e wo k.
Chap e 2
S a e-o - he-A
This chap e aims o p esen he esul s o a bibliog aphic sea ch, in o de o in e nalize, in a simple
way, he concep s ha will be add essed h oughou he disse a ion.
Fi s ly, in sec ion 2.1, he concep s o indus ial p oduc ion sys ems will be in oduced in
gene al and, la e , in sec ion 2.2, he job-shop p oduc ion sys ems will be add essed. Then, sec ion
2.3 p esen s some in e p e a ions o he logis ics sys ems associa ed wi h Ma e ial Handling in
indus ial en i onmen s.
In sec ion 2.4 he ques ion o Ma e ial Handling Sys ems (MHS) will be add essed, and in he
nex sec ion (2.5) he p inciples e e ing o Milk-Run sys ems will be in oduced.
Sec ion 2.6 in oduces he subjec o Machine Lea ning, wi h special e e ence o he Rein-
o cemen Lea ning echnique.
Finally, sec ion 2.7 p esen s he main concep s associa ed wi h simula ion app oaches.
2.1 Indus ial P oduc ion Sys ems
The cons an e olu ion o he ma ke means ha , nowadays, socie y seeks o di e en ia ed p od-
uc s, which ollows a pe spec i e o di e si y a he han quan i y. In his way, he specialized
indus ial o ganiza ions ha e as main objec i e he inc ease o he e ec i eness and e iciency o
hei p oduc ion p ocesses.
Abou ypologies, he e a e sys ems whose objec i e is o p oduce p oduc s on a la ge scale
wi h a low deg ee o a ie y and a e also cha ac e ized by i s high p oduc i i y, low quali ica ion
o hei ope a o s, educed complexi y o ac o y managemen and educed lexibili y. This ype
o sys em is called as p oduc o ien ed.
On he o he hand, he e a e sys ems ha a e o ien ed o he p ocess and o he cus ome , in
which i s main goal is o sa is y he cus ome ’s needs, acco ding o a pull pe spec i e, and o
alue he quali y o he p oduc . This ype o sys em gi es mo e impo ance o he p oduc a ie y
ins ead o quan i y, and i is conside ed mo e lexible and deno ed by g ea e complexi y o ac o y
managemen .
7
14 S a e-o - he-A
•De e mined ime pe iods assignmen p oblem - known ou es and imes.
Figu e 2.7: Rep esen a ion o he ca ego ies o milk- un in-plan dis ibu ion p oblems, adap ed
om [4]
2.5 Milk-Run
A Milk-Run Sys em is di ec ly ela ed o he concep o anspo a ion in he indus y. The need o
educe s ocks and cos s in anspo was he eason o he c ea ion o his model. This idea o cos
educ ion ollows he JIT philosophy o Lean undamen als. B ie ly, his is a deli e y sys em ha
allows us o op imize ou es and educe was es be ween wo ks a ions in a ac o y plan (in-plan ).
Wi h ega d o he Milk-Run sys em, Baudin [22] s a es:
“This concep allows o mo e small quan i ies o a la ge numbe o di e en i ems
wi h p edic able lead imes and wi hou mul iplying anspo cos s" [22] - (Baudin,
2004 [13])”
All hese aspec s led Lean manu ac u e s o choose o o ganize hei anspo acco ding o
ixed imes and ou es, in he o m o a Milk-Run sys em. The design o hese sys ems in ol es a
highe complexi y, which acco ding o Meye [21] can be desc ibed in he ollowing h ee ac o s:
•T anspo o ma e ials;
•F equency o anspo s;
•Rou e scheduling.
2.5.1 Ad an ages o a Milk-Run Sys em
Compa ing o he adi ional app oach, shown in Figu e 2.4, he Milk-Run sys em p esen s he
ollowing ad an ages:

2.5 Milk-Run 15
In en o y Reduc ion
As Figu e 2.8 shows, he Milk-Run ypology, in cases X and Z, allowed he in en o y le el o be
educed by 1/3, because he equency o anspo was inc eased by h ee uni s. In case Y, he
equency o anspo inc eased wice, esul ing in a dec ease in he s ock o 2/3 o he le el o
he poin - o-poin ypology.
Figu e 2.8: Compa ison be ween he in en o y le els o a Poin - o-poin and Milk-Run ypology,
adap ed om [5]
Replenishmen wi h P edic able Lead Times
On a daily basis he e a e housands o p oduc s wi h di e en anspo equencies. Howe e ,
h ough Figu e 2.8, which p esen s only h ee di e en p oduc s (X, Y, Z), i is possible o gene -
alize o o he cases. So, i can be concluded ha hei lead imes a e p edic able.
Because we ha e access o da a which indica es an inc ease o dec ease in he equency o
supply, i becomes p edic able o es ima e he s ock a ia ions ha will occu in he u u e.
Be e In en o y Visibili y
In he case o he poin - o-poin deli e ies, he e may be a case whe e only he p oduc X is
anspo ed on a la ge scale, causing an almos o al emp ying o he espec i e shel es. Howe e ,
his si ua ion does no ep esen any ype o anomaly in he in en o y.
In he case o he Milk-Run sys em, because he quan i ies anspo ed a e p ac ically he
same o all p oduc s, any signi ican a ia ion in he amoun o any ype o p oduc p esen on
he co esponding shel is an immedia e sign o he p esence o an anomaly. So, he Milk-Run
ypology has a be e isibili y o he in en o y, allowing o ac quickly in case o abno mali y.
16 S a e-o - he-A
Imp o e Communica ion Skills wi h Supplie s
Using Milk-Run enables supplie s o be in egula con ac wi h he cus ome s, because o he e-
quency o supply. The e o e, i is possible o ob ain eedback om consume s abou he quan i ies
and quali y o he deli e ed p oduc s, which makes i possible o imp o e he deli e y sys em and
he u u e quali y o he p oduc s. [13]
2.6 Machine Lea ning
The Milk-Run sys ems need o adap since he cu en eali y o he ac o y layou s has led his
sys em o become dynamic so ha i is necessa y o de elop decision algo i hms. So, he con-
cep o Machine Lea ning was in oduced in he scope o he anspo o ma e ials in a job-shop
en i onmen , mo e speci ically he Rein o cemen Lea ning echnique.
This concep a gues ha lea ning om in e ac ions is a undamen al idea subjacen o almos
all heo ies o lea ning and in elligence. [23] Basically, Machine Lea ning aims o lea n based on
p e ious da a and make p edic ions o decisions o he u u e. [24]
Acco ding o A hu Samuel, pionee in a i icial in elligence:
“Machine Lea ning is he ield o s udy ha gi es compu e s he abili y o lea n
wi hou being explici ly p og ammed" - A hu Samuel, 1959 [23]
Enume a ing some o i s applica ions [23]:
•Analyse p oduc images on a p oduc ion line o au oma ically classi y hem;
•De ec umou s h ough b ain scans;
•Summa ize long documen s au oma ically.
The Machine Lea ning sys ems can be classi ied in o h ee ca ego ies, acco ding o i s lea ning
p ocesses:
•Supe ised Lea ning
A se ies o examples (inpu s) wi h he co ec answe (ou pu s) is p o ided by an ex e nal
supe iso y Agen and, based on aining, he implemen ed algo i hm gene alizes he co ec
answe o ano he se o inpu s, a e wa ds. [25]
•Unsupe ised Lea ning
The de eloped algo i hm ies o iden i y simila i ies be ween he inpu s, ca ego izing hem.
One o he bes -known echniques is he clus e ing. [25]
•Rein o cemen Lea ning
I is loca ed be ween Supe ised Lea ning and Unsupe ised Lea ning. The algo i hm is
in o med abou he quali y o he esponse, bu i is no in o med abou how o co ec i .
2.6 Machine Lea ning 17
So, i is necessa y o explo e and expe imen o he di e en possibili ies un il he Agen
disco e s how o ob ain a highe quali y esponse.
In his pa icula disse a ion, i is pe inen , wi h ega d o he anspo o ma e ials in a
job-shop en i onmen , o analyse, essen ially, he Rein o cemen Lea ning ield.
2.6.1 Rein o cemen Lea ning
Lea ning how o con ol Agen s di ec ly om high-le el senso y in o ma ion, such as ision and
speech, is one o RL’s longs anding challenges. [26]
The Agen is no speci ically old wha ac ions o ake, unlike wha happens in o he o ms o
Machine Lea ning, like Supe ised Lea ning. The e o e, he Agen will ha e o ind ou which
ac ions, delibe a ed so a , allowed him o ob ain g ea e ewa ds, a he end o he lea ning phase.
In e es ingly, he ac ions aken in he p esen will a ec he espec i e ewa d, as well as he u u e
ones.
The concep s o " ial and e o sea ch" and "delayed ewa d" a e he wo mos impo an
cha ac e is ics o RL.
Unlike Supe ised Lea ning, which is a way o lea ning based on examples p o ided aking
in o accoun he knowledge o an ex e nal supe iso , his is no appliable o an in e ac i e lea ning.
This is because, in in e ac i e p oblems, in mos cases, i is impossible o ob ain examples o he
desi ed beha iou ha a e co ec and ep esen all he si ua ions in which he Agen needs o ac .
Hence, he RL allows he Agen o decide wha ac ion o ake, aking in o accoun his own
expe ience. The Agen will ha e o check he decisions he has made in he pas and ind ou i , in
ac , he ob ained a bene icial ewa d, so ha he can hen la e ca y ou his ac ion. In his sense, a
new pa adigm appea s, in which he Agen , in addi ion o explo ing knowledge ha he al eady has
om p e ious si ua ions, also needs o explo e new decisions ne e made be o e, o see i he ge s
a highe ewa d. Nei he o hese pa adigms is conside ed be e han he o he because in bo h
cases, he Agen will ail (ob ain a lowe ewa d) and he solu ion s a es in he c i ical capaci y o
he Agen , so he has o pe o m se e al es s, in o de o ind he solu ion ha p o ides him wi h a
inal alue co esponding o he bigges ewa d. [6]
2.6.2 Rein o cemen Lea ning Cha ac e is ics
Agen
The Agen is he en i y ha i is esponsible o make he decisions and i is called “lea ne ” and
“decision make ”.
Mo e speci ically, he Agen and he en i onmen in e ac wi h each o he in he o m o dis-
c e e ime in e als ( =0,1,2, ...). As Figu e 2.9 p esen s, o each , he Agen ecei es a ep e-
sen a ion o he s a e o he en i onmen s , such ha s ∈S ep esen s he se o all possible s a es.
Consequen ly, he Agen ecei es a esponse in he o m o a ewa d +1∈Rand a new s a e o
he en i onmen , s +1, which is a eedback o he nex decision o make a +1. [6]
18 S a e-o - he-A
En i onmen
The en i onmen is esponsible o in o ming he Agen o he cu en s a e and he ewa d ob-
ained o he ac ion aken p e iously. I also ells he Agen a se o all possible s a es.
Ac ion
The ac ion is he esul o he decision made by he Agen . The objec i e is o ind he bes
solu ion, which co esponds o choosing he ac ion ha allows him o ob ain he highes ewa d
because each ac ion o igina es di e en ewa d alues.
Rewa d
The ewa d is a eedback in which he Agen e alua es he consequences o his ac ion aken in
he p e ious s a e. I is impo an o e e again ha he objec i e is o ob ain he g ea es possible
accumula ion o ewa ds, keeping in mind ha a la ge ewa d ob ained in a gi en s a e does no
necessa ily mean ha he inal accumula ion o ewa ds will be he bes . This is because, al hough
a speci ic ewa d in a gi en s a e is he la ges one, i may lead o a non-ideal si ua ion in he u u e
and in luence nega i ely he ollowing ewa ds. [27]
Policy π
I is a mapping s a egy ha allows he Agen o decide he nex ac ion o ake, in o de o ob ain a
good accumula ed ewa d in he long e m.
The RL speci ies how he Agen can change his policy, aking in o accoun his expe ience.
The Agen can also be classi ied acco ding o policy, alue unc ion and model. A policy, π,
is a mapping o s a es s∈Sand ac ions a∈A(s), o he p obabili y π(s,a)o aking an ac ion a
he ime o a s a e s. The alue o he s a e sunde a policy πis s ill deno ed by Vπ(s). [6]
Figu e 2.9: Agen -En i onmen in e ac ion diag am, adap ed om [6]
2.7 Simula ion
Modelling is a ool ha allows us o sol e eal con ex p oblems. Mos o he ime we canno
a o d o expe imen and es eal objec s, in o de o ob ain he bes solu ion, since hese objec s
a e, in gene al, expensi e and e en sca ce. Thus, he simula ion assumes a undamen al ole in his
2.7 Simula ion 19
con ex , in o de o be able o sol e p oblems in a p ac ical way and wi hou colla e al damage,
con i med by he ollowing quo e adap ed om [28].
“Modeling consis s o inding he pa h o he p oblem o i s solu ion, in a isk- ee
wo ld whe e we can make mis akes, undo hings, go back in ime and s a again.”
[28]
In addi ion o he pu pose o modelling, he e a e mo e bene i s o using his me hod, such as
[29]:
•Simula ion models allow us o analyse sys ems and ind solu ions in which he analy ical
models ail;
•I allows es ing new policies, ope a ing p ocedu es, decision ules, in o ma ion lows wi h-
ou making changes o he eal sys em;
•Disco e he bo lenecks.
Howe e , despi e all hese gains, he inal esul s can some imes be di icul o in e p e . The
ac ha he simula ion equi es a lo o aining is one o he leas a ou able poin s.
2.7.1 Componen s o a Sys em
In o de o be e unde s and he cons i u ion o a simula ion sys em, his sec ion desc ibes i s main
elemen s. [29]
Sys em
A se o en i ies ha in e ac wi h each o he in o de o achie e he ou lined objec i es.
Model
An abs ac ep esen a ion o a sys em, which allows i o desc ibe in e ms o i s s a e, en i ies,
p ocesses, e en s, ac i i ies, and o he s.
Sys em S a e
A necessa y se o a iables o desc ibe he sys em.
En i y
An objec o in e es which equi es an explici ep esen a ion.
A ibu e
An en i y p ope y.

20 S a e-o - he-A
E en
An ins an occu ence ha can change he s a e o he sys em.
Ac i i y
The ime du a ion ha a ask needs o be execu ed, known a he momen i s a s.
Delay
The excess ime in e al, only known when i ends.
Clock
A a iable ha ep esen s he simula ed ime.
A me hod is a s uc u e used o map eal sys ems in simula ion models. Wi h ega d o he
simula ion modelling me hods, his can be o ganized in o h ee ca ego ies: Agen -Based, Sys em
Dynamics and Disc e e-E en Modelling. The use o each one o hese me hods depends on he
sys em o be implemen ed and i s objec i es.
Agen -Based
I is pa o he class o compu a ional models o simula ing ac ions and in e ac ions be ween
au onomous Agen s (indi idual o collec i e). Despi e he lack o knowledge o he sys em’s
beha iou and inabili y o ep esen he p ocess low, he main objec i e is o e i y and s udy he
e ec o he Agen s on he sys em as a whole. [28]
Sys em Dynamics
John S e man [28] s a es ha his model is a pe spec i e and a se o concep ual ools ha enables
he unde s andmen o he s uc u e and dynamics o complex sys ems. This model is also a
igo ous modelling me hod ha allows o build o mal simula ions o complex sys ems and use
hem o design mo e e ec i e policies and o ganiza ions.
Disc e e-E en Modelling
I allows concei ing he modelling o a sys em as a disc e e sequence o e en s in ime. Each e en
occu s a a pa icula ime and causes a change in he s a e o he sys em. This sys em equi es ha
modelling has o be seen as a p ocess so ha i is a sequence o ope a ions pe o med by Agen s.
[28]
2.7 Simula ion 21
2.7.2 Disc e e-E en Simula ion
Nowadays, his simula ion me hod is widely used in he modelling and analysis o p oblems in
he a ea o logis ics sys ems, as i allows he s udy o aspec s such as p ocesses, scheduling and
esou ce alloca ion. Heal h, business p ocesses and mili a y applica ions a e also a eas co e ed by
his modelling me hod. Consequen ly, all hese ad ances ha e led o a so wa e de elopmen . [30]
In he speci ic case o his disse a ion, he simula ion will be used o e alua e he in e nal
ma e ial low o a manu ac u ing plan , in o de o make conclusions ega ding he ac o y’s p o-
duc i i y and alues o makespan, so we a e able o iden i y possible bo lenecks.
The e o e, his ool is used as a me hod o analysing and sol ing he ollowing p oblems,
associa ed wi h job-shop en i onmen s:
•E alua e he e ec o changing ma e ial anspo ou es be ween wo ks a ions;
•Analyse he phenomenon o esou ce alloca ion and bo leneck p e en ion;
•Assess he impac o changing he elemen s in he layou on pe o mance.
Th ough published a icles dedica ed o he s udy o his a ea o simula ion, i is possible o
d aw examples o applica ions in dis ibu ion and anspo sys ems.
Hugan (2001) elabo a ed a s udy ha allowed him o e alua e he in e nal a ic o a Gene al
Mo o s au omobile plan , in he USA, which was based on he JIT model o Lean manu ac u e.
The simula ion allowed o imp o e he in e nal ou es o each ype o p oduc , as well as o
es ima e he a e age ime spen by a p oduc in he ac o y, om i s en y o i s exi . Kuo, Chen,
Selikson and Lee (2001) used he simula ion o disc e e-e en s o s udy he low o ma e ials
also in a manu ac u ing plan , which allowed hem o ha e a deepe knowledge o ope a ions and
logis ics p ocesses. [30]
2.7.3 Cons uc ion S eps o a Simula ion Model
P oblem Fo mula ion
The p oblem mus be well o mula ed so ha he e is no doub . The e a e s ill cases whe e i is
necessa y o p oceed wi h a o al o pa ial e o mula ion o he p oblem. [29]
Es ablishmen o Objec i es and Gene al P ojec Plan
The objec i es will be he ques ions o be answe ed h ough he simula ion. A e deciding which
simula ion me hod is he mos sui able, i is necessa y o lis a se ies o al e na i es o he simu-
la ion and ind ways o e alua e he e ec i eness o hese same solu ions. Besides he objec i es
de ined o he end o each s a e, i is also impo an o men ion in he plan he numbe o people
in ol ed, as well as he associa ed cos s and he es ima ed ime o each phase o he p ojec . [29]
22 S a e-o - he-A
Model Concep ualiza ion
Acco ding o P i ske (1998), al hough i is no possible a p io i o de ine he ins uc ions ha will
lead o a success ul model, he e a e some poin s o iew ha mus be ollowed o he model o
be success ul. I is necessa y o s a by de ining a simplis ic model and, om he e, make he
necessa y changes s ep by s ep o ob ain good esul s. [29]
Da a Collec ion
Da a collec ion is di ec ly associa ed wi h he cons uc ion o he model, since he da a collec ed
will se e as inpu o he model. As his collec ion ills la ge in e als o ime and his is a e y
impo an aspec in he elabo a ion o he model, i is essen ial o s a he collec ion as soon as
possible. [29]
Model T ansla ion
This phase consis s o con e ing he model in o a simula ion language using speci ic so wa e
p og ams. In he speci ic case o his disse a ion, he so wa e used is he Flexim p og am. [29]
Ve i ica ion and Valida ion
This s ep enables o check i he p og am is p epa ed o he simula ion model, ca ying ou e -
i ica ion and debugging es s. By compa ing he model wi h he beha iou o he cu en sys em,
i is use ul o use his eedback in o de o imp o e he model. The p ocess is epea ed i e a i ely
un il he esul ob ained is sa is ac o y. [29]
Expe imen al Design
The al e na i es p e iously de ined in he “Es ablishmen o Objec i es and Gene al P ojec Plan”
phase ha mus be simula ed, mus be de e mined. [29]
P oduc ion and Analysis o Resul s
A e se e al es s o he model, wi h di e en da a, he esul ed analyses a e used o es ima e he
pe o mance o he sys em ha was simula ed. [29]
Documen a ion
A e he simula ion, i is necessa y o epo wo ypes o da a: p og am and p og ess. I he p o-
g am is used by o he s in he u u e, he p og am documen a ion indica es he modes o ope a ion
and beha iou o he p og am, as well as o he undamen al aspec s. In ela ion o he p og ess
epo , his is essen ial o he model o ob ain c edibili y and ce i ica ion. [29]
2.7 Simula ion 23
Implemen a ion
The success o he implemen a ion will depend on each phase p e iously e e ed. [29]
30 P oblem and Me hodology
Figu e 3.4: G aphical ep esen a ion o a WS – Flexsim
3.3.4 T anspo
As p e iously indica ed, he anspo o he en i ies is ca ied ou by an AGV, which is c ea ed
om an objec o he Task Execu e class (Flexsim Lib a y). The eby, i is only necessa y o
indica e which s a ions he AGV will collec (OB) and deli e (IB) pa s. This in o ma ion can
be ound in de ail in he p e ious chap e s o P ocessing, 3.2.2, Sequencing, 3.2.3 and P oduc ion
Plan 3.2.4.
I is also wo h no ing ha he AGV speed will be 1.50 m/s o all simula ion models.
3.3.5 Load
All he pa s come in in o he sys em h ough he Sou ce, howe e , in his pa icula case, his
esou ce has been eplaced by a Queue, called "Sou ce1", which admi s he same beha iou . The
main eason o using a Queue objec ins ead o a Sou ce is ha i is possible o obse e he
accumula ion o pa s h oughou he simula ion. A las , he esou ces whe e he pa s a e loaded
a e he Sou ce and he ou pu bu e s.
3.3.6 Unload
The inal unloading o he en i ies is ca ied ou a Sink, and he in e media e unloads be ween
WSs ake place in he espec i e inpu bu e s (IBWS).
3.3.7 P ocesses
The e a e essen ially wo ools ha allow he building o a simula ion model: 3D model and
P ocess Flow. Rega ding P ocess Flow, i always and in any ci cums ance o e laps he 3D model.
Howe e , he coope a ion be ween bo h is essen ial, al hough, in his pa icula si ua ion, i is in
he P ocess Flow ha he low o en i ies and esou ces will be de ined.
In gene al, all anspo sys ems in ol e h ee la ge blocks: C ea ion o Pa s (Figu e 3.5),
P ocessing o Pa s (Figu e 3.6) and T anspo o Pa s. Bo h phases o c ea ion and p ocessing o
pa s a e common o all models co e ed in his disse a ion.

3.3 Disc e e-E en Simula ion Model – Flexsim 31
3.3.7.1 C ea ion o Pa s
The "Sou ce" block de ines he ime when he p oduc ion o de s will be eleased, using an Excel
ile ha con ains he in o ma ion o his pu pose. Subsequen ly, he “C ea e Objec and Type”
block allows he c ea ion o he en i ies and assigns hem o he co esponding pa ype, aking
in o accoun he Label e e ing o he pa ype o he P oduc ion Plan able (Table 3.4). Taking
in o accoun he 5 di e en pa ypes, he e is a need o de ine colou s o each pa ype, in
o de o dis inguish hem du ing he pe o mance o he simula ion, accomplished by he “Change
Colo ” block:
•1 - Aqua;
•2 - Red;
•3 - Blue;
•4 - Yellow:
•5 - Lime.
Conside ing he ype o he cu en pa , he block “De e mine Line” allows o go h ough he
Rou ings Table (Table 3.5) and iden i y he line in ques ion, in o de o ollow he WS sequence
o he pa ype in e idence.
Subsequen ly, a e iden i ying he WS whe e he i s unload will be pe o med, i is necessa y
o acqui e he AGV in o de o alloca e i , so ha he ini ial load can be execu ed in he Sou ce
and he unload in he IB o he ollowing WS. A he end o he p ocess, he esou ce is eleased
("Release").
Figu e 3.5: Rep esen a ion o he “C ea ion o Pa s” Block
3.3.7.2 P ocessing o Pa s
When an en i y en e s he IB o a WS, i is necessa y o mo e i o he WS i sel , hen ope a e
i and, inally, mo e i one mo e ime o he espec i e OB. Howe e , i mus be no ed ha he
en i y will be jus mo ed om he cu en IB o he co esponding WS i he e a e no pa s in
32 P oblem and Me hodology
he espec i e WS, because i s maximum capaci y is uni a y. These es ic ions a e de ined by he
c ea ion o zones, ocused on each one o he exis ing WS. A he end o he p ocessing, he block
“Exi Zone” will elease he espec i e WS om he due en i y al eady p ocessed, and i will be
a ailable o ecei e a new pa . Then he line in he Rou ing Table (Table 3.5) is inc emen ed by
one, in o de o ollow he p e-de ined sequence o he pa .
Figu e 3.6: Rep esen a ion o he “P ocessing o Pa s” Block
3.4 Implemen a ion o a FIFO T anspo Sys em
Amongs all he anspo sys ems co e ed in he disse a ion, his model is conside ed he simples
one since he decision ule o apply o he AGV ollows a Fi s -in, Fi s -ou pa adigm. This means
ha he AGV will decide o ake o wa d he anspo o a pa ha was p ocessed i s , ins ead o
a pa ha has been p ocessed mo e ecen ly.
Acco ding o he P ocess Flow me hod, in addi ion o he wo blocks discussed in subchap e s
3.3.7.1 (C ea ion o Pa s) and 3.3.7.2 (P ocessing o Pa s), he block ha men ions he anspo
o pa s is also essen ial (3.4.1).
3.4.1 T anspo o Pa s
Apa om he ini ial anspo om Sou ce o he IB o he i s WS o each p oduc ion o de ,
add essed in he "C ea ion o Pa s" block (3.3.7.1), his block includes all he o he anspo s. In
his segmen , as soon as an en i y eaches he OB o any o he 7 WSs, he AGV will be eques ed
and, as soon as a ailable, i will anspo he pa om he cu en WS o he nex one, espec ing
he pa icula sequences. Finally, he AGV will be eleased, o igina ing he c ea ion o a new
anspo p ocess.
3.5 Implemen a ion o an Op imized Milk-Run Sys em 33
Figu e 3.7: Rep esen a ion o he “T anspo o Pa s” Block – FIFO Model
Table 3.5: Rou ings Table
Line Pa Type Wo ks a ion P ocessing Time (in minu es)
1 1 WS2 2
2 1 WS1 9
3 1 WS7 7
4 1 WS6 8
5 1 Sink1 0
6 2 WS4 10
7 2 WS2 9
8 2 WS5 5
9 2 WS1 5
10 2 Sink1 0
11 3 WS2 6
12 3 WS4 10
13 3 WS7 7
14 3 WS3 7
15 3 Sink1 0
16 4 WS7 3
17 4 WS3 7
18 4 WS2 7
19 4 WS4 5
20 4 Sink1 0
21 5 WS4 10
22 5 WS5 6
23 5 WS1 9
24 5 WS2 2
25 5 Sink1 0
3.5 Implemen a ion o an Op imized Milk-Run Sys em
The main di e ence be ween his Milk-Run sys em and he one p e iously p esen ed in he ea lie
subsec ion is he decision ule o be applied o he AGV. While in he p e ious sys em he e was no
p io i y ega ding he anspo o en i ies, ollowing he FIFO pa adigm, in his speci ic con ex
34 P oblem and Me hodology
he AGV has a p ede ined WS sequence, (iden i ied in he ollowing Table 3.6 and illus a ed in
Figu e 3.8), ha i ollows in o de o load he en i ies. In o he wo ds, i he AGV is, o example,
in WS2, bu ha same s a ion does no con ain en i ies in i s OB o pe o m he load, he AGV will
check i he nex WS in he AGV Rou ing Table (WS4) includes pa s in he espec i e OB, and
so on.
Wha dis inguishes his op imized Milk-Run algo i hm om he classic one is he ac ha
he AGV pe o ms he load di ec ly on he nea es WS, in a clockwise di ec ion, which mus
necessa ily ha e pa s in i s OB. On he o he hand, wha happens in he classic Milk-Run sys em
is ha he AGV mus isi he nea es WS, also clockwise, e en i i does no con ain pa s in he
espec i e OB.
Figu e 3.8: AGV T ajec o y – Milk-Run Model
Table 3.6: AGV Rou ing
Wo ks a ion
Sou ce
WS2
WS4
WS5
WS1
WS7
WS3
WS6
Consequen ly, using one mo e ime he P ocess Flow me hod, h ee la ge blocks we e also
es ablished: C ea ion o Pa s (3.3.7.1), P ocessing o Pa s (3.3.7.2) and T anspo o Pa s (3.5.1).
3.6 Implemen a ion o he Nea es WS Rule 35
3.5.1 T anspo o Pa s
The ype o he “Release oken.WS” block is he “Schedule Sou ce” and his block allows he
de ini ion o he cu en WS and associa es i wi h a oken called oken.WS. A oken is conside ed
a class, isibly ep esen ed by a ci cle, wi h he abili y o be upda ed h oughou he execu ion o
he P ocess Flow. The “Nex WS?” decision block iden i ies he nex WS whe e he AGV has o
pe o m he load, mo e speci ically i s OB, as well as an indica ion o he co esponden IB whe e
he en i y will be unloaded.
A e iden i ying he subsequen WS, i is necessa y o ese e he AGV and hen iden i y he
oldes “en i y” o he nex WS. A e ha , i is essen ial o load and unload he cu en pa and,
inally, upda e he cu en WS oken. Finally, he esou ce is eleased.
I should also be no ed ha when he AGV Rou ing able is comple ely c ossed and he cu en
line co esponds again o he cu en WS, i means ha none o he WSs has pa s in he espec i e
OBs o be loaded and, he e o e, he AGV is edi ec ed o a s a e whe e i will be wai ing un il a
pa is p ocessed ("Wai o E en ").
Figu e 3.9: Rep esen a ion o he “T anspo o Pa s” Block – Op imized Milk-Run Model
3.6 Implemen a ion o he Nea es WS Rule
This sec ion in ends o demons a e a new heu is ic also de eloped in he disse a ion, called Nea -
es WS Rule, whose main objec i e is o minimize he dis ances co e ed by he AGV. In e ms
o he decision p inciple, wha dis inguishes his heu is ic om he Milk-Run algo i hm (Chap e
3.5) is he ac ha he AGV mo es o he nea es WS wi h pa s wai ing o be anspo ed, no
ollowing any so o ou e.
As Figu e 3.11 displays, i he cu en s a ion is he WS4 and he e a e only pa s al eady
p ocessed wai ing o anspo in he ou pu bu e s o WS2 and WS3, since he closes s a ion o
WS4 is WS2, he AGV will load he pa p esen in he WS2 OB.

36 P oblem and Me hodology
Figu e 3.10: Illus a i e example o he Nea es WS Rule
One o he di e en ia ing ea u es o his new anspo sys em is he ac ha he e is an
ex e nal p og am ha con ols he simula ion en i onmen , ecei ing in o ma ion om i in he
o m o obse a ions, and sending ac ions ha indica e he OB o he s a ion whe e he load will
be ca ied ou . This app oach is comple ely di e en om hose p e iously s udied, like he FIFO
and op imized Milk-Run sys ems, since Flexsim will no be esponsible o making he decisions,
bu an ex e nal p og am, called Se e . This en i y is p og ammed using he Py hon language and
i is ep esen ed in Figu e 3.11.
Figu e 3.11: Se e -Flexsim ela ionships
The communica ions be ween he Se e and Flexsim en i onmen , which will be a clien ,
ha e o be success ul. In his sense, he T ansmission Con ol P o ocol (TCP) was chosen o
accomplish hese communica ions equi emen s. Fo his, i is c ucial o c ea e Clien -Se e
socke s, ha will be discussed in he nex Sec ion 3.6.1.
3.6.1 Clien -Se e TCP Socke s
Socke s a e an impo an ool used o send messages o e a ne wo k. This ne wo k can be logical,
local o he compu e o e en physically connec ed o an ex e nal ne wo k.
3.6 Implemen a ion o he Nea es WS Rule 37
The TCP p o ocol, in addi ion o being e y eliable, allows ha he da a sen by he clien can
be ecei ed and in e p e ed acco ding o he o de o sending by he Se e . Howe e , ega ding
he UDP p o ocol (Use Da ag am P o ocol), i does no gua an ee ha he da a will be ecei ed
by he Se e in he same sequence in which i was sen by he clien . [32]
Wi hin he scope o his disse a ion, he communica ion be ween he Se e and Flexsim
en i ies equi es he c ea ion o socke s o exchange messages, espec ing he TCP communica ion
p o ocol. In his sec ion, Figu e 3.12 illus a es he global s eps ha cons i u e he c ea ion and
de elopmen o a Clien -Se e communica ion. The Se e is conside ed a passi e Agen because
i wai s o he connec ion eques om he clien . On he o he hand, he clien is cha ac e ized
o aking he ini ia i e and making a communica ion eques , so i is an ac i e en i y in his whole
p ocess. [33]
B ie ly, in a i s ins ance, he Se e is esponsible o c ea ing he espec i e socke , and hen,
an add ess and a po a e immedia ely associa ed wi h i . Subsequen ly, he Se e wai s o he
momen he clien eques s he connec ion. A e he connec ion is success ul, i is possible o
send and ecei e messages be ween bo h en i ies, and a he end o his whole p ocess o sending
and ecei ing da a, he connec ions a e closed.
Figu e 3.12: Clien -Se e TCP/IP communica ion diag am
3.6.2 Se e – Ex e nal P og am
F om an obse a ion sen by he simula ion en i onmen ha indica es he cu en s a e o each
WS in e ms o he p esence o pa s in hei espec i e OB, as well as he cu en s a ion whe e
he AGV is placed, he Se e will hen send an ac ion wi h he espec i e WS whe e he AGV
will ealize he load.
To s o e he cu en da a o he simula ion model in he Se e , he a chi ec u e ep esen ed in
he ollowing UML diag am in Figu e 3.13 was c ea ed, whe e h ee classes a e de ined:
38 P oblem and Me hodology
•Sys em_Da a - con ains he dic iona ies e e ing o he dis ances be ween WSs and he
own WSs and allows an associa ion wi h he AGV class, in o de o know he cu en s a ion
whe e he AGV is loca ed;
•Wo ks a ion - allows o know i a speci ic WS has pa s in i s OB;
•AGV - con ains he AGV cu en loca ion;
Figu e 3.13: UML class diag am – Nea es WS Rule
The obse a ion sen by Flexsim is in he ollowing o m: 8 booleans + cu en WS, "10010000WS1",
in which he 8 booleans allow o iden i y i each o he 7 WSs and Sou ce ha e pa s in hei espec-
i e OB. Finally, he s ing allows o iden i y he cu en WS. Fo he speci ic example men ioned
("10010000WS1"), he conclusions d awn a e as ollows:
Table 3.7: In e p e a ion o he obse a ion sen by Flexsim
Wo ks a ion Does i ha e pa s on OB? Cu en WS
WS1 Yes X
WS2 No -
WS3 No -
WS4 Yes -
WS5 No -
WS6 No -
WS7 No -
Sou ce No -
I is impo an o no e ha he cu en s a ion is upda ed whene e an unload is pe o med on
one o he WSs, including Sink and excluding Sou ce. In his way, Sou ce will ne e assume he
ole o he cu en WS, as i is only used o pe o m loads. Howe e , i is essen ial o know i he e
a e pa s o anspo in i .
Re u ning o he diag am o he classes ha cons i u e he Se e , a e ex ac ing he in o ma-
ion om he obse a ion, i is necessa y o ill all he a ibu es o he h ee classes, namely he
"boolean_has_pa _OB" a ibu e o he "Wo ks a ion" objec s, as well as he cu en WS alue,
e e ing o he "AGV" objec . Rega ding he dis ance alues, hey ha e p e iously been ead om
an Excel ile and inse ed in he dis ance dic iona y o he "Sys em_Da a" class.
3.6 Implemen a ion o he Nea es WS Rule 39
A e he ea men and o ganiza ion o all his in o ma ion, i is necessa y o p oceed wi h he
implemen a ion o he decision ule al eady men ioned (Figu e 3.11), which allows he AGV o
ca y ou he load in he nea es WS ha con ains pa s in i s OB.
Finally, his in o ma ion ha e lec s he ou pu bu e o he chosen s a ion is sen back o he
simula ion en i onmen , in he o m o "OBWS" ( o example "OBWS2").
3.6.3 Flexsim – Simula ion En i onmen
3.6.3.1 T anspo o Pa s
A e sending he obse a ion o he Se e , Flexsim wai s o he esponse wi h he ac ion con-
aining he OBWS om he Se e . Howe e , i he Se e de ec s ha he e a e no pa s in any
o he OBs, i sends a message ha o de s he simula ion o go o a wai s a e ("Wai o pa s on
OB"), as shown in Figu e 3.14. I he e a e pa s in a leas one o he s a ions, he Se e sends
he ac ion wi h he OBWS and he Flexsim accesses he oldes objec p esen in ha OB. Then,
he nex WS whe e he pa will be unloaded is ound.
Finally, in case ha he numbe o pa s ha en e a Sink eaches he alue 2080, which
co esponds o he o al numbe o eleased o de s, he simula ion ends and i s s a e is sen o he
"Sink" block.
Figu e 3.14: Rep esen a ion o he “T anspo o Pa s” Block – Nea es WS Rule
46 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms
Figu e 4.6: Neu on gene al s uc u e, adap ed om [7]
The main idea o he PPO algo i hm is o make su e ha he new policy is no oo a om he
p e ious one. The e sion used cons i u es an implemen a ion o S able Baselines and in addi ion
o using ec o ized en i onmen s, i also allows ha he obse a ion and he esul ing ac ion can be
o a disc e e, box, mu idisc e e o mul ibina y ype. In his speci ic scope, he obse a ion assumes
he box ype (16,) and he ac ion admi s he disc e e ype (0-7). The nex Figu e 4.7 ep esen s he
ne wo k a chi ec u e o his disse a ion.
Figu e 4.7: SLP ne wo k a chi ec u e
Wi hin he obse a ion, he i s 8 booleans [0-7] indica e he WSs ha ha e o ha e no pa s
in hei espec i e OB (Table 4.2). The las 8 booleans [8-15] s a e he cu en AGV loca ion,
co esponding o a "one-ho encoding" (Table 4.3). Acco dingly o he Nea es WS Rule (Sec ion
3.6), he s a ions s a e is upda ed jus a e an unload was pe o med. Bu , whene e an ac ion
co esponds o a WS ha does no con ain pa s, he AGV goes he e in he same way and he
cu en WS is also upda ed.
As an example, i he obse a ion is: [1001000010000000], i means ha only WS1 and WS4
ha e pa s in hei OB and he cu en WS is WS1.

4.4 App oach I – Ini ializa ion o Weigh s 47
Table 4.2: Obse a ion segmen a ion [0-7]
Obse a ion [0-7] Exis ence o pa s
10000000 WS1
01000000 WS2
00100000 WS3
00010000 WS4
00001000 WS5
00000100 WS6
00000010 WS7
00000001 Sou ce
00000000 The e a e no pa s
Table 4.3: Obse a ion segmen a ion [8-15]
Obse a ion [8-15] Ac ual WS
10000000 WS1
01000000 WS2
00100000 WS3
00010000 WS4
00001000 WS5
00000100 WS6
00000010 WS7
00000001 Sink
00000000 Sou ce
4.4 App oach I – Ini ializa ion o Weigh s
Fi s ly, i was decided o de ine a neu al ne wo k a chi ec u e and a se o weigh s ha would allow
he eplica ion o he Nea es WS Rule. This phase aimed o ensu e ha an SLP ne wo k, wi h he
igh se o weigh s, would be su icien o ob ain a pe o mance equi alen o he esul s o he
Nea es WS Rule.
Fo demons a i e pu poses, in Figu e 4.8 i is possible o obse e he beha iou o his p ac-
ice, aking in o accoun he obse a ion con en and he esul ing ac ion. Since i is jus an ex-
ample, he e we e only conside ed 3 WSs, whose dis ances be ween hem a e also illus a ed in
Figu e 4.8.
In o de o simpli y he isualiza ion, only he weigh s o he ou pu neu ons 1 and 2, co e-
sponding o OBWS1 and OBWS2, a e iden i ied in he espec i e igu es.
As he Figu e 4.8 and Equa ion 4.3 show, he ou pu neu on ha will be ac i a ed is he i s
one, since i ob ains he highes ou pu alue. As i makes sense, i he e a e pa s in he WS1 and
WS3 ou pu bu e s, and he cu en s a ion is he WS1, hen he AGV will load he pa on he OB
o he WS1.
Figu e 4.8: Neu al Ne wo k – Example 1
48 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms
Dmax =3 (4.1)
Bias =−Dmax−2=−3−2=−5 (4.2)
OB_WS1= (1∗5+0∗0+1∗0)+(1∗3+0∗2+0∗1)+(1∗−5) = 3 (4.3)
OB_WS2= (1∗0+0∗0+1∗0)+(1∗2+0∗3+0∗0)+(1∗−5) = −3 (4.4)
OB_WS3= (1∗0+0∗0+1∗5)+(1∗1+0∗0+0∗3)+(1∗−5) = 1 (4.5)
Like he Figu e 4.9 deno es, he only case in which his ne wo k would no ha e he same
beha io as he ule would be i he cu en WS was he Sou ce, whe e i would be a d aw be ween
all s a ions wi h a pa on hei OBs (in his case: WS1 and Sou ce).
Figu e 4.9: Neu al Ne wo k – Example 2
Dmax =3 (4.6)
Bias =−Dmax−2=−3−2=−5 (4.7)
OB_WS1= (1∗5+0∗0+1∗0)+(0∗3+0∗2+0∗1)+(1∗−5) = 0 (4.8)
OB_WS2= (1∗0+0∗0+1∗0)+(0∗2+0∗3+0∗0)+(1∗−5) = −5 (4.9)
4.5 App oach II – Neu al Ne wo k P e-T aining 49
OB_Sou ce = (1∗0+0∗0+1∗5)+(0∗1+0∗0+0∗3)+(1∗−5) = 0 (4.10)
The S ableBaselines amewo k, which will be used o Rein o cemen Lea ning algo i hms,
does no allow he de ini ion o ini ial ne wo k weigh s. Thus, i was no possible o c ea e a policy
wi h he calcula ed weigh alues. S ill, his was an impo an exe cise o ensu e ha his ne wo k
a chi ec u e is capable o achie ing pe o mance a leas equi alen o he p e iously de ined ule.
An al e na i e o ini ializing he weigh s is using a p e- aining phase o he ne wo k, which is
co e ed in sec ion 4.5.
Thus, appea s he necessi y o s udying he in luence o he p e- aining algo i hms o obse e
i s in e e ence in he cu en p oblem.
4.5 App oach II – Neu al Ne wo k P e-T aining
This app oach allows, h ough he implemen a ion o an algo i hm wi h esul s al eady isible, in
his pa icula case he Nea es WS Rule, o ha e as s a ing poin ini ial solu ions ha come om
ha ule.
In his phase, he beha iou ha he Agen mus ha e o pe o m a s ep in o de o eplica e
he Nea es WS Rule is de ined. Then, se e al simula ions a e un using his Agen o ob ain a
da a se wi h obse a ion-ac ion pai s. Finally, hese da a a e used o ain he SLP neu al ne wo k
acco ding o Supe ised Lea ning algo i hms. This p ocess ensu es ha he neu al ne wo k s a s
i s lea ning p ocess wi h a se o weigh s ha eplica e he unc ioning o he ule.
Fo his, in o de o e i y he in luence o his p e- aining in he inal solu ion, wo mod-
els wi h di e en objec i es will be app oached, namely he minimiza ion o makespan and he
maximiza ion o p oduc i i y, aking in o accoun he PPO algo i hm ("Makespan" and "Base"
Models).
4.6 Makespan Model
In o de o e i y he co ec beha iou o he Rein o cemen Lea ning algo i hm, PPO, he ini ial
objec i e was o minimize he makespan. Thus, only one elease o de was de ined o a andom
en i y, mo e speci ically a pa ype o numbe 4.
The main pu pose o his model is o e i y he co ec beha iou o his anspo sys em,
obse ing he endency o he inal solu ions o ge he op imal solu ion. Associa ed wi h he
Agen ’s lea ning p ocess, he o al numbe o ime s eps de ined was 250.000 (s eps), so ha i is
belie ed o be mo e han enough o achie e he op imal solu ion.
As a compa able poin o he solu ions achie ed in he op imiza ion p ocess, he op imal solu-
ion can be ob ained h ough he Nea es WS and op imized Milk-Run ules, because he e is jus
one pa in he en i e simula ion p ocess.
50 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms
In his case, he p e- aining o he ne wo k does no apply o his model since he subjacen
ule would lead o an ideal beha iou al eady du ing he ini ial phase o he lea ning p ocess, which
is no in ended. Tha said, since he objec i e is o analyse he beha iou o he op imiza ion cu e
o he RL algo i hm, he idea is ha he inal makespan alues o each simula ion pe o med
app oach he op imal solu ion. In his speci ic case, he ideal solu ion o a pa ype 4 is: 1486
seconds.
Figu e 4.10 shows he P ocess Flow o he anspo p ocess implemen ed in he Flexsim
simula ion en i onmen . The “Decide” block de ines he nex WS whe e he AGV has o go,
aking in o accoun he in o ma ion sen by he Agen o e he a ious s eps i sends.
I he e a e no pa s al eady p ocessed wai ing o be anspo ed o he nex WS, he AGV goes
o a s a e whe e i is wai ing o he pa in ques ion o be p ocessed (gi en ha in his pa icula
model he e will only be one pa h oughou he en i e simula ion).
In he case ha he Agen sends a s ep e e ing o a WS ha does no con ain pa s in his OB,
since he ne wo k is in he lea ning p ocess, he AGV will also go he e and he cu en WS will
be upda ed.
As soon as he pa en e s a Sink, he Flexsim will send he alue co esponding o he
makespan as a ewa d o he Se e , bu wi h he nega i e signal (-makespan). This pa icula -
i y is jus i ied by he ac ha he pu pose o he PPO algo i hm is always o maximize some hing.
In his case, he goal is o maximize (-) makespan, in o he wo ds, minimize he makespan. S ill
in his scena io, he hi d pa ame e , called "done", is also sen as a uni , since he pa has al eady
been en e ed in o he Sink and he simula ion is inished.
Figu e 4.10: Rep esen a ion o he “T anspo o Pa s” Block – Makespan Model
A e unning se e al simula ions, i was ound ha he model, unde he cu en condi ions in
which he makespan is only sen a he end o he simula ion when he pa en e s a Sink, does
no allow eaching he op imal solu ion. I was concluded, he e o e, ha he ac ha he ewa ds
always assume null alues, wi h he excep ion o he esponse o he las s ep ha includes he
makespan, is no enough o he ne wo k o be able o make an adequa e lea ning and ind he
ideal solu ion wi hin he de ined 250k ime s eps.
4.6 Makespan Model 51
The e o e, some c i e ia we e in oduced in o de o check whe he i is possible o achie e he
op imal solu ion, in pa icula :
• he sending o ewa ds be ween ime s eps wi h he alue o a ime in e al ini ia ed by a
load and ended wi h an unload (including he p ocessing ime), in de imen o sending he
o al makespan only a he end o he simula ion;
• he in oduc ion o penal ies, i he mo emen s made do no in ol e he anspo a ion o
pa s;
• he inse ion o no maliza ions, modi ying he alue o he ewa ds, con e ing hem in he
in e al [-1,0].
La e , in o de o s udy he in luence o hese changes on lea ning pe o mance, all o hese
scena ios p esen ed in Table 4.4 we e implemen ed and simula ed.
Table 4.4: Scena ios o he Makespan Model
Scena io Time s eps No maliza ion No maliza ion Fac o Type o Rewa d Penal y
1 250k No 0 makespan 0
2 250k No 0 makespan -FN/2
3 250k Yes 15k makespan 0
4 250k Yes 15k makespan -0.5
5 250k No 0 be ween ime s eps 0
6 250k No 0 be ween ime s eps -FN/2
7 250k Yes FN be ween ime s eps 0
8 250k Yes FN be ween ime s eps -0.5
FN = 4*T a el ime be ween he wo mos dis an s a ions + 2*Highe P ocessing Time (Pa ype
no4).
Fo all he cases p esen ed abo e in Table 4.4, he model con e s he solu ions ob ained o
he op imal solu ion, excep o he scena io in which he ewa ds a e sen be ween ime s eps
wi hou no maliza ion o penal y. The ques ion ha immedia ely a ises is ela ed o he ime ha
is necessa y o each he op imal solu ion and, in his con ex , i was ound ha no maliza ion is
he ac o ha allows he model o end mo e quickly owa ds he op imal solu ion. These esul s
a e p esen ed and discussed wi h mo e de ail in Chap e 5.
In he ollowing subsec ions a e p esen ed he s eps used in he no maliza ion and penal y
me hods o he ypes o ewa ds: be ween ime s eps and makespan.
4.6.1 No maliza ion
The main objec i e o no maliza ion is o e i y whe he , by adjus ing he alues o he ewa ds
o alues be ween [-1, 0], he algo i hm will be able o con e ge be e o a good solu ion and
inc ease i s lea ning capaci y.

52 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms
Fo his, he no maliza ion ac o depends on he ype o ewa d (be ween ime s eps o
makespan) and applica ion o penal ies, cha ac e ized by he ollowing Equa ions 4.11 and 4.12:
4.6.1.1 Type o Rewa d: Makespan
No maliza ion ac o =2∗7.5k=15k(seconds)(4.11)
Rewa d ∈[−0.5,0],wi hou penal y (4.12)
This alue a ibu ed o he no maliza ion ac o is jus i ied by he ac ha i was obse ed
du ing all he simula ions p e iously ca ied ou ha he inal alue o he makespan, o a single
pa o ype 4, ne e exceeded he alue o 7.5k seconds, wi h a e excep ions. The e o e, aking
in o accoun he subsequen applica ion o penal ies (-0.5), he alue conside ed o he no mal-
iza ion ac o was 15k seconds, as i is wice 7.5k. The e o e, wi hou applying he penal y, he
ewa d assumes alues be ween -0.5 and 0, since he pu pose o op imiza ion is o maximize (-)
makespan.
4.6.1.2 Type o Rewa d: Be ween Time s eps
No maliza ion ac o =4∗T1+2∗T2 (4.13)
in which T1=T a el ime be ween he wo mos dis an s a ions; T2=Highe P ocessing Time
(Pa o ype 4).
Rewa d ∈[−0.5,0],wi hou penal y (4.14)
As in he p e ious case ega ding makespan, he no maliza ion ac o applied allows ewa ds
o assume alues be ween -0.5 and 0, wi hou he applica ion o penal ies.
In he speci ic case o a pa ype 4:
•T1 = 48.827 seconds;
•T2 = 420 seconds.
No maliza ion ac o =1035.308 (4.15)
4.6.2 Penal ies
Penal ies a e applied when he AGV pe o ms mo emen s wi hou a pa . The objec i e is, he e-
o e, h ough he applica ion o his penal y wi h he alue o -0.5, o make i clea o he Agen
ha , in his speci ic case, i is no a good p ac ice o make mo emen s wi hou a pa , gi en ha i
4.7 Base Model 53
is only one pa in he simula ion en i onmen . Tha said, aking in o accoun he ype o ewa d,
hei alues will be included in he ollowing ange:
Rewa d ∈[−1,0](4.16)
4.7 Base Model
A e analysing he Agen ’s beha iou owa ds he model whose objec i e was o minimize he
makespan, i was also decided o analyse he issue o maximizing p oduc i i y and c ea e a new
model ha allows s udying he PPO algo i hm.
Wi h ega d o he P ocess Flow o he simula ion model, all he blocks a e coinciden wi h he
blocks o he p e ious model, e e ing o he makespan. The only ele an di e ence is he issue
o he ewa ds ha a e sen om Flexsim o he Agen , in e media ed by he Se e . In esponse
o a s ep om he Agen , he simula ion en i onmen cons uc s he obse a ion ha con ains he
cu en s a us o each WS, in e ms o he p esence o pa s, and he cu en posi ion o he AGV.
Howe e , he ewa ds assume a uni alue whene e a pa is en e ed a Sink. O he wise, hey a e
assigned a null alue. In ela ion o he pa ame e “done”, i will be uni a y whene e he de ined
ime ho izon is eached, ending he simula ion.
Mo eo e , whene e a ewa d assumes a alue o 1, he e is an in e nal coun e in he sim-
ula ion en i onmen ha is inc eased, p o iding an upda ed indica ion o he numbe o pa s
manu ac u ed so a , o a la e compa ison o esul s wi h he p e ious p oduc i i y models.
In his con ex , unlike wha happens wi h he makespan model, he use o p e- aining becomes
impo an in o de o e i y whe he i has had an e ec o no . The ule implemen ed and asso-
cia ed wi h he p e- aining is based on one o he ules p e iously desc ibed in his disse a ion,
be e known as he Nea es WS Rule, which allows AGV o pe o m a load on he nea es WS ha
con ains pa s in he espec i e OB.
A no less impo an issue is o unde s and he e ec s o he in oduc ion o penal ies on p o-
duc i i y. Fo his, gi en ha he ewa ds a e al eady no malized in he in e al [0, 1], mo e
speci ically assuming he alues 0 o 1, he de ined penal y was (-) 0.5, simila o he p e ious
Makespan Model. Tha said, whene e he AGV makes an emp y mo emen , o de ed by he
Agen , he ewa d e u ned will be -0.5.
Rega ding he numbe o s eps sen by he Agen , a e se e al uns, i was concluded ha he
alue o 10M s eps would be he mos indica ed alue and mo e han enough o each he inal
solu ion. Fo u he compa ison o esul s, he ime ho izons chosen we e 36 and 52 hou s.
Finally, i should be no ed ha , whene e a "done" alue is assigned o 1, he p oduc i i y
alue ela ed o he simula ion in ques ion is also w i en in a ex ile, o la e analysis.
54 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms
4.8 Robus ness o he Simula ion Models
This sec ion was c ea ed wi h he in en ion o e i ying whe he he models co e ed in he p e ious
sec ions, namely he FIFO, op imized Milk-Run, Nea es WS Rule and Base Model, e e ing o
p oduc i i y, ha e an app op ia e beha iou agains possible changes in he p oduc ion mix o in
he p ocessing imes.
4.8.1 P oduc ion Mixes
Wi h ega d o changes in p oduc ion mixes, he o de in which he di e en pa ypes a e launched
will su e some a ia ions and i will be andomly gene a ed, using he Mic oso Excel so wa e
ool.
Jus like wha happens in he o iginal mix, in which he p obabili y o each pa ype being
eleased is nea ly 20% (Table 4.5), he same es ic ion applies o he nex 5 di e en mixes gen-
e a ed.
B ie ly, he di e en p oduc ion mixes a e gene a ed so ha he pa ype associa ed wi h each
p oduc ion o de is de e mined andomly, bu wi h he same inal p obabili y o being c ea ed
(20%).
Table 4.5: Quan i y o he di e en pa ypes – O iginal Mix
Pa Type Quan i y (uni s)
1 448
2 410
3 429
4 424
5 369
The e o e, as shown in Chap e 5, he e a e small di e ences in he inal quan i ies o pa s
p oduced o each one o he mixes, due o he way hey we e c ea ed.
4.8.2 P ocessing Times
The p ocessing imes co esponding o each pa ype and WS we e in oduced in a s ochas ic
en i onmen , in o de o ollow, mo e conc e ely, a iangula p obabili y dis ibu ion.
Assuming ha Ti j ep esen s he p ocessing ime o he pa ype iin he WS j, i was adap ed
o espec he iangula dis ibu ion, shown in Figu e 4.11. In his segmen , a a iabili y ac o α
was also added o simula e possible a ia ions ha may occu in a eal con ex .
4.8 Robus ness o he Simula ion Models 55
Figu e 4.11: T iangula dis ibu ion unc ion, e e ing o he p ocessing imes
In a igo ous way, he espec i e iangula dis ibu ion is de ined by he ollowing exp essions:
(x|a,b,c) =























2(x−a)
(b−a)(c−a),se a≤x<c
2
(b−a),se x=c
2(b−x)
(b−a)(b−c),se c<x≤b
0,o he wise
(4.17)
Whe e a, b and c ake he nex alues:







a=Ti j(1-α)
b=Ti j(1+α)
c=Ti j







(4.18)
The esul s o his p ocedu e, which is based on he in oduc ion o he p ocessing imes in a
s ochas ic en i onmen , a e desc ibed in Chap e 5.
62 Resul s Analysis
Figu e 5.5: Makespan ela ed o Scena io 5, o a o al o 250k ime s eps
Table 5.8: Nume ical in e p e a ion o makespan e e ing o Scena io 5 (in seconds)
Minimum Value Maximum Value Final Value
1630 17202 6645
5.4.6 Scena io 6 – Type o ewa d: be ween ime s eps, wi h penal ies and wi hou
no maliza ion
Analysing he scena ios 2 and 6, bo h using penal ies, i appea s ha he wo ha e simila be-
ha iou s, wi h a g ea e numbe o simula ions ca ied ou in he second scena io. These ques ions
can be obse ed in Tables 5.5 (scena io 2) and 5.9 (scena io 6).
Figu e 5.6: Makespan ela ed o Scena io 6, o a o al o 250k ime s eps
Table 5.9: Nume ical in e p e a ion o makespan e e ing o Scena io 6 (in seconds)
Minimum Value Maximum Value Final Value
1486 3945 1486

5.4 Makespan Model 63
5.4.7 Scena io 7 – Type o ewa d: be ween ime s eps, wi hou penal ies and wi h
no maliza ion
As in he p e ious scena io, he op imal alue is also achie ed (Figu e 5.7). Ne e heless, as
e idenced in scena ios 2 and 3, he use o he no maliza ion in a singula way allows ob aining he
bes solu ion ea lie when compa ed o he isola ed applica ion o penal ies.
Figu e 5.7: Makespan ela ed o Scena io 7, o a o al o 250k ime s eps
Table 5.10: Nume ical in e p e a ion o makespan e e ing o Scena io 7 (in seconds)
Minimum Value Maximum Value Final Value
1486 4096 1486
5.4.8 Scena io 8 – Type o ewa d: be ween ime s eps, wi h penal ies and wi h
no maliza ion
As well as he ou h scena io, and as expec ed, when he no maliza ions and penal ies a e applied
simul aneously, he op imal solu ion is ob ained in ad ance in ela ion o he models in which he
penal ies o no maliza ion a e applied sepa a ely.
Figu e 5.8: Makespan ela ed o Scena io 8, o a o al o 250k ime s eps
64 Resul s Analysis
Table 5.11: Nume ical in e p e a ion o makespan e e ing o Scena io 8 (in seconds)
Minimum Value Maximum Value Final Value
1486 4116 1486
5.4.9 Syn hesis
In sho , ega ding he models p esen ed in Sec ion 5.4 ela ed o makespan, i is possible o s a e
ha he Rein o cemen Lea ning PPO algo i hm was co ec ly implemen ed, gi en ha he models
ollow hei cha ac e is ic cu e, allowing o maximize (-) makespan.
The esul s also sugges ha he no maliza ion o ewa ds and he applica ion o penal ies
imp o e he lea ning capaci y o he model.
5.5 Base Model
Re u ning o he ques ion o he p oduc i i y and aking in o accoun he PPO algo i hm, he esul s
achie ed o he ime ho izons o 36 and 52 hou s we e di ided acco ding o he exis ence o no
o a p e- aining o he neu al ne wo k and a e ep esen ed and indica ed, espec i ely, in Figu es
5.9,5.10,5.11,5.12 and Tables 5.12,5.13,5.14,5.15. Wi h espec o he Agen ’s lea ning pe iod,
as e e ed o in Sec ion 4.7, he numbe o ime s eps was se o 10M. Rega ding he p e- aining,
i espec s he Nea es WS Rule, discussed in Sec ion 3.6.
Fo a be e unde s anding o he in luence o he p e- aining in he lea ning phase, some se s
o obse a ion-ac ion pai s esul ing om his same s age will also be p esen ed. Fo his pu pose,
a p obabili y unc ion will be applied o he ac ions, a e he lea ning phase is pe o med. The
objec i e o his s ep is o e i y in which WS he AGV will ca y ou he espec i e load.
Subsequen ly, simila ly o he p e ious Sec ion (5.4) e e ing o he makespan, he esul s
de i ed om he applica ion o a penal y ac o (-0.5) a e also p esen ed.
5.5.1 Wi hou p e- aining, o a ime ho izon o 36 hou s
Figu e 5.9 allows us o in e ha he model’s beha iou , wi h ega d o p oduc i i y alues, ollows
he PPO op imiza ion algo i hm.
As can be seen, he model eaches he maximum p oduc i i y alues in he i s simula ions.
This phenomenon is explained by he weigh s which a e a bi a ily assigned in he ini ial lea ning
phase o he ne wo k. Subsequen ly, a e a ce ain numbe o simula ions, he model p esen s
lowe p oduc i i y alues, also due o he weigh s de ined by he Agen and because a he begin-
ning o he in o ma ion collec ion, he ne wo k has li le knowledge ye . Howe e , he p oduc i i y
cha ac e is ic cu e allows e i ying ha he p oduc i i y alues end o a inal maximum solu ion,
which app o es ha he lea ning phase was pe o med co ec ly.
The inal alue o p oduc i i y, which comes om he ne wo k’s lea ning p ocess, is 343 pa s
ha en e a Sink.
5.5 Base Model 65
Figu e 5.9: P oduc i i y e e ing o he Base Model wi hou p e- aining, o a o al o 10M ime
s eps and a ime ho izon o 36h
Table 5.12: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model
wi hou p e- aining, o a o al o 10M ime s eps and a ime ho izon o 36h
Minimum Value Maximum Value Final Value
250 347 343
The se o obse a ion-ac ion pai s esul ing om he lea ning phase, wi hou he p e- aining,
a e exempli ied below:
•Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4
Obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [8.3619851e-01, 1.9839258e-04, 7.7551082e-02, 2.0058152e-04, 1.7890350e-04,
6.3502125e-02, 1.9391190e-02, 2.7792549e-03]
In his pa icula si ua ion, he beha iou o he ne wo k is as expec ed, gi en ha he highes
p obabili y is associa ed wi h he WS1.
•Obse a ion-Ac ion pai no2: Pa s in WS3, WS4 | Cu en WS: 4
Obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [1.4122830e-03, 1.3475084e-05, 9.6138531e-01, 2.4895007e-03, 7.2637980e-05,
2.9261060e-02, 3.8612273e-03, 1.5044548e-03]
Wi h ega d o his obse a ion-ac ion pai , he ne wo k’s beha iou does no co espond o
wha would be done by he Nea es WS Rule, gi en ha , in his si ua ion, he AGV would
load a pa in he WS3 and no in he WS4, as he high p obabili y sugges s.
5.5.2 Wi h p e- aining, o a ime ho izon o 36 hou s
Like he model wi hou p e- aining, his one also espec s he cu e e e ing o he PPO op i-
miza ion algo i hm, ep oduced in Figu e 5.10.
66 Resul s Analysis
Howe e , in compa ison wi h he model wi hou p e- aining, i appea s ha he esul s o his
model be o e he lea ning s age a e lowe , in e ms o p oduc i i y. Despi e ha , simila ly o wha
happened in he op imized Milk-Run and Nea es WS Rules, his episode can be explained by he
ac ha , some imes, he emp y a els o he anspo o pa s o dis an WSs, mo e common in
he model wi hou p e- aining, can b ing ad an ages in e ms o he inal esul o p oduc i i y,
educing possible bo lenecks and balancing he machine occupancy a es.
Figu e 5.10: P oduc i i y e e ing o he Base Model wi h p e- aining, o a o al o 10M ime
s eps and a ime ho izon o 36h
Table 5.13: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model
wi h p e- aining, o a o al o 10M ime s eps and a ime ho izon o 36h
Minimum Value Maximum Value Final Value
257 277 274
One mo e ime, he examples o he obse a ion-ac ion pai s, esul ing om he lea ning p o-
cess, e idence he p e ious accomplishmen o he lea ning phase:
•Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4
Obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [9.9999833e-01, 6.0297623e-07, 9.3485424e-22, 1.0542382e-06, 3.6167533e-10,
4.3565872e-16, 4.4415426e-18, 2.8845209e-15]
The beha iou o he ne wo k is he one expec ed, since he WS1 has he highes p obabili y.
•Obse a ion-Ac ion pai no2: Pa s in WS3, WS4 | Cu en WS: 4
Obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [9.00684897e-25, 1.09724185e-16, 1.59646764e-13, 1.00000000e+00, 1.18856031e-
19, 6.28844218e-25, 2.55962060e-27, 2.96594020e-27]
In con as wi h wha happened in he p e ious scena io in which he p e- aining did no
occu , he highe p obabili y is now consis en wi h he expec ed (WS4), acco ding o he
Nea es WS Rule.
5.5 Base Model 67
5.5.3 Wi hou p e- aining, o a ime ho izon o 52 hou s
Acco ding o he g aph in Figu e 5.11, he beha iou o he lea ning phase, like he Model wi hou
p e- aining o he 36-hou ime ho izon, ollows he PPO algo i hm, despi e he ini ial d op o
he p oduc i i y alues.
Rega ding he inal alue o p oduc i i y, his is logically highe han he p oduc i i y e e ing
o he pe iod o 36 hou s.
Figu e 5.11: P oduc i i y e e ing o he Base Model wi hou p e- aining, o a o al o 10M ime
s eps and a ime ho izon o 52h
Table 5.14: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model
wi hou p e- aining, o a o al o 10M ime s eps and a ime ho izon o 52h
Minimum Value Maximum Value Final Value
398 507 501
Fo he same obse a ion-ac ion pai s, he solu ions a e he ollowing ones:
•Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4
obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [8.6369443e-01, 2.2193830e-04, 4.7343463e-02, 6.8649366e-05, 1.9357287e-04,
7.4317701e-02, 1.2128289e-02, 2.0318190e-03]
Fo his case, he beha iou o he ne wo k is he one expec ed (WS1).
•Obse a ion-Ac ion pai no2: Pa s in na WS3, WS4 | Cu en WS: 4
obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [2.6672529e-03, 7.9401660e-05, 9.7478318e-01, 2.3247360e-03, 1.2263234e-04,
1.7269300e-02, 2.2482565e-03, 5.0515484e-04]
In acco dance wi h he esul s o he las model (36h), wi hou p e- aining, i is concluded
ha he lea ning o he ne wo k leads he AGV o do he load in he u he s a ion (WS3)
ins ead o he nea es one (WS4).

68 Resul s Analysis
5.5.4 Wi h p e- aining, o a ime ho izon o 52 hou s
In his las scena io, Figu e 5.12 p o es he expec ed beha iou o he model. As Table 5.15
indica es, he inal alue o p oduc i i y is lowe han he alue ob ained in he scena io in which
he p e- aining did no occu . The mos admissible easons a e p esen ed in Sec ion 5.6 ha
compa es he esul s o all models.
Figu e 5.12: P oduc i i y e e ing o he Base Model wi h p e- aining, o a o al o 10M ime
s eps and a ime ho izon o 52h
Table 5.15: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model
wi h p e- aining, o a o al o 10M ime s eps and a ime ho izon o 52h
Minimum Value Maximum Value Final Value
375 409 404
The ollowing obse a ion-ac ion pai s ollow he conjunc u es pe o med p e iously, simi-
la ly wi h he esul s ob ained in he model wi h p e- aining, e e ing o he ime ho izon o 36
hou s:
•Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4
Obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [9.9983370e-01, 1.7993931e-06, 4.0928912e-19, 1.6300578e-04, 1.4105979e-06,
4.8301262e-15, 1.4212710e-17, 2.4972540e-14]
•Obse a ion-Ac ion pai no2: Pa s in WS3, WS4 | Cu en WS: 4
Obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]
Ac ion: [8.7494940e-28, 5.5310151e-18, 3.6004825e-13, 1.0000000e+00, 3.2771040e-19,
4.0131407e-26, 4.2842155e-29, 7.9547190e-29]
5.6 Compa ison o he FIFO, Op imized Milk-Run, Nea es WS Rule and Base Models 69
5.5.5 Applica ion o Penal ies
A e he penal ies o -0.5 we e applied whene e an emp y mo emen is ca ied ou , i was con-
cluded ha he ou models ollow he expec ed beha iou in ela ion o he PPO algo i hm, as i s
cu e p esen s. Howe e , as Table 5.16 exp esses, he esul s a e in e io .
Since pe o ming emp y mo emen s is no necessa ily a bad p ac ice, his is one o he easons
o which he p oduc i i y alues a e lowe using penal ies.
Table 5.16: P oduc i i y (in pa s) e e ing o he Base Model, wi h penal ies
Base Model P oduc i i y (36h) P oduc i i y (52h)
Wi hou p e- aining 343 501
Wi h p e- aining 274 404
Wi hou p e- aining, wi h penal ies 304 454
Wi h p e- aining, wi h penal ies 271 403
5.6 Compa ison o he FIFO, Op imized Milk-Run, Nea es WS Rule
and Base Models
Ini ially, compa ing all he models p e iously s udied in Sec ions 5.1,5.2,5.3 and 5.5, wi h ega d
o p oduc i i y, he Base Model was he one ha p esen ed he bes esul s. Tha said, he use
o Machine Lea ning echniques combined wi h simula ion app oaches has enabled us o achie e
good solu ions.
Rega ding he Base Model, he di e ence be ween he usage o no o p e- aining has o do
wi h he ac ha , a he beginning o he lea ning p ocess, he esul ing solu ions a e al eady
o no close o a local op imum. I p e- aining occu s, he solu ions de i ed om he lea ning
p ocess will be in close o a local op imum, unlike wha happens in he si ua ion whe e he e is no
p e- aining.
Since he solu ions s a nea a local op imum, he Agen has di icul ies o explo e and ind
new be e solu ions han he ones se led ou side his egion o he solu ion space.
In his sense, i was ound ha he inal p oduc i i y esul s ob ained we e highe in he case
whe e he p e- aining was no ca ied ou (Table 5.17). This is explained by he ac ha , in he
absence o p e- aining, he occu ence o anspo s o dis an s a ions o he de imen o close
s a ions allows he educ ion o bo lenecks and he balancing o he machine occupancy a es.
The ac ha he e is no p e- aining also allows he Agen no o be so easily s agna ed in a local
op imum gi en by he p e- aining, so he e is mo e eedom o explo e new solu ions, which will
allow he Agen o achie e be e solu ions, con a ily o wha happens in he model whe e he e
is p e- aining.
70 Resul s Analysis
Table 5.17: P oduc i i y (in pa s) e e ing o he models in s udy
Model P oduc i i y (36h) P oduc i i y (52h)
FIFO 177 262
Op imized Milk-Run 266 389
Nea es WS Rule 258 377
Base, wi h p e- aining 274 404
Base, wi hou p e- aining 343 501
5.7 Robus ness o he Simula ion Models
A e analysing he models when applied o de e minis ic en i onmen s, i is pe inen o e i y
hei beha iou when applied o o he si ua ions. In pa icula , i is impo an o s udy he in luence
o di e en p oduc ion mixes and p ocessing imes, in o de o examine he adap abili y o he
models, bu only ega ding p oduc i i y, which he e o e excludes he model ela ed o he s udy
o makespan.
I is also impo an o men ion ha , acco ding o he Base Model, he e was no cus omized
e-lea ning o hese scena ios, because he weigh s ob ained in he base scena io (o iginal mix)
we e used.
5.7.1 P oduc ion Mixes
Fi e di e en mixes we e de ined, andomly gene a ed, so ha , as desc ibed in Sec ion 4.8.1, he
numbe o o de s o each pa ype is balanced (≃20%).
The ime ho izons unde s udy a e 36 and 52 hou s o compa ison p inciples, and he AGV
speed also emains cons an a 1.5 m/s, since speed does no cons i u e a ac o unde conside a ion.
Tables 5.18 and 5.19 enable us o conclude ha , o bo h empo al pe spec i es, he Agen
when applied o di e en p oduc ion mixes ensu es ha he p oduc i i y esul s emain, on a -
e age, close o each o he . So, i is possible o alida e he Base Model and i s adap abili y in
di e en p oduc ion con ex s, jus like he o he models.
I should also be no ed ha he Base Model, wi hou p e- aining, emains he one ha ob ains
he bes esul s.
Table 5.18: P oduc i i y (in pa s) analysis e e ing o di e en p oduc ion mixes, o a ime
ho izon o 36 hou s
Model Ini ial Mix Mix 1 Mix 2 Mix 3 Mix 4 Mix 5
FIFO 177 196 179 171 178 174
Op imized Milk-Run 266 261 251 257 261 242
Nea es WS Rule 258 252 239 250 252 235
Base, wi h p e- aining 274 273 259 266 278 252
Base, wi hou p e- aining 343 341 328 314 325 329
5.7 Robus ness o he Simula ion Models 71
Table 5.19: P oduc i i y (in pa s) analysis e e ing o di e en p oduc ion mixes, o a ime
ho izon o 52 hou s
Model Ini ial Mix Mix 1 Mix 2 Mix 3 Mix 4 Mix 5
FIFO 262 271 254 256 268 253
Op imized Milk-Run 389 385 359 379 386 370
Nea es WS Rule 377 366 344 362 371 355
Base, wi h p e- aining 404 409 376 388 394 382
Base, wi hou p e- aining 501 473 471 473 477 478
5.7.2 P ocessing Times
The second c i e ion o e alua e he adap abili y o hese models ela ed o p oduc i i y includes
he in oduc ion o hem in a s ochas ic en i onmen in which, acco ding o Sec ion 4.8.2, he
de ini ion o p ocessing imes ollows a iangula dis ibu ion.
The a iabili y ac o , α, applied o he p ocessing imes o each pa was 0.4. I is also
impo an o compa ison pu poses o main ain he ime ho izons in 36 and 52 hou s and he AGV
speed in 1.5 m/s.
The ollowing Tables 5.20 and 5.21 p o ides he esul s o he 10 simula ions gene a ed and
indi idually applied o each one o he implemen ed models. These esul s a e impo an o analyse
he a e age and s anda d de ia ions o he esul ing p oduc i i y alues.
The low s anda d de ia ion as well as he simila i y be ween he a e age alues o p oduc i i y
ob ained o each one o he models and he alues associa ed wi h he o iginal mix, also indica e
ha all he alues a e in ag eemen wi h each o he . So, simila ly o wha happened wi h he
p oduc ion mixes, we can conclude ha all models p esen good adap abili y when subjec o
s ochas ic en i onmen s.
To conclude, he Base Model, wi hou p e- aining, is he one ha allows he achie emen o
he highe p oduc i i y alues.
78 REFERENCES
[14] R. D ießel and L. Mönch, “An in eg a ed scheduling and ma e ial-handling app oach o
complex job shops: a compu a ional s udy,” In e na ional Jou nal o P oduc ion Resea ch,
ol. 50, no. 20, pp. 5966–5985, 2012.
[15] R. M. Logís ica, “Milk un: Concei os, an agens e ganhos pa a a ope ação
logís ica.” h ps:// e is amundologis ica.com.b /blog/achiles/
milk- un-concei os- an agens-e-ganhos-pa a-a-ope acao-logis ica.
Accessed in 2019-11-23.
[16] R. F. D. San os, “Es a égias híb idas de machine lea ning e simulação pa a a esolução de
p oblemas de escalonamen o,” 2018.
[17] S. Huang, “Op imiza ion o job shop scheduling wi h ma e ial handling by au oma ed guided
ehicle,” 2018.
[18] A. de F ei as Ribei o, “Taylo ismo, o dismo e oyo ismo,” Lu as Sociais, ol. 19, no. 35,
pp. 65–79, 2015.
[19] A. Aze edo, “Lean hinking: p incípios, undamen os, p incipais e amen as.” Aze edo,
Amé ico, Lean hinking: p incípios, undamen os, p incipais e amen as. Sis ema de P o-
dução Toyo a Pa e I, 2013. 28 slides. Suppo ma e ial o he GOPE cou se o MIEEC o
FEUP. Accessed in 2020-01-23.
[20] G. Zäp el, R. B aune, and M. Bögl, Me aheu is ic Sea ch Concep s: A Tu o ial wi h Appli-
ca ions o P oduc ion and Logis ics. Sp inge Be lin Heidelbe g, 2010.
[21] A. Meye , “Milk un design (de ini ions, concep s and solu ion app oaches). disse a ion,
ka ls uhe ins i u ü echnologie (ki ) akul ä ü maschinenbau, ki scien i ic publishing,
ka ls uhe (2015),” 2015.
[22] . S. G. B a , G. S., “Milk un logis ics: Li e a u e e iew and di ec ions, london, england,”
2011.
[23] A. Gé on, Hands-On Machine Lea ning wi h Sciki -Lea n, Ke as, and Tenso Flow: Con-
cep s, Tools, and Techniques o Build In elligen Sys ems. O’Reilly Media, 2019.
[24] Y. Li, “Deep ein o cemen lea ning: An o e iew,” a Xi p ep in a Xi :1701.07274, 2017.
[25] S. Ma sland, Machine lea ning: an algo i hmic pe spec i e. Chapman and Hall/CRC, 2014.
[26] V. Mnih, K. Ka ukcuoglu, D. Sil e , A. G a es, I. An onoglou, D. Wie s a, and M. Ried-
mille , “Playing a a i wi h deep ein o cemen lea ning,” a Xi p ep in a Xi :1312.5602,
2013.
[27] V. H. de Sousa Ca nei o, “Abo dagens híb idas de" machine lea ning"/simulação pa a sis-
emas logís icos dinâmicos,” 2019.
[28] I. G igo ye , Anylogic in h ee days. Fi h Edi .C, 2018.
[29] J. Banks, J. S. Ca son II, L. Ba y, e al., “Disc e e-e en sys em simula ion ou h edi ion,”
2005.
[30] A. K. d. Sil a, Mé odo pa a a aliação e seleção de so wa es de simulação de e en os dis-
c e os aplicados à análise de sis emas logís icos. PhD hesis, Uni e sidade de São Paulo,
2006.

REFERENCES 79
[31] V. H. S. Ca nei o, Hyb id Machine Lea ning/Simula ion App oaches o Decen alized P o-
duc ion Scheduling. 2018.
[32] “Socke p og amming in py hon (guide) by na han jennings.” Accessed in 2020-03-02.
[33] M. Almeida, Luís; San os Ped o; Sousa, “Suppo in o ma ion o lab assignmen 1a cp
socke s in c.” Suppo in o ma ion o lab assignmen 1a TCP socke s in C, Luís Almeida,
Ped o San os, Ma io Sousa, 18 slides. Suppo ma e ial o he ACI cou se o MIEEC o
FEUP. Accessed in 2020-04-15.
[34] A. Hill, A. Ra in, M. E nes us, A. Glea e, A. Kane is o, R. T ao e, P. Dha iwal, C. Hesse,
O. Klimo , A. Nichol, M. Plappe , A. Rad o d, J. Schulman, S. Sido , and Y. Wu, “S able
baselines.” h ps://gi hub.com/hill-a/s able-baselines, 2018.