Hybrid Machine Learning/Simulation Approaches for Logistic Systems Optimization

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Hyb id Machine Lea ning/Simula ion App oaches o Logis ics Sys ems Op imiza ion F ancisco Alexand e Lou enço Maia FINAL VERSION Mes ado In eg ado em Engenha ia Ele o écnica e de Compu ado es Supe iso : Amé ico Lopes de Aze edo Second Supe iso : João Ped o Ta a es Viei a Bas o July 22, 2020 c F ancisco Alexand e Lou enço Maia, 2020 Resumo Hoje em dia, em-se es emunhado um ab up o c escimen o e desen ol imen o da indús ia, e- le ido no ele ado g au de complexidade e in eligência que os sis emas de p odução co en es ap esen am, onde se des acam os sis emas logís icos. Es a incessan e p ocu a pela ino ação e melho amen o con ínuo são mui o eco en es na época a ual, aduzindo-se em cons an es ans- o mações no concei o da qualidade de um p odu o. Des e modo, eme ge a necessidade em o imiza os layou s ab is conduzindo a um aumen o da lexibilidade ace aos seus compo amen os dinâmicos. Nes e seguimen o su ge a imp escindibili- dade de ap imo amen o do compo amen o do eículo au ónomo associado, com is a a inalidades comuns como o aumen o da p odu i idade e minimização de cus os e lead imes. Nes e âmbi o, o obje i o des a disse ação é a combinação de écnicas de Rein o cemen Lea ning com abo dagens de simulação pa a a o imização de um sis ema logís ico job-shop, no que à p odu i idade diz espei o. Pa a além da implemen ação do modelo de simulação do sis ema logís ico, es a disse ação desen ol e ambém numa ase inicial compo amen os elemen a es a aplica ao eículo, imple- men adas no p óp io ambien e de simulação. Pos e io men e, dado que a á ea de Machine Lea ning em ob ido an o sucesso nou as á eas ecnológicas, su giu o desa io da in odução do concei o de ede neu onal, a a és da c iação de uma no a en idade designada Agen e e ca a e izada pela écnica de ap endizagem baseada em Rein o cemen Lea ning. Po im, nes a disse ação, pa a além de se conclui que a abo dagem baseada em Rein o ce- men Lea ning p opo cionou os melho es esul ados de p odu i idade, e i a am-se ainda con- clusões no que à obus ez des es modelos diz espei o, a im de a alia a sua lexibilidade quando sujei os a di e en es con ex os, simulando um ambien e eal. i ii Abs ac Nowadays, we ha e been wi nessing an ab up g ow h and de elopmen o he indus y, e lec ed in he high le el o complexi y and in elligence ha he cu en p oduc ion sys ems p esen , in which he logis ics sys ems s and ou . This incessan sea ch o inno a ion and con inuous imp o emen a e e y common oday, ep oducing in o cons an changes in he p oduc quali y concep . In his sense, he need o op imize he ac o y layou s eme ges, leading o an inc ease in lexi- bili y because o hei dynamic beha iou s. In his segmen , he e is an essen ial need o imp o e he beha iou o he associa ed au onomous ehicle, o each common objec i es such as inc eas- ing he p oduc i i y and minimizing cos s and lead imes. In his con ex , he objec i e o his disse a ion is he combina ion o Rein o cemen Lea n- ing echniques wi h simula ion app oaches o he op imiza ion o a job-shop logis ics sys em, ega ding p oduc i i y. Beyond he implemen a ion o he simula ion model o he logis ics sys em, his disse a ion de elops, in an ini ial phase, elemen a y beha iou s o be applied o he ehicle, implemen ed in he simula ion en i onmen i sel . Subsequen ly, gi en ha he Machine Lea ning a ea has been so success ul in o he echno- logical a eas, he challenge o in oducing he concep o he neu al ne wo k appea s, h ough he c ea ion o a new en i y called Agen and cha ac e ized by he Rein o cemen Lea ning echnique. Finally, in his disse a ion, in addi ion o concluding ha he Rein o cemen Lea ning-based app oach p o ided he bes p oduc i i y esul s, conclusions we e also d awn ega ding he o- bus ness o hese models, in o de o assess hei lexibili y when subjec o di e en con ex s, simula ing a eal en i onmen . iii i Acknowledgemen s My i s wo ds go o Enginee João Bas o and P o esso Amé ico Aze edo o all hei a ailabili y, comp ehension and suppo du ing his mas e ’s disse a ion. Thei help was undamen al in his whole p ocess and I am e y g a e ul o hem o ha . Secondly, I would also like o acknowledge he suppo o he Enginee s Romão San os and Na ciso Caldas o all he a ailabili y and incen i e du ing he disse a ion. In his segmen , I also ha e a wo d add essed o INESC-TEC, in pa icula o he Cen e o En e p ise Sys ems Enginee ing and all i s elemen s o he acili ies o e ed h oughou he disse a ion and o he excellen en i onmen p o ided. I would like o hank he Facul y o Enginee ing o he Uni e si y o Po o and i s p o esso s o e e y hing ha has been ansmi ed o me o e hese 5 yea s, no only academically bu also pe sonally. I ha e o highligh all he iendly ela ionships I ha e c ea ed in hese 5 yea s, namely Ca los Ca alho, wi h whom I sha ed his expe ience in he INESC-TEC du ing his inal semes e , as well as Fábio Quei ós, Edua do Caldas, F ancisco Pi es, And é Oli ei a, And é Cip iano, Lídio Ribei o, Ma ia Pe ei a, Alexand a San os and A u Almeida, among many o he s ha I ake wi h me o he es o my li e. I also hank my amily o all he suppo and mo i a ion ha helped me o go h ough his jou ney, namely my pa en s F ancisco Maia, Ma ia Isilda and sis e Ca olina Maia. Finally, I add ess a wo d o app ecia ion o all my iends o all he encou agemen and s imula ion du ing his pe iod o my li e, no only in he bad momen s bu also in he good ones. F ancisco Maia i A momen o pain is wo h a li e ime o glo y. Louis Zampe ini ii xi LIST OF TABLES 5.19 P oduc i i y (in pa s) analysis e e ing o di e en p oduc ion mixes, o a ime ho izono 52hou s................................. 71 5.20 P oduc i i y (in pa s) in s ochas ic en i onmen s, o a ime ho izon o 36 hou s . 72 5.21 P oduc i i y (in pa s) in s ochas ic en i onmen s, o a ime ho izon o 52 hou s . 72 Abb e ia ions and Symbols AGV Au oma ed Guided Vehicle AI A i icial In elligence FIFO Fi s In, Fi s Ou IB Inpu Bu e IBWSx Inpu Bu e o Wo ks a ion x JIT Jus -in-Time MHS Ma e ial Handling Sys em OB Ou pu Bu e OBWSx Ou pu Bu e o Wo ks a ion x PPO P oximal Policy Op imiza ion RL Rein o cemen Lea ning VRP Vehicle Rou ing P oblem WIP Wo k-in-P og ess WS Wo ks a ion x Chap e 1 In oduc o y Analysis 1.1 Con ex ualiza ion La ely, he e has been an ab up g ow h and de elopmen o he indus y, in which he concep “In- dus y 4.0” appea ed, which allowed he exchange o in o ma ion be ween a a ie y o equipmen s in a ac o y. Namely ega ding he op imiza ion o in e nal p ocesses, as well as a company’s p od- uc s and e en se ices, his pa adigm is e olu ionizing he indus y wo ldwide. Consequen ly, he cu en need and demand o inno a ion and imp o emen ollow he model based on con inuous imp o emen . [A yea wi hou imp o ing is a yea won by compe i o s - J. M. Ju an]. [8] In his sense, he concep o Lean Thinking, ounded by Taiichi Ohno and Eiji Toyoda, which combines he elimina ion o was e (Jus -in- ime - "Any ac i i y ha he cus ome is no willing o pay" - Taiichi Ohno) wi h he immedia e eac ion o any p oblem ha could a ise in du ing a p ocess (Jidoka - Japanese e m), has eme ged in his con ex . The main objec i e o his idea is o inc ease cus ome sa is ac ion, c ea ing signi ican changes in he manu ac u ing p ocesses ha con ibu e o hei be e unc ioning (Kaizen), inc easing p oduc i i y and e iciency. [9] And wha is he eason o his incessan sea ch o inno a ion and op imiza ion o indus ial p ocesses? The answe is p e y simple, cus ome s a e he main eason. Thei demands and needs ha e also been inc easing o e he pas ew yea s, bo h in e ms o a ie y and quali y. A he beginning o he s udy o hese subjec s, i was conside ed ha quali y would only be e- la ed o he p oduc speci ica ions ("Quali y is con o mance o he speci ica ions" - Philip C osby). Howe e , his concep has been cons an ly upda ed, conside ing ha he p ima y ac o is he sa - is ac ion o he cus ome ’s needs ("Quali y is i ness o use" - Joseph Ju an). Fo his, i is i s ly necessa y o in e he p oduc speci ica ions, ollowed by he iden i ica ion o he p obable e o s ha may a ise du ing he p ocesses and hei causes, be o e p oceeding o hei elimina ion. [10] The need o inc ease quali y, dec ease cos s and educe deli e y imes, led o he c ea ion o se e al ypes o ac o y layou s, namely he unc ional (p ocess-o ien ed, job-shop), line (linea lows, low-shop) and ixed ( he p oduc canno be mo ed). In his sense, he indus ies s a ed o adop di e en modes o p oduc ion, namely he job- shop, cha ac e ized by he exis ence o specialized a eas by unc ion, and low-shop, de ined by he 1 2In oduc o y Analysis p oduc ion lines. O e all, he layou s a e designed o minimize Ma e ial Handling cos s, elimina e bo lenecks, educe cycle imes, elimina e was e and inc ease p ocess lexibili y. Cu en ly, many indus ies make use o he job-shop p oduc ion mode, because i is ela ed o a high di e si y o p oduc s p oduced in low olume (p oduc ion o o de ). I also allows he inc ease o he lexibili y o p oduc ion p ocesses, howe e , hey p esen high WIP and queues. [11] In o de o op imize hese logis ics sys ems, Ma e ial Handling is in g ea ocus a he p esen . In o de o ob ain sho e cycle imes and lowe cos s in anspo ing aw ma e ials be ween wo k- s a ions (WSs), he e is a need o de elop new op imiza ion algo i hms. Fo his, i is necessa y o ake in o accoun he ime and space (o he wa ehouse, o example). In o he wo ds, i is necessa y o coo dina e all he asks o each wo ks a ion in o de , o example, o sa is y all o de s wi h he sho es possible lead ime. [12] Di ec ly associa ed wi h hese ma e ial managemen sys ems a e he AGVs (Au oma ed Guided Vehicle) ha allow he ma e ials o be anspo ed be ween s a ions, and a e gene ally unmanned. Cu en ly, ehicle ou ing p oblems a e in he spo ligh and hei objec i e is o a el he sho es dis ance possible, minimizing cos s. This ype o p oblem has some common cha ac e is ics o he "T a eling Salesman" p oblem, in which i is supposed o isi a ce ain numbe o ci ies co e ing he sho es possible dis ance. In he case o indus ies, each ehicle has a maximum capaci y. Mos o hese p oblems can be sol ed using he Milk-Run sys em, which is a deli e y sys em ha allows us o educe s ocks, educe was e and op imize ou es. I is a deli e y sys em in which one p oduc is deposi ed and ano he one is collec ed igh a e , in o de o sa e ime. I should also be no ed ha he AGV’s ou e is ixed. The collec ion o p oduc s om supplie s is ca ied ou on a scheduled basis, in s ipula ed quan i ies, making cycle imes mo e p edic able. [13] Besides heu is ics such as Milk-Run, o he app oaches, based on me aheu is ics, ha e been also de eloped o sol e his kind o p oblems. [14] Allied o hese app oaches, he concep o Machine Lea ning eme ges. Since i has been qui e success ul in o he a eas o echnology, why no make use o his ool and apply i o p oduc ion sys ems? I is p ecisely hese issues ha a e cu en ly being s udied and de eloped. 1.2 Mo i a ion The g owing e olu ion o he indus y due o he con inued inc ease in compe i i eness has led logis ics managemen o be in ogue hese days. In o he wo ds, i s p ocesses ha e unde gone signi ican changes, bo h inancially and empo a ily, in addi ion o he objec i e o making he sys em mo e obus and secu e. The impo ance o educing human-made ailu es has led o he use o AGVs, which, besides he sa e y and p ecision issues, also con ibu e o he au oma iza ion o p oduc ion sys ems, wi h ega d o he sequence o ope a ions. Fo his pu pose, ce ain heu is ics we e c ea ed and de eloped ha allowed, o example, o minimize he cycle imes o a p oduc ion line as well as he cos s associa ed wi h Ma e ial 1.3 Objec i es and Resea ch Ques ions 3 Handling. I should also be no ed ha a la ge pa o he lead ime is spen on anspo , wi h only a mino i y ocusing on p oduc ion p ocesses. One o he mos impo an o mula ions ha eme ged, aking in o accoun he need o accel- e a e he low o ma e ials be ween loca ions, was he concep o Milk-Run sys ems. These ones allow ha , in a deli e y sys em, when a ce ain p oduc is deli e ed o a s ipula ed loca ion, ano he one is also collec ed, sa ing ime in anspo a ion. In sho , hese ypes o sys ems con ibu e o he in eg a ion be ween he logis ics sys ems and supply chains. [15] Tha said, he Milk-Run concep was la e anspo ed o a ac o y layou con ex , in which each loca ion co esponds o a wo ks a ion. As men ioned, hese heu is ics ensu e he esolu ion o p oblems such as Job-Shop Scheduling and Ma e ial Handling. In o de o sol e hese op imiza ion p oblems, some ools like Machine Lea ning (lea ning algo i hms such as neu al ne wo ks) can be combined o sol e ma e ial mo emen p oblems. This ool has a g ea p ominence nowadays and allows o make p edic ions aking in o accoun da a collec ed p e iously, e en in highly complex en i onmen s. In his segmen , one o he pa adigms o Machine Lea ning ha eme ges is Rein o cemen Lea ning (RL). Based on he en i onmen in ques ion and he decisions aken by he Agen , his lea ning me hod allows ecei ing eedback ha indica es he bes decision made so a . [16] In summa y, he s udy o he in luence ha hese me hods ha e on sol ing dynamic ehicle ou ing p oblems associa ed wi h he anspo o ma e ials be ween wo ks a ions in a ac o y is e y in e es ing and allows o c ea e e y e ec i e and adap able algo i hms. These algo i hms based on Rein o cemen Lea ning can be applied ega dless o he complex- i y o he sys em, and he e is no need o change he ac o y layou . In addi ion o all o his, he in es men is low and he esul s achie ed a e sa is ac o y. In conclusion, he cons an need o op imize bo h p oduc ion imes and anspo , imp o ing he ou es, leads us o ques ion he ac ha , i Machine Lea ning has been so success ul in o he a eas, why no adap i o he p oduc i e sys ems and combine i wi h simula ion app oaches o he op imiza ion o logis ics sys ems. [17] 1.3 Objec i es and Resea ch Ques ions The main objec i e o his disse a ion is he combina ion o echniques based on Machine Lea n- ing wi h simula ion app oaches o logis ics sys ems op imiza ion. Fi s ly, i is necessa y o model he p oblem, unde s and he ac o y’s ope a ion (layou ), c ea e he simula ion model ( h ough he Flexsim so wa e ool) and, inally, de ine simple decision ules o command he AGV. Subsequen ly, i is possible o in eg a e aining algo i hms based on Rein o cemen Lea ning echniques, which de ine a comple ely dynamic beha iou o he logis ics sys em, o inc ease he p oduc i i y and minimize he makespans ( he o al ime o comple e a sequence o asks). These algo i hms, made a ailable by OpenAI Baselines, a e conside ed he s a e-o - he-a o Rein o ce- men Lea ning, nowadays. 4In oduc o y Analysis Finally, i is essen ial o es whe he he algo i hm adap s o changes in he ac o y, wi h espec o changes in he p oduc ion mix o in he s ochas ici y o p ocessing imes. In o de o be able o accomplish hese objec i es, we need o answe he ollowing esea ch ques ions (RQ): •RQ 1: How can we model a ac o y’s ope a ion in a simula ion model wi h simple decision ules o command he AGV? •RQ 2: How can we in eg a e aining algo i hms based on RL echniques wi h a simula ion model o s udy he ac o y’s p oduc i i y and makespan? •RQ 3: How can we e alua e he obus ness o he p oposed app oaches? 1.4 Me hodological App oach This disse a ion ollows he s udy o a se o hyb id app oaches o simula ion and Machine Lea n- ing, whose consequen pu pose is he op imiza ion o a logis ics sys em, leading o an inc ease in i s p oduc i i y and minimiza ion o makespan. The ini ial phase o he disse a ion is alloca ed o he cons uc ion o he simula ion model o he job-shop layou , using he Flexsim so wa e, in which an au oma ed guided ehicle will also be inco po a ed. Pos e io ly, he e a e wo elemen a y decision ules ha will be implemen ed, which p o ide he indica ion o he wo ks a ion (WS) whe e a load o an en i y will be pe o med. The i s ule, e e ing o he inaugu al simula ion model, is based on he Fi s -in Fi s -ou (FIFO) algo i hm, while he second model is a Milk-Run op imized anspo sys em. Then, he concep o communica ion be ween he simula ion en i onmen and an ex e nal p o- g am will be in oduced, which will allow he c ea ion o a dis ibu ed sys em, because he las wo ules add essed we e implemen ed in he simula ion model. This ex e nal p og am, called Se e , uses he Py hon language and i will communica e wi h he simula ion en i onmen , becoming e- sponsible o AGV decision making. This new s age also equi es ha he communica ion be ween he wo en i ies has o be p elimina ily es ablished, using he TCP communica ion p o ocol. A he beginning o he inal s age, he concep s o Machine Lea ning will be in oduced and discussed, namely he in oduc ion o a hi d en i y called Agen , which implemen s a neu al ne wo k o decision making. This ne wo k will be esponsible o de ining he WS whe e he AGV will load he espec i e en i ies. Fo ha , he Agen ecei es om Flexsim, h ough he models subsequen ly de eloped, a se o ele an in o ma ion, namely obse a ions o he cu en s a e o he plan and he cu en loca ion o he AGV. Howe e , i should be no ed ha he Se e is s ill p esen , assuming he ole o an in e media y be ween he Agen and Flexsim. Consequen ly, h ough a Rein o cemen Lea ning algo i hm, called P oximal Policy Op imiza- ion (PPO), he neu al ne wo k will de e mine he espec i e WS whe e he AGV has o load an en i y. A p e- aining p ocess o he Agen will also be s udied and applied, which will allow he 1.5 Disse a ion O ganiza ion 5 imp o emen o i s lea ning phase and, consequen ly, lead o a possible maximiza ion o p oduc- i i y ("Base Model") and minimiza ion o makespan ("Makespan Model"). Finally, he p oduc ion mixes will be changed and he concep o p obabili y dis ibu ion will be in oduced. In his sense, he models p e iously discussed will be applied o a s ochas ic en i- onmen , in o de o e i y i hey p esen a high le el o obus ness so ha hey can be applied in a eal con ex , whe e he p ocessing imes a e subjec o cons an changes. All hese phases a e ep esen ed in Figu e 1.1. Figu e 1.1: Disse a ion app oach me hod 1.5 Disse a ion O ganiza ion Wi h ega d o he o ganiza ion and s uc u e o his documen , i is di ided in o six chap e s. The i s chap e p e ends o in oduce he heme o he disse a ion, including i s con ex ual- iza ion, mo i a ion, objec i es and esea ch ques ions, as well as he schema ic o he me hodolog- ical app oach ollowed. Chap e 2p esen s he s a e-o - he-a o he subjec s co e ed by he disse a ion, like he in- dus ial p oduc ion sys ems, in pa icula he job-shop ype, Ma e ial Handling Sys ems, Machine Lea ning concep s and disc e e-e en simula ion. The desc ip ion and cha ac e is ics o he p oblem, as well as he desc ip ion o i s me hod- ology, a e co e ed in chap e 3, e en as he implemen a ion o he i s h ee anspo sys ems conside ed h oughou he disse a ion. Chap e 4 e lec s wo addi ional anspo sys ems, in oducing Rein o cemen Lea ning ech- niques in combina ion wi h simula ion app oaches. The obus ness o he sys ems is ela ed o he p oduc ion mixes and p ocessing imes, which a e also add essed. 6In oduc o y Analysis The esul s ob ained a e exposed and ea ed in chap e 5wi h ega d o he i e anspo sys ems de eloped and hei obus ness. Finally, chap e 6p esen s he conclusions o he disse a ion p ojec and he sugges ions o a u u e wo k. Chap e 2 S a e-o - he-A This chap e aims o p esen he esul s o a bibliog aphic sea ch, in o de o in e nalize, in a simple way, he concep s ha will be add essed h oughou he disse a ion. Fi s ly, in sec ion 2.1, he concep s o indus ial p oduc ion sys ems will be in oduced in gene al and, la e , in sec ion 2.2, he job-shop p oduc ion sys ems will be add essed. Then, sec ion 2.3 p esen s some in e p e a ions o he logis ics sys ems associa ed wi h Ma e ial Handling in indus ial en i onmen s. In sec ion 2.4 he ques ion o Ma e ial Handling Sys ems (MHS) will be add essed, and in he nex sec ion (2.5) he p inciples e e ing o Milk-Run sys ems will be in oduced. Sec ion 2.6 in oduces he subjec o Machine Lea ning, wi h special e e ence o he Rein- o cemen Lea ning echnique. Finally, sec ion 2.7 p esen s he main concep s associa ed wi h simula ion app oaches. 2.1 Indus ial P oduc ion Sys ems The cons an e olu ion o he ma ke means ha , nowadays, socie y seeks o di e en ia ed p od- uc s, which ollows a pe spec i e o di e si y a he han quan i y. In his way, he specialized indus ial o ganiza ions ha e as main objec i e he inc ease o he e ec i eness and e iciency o hei p oduc ion p ocesses. Abou ypologies, he e a e sys ems whose objec i e is o p oduce p oduc s on a la ge scale wi h a low deg ee o a ie y and a e also cha ac e ized by i s high p oduc i i y, low quali ica ion o hei ope a o s, educed complexi y o ac o y managemen and educed lexibili y. This ype o sys em is called as p oduc o ien ed. On he o he hand, he e a e sys ems ha a e o ien ed o he p ocess and o he cus ome , in which i s main goal is o sa is y he cus ome ’s needs, acco ding o a pull pe spec i e, and o alue he quali y o he p oduc . This ype o sys em gi es mo e impo ance o he p oduc a ie y ins ead o quan i y, and i is conside ed mo e lexible and deno ed by g ea e complexi y o ac o y managemen . 7 14 S a e-o - he-A •De e mined ime pe iods assignmen p oblem - known ou es and imes. Figu e 2.7: Rep esen a ion o he ca ego ies o milk- un in-plan dis ibu ion p oblems, adap ed om [4] 2.5 Milk-Run A Milk-Run Sys em is di ec ly ela ed o he concep o anspo a ion in he indus y. The need o educe s ocks and cos s in anspo was he eason o he c ea ion o his model. This idea o cos educ ion ollows he JIT philosophy o Lean undamen als. B ie ly, his is a deli e y sys em ha allows us o op imize ou es and educe was es be ween wo ks a ions in a ac o y plan (in-plan ). Wi h ega d o he Milk-Run sys em, Baudin [22] s a es: “This concep allows o mo e small quan i ies o a la ge numbe o di e en i ems wi h p edic able lead imes and wi hou mul iplying anspo cos s" [22] - (Baudin, 2004 [13])” All hese aspec s led Lean manu ac u e s o choose o o ganize hei anspo acco ding o ixed imes and ou es, in he o m o a Milk-Run sys em. The design o hese sys ems in ol es a highe complexi y, which acco ding o Meye [21] can be desc ibed in he ollowing h ee ac o s: •T anspo o ma e ials; •F equency o anspo s; •Rou e scheduling. 2.5.1 Ad an ages o a Milk-Run Sys em Compa ing o he adi ional app oach, shown in Figu e 2.4, he Milk-Run sys em p esen s he ollowing ad an ages: 2.5 Milk-Run 15 In en o y Reduc ion As Figu e 2.8 shows, he Milk-Run ypology, in cases X and Z, allowed he in en o y le el o be educed by 1/3, because he equency o anspo was inc eased by h ee uni s. In case Y, he equency o anspo inc eased wice, esul ing in a dec ease in he s ock o 2/3 o he le el o he poin - o-poin ypology. Figu e 2.8: Compa ison be ween he in en o y le els o a Poin - o-poin and Milk-Run ypology, adap ed om [5] Replenishmen wi h P edic able Lead Times On a daily basis he e a e housands o p oduc s wi h di e en anspo equencies. Howe e , h ough Figu e 2.8, which p esen s only h ee di e en p oduc s (X, Y, Z), i is possible o gene - alize o o he cases. So, i can be concluded ha hei lead imes a e p edic able. Because we ha e access o da a which indica es an inc ease o dec ease in he equency o supply, i becomes p edic able o es ima e he s ock a ia ions ha will occu in he u u e. Be e In en o y Visibili y In he case o he poin - o-poin deli e ies, he e may be a case whe e only he p oduc X is anspo ed on a la ge scale, causing an almos o al emp ying o he espec i e shel es. Howe e , his si ua ion does no ep esen any ype o anomaly in he in en o y. In he case o he Milk-Run sys em, because he quan i ies anspo ed a e p ac ically he same o all p oduc s, any signi ican a ia ion in he amoun o any ype o p oduc p esen on he co esponding shel is an immedia e sign o he p esence o an anomaly. So, he Milk-Run ypology has a be e isibili y o he in en o y, allowing o ac quickly in case o abno mali y. 16 S a e-o - he-A Imp o e Communica ion Skills wi h Supplie s Using Milk-Run enables supplie s o be in egula con ac wi h he cus ome s, because o he e- quency o supply. The e o e, i is possible o ob ain eedback om consume s abou he quan i ies and quali y o he deli e ed p oduc s, which makes i possible o imp o e he deli e y sys em and he u u e quali y o he p oduc s. [13] 2.6 Machine Lea ning The Milk-Run sys ems need o adap since he cu en eali y o he ac o y layou s has led his sys em o become dynamic so ha i is necessa y o de elop decision algo i hms. So, he con- cep o Machine Lea ning was in oduced in he scope o he anspo o ma e ials in a job-shop en i onmen , mo e speci ically he Rein o cemen Lea ning echnique. This concep a gues ha lea ning om in e ac ions is a undamen al idea subjacen o almos all heo ies o lea ning and in elligence. [23] Basically, Machine Lea ning aims o lea n based on p e ious da a and make p edic ions o decisions o he u u e. [24] Acco ding o A hu Samuel, pionee in a i icial in elligence: “Machine Lea ning is he ield o s udy ha gi es compu e s he abili y o lea n wi hou being explici ly p og ammed" - A hu Samuel, 1959 [23] Enume a ing some o i s applica ions [23]: •Analyse p oduc images on a p oduc ion line o au oma ically classi y hem; •De ec umou s h ough b ain scans; •Summa ize long documen s au oma ically. The Machine Lea ning sys ems can be classi ied in o h ee ca ego ies, acco ding o i s lea ning p ocesses: •Supe ised Lea ning A se ies o examples (inpu s) wi h he co ec answe (ou pu s) is p o ided by an ex e nal supe iso y Agen and, based on aining, he implemen ed algo i hm gene alizes he co ec answe o ano he se o inpu s, a e wa ds. [25] •Unsupe ised Lea ning The de eloped algo i hm ies o iden i y simila i ies be ween he inpu s, ca ego izing hem. One o he bes -known echniques is he clus e ing. [25] •Rein o cemen Lea ning I is loca ed be ween Supe ised Lea ning and Unsupe ised Lea ning. The algo i hm is in o med abou he quali y o he esponse, bu i is no in o med abou how o co ec i . 2.6 Machine Lea ning 17 So, i is necessa y o explo e and expe imen o he di e en possibili ies un il he Agen disco e s how o ob ain a highe quali y esponse. In his pa icula disse a ion, i is pe inen , wi h ega d o he anspo o ma e ials in a job-shop en i onmen , o analyse, essen ially, he Rein o cemen Lea ning ield. 2.6.1 Rein o cemen Lea ning Lea ning how o con ol Agen s di ec ly om high-le el senso y in o ma ion, such as ision and speech, is one o RL’s longs anding challenges. [26] The Agen is no speci ically old wha ac ions o ake, unlike wha happens in o he o ms o Machine Lea ning, like Supe ised Lea ning. The e o e, he Agen will ha e o ind ou which ac ions, delibe a ed so a , allowed him o ob ain g ea e ewa ds, a he end o he lea ning phase. In e es ingly, he ac ions aken in he p esen will a ec he espec i e ewa d, as well as he u u e ones. The concep s o " ial and e o sea ch" and "delayed ewa d" a e he wo mos impo an cha ac e is ics o RL. Unlike Supe ised Lea ning, which is a way o lea ning based on examples p o ided aking in o accoun he knowledge o an ex e nal supe iso , his is no appliable o an in e ac i e lea ning. This is because, in in e ac i e p oblems, in mos cases, i is impossible o ob ain examples o he desi ed beha iou ha a e co ec and ep esen all he si ua ions in which he Agen needs o ac . Hence, he RL allows he Agen o decide wha ac ion o ake, aking in o accoun his own expe ience. The Agen will ha e o check he decisions he has made in he pas and ind ou i , in ac , he ob ained a bene icial ewa d, so ha he can hen la e ca y ou his ac ion. In his sense, a new pa adigm appea s, in which he Agen , in addi ion o explo ing knowledge ha he al eady has om p e ious si ua ions, also needs o explo e new decisions ne e made be o e, o see i he ge s a highe ewa d. Nei he o hese pa adigms is conside ed be e han he o he because in bo h cases, he Agen will ail (ob ain a lowe ewa d) and he solu ion s a es in he c i ical capaci y o he Agen , so he has o pe o m se e al es s, in o de o ind he solu ion ha p o ides him wi h a inal alue co esponding o he bigges ewa d. [6] 2.6.2 Rein o cemen Lea ning Cha ac e is ics Agen The Agen is he en i y ha i is esponsible o make he decisions and i is called “lea ne ” and “decision make ”. Mo e speci ically, he Agen and he en i onmen in e ac wi h each o he in he o m o dis- c e e ime in e als ( =0,1,2, ...). As Figu e 2.9 p esen s, o each , he Agen ecei es a ep e- sen a ion o he s a e o he en i onmen s , such ha s ∈S ep esen s he se o all possible s a es. Consequen ly, he Agen ecei es a esponse in he o m o a ewa d +1∈Rand a new s a e o he en i onmen , s +1, which is a eedback o he nex decision o make a +1. [6] 18 S a e-o - he-A En i onmen The en i onmen is esponsible o in o ming he Agen o he cu en s a e and he ewa d ob- ained o he ac ion aken p e iously. I also ells he Agen a se o all possible s a es. Ac ion The ac ion is he esul o he decision made by he Agen . The objec i e is o ind he bes solu ion, which co esponds o choosing he ac ion ha allows him o ob ain he highes ewa d because each ac ion o igina es di e en ewa d alues. Rewa d The ewa d is a eedback in which he Agen e alua es he consequences o his ac ion aken in he p e ious s a e. I is impo an o e e again ha he objec i e is o ob ain he g ea es possible accumula ion o ewa ds, keeping in mind ha a la ge ewa d ob ained in a gi en s a e does no necessa ily mean ha he inal accumula ion o ewa ds will be he bes . This is because, al hough a speci ic ewa d in a gi en s a e is he la ges one, i may lead o a non-ideal si ua ion in he u u e and in luence nega i ely he ollowing ewa ds. [27] Policy π I is a mapping s a egy ha allows he Agen o decide he nex ac ion o ake, in o de o ob ain a good accumula ed ewa d in he long e m. The RL speci ies how he Agen can change his policy, aking in o accoun his expe ience. The Agen can also be classi ied acco ding o policy, alue unc ion and model. A policy, π, is a mapping o s a es s∈Sand ac ions a∈A(s), o he p obabili y π(s,a)o aking an ac ion a he ime o a s a e s. The alue o he s a e sunde a policy πis s ill deno ed by Vπ(s). [6] Figu e 2.9: Agen -En i onmen in e ac ion diag am, adap ed om [6] 2.7 Simula ion Modelling is a ool ha allows us o sol e eal con ex p oblems. Mos o he ime we canno a o d o expe imen and es eal objec s, in o de o ob ain he bes solu ion, since hese objec s a e, in gene al, expensi e and e en sca ce. Thus, he simula ion assumes a undamen al ole in his 2.7 Simula ion 19 con ex , in o de o be able o sol e p oblems in a p ac ical way and wi hou colla e al damage, con i med by he ollowing quo e adap ed om [28]. “Modeling consis s o inding he pa h o he p oblem o i s solu ion, in a isk- ee wo ld whe e we can make mis akes, undo hings, go back in ime and s a again.” [28] In addi ion o he pu pose o modelling, he e a e mo e bene i s o using his me hod, such as [29]: •Simula ion models allow us o analyse sys ems and ind solu ions in which he analy ical models ail; •I allows es ing new policies, ope a ing p ocedu es, decision ules, in o ma ion lows wi h- ou making changes o he eal sys em; •Disco e he bo lenecks. Howe e , despi e all hese gains, he inal esul s can some imes be di icul o in e p e . The ac ha he simula ion equi es a lo o aining is one o he leas a ou able poin s. 2.7.1 Componen s o a Sys em In o de o be e unde s and he cons i u ion o a simula ion sys em, his sec ion desc ibes i s main elemen s. [29] Sys em A se o en i ies ha in e ac wi h each o he in o de o achie e he ou lined objec i es. Model An abs ac ep esen a ion o a sys em, which allows i o desc ibe in e ms o i s s a e, en i ies, p ocesses, e en s, ac i i ies, and o he s. Sys em S a e A necessa y se o a iables o desc ibe he sys em. En i y An objec o in e es which equi es an explici ep esen a ion. A ibu e An en i y p ope y. 20 S a e-o - he-A E en An ins an occu ence ha can change he s a e o he sys em. Ac i i y The ime du a ion ha a ask needs o be execu ed, known a he momen i s a s. Delay The excess ime in e al, only known when i ends. Clock A a iable ha ep esen s he simula ed ime. A me hod is a s uc u e used o map eal sys ems in simula ion models. Wi h ega d o he simula ion modelling me hods, his can be o ganized in o h ee ca ego ies: Agen -Based, Sys em Dynamics and Disc e e-E en Modelling. The use o each one o hese me hods depends on he sys em o be implemen ed and i s objec i es. Agen -Based I is pa o he class o compu a ional models o simula ing ac ions and in e ac ions be ween au onomous Agen s (indi idual o collec i e). Despi e he lack o knowledge o he sys em’s beha iou and inabili y o ep esen he p ocess low, he main objec i e is o e i y and s udy he e ec o he Agen s on he sys em as a whole. [28] Sys em Dynamics John S e man [28] s a es ha his model is a pe spec i e and a se o concep ual ools ha enables he unde s andmen o he s uc u e and dynamics o complex sys ems. This model is also a igo ous modelling me hod ha allows o build o mal simula ions o complex sys ems and use hem o design mo e e ec i e policies and o ganiza ions. Disc e e-E en Modelling I allows concei ing he modelling o a sys em as a disc e e sequence o e en s in ime. Each e en occu s a a pa icula ime and causes a change in he s a e o he sys em. This sys em equi es ha modelling has o be seen as a p ocess so ha i is a sequence o ope a ions pe o med by Agen s. [28] 2.7 Simula ion 21 2.7.2 Disc e e-E en Simula ion Nowadays, his simula ion me hod is widely used in he modelling and analysis o p oblems in he a ea o logis ics sys ems, as i allows he s udy o aspec s such as p ocesses, scheduling and esou ce alloca ion. Heal h, business p ocesses and mili a y applica ions a e also a eas co e ed by his modelling me hod. Consequen ly, all hese ad ances ha e led o a so wa e de elopmen . [30] In he speci ic case o his disse a ion, he simula ion will be used o e alua e he in e nal ma e ial low o a manu ac u ing plan , in o de o make conclusions ega ding he ac o y’s p o- duc i i y and alues o makespan, so we a e able o iden i y possible bo lenecks. The e o e, his ool is used as a me hod o analysing and sol ing he ollowing p oblems, associa ed wi h job-shop en i onmen s: •E alua e he e ec o changing ma e ial anspo ou es be ween wo ks a ions; •Analyse he phenomenon o esou ce alloca ion and bo leneck p e en ion; •Assess he impac o changing he elemen s in he layou on pe o mance. Th ough published a icles dedica ed o he s udy o his a ea o simula ion, i is possible o d aw examples o applica ions in dis ibu ion and anspo sys ems. Hugan (2001) elabo a ed a s udy ha allowed him o e alua e he in e nal a ic o a Gene al Mo o s au omobile plan , in he USA, which was based on he JIT model o Lean manu ac u e. The simula ion allowed o imp o e he in e nal ou es o each ype o p oduc , as well as o es ima e he a e age ime spen by a p oduc in he ac o y, om i s en y o i s exi . Kuo, Chen, Selikson and Lee (2001) used he simula ion o disc e e-e en s o s udy he low o ma e ials also in a manu ac u ing plan , which allowed hem o ha e a deepe knowledge o ope a ions and logis ics p ocesses. [30] 2.7.3 Cons uc ion S eps o a Simula ion Model P oblem Fo mula ion The p oblem mus be well o mula ed so ha he e is no doub . The e a e s ill cases whe e i is necessa y o p oceed wi h a o al o pa ial e o mula ion o he p oblem. [29] Es ablishmen o Objec i es and Gene al P ojec Plan The objec i es will be he ques ions o be answe ed h ough he simula ion. A e deciding which simula ion me hod is he mos sui able, i is necessa y o lis a se ies o al e na i es o he simu- la ion and ind ways o e alua e he e ec i eness o hese same solu ions. Besides he objec i es de ined o he end o each s a e, i is also impo an o men ion in he plan he numbe o people in ol ed, as well as he associa ed cos s and he es ima ed ime o each phase o he p ojec . [29] 22 S a e-o - he-A Model Concep ualiza ion Acco ding o P i ske (1998), al hough i is no possible a p io i o de ine he ins uc ions ha will lead o a success ul model, he e a e some poin s o iew ha mus be ollowed o he model o be success ul. I is necessa y o s a by de ining a simplis ic model and, om he e, make he necessa y changes s ep by s ep o ob ain good esul s. [29] Da a Collec ion Da a collec ion is di ec ly associa ed wi h he cons uc ion o he model, since he da a collec ed will se e as inpu o he model. As his collec ion ills la ge in e als o ime and his is a e y impo an aspec in he elabo a ion o he model, i is essen ial o s a he collec ion as soon as possible. [29] Model T ansla ion This phase consis s o con e ing he model in o a simula ion language using speci ic so wa e p og ams. In he speci ic case o his disse a ion, he so wa e used is he Flexim p og am. [29] Ve i ica ion and Valida ion This s ep enables o check i he p og am is p epa ed o he simula ion model, ca ying ou e - i ica ion and debugging es s. By compa ing he model wi h he beha iou o he cu en sys em, i is use ul o use his eedback in o de o imp o e he model. The p ocess is epea ed i e a i ely un il he esul ob ained is sa is ac o y. [29] Expe imen al Design The al e na i es p e iously de ined in he “Es ablishmen o Objec i es and Gene al P ojec Plan” phase ha mus be simula ed, mus be de e mined. [29] P oduc ion and Analysis o Resul s A e se e al es s o he model, wi h di e en da a, he esul ed analyses a e used o es ima e he pe o mance o he sys em ha was simula ed. [29] Documen a ion A e he simula ion, i is necessa y o epo wo ypes o da a: p og am and p og ess. I he p o- g am is used by o he s in he u u e, he p og am documen a ion indica es he modes o ope a ion and beha iou o he p og am, as well as o he undamen al aspec s. In ela ion o he p og ess epo , his is essen ial o he model o ob ain c edibili y and ce i ica ion. [29] 2.7 Simula ion 23 Implemen a ion The success o he implemen a ion will depend on each phase p e iously e e ed. [29] 30 P oblem and Me hodology Figu e 3.4: G aphical ep esen a ion o a WS – Flexsim 3.3.4 T anspo As p e iously indica ed, he anspo o he en i ies is ca ied ou by an AGV, which is c ea ed om an objec o he Task Execu e class (Flexsim Lib a y). The eby, i is only necessa y o indica e which s a ions he AGV will collec (OB) and deli e (IB) pa s. This in o ma ion can be ound in de ail in he p e ious chap e s o P ocessing, 3.2.2, Sequencing, 3.2.3 and P oduc ion Plan 3.2.4. I is also wo h no ing ha he AGV speed will be 1.50 m/s o all simula ion models. 3.3.5 Load All he pa s come in in o he sys em h ough he Sou ce, howe e , in his pa icula case, his esou ce has been eplaced by a Queue, called "Sou ce1", which admi s he same beha iou . The main eason o using a Queue objec ins ead o a Sou ce is ha i is possible o obse e he accumula ion o pa s h oughou he simula ion. A las , he esou ces whe e he pa s a e loaded a e he Sou ce and he ou pu bu e s. 3.3.6 Unload The inal unloading o he en i ies is ca ied ou a Sink, and he in e media e unloads be ween WSs ake place in he espec i e inpu bu e s (IBWS). 3.3.7 P ocesses The e a e essen ially wo ools ha allow he building o a simula ion model: 3D model and P ocess Flow. Rega ding P ocess Flow, i always and in any ci cums ance o e laps he 3D model. Howe e , he coope a ion be ween bo h is essen ial, al hough, in his pa icula si ua ion, i is in he P ocess Flow ha he low o en i ies and esou ces will be de ined. In gene al, all anspo sys ems in ol e h ee la ge blocks: C ea ion o Pa s (Figu e 3.5), P ocessing o Pa s (Figu e 3.6) and T anspo o Pa s. Bo h phases o c ea ion and p ocessing o pa s a e common o all models co e ed in his disse a ion. 3.3 Disc e e-E en Simula ion Model – Flexsim 31 3.3.7.1 C ea ion o Pa s The "Sou ce" block de ines he ime when he p oduc ion o de s will be eleased, using an Excel ile ha con ains he in o ma ion o his pu pose. Subsequen ly, he “C ea e Objec and Type” block allows he c ea ion o he en i ies and assigns hem o he co esponding pa ype, aking in o accoun he Label e e ing o he pa ype o he P oduc ion Plan able (Table 3.4). Taking in o accoun he 5 di e en pa ypes, he e is a need o de ine colou s o each pa ype, in o de o dis inguish hem du ing he pe o mance o he simula ion, accomplished by he “Change Colo ” block: •1 - Aqua; •2 - Red; •3 - Blue; •4 - Yellow: •5 - Lime. Conside ing he ype o he cu en pa , he block “De e mine Line” allows o go h ough he Rou ings Table (Table 3.5) and iden i y he line in ques ion, in o de o ollow he WS sequence o he pa ype in e idence. Subsequen ly, a e iden i ying he WS whe e he i s unload will be pe o med, i is necessa y o acqui e he AGV in o de o alloca e i , so ha he ini ial load can be execu ed in he Sou ce and he unload in he IB o he ollowing WS. A he end o he p ocess, he esou ce is eleased ("Release"). Figu e 3.5: Rep esen a ion o he “C ea ion o Pa s” Block 3.3.7.2 P ocessing o Pa s When an en i y en e s he IB o a WS, i is necessa y o mo e i o he WS i sel , hen ope a e i and, inally, mo e i one mo e ime o he espec i e OB. Howe e , i mus be no ed ha he en i y will be jus mo ed om he cu en IB o he co esponding WS i he e a e no pa s in 32 P oblem and Me hodology he espec i e WS, because i s maximum capaci y is uni a y. These es ic ions a e de ined by he c ea ion o zones, ocused on each one o he exis ing WS. A he end o he p ocessing, he block “Exi Zone” will elease he espec i e WS om he due en i y al eady p ocessed, and i will be a ailable o ecei e a new pa . Then he line in he Rou ing Table (Table 3.5) is inc emen ed by one, in o de o ollow he p e-de ined sequence o he pa . Figu e 3.6: Rep esen a ion o he “P ocessing o Pa s” Block 3.4 Implemen a ion o a FIFO T anspo Sys em Amongs all he anspo sys ems co e ed in he disse a ion, his model is conside ed he simples one since he decision ule o apply o he AGV ollows a Fi s -in, Fi s -ou pa adigm. This means ha he AGV will decide o ake o wa d he anspo o a pa ha was p ocessed i s , ins ead o a pa ha has been p ocessed mo e ecen ly. Acco ding o he P ocess Flow me hod, in addi ion o he wo blocks discussed in subchap e s 3.3.7.1 (C ea ion o Pa s) and 3.3.7.2 (P ocessing o Pa s), he block ha men ions he anspo o pa s is also essen ial (3.4.1). 3.4.1 T anspo o Pa s Apa om he ini ial anspo om Sou ce o he IB o he i s WS o each p oduc ion o de , add essed in he "C ea ion o Pa s" block (3.3.7.1), his block includes all he o he anspo s. In his segmen , as soon as an en i y eaches he OB o any o he 7 WSs, he AGV will be eques ed and, as soon as a ailable, i will anspo he pa om he cu en WS o he nex one, espec ing he pa icula sequences. Finally, he AGV will be eleased, o igina ing he c ea ion o a new anspo p ocess. 3.5 Implemen a ion o an Op imized Milk-Run Sys em 33 Figu e 3.7: Rep esen a ion o he “T anspo o Pa s” Block – FIFO Model Table 3.5: Rou ings Table Line Pa Type Wo ks a ion P ocessing Time (in minu es) 1 1 WS2 2 2 1 WS1 9 3 1 WS7 7 4 1 WS6 8 5 1 Sink1 0 6 2 WS4 10 7 2 WS2 9 8 2 WS5 5 9 2 WS1 5 10 2 Sink1 0 11 3 WS2 6 12 3 WS4 10 13 3 WS7 7 14 3 WS3 7 15 3 Sink1 0 16 4 WS7 3 17 4 WS3 7 18 4 WS2 7 19 4 WS4 5 20 4 Sink1 0 21 5 WS4 10 22 5 WS5 6 23 5 WS1 9 24 5 WS2 2 25 5 Sink1 0 3.5 Implemen a ion o an Op imized Milk-Run Sys em The main di e ence be ween his Milk-Run sys em and he one p e iously p esen ed in he ea lie subsec ion is he decision ule o be applied o he AGV. While in he p e ious sys em he e was no p io i y ega ding he anspo o en i ies, ollowing he FIFO pa adigm, in his speci ic con ex 34 P oblem and Me hodology he AGV has a p ede ined WS sequence, (iden i ied in he ollowing Table 3.6 and illus a ed in Figu e 3.8), ha i ollows in o de o load he en i ies. In o he wo ds, i he AGV is, o example, in WS2, bu ha same s a ion does no con ain en i ies in i s OB o pe o m he load, he AGV will check i he nex WS in he AGV Rou ing Table (WS4) includes pa s in he espec i e OB, and so on. Wha dis inguishes his op imized Milk-Run algo i hm om he classic one is he ac ha he AGV pe o ms he load di ec ly on he nea es WS, in a clockwise di ec ion, which mus necessa ily ha e pa s in i s OB. On he o he hand, wha happens in he classic Milk-Run sys em is ha he AGV mus isi he nea es WS, also clockwise, e en i i does no con ain pa s in he espec i e OB. Figu e 3.8: AGV T ajec o y – Milk-Run Model Table 3.6: AGV Rou ing Wo ks a ion Sou ce WS2 WS4 WS5 WS1 WS7 WS3 WS6 Consequen ly, using one mo e ime he P ocess Flow me hod, h ee la ge blocks we e also es ablished: C ea ion o Pa s (3.3.7.1), P ocessing o Pa s (3.3.7.2) and T anspo o Pa s (3.5.1). 3.6 Implemen a ion o he Nea es WS Rule 35 3.5.1 T anspo o Pa s The ype o he “Release oken.WS” block is he “Schedule Sou ce” and his block allows he de ini ion o he cu en WS and associa es i wi h a oken called oken.WS. A oken is conside ed a class, isibly ep esen ed by a ci cle, wi h he abili y o be upda ed h oughou he execu ion o he P ocess Flow. The “Nex WS?” decision block iden i ies he nex WS whe e he AGV has o pe o m he load, mo e speci ically i s OB, as well as an indica ion o he co esponden IB whe e he en i y will be unloaded. A e iden i ying he subsequen WS, i is necessa y o ese e he AGV and hen iden i y he oldes “en i y” o he nex WS. A e ha , i is essen ial o load and unload he cu en pa and, inally, upda e he cu en WS oken. Finally, he esou ce is eleased. I should also be no ed ha when he AGV Rou ing able is comple ely c ossed and he cu en line co esponds again o he cu en WS, i means ha none o he WSs has pa s in he espec i e OBs o be loaded and, he e o e, he AGV is edi ec ed o a s a e whe e i will be wai ing un il a pa is p ocessed ("Wai o E en "). Figu e 3.9: Rep esen a ion o he “T anspo o Pa s” Block – Op imized Milk-Run Model 3.6 Implemen a ion o he Nea es WS Rule This sec ion in ends o demons a e a new heu is ic also de eloped in he disse a ion, called Nea - es WS Rule, whose main objec i e is o minimize he dis ances co e ed by he AGV. In e ms o he decision p inciple, wha dis inguishes his heu is ic om he Milk-Run algo i hm (Chap e 3.5) is he ac ha he AGV mo es o he nea es WS wi h pa s wai ing o be anspo ed, no ollowing any so o ou e. As Figu e 3.11 displays, i he cu en s a ion is he WS4 and he e a e only pa s al eady p ocessed wai ing o anspo in he ou pu bu e s o WS2 and WS3, since he closes s a ion o WS4 is WS2, he AGV will load he pa p esen in he WS2 OB. 36 P oblem and Me hodology Figu e 3.10: Illus a i e example o he Nea es WS Rule One o he di e en ia ing ea u es o his new anspo sys em is he ac ha he e is an ex e nal p og am ha con ols he simula ion en i onmen , ecei ing in o ma ion om i in he o m o obse a ions, and sending ac ions ha indica e he OB o he s a ion whe e he load will be ca ied ou . This app oach is comple ely di e en om hose p e iously s udied, like he FIFO and op imized Milk-Run sys ems, since Flexsim will no be esponsible o making he decisions, bu an ex e nal p og am, called Se e . This en i y is p og ammed using he Py hon language and i is ep esen ed in Figu e 3.11. Figu e 3.11: Se e -Flexsim ela ionships The communica ions be ween he Se e and Flexsim en i onmen , which will be a clien , ha e o be success ul. In his sense, he T ansmission Con ol P o ocol (TCP) was chosen o accomplish hese communica ions equi emen s. Fo his, i is c ucial o c ea e Clien -Se e socke s, ha will be discussed in he nex Sec ion 3.6.1. 3.6.1 Clien -Se e TCP Socke s Socke s a e an impo an ool used o send messages o e a ne wo k. This ne wo k can be logical, local o he compu e o e en physically connec ed o an ex e nal ne wo k. 3.6 Implemen a ion o he Nea es WS Rule 37 The TCP p o ocol, in addi ion o being e y eliable, allows ha he da a sen by he clien can be ecei ed and in e p e ed acco ding o he o de o sending by he Se e . Howe e , ega ding he UDP p o ocol (Use Da ag am P o ocol), i does no gua an ee ha he da a will be ecei ed by he Se e in he same sequence in which i was sen by he clien . [32] Wi hin he scope o his disse a ion, he communica ion be ween he Se e and Flexsim en i ies equi es he c ea ion o socke s o exchange messages, espec ing he TCP communica ion p o ocol. In his sec ion, Figu e 3.12 illus a es he global s eps ha cons i u e he c ea ion and de elopmen o a Clien -Se e communica ion. The Se e is conside ed a passi e Agen because i wai s o he connec ion eques om he clien . On he o he hand, he clien is cha ac e ized o aking he ini ia i e and making a communica ion eques , so i is an ac i e en i y in his whole p ocess. [33] B ie ly, in a i s ins ance, he Se e is esponsible o c ea ing he espec i e socke , and hen, an add ess and a po a e immedia ely associa ed wi h i . Subsequen ly, he Se e wai s o he momen he clien eques s he connec ion. A e he connec ion is success ul, i is possible o send and ecei e messages be ween bo h en i ies, and a he end o his whole p ocess o sending and ecei ing da a, he connec ions a e closed. Figu e 3.12: Clien -Se e TCP/IP communica ion diag am 3.6.2 Se e – Ex e nal P og am F om an obse a ion sen by he simula ion en i onmen ha indica es he cu en s a e o each WS in e ms o he p esence o pa s in hei espec i e OB, as well as he cu en s a ion whe e he AGV is placed, he Se e will hen send an ac ion wi h he espec i e WS whe e he AGV will ealize he load. To s o e he cu en da a o he simula ion model in he Se e , he a chi ec u e ep esen ed in he ollowing UML diag am in Figu e 3.13 was c ea ed, whe e h ee classes a e de ined: 38 P oblem and Me hodology •Sys em_Da a - con ains he dic iona ies e e ing o he dis ances be ween WSs and he own WSs and allows an associa ion wi h he AGV class, in o de o know he cu en s a ion whe e he AGV is loca ed; •Wo ks a ion - allows o know i a speci ic WS has pa s in i s OB; •AGV - con ains he AGV cu en loca ion; Figu e 3.13: UML class diag am – Nea es WS Rule The obse a ion sen by Flexsim is in he ollowing o m: 8 booleans + cu en WS, "10010000WS1", in which he 8 booleans allow o iden i y i each o he 7 WSs and Sou ce ha e pa s in hei espec- i e OB. Finally, he s ing allows o iden i y he cu en WS. Fo he speci ic example men ioned ("10010000WS1"), he conclusions d awn a e as ollows: Table 3.7: In e p e a ion o he obse a ion sen by Flexsim Wo ks a ion Does i ha e pa s on OB? Cu en WS WS1 Yes X WS2 No - WS3 No - WS4 Yes - WS5 No - WS6 No - WS7 No - Sou ce No - I is impo an o no e ha he cu en s a ion is upda ed whene e an unload is pe o med on one o he WSs, including Sink and excluding Sou ce. In his way, Sou ce will ne e assume he ole o he cu en WS, as i is only used o pe o m loads. Howe e , i is essen ial o know i he e a e pa s o anspo in i . Re u ning o he diag am o he classes ha cons i u e he Se e , a e ex ac ing he in o ma- ion om he obse a ion, i is necessa y o ill all he a ibu es o he h ee classes, namely he "boolean_has_pa _OB" a ibu e o he "Wo ks a ion" objec s, as well as he cu en WS alue, e e ing o he "AGV" objec . Rega ding he dis ance alues, hey ha e p e iously been ead om an Excel ile and inse ed in he dis ance dic iona y o he "Sys em_Da a" class. 3.6 Implemen a ion o he Nea es WS Rule 39 A e he ea men and o ganiza ion o all his in o ma ion, i is necessa y o p oceed wi h he implemen a ion o he decision ule al eady men ioned (Figu e 3.11), which allows he AGV o ca y ou he load in he nea es WS ha con ains pa s in i s OB. Finally, his in o ma ion ha e lec s he ou pu bu e o he chosen s a ion is sen back o he simula ion en i onmen , in he o m o "OBWS" ( o example "OBWS2"). 3.6.3 Flexsim – Simula ion En i onmen 3.6.3.1 T anspo o Pa s A e sending he obse a ion o he Se e , Flexsim wai s o he esponse wi h he ac ion con- aining he OBWS om he Se e . Howe e , i he Se e de ec s ha he e a e no pa s in any o he OBs, i sends a message ha o de s he simula ion o go o a wai s a e ("Wai o pa s on OB"), as shown in Figu e 3.14. I he e a e pa s in a leas one o he s a ions, he Se e sends he ac ion wi h he OBWS and he Flexsim accesses he oldes objec p esen in ha OB. Then, he nex WS whe e he pa will be unloaded is ound. Finally, in case ha he numbe o pa s ha en e a Sink eaches he alue 2080, which co esponds o he o al numbe o eleased o de s, he simula ion ends and i s s a e is sen o he "Sink" block. Figu e 3.14: Rep esen a ion o he “T anspo o Pa s” Block – Nea es WS Rule 46 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms Figu e 4.6: Neu on gene al s uc u e, adap ed om [7] The main idea o he PPO algo i hm is o make su e ha he new policy is no oo a om he p e ious one. The e sion used cons i u es an implemen a ion o S able Baselines and in addi ion o using ec o ized en i onmen s, i also allows ha he obse a ion and he esul ing ac ion can be o a disc e e, box, mu idisc e e o mul ibina y ype. In his speci ic scope, he obse a ion assumes he box ype (16,) and he ac ion admi s he disc e e ype (0-7). The nex Figu e 4.7 ep esen s he ne wo k a chi ec u e o his disse a ion. Figu e 4.7: SLP ne wo k a chi ec u e Wi hin he obse a ion, he i s 8 booleans [0-7] indica e he WSs ha ha e o ha e no pa s in hei espec i e OB (Table 4.2). The las 8 booleans [8-15] s a e he cu en AGV loca ion, co esponding o a "one-ho encoding" (Table 4.3). Acco dingly o he Nea es WS Rule (Sec ion 3.6), he s a ions s a e is upda ed jus a e an unload was pe o med. Bu , whene e an ac ion co esponds o a WS ha does no con ain pa s, he AGV goes he e in he same way and he cu en WS is also upda ed. As an example, i he obse a ion is: [1001000010000000], i means ha only WS1 and WS4 ha e pa s in hei OB and he cu en WS is WS1. 4.4 App oach I – Ini ializa ion o Weigh s 47 Table 4.2: Obse a ion segmen a ion [0-7] Obse a ion [0-7] Exis ence o pa s 10000000 WS1 01000000 WS2 00100000 WS3 00010000 WS4 00001000 WS5 00000100 WS6 00000010 WS7 00000001 Sou ce 00000000 The e a e no pa s Table 4.3: Obse a ion segmen a ion [8-15] Obse a ion [8-15] Ac ual WS 10000000 WS1 01000000 WS2 00100000 WS3 00010000 WS4 00001000 WS5 00000100 WS6 00000010 WS7 00000001 Sink 00000000 Sou ce 4.4 App oach I – Ini ializa ion o Weigh s Fi s ly, i was decided o de ine a neu al ne wo k a chi ec u e and a se o weigh s ha would allow he eplica ion o he Nea es WS Rule. This phase aimed o ensu e ha an SLP ne wo k, wi h he igh se o weigh s, would be su icien o ob ain a pe o mance equi alen o he esul s o he Nea es WS Rule. Fo demons a i e pu poses, in Figu e 4.8 i is possible o obse e he beha iou o his p ac- ice, aking in o accoun he obse a ion con en and he esul ing ac ion. Since i is jus an ex- ample, he e we e only conside ed 3 WSs, whose dis ances be ween hem a e also illus a ed in Figu e 4.8. In o de o simpli y he isualiza ion, only he weigh s o he ou pu neu ons 1 and 2, co e- sponding o OBWS1 and OBWS2, a e iden i ied in he espec i e igu es. As he Figu e 4.8 and Equa ion 4.3 show, he ou pu neu on ha will be ac i a ed is he i s one, since i ob ains he highes ou pu alue. As i makes sense, i he e a e pa s in he WS1 and WS3 ou pu bu e s, and he cu en s a ion is he WS1, hen he AGV will load he pa on he OB o he WS1. Figu e 4.8: Neu al Ne wo k – Example 1 48 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms Dmax =3 (4.1) Bias =−Dmax−2=−3−2=−5 (4.2) OB_WS1= (1∗5+0∗0+1∗0)+(1∗3+0∗2+0∗1)+(1∗−5) = 3 (4.3) OB_WS2= (1∗0+0∗0+1∗0)+(1∗2+0∗3+0∗0)+(1∗−5) = −3 (4.4) OB_WS3= (1∗0+0∗0+1∗5)+(1∗1+0∗0+0∗3)+(1∗−5) = 1 (4.5) Like he Figu e 4.9 deno es, he only case in which his ne wo k would no ha e he same beha io as he ule would be i he cu en WS was he Sou ce, whe e i would be a d aw be ween all s a ions wi h a pa on hei OBs (in his case: WS1 and Sou ce). Figu e 4.9: Neu al Ne wo k – Example 2 Dmax =3 (4.6) Bias =−Dmax−2=−3−2=−5 (4.7) OB_WS1= (1∗5+0∗0+1∗0)+(0∗3+0∗2+0∗1)+(1∗−5) = 0 (4.8) OB_WS2= (1∗0+0∗0+1∗0)+(0∗2+0∗3+0∗0)+(1∗−5) = −5 (4.9) 4.5 App oach II – Neu al Ne wo k P e-T aining 49 OB_Sou ce = (1∗0+0∗0+1∗5)+(0∗1+0∗0+0∗3)+(1∗−5) = 0 (4.10) The S ableBaselines amewo k, which will be used o Rein o cemen Lea ning algo i hms, does no allow he de ini ion o ini ial ne wo k weigh s. Thus, i was no possible o c ea e a policy wi h he calcula ed weigh alues. S ill, his was an impo an exe cise o ensu e ha his ne wo k a chi ec u e is capable o achie ing pe o mance a leas equi alen o he p e iously de ined ule. An al e na i e o ini ializing he weigh s is using a p e- aining phase o he ne wo k, which is co e ed in sec ion 4.5. Thus, appea s he necessi y o s udying he in luence o he p e- aining algo i hms o obse e i s in e e ence in he cu en p oblem. 4.5 App oach II – Neu al Ne wo k P e-T aining This app oach allows, h ough he implemen a ion o an algo i hm wi h esul s al eady isible, in his pa icula case he Nea es WS Rule, o ha e as s a ing poin ini ial solu ions ha come om ha ule. In his phase, he beha iou ha he Agen mus ha e o pe o m a s ep in o de o eplica e he Nea es WS Rule is de ined. Then, se e al simula ions a e un using his Agen o ob ain a da a se wi h obse a ion-ac ion pai s. Finally, hese da a a e used o ain he SLP neu al ne wo k acco ding o Supe ised Lea ning algo i hms. This p ocess ensu es ha he neu al ne wo k s a s i s lea ning p ocess wi h a se o weigh s ha eplica e he unc ioning o he ule. Fo his, in o de o e i y he in luence o his p e- aining in he inal solu ion, wo mod- els wi h di e en objec i es will be app oached, namely he minimiza ion o makespan and he maximiza ion o p oduc i i y, aking in o accoun he PPO algo i hm ("Makespan" and "Base" Models). 4.6 Makespan Model In o de o e i y he co ec beha iou o he Rein o cemen Lea ning algo i hm, PPO, he ini ial objec i e was o minimize he makespan. Thus, only one elease o de was de ined o a andom en i y, mo e speci ically a pa ype o numbe 4. The main pu pose o his model is o e i y he co ec beha iou o his anspo sys em, obse ing he endency o he inal solu ions o ge he op imal solu ion. Associa ed wi h he Agen ’s lea ning p ocess, he o al numbe o ime s eps de ined was 250.000 (s eps), so ha i is belie ed o be mo e han enough o achie e he op imal solu ion. As a compa able poin o he solu ions achie ed in he op imiza ion p ocess, he op imal solu- ion can be ob ained h ough he Nea es WS and op imized Milk-Run ules, because he e is jus one pa in he en i e simula ion p ocess. 50 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms In his case, he p e- aining o he ne wo k does no apply o his model since he subjacen ule would lead o an ideal beha iou al eady du ing he ini ial phase o he lea ning p ocess, which is no in ended. Tha said, since he objec i e is o analyse he beha iou o he op imiza ion cu e o he RL algo i hm, he idea is ha he inal makespan alues o each simula ion pe o med app oach he op imal solu ion. In his speci ic case, he ideal solu ion o a pa ype 4 is: 1486 seconds. Figu e 4.10 shows he P ocess Flow o he anspo p ocess implemen ed in he Flexsim simula ion en i onmen . The “Decide” block de ines he nex WS whe e he AGV has o go, aking in o accoun he in o ma ion sen by he Agen o e he a ious s eps i sends. I he e a e no pa s al eady p ocessed wai ing o be anspo ed o he nex WS, he AGV goes o a s a e whe e i is wai ing o he pa in ques ion o be p ocessed (gi en ha in his pa icula model he e will only be one pa h oughou he en i e simula ion). In he case ha he Agen sends a s ep e e ing o a WS ha does no con ain pa s in his OB, since he ne wo k is in he lea ning p ocess, he AGV will also go he e and he cu en WS will be upda ed. As soon as he pa en e s a Sink, he Flexsim will send he alue co esponding o he makespan as a ewa d o he Se e , bu wi h he nega i e signal (-makespan). This pa icula - i y is jus i ied by he ac ha he pu pose o he PPO algo i hm is always o maximize some hing. In his case, he goal is o maximize (-) makespan, in o he wo ds, minimize he makespan. S ill in his scena io, he hi d pa ame e , called "done", is also sen as a uni , since he pa has al eady been en e ed in o he Sink and he simula ion is inished. Figu e 4.10: Rep esen a ion o he “T anspo o Pa s” Block – Makespan Model A e unning se e al simula ions, i was ound ha he model, unde he cu en condi ions in which he makespan is only sen a he end o he simula ion when he pa en e s a Sink, does no allow eaching he op imal solu ion. I was concluded, he e o e, ha he ac ha he ewa ds always assume null alues, wi h he excep ion o he esponse o he las s ep ha includes he makespan, is no enough o he ne wo k o be able o make an adequa e lea ning and ind he ideal solu ion wi hin he de ined 250k ime s eps. 4.6 Makespan Model 51 The e o e, some c i e ia we e in oduced in o de o check whe he i is possible o achie e he op imal solu ion, in pa icula : • he sending o ewa ds be ween ime s eps wi h he alue o a ime in e al ini ia ed by a load and ended wi h an unload (including he p ocessing ime), in de imen o sending he o al makespan only a he end o he simula ion; • he in oduc ion o penal ies, i he mo emen s made do no in ol e he anspo a ion o pa s; • he inse ion o no maliza ions, modi ying he alue o he ewa ds, con e ing hem in he in e al [-1,0]. La e , in o de o s udy he in luence o hese changes on lea ning pe o mance, all o hese scena ios p esen ed in Table 4.4 we e implemen ed and simula ed. Table 4.4: Scena ios o he Makespan Model Scena io Time s eps No maliza ion No maliza ion Fac o Type o Rewa d Penal y 1 250k No 0 makespan 0 2 250k No 0 makespan -FN/2 3 250k Yes 15k makespan 0 4 250k Yes 15k makespan -0.5 5 250k No 0 be ween ime s eps 0 6 250k No 0 be ween ime s eps -FN/2 7 250k Yes FN be ween ime s eps 0 8 250k Yes FN be ween ime s eps -0.5 FN = 4*T a el ime be ween he wo mos dis an s a ions + 2*Highe P ocessing Time (Pa ype no4). Fo all he cases p esen ed abo e in Table 4.4, he model con e s he solu ions ob ained o he op imal solu ion, excep o he scena io in which he ewa ds a e sen be ween ime s eps wi hou no maliza ion o penal y. The ques ion ha immedia ely a ises is ela ed o he ime ha is necessa y o each he op imal solu ion and, in his con ex , i was ound ha no maliza ion is he ac o ha allows he model o end mo e quickly owa ds he op imal solu ion. These esul s a e p esen ed and discussed wi h mo e de ail in Chap e 5. In he ollowing subsec ions a e p esen ed he s eps used in he no maliza ion and penal y me hods o he ypes o ewa ds: be ween ime s eps and makespan. 4.6.1 No maliza ion The main objec i e o no maliza ion is o e i y whe he , by adjus ing he alues o he ewa ds o alues be ween [-1, 0], he algo i hm will be able o con e ge be e o a good solu ion and inc ease i s lea ning capaci y. 52 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms Fo his, he no maliza ion ac o depends on he ype o ewa d (be ween ime s eps o makespan) and applica ion o penal ies, cha ac e ized by he ollowing Equa ions 4.11 and 4.12: 4.6.1.1 Type o Rewa d: Makespan No maliza ion ac o =2∗7.5k=15k(seconds)(4.11) Rewa d ∈[−0.5,0],wi hou penal y (4.12) This alue a ibu ed o he no maliza ion ac o is jus i ied by he ac ha i was obse ed du ing all he simula ions p e iously ca ied ou ha he inal alue o he makespan, o a single pa o ype 4, ne e exceeded he alue o 7.5k seconds, wi h a e excep ions. The e o e, aking in o accoun he subsequen applica ion o penal ies (-0.5), he alue conside ed o he no mal- iza ion ac o was 15k seconds, as i is wice 7.5k. The e o e, wi hou applying he penal y, he ewa d assumes alues be ween -0.5 and 0, since he pu pose o op imiza ion is o maximize (-) makespan. 4.6.1.2 Type o Rewa d: Be ween Time s eps No maliza ion ac o =4∗T1+2∗T2 (4.13) in which T1=T a el ime be ween he wo mos dis an s a ions; T2=Highe P ocessing Time (Pa o ype 4). Rewa d ∈[−0.5,0],wi hou penal y (4.14) As in he p e ious case ega ding makespan, he no maliza ion ac o applied allows ewa ds o assume alues be ween -0.5 and 0, wi hou he applica ion o penal ies. In he speci ic case o a pa ype 4: •T1 = 48.827 seconds; •T2 = 420 seconds. No maliza ion ac o =1035.308 (4.15) 4.6.2 Penal ies Penal ies a e applied when he AGV pe o ms mo emen s wi hou a pa . The objec i e is, he e- o e, h ough he applica ion o his penal y wi h he alue o -0.5, o make i clea o he Agen ha , in his speci ic case, i is no a good p ac ice o make mo emen s wi hou a pa , gi en ha i 4.7 Base Model 53 is only one pa in he simula ion en i onmen . Tha said, aking in o accoun he ype o ewa d, hei alues will be included in he ollowing ange: Rewa d ∈[−1,0](4.16) 4.7 Base Model A e analysing he Agen ’s beha iou owa ds he model whose objec i e was o minimize he makespan, i was also decided o analyse he issue o maximizing p oduc i i y and c ea e a new model ha allows s udying he PPO algo i hm. Wi h ega d o he P ocess Flow o he simula ion model, all he blocks a e coinciden wi h he blocks o he p e ious model, e e ing o he makespan. The only ele an di e ence is he issue o he ewa ds ha a e sen om Flexsim o he Agen , in e media ed by he Se e . In esponse o a s ep om he Agen , he simula ion en i onmen cons uc s he obse a ion ha con ains he cu en s a us o each WS, in e ms o he p esence o pa s, and he cu en posi ion o he AGV. Howe e , he ewa ds assume a uni alue whene e a pa is en e ed a Sink. O he wise, hey a e assigned a null alue. In ela ion o he pa ame e “done”, i will be uni a y whene e he de ined ime ho izon is eached, ending he simula ion. Mo eo e , whene e a ewa d assumes a alue o 1, he e is an in e nal coun e in he sim- ula ion en i onmen ha is inc eased, p o iding an upda ed indica ion o he numbe o pa s manu ac u ed so a , o a la e compa ison o esul s wi h he p e ious p oduc i i y models. In his con ex , unlike wha happens wi h he makespan model, he use o p e- aining becomes impo an in o de o e i y whe he i has had an e ec o no . The ule implemen ed and asso- cia ed wi h he p e- aining is based on one o he ules p e iously desc ibed in his disse a ion, be e known as he Nea es WS Rule, which allows AGV o pe o m a load on he nea es WS ha con ains pa s in he espec i e OB. A no less impo an issue is o unde s and he e ec s o he in oduc ion o penal ies on p o- duc i i y. Fo his, gi en ha he ewa ds a e al eady no malized in he in e al [0, 1], mo e speci ically assuming he alues 0 o 1, he de ined penal y was (-) 0.5, simila o he p e ious Makespan Model. Tha said, whene e he AGV makes an emp y mo emen , o de ed by he Agen , he ewa d e u ned will be -0.5. Rega ding he numbe o s eps sen by he Agen , a e se e al uns, i was concluded ha he alue o 10M s eps would be he mos indica ed alue and mo e han enough o each he inal solu ion. Fo u he compa ison o esul s, he ime ho izons chosen we e 36 and 52 hou s. Finally, i should be no ed ha , whene e a "done" alue is assigned o 1, he p oduc i i y alue ela ed o he simula ion in ques ion is also w i en in a ex ile, o la e analysis. 54 Implemen a ion o a Dynamic T anspo Sys em, using Rein o cemen Lea ning Algo i hms 4.8 Robus ness o he Simula ion Models This sec ion was c ea ed wi h he in en ion o e i ying whe he he models co e ed in he p e ious sec ions, namely he FIFO, op imized Milk-Run, Nea es WS Rule and Base Model, e e ing o p oduc i i y, ha e an app op ia e beha iou agains possible changes in he p oduc ion mix o in he p ocessing imes. 4.8.1 P oduc ion Mixes Wi h ega d o changes in p oduc ion mixes, he o de in which he di e en pa ypes a e launched will su e some a ia ions and i will be andomly gene a ed, using he Mic oso Excel so wa e ool. Jus like wha happens in he o iginal mix, in which he p obabili y o each pa ype being eleased is nea ly 20% (Table 4.5), he same es ic ion applies o he nex 5 di e en mixes gen- e a ed. B ie ly, he di e en p oduc ion mixes a e gene a ed so ha he pa ype associa ed wi h each p oduc ion o de is de e mined andomly, bu wi h he same inal p obabili y o being c ea ed (20%). Table 4.5: Quan i y o he di e en pa ypes – O iginal Mix Pa Type Quan i y (uni s) 1 448 2 410 3 429 4 424 5 369 The e o e, as shown in Chap e 5, he e a e small di e ences in he inal quan i ies o pa s p oduced o each one o he mixes, due o he way hey we e c ea ed. 4.8.2 P ocessing Times The p ocessing imes co esponding o each pa ype and WS we e in oduced in a s ochas ic en i onmen , in o de o ollow, mo e conc e ely, a iangula p obabili y dis ibu ion. Assuming ha Ti j ep esen s he p ocessing ime o he pa ype iin he WS j, i was adap ed o espec he iangula dis ibu ion, shown in Figu e 4.11. In his segmen , a a iabili y ac o α was also added o simula e possible a ia ions ha may occu in a eal con ex . 4.8 Robus ness o he Simula ion Models 55 Figu e 4.11: T iangula dis ibu ion unc ion, e e ing o he p ocessing imes In a igo ous way, he espec i e iangula dis ibu ion is de ined by he ollowing exp essions: (x|a,b,c) =                        2(x−a) (b−a)(c−a),se a≤x<c 2 (b−a),se x=c 2(b−x) (b−a)(b−c),se c<x≤b 0,o he wise (4.17) Whe e a, b and c ake he nex alues:        a=Ti j(1-α) b=Ti j(1+α) c=Ti j        (4.18) The esul s o his p ocedu e, which is based on he in oduc ion o he p ocessing imes in a s ochas ic en i onmen , a e desc ibed in Chap e 5. 62 Resul s Analysis Figu e 5.5: Makespan ela ed o Scena io 5, o a o al o 250k ime s eps Table 5.8: Nume ical in e p e a ion o makespan e e ing o Scena io 5 (in seconds) Minimum Value Maximum Value Final Value 1630 17202 6645 5.4.6 Scena io 6 – Type o ewa d: be ween ime s eps, wi h penal ies and wi hou no maliza ion Analysing he scena ios 2 and 6, bo h using penal ies, i appea s ha he wo ha e simila be- ha iou s, wi h a g ea e numbe o simula ions ca ied ou in he second scena io. These ques ions can be obse ed in Tables 5.5 (scena io 2) and 5.9 (scena io 6). Figu e 5.6: Makespan ela ed o Scena io 6, o a o al o 250k ime s eps Table 5.9: Nume ical in e p e a ion o makespan e e ing o Scena io 6 (in seconds) Minimum Value Maximum Value Final Value 1486 3945 1486 5.4 Makespan Model 63 5.4.7 Scena io 7 – Type o ewa d: be ween ime s eps, wi hou penal ies and wi h no maliza ion As in he p e ious scena io, he op imal alue is also achie ed (Figu e 5.7). Ne e heless, as e idenced in scena ios 2 and 3, he use o he no maliza ion in a singula way allows ob aining he bes solu ion ea lie when compa ed o he isola ed applica ion o penal ies. Figu e 5.7: Makespan ela ed o Scena io 7, o a o al o 250k ime s eps Table 5.10: Nume ical in e p e a ion o makespan e e ing o Scena io 7 (in seconds) Minimum Value Maximum Value Final Value 1486 4096 1486 5.4.8 Scena io 8 – Type o ewa d: be ween ime s eps, wi h penal ies and wi h no maliza ion As well as he ou h scena io, and as expec ed, when he no maliza ions and penal ies a e applied simul aneously, he op imal solu ion is ob ained in ad ance in ela ion o he models in which he penal ies o no maliza ion a e applied sepa a ely. Figu e 5.8: Makespan ela ed o Scena io 8, o a o al o 250k ime s eps 64 Resul s Analysis Table 5.11: Nume ical in e p e a ion o makespan e e ing o Scena io 8 (in seconds) Minimum Value Maximum Value Final Value 1486 4116 1486 5.4.9 Syn hesis In sho , ega ding he models p esen ed in Sec ion 5.4 ela ed o makespan, i is possible o s a e ha he Rein o cemen Lea ning PPO algo i hm was co ec ly implemen ed, gi en ha he models ollow hei cha ac e is ic cu e, allowing o maximize (-) makespan. The esul s also sugges ha he no maliza ion o ewa ds and he applica ion o penal ies imp o e he lea ning capaci y o he model. 5.5 Base Model Re u ning o he ques ion o he p oduc i i y and aking in o accoun he PPO algo i hm, he esul s achie ed o he ime ho izons o 36 and 52 hou s we e di ided acco ding o he exis ence o no o a p e- aining o he neu al ne wo k and a e ep esen ed and indica ed, espec i ely, in Figu es 5.9,5.10,5.11,5.12 and Tables 5.12,5.13,5.14,5.15. Wi h espec o he Agen ’s lea ning pe iod, as e e ed o in Sec ion 4.7, he numbe o ime s eps was se o 10M. Rega ding he p e- aining, i espec s he Nea es WS Rule, discussed in Sec ion 3.6. Fo a be e unde s anding o he in luence o he p e- aining in he lea ning phase, some se s o obse a ion-ac ion pai s esul ing om his same s age will also be p esen ed. Fo his pu pose, a p obabili y unc ion will be applied o he ac ions, a e he lea ning phase is pe o med. The objec i e o his s ep is o e i y in which WS he AGV will ca y ou he espec i e load. Subsequen ly, simila ly o he p e ious Sec ion (5.4) e e ing o he makespan, he esul s de i ed om he applica ion o a penal y ac o (-0.5) a e also p esen ed. 5.5.1 Wi hou p e- aining, o a ime ho izon o 36 hou s Figu e 5.9 allows us o in e ha he model’s beha iou , wi h ega d o p oduc i i y alues, ollows he PPO op imiza ion algo i hm. As can be seen, he model eaches he maximum p oduc i i y alues in he i s simula ions. This phenomenon is explained by he weigh s which a e a bi a ily assigned in he ini ial lea ning phase o he ne wo k. Subsequen ly, a e a ce ain numbe o simula ions, he model p esen s lowe p oduc i i y alues, also due o he weigh s de ined by he Agen and because a he begin- ning o he in o ma ion collec ion, he ne wo k has li le knowledge ye . Howe e , he p oduc i i y cha ac e is ic cu e allows e i ying ha he p oduc i i y alues end o a inal maximum solu ion, which app o es ha he lea ning phase was pe o med co ec ly. The inal alue o p oduc i i y, which comes om he ne wo k’s lea ning p ocess, is 343 pa s ha en e a Sink. 5.5 Base Model 65 Figu e 5.9: P oduc i i y e e ing o he Base Model wi hou p e- aining, o a o al o 10M ime s eps and a ime ho izon o 36h Table 5.12: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model wi hou p e- aining, o a o al o 10M ime s eps and a ime ho izon o 36h Minimum Value Maximum Value Final Value 250 347 343 The se o obse a ion-ac ion pai s esul ing om he lea ning phase, wi hou he p e- aining, a e exempli ied below: •Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4 Obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [8.3619851e-01, 1.9839258e-04, 7.7551082e-02, 2.0058152e-04, 1.7890350e-04, 6.3502125e-02, 1.9391190e-02, 2.7792549e-03] In his pa icula si ua ion, he beha iou o he ne wo k is as expec ed, gi en ha he highes p obabili y is associa ed wi h he WS1. •Obse a ion-Ac ion pai no2: Pa s in WS3, WS4 | Cu en WS: 4 Obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [1.4122830e-03, 1.3475084e-05, 9.6138531e-01, 2.4895007e-03, 7.2637980e-05, 2.9261060e-02, 3.8612273e-03, 1.5044548e-03] Wi h ega d o his obse a ion-ac ion pai , he ne wo k’s beha iou does no co espond o wha would be done by he Nea es WS Rule, gi en ha , in his si ua ion, he AGV would load a pa in he WS3 and no in he WS4, as he high p obabili y sugges s. 5.5.2 Wi h p e- aining, o a ime ho izon o 36 hou s Like he model wi hou p e- aining, his one also espec s he cu e e e ing o he PPO op i- miza ion algo i hm, ep oduced in Figu e 5.10. 66 Resul s Analysis Howe e , in compa ison wi h he model wi hou p e- aining, i appea s ha he esul s o his model be o e he lea ning s age a e lowe , in e ms o p oduc i i y. Despi e ha , simila ly o wha happened in he op imized Milk-Run and Nea es WS Rules, his episode can be explained by he ac ha , some imes, he emp y a els o he anspo o pa s o dis an WSs, mo e common in he model wi hou p e- aining, can b ing ad an ages in e ms o he inal esul o p oduc i i y, educing possible bo lenecks and balancing he machine occupancy a es. Figu e 5.10: P oduc i i y e e ing o he Base Model wi h p e- aining, o a o al o 10M ime s eps and a ime ho izon o 36h Table 5.13: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model wi h p e- aining, o a o al o 10M ime s eps and a ime ho izon o 36h Minimum Value Maximum Value Final Value 257 277 274 One mo e ime, he examples o he obse a ion-ac ion pai s, esul ing om he lea ning p o- cess, e idence he p e ious accomplishmen o he lea ning phase: •Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4 Obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [9.9999833e-01, 6.0297623e-07, 9.3485424e-22, 1.0542382e-06, 3.6167533e-10, 4.3565872e-16, 4.4415426e-18, 2.8845209e-15] The beha iou o he ne wo k is he one expec ed, since he WS1 has he highes p obabili y. •Obse a ion-Ac ion pai no2: Pa s in WS3, WS4 | Cu en WS: 4 Obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [9.00684897e-25, 1.09724185e-16, 1.59646764e-13, 1.00000000e+00, 1.18856031e- 19, 6.28844218e-25, 2.55962060e-27, 2.96594020e-27] In con as wi h wha happened in he p e ious scena io in which he p e- aining did no occu , he highe p obabili y is now consis en wi h he expec ed (WS4), acco ding o he Nea es WS Rule. 5.5 Base Model 67 5.5.3 Wi hou p e- aining, o a ime ho izon o 52 hou s Acco ding o he g aph in Figu e 5.11, he beha iou o he lea ning phase, like he Model wi hou p e- aining o he 36-hou ime ho izon, ollows he PPO algo i hm, despi e he ini ial d op o he p oduc i i y alues. Rega ding he inal alue o p oduc i i y, his is logically highe han he p oduc i i y e e ing o he pe iod o 36 hou s. Figu e 5.11: P oduc i i y e e ing o he Base Model wi hou p e- aining, o a o al o 10M ime s eps and a ime ho izon o 52h Table 5.14: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model wi hou p e- aining, o a o al o 10M ime s eps and a ime ho izon o 52h Minimum Value Maximum Value Final Value 398 507 501 Fo he same obse a ion-ac ion pai s, he solu ions a e he ollowing ones: •Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4 obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [8.6369443e-01, 2.2193830e-04, 4.7343463e-02, 6.8649366e-05, 1.9357287e-04, 7.4317701e-02, 1.2128289e-02, 2.0318190e-03] Fo his case, he beha iou o he ne wo k is he one expec ed (WS1). •Obse a ion-Ac ion pai no2: Pa s in na WS3, WS4 | Cu en WS: 4 obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [2.6672529e-03, 7.9401660e-05, 9.7478318e-01, 2.3247360e-03, 1.2263234e-04, 1.7269300e-02, 2.2482565e-03, 5.0515484e-04] In acco dance wi h he esul s o he las model (36h), wi hou p e- aining, i is concluded ha he lea ning o he ne wo k leads he AGV o do he load in he u he s a ion (WS3) ins ead o he nea es one (WS4). 68 Resul s Analysis 5.5.4 Wi h p e- aining, o a ime ho izon o 52 hou s In his las scena io, Figu e 5.12 p o es he expec ed beha iou o he model. As Table 5.15 indica es, he inal alue o p oduc i i y is lowe han he alue ob ained in he scena io in which he p e- aining did no occu . The mos admissible easons a e p esen ed in Sec ion 5.6 ha compa es he esul s o all models. Figu e 5.12: P oduc i i y e e ing o he Base Model wi h p e- aining, o a o al o 10M ime s eps and a ime ho izon o 52h Table 5.15: Nume ical in e p e a ions o he p oduc i i y (in pa s) e e ing o he Base Model wi h p e- aining, o a o al o 10M ime s eps and a ime ho izon o 52h Minimum Value Maximum Value Final Value 375 409 404 The ollowing obse a ion-ac ion pai s ollow he conjunc u es pe o med p e iously, simi- la ly wi h he esul s ob ained in he model wi h p e- aining, e e ing o he ime ho izon o 36 hou s: •Obse a ion-Ac ion pai no1: Pa s in WS1 | Cu en WS: 4 Obse a ion: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [9.9983370e-01, 1.7993931e-06, 4.0928912e-19, 1.6300578e-04, 1.4105979e-06, 4.8301262e-15, 1.4212710e-17, 2.4972540e-14] •Obse a ion-Ac ion pai no2: Pa s in WS3, WS4 | Cu en WS: 4 Obse a ion: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Ac ion: [8.7494940e-28, 5.5310151e-18, 3.6004825e-13, 1.0000000e+00, 3.2771040e-19, 4.0131407e-26, 4.2842155e-29, 7.9547190e-29] 5.6 Compa ison o he FIFO, Op imized Milk-Run, Nea es WS Rule and Base Models 69 5.5.5 Applica ion o Penal ies A e he penal ies o -0.5 we e applied whene e an emp y mo emen is ca ied ou , i was con- cluded ha he ou models ollow he expec ed beha iou in ela ion o he PPO algo i hm, as i s cu e p esen s. Howe e , as Table 5.16 exp esses, he esul s a e in e io . Since pe o ming emp y mo emen s is no necessa ily a bad p ac ice, his is one o he easons o which he p oduc i i y alues a e lowe using penal ies. Table 5.16: P oduc i i y (in pa s) e e ing o he Base Model, wi h penal ies Base Model P oduc i i y (36h) P oduc i i y (52h) Wi hou p e- aining 343 501 Wi h p e- aining 274 404 Wi hou p e- aining, wi h penal ies 304 454 Wi h p e- aining, wi h penal ies 271 403 5.6 Compa ison o he FIFO, Op imized Milk-Run, Nea es WS Rule and Base Models Ini ially, compa ing all he models p e iously s udied in Sec ions 5.1,5.2,5.3 and 5.5, wi h ega d o p oduc i i y, he Base Model was he one ha p esen ed he bes esul s. Tha said, he use o Machine Lea ning echniques combined wi h simula ion app oaches has enabled us o achie e good solu ions. Rega ding he Base Model, he di e ence be ween he usage o no o p e- aining has o do wi h he ac ha , a he beginning o he lea ning p ocess, he esul ing solu ions a e al eady o no close o a local op imum. I p e- aining occu s, he solu ions de i ed om he lea ning p ocess will be in close o a local op imum, unlike wha happens in he si ua ion whe e he e is no p e- aining. Since he solu ions s a nea a local op imum, he Agen has di icul ies o explo e and ind new be e solu ions han he ones se led ou side his egion o he solu ion space. In his sense, i was ound ha he inal p oduc i i y esul s ob ained we e highe in he case whe e he p e- aining was no ca ied ou (Table 5.17). This is explained by he ac ha , in he absence o p e- aining, he occu ence o anspo s o dis an s a ions o he de imen o close s a ions allows he educ ion o bo lenecks and he balancing o he machine occupancy a es. The ac ha he e is no p e- aining also allows he Agen no o be so easily s agna ed in a local op imum gi en by he p e- aining, so he e is mo e eedom o explo e new solu ions, which will allow he Agen o achie e be e solu ions, con a ily o wha happens in he model whe e he e is p e- aining. 70 Resul s Analysis Table 5.17: P oduc i i y (in pa s) e e ing o he models in s udy Model P oduc i i y (36h) P oduc i i y (52h) FIFO 177 262 Op imized Milk-Run 266 389 Nea es WS Rule 258 377 Base, wi h p e- aining 274 404 Base, wi hou p e- aining 343 501 5.7 Robus ness o he Simula ion Models A e analysing he models when applied o de e minis ic en i onmen s, i is pe inen o e i y hei beha iou when applied o o he si ua ions. In pa icula , i is impo an o s udy he in luence o di e en p oduc ion mixes and p ocessing imes, in o de o examine he adap abili y o he models, bu only ega ding p oduc i i y, which he e o e excludes he model ela ed o he s udy o makespan. I is also impo an o men ion ha , acco ding o he Base Model, he e was no cus omized e-lea ning o hese scena ios, because he weigh s ob ained in he base scena io (o iginal mix) we e used. 5.7.1 P oduc ion Mixes Fi e di e en mixes we e de ined, andomly gene a ed, so ha , as desc ibed in Sec ion 4.8.1, he numbe o o de s o each pa ype is balanced (≃20%). The ime ho izons unde s udy a e 36 and 52 hou s o compa ison p inciples, and he AGV speed also emains cons an a 1.5 m/s, since speed does no cons i u e a ac o unde conside a ion. Tables 5.18 and 5.19 enable us o conclude ha , o bo h empo al pe spec i es, he Agen when applied o di e en p oduc ion mixes ensu es ha he p oduc i i y esul s emain, on a - e age, close o each o he . So, i is possible o alida e he Base Model and i s adap abili y in di e en p oduc ion con ex s, jus like he o he models. I should also be no ed ha he Base Model, wi hou p e- aining, emains he one ha ob ains he bes esul s. Table 5.18: P oduc i i y (in pa s) analysis e e ing o di e en p oduc ion mixes, o a ime ho izon o 36 hou s Model Ini ial Mix Mix 1 Mix 2 Mix 3 Mix 4 Mix 5 FIFO 177 196 179 171 178 174 Op imized Milk-Run 266 261 251 257 261 242 Nea es WS Rule 258 252 239 250 252 235 Base, wi h p e- aining 274 273 259 266 278 252 Base, wi hou p e- aining 343 341 328 314 325 329 5.7 Robus ness o he Simula ion Models 71 Table 5.19: P oduc i i y (in pa s) analysis e e ing o di e en p oduc ion mixes, o a ime ho izon o 52 hou s Model Ini ial Mix Mix 1 Mix 2 Mix 3 Mix 4 Mix 5 FIFO 262 271 254 256 268 253 Op imized Milk-Run 389 385 359 379 386 370 Nea es WS Rule 377 366 344 362 371 355 Base, wi h p e- aining 404 409 376 388 394 382 Base, wi hou p e- aining 501 473 471 473 477 478 5.7.2 P ocessing Times The second c i e ion o e alua e he adap abili y o hese models ela ed o p oduc i i y includes he in oduc ion o hem in a s ochas ic en i onmen in which, acco ding o Sec ion 4.8.2, he de ini ion o p ocessing imes ollows a iangula dis ibu ion. The a iabili y ac o , α, applied o he p ocessing imes o each pa was 0.4. I is also impo an o compa ison pu poses o main ain he ime ho izons in 36 and 52 hou s and he AGV speed in 1.5 m/s. The ollowing Tables 5.20 and 5.21 p o ides he esul s o he 10 simula ions gene a ed and indi idually applied o each one o he implemen ed models. These esul s a e impo an o analyse he a e age and s anda d de ia ions o he esul ing p oduc i i y alues. The low s anda d de ia ion as well as he simila i y be ween he a e age alues o p oduc i i y ob ained o each one o he models and he alues associa ed wi h he o iginal mix, also indica e ha all he alues a e in ag eemen wi h each o he . So, simila ly o wha happened wi h he p oduc ion mixes, we can conclude ha all models p esen good adap abili y when subjec o s ochas ic en i onmen s. To conclude, he Base Model, wi hou p e- aining, is he one ha allows he achie emen o he highe p oduc i i y alues. 78 REFERENCES [14] R. D ießel and L. Mönch, “An in eg a ed scheduling and ma e ial-handling app oach o complex job shops: a compu a ional s udy,” In e na ional Jou nal o P oduc ion Resea ch, ol. 50, no. 20, pp. 5966–5985, 2012. [15] R. M. Logís ica, “Milk un: Concei os, an agens e ganhos pa a a ope ação logís ica.” h ps:// e is amundologis ica.com.b /blog/achiles/ milk- un-concei os- an agens-e-ganhos-pa a-a-ope acao-logis ica. Accessed in 2019-11-23. [16] R. F. D. San os, “Es a égias híb idas de machine lea ning e simulação pa a a esolução de p oblemas de escalonamen o,” 2018. [17] S. Huang, “Op imiza ion o job shop scheduling wi h ma e ial handling by au oma ed guided ehicle,” 2018. [18] A. de F ei as Ribei o, “Taylo ismo, o dismo e oyo ismo,” Lu as Sociais, ol. 19, no. 35, pp. 65–79, 2015. [19] A. Aze edo, “Lean hinking: p incípios, undamen os, p incipais e amen as.” Aze edo, Amé ico, Lean hinking: p incípios, undamen os, p incipais e amen as. Sis ema de P o- dução Toyo a Pa e I, 2013. 28 slides. Suppo ma e ial o he GOPE cou se o MIEEC o FEUP. Accessed in 2020-01-23. [20] G. Zäp el, R. B aune, and M. Bögl, Me aheu is ic Sea ch Concep s: A Tu o ial wi h Appli- ca ions o P oduc ion and Logis ics. Sp inge Be lin Heidelbe g, 2010. [21] A. Meye , “Milk un design (de ini ions, concep s and solu ion app oaches). disse a ion, ka ls uhe ins i u ü echnologie (ki ) akul ä ü maschinenbau, ki scien i ic publishing, ka ls uhe (2015),” 2015. [22] . S. G. B a , G. S., “Milk un logis ics: Li e a u e e iew and di ec ions, london, england,” 2011. [23] A. Gé on, Hands-On Machine Lea ning wi h Sciki -Lea n, Ke as, and Tenso Flow: Con- cep s, Tools, and Techniques o Build In elligen Sys ems. O’Reilly Media, 2019. [24] Y. Li, “Deep ein o cemen lea ning: An o e iew,” a Xi p ep in a Xi :1701.07274, 2017. [25] S. Ma sland, Machine lea ning: an algo i hmic pe spec i e. Chapman and Hall/CRC, 2014. [26] V. Mnih, K. Ka ukcuoglu, D. Sil e , A. G a es, I. An onoglou, D. Wie s a, and M. Ried- mille , “Playing a a i wi h deep ein o cemen lea ning,” a Xi p ep in a Xi :1312.5602, 2013. [27] V. H. de Sousa Ca nei o, “Abo dagens híb idas de" machine lea ning"/simulação pa a sis- emas logís icos dinâmicos,” 2019. [28] I. G igo ye , Anylogic in h ee days. Fi h Edi .C, 2018. [29] J. Banks, J. S. Ca son II, L. Ba y, e al., “Disc e e-e en sys em simula ion ou h edi ion,” 2005. [30] A. K. d. Sil a, Mé odo pa a a aliação e seleção de so wa es de simulação de e en os dis- c e os aplicados à análise de sis emas logís icos. PhD hesis, Uni e sidade de São Paulo, 2006. REFERENCES 79 [31] V. H. S. Ca nei o, Hyb id Machine Lea ning/Simula ion App oaches o Decen alized P o- duc ion Scheduling. 2018. [32] “Socke p og amming in py hon (guide) by na han jennings.” Accessed in 2020-03-02. [33] M. Almeida, Luís; San os Ped o; Sousa, “Suppo in o ma ion o lab assignmen 1a cp socke s in c.” Suppo in o ma ion o lab assignmen 1a TCP socke s in C, Luís Almeida, Ped o San os, Ma io Sousa, 18 slides. Suppo ma e ial o he ACI cou se o MIEEC o FEUP. Accessed in 2020-04-15. [34] A. Hill, A. Ra in, M. E nes us, A. Glea e, A. Kane is o, R. T ao e, P. Dha iwal, C. Hesse, O. Klimo , A. Nichol, M. Plappe , A. Rad o d, J. Schulman, S. Sido , and Y. Wu, “S able baselines.” h ps://gi hub.com/hill-a/s able-baselines, 2018.