Data-driven identification of optimal substitutions in soccer

Degree in Mathematics Degree in Engineering Physics Bachelor’s Degree Thesis Data-driven identification of optimal substitutions in soccer Joan Hernanz i Ib´ a˜ nez Director (KU Leuven): Jesse Davis Tutor (UPC): Enrique Romero Merino May 2023 Abstract Data-driven football analytics is a rising field, and clubs are spending more and more resources on gaining a competitive edge through data-driven techniques. Substitutions are the main tool a coach has to intervene in the course of the game, and their limitation and relevance have attracted interest to their study. In this thesis, we seek a data-informed approach to identifying and predicting optimal substitutions. With the recent change in football legislation, now each team is permitted up to five substitutions per match. We compare the new paradigm with the prior one and look for optimal substitutions with the use of data-based models. With machine learning classifiers and a substitution sensible in-game win probability model, player changes have been assessed. The same models have been used for simulating alternative types of substitution and timing, in order to make data-driven approaches that increase the chances of success. The addition of extra substitutions has resulted in an increase in tactical interventions by coaches, but the match dynamics have remained the same. Machine learning models obtain good results in the prediction of substitution assessment. The win probability model is sensible to substitutions, which have generally a better effect on the substituting team. Offensive substitutions generally increase the winning probabilities, especially for losing teams. No timing is observed to be significantly better for doing substitutions, but in particular cases can be very relevant. Keywords: Football Analytics, Football, Big Data, Substitutions, Machine Learning, Win Probability Model, Bayesian Model, Data-Driven Decisions. American Mathematical Society 2020 Mathematics Subject Classification: 62-07, 60E99 1 Resum L’an`alisi basada en dades del futbol ´es un camp en auge, i els clubs dediquen cada vegada m´es recursos a obtenir un avantatge competitiu mitjan¸cant t`ecniques basades en dades. Les substitucions s´on la principal eina de la qual disposa un entrenador per a intervenir en el transcurs del joc, i la seva limitaci´o i rellev`ancia han despertat l’inter`es pel seu estudi. En aquesta tesi busquem un enfocament anal´ıtic per a la identificaci´o i predicci´o de substitucions `optimes. Amb el recent canvi en la legislaci´o futbol´ıstica, ara es permeten a cada equip fins a cinc substitucions per partit. Comparem el nou paradigma amb l’anterior i busquem les substitucions `optimes amb l’´us de models basats en dades. Amb classificadors d’aprenentatge autom`atic i un model de probabilitat de vict`oria en directe sensible a les substitucions, s’han avaluat els canvis de jugadors. Els mateixos models s’han utilitzat per a simular substitucions de diferent tipus i en moments alternatius, amb la finalitat de realitzar plantejaments basats en dades que augmentin les probabilitats d’`exit. L’addici´o de canvis addicionals ha provocat un augment de les intervencions t`actiques dels entrenadors, per`o la din`amica del partit es mant´e igual. Els models d’aprenentatge autom`atic obtenen bons resultats en la predicci´o de la valoraci´o de les substitucions. El model de probabilitat de vict`oria ´es sensible a les substitucions, que solen tenir un millor efecte per a l’equip que les realitza. Les substitucions ofensives solen augmentar les probabilitats de vict`oria, especialment per als equips que van perdent. No s’observa que cap moment sigui significativament millor per a fer el canvi, per`o en casos particulars pot ser rellevant. Paraules clau: Anal´ıtica de Futbol, Futbol, Big Data, Aprenetatge Autom`atic, Model de Probabilititat de Victoria, Model Bayesi`a, Decisions Basades en Dades. 2 Resumen El an´alisis basado en datos del f´utbol es un campo en auge, y los clubes dedican cada vez m´as recursos a obtener una ventaja competitiva mediante t´ecnicas basadas en datos. Las sustituciones son la principal herramienta de la que dispone un entrenador para intervenir en el transcurso del juego, y su limitaci´on y relevancia han despertado el inter´es por su estudio. En esta tesis buscamos un enfoque anal´ıtico para la identificaci´on y predicci´on de sustituciones ´optimas. Con el reciente cambio en la legislaci´on futbol´ıstica, ahora se permiten a cada equipo hasta cinco sustituciones por partido. Comparamos el nuevo paradigma con el anterior y buscamos las sustituciones ´optimas con el uso de modelos basados en datos. Con clasificadores de aprendizaje autom´atico y un modelo de probabilidad de victoria en directo sensible a las sustituciones, se han evaluado los cambios de jugadores. Los mismos modelos se han utilizado para simular sustituciones de distinto tipo y en momentos alternativos, con el fin de realizar planteamientos basados en datos que aumenten las probabilidades de ´exito. La adici´on de cambios adicionales ha provocado un aumento de las intervenciones t´acticas de los entrenadores, pero la din´amica del partido se mantiene igual. Los modelos de aprendizaje autom´atico obtienen buenos resultados en la predicci´on de la valoraci´on de las sustituciones. El modelo de probabilidad de victoria es sensible a las sustituciones, que suelen tener un mejor efecto para el equipo que las realiza. Las sustituciones ofensivas suelen aumentar las probabilidades de victoria, especialmente para los equipos que van perdiendo. No se observa que ning´un momento sea significativamente mejor para hacer el cambio, pero en casos particulares puede ser relevante. Palabras clave: Anal´ıtica de F´utbol, F´utbol, Big Data, Arendizaje Autom´atico, Modelo de Probabilidad de Victoria, Modelo Bayesiano, Decisiones Basadas en Datos. 3 Acknowledgements First of all, I would like to thank Jesse Davis for his support, for always finding a moment to talk and guide me with his ideas and overall clear picture, and Maaike Van Roy, for the weekly feedback and always encouraging my work. I have very much enjoyed working in the football analytics topic with such a world-leading group. Thanks to KU Leuven for offering all of the services of one of the most prestigious universities in mainland Europe. Gracias a Enrique Romero, mi tutor por correspondencia electr´onica, por toda la ayuda, correciones y ´animo a distancia. Gr`acies en especial al CFIS, per la confian¸ca i l’ajuda econ`omica durant tota la carrera, i per oferir-me l’oportunitat de passar un any on volia investigant el que m’agrada. Gr`acies als que heu fet que la meva est`ancia a B`elgica valgui molt la pena: Leuven Team, la Resistencia i els Earls of Leuven. I als que m’heu acompanyat tota la carrera: els matfis, i la gent de la FME i EF i la gent del Thau. Una abra¸cada molt forta pel Marc, que s´e que estas molt orgull´os de tots nosaltres. Gr`acies a l’Andreu pels memes, i pare, mare i familia en general per les visites i videotrucades absurdes. Gr`acies, en especial, a la Martina, que hi cap en totes les categories i que es mereix molt m´es que aquesta frase. Joan, Leuven, May 2023 4 Contents 1 Introduction 7 1.1 Problem Description .......................... 7 1.2 Research Questions ........................... 8 2 Background 10 2.1 Introduction to Football Analytics ................... 10 2.2 The Substitution Problem ....................... 11 2.3 Related Work .............................. 13 3 Data Overview and Analysis 14 3.1 Data Description ............................ 14 3.1.1 Data providers ......................... 14 3.1.2 Training and test data ..................... 16 3.2 Data Analysis .............................. 17 3.2.1 Substitutions and goal scoring ................. 18 3.2.2 Players’ performance through time .............. 19 3.3 Comparison Between 3-subs and 5-subs Eras ............. 21 4 Substitution Evaluation 24 4.1 Pre-Study Analysis ........................... 24 4.1.1 Assessment rules definition ................... 24 4.1.2 Rules correlation ........................ 26 4.2 Classification Models .......................... 27 4.2.1 Offensive and defensive substitutions ............. 28 4.2.2 Scoreline conditioned evaluation ................ 28 4.2.3 Advanced metrics evaluation .................. 29 4.3 Win Probability Model ......................... 30 4.3.1 Model definition and validation ................ 30 4.3.2 Effect of substitutions ..................... 32 4.4 Results and Discussion ......................... 35 5 Substitute Selection 36 5.1 Bench Analysis ............................. 36 5.2 Type of Substitution .......................... 39 5.2.1 Win probability simulations .................. 39 5.3 Player Selector. Theoretical Framework ................ 42 5 6 Optimal Timing 44 6.1 Timing Analysis of Positive Substitutions ............... 44 6.1.1 Substitution classification ................... 44 6.1.2 Win probability model ..................... 46 6.2 Simulation of Alternative Timing ................... 47 6.2.1 Substitution classification models ............... 47 6.2.2 Win probability model ..................... 48 6.3 Results and Discussion ......................... 50 7 Conclusions 52 References 57 6 Chapter 1 Introduction 1.1 Problem Description Football is the most important sport in the world, with fans all over the globe tuning in every day to follow games, and shows discussing everything related to those games. With the increasing amount of data being generated in football, the industry has started promoting data-driven and data-informed [13] techniques to optimize each and every task. In recent years, football analytics has emerged as a powerful tool for understanding the game and gaining a competitive advantage [30]. While the use of analytics in sports is not new, it has been most prominently associated with isolated events and higher-scoring games (such as baseball or basketball). Data analysis has been used for decades in these sports to evaluate players and teams, inform strategy, and make informed decisions [20]. However, the traditional mindset of football coaches and fans, combined with the naturally random spirit of football due to its low-scoring nature, has often made football resistant to the use of data and statistics in the sport. In contrast, many football analysts and experts believe that the use of analytics is essential for understanding the game and improving performance. The success of teams that have embraced analytics in all areas provides compelling evidence for the value of this approach. Substitutions, in soccer, are important resources, because they alter the ability to change tactics [23], which can often directly influence the final outcome of the game. Through substitutions, the coach explicitly defines what his intention is in relation to the game, and can show their ability to maneuver mid-match. Nowadays, coaches can perform up to five substitutions per match, in a total of three windows (and possibly at half-time) [19]. Coaches have to use these substitutions for possible injuries, fatigued players, protection for booked players and, potentially, change the course of a match through a tactical or player change. In this thesis, we will focus on the latter. Changes in formation, playing style and even a team’s mentality can be accomplished without substituting a player. We are interested 7 in identifying optimal substitutions and knowing the effectiveness of substitutions, according to the progress of the match. The type of substitution, whether it must change the team formation of style, picking the correct player and being spot on with timing are very important for a coach to accomplish their objective. Accomplishing this can result in altering the outcome if they are losing, or maintaining the victory if they are already winning. Multiple strategies can be followed, since tactical decisions are simultaneously factors for the team stabilization and the opposition’s destabilization ([12], [4], [28], [14]). 1.2 Research Questions Three substitutions per team and per match were the rule until a few years ago. With the COVID break, many leagues accepted that teams could do up to five substitutions per match, due to schedule overload. Even though some major leagues went back to the three substitutions per match, from this season (2022/23) all major leagues in Europe and continental competitions have opted to allow five changes per match [21]. Since there is no apparent extra match load, we pose the following question. Research Question 1 Has the new 5-substitutions per match rule made that more substitutions become relevant? Do coaches have more ways to maneuver? We will focus on this question from an observational point of view, which will be addressed in section 3.3, by comparing of the last complete season previous to COVID with the seasons that have followed that allowed five substitutions per match. Next, we will focus on assessing substitutions. This key part of the investigation is the base for the whole Thesis, as investigating optimal substitutions depends on how you define them. We pose the following research question. Research Question 2 How can we determine if a substitution is useful? What is the performance of the subbed player in comparison to the starting one? Do substitutions have an effect on a team’s win probability? The question will be addressed in chapter 4. Different evaluation methods will be discussed and analyzed, including changes in result, match momentum or in substitute performance. Machine learning models will be trained for the prediction of such optimal substitutions. A model based on [29] will be developed, using substitution information to calculate the win probability during the match and discuss how sensible the model is to such features. The models developed in this chapter will be used during the course of this report. The next topic that will be addressed is the selection of the adequate player to come in. Given the moment at which a coach decides to introduce some kind of 8 But SCA are important because they provide information on the game previous to a goal chance which is just a shot. Match Shooting sheet xG, SCA per each shot Lineup sheet Starting and bench players Match report sheet Goals, substitutions and bookings Figure 3.1: Diagram of information obtained from a match The shooting sheet is of the utmost importance since it gives xG data for each shot. This recent feature, that fbref.com has included due to a recent change in data-providers 1 , enables us to do a much more in-depth analysis. The low-scoring nature of football means that goals are a rarity, so xG is a weighted way to study the teams’ performance. Since the sheet provides the SCA, we can analyze the influence of players at the shot level, not only in shots but also in the actions leading to them, opening the doors to many analyses. Following the definition of xG, even though the value assigned to a shot might differ depending on the provider of the data, a series of metrics [15] and terminology is derived from it, which are directly provided by the sheets or easily computed. Definition 3.3 Expected Goals Against (xGA) is the amount of xG amassed by the opposition during a period of time (period of a match, complete match or season). Definition 3.4 Expected Goals Difference (xGD) is the difference in the xG amassed by a team and the opposition during a period of time (period of a match, complete match or season). Definition 3.5 To the player that does the pass previous to the shot, we assign an Expected Assists (xA) value equal to the xG assigned to the shot. Shot-by-shot data gives this report a much more profound scope. This falls short of the event-by-event and tracking data that clubs and industry leaders are currently working on, which gives the opportunity for a more in-depth analysis, which ultimately can lead to a competitive advantage [27]. Next on, we have access to the lineups of both teams. Further analysis of this data will be conducted in section 5.1. Lineups include the position the player has played during the match, including multiple positions if the coach has considered tactical changes throughout the match. For unused bench players, position data is void, 1 https://www.sports-reference.com/blog/2022/11/fbref-shot-level-xg-now-on-match-reportsfor-20-competitions/ 15 since they did occupy any position. Nonetheless, we can study their positions during the season or get access to the players’ transfermarkt.com page, where their usual position is available. Finally, from the match report we get the commonly known match features: goals, yellow and red cards, and substitutions. This information gives us an overview of the match state. Such data has been available for much more time than the new in-depth spatiotemporal data, but this report shows that important analysis can be done with such simple, and widely available, data points. One missing feature from fbref.com ’s data is the finishing times of both first-halves and matches. Such information can be relevant, in particular when large injury time is introduced, but since we have the minute of all shots, bookings and substitutions, it will be possible to assess the final minute of playing time with sufficient accuracy. clubelo.com Elo ratings Elo ratings are obtained from clubelo.com page, a reference page for European Football Club Rankings based on the Elo system. We can access the complete rankings on any given day, enabling us to obtain the Elo ratings of the teams on every match day. These rankings are comparable through different leagues since they are weighted based on international matches. We do not have the particular Home Field Advantage (HFA) of each competition, so we must add a general HFA on all computations. 3.1.2 Training and test data As training data, we have selected to study the top leagues in European men’s football. We will divide between data for the study of player performance and the data for the study of substitutions. To study players’ productivity, we have studied the shot-by-shot xG of four complete seasons, from 2018/19 up to 2021/22, in the following leagues: the English Premier League, the Spanish LaLiga, the Italian Serie A, the German Bundesliga and the French Ligue 1. These five competitions are known as the 5 big leagues and will be referred to as so during this Thesis. This sums up to a total of 7203 matches and 177980 shots taken into account for the training data. Unfortunately, we cannot use all this data when training models that evaluate substitutions, since the 5-changes per match rule was implemented after the COVID stop. This was through the 2019/20 season. The last matches of such had 5 substitutions, but the change was just implemented, so coaches might not have it used to fully profit from such a rule. Thus, we are counting only the 2020/21 and 2021/22 seasons for substitution model training. The English Premier League decided to go back to the 3-subs per match ruling, so we cannot include it in the training. To minimize the effect of the reduction on matches, we have added these seasons of the Dutch Eredivisie and the Portuguese Primeira Liga, the next leagues 16 according to the UEFA coefficient 2 . Thus we are evaluating 4116 matches and 34368 substitutions. The number of matches per league is 380 for a 20-team competition, which are the top-tier leagues in England, Spain, Italy and France, and 306 matches for the 18-teams leagues, such as the top-tier German, Dutch and Portuguese leagues. The French first division, during the COVID break, had only 279 matches played, which explains the odd number of matches in the corresponding dataset. The test data are all matches played this season (2022/23) at the male European top 5 leagues up to the FIFA match day of finals of March since the English league has used 5 substitutions per match this year [21]. We consider coaches are used to this ruling due to being in the majority of Europe and international competitions. Thus, our test datasets include n = 1305 matches played in the first divisions of England, Spain, Italy, Germany and France since the start of the season up to the 20 th of March. Table 3.1 provides a summary of the datasets used in this Thesis, and the relevant number of data points each of them offers. Seasons Data Matches Data points Top5 18/22 Training Data-Players 7203 177980 shots ES, IT, DE, FR, NL, PT 20/22 Training Data-Subs 4116 34368 subs Top5 22/23 (up to 20th March) Test Data 1305 11377 subs 32461 shots Table 3.1: Summary of the data sets used for training and testing. We have discarded international competitions such as the Champions League since its knockout stage and short group phase can produce situations where the objective of the team is not to win (depending on results from previous matches) or that reserve players are used (teams already classified or eliminated). In a similar way, we have not taken into account any of the relegation or title playoffs in Germany, France, the Netherlands or Portugal. Thus, we are considering only regular league games, where these situations can happen in a much more sporadic way. 3.2 Data Analysis In a normal match, during the first-half, lineups are what coaches decided previous to the encounter, and most of the tactics are previously decided and given to the players. in the second-half, players might receive detailed instructions at half-time on how to attack the opposition’s weaknesses, and new players are introduced to potentially modify the tactics of a team. Thus, we can distinguish between first-halves, where the initial approach plays a major role, and second-halves where adaptability becomes a main asset. 2https://www.uefa.com/nationalassociations/uefarankings/country/ 17 3.2.1 Substitutions and goal scoring Comparing the first and second-halves can provide us with an insight into the effect of substitutions in a game. Although it is true that second-halves are usually longer, it is assumed that the injury time is intended to correct for lost time, so we can assume that the effective playing time is similar between the two halves. But the facts are that 55.6% of the xG and 56.1% of the goals happened in second-halves, which is a significant increase. When there were three substitutions, goal-scoring intensity increased after the first and second substitutions, but was reduced after the opposition’s third [1]. When comparing this to last season, 21/22, with five substitutions through 3 substitution windows we obtain similar results. In Figure 3.2 we see how the xG generated evolves after each substitution window, normalized by minutes. xG intensity, and consequently goal probability, increases after the first and second coach interventions, due to the fresh legs and new tactical information brought into the pitch. It also decreases after the third, which is most probably due to the overlapping with the opposition’s substitutions, which decreases goal-scoring probability [1]. Figure 3.2: Average xG generated after each substitution window, grouped by leagues. Matches do tend to be more open in the second-half, as most of the goals and goal opportunities happen there. Coaches can introduce tactical variations and therefore increase their attack production. But at the end of the match, tactical decisions oppose themselves and the goal-scoring intensity decreases a little bit, even though the number of minutes after the last window is smaller and might be less significant. 18 3.2.2 Players’ performance through time The main metric for assessing the actions of players will be the newly defined expected Value. Definition 3.6 We define a new metric Expected Value (xV) as the amount of xG amassed by a player in all of their shots and SCA. We are aware of the shortcomings of xV. Metrics such as expected threat [32] or VAEP [11] give value to all actions, and are more sensitive to the amount of real hazard the action posed on the defending team. We are not valuing all the actions, just shots and the two previous teammates’ actions; and we are giving actions the value of the shot they helped produce, which may not be directly related to the quality of the actions. Extreme examples can also happen, such as a player doing a dribble, getting fouled and taking the penalty. Since the penalty has an xG of 0.76, and the player produced the two SCA, they would amass 2.18 xV, which is much higher than the value of the player in that circumstance, but such cases are only theoretical and in practice are treated as outliers. Knowing the faults this metric can have, it gives much more information than just xG or even xG+xA, since it takes into account many more types of actions, and since we do not work with event-by-event data, it gives us the broader view possible. Expected Value will be used, through the Thesis, when talking about players, while expected Goals refers to the team since otherwise we would be double, or even triple, counting each goal-scoring chance. Substitutions can have multiple reasons, and fatigue is usually indicated as the cause of many of the substitutes. By having the xG data of every shot, we can compute the xV that players generate through time. On Figure 3.3 we can compare the Predicted xV of an average top 5 leagues player, depending on whether they start on the pitch or the bench. Definition 3.7 Predicted xV is the amount of xV expected to generate in the number of minutes played, by multiplying the rate of xV per minute by the number of minutes. 19 Figure 3.3: xV generated through a match at the average player rate of xV per minute. For bench players, in red, minutes are counted from the moment they enter the pitch There are two main observations to be made. On the one hand, we can see that the rate of xV generated is almost constant through time. This happens for both starting players and bench players. This is, the rate of attack generated by a starting player does not decay, which would indicate that fatigue does not really apply, or is compensated by the increase in goal opportunities overall. On the other hand, substitute players produce a higher amount of xV. This is a bit of a counter-intuitive fact since one would expect that the best attacking players generate most of the occasions, and that the best players are starters for their teams. This happens due to the overlapping of two phenomenons: an increase in overall generated xG and fatigue of starting players. Matches tend to become more open and have more xG when the match is ending, and starting players produce fewer xV in relation to the total generated. Thus, substitute players usually play in more open matches and can produce more attacking opportunities, independently of their quality in comparison to starters. 20 3.3 Comparison Between 3-subs and 5-subs Eras In Research question 1 we wondered about the effect of the new rule that allows five substitutions per team per match. This rule, adopted as an emergency at the restart after COVID lockdown, seems to establish the norm for the future. Substitutions are the main tool coaches have to change the course of the match. Since they usually happen in the second-half, we can focus on sign changes as a measure of coaches’ influence. We found that over the last compete season before COVID, 2018/19, a 42.6% of the matches changed their result from half-time to the end of the match, and also that 61.7% of the matches drawn after the first-half ended up being a victory for either team. When we check the two following complete seasons with 5 substitutions, we observe that the percentage of matches that changed their result were 39.3% in 20/21 and 41.4% in 21/22, very similar numbers and if something, smaller. Similarly, a 60.9% of half-time drawn matches ended up not being a draw in 20/21 and 61.5% in 21/22, which are the same percentages with a bit of variation. Players distribute themselves through the pitch, with different roles depending on their position [23]. For each position, the data providers define, we have assigned a value, the offensiveness, related to their proximity to the opposition’s goal. This way, we can assign an offensive value to the player depending on the positions they have occupied during a match. If they have occupied multiple positions, we compute the average of the offensiveness of all the positions filled. In Table 3.2 a relation between positions and their offensiveness value can be found. Position Abbreviation Offensiveness Goalkeeper GK 0 Centre, Left or Right back, Defender CB, LB, RB, DF 1 Defensive Midfielder, Wingback DM, WB 2 Central, Left or Right Midfielder CM, LM, RM, MF 3 Attacking Midfielder, Left or Right Winger AM, LW, RW 4 Forward, Second Striker FW, SS 5 Table 3.2: Different positions, with the abbreviations used by the data providers and the relative offensiveness assigned. Definition 3.8 We define a player change as an offensive substitution if the offensiveness of the player coming in is more than 0.5 units higher than the player going out. Similarly, a defensive substitution is where the positions occupied by the new player are less offensive by at least 0.5 units. All other substitutions are considered neutral. There are multiple reasons to substitute a player: protecting a booked player, injuries, fatigue, or tactical substitutions. According to the types of substitution, all offensive 21 and defensive substitutions carry tactical adjustments. A neutral substitution can also be tactical since different players have different profiles, but they might also be just replacements due to physical conditions or the context of the match. Due to our usage of widely available data, we cannot distinguish between these two cases. Thus, we will compare neutral and non-neutral substitutions. The percentage of non-neutral substitutions was 24.6% during season 18/19, and continued to be 24.3% in 19/20 up to the COVID lockdown. When leagues resumed, allowing five substitutions, only 18.0% of substitutions were non-neutral. The same rate continued in 20/21 with a 17.2%. The number of non-neutral substitutions per team per match remained the same, close to 0.7. This means that coaches made a similar number of tactical changes, keeping their usual number of tactical alterations through a match, but increased non-tactical ones, probably due to fatigue, COVID sequels and a more compact schedule. A summary of these rates is in Table 3.3. These percentages may vary depending on the definition of offensive and defensive substitutions [7]. On season 21/22, with coaches having time to adapt, the percentage of non-neutral subs raised back to 24.4% of the total ones, and coaches did 1.1 non-neutral substitutions per team per match. This trend seems to be contradicted by the results of 22/23 season so far, going back to 17.8% of totals. This might be due to the World Cup being held in December, which has made teams play more frequently and players play more matches, thus increasing the need for physical substitutions. Season Non-neutral subs percentage Non-neutral subs per match Subs per match 18/19 24.6% 0.72 2.93 19/20 pre-COVID 24.3% 0.70 2.87 19/20 post-COVID 18.0% 0.77 4.29 20/21 17.2% 0.72 4.17 21/22 24.4% 1.1 4.33 22/23 17.8% 0.77 4.35 Table 3.3: Summary of the non-neutral substitutions per season, both as percentage of total substitutions and per match, and also the number of substitutions per team per game. Finally, we would like to see if teams are adapting to the new rule, so we look at the number of overall substitutions made by the teams over time. After performing a Welch Two Sample t-test on the substitutions per team per match through seasons 20/21 and 21/22 we obtain that the mean of the latter, 4.33 subs per team with a standard deviation of 0.88, is statistically significant (p-value 4 . 328 e− 15) bigger than the 4.17 of the first season, with 0.92 deviation. The comparison with the ongoing 22/23 season shows that the value remains stable at around 4.35 subs per match, and with a similar standard deviation. 22 Including a new substitution is a very relevant change in the rules [34]. We have observed that even though coaches have more maneuverability, the change in dynamics in second-halves remains very similar. A possible explanation is a Red Queen effect [33], where both teams evolve so that the coaches’ effect cancels itself. When the new rule was introduced, the tactical substitutions per match remains the same, but the percentage decreased. With the adaptability of the teams’ staff, which can be seen in the increase of overall substitutions a team uses, the amount of non-neutral substitutions has increased. 23 Chapter 4 Substitution Evaluation This section focus on Research question 2 from different perspectives. First, we will try to define what is a useful or positive substitution according to different criteria. We will also try to compute the winning probability and thus derive the impact substitutions have. Football’s low-scoring nature generates multiple situations where better-playing teams do not necessarily convert into a positive result. Thus, if a substitution has an impact on the game that is not reflected in the data we have available, we might not be able to evaluate it properly. In this chapter, we use the available information to try and assess substitutions in the best possible way. 4.1 Pre-Study Analysis Evaluating a substitution can be a difficult task, and more so not having event-byevent data, so we cannot get valuations on the effect of a particular player’s actions, such as xT [32] or VAEP [11]. What we do have is shot-by-shot data, and therefore we can compute the expected Value generated by a player with their shots and Shot Creating Actions. We are going to pack substitutions into windows, as the effect of two simultaneous changes can not be discerned, so in this section the term substitution will be used also to refer either to a single sub or a window of multiple player swaps, according to context. The analysis in this section is new, with rules developed especially for this thesis. Even though some other studies use similar assessment rules ([26], [31], [7]), we introduce a wider range of ways to study and evaluate substitutions. 4.1.1 Assessment rules definition For those windows including an offensive or defensive substitution, we are going to define simple rules: an offensive change is successful if a goal is scored afterwards; a defensive one is successful if no goal is conceded from that moment to the end of the 24 gi∼B ( t, θt,i ). By adding that to the already scored goals, Gt,i , we can simulate the final result. Different approaches were taken in the way to predict θt,i . The intuitive idea is that the goal-scoring probability at the next minute increases with time, but different features can affect it in different ways. It has been shown that the relative importance of variables evolves non-linearly of time [29], but also intuitively close time frames should have a close relationship between the variables. The four algorithms were: •Multiple Logistic Regressions (mLR) A logistic regression is calculated for each time remaining t . We use this approach and not a general logistic regressor because the importance of certain features varies with time. θt,i =ewtxt,i 1 + ewtxt,i (4.1) •Random Forest (RF) A Random Forest regressor. Random forest is able to deal with non-linear interactions between the features. •Multiple Random Forests (mRF) A different Random Forest for each time frame, but using also close-time values also as training to prevent a certain time frame to be an outlier. This approach is basically the same as a simple random forest but with a very much increased number of trees. •Gaussian Walk (GW) This Bayesian approach also uses an inverse logistic function with a series of weights αt,i which take as a prior the weights of the preceding time frame, when the minutes remaining are t. θt,i = invlogit(αt·xt,i +β)αt∼N(αt+1,2) β∼N(0,10) (4.2) Comparing this particular model to the one developed by Robberechts, Van Haaren and Davis [29], we see a few important differences. Since we want to look at substitutions, we only study the probabilities for the second-half of matches. We work with the number of minutes remaining t , while Robberechts et al. divide the match into 100 time frames. Our approach gives each minute the same importance, even if the match counted with a big amount of stoppage time. But the model faces the same problem if it was to be implemented in real life since injury time is unknown. For the test data, we assume the 96 th minute to be the last, since it is the mode, and we believe this model should be evaluated without knowing the real last minute. The second main difference is we predict θt,i and then simulate the 31 match, while Robberechts et al. implement it in their model and directly predict the probabilities of each result. In order to validate the performance of a win probability model, we need to look at the whole picture. If the model gives a team an 8% of win probability on a particular match, that result is either going to happen or not, because the match only happens one time. Thus, we group all the predictions of an 8% and we expect that such a percentage of the cases actually happen. We call predicted probability, to the probability the model predicts for a certain result, and actual probability the percentage of times such result comes true. In Figure 4.2 we see the graphs for the four algorithms and their performances on the three different results: home win, away win and draw. The size of the point represents the number of such predictions that were made. This is a difference with respect to the model of Robberechts et al. Since we calculate probabilities as a result of the simulation, given the αt,i , and not as a direct prediction of the model, we get a much more even distribution of predictions. This is, for the Random Forest or multiple Logistic Regressor, they got a non-uniform distribution of predictions, having few predictions with high probabilities. When implementing the Bayesian approach, that problem was partially solved. In our case, for all models we have a much more uniform distribution, even though predictions of 0-2% are a bit more typical. 4.3.2 Effect of substitutions A team might take multiple approaches for a match, such as maximizing their win probability or minimizing the losing one [3]. These decisions might depend on many factors: league positions, relative strength, psychological momentum, etc... In this thesis, we will focus on regular league games, and based on that 3-point per win, 1-point per draw rule, we will measure expected points as a unified approach. Definition 4.7 Expected Points (EP) is the expected value of points for a team given their winning, drawing and losing probabilities in a game. It is logical that a team would like to maximize the points they obtained, since it may give them better long-term results. Thus, we will use EP as a min-max-style approach, and we will assess possible strategies as a result of their effect on EP. This metric weights the draw and win probabilities in the same way the league scoring system does, and therefore we believe it can be accepted as a general strategy for all teams. We compare the winning probability after the substitution to the minute before it happened. Consequently, when a goal and a change happen at the same minute, so it reports fbref.com . But we cannot discern what happened before. This produces a massive change in win probability, and a very high ∆ EP , which is due to the goal and not the substitution, so we discard those substitutions from the analysis. To ensure that the substitution features were relevant we trained the same model but 32 (a) Multiple Logistic Regressors (b) Random Forest (c) Multiple Random Forest (d) Gaussian Walk Figure 4.2: Evaluation of different techniques to calculate θt,i and the final performance of the in-game win probability model, comparing predicted probability to actual probability for each type of prediction. Size indicates the number of predictions. without those features. This is, predict θt,i from xt,i = ( t, i, τt,i ). We shall call this model simple as opposed to the complete model which we will refer to as complex. In Figure 4.3 we see the distribution of the variation of expected points when a substitution is made for the two models. The simple model is oblivious to the fact that the substitutions have been made, so the distribution should be as if we picked random moments in matches’ second-halves and calculated their ∆EP . 33 Figure 4.3: Distribution of the variation of EP after a substitution. Comparison between the simple and the complex model. Instead, in the distribution of the complex model we see how the number of substitutions of none or very little impact in EP is much lower, and that the substitutions with an increase or decrease in EP grow, as it can be seen in both tails of the complex distribution curve. If we separate the substitutions in positive and negatives, as in Figure 4.4, we can focus on the difference between the positive and negative tail. In order to better visualize these tails in smaller numbers, we apply the transformation to the x-axis and expose √∆EP . Figure 4.4: Distribution of √∆EP for positive and negative substitutions in the complex model. 34 Here we can see how the two curves are not symmetric, which means substitutions are not just random moments according to the model. We can also see how the density is bigger for the substitutions with a positive impact on the match. One of the conclusions is that the majority of substitutions have a positive effect on the team’s goal-scoring probability, while not diminishing the defense so match, so the win and draw probability increase and so do the expected points. 4.4 Results and Discussion In this chapter, we wanted to study the problem of properly evaluating the effectiveness of a substitution. This is a difficult thing to do with the available data since many of the player’s actions go unnoticed. Different approaches have been taken for the assessment, and even though they are correlated, in many cases a single player’s performance, or even the whole team, might not translate into a palpable result in the form of goals. We trained many models to predict the success of the substitutions. In a similar study on the Brazilian League [7], they obtained model performances of 75-85%. Our results oscillate between the high sixties and high seventies. As we described, methods in both studies differ, and this might explain why a more detailed dataset gives worse predictions. It is true, though, that Brutti et al. focus much on the team’s strength, while we only counted for Elo ratings. A more team-based approach might be necessary to improve predictions on substitutions. We observed that player-specific information did better the results of the models predicting xV Change Rule. The second approach consisted in calculating the importance of substitutions as a byproduct of the variation in win probability. Our in-game win probability model, based on the one developed by Robberechts, Van Haaren and Davis [29], proved to be sensitive to substitutions, as seen in Figure 4.3. Our change in the model, trying to predict future goals and then simulating the match, instead of predicting result percentages, turned out to be effective in having predictions of all possible percentages for all results. Similarly, as in Robberechts et al., the Bayesian approach with the Gaussian Walk has resulted as the most effective, having a high relation between predicted and actual probability. One aspect to discuss about the win probability model is the effect that is shown that substitutions have. Most of the substitutions have a small impact, both for positive and negative outcomes. In actual matches, a substitution can have a much bigger effect, since the player that comes in might be especially inspired and have an above-average finishing, or on the contrary have a bad day. The model averages the effect of the substitutions given a game state. It does not account for other contextual features that might have an impact on the development of the game, such as the psychological aspects of a substitution. In general, though, the impact of a player replacement is usually beneficial, as seen from Figure 4.4. 35 Chapter 5 Substitute Selection A natural question a coach may come up with is which player should be introduced. Research question 3 is the formulation of such doubt and this chapter aims to provide an answer. With the new regulations, in many of the leagues, the number of players on the bench has now increased, though it is different for each country. Deciding which player comes into the match depends mostly on the available resources. Two players are involved in a substitution: the player on the field that is removed, and a bench player who is introduced. Deciding on which player should leave the pitch is a very interesting topic. Unfortunately, we cannot address it since we are working with public and widely available data, that does not include event-by-event, therefore, a detailed analysis of the player’s performance through the game cannot be made, making it difficult to properly assess which player is having a below par match. Moreover, many other factors, such as physical form or the tactical needs of the team need to be taken into account. Of course, some of these problems arise as well when selecting the bench player that will participate in the match. Physical problems are not a concern for substitutes, professionals can play a match as bench players, and have been proven to cover more spaces and do more sprints than starting players [4]. We give a data-based approach to substitute selection, even though tactical knowledge and specific onmatch situations cannot be captured into data are still important. 5.1 Bench Analysis For the purpose of simulations and proper substitute selection, we need to know which players are on the bench. On a general basis, if we want to decide between various possible types of substitutions, knowing a team’s formation and the available players is very important. In this section, we aim to provide some ground rules for simulation. In the 2020/21 and 2021/22 seasons, teams could have up to 12 players on the bench for the Spanish LaLiga, the Italian Serie A and the Dutch Eredivisie; but only up to 36 (a) Goalkeepers Histogram (b) Defenders Histogram (c) Midfielders Histogram (d) Forwards Histogram Figure 5.1: Histograms of the distribution of each position, separated by maximum bench size. 9 in the German Bundesliga, the French Ligue 1 and the Portuguese Primeira Liga. Thus, our analysis will have to be dual. We first start by seeing which is the typical distribution of a bench to see which assets a coach has at their disposal. Figure 5.1 shows the distribution of the main positions a player can be: goalkeeper, defender, midfielder or forward. A few observations about the distribution of players. In Figure 5.1a we see that, while in smaller benches teams mainly have just one bench goalkeeper, when twelve spots are available, usually two goalkeepers stand at the bench. Substitute goalkeepers rarely come into play (roughly 0.3%), and needing two extra goalkeepers is unheardof, but teams rather have them both on the bench, possibly due to not needing 11 substitute field players. In Figure 5.1c we observe that some benches do not have midfielders at all. While this sounds odd, some teams might play their midfielders in other positions, since they are the most versatile, or plan on playing in midfield a player which the data providers tag as either defender or forward. Additionally, through the season squads get affected by injuries and the number of players initially planned could not be available in certain matches 37 Selecting adequate bench players is key to the coach’s success because it defines their tools during the match, the pieces they can fit into the playing XI to change the game dynamics. Beal et al. [3], develop a formal model for a pre-Match Bayesian game. Teams have different tactics, which are the formations that they use, and a multi-class classification deep neural network is trained to learn the payoffs of the different strategies. In such a way, they try to predict the opposition’s strategy and suggested an optimal tactic. The results showed teams with tactics similar to the suggested ones achieved better results. The pre-match modeling of the game is an area where bench management could be an important asset. Although we were unable to explore the idea in this report, we believe it is an important area for future research. Including the information on the opposition’s bench could improve the model by giving more valuable information. In this analysis, we aim to give an approximate idea of the bench composition for accurate and real simulations. 12-Player Bench 9-Player Bench Formation Gk Def Mid For Gk Def Mid For 4-5-1 2 3 3 4 1 3 2 3 4-4-2 2 4 2 4 1 3 2 3 4-3-3 2 3 3 4 1 3 2 3 3-5-2 2 4 3 3 1 2 2 4 3-4-3 2 4 2 4 1 2 2 4 5-3-2 2 3 3 4 1 2 2 4 Table 5.1: Number of players per position in the bench, separated by the two bench sizes. In Table 5.1 we have summed up the bench that we will be working on during the simulations, based on the averages of players from such position and such that they are a full-sized bench. The types of formations are ordered by appearances on our dataset, and all types of formations that appear more than 50 times can be included in one such formation. Formations are defined by the number of defenders, midfielders and forwards separated by a dash. This is a simplification of the previously defined positions in Table 3.2, so multiple tactical approaches are lost. For example, both the 4-2-3-1 and 4-1-4-1, two widely used formations, are packed under 4-5-1, even though they might have more players defined as forwards. Following the example of [2], we will assume that on the pitch we can have a minimum of 3 and a maximum of 5 defenders, same with midfielders, and 1 up to 3 attackers. Even though the number of forwards on the pitch is smaller, we know that most of the substitutions involve attacking players [4] so coaches want to have more options in these positions. These constraints will be applied for the simulation of full matches. 38 5.2 Type of Substitution We have classified substitutions as offensive, defensive or neutral, according to Definition 3.8, depending if there is a reasonable increase or decrease in the offensiveness on the position of the incoming player. We can keep the moment a coach decides to make a substitution constant. For this, we will assume that no substitutions are due to an injury, and hence require a similar player as a replacement. During chapter 4 we studied different ways to evaluate the effectiveness of a substitution. We are going to use those same models to evaluate if another type of substitution could have been better according to the models. For each window substitution, we have simulated the expected result if those substitutions had had another nature. For offensive substitutions, we simulate them being neutral or defensive, and evaluate what the performance would have been. Similarly, we have done the same for neutral and defensive substitutions. The wide spectrum of targets defined in subsection 4.1.1, multiplies the opportunities we have here, so we are just going to explore the options we find more interesting. A real-life application of this kind of model could be for coaches, to decide the type of substitution they have to do or to be data-informed as to which is more likely to accomplish their goals. Additional models, that not only differentiate between offensive and defensive substitutions but also incorporate other factors, have the potential to equip technical staff with a broader range of tools for conducting simulations. 5.2.1 Win probability simulations Simulations with the in-game win probability model, detailed in section 4.3, not only allow us to see if certain simulated substitutions would be successful but since we work with Expected Points (Definition 4.7), we are provided with a numerical weighing of the effect of changing the type of player swap. For each substitution, we have simulated that modification in an offensive, neutral and defensive way. The Bayesian win probability model is applied to each generated game state. This is, given a game state xt,i = ( t, i, τt,i, st,i ) we modify st,i , so we have st,i,off , st,i,neu and st,i,def . Similarly, for that same t , we modify the game state of the opposing team i∗ into xt,i∗ = ( t, i∗, τt,i∗, st,i∗,off/neu/def ), depending on the case. Thus, for every case, we get three pairs for each case: ( θt,i,off, θt,i∗,off ) for the offensive substitution state, ( θt,i,neu, θt,i∗,neu ) for the neutral one and ( θt,i,def, θt,i∗,def ) for defensive. With the definition of gt,i ∼B ( t, θt,i ) and adding the current scoreline ( Gt,i, Gt,i∗ ), we simulate and get the home win, draw and away win probabilities. Note that this analysis is independent of which of the nature of the substitutions. For either offensive, defensive or neutral substitution, we simulate the three states. 39 Figure 5.2: Variation in the amount of Expected Points. x-axis is square root scaled. Separation between the comparison of offensive substitutions with respect to neutral ones (blue) and negative ones with respect to neutral (red). The graphical representation of the simulation results can be observed in Figure 5.2. From the previous simulations we obtain EPoff , EPneu and EPdef . In blue, we have the distribution of ∆ EPoff = EPoff −EPneu , and in red we have ∆ EPdef = EPdef −EPneu . The x-Axis has been rescaled by a square root transformation for better visualization, but maintaining the sign of the change, so the amount plotted is actually sgn ( EP ) √∆EP . If we plot simply ∆ EP the curve is centered and spiked. This is due to 60.3% of the substitutions having a ∆ EP of less than 0.03, which corresponds to less than 1% change in EP since we work with a 3-point system. This is important since we see that almost two of each five substitutions could have a significant displacement in terms of win probability if the proper type of substitution is taken. In Figure 5.2 we see that, on the one hand, the blue curve is displaced to the right, which means that, in general, an offensive substitution gives you better odds of winning than a neutral one. On the other hand, the red curve is slightly displaced to the left, giving you worse win probability percentages. This does not mean it is always the case since we see a big part of the ∆ EPoff curve is negative, giving you worse chances than a neutral sub, and similarly with the ∆ EPdef in the positive changes. Looking at the numbers, we see that over 20% of the substitutions could have very significant, more than 0.3 expected points, swing in expected points. This means that, by properly selecting the substitution type, one of every five substitutions could have a real impact on the match and the winning probability. 40 obtain a win probability advantage. We do notice that the model is more sensitive to substitutions early in the second-half. Figure 6.3: Distribution of the variation of EP after a substitution. Distribution at each minute. 6.2 Simulation of Alternative Timing In this section we will study the alternative timing for the substitutions coaches did during the studied matches. This is, for each substitution that had been made, we are simulating moving it forward and backward through time. We have respected the coaches’ choices in terms of the type of substitution and the different substitutions windows used, so if a team made the first two changes at the same time and then two isolated ones, we have also done so in the simulations, while also including a five-minute minimum space between the simulated substitutions, since it is no real scenario wasting two windows in two consecutive player changes, except for extraordinary cases such as an injury, a red card, or trying to lose time. Simulations have followed the same structure for our two types of models. Knowing how the substitutions were packed into windows and their offensive or defensive nature, we were able to generate new game states xt,i by varying the substitution information. This applied both to the classification models, which had the updated state input, and the win probability model, in a similar way to the simulations previously explained in this thesis. 6.2.1 Substitution classification models After generating the adequate game states for simulation, we applied the trained substitution classification models. We grouped substitutions according to their 47 proximity to the real match case, and the results are shown in Table 6.1. Time groupings are not homogeneous because we want to remark the minutes closer to the real substitution. We show the change in the percentage of positive or successful substitutions. This is, offensive substitutions, when simulated from 6 to 10 minutes prior to their real minute, obtained better predictions 3.6% of the time. Change in positive/successful substitutions (%) Time to real sub [-20,-11] [-10,-6] [-5,-1] [1,5] [6,10] [11,20] Offensive Subs +8.2 +3.6 +0.4 -0.2 -1.3 -6.5 Defensive Subs -7.6 -3.4 -1.2 +0.9 +2.5 +7.3 2C-SER +3.2 +2.5 +0.4 +0.2 -0.1 -0.4 3C-SER +1.5 +0.7 0.0 -0.2 -0.7 -1.1 xV-Change Rule +10.2 +6.1 +1.4 -2.1 -4.2 -8.3 Table 6.1: Results of timing simulations with the classification models. Results grouped by different assessment techniques and time with respect to the real substitution. Numbers represent the variation in the percentage of positive or successful substitutions. Negative numbers represent a decrease in the number of successful outcomes Results in Table 6.1 are in line with most of the other results. Offensive substitutions have a higher probability of success if done earlier, while defensive ones behave the other way around: success increases as they are done later in the match. This agrees with the results in Figure 6.2, and the main reasons are the same: time left to achieve the goal. The Scoreline Evaluation Rule, both for the two and three-class problems, seems to be a bit better when substitutions are done earlier than in real life, but the numbers are very small and not very significant. The biggest results are for the xV Change Rule, which compared the incoming player performance to the outgoing one. We see that having more time as a substitute is correlated with a higher probability of a successful substitution. 6.2.2 Win probability model From Figure 6.3, we observed that there is no apparent general better moment to do a substitution, but on a single match we have observed significant changes in the win probabilities. Thus we simulate for each substitution an alternative moment. The results, in general, agree that substitutions should be done, if anything, previously to the moment when they happen in reality. In Figure 6.4 we have plotted the distribution of ∆ EP depending on the time previous to which the substitution took place. The biggest changes are when substitutions are made 30 minutes prior to their real-time, which practically could never happen. 48 Figure 6.4: Distribution of the variation of EP depending on the time previous to the real substitution Again, substitutions in a match are related, so considering their relations is relevant. If we apply it to the Levante 3-3 Real Madrid match we discussed earlier, whose expected points evolution is in Figure 5.3, we can see that Madrid could have started their comeback before, according to the model. This is due to Madrid having a higher Elo rating, so the model gives a higher goal-scoring probability by doing the substitutions, so it suggests doing it before and thus enhancing the winning probabilities. On the other hand, for Levante, it shows very little change when just changing the timing. Rey et al. [28] showed that losing teams substitute before while leading teams tend to delay changes, and our analysis suggests it is the optimal strategy. While the previous match may have been more frenetic and high-scoring, we can focus also on a quieter and more measured game, which now presents an equally valuable opportunity for analysis. In fact, by studying the strategies and tactics used in a more low-key setting, we may be able to uncover new insights and approaches that could prove useful in future matches. This is the case for Osasuna 0-0 Espanyol, played also at the start if the 2021/22 LaLiga. Both teams had very similar Elo ratings, and it was a close goal-less match. Espanyol made a half-time sub, and then a double substitution at the 63 rd minute, after which they had 1.07 EP. Had they done the substitution before, the EP after the substitution would have been 0.02 EP above the value they got in the real match. When arrived to the 63 rd minute, the simulation gives the same value, since the game state xt,i is the same. The fact is, during the minutes the substitution had been done, Espanyol had a higher win 49 probability. When we make this double substitution later in the match, Espanyol’s EP goes lower while Osasuna’s goes up. In that match, Osasuna just made three changes. In this particular case, the variation with time of those substitutions has almost no effect, less than 0.01 EP on the winning probabilities. What simulations do say is that, if those changes had been offensive, a real change could have been seen, as we discussed in section 5.2. Had Osasuna’s coach decided to introduce a fourth and fifth bench player into the match, their EP would have gone down by close to 0.02 EP, so just staying with the players on the field is the decision the model would have suggested. 6.3 Results and Discussion Timing of substitutions is key. Most of the substitutions occur between the 60 th and the 85 th minute [4], in our case we found that 65.3% of the substitutions were on that time period. Early substitutions showed a higher tendency to turn the result into a desired one, while late substitutions have a higher percentage of failure. By bringing fresh legs and new tactics onto the field, coaches have the opportunity to inject energy and creativity into their team’s performance, potentially shifting the momentum in their favor. Even though generated chances, measured by xG, and actual goal scoring are not directly related, the analysis for both cases is very similar. The simulations with alternative timing on the Scoreline Evaluation Rule resulted in no significant change. The distribution in time of success for offensive and defensive substitutions is the logical consequence of their definition. Since we need a goal to consider the offensive substitution as accomplished, the probability is higher if the change is done earlier. Similarly, the probability of keeping a clean sheet after a defensive change increases the less time the opposition has for goal-scoring. Moreover, the results align with the variation of goal-scoring intensity by Amez et al. [1]. Similarly, the number of inconsequential substitutions, those that do not vary the result, increases as the final whistle gets closer. Simulations with alternative timing gave according to results, both for offensive and defensive substitutions. Offensive and defensive substitutions, which we defined as a change in the position occupied by the players has an effect on the centroid of the whole team shifting [23]. In subsection 3.2.2, especially captured in Figure 3.3, we discussed how substitutes tend to produce more expected value, which makes a logical decision to advance an offensive substitution if a player is performing poorer. While defensive substitutions have higher chances of being successful if done later in the match, it does not mean that the optimal strategy is such. Teams that are winning, though, have been observed to tend to delay their substitutions [28]. The in-game win probability model seems to have a higher sensibility towards early 50 substitutions. This is logical due to the effect of the substitutions being able to have repercussions in the game, both for good and bad substitutions. This result, together with the analysis of the different assessments, points towards the same direction: early substitutions have a higher impact on the game. The higher rate of xV per minute of substitutes is one of the reasons we obtain a tendency to better results for earlier substitutions. In the first stage of the investigation, no constraints were included, and the win probability model suggested the winning combination was to burn all substitutions as soon as possible. This, with the proper amount of information, does not make sense, since playing over 30 minutes with no substitutions left can be harmful to injury and fatigue wise. Since teams never use this strategy, and the last substitutions are done very late into the game ([4], [28], [14]), the model has not learned that it could be a problem. Adding fresh players does increase your goal-scoring probabilities, so the model suggests adding even more fresh players into the pitch. When doing time simulations with the xV Change Rule we obtained that doing earlier the substitutions resulted in more success. This means the incoming players are expected to outperform their colleagues if enough time is given to them because bench players outperform starters on average. Simulations on complete matches exposed the problem of the model suggesting too early substitutions. We could not find any evidence of Myers’ substitution rule [26] as an optimal strategy. Even though we have now five substitutions and study a different dataset of matches, the problems discussed by Silva and Swartz [31] still apply: similar strategies are equally useful and some bad patterns can bias the analysis. 51 Chapter 7 Conclusions Substitutions are the main asset a coach has to directly influence the game. By substituting players on the field with fresher bench players, clubs’ technical staff are able to introduce new tactics, make more adjustments and possibly change momentum. The objective of this thesis is to study optimal substitutions through the use of the data. In order to do so, we studied the substitution problem from different points of view. Following the COVID pandemic, the majority of leagues introduced a fourth and fifth substitution for each team, as a temporary measure for the big amount of matches played in a small period of time. Some leagues decided to keep these extra substitutions, and this now seems to be the new rule of international top-tier football. Through the analysis of the European male top leagues, we responded to Research question 1, in which way did the introduction of the additional substitutions affect the coaches’ maneuverability and the relative importance of substitutions. The results show how clubs throughout Europe have needed a period of adjustment. The current season, 2022/23, is the third complete season where the 5 substitutions per match rule is valid. The number of actual player changes done by a team has increased significantly speaking since this rule was introduced and now seems to be stable, with 4.35 substitutions per team per game in average. Even though substitutions have increased, the percentage of matches that changed results after the second-half, which is the one affected by substitutions, has remained the same to pre-COVID era. The percentage of matches that are drawn at half-time and finish with a winner has also remained very similar. This is the result of teams evolving to continue to be competitive with each other, and we cannot see an effect of the increase of substitutions in match scorelines. We defined substitutions as offensive or defensive depending on the positions the incoming and outgoing players occupy. If the positions are distinct, it means that the substitution has a tactical component, and the behavior of the team is changed [23]. Of course, same position substitutions can be due to tactical reasons or simply 52 fatigue, injuries or bookings, but due to the data we are working with, we have no way to know. When the new 5-subs rule was introduced, coaches maintained their number of non-neutral substitutions per match in absolute value, which by the second season had been raised, equaling the percentage of almost 1 of every 4 substitutions being non-neutral. This is further proof of the adaptation process teams have taken since the COVID outbreak, and how the new 5-subs rule has changed the tactical landscape, with non-neutral substitutions increasing by a 50% in absolute numbers. During the whole thesis, we have worked with public data available for many leagues, not only European male top-tier competitions, so that this analysis could be assimilated into other leagues. The counterpart is the lack of data, which in this case means many substitutions have a tactical effect we cannot see through the data. Having a better description of the substitutions would be a great asset to study how the change between 3-subs per match and 5 has changed the coaches’ tactics. In particular, having event-by-event data is the main point of improvement of this thesis and throughout the conclusions will be mentioned multiple times. Moving forward, we explored Research question 2, where we focus on the evaluation of substitutions. The aim of the thesis is to identify optimal substitutions, so first we needed to state what we define as a successful one. We based our assessment on three points of view: the scoreboard and its evolution, the goal-scoring chances and the players’ performance. All three ways of assessing substitutions are correlated, but on a particular game or substitution the assessments can be distinct. Also, for all the perspectives we looked at the problem as a two-class problem, where substitutions are either positive or negative, and a three-class problem where we introduced inconsequential tags to the substitution without a sufficiently relevant effect. Once we had the rules where we defined what we consider as successful, we trained different machine learning models, mainly k Nearest Neighbors, Support Vector Machines and Random Forest classifiers. The results were close, but relatively worse, than the ones obtained by Brutti et al. in a similar study [7], even though methodologies are slightly different and use distinct data. The best predictions were obtained for the three-class problems, due to accuracy being higher for the amount of inconsequential substitutions. We did not find any significant difference between the algorithms used to predict the classification of substitutions. The second way in which we assessed substitutions was as a byproduct of an in-game win probability model, based on the idea by Robberechts et al. [29]. Our model predicts, given a game state, the probability of scoring a goal and simulates the rest of the match. Of the different techniques tried for the prediction of goal scoring, the one that resulted in more accurate predictions was the Gaussian Walk, as the Bayesian approach really transfers information between time frames. The validation curves shown by the other methods, the Multiple Logistic Regressors, Random 53 Forest and Multiple Random Forest were worse than the Bayesian method but still achieved a reasonable level of accurate predictions. From this model, we studied how the win, tie and loss probabilities changed with the substitutions. While comparing to a model without the substitution information, we proved that player modifications are relevant to predicted goal scoring and, therefore, winning probability. We also observed that substitutions tend, in a majority, to give a boost to the teams’ performance, given that the majority of the changes produced a positive swing in expected points. These two main models, the classifiers and the win probability, are the base of the thesis. Classification models could be improved with more information about teams, such as their offensive and defensive performances, or a more detailed game state. The win probability model has a main point of improvement, which is that due to the observed higher performance of substitutes, it usually suggests more substitutions to be done earlier. Teams always wait until the end of the match to finish their substitution possibilities, for eventual red cards or injuries. The model has not learned the consequences of having the substitutions done so early, because there are no such cases in the training set. Turning to Research question 3, we investigated the problem of knowing the expected performance of the bench players. Before proceeding to the simulations, we did an analysis of the distributions of the bench, depending on the number of allowed players and the starting formation. This allows us to make a general analysis of the substitutions without depending on each particular bench from each match. We have worked with offensive, neutral and defensive substitutions for the type of substitution in the simulations, which are proven to modify the team’s tactics [23]. For every substitution that happened in a match, we simulated as if it was any of the three options, using the win probability model to obtain a numeric value of the impact of the substitution. Having access to more detailed data, such as the event-by-event or tracking data, would allow these same models to introduce more features and different kinds of substitutions. If a similar model were to be implemented for real use, coaches would like a wider range for the types of available substitutions, with more tactical nuances than a simple offensive-defensive choice. The results of the simulations were interesting. First, we saw that the change in expected points for offensive substitutions with respect to neutral was in majority positive. Instead, doing the substitution defensively gives, on average, a worse EP value than neutral. This means that the model tends to better value the offensive substitutions. But we saw that defensive changes can actually induce a positive swing in EP. It was the case of the Levante - Real Madrid match, where a lower Elo rated team, Levante, with a favorable result, the model suggested that a defensive player swap was the correct option. Overall, the results of the simulations showed 54 that a reasonable amount of substitutions can have a significant change in their expected points value depending on the type of substitution. The selection of a substitute has many improvement ways. The expected performance of a player is an analysis of its own, and in this thesis could only be theorized and very small worked with. Many metrics from extended seasons can be taken into account to properly assess the player, and football clubs could even include training data which is evidently never public. Moreover, studying relationships between teammates on the football pitch [6] can also be a powerful asset to identify optimal substitutions. The creation of a Bayesian model, where every piece of information about a player can be inputted for a better performance prediction, is a promising way to start the investigation on bench players’ achievement forecast. Finally, we addressed Research question 4 and dealt with the timing problem. We moved the substitutions through the time, respecting the windows on which they were made, and applying reasonable constraints for the simulations. The generated game states were fed into the models, both the classifiers and the win probability. The result for offensive substitutions was that it was better to make them as soon as possible, which is logical since it gives the team more time to score a goal. In a similar way, when assessing substitutions by xV of the player, early substitutions are much more effective that the end of the match. Reversely, the defensive substitutions are proven more effective in the latest part of the match, since the opposition has less time to score. These findings are consistent with other analyses of goal-scoring frequencies and their correlation with substitutions [1]. Obviously, the results are a consequence of the definition. While bench players generate more xV and is logical to substitute a player performing poorly, it is not clear that delaying defensive substitutions is the optimal strategy, even though is what teams do [28]. When studying the substitutions with respect to the Scoreline Evaluation Rules, we see that positive subs happen before, in general, than negative ones. But when doing the simulations, we did not find a very significant change in the amount of positive and negative substitutions, neither for the two-class or three-class problem. Similarly, the simulations with the win probability model did not show much variation in time. We did observe that the substitutions that have a higher effect on the winning probabilities are the ones made early in the second-half and at half-time. These results align with the study by Silva and Swartz [31], where they found no apparent better moment to make the substitution, contradicting Myers’ [26] timing rule. While with the type of substitution we obtained a high proportion of substitutions with a relevant change in expected points, with the timing the results are much more restrained. In some cases, we obtain that doing the substitution earlier can provide the team with a higher winning probability during a few minutes, but the increments in EP are lower than with the type of substitute. 55 During the thesis we have worked separately on two different parts of the substitution problem: the incoming player and the time at which they do. A complete system of data-driven forecast of substitutions, as future work on this problem, should include a coordination of those two problems, and also the decision of the outgoing player. With access to event-by-event or tracking data, a more profound analysis can be done of a player’s actions ([32], [11]) and their effect on the match, while joining with physical information through GPS [36] or multi-camera tracking systems ([8], [5], [4]), the system would be able to provide with a more clear view of the under-performers of the game. Such a complete model could implement various ideas: trying to predict the decline in physical performance, identifying defensive under-performance from an opposition high value. Such information, adequately combined with a timing analysis and a proper substitute selection, could provide coaches and their staff with a powerful data-informed tool to gain a competitive advantage and to know the best databased decisions. The key idea is to weave all the analysis for more real information about the problem. In this thesis, we simulated alternative substitute selection and alternative timing separately. Being able to introduce simulations of the outgoing player and combining the three factors could lead to a well-functioning complete model for forecasting and identification of optimal substitutions in soccer. In this thesis we developed two types of models: classification models and a substitute sensible win probability model. We have used them to study the most beneficial substitutions and simulate alternative settings for better performance. We saw that coaches can definitely use substitutions to change the course of matches and that by selecting mostly the appropriate type of substitution can obtain a big increase in their win probabilities. Timing is less important numerically wise, but can play a relevant role in some cases. In conclusion, this thesis has provided a comprehensive analysis of substitutions in soccer, exploring various approaches and methods for evaluating their impact on the game. Through our investigation, we have studied some factors that influence the effectiveness of substitutions, including the timing of the substitution and the tactical approach of the team from the type of substitution they choose. Our findings provide insights that can inform decision-making processes for coaches and analysts alike. Moving forward, it is clear that further research is needed to fully understand the complexities of substitutions in soccer and to develop more sophisticated methods for evaluating their impact. Nonetheless, we believe that this thesis represents a valuable contribution to the field of football analytics, and offers a solid foundation for future work on the substitution problem. 56