Contextual multi-armed bandits for non-stationary wireless network selection

Contextual Multi-Armed Bandits for Non-Stationary Wireless Network Selection Llu´ ıs Mart´ ınez, Josep Vidal and Margarita Cabrera-Bean Dept. of Signal Theory and Communications, Universitat Polit` ecnica de Catalunya, Barcelona, Spain Email: {lluis.martinez.casanovas, josep.vidal, marga.cabrera}@upc.edu Abstract—As the number of wireless technologies (4G, 5G, 802.11ax and other) have been rapidly increasing, so have the number of wireless networks concurrently deployed on a given coverage area. As a result, the judicious selection of the network that maximizes the quality perceived by a user terminal has become a significantly relevant problem. Contextual Multi- Armed Bandits (CMAB) are viable models to approach the problem. While multiple CMAB algorithms have been designed, most of them are only suited for stationary environments. This work proposes a new set of network selection algorithms that relate the traffic type (used as contextual information) to the perceived quality of the available networks for non-stationary scenarios. Results show significantly improved performance when compared to non-adaptive approaches. Index Terms—Network Selection, Multi-Armed Bandit, Nonstationarity I. INTRODUCTION The appearance of wireless technologies has eased the access to the Internet to a large portion of the world’s population. As more and more standards are developed and deployed (LTE, 5G and many new Wi-Fi versions), more wireless technologies are in reach of any user terminal at a certain location. This growth in number of technologies entails the opportunity for user terminals to choose the best network to connect to. Making the choice is particularly challenging in this context, where the quality of each network may rapidly change due to user movement, the number of connected devices, the backhaul load, etc. The request of a certain service (and hence traffic type, such as texting or video streaming) is relevant in the selection because each traffic type is more or less dependent on some specific network parameters. The amount of studies focused in solving this problem has been increasing in recent years [1] [2]. Some have shown the effectiveness of using Multi-Armed Bandits (MAB) [3] [19]. In the MAB model, at each time step tan agent chooses an action (so called arm a) from a set Aof possible actions. When an arm ais selected, a reward rassociated to the quality of the selection is obtained. The goal of MAB algorithms is to make the adequate decisions in order to maximize the sum of rewards over time, by following a policy that combines exploration of qualities of all the arms and exploitation of the knowledge acquired so far [4]. This work has been funded through the project ROUTE56 (Agencia Estatal de Investigaci´ on, Ministerio de Ciencia e Innovaci´ on, PID2019-104945GB- I00/ AEI / 10.13039/501100011033) and the grants 22CO1/008248 and 2021 SGR 01033 (AGAUR, Generalitat de Catalunya). In Contextual Multi-Armed Bandits (CMAB) a contextual information is provided to the agent before making the choice of an arm. In our application to wireless network selection, each arm represents a wireless network the agent can connect to. The rewards rgenerated are a measure of the overall perceived quality of the selected network. The contextual information is a set of measurable parameters that link the type of service and the reward. This makes CMAB algorithms more complex than regular MAB algorithms, as they need to keep track of the relationship between the context and the reward. CMAB models have been successfully applied in many other fields, e.g. medicine, financial investment, etc. [6]. Most MAB and CMAB algorithms are designed for stationary environments, where the mean reward of arms does not change over time. However, not all situations where the MAB paradigm may be useful are stationary, so in recent years new non-stationary algorithms have been developed. In [7], a nonstationary CMAB algorithm is presented. Its strategy consists in detecting non-stationarity by randomly entering into replay phases, where it constantly monitors the rewards of an arm and it compares this figure with its current estimate. If the current estimate is far away from the observed rewards, it is identified as a likely change of the mean reward of the arm. On another work, [8] proposes to keep multiple ”slave” models with different strategies at the same time. The ”master model” will use the most promising model at each step. The main drawback of both approaches is that the hyperparameters associated to non-stationarity are static and defined in advance, whilst the optimal value depend on the behavior of the nonstationarity (e.g. how fast and often the environment changes) which is unknown beforehand and may vary over time. A third approach [20] adapts the exploration parameter of ϵ-greedy MAB as a function of the temporal-difference error observed from value functions, which is considered as a measure of the agent’s uncertainty about the environment. Based on this later approach and on our previous work [9], we propose an adaptive algorithm for a CMAB. More specifically, our contributions are twofold. First, we adopt a polinomial CMAB framework for the best wireless network selection by using the parameters of the traffic type as context. Second, we propose adaptive CMAB algorithms for non-stationary scenarios by modifying the well-known linear UCB1 (LinUCB1) [5]. The CMAB model is discussed in section II and a procedure for stabilization of the LinUCB1 algorithm is proposed. In © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. https://dx.doi.org/10.1109/GLOBECOM54140.2023.10437363 sections III and IV, the polinomial model and the adaptive solution proposed are explained respectively. Test results in realistic environments are discussed in section V. Finally, some conclusions are drawn in section VI. II. CONTEXTUAL MULTI-ARMED BANDITS A. The contextual model In the CMAB model, at every time step twe have to make a choice among kdifferent actions or arms from an action set A. At each t, and before taking the decision, the agent also receives extra information (the context) of the state. This context is represented by a vector ϕsof size dthat represents one among the possible contexts S. It is important to remark that decisions are independent. Every time an action ais chosen, the environment generates a reward that is characterized by a probability density function related to the selected action and context: r∼f(r|S =s, A=a). The goal is to choose the sequence of actions that maximize the cumulative reward over time. In CMAB problems, this is usually done by estimating the relationship between the observed rewards and contexts. The mean reward for arm a and context sis defined as: Qs,a =E{r|S =s, A=a}.(1) In MAB problems, the total regret measures performance. The regret is the expectation of the difference between the reward obtained when taking the best possible action r∗ tand the reward of the action actually chosen, rt, at a specific time step t: lt=E{r∗ t−rt},(2) The total regret is defined as the sum of regrets over time: Rt= t X τ=1 lτ.(3) For non-stationary scenarios, the term dynamic regret is used instead, as the optimal arm can change at each time step t. B. QoE used as reward In the case of the wireless network selection problem, each arm is a possible network to connect to and the agents taking decisions may run in a user terminal. In our work, the values of rewards rtare Mean Opinion Scores (MOS), which measure the Quality of Experience (QoE) perceived by the user. In general, the QoE has to be mapped from objective measurements observed in the flow, like throughput (TH), delay (DL) and packet loss rate (LR), that provide the Quality of Service (QoS). Mapping functions were derived in [16] for 11 traffic types such as live video, instant message, meeting video, etc. according to the following expressions: QoS =b·exp(c·DL) + g LR +h+p·log(TH) + q(4) r=QoE =< α·exp(−β·QoS) + γ > (5) where <·>is the rounding function and the values of b, c, g, h, p, q, α, β and γare characteristic of every traffic type [16] and of every radio access technology, and are assumed to be known by the user. The model behaves in such a way that the higher is the throughput and the lower is the loss and delay generated by a network, the higher is the QoE (that is the reward r) generated by the action of choosing such network. As it was proposed in our previous work [9], the bitflow parameter values (T H,DL and LR) used in (4) and (5) are generated from different realistic random distribution functions designed by observing and fitting multiple histograms of real LTE traffic from various LTE datasets. See section V.A for further details. C. Linear contextual multi-armed bandits Contextual algorithms need to establish a model of the relationship between the context and the reward. In recent studies such as [11], Bayesian Neural Networks (BNN) are used to model this relationship. This BNNs output an estimation of the reward for an inputted arm and context. Despite this is a very promising approach, it involves heavy computations and long training periods for the coefficients of the neural network. A simpler and less accurate approach for contextual algorithms is to define a linear relationship between the context and the reward. It is assumed that the reward generation at step tfor context sand arm afollows the equation: ra,t =ϕT stθa,+wt(6) where ϕstis the column vector for context sat step tand θa is a vector of dparameters for arm athat relate the context and the reward. wtis a zero-mean random disturbance. In our wireless network selection application, ϕstcontains the values of the coefficients that map objective measurementes to QoE (b, c, g, h, p, q, α, β, γ) in (4)(5) for the traffic type at hand. Taking this model into account, the predicted action value given the context is: ˆ Qa,t =ϕT st ˆ θa,t.(7) With this model, agents have to estimate ˆ θa,t for each action before making a decision. Given ϕst, associated to the traffic type, and the estimated ˆ θa,t, agents are able to decide on the most convenient arm. Estimation using least squares over t observations departs from the definitions: Aa,t =ϕs1ϕs2. . . ϕstTra,t =ra,1ra,2. . . ra,tT,(8) where Aa,t gathers the contexts when arm awas selected and ra,t contains the rewards in those cases (and hence the number of rows of Aa,t is different for every a). The least squares solution for θais: min θa ||Wt(ra,t −Aa,tθa)||2(9) ˆ θa,t = (AT a,tWT tWtAa,t)−1AT a,tWT tWtra t(10) where Wt=diag(λt, λt−1,...,1) contains the forgetting factor λ∈(0,1] that accounts for the system memory and becomes useful for non-stationary scenarios. This can be interpreted as a linear contextual counterpart of the discounted- UCB algorithm. In [18] (see also references therein) it is shown that its non-contextual version is almost optimal as its regret nearly matches a lower-bound. Using (10), an algorithm could be defined to make decisions depending on the context. It is worth noting that the inverse of (AT a,tWT tWtAa,t)−1 only exists after the agent has selected different types of traffic. In the meantime, the following update is used: (AT a,tWT tWtAa,t +δI)−1, where δis a positive small value. This formulation has one critical flaw: it deals with matrices and vectors whose size grows over time. The issue can be solved by resorting to the Sherman-Morrison (SM) formula [15] for the recursive computation of the inverse, allowing fixed-sized matrices and vectors instead: Ba,t = (AT a,tWT tWtAa,t)−1(11) za,t =AT a,tWT tWtra,t (12) These matrices and vectors have the same size as ϕst,a and can be updated in a very efficient way: Ba,t =1 λ2Ba,t−1−Ba,t−1ϕstϕT stBa,t−1 λ2+ϕT stBa,t−1ϕst(13) za,t =λ2za,t−1+ϕstra,t (14) with this formulation, ˆ θa,t is computed as: ˆ θa,t =Ba,tza,t (15) Like in the classical MAB, a trade-off between exploitation and exploration can be defined. When exploiting, the arm a with the highest estimated mean reward ˆ Qa,t is selected by the agent. On the other hand, exploring entails obtaining more accurate estimates for all actions by choosing non-optimal arms, so that better decisions can be made in the future. When exploring, it is assumed that the user terminal connects to different networks each time it wants to establish a flow, following some exploratory policy. In the beginning it has little knowledge about the best network, but over time this knowledge improves through the estimation of the best values for θa. These handovers should be done with a periodicity that is below the coherence time of the network characteristics. How the exploitation and exploration trade-off is managed is key in MAB algorithms. The Upper-Confidence-Bound action selection 1 (UCB1) algorithm [12] proposes to compute an uncertainty interval to ˆ Q(s, a)that will decrease as more information about arm ais gathered. Best arm is the one with the highest estimated mean ˆ Q(s, a)plus its associated uncertainty interval. The UCB1 algorithm can be adapted for the contextual MAB with a linear model, thus obtaining the so called LinUCB1 (we will call it in section V as SM Linear Contextual UCB1, and use it with λ= 1). The way this algorithm selects an arm is shown in (16): at+1 = argmax aˆ Qa,t +qαln(t)ϕT stBa,tϕst(16) where αcontrols the level of exploration. D. Stable unbiased linear contextual multi-armed bandits It is well known that the SM-based update algorithm in (13) may yield a Ba,t that is no longer positive semi-definite due to round-off errors, specially when λ < 1. Then, the iterative algorithm becomes unstable. A possible solution adopted in [13] is to apply ridge regression and regularize Ba,t by summing an identity matrix, at the expenses of biasing the estimated ˆ Q(s, a)[5]. Alternatively, we can resort to the QR Decomposition (QRD)-based least squares solution [14]. The error minimization problem can be recast as follows: min θa ||ra,t −Aa,tθa||2= min θa ||QT a,t(ra,t −Aa,tθa)||2(17) where Qa,t is a unitary matrix. In particular, we are interested in Qa,t such that Aa,t =Qa,tRa,t where Ra,t is an uppertriangular matrix. In this way, the least squares estimate ˆ θa,t at each time step is: ˆ θa,t = (AT a,tAa,t)−1AT a,tra,t =R−1 a,t QT a,tra,t =R−1 a,t va,t (18) When applying QRD, given the initial Ra,t−1and va,t−1, we just need to triangularize the matrix where the newly observed context vector ϕstand the reward ra,t are appended: λRa,t−1λva,t−1 ϕT stra,t =˜ Qa,t Ra,t va,t 0T∗(19) The forgetting factor λ∈(0,1] accounts for an exponential weighting of the memory of the algorithm, and can be used in non-stationary scenarios as shown in section IV. The values in ∗are residuals that are not relevant in the choice of a(see (20)). We can easily adapt the LinUCB1 rule in (16) to include the vectors and matrices used by QRD: at+1 = argmax aˆ Qa,t +qαln(t)ϕT st(RT a,tRa,t)−1ϕst (20) The complexity of the algorithm is linear in the number of arms kand quadratic in the size of the contextual vector d. As proved in [5], the regret is upperbounded as Oqtd ln3(2kt)if λ= 1, as long as the average reward of each action satisfies the model in (7). Applying QRD to UCB1, we define the QRD Linear Contextual UCB1 algorithm. III. BEYOND LINEAR CMAB, A POLYNOMIAL MODEL Linear models relating the reward and context have limited representation capabilities. In general, (6) may not capture the complex relation between the context ϕstand the rewards that equation (5) suggests. We can improve the model by introducing a polynomial approximation whose coefficients are obtained from the linear recursive procedure in section II. In other words, a polynomial approximation can capture the Taylor series expansion of the true relation between the context vector ϕsand the reward. This means that a larger context vector ˜ ϕswill be used than the one given in (6). As an example, if the context contain two values: ϕT s=ϕ1, ϕ2 this vector will be the one used by the linear model, while for a quadratic model we take: ˜ ϕT s=1, ϕ1, ϕ2,(ϕ1)2,(ϕ2)2, ϕ1ϕ2 By implementing this concept, the algorithm is expected to model more accurately the unknown relationship between the observed rewards and the context, thus taking more accurate decisions. All CMAB concepts seen so far apply. However, this improvement also comes at the cost of enlarging the size of the vectors and matrices and requiring more computational burden, which can be controlled by the degree of the polynomial used. The application to the UCB1 algorithm yields the SM Quadratic Contextual UCB1 and QRD Quadratic Contextual UCB1 algorithms, a naming that will be used in section V below. IV. ADAPTIVE CMAB Up to this point, all the algorithms include a parameter λthat accounts for the non-stationarity of the scenario, and usually there is no clue about how to select λ. In terms of the network selection problem, the nonstationarity reflects a change on network parameters such as throughput or loss rate of a specific network. This can occur because of user terminal movement, a change in the amount of connected devices, etc. In our simulations, this change is implemented by time varying the parameters that characterize the probability density functions that we use to generate the network parameter values T H,DL and LR, introduced in section II.B and described in section V.A. Therefore, in contextual non-stationary environments, the optimum θavector changes over time. Stationary algorithms are not suited to track these changes because they have longterm memory: the first observed reward is as important as a recent one. For non-stationary environments, the memory of the algorithm needs to be tweaked to give more value to recent observations. Borrowing the adaptive concept presented in [9], we propose to continuously modify the forgetting factor λthat controls the memory in (19) by detecting a non-stationary situation. Our design is able to track if the current estimate ˆ Qa,t is far or close to convergence by measuring the history of signs of the error function et=ra,t −˜ ϕT st ˆ θa,t. Obtaining the same error sign during consecutive steps is interpreted as the current estimate is far from the true value, that is, the algorithm is far from convergence and the forgetting factor λhas to be decreased for faster convergence. On the other hand, the observation of consecutive alternate signs is a hint of the current estimate being close to the true value, the algorithm is close to convergence and λshould be enlarged. We propose the following rule: λis decreased when ethas the same sign on m1consecutive samples, and increased after m0consecutive alternate signs of et. Updates are made by adding or subtracting a predefined constant λstep. A maximum (λmax) and minimum (λmin) value for the forgetting factor must also be set. A pseudocode is described in algorithm 1. By applying this adaptive approach to λin the previous algorithms, we obtain: QRD Adaptive Linear Contextual UCB1, QRD Adaptive Quadratic Contextual UCB1, etc. Algorithm 1 QRD Contextual UCB1 Multi-Armed Bandit Intialization: initialize matrices and vectors. while t<Tsteps do ϕst←Get context at step t Modify ϕstif polynomial model is used for iin 1, k do ˆ Qi,t =ϕT st ˆ θi,t Ii=ˆ Qi,t +qαln(t)ϕT st(RT i,tRi,t)−1ϕst end for a= argmax a Ia ra,t =observe reward(a, t) et=ra,t −ϕT st ˆ θa,t if (m1equal signs on et) & (λ>λmin)then λ←λ−λstep ▷Decrease memory end if if (m0changing signs on et) & (λ<λmax)then λ←λ+λstep ▷Increase memory end if Update va,t,˜ Qa,t and Ra,t using (19) ˆ θa,t =R−1 a,t va,t t←t+ 1 Update total regret using (3) ▷For display purposes end while V. RESULTS In this section, the proposed algorithms are tested and compared. Results are discussed both in stationary and nonstationary conditions. A. QoE model The objective measurements in the flow T H,DL and LR used in the evaluation of the QoE are randomly generated from distribution functions designed by fitting histograms of real LTE traffic from various datasets [21] [22]. Those functions are used to generate random rewards ra,t from (4) and (5). For TH and DL multiple LTE datasets were used to generate the histograms. For LR the histograms in [17] were used instead. After testing multiple distributions, the following ones were chosen: •Throughput (TH): an inverse Gaussian distribution with µ= 3 and λ= 10. The generated values are in Mbps. •Delay (DL): a truncated Gaussian distribution with µ= 0.6511 and σ= 0.07. The generated values are in seconds. •Packet loss Rate (LR): a Beta distribution with α= 0.075 and β= 10. The values of the parameters for each objective measurement were slightly modified for every arm and on each nonstationary situation. Obviously, some other parameters might be in order for other technologies, such as WiFi or 5G. B. Simulation set-up The algorithms have been tested in both stationary and nonstationary scenarios, as described below. Scenario 1. Stationary case. It has k= 5 available wireless networks whose parameters are obtained from stationary density functions unique to each arm that follow the model proposed in [9]. In this scenario there are 11 different traffic types: Web Browsing, Instant Messaging, Voice Call, Online Game, Meeting Video, On-Demand Audio, File Sharing, Location, Live Audio, On-Demand Video and Live Video. The traffic types change at every step, so the QoE values obtained as rewards change accordingly. Hence, the selection of the optimal network also change as it depends on the traffic type. Scenario 2. Non-stationary case. It also has k= 5 actions. This scenario only uses 3 different traffic types: On-Demand Video, Live Video and Live Audio. The traffic type also changes at each step. We consider the situation where the distributions of rewards remain constant over epochs and change at unknown time instants: the parameters of the density functions for TH,DL and LR in section V.A make abrupt changes every 30.000 steps, and so does the generated rewards. Algorithms tested. Firstly, non-contextual UCB1 has been used to serve as a reference for the performance of the rest of the algorithms. As for the stationary contextual algorithms, SM Linear Contextual UCB1,SM Quadratic Contextual UCB1 and SM Cubic Contextual UCB1 have been tested with forgetting factor λ= 1 to account for stationary conditions. The QRD Linear Contextual UCB1 and QRD Quadratic Contextual UCB1 algorithms have been tested with a forgetting factor of λ= 0.995 in both scenarios. Furthermore, as adaptive contextual algorithms (see section IV), QRD Adaptive Linear Contextual UCB1 and QRD Adaptive Quadratic Contextual UCB1 have been used. The following parameters have been chosen for the QRD Adaptive Linear Contextual UCB1:λ= 0.995,λmin = 0.990,λmax = 1,λstep = 0.0001, m0= 15 and m1= 25. As for the QRD Adaptive Quadratic Contextual UCB1 algorithm: λ= 0.995,λmin = 0.990, λmax = 1,λstep = 0.0001,m0= 10 and m1= 15. These values have been decided after thorough testing in both scenarios. Regret plots are obtained after averaging over 50 independent runs. C. Evaluation of regret Scenario 1. In figure 1 the results for the stationary scenario are displayed in terms of the total regret Rt. Notice that the algorithm exhibiting the best performance is the one with the lowest regret. UCB1 is the algorithm with the worst performance. This behavior is expected, as it does not exploit the context to select the best action depending on the traffic type. The adaptive algorithms (denoted as QRD adaptive) are able to perform better than UCB1 and the non-stationary contextual algorithms, as they can detect that the environment is stationary and increase its memory accordingly. However, they perform worse than the stationary contextual algorithms (denoted as SM), as they need some time to adapt the forgetting factor. It is worth remarking that both polynomial contextual Fig. 1. Evolution of the total regret in the stationary scenario 1. algorithms behave better than the linear one, demonstrating the advantage of using higher degree polynomials in these contextual scenarios. It is also interesting to mention that the SM Quadratic Contextual UCB1 and SM Cubic Contextual UCB1 have very similar results. Even though the former has fewer degrees of freedom, it is more computationally efficient. Scenario 2. Results from the non-stationary scenario are presented in figure 2. It can be seen that the performance of the algorithms change drastically. Firstly, all stationary algorithms perform poorly, as they take very long to detect the abrupt changes due to their long-term memory. Even UCB1 is able to adapt much quicker to these changes compared to the stationary contextual algorithms. This occurs because the stationary contextual algorithms use λ= 1 in matrix Ba,t and vector za,t, that take long to adapt to the nonstationary changes. The best results are obtained with the adaptive QRD algorithms. They deliver better performance than the regular SM contextual algorithms, as they are able to tune the forgetting factor when there is a change in the network quality. It is also interesting to remark that the proposed nonstationary algorithms quickly converge at every change in stationarity. Discussion. Notice that the regrets of scenario 1 cannot be directly compared with those of scenario 2. This is because the regret function in (2) depends on the type of traffic and this is generated randomly for each type of scenario. Therefore, the values of r∗are random. Algorithms can however be compared in terms of the percentage of times the best possible network is selected at every time step over 50 runs. This is shown in table I. In the first scenario, the stationary contextual algorithms have the best performance, followed by the adaptive contextual algorithms. It is also clearly seen that polynomial algorithms present better results than linear ones. In the second scenario, it can be seen that the adaptive algorithms exhibit better results, showing total regret stabilization after each sudden change of environment. Fig. 2. Evolution of the total regret in the non-stationary scenario 2. TABLE I BEST-NETWORK SELECTION PERCENTAGE. Algorithm Scenario 1 Scenario 2 UCB1 24.9% 75.5% SM Linear Contextual UCB1 60.9% 53.8% SM Quadratic Contextual UCB1 80.4% 52.0% SM Cubic Contextual UCB1 80.0% 53.7% QRD Linear Contextual UCB1 52.7% 93.8% QRD Quadratic Contextual UCB1 56.4% 96.8% QRD Adaptive Linear Contextual UCB1 64.8% 93.8% QRD Adaptive Quadratic Contextual UCB1 67.0% 98.2% VI. CONCLUSIONS In this study, the wireless network selection problem has been modeled using a contextual multi-armed bandit model, where the context is given by the traffic type, allowing to take best decisions depending on the traffic type. Firstly, the classic contextual LinUCB1 algorithm has been improved by introducing a polynomial model for the context. Moreover, it has been demonstrated that all algorithms perform poorly in non-stationary environments. To this end, a forgetting factor has been introduced in the iterative solution using QR decomposition which does not present the instability problems of other matrix decomposition algorithms, as SM, allowing adaptation to model drifting in the scenario. A further improvement has been proposed by introducing an adaptive mechanism that detects non-stationatity and controls the memory of the algorithm by changing the value of the forgetting factor. Finally, all algorithms have been tested in realistic models-based environments where the agent must select the best wireless network with changing traffic types and non-stationary network parameters. An excellent performance has been confirmed for the adaptive algorithms. REFERENCES [1] Q. Wu, Z. Du, P.Yang, Y. Yao and J. Wang, “Traffic-Aware Online Network Selection in Heterogeneous Wireless Networks”, IEEE Transactions on Vehicular Technology, vol. 65, pp. 381–3976, 2016. [2] M. S. Allahham, A. A. Abdellatif, N. Mhaisen, A. Mohamed, A. Erbad and M. Guizani, ”Multi-Agent Reinforcement Learning for Network Selection and Resource Allocation in Heterogeneous Multi-RAT Networks,” in IEEE Transactions on Cognitive Communications and Networking, vol. 8, no. 2, pp. 1287-1300, 2022. [3] S. Boldrini, L. De Nardis, G. Caso, M. Le, J. Fiorina and M. Di Benedetto, “muMAB: A Multi-Armed Bandit Model for Wireless Network Selection”, Algorithms, vol. 11, art. 13, 2018. [4] D. A. Berry and B. Fristedt. ”Bandit Problems: Sequential Allocation of Experiments”. Monographs on Statistics and Applied Probability. Chapman and Hall, 1985. [5] L. Li, W. Chu, J. Langford, R. E. Shapire, ”A Contextual-Bandit Approach to Personalized News Article Recommendation”, The Nineteenth International WWW Conference, 26-30 April 2010, Raleigh, NC, USA. [6] D. Bouneffouf, I. Rish and C. Aggarwal, ”Survey on Applications of Multi-Armed and Contextual Bandits,” 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8, 2020. [7] Y. Cheng, C. Lee, H. Luo and C. Wei, ”A New Algorithm for Nonstationary Contextual Bandits: Efficient, Optimal, and Parameter-free”, Proceedings of the Thirty-Second Conference on Learning Theory, 2019, pp. 696-726. [8] Wu, Qingyun et al. “Learning Contextual Bandits in a Non-stationary Environment.” The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018. [9] L. Mart´ ınez, M. Cabrera-Bean and J. Vidal, ”A Multi-Armed Bandit Model for Non-Stationary Wireless Network Selection,” 2021 IEEE Globecom Workshops (GC Wkshps), 2021, pp. 1-6. [10] Z. Xu and A. Zhang, “Network Traffic Type-Based Quality of Experience (QoE) Assessment for Universal Services”, Applied Sciences, vol. 9, pp. 4107, 2019. [11] M. Collier and H. Llorens, ”Deep Contextual Multi-armed Bandits”, CoRR, 2018. [12] P. Auer, N. Cesa-Bianchi and P. Fischer, ”Finite-time Analysis of the Multiarmed Bandit Problem”, Machine Learning, vol. 47, pp. 235–256, 2002. [13] Y. Jun-Feng, ”Preconditioner based on the Sherman–Morrison formula for regularized least squares problems”, Applied Mathematics and Computation, 2009. [14] S. Hammarling and C. Lucas, ”Updating the QR factorization and the least squares problem”, 2008. MIMS EPrint 2008.111. [15] D. Bindel, ”Sherman-Morrison-Woodbury”, 2009. Cornell CS, Matrix Computations, chapter 5. [16] Z. Xu and A. Zhang, ”Network Traffic Type-Based Quality of Experience (QoE) Assessment for Universal Services”, Applied Sciences, vol. 9, pp. 4107, 2019. [17] D. Baltrunas, A. Elmokashfi, A. Kvalbein and ¨ O. Alay, “Investigating packet loss in mobile broadband networks under mobility”, 2016 IFIP Networking Conference and Workshops, pp. 225-233, 20168. [18] A. Garivier and E. Moulines, ”On Upper-Confidence Bound Policies for Switching Bandit Problems”, ALT 2011 Intl. Conf. on Algorithmic Learning Theory, Springer Berlin Heidelberg, 2011, pp.174-188. [19] A. Bozorgchenani, S. Maghsudi, D. Tarchi, and E. Hossain, ”Computation Offloading in Heterogeneous Vehicular Edge Networks: On-line and Off-policy Bandit Solutions”, IEEE Transactions on Mobile Computing, Vol. 21, No. 12, Dec. 2022, pp: 4233-4248 [20] M. Tokic, ”Adaptive ϵ-greedy exploration in reinforcement learning based on value differences” in Advances in Artificial Intelligence, Lecture Notes in Computer Science, vol. 6359, Springer-Verlag, 2010, pp. 203–210 [21] LTE and NS3 dataset, https://www.ucc.ie/en/misl/research/datasets/ivid 4g lte dataset/ [22] Univ-latencies dataset, https://sourceforge.net/projects/bandit/ files/Datasets/Univ.%20webpage%20latencies%20%28txt%29/ univ-latencies.zip/download