Recurrent neural networks for ornithopter trajectory optimization

Máster Universitario en Matemática Avanzada Trabajo Fin de Máster Recurrent neural networks for ornithopter trajectory optimization Presented by: Luis David Pascual Callejo Supervised by: Dr. José Miguel Díaz Báñez Dpto de Matemática Aplicada II Universidad de Sevilla November, 2020 2 Abstract Path planning is a widely studied subject due to its vast number of applications, specially for robots and unmanned vehicles. Strategies to solve it can be categorised as classical methods and heuristic methods, each one with its own advantages and disadvantages. Generally speaking, analytical methods are very complex for actual applications, whereas the heuristic methods are penalized by the size of the search space. For the case of unmanned aerial vehicles this penalization cannot be afforded, since due to weight and reaction time constrains, paths should be computed on line with fast and computationally light algorithms. In this work the use recurrent neuronal networks to contour this problem is proposed. The neuronal network is tasked with learning the underlying optimal trajectory flight dynamics, which are in turn numerically estimated by a time consuming heuristic method. More precisely, a recent heuristic method (OSPA) is used to compute a set of optimal trajectories for the ornithopter and then, the neuronal network is tasked with learning the underlying function from it. The goal is to obtain similar performances to the heuristic method with much faster computation times. The effectiveness and efficiency of the proposed algorithm are demonstrated through numerical simulations on validation data sets. In addition, far from blindly applying a recurrent neuronal network, a mathematical framework will be developed in other to justify the choices made and the resulting performance. Such framework will be supported by the universal approximation theorem, the algebraic feedforward neuronal network equations and the maximum likelihood method. 3 4 Resumen El cálculo de trayectorias es un área con múltiples aplicaciones, especialmente para robots y vehículos no tripulados. Las estrategias para su resolución pueden ser categorizadas en métodos clásicos y heurísticos, cada uno con sus ventajas e inconvenientes. De manera general, los métodos analíticos son demasiado complicados para aplicaciones reales mientras que los heurísticos son penalizados por el espacio de búsqueda. En el caso de vehículos aéreos no tripulados, dicha penalización no es aceptable debido a los requerimientos de tiempo de reacción y peso, lo que conlleva la necesidad de algoritmos ligeros y rápidos. En este trabajo se propone el uso de redes neuronales recurrentes para salvar dichas limitaciones. La red es encomendada con la tarea de aprender la dinámica de vuelo de trayectorias óptimas que han sido previamente estimadas numéricamente por un algoritmo heurístico. En concreto, se utiliza un novedoso algoritmo heurístico (OSPA) para calcular un set de trayectorias óptimas y la red neuronal recurrente tiene la tarea de aprender de él la función subyacente. El objetivo es obtener métricas similares en tiempos de computación mucho menores. La efectividad y eficiencia de la red será validada mediante simulaciones en un set se validación. Adicionalmente, lejos de aplicar indiscriminadamente una red neuronal, un marco matemático será desarrollado para justificar las elecciones hechas y los resultados obtenidos. Dicho marco será desarrollado a partir del teorema de aproximación universal, las ecuaciones algebraicas de la red y el método de máxima verosimilitud. 5 6 Acknowledgements I would like to thank Dr. José Miguel Díaz-Báñez for giving me the opportunity to work on this project, for his guidance and patience throughout the project and for their instructions and support over this time. 7 8 Contents 1 Introduction 15 1.1 Motivation and goal .......................... 15 1.2 Related work .............................. 17 1.3 Preliminaries .............................. 18 1.3.1 Introduction to neural networks ................ 18 1.3.2 Neural network architecture .................. 22 1.3.3 Recurrent neural networks ................... 24 2 The ornithopter trajectory optimization problem 29 2.1 Problem statement ........................... 29 2.2 Data set ................................. 31 3 The recurrent neural network 33 3.1 Input ................................... 33 3.2 Recurrent neural network layer .................... 33 3.3 Output .................................. 35 4 RNN Implementation 41 4.1 Input pre-processing .......................... 41 4.2 Recurrent Neural layer ......................... 42 4.3 Output layer ............................... 43 4.3.1 Regression model ........................ 43 4.3.2 Classification model ....................... 44 4.4 Training and evaluation ........................ 45 5 Results 47 5.1 Action prediction RNN ......................... 47 5.1.1 Network architecture and path computation ......... 47 5.1.2 Training ............................. 49 5.1.3 Results analysis ......................... 49 5.2 Sequence to sequence RNN ....................... 52 5.2.1 Network architecture and path computation ......... 52 5.2.2 Training ............................. 53 9 16 CHAPTER 1. INTRODUCTION Figure 1.1: Machine Learning to fractions of a second. This is where neuronal networks appear into scene, as they are widely known for solving complex problems with simple linear algebra operations. If a neuronal network can be trained to estimate an optimal trajectory, then it can be embarked in the ornithopter to efficiently compute the trajectory with little need of computation resources and time. Building on the ability of the OSPA algorithm to compute optimal trajectories for the ornithopter, our recurrent neuronal network will be tasked with learning the underlying flight dynamics from it. The goal is to obtain similar performances to the heuristic method with much faster computation times. Instead of applying blindly a neural network, a mathematical framework will be given in other to justify the choices made and the corresponding performance results. Despite the wide spread use of neuronal networks, there are not many mathematical models able to explain their effectiveness. Therefore, many times neuronal networks are treated as black boxes where data is poured in one side and answers are obtained from the other side as depicted on Figure 1.1. When linked to math, neuronal networks are often associated with linear algebra as most of the common algorithms make extensive use of it. However, when interested in explaining their effectiveness, other mathematical approaches as probability or information theory are more suitable Shwartz-Ziv & Tishby (2017). To sum-up, the goal of this work is to define, train and evaluate a recurrent neuronal network to the concrete case of the computation of the optimal trajectory for an ornithopter, so the complex underlaying flight dynamics are learnt. The training data is obtained from the results of the OSPA heuristic approach proposed in Rodríguez et al. (2020). The choices made and results will be discussed and justified using a probabilistic approach. 1.2. RELATED WORK 17 1.2 Related work The ornithopter optimal trajectory problem itself is defined in Rodríguez et al. (2020), where the authors have developed an algorithm (OSPA) to compute minimal energy consumption trajectories. This thesis builds upon their experience in optimal algorithms and explores Recurrent Neural Networks as a way to contour the problems they found, especially the long computation times. The application of neural networks to trajectory optimization is not new, see Glasius et al. (1995), Horn et al. (2012), Yang & Meng (2000) amongst others. Even the application of Recurrent Neural Networks (RNNs) to trajectory optimization is not new Wang (1999). However, most authors tackle this problem directly, embedding the trajectory constraints in the neural network equations and using the NN’s loss function to optimize the trajectory. Generally speaking, a lot of work exists on using neural networks to directly solve Non-Linear Programming problems (NLP), for example approximating an optimal controller for Unmanned Aerial Vehicles Xu et al. (2007). However this approach is very complex and can only be afforded if the vehicle mechanics are simple. As introduced in section 1.1, the flight dynamics from our ornithopter are by no means simple, but rather it is composed by a complex nonlinear differential equation system Rodríguez et al. (2020). Including such complexity in our network’s loss function or network’s optimization algorithm would be an impossible endeavour. Therefore the approach of this work consists on designing a NN that will mimic the behaviour of the OSPA algorithm just by learning from a data set of optimal trajectories generated by OSPA. This type of NN application is called function approximation. Previous work on NN function approximation for trajectory optimization can be found in Mordatch & Todorov (2014). In this thesis we are going to apply a very special type of neural networks, the recurrent neural networks. The reason behind is because this type of networks are able of modeling dynamic systems, from electronics Luongvinh & Kwon (2005) to engines Tan & Saif (2000). Instead of a blind implementation, a mathematical framework is given by the use of three pillars: •The universal estimator theorem. •The development of the NN algebraic equations derived from the neuron ones. •The use of the maximum likelihood There are many attempts to understand how neural networks work, but not that much results. A first, general overview on the mathematical aspects of neural networks can be found on Goodfellow et al. (2016). Although NN are driven by algebra, this book already introduces probably and information theory concepts 18 CHAPTER 1. INTRODUCTION as a mean to understand NN behaviour. In Murphy (2012) the reader can get a deeper dive into a probabilistic perspective of machine leaning and neural networks in particular. However, the best insights are given in Shwartz-Ziv & Tishby (2017), where the author uses information theory to present a comprehensive theoretical understanding on how Deep Neural Networks learn. This approach is further explained on Saxe et al. (2019) by means of the information bottleneck theorem. The RNN proposed in this work is extremely simple: only one recurrent layer with 11 neurons. Therefore, there is no point in applying the results from ShwartzZiv & Tishby (2017), which are intended for deep configurations. Instead, we have taken the fundamental concepts from probability and information theory and developed them into a suitable mathematical framework for our simple model. 1.3 Preliminaries 1.3.1 Introduction to neural networks Neural networks overview The goal of a feedforward network is to approximate some unknown function fby learning the values of the parameters θthat result in the best function approximation f∗(x;θ). It’s goal is thus to output for each value of x(features) a value ˆy=f∗(x)that is closest to y(label). The network is associated with a directed acyclic graph (DAC) describing how the functions are composed together. For example, if we have three consecutive layers f1(x),f2(x)and f3(x)connected in a chain, their composed function is f(x) = f3(f2(f1(x))) Goodfellow et al. (2016). Furthermore, in accordance with the universal approximation theorem, standard multilayer feedforward networks are capable of approximating any measurable function to any desired degree of accuracy. Theorem 1.3.1. Hornik et al. (1989). A feedforward network with a linear output layer and at least one hidden layer with any continuous squashing function can approximate any Borel measurable function from one finite-dimensional space to another with any desired non-zero amount of error, provided that the network is given enough hidden units. The derivatives of the feedforward network can also approximate the derivatives of the function arbitrarily well, see Hornik et al. (1990). This can be extended to the use of rectified linear activation functions (RLUs)(Leshno et al. (1993)). Definition 1.3.2. A function Ψ : R→[0,1] is a squashing function if it is nondecreasing, limλ→∞ Ψ(λ) = 1 and limλ→−∞ Ψ(λ) = 0 1.3. PRELIMINARIES 19 Figure 1.2: Feedforward neuron network schema. These models are called feedforward because information flows through the function being evaluated from x, through the intermediate computations, until the output y. There are no feedback connections in which outputs of the model are fed back into itself. When feedforward neural networks are extended to include feedback connections, they are called recurrent neural networks, Goodfellow et al. (2016). In Figure 1.2, a feedforward neuronal schema can be seen. Each color represents a layer, which is composed by several neurons. The first layer is called the input layer and the last layer is called the output layer. Any other layer in between is called a hidden layer. As it can be seen, for every layer k, each neuron nk iin position iis connected to each of the neurons of the next layer via a set of weights wk ji. Therefore, wk ji denotes the connection weight from neuron ibelonging to layer kto the neuron j belonging to the layer k−1. At every neuron, the weighted sum of the outputs from the neurons of previous layer is used to compute its output, as we will be see next in detail. Neuron equations Figure 1.3 shows how a single neuron works. Every neuron nk ihas: •A set of weights wk ji, that connect the neuron to the outputs of the precedent layer. •A bias bk i, which can we rewritten as wk 0iif we add for every neuron an additional input ok−1 0= 1. 20 CHAPTER 1. INTRODUCTION Figure 1.3: Single Neuron diagram in Guarnieri et al. (2006). •An activation function g(·)which should be continuous and inversible. This functions transforms the sum of all weighted inputs including the bias (ak i), into the final neuron output ok i. All these transformation can be summed-up in the equations below: ak i=bk i+ rk−1 X j=1 wk jiok−1 j= rk−1 X j=0 wk jiok−1 j ok i=g(ak i) (1.1) As it can be deduced, the parameters θto be optimized in the neuronal network corresponds to the weights wk ji, including the bias wk 0i. Neural Network parameter optimization In 1.3.1 it has been introduced how a neural network form a parametric family {f∗(·;θ)|θ∈Θ}, where Θis called the parameter space. The family f∗(·;θ) is given by the neural network architecture, which will be covered in 1.3.2. Our goal here is to find the parameter ˆ θ(which corresponds to the optimal weights ˆwk ji) so that f∗(x;ˆ θ)best approximates the true underlying function f. As in any optimization problem, a cost function has to be defined. Although any cost function can be chosen, the usage of the maximum log likelihood framework is selected here because it is the best estimator asymptotically in terms of its rate of convergence as the number of examples m→ ∞, see Goodfellow et al. (2016). 1.3. PRELIMINARIES 21 Definition 1.3.3. Goodfellow et al. (2016) Consider a set of mexamples X= x1, ..., xmdrawn independently from the true but unknown data generating distribution pdata(x). Let pmodel(X;θ)be a parametric family of probability distributions over the same space indexed by θ. The maximum likelihood estimator for θis then defined as: θML = arg max θpmodel(X;θ) = arg max θ m Y i=1 pmodel(xi;θ) = arg max θ m X i=1 log pmodel(xi;θ) Note that the Likelihood function it is just the joint probability density function for the independent and identically distributed random variables pmodel(xi;θ). Here our model tries to approximate the underlying data generation distribution pdata(x). Finally, the Log of the likelihood has been applied for convenience reasons, which is possible since the log is monotonic and thus the θML will not differ. If we rescale the expression above by the number of samples m, it can be rewritten as: θML = arg max θ E X[log pmodel(xi;θ)] In the cases we are interested in predicting Y given X, we generalize it to the conditional probability P(y|x;θ)as: θML = arg max θL(x;θ) = arg max θ m X i=1 log pmodel(yi|xi;θ) Neural Network optimization algorithm The simplest optimization algorithm (and the one we are going to use) is the gradient descendent back propagation. Actually, the most used optimization algorithms are built upon it. The gradient descendent simply consist on changing the value of the parameters θin the negative direction of the gradient of the cost function J(θ)with regards to θ. In our case, the cost function is the negative of the log likelihood. θt+1 =θt−α∂J(x, θt) ∂θ J(X, θ) = −L(x, θ) (1.2) The term back propagation comes from the fact that calculation of the gradient proceeds backwards through the network, with the gradient of the final layer of weights being calculated first and the gradient of the precedent layer’s of weights being calculated one at a time until the first layer is reached. The efficiency of the algorithms comes from the fact that partial computations of the gradient in one layer are reused for computation of the gradient in precedent layers recursively. 22 CHAPTER 1. INTRODUCTION This is a consequence of applying the chain rule to the partial derivative of the cost function with regards to every weight. For every input-output pair (xi, yi)we obtain the following result: ∂J ∂wk ij =∂J ∂ak j ∂ak j ∂wk ij The first term can be rewritten as: δk j=∂J ∂ak j = rk+1 X l=1 ∂J ∂ak+1 l ∂ak+1 l ∂ak j = rk+1 X l=1 δk+1 j ∂ak+1 l ∂ak j where rk+1 denotes the number of nodes in the next layer (k+1). Taking into account that by definition ak+1 l=Prk j=1 wk+1 jl gak j, this term can be expressed as: δk j= rk+1 X l=1 δk+1 j ∂ak+1 l ∂ak j = rk+1 X l=1 δk+1 jwk+1 jl g0ak j The second term can be rewritten as: ∂ak j ∂wk ij =∂ ∂wk ij rk−1 X l=0 wk ljok−1 l!=ok−1 i Combining again the first and second terms, we finally obtain the partial derivative of the error function J with respect to a weight wk ij: ∂J ∂wk ij =δk jok−1 i=g0ak jok−1 i rk+1 X l=1 wk+1 jl δk+1 l Note that this partial derivative only depends on the errors δk+1 lat the next layer, whereas the rest of the values were calculated during the forward propagation of the input values xithrough the network. To wrap-up, feed forward neural networks are trained in two phases: •The forward phase, where input values xiflow forward to compute the output value yi •The backwards phase, where for each pair input-output, error values flow backwards to compute the gradient. 1.3.2 Neural network architecture The neuronal network architecture and activation functions will define the parametric family {f∗(·;θ)|θ∈Θ}. They will be chosen in order to have a balanced network capacity. The capacity is defined by: 1.3. PRELIMINARIES 23 •Depth and width of the network: generally speaking, the deepest (more layers) and widest (more neurons per layer) the greatest number of parameters will be available and thus a greater extent of the training information can be learned. •Activation functions and general architecture: they will define the representational capacity of the model, or in other words, the set of functions that can be learned by the NN. For example, if we only use linear functions as activation functions g(∆), we will not be able to capture non-linear behaviours in our training data. Regarding the general architecture, adding feedback loops in our architecture will allow us to capture sequential/dynamical behaviours as it will be explained for the recurrent neuronal networks in 1.3.3. The adjective balanced is used since NN with high capacity are prone to overfitting, whereas NN with low capacity are prone to underfitting •Underfitting occurs when the network is not able to capture enough information from the training set and thus the learned model f∗(x;θ)is too simple to correctly approximate the underlaying function f. •Overfitting occurs when the network is not able to generalize the information learned from the training data. That is, the learned model f∗(x;θ)is too specific for a particular set of training data and when a new unseen input x is fed into the network it is not able to make a good estimation. This problem can be described as a bias-variance problem as follows: First, we are going to use the Mean Squared Error (MSE) as a measure of our performance. We will see later on on 3.3 how minimizing the MSE is equivalent to maximizing the Likelihood under certain hypothesis. The MSE is defined as: MSE =Ex∈data[(y−f∗(x))2] We also define bias and variance of our approximation function f∗(x;θ)as: bias[f∗(x)] = ED⊂data[f∗(x)]−f(x); var[f∗(x)] = ED⊂data[f∗(x)−ED⊂data[f∗(x)])2] Here ED⊂data[f∗(x;θ)] is the expectancy of f∗(x;θ)over the different training sets D. Every different training set would lead to different optimal parameters θand thus different values of f∗(x;θ)for the same sample x. Finally we can assume that our data has some data noise with regards to the true values of the unknown underlaying function f(x)and that this noise has zero mean. y=f(x) +  E[] = 0 24 CHAPTER 1. INTRODUCTION We can expand the expectancy of the Squared Error of a sample over the different training sets Das: ED(y−f∗(x))2=ED(f(x) + −f∗(x))2 =ED[(f(x)−f∗(x))2] + ED[2]+2ED[(f(x)−f∗(x))]ED[] =ED[(f(x)−f∗(x))2] + σ2  =ED[(f(x) + ED[f∗(x)] −ED[f∗(x)] −f∗(x))2] + σ2  =ED[(f(x)−ED[f∗(x)])2] + ED[(f∗(x)−ED[f∗(x)])2]−2ED[(f(x) −ED[f∗(x)])(f∗(x)−ED[f∗(x)])] + σ2  =bias[f∗(x)]2+var[f∗(x)] −2(f(x)−ED[f∗(x)])(ED[f∗(x)] −ED[f∗(x)]) + σ2  =bias[f∗(x)]2+var[f∗(x)] + σ2  Therefore, averaging over all test samples, we obtain the MSE as: MSE =Ex[Ef∗(y−f∗(x))2] = Ex[bias[f∗(x)]2] + Ex[var[f∗(x)]] + σ2  As it can be deduced, big (or at least statistically representative) training sets Dreduce both the bias and the variance. Unfortunately, as we will see in subsection 2.2, our available data set is small and thus we will be very susceptible to a generalization error if the NN model is not wisely chosen Neal et al. (2018). Finally, the error σcannot be reduced by any means and it will be a lower bound. On our specific ornithopter trajectory problem this error will be the inherited error from the OSPA heuristic algorithm trajectories given as input. In section 3.1 we will try to partially overcome this error by adding the true target value at the end of each OSPA trajectory. Figure 1.4 shows the trade off between NN model bias and variance Papachristoudis (2019). 1.3.3 Recurrent neural networks A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence. These connections allow previous outputs to be used as inputs while having hidden states. Therefore, this type of neuronal network exhibits temporal dynamic behavior. Mandic & Chambers (2001) This property will allow us to capture the underlying flight dynamics contained in the OSPA optimal trajectories. Figure 1.5 shows the schema of a single recurrent neuron as in Amidi (2019). •The neuron output at time t, yt, is a function of the hidden state a time t, atthrough a set of coefficients Wya, a bias byand an output activation gate g2. 1.3. PRELIMINARIES 25 Figure 1.4: NN model bias-variance trade off Papachristoudis (2019). Figure 1.5: Single Recurrent Neuron schema 32CHAPTER 2. THE ORNITHOPTER TRAJECTORY OPTIMIZATION PROBLEM •The sequence of flight states S= (s0, s1, s2, s3...sn). •The sequence of flight maneuvers A= (a0, a1, a2, a3...an−1). •The cost of the trajectory. •The computation time for the OSPA method. Out of the 236 trajectories, the 80% of them are randomly selected for for training whereas the remaining 20% is used as validation set to compute the generalization error and RNN performance metrics. Variable Initial State Target State X(m)0[25,100] Z(m)0[−40,0] Θ (◦) [−30,30] 30 Ub(m/s) [1,4] 0 Table 2.1: Ranges of initial and target state variables used in the experiments. Chapter 3 The recurrent neural network 3.1 Input The data set introduced in section 2.2 is pre-processed before ingestion by the RNN. More precisely, the inputs to train our RNN are the sequences of flight states S= (s0, s1, s2, s3...sn), which will be transformed to the distance sequences X= (x0, x1, x2, x3...xn)by applying the following transformations: •First, in order to include information about the target in each time step, the flight state sequences have been changed to the distances to the target: xi=starget −si •Second, due to the fact that the different flight components in the flight state (x, z, u, w, θ, q)have a great disparity in values, they have been normalized using their standard deviations as follows: xi=zi σ Regarding the actions a= (δ, f), they will be used as they are since both orders of magnitude are similar. This input data follows a probability distribution which will be represented throughout this work as pdata(x). 3.2 Recurrent neural network layer Feedforward neural networks learn from the training samples as if each of them were independent and identically distributed random variables. However, we know that this is not the case when dealing with our trajectory problem, or more generally, with sequences. In other words, the current state xtis not independent 33 34 CHAPTER 3. THE RECURRENT NEURAL NETWORK from the previous state xt−1. Thus, it seems logical to use the information in the previous states to better estimate the next action. Note that for the specific case of our trajectories, the next action is only function of the current state and the target state. Thus, a simple neuronal network could do this job as it is not mandatory to consider the previous states to compute the next action. In more general sequence cases, like for example in sentence translations, it is necessary to consider all previous words and even future ones to fully put in context the current word. As discussed in the introduction, the recurrent neural networks exhibits temporal dynamic behavior. Since the ornithopter trajectory can be described by a dynamical system, a RNN seems then the perfect artificial neural network class choice. Furthermore, in our case we will show that we comply within Theorem 1.3.4. This will in turn allow us to apply all the good properties of a feedforward network. In deed, within any given trajectory, the conditional probability distribution over the variables at t+ 1 given the variables at tis stationary. This is true since: •For every tuple of starting and target points, it is expected to obtain the same optimal trajectory. Therefore, considering the starting point as an intermediate point at time t of a longer trajectory with same target point, the following intermediary points are expected to be same. •When dealing with discrete trajectory state values, due to the fact that the time step is constant, the same intermediate state values are expected, making their probability distribution stationary with time. Finally, we can also state that our problem complies the universal approximation theorem. Following Theorem 1.3.1, this is true since: •Any continuous function on a closed and bounded subset of RNis Borel measurable. •Our underlaying function fmust be continuous as two optimal trajectories can be arbitrarily close as far as the starting and target points are respectively as close as needed. •Our subset is bounded in IR6as no infinite trajectory can be optimal (nor feasible). Note that the universal approximation theorem ensures that a NN can approximate the optimal trajectory function to any degree of accuracy, but nothing is said about how the NN should be in terms of configuration or needed number of parameters (which can be lead in practice to unfeasible NN). Finally, the gradient descendent algorithm can be applied as well to our RNN as introduced in section 1.3.1 for feedforward networks. ) 3.3. OUTPUT 35 3.3 Output The NN approximation task can be divided into regression or classification problems Classification model As we will see in the implementation and results, the ornithopter possible action data set is actually a finite set of 35 different action tuples. This is due to the methodology of the heuristic method used to compute the optimal path, which requires a finite set of possible action outcomes in order to obtain a search tree. In the task of predicting the next action, the RNN can either output the tuple a= (δ, f), leading to a regression model, or it can predict which is the most probable action to apply from the aforementioned finite set of actions, leading to a classification model. The outcome thus will be the probability to apply each of the possible actions and one with the highest predicted probability will be selected. Therefore, in the classification problem, the NN aims to output the most probable action to take amongst a given set of finite options, given the corresponding preceding flight states. In the classification model each action akis treated as a different category, leading to 35 different categories: ak= (δk, fk), k = 1, ..., 35 In this case, we want the RNN to output the probability that each category has to be selected. This gives the following relationship between the real probability distribution ykand our prediction ˆykto be: ˆyk=p(yk|x, θ) where ˆykis the vector with the predicted probabilities for each category and ykis the "one hot" representation of each category. A "one-hot" representation of the category kconsist on a vector with size the number categories where all elements are zero except the kth element with value 1. For instance, a3is represented by [0,0,1,0, ..., 0] . This representation indicates that the probability for category 3 is 1 whereas is zero for the rest. Proposition 3.3.1. It is equivalent to use the Log-likelihood or the Categorical Cross Entropy as loss function for our NN classification problem. Proof. As for the regression case, when computing the maximum log likelihood we end up with: L= m X i=1 log p(y|xi;θ) = m X i=1 log ˆyi= m X i=1 N X k=1 p(yik) log ˆyik 36 CHAPTER 3. THE RECURRENT NEURAL NETWORK The later equality is true since each probability p(yik)is zero except for the observed category kthat is equal to 1. Note that this last term corresponds to the negative cross-entropy of the distribution ˆyirelative to a distribution p(yi)which, for discrete probability distributions, can be written as: H(p(yi),ˆyi) = − N X k=1 p(yik) log ˆyik This gives us the following result θML = arg max θL= arg max θ− m X i=1 H(p(yi),ˆyi) The cross entropy loss function can be interpreted as the expected messagelength per datum when a wrong distribution pmodel is assumed while the data actually follows a distribution pdata. Or in other words, when our neural network f∗(·;θ∗)is used instead of the real source of data, which is our OSPA planner. Therefore, the higher the cross-entropy is, the further our NN is from generating the real data probability distribution. Here the locally optimal parameter θ∗is found by the loss optimization algorithm so that the cross-entropy values is locally minimized for the parametric family f∗(·;θ). We have just seen a way to interpret the Likelihood loss function. However, we may be interested in an actual measure of dissimilarity between our model and the underlying true data distribution. A measure of the dissimilarity between any two distribution is the KL divergence, and it is defined for discrete distributions as: DKL(pdata kpmodel) = X x∈X pdata(x) log pdata(x) pmodel(x;θ) Although the KL divergence is not a true metric, since for instance DKL(pkq)6= DKL(qkp), it is intuitively a pertinent loss function since it represents the amount of information lost when pmodel is used to approximate pdata. Proposition 3.3.2. It is equivalent to use the Log-likelihood or the KL divergence as loss functions to compute the optimal parameters for our NN classification problem. 3.3. OUTPUT 37 Proof. This divergence can be rewritten as: DKL(pdata kpmodel) = DKL(p(y)kˆy) = m X i=1 p(y(xi)) log p(y(xi)) ˆy(xi;θ) = m X i=1 p(y(xi)) log p(y(xi)) − m X i=1 p(y(xi)) log ˆy(xi;θ) The first term of the right corresponds to the negative of the entropy Hof p(y)and does not depend on the parameters θ. The second term on the right correspond to the cross-entropy of ˆy(x;θ)relative to p(y). DKL(p(y)kˆy) = −Hy+Hyˆy(x;θ) Here we can see that the amount of information lost when ˆyis used to approximate yactually corresponds to the difference of Entropy. Therefore, maximizing the log likelihood is equivalent to maximizing the similarity which in turn corresponds to minimize the cross-entropy. θmDKL =θmCH =θML When using the KL divergence as a loss function, in addition to find the locally optimal parameter θ∗, we can actually asses the quality of our model. Knowing that a zero KL divergence value corresponds to a perfect match between model and the true distribution, we can consider the values in Table 3.1 as a guideline Brownlee (2019): KL Divergence Value (nats) Model valuation 0.00 perfect <0.02 good <0.05 on track <0.2 fine <1.0 poor >2.0 broken Table 3.1: Model valuation by KL divergence loss values. 38 CHAPTER 3. THE RECURRENT NEURAL NETWORK Regression model In the regression problem, the RNN aims to output the best possible approximation to the values of the true states in IR6given by an OSPA trajectory. Therefore, after training, the RNN will output a prediction ˆyigiven the input xi. The relationship between the real value yiand our prediction ˆyican be written as: yi= ˆyi+ei where eiis the error due to either non modeled aspects. Assuming that all these aspects are independent, we can apply the central limit theorem to rewrite this error as a Gaussian distribution with the form: p(ei) = 1 σ√2πe−1 2(ei σ)2 =1 σ√2πe−1 2(yi−ˆyi σ)2 Here, it is assumed that the error has standard deviation σand null mean. This implies that p(yi|ˆyi)follows a Normal distribution N(ˆyi, σ2I). Due to the fact that ˆyidepends on xiand θit can be rewritten as: p(yi|ˆyi;σ) = p(yi|xi;θ;σ) = 1 σ√2πe−1 2(yi−ˆyi σ)2 Proposition 3.3.3. Given the above mentioned hypothesis, it is equivalent to use the Log Likelihood or the Mean Squared Error as loss functions for our NN. Proof. If we compute the Maximum log likelihood, we end up with: L= m X i=1 log p(yi|xi;θ;σ) = m X i=1 log 1 σ√2πe−1 2(yi−ˆyi σ)2 =mlog 1 σ√2π− m X i=1 1 2yi−ˆyi σ2 θML = arg max θL= arg max θmlog 1 σ√2π− m X i=1 1 2yi−ˆyi σ2 = arg max θ m X i=1 1 2(yi−ˆyi)2, which turns to be equivalent to minimizing the Mean Squared Error. In our Neural Network, we will use a Linear Activation to output the prediction as we expect it to be continuous in IR6. The MSE is already a good measure of how well our model fits the true distribution and it gives a measure of the estimation error from our model. Nevertheless, we can also have the value of the measure of the dissimilarity between the model and true distributions via the KL divergence: DKL(pdata kpmodel) = X x∈X pdata(x) log pdata(x) pmodel(x;θ) 3.3. OUTPUT 39 Proposition 3.3.4. It is equivalent to use the Log-likelihood or the KL divergence as loss functions to compute the optimal parameters for our NN regression problem. Proof. This divergence can be rewritten as: DKL(pdata kpmodel) = X x∈X pdata(x) log pdata(x)−X x∈X pdata(x) log pmodel(x;θ) The first term of the right corresponds to the negative of the entropy Hof pdata(x) and does not depend on the parameters θ. The second term on the right correspond to the negative log likelihood of pdata(x)and pmodel(x;θ). DKL(pdata kpmodel) = −Hpdata −L(x;θ) Therefore, maximizing the similarity corresponds to maximizing the log likelihood. θmDKL =θML = arg max θ E Xlog pmodel(x;θ) 40 CHAPTER 3. THE RECURRENT NEURAL NETWORK Chapter 4 RNN Implementation This chapter contains an overview on how our RNN problem statement has been actually implemented, layer by layer. Overall, all the scripts are programmed using Jupyter notebooks and the neural networks are computed using the TensorFlow library in Python. Jupyter Notebook allows us to explain how the code works on the same script file and TensorFlow is a widely used Machine Learning Library developed by Google. All RNN implementation code is published in https:// github.com/paskymail/Neuronal_Networks/tree/master/Simple_RNN. 4.1 Input pre-processing Prior to the ingestion of the training data, the training set is pre-processed using the script "Process_Training_Data_2.ipynb" This pre-process consist on: •Transforming Flight state sequences into its normalized trajectory distances sequence as explained in Section 3.1. •Creating distance sequences of equal length by truncation when the trajectory is too long or by padding with zero distance vectors when the trajectory is too short. The reasons for setting equal length trajectories are: •The RNN is trained by input batches and each batch should be of equal length due to computation constraints. •Only flight states belonging to the same trajectory be feed to the RNN in order to comply with theorem 1.3.4. As explained in Figure 4.1, the padding consist on adding zero distance vectors when the original trajectory sequence is not long enough. For instance, if the 41 48 CHAPTER 5. RESULTS Figure 5.1: Next action prediction from previous states. •In the input layer we will input the last predicted state. •The output layer is expected to output the probabilities for the next action to apply. As explained during the problem statement, a softmax activation function is applied. •The fixed function xt=F(xt−1;at)will compute the next state based on the predicted action at. •The recurrent layer is expected to learn the function at+1 =f(ht;xt), where xtis the predicted distance at time tand ht=fr(ht−1;xt−1)is the hidden layer state which contains information from previous steps. These dependencies can be inferred from the recurrent neuron equations (1.3). Taking the aforementioned points in mind, the algorithm 1is used to construct the trajectory step by step. In Table 5.1 a summary of the Keras Neural Network can be found. It can be seen how the output layer has 35 neurons, one per category so its probabilities can be compared with the real category "one-hot" representation. The neural network contains a total of 1212 trainable parameters. It is important to note that this approach does not take advance of the recurrent nature of the neural network to build a complete sequence, but rather the network 5.1. ACTION PREDICTION RNN 49 Algorithm 1: trajectory loop Result: X= (x0, x1, x2, x3...xn), Cost x0; while x6=Zero do ai=RNN(~x); Cost =C(ai); xi=F(ai); ~x.append(xi); end is used as an action classification. Any other simple network is suitable to do this job, but this architecture is used as a stating point to be able to evaluate further on the benefits of other architectures capable of exploiting the recurrent properties of the network. 5.1.2 Training Now, we have to define the network cost function. Our goal here is to estimate the next action to apply, therefore our target it to get the action category estimate ˆa=p(a|x;θ)as close as possible to the true action category value p(a|x). We are going to apply what it has been already developed on subsection 3.3 and the we use the KL divergence loss function as the measure of the dissimilarity between these two distributions. which is defined for a discrete distributions as: DKL(pdata kpmodel) = X x∈X pdata(x) log pdata(x) pmodel(x;θ) The computation of such loss function is implemented in the TensorFlow API by the KL Divergence Loss function. Alternatively, the Categorical Cross-entropy Loss function could be used with the same results. Despite the high number of parameters, no overfitting effects have been observed. In Figure 5.2 the loss (KL divergence) decrease during training is shown. It has to be noted that the validation loss decreases jointly with the training loss, which implies that the networks is learning properly and it is able to generalize to unseen validation data. Finally, the KL divergence loss value reached is close to 0.02, which in accordance with Table 3.1 corresponds to a good model fit. 5.1.3 Results analysis Figure 5.3 plots the RNN predicted X, Z trajectory again the OSPA trajectory. We see I that the RNN trajectory is quite similar to the OSPA behavior. 50 CHAPTER 5. RESULTS Figure 5.2: Loss (KL-divergence) decrease during training Figure 5.3: X,Z components trajectory comparison. 5.1. ACTION PREDICTION RNN 51 Layer Output shape parameter number LSTM [None, None, 11] 792 Dense [None, None, 35] 420 Table 5.1: Action classifier network summary. Figure 5.4: Comparison on the u,v,θ,qcomponents. This property is also true for the complete 6 components as shown in Figure 5.4, where the u,v,θand qcomponents of the flight state are plotted against the xcomponent. This is expected since there is no special priority treatment for any component and thus the performance for each of them should be similar. In case of any component to be prioritized, a set of weights can be assigned for each one when computing the loss function in such a way errors on some components are specially penalized. The overall results can be seen in Table 5.2, which contains the following performance metrics already introduced in section 2.1: •Cost (W): is the total energy cost (battery) consumed by the ornithopter, and it is determined by the maneuvers (actions) performed. •Time (s): is the total elapsed time in computing the full trajectory. •Precision (s): is the euclidean distance in the XZ-plane from the final state to the target •Trajectory error (m): is the mean of distance in the Z axis between the OSPA and the RNN trajectories at 10 given intermediary X-positions. 52 CHAPTER 5. RESULTS These metrics were identified as relevant for the OSPA performance itself and are not used to optimize the network parameters, they are computed only for comparison purposes. Algorithm Cost (W) Time (s) Precision (m) Trajectory error (m) OSPA 34.68 520 2.84 na Action prediction 37.25 0.43 6.29 3.85 Table 5.2: Comparison with OSPA algorithm. As an overall conclusion, it can be stated that the RNN is able to generate new optimal trajectories from unseen start and target points. Even if the RNN is much faster (0.43s vs 520s), precision has been worsened compared to the OSPA planer in terms of distance to the target (6.29m vs 2.84m). It has to be noted that although the overall trajectory energy consumption has also increased (37.25 vs 34.68), it has been done in less proportion, indicating that similar trajectories also have similar costs and that the energy consumption is not specially sensitive within small deviations. This fact will turn useful when the states are predicted instead of the actions and thus the cost cannot be computed. In the next sections we will exploit the dynamic properties of the RNN to improve our results. We will maintain the same 11 neurons in the recurrent layer and reduce the neurons in the output layer, reducing the number of trainable parameters. Although a reduction in performance could be expected, we will see how the new architectures will actually improve the precision, even beyond the OSPA algorithm. In other words, even if the number of parameters will be reduced, we will change the parametric family {f∗(·;θ)}for a new and better suited one, with increased capacity. 5.2 Sequence to sequence RNN 5.2.1 Network architecture and path computation In our first attempt for a real recurrent network we are going to use a sequence to sequence configuration as the one shown in Figure 5.5. This configuration will be also the set up used for the training of the next RNN architectures. Therefore the configuration is as follows: •In the input layer we will input the true flight state distances trajectory sequence •The output layer is expected to output the same trajectory sequence shifted by one time-step in the future 5.2. SEQUENCE TO SEQUENCE RNN 53 Figure 5.5: Sequence to sequence architecture. •The hidden layer (corresponding to the recurrent layer) is the expected to learn the function xt+1 =f(ht;xt), where xtis the true distance at time t and ht=fr(ht−1;xt−1)is the hidden layer state which contains information from previous steps. This dependencies can be inferred from the recurrent neuron equations 1.3. Note that if we do not perform the sequence shift, the NN would learn an identity function. In Table 5.3 a summary of the Keras Neural Network can be found. It can be seen how the output layer has 6 neurons, one per flight state so their estimated values can be compared with the true ones at each time step. The neural network contains a total of 864 trainable parameters. Layer Output shape parameter number LSTM [None, None, 11] 792 Dense [None, None, 6] 72 Table 5.3: Sequence to sequence network summary. Finally, the network schema for a given time step can be seen in the Figure 5.6: 5.2.2 Training Now, we define as usual the network cost function. Our goal here is to estimate the same input sequence shifted by one time step, therefore our target it to get pmodel(x;θ)as close as possible as pdata(x). In this case, due to the fact that we are dealing with a regression model, we are going to use the equivalence between the Maximum Likelihood and the minimum Mean Squared Error seen in 3.3 to use the Mean Square Error as our Loss function. 54 CHAPTER 5. RESULTS Figure 5.6: Sequence to sequence time step. Therefore: θML =θmMSE The computation of such loss function is implemented in the TensorFlow API by the MSE Loss function. A smooth MSE loss decrease can be seen in 5.7 with no overfitting effects. It has to be noted that the validation loss values are very close to the training ones, which implies that the networks is able to generalize the results perfectly. 5.2.3 Results analysis As shown in Figure 5.8, the predicted X, Z trajectory correctly matches the OSPA planner. Even when a change in direction occurs, the RNN is able to predict it. Finally, the fact that the goal state is not reached is because the trajectory is truncated to an arbitrary number of steps (10 in this case) as explained in subsection 4.1. This behavior is also true for the complete 6 components as shown in Figure 5.9. Again, the RNN is able to correctly predict a sudden change for the θand qcomponents. Table 5.4 shows the overall results. There are two scores to be highlighted. •The mean distance between the OSPA and RNN predicted trajectories is 1.59 meters, which implies that both trajectories are very close to each other and thus very similar trajectory energy costs are expected as shown in the results analysis of subsection 5.2. 5.2. SEQUENCE TO SEQUENCE RNN 55 Figure 5.7: Loss (MSE) decrease during training Figure 5.8: X,Z components trajectory comparison 56 CHAPTER 5. RESULTS Figure 5.9: Comparison with components u,v,θ,q. •The precision has exceeded the one reached by the OSPA planner (1.40m vs 2.84m). The later statement may seem to be impossible at first glance, since in accordance with bias-var variance trade off explained in section 1.3.2, we are bounded by the OSPA data noise σ2 due to equation MSE =Ex[bias[f∗(x)]2]+Ex[var[f∗(x)]]+ σ2 . However, the fact of having introduced the real target value at the end of each sequence as explained in subsection 4.1 has allowed us to overcome such limitation and exceed the OSPA performance. Algorithm Cost (W) Time (s) Precision (m) Trajectory error (m) OSPA 34.68 520 2.84 na RNN sequential na 0.46 1.40 1.59 Table 5.4: Comparison with OSPA algorithm. 5.3 Use of RNN as decoder 5.3.1 Network architecture and path computation In the previous configuration we have considered that the previous true distances to the target are known and that we are only interested in predicting the next distance state. However, a planner should predict the flight states of multiple steps in the future, not only the next one. This will be the Decoder case, where 5.3. USE OF RNN AS DECODER 57 Figure 5.10: Decoder architecture. Figure 5.11: Decoder neuron. only the initial distance is feed and the task is to develop a complete trajectory from it so it can be embarked in an ornithopter for trajectory planning purposes. As can be seen in the Figure 5.10, a decoder has a single input and develops a sequence starting from this input. For our specific case, we will input the initial distance and we will obtain the full trajectory. This would be possible as long as the initial input contains enough information in such a way that the optimal trajectory is defined, which is true in our example. Given an starting point Aand end point B, there should be a single optimal trajectory connecting them. Actually, the only difference with the previous networks is that at every time step, the input consists in the prediction of the previous states instead of the true previous states, see Figure 5.11. Taking the aforementioned points in mind, the algorithm 2is used to construct the trajectory. 64 CHAPTER 5. RESULTS Chapter 6 Conclusion and future work First, we are going to sum-up what has been achieved so far: 1. It has been introduced what feedforward neural networks are, what are they for, their architecture and how their neurons work. 2. Once connected, the equations of these neurons have been developed so that the general NN algebraic equations are obtained. This was done with the aim of providing a general understanding on how actually NN are able to output a prediction given an input. 3. These equations, in turn, have shown how a given neural network form a parametric family {f∗(·;θ)|θ∈Θ}, where the family f∗(·;θ)is given by the NN architecture and the parameters θcorrespond to the NN weights. 4. It has been introduced the maximum likelihood framework as mean to estimate the parameter value ˆ θfor a given family, so that under the assumed model f∗(x;ˆ θ), the observed data is the most probable. Or in other words, f∗(x;ˆ θ)is the best approximate to the true underlying function f. 5. It has been introduced that NNs are universal approximators up to any arbitrarily small error under certain conditions, and how our ornithopter trajectory optimization meets them. 6. The back propagation algorithm has been developed from the feedforward NN algebraic equations as a mean to compute the parameter ˆ θthat gives the (local) maximum likelihood for our model. 7. The general maximum likelihood expression θML = arg maxθQm i=1 pmodel(xi;θ) has been further developed into more suitable expressions depending on the problem. Actually, the ML was not possible to compute for our regression problem. 65 66 CHAPTER 6. CONCLUSION AND FUTURE WORK 8. For our regression problem, the MSE optimization has been proven equivalent to the ML optimization under certain hypothesis. 9. For our classification problem, the minimum KL divergence or the crossentropy have been proven equivalent to the ML. Additionally, these expressions are shown meaningful loss functions since they quantify the loss of information incurred in using our NN model instead of the true distribution. 10. The recurrent neurons have been introduced as a way to give our NN the capacity to learn temporal dynamic behaviours and thus increase the capacity of our parametric family. 11. The "virtual" unfolding of the recurrent neurons layer into feed-forwarded ones has been introduced as a way to apply the aforementioned properties and proofs. 12. The ornithopter trajectory optimization has been introduced as a physical application of all the aforementioned points. 13. The ornithopter trajectory optimization data has been presented and explained as well as the necessary data pre-processing. 14. A concrete Recurrent Neural Network candidate for the problem has been presented and explained. 15. This candidate has been adapted to both the regression and classification problems, explaining the reasons behind each output layer configuration. 16. This RNN has been applied to the ornithopter trajectory optimization problem using 4 different architectures. 17. A first architecture consisting on a classifier for the next action has shown good results improving largely the OSPA computational times while keeping a reasonable performance. 18. A second architecture consisting on a regression sequence to sequence configuration has shown even better results, outperforming the OSPA performance regarding precision to the target. 19. A third architecture consisting on a decoder configuration has exceeded the previous performance, proving the fact that the RNN is able to learn the underlaying trajectory flight dynamics. 20. A fourth and last architecture creating an Ordinary Differential Equation Integrator has shown how RNN can be designed so they behaviour can be interpreted in physical terms. 6.1. CONCLUSION 67 21. Finally, the bias-variance trade-off has been deduced from the MSE expression, which in turn has been used to explain undesired training behaviours. 6.1 Conclusion The goals stated in the motivation have been achieved throughout this work: •On the theoretical side, all neural network expressions or choices have been mathematically derived or supported by three pillars: 1. The universal estimator theorem. 2. The development of the NN algebraic equations derived from the neuron ones. 3. The use of the maximum likelihood and its derivations to determine the optimal parameters for the paramedic family formed by the NN. •On the applications side, several RNN architectures have been applied to the specific problem of the ornithopter trajectory optimization, leading to the following results: 1. The RNN has outperformed the OSPA method both in time (0.5s vs 520s) and precision to the target (1.19m vs 2.84m). 2. The RNN has been able to learn the underlying flight dynamics of the problem. 3. The ODE Integrator architecture has given a physically interpreted RNN behaviour. 4. All the results and choices have been justified using the mathematical background developed at the beginning of this thesis. 6.2 Future work Some investigation lines for future work are the following: •In our work, the use of information theory concepts as the cross-entropy and KL divergence has been introduced to explain the output of information of our NN with regards to the true data distribution. The use of information theory to explain how NNs work can be further developed by the use of the information bottleneck theorem Saxe et al. (2019). This theory attempts to explain the behavior of the deep learning via the transfer of information though the successive NN layers. 68 CHAPTER 6. CONCLUSION AND FUTURE WORK •One of the reasons of not having applied this theorem to this work, is due to the fact that our RNN is very simple (11 neurons) and shallow (only recurrent and output layer) to be consider deep learning. The use of a deeper and wider network can open new fields and questions as well as improve the network’s performance. •The use of deep learning is subject to having enough data. Due to the limit availability of it (236 trajectories), its usage has not been possible in this work. However, if additional data is gathered, deeper versions of the RNN can be considered. •In line with the availability of additional trajectories, other trajectory types can be added so the capacity of the RNN to learn simultaneously different types can be tested. •Finally, the integration of additional features to the loss function can be considered so other elements as obstacles or additional trajectory optimization metrics can be included. Bibliography Amidi, A. (2019), ‘Cs 230 - deep learning’. URL: https://stanford.edu/ shervine/teaching/cs-230/cheatsheet-recurrentneural-networks Baek, S. S., Bermudez, F. L. G. & Fearing, R. S. (2011), Flight control for target seeking by 13 gram ornithopter, in ‘2011 IEEE/RSJ International Conference on Intelligent Robots and Systems’, IEEE, pp. 2674–2681. Brownlee, J. (2019), ‘A gentle introduction to cross-entropy for machine learning’. URL: https://machinelearningmastery.com/cross-entropy-for-machinelearning/ Chen, R. T., Rubanova, Y., Bettencourt, J. & Duvenaud, D. K. (2018), Neural ordinary differential equations, in ‘Advances in neural information processing systems’, pp. 6571–6583. DeLaurier, J. D. (1994), ‘An ornithopter wing design’, Canadian aeronautics and space journal 40(1), 10–18. Glasius, R., Komoda, A. & Gielen, S. C. (1995), ‘Neural network dynamics for path planning and obstacle avoidance’, Neural Networks 8(1), 125–133. Goodfellow, I., Bengio, Y. & Courville, A. (2016), ‘Deep learning book’, MIT Press 521(7553), 800. Guarnieri, R. A., Pereira, E. B. & Chou, S. C. (2006), ‘Solar radiation forecast using artificial neural networks in south brazil’, Proceedings of the 8th ICSHMO pp. 24–28. Horn, J. F., Schmidt, E. M., Geiger, B. R. & DeAngelo, M. P. (2012), ‘Neural network-based trajectory optimization for unmanned aerial vehicles’, Journal of Guidance, Control, and Dynamics 35(2), 548–562. Hornik, K., Stinchcombe, M. & White, H. (1990), ‘Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks’, Neural networks 3(5), 551–560. 69 70 BIBLIOGRAPHY Hornik, K., Stinchcombe, M., White, H. et al. (1989), ‘Multilayer feedforward networks are universal approximators.’, Neural networks 2(5), 359–366. Leshno, M., Lin, V. Y., Pinkus, A. & Schocken, S. (1993), ‘Multilayer feedforward networks with a nonpolynomial activation function can approximate any function’, Neural networks 6(6), 861–867. Luongvinh, D. & Kwon, Y. (2005), Behavioral modeling of power amplifiers using fully recurrent neural networks, in ‘IEEE MTT-S International Microwave Symposium Digest, 2005.’, IEEE, pp. 4–pp. Mandic, D. & Chambers, J. (2001), Recurrent neural networks for prediction: learning algorithms, architectures and stability, Wiley. Mordatch, I. & Todorov, E. (2014), Combining the benefits of function approximation and trajectory optimization., in ‘Robotics: Science and Systems’, Vol. 4. Murphy, K. P. (2012), Machine learning: a probabilistic perspective, MIT press. Neal, B., Mittal, S., Baratin, A., Tantia, V., Scicluna, M., Lacoste-Julien, S. & Mitliagkas, I. (2018), ‘A modern take on the bias-variance tradeoff in neural networks’, arXiv preprint arXiv:1810.08591 . Papachristoudis, G. (2019), ‘The bias-variance tradeoff’. URL: https://towardsdatascience.com/the-bias-variance-tradeoff-8818f41e39e9 Rodríguez, F., Díaz-Báñez, J. M., Sanchez-Laulhe, E., Capitán, J. & Ollero, A. (2020), ‘Kinodynamic planning for an energy-efficient autonomous ornithopter’, arXiv 2010.12273. Ross, I. M. (2009), A primer on Pontryagin’s principle in optimal control, Collegiate Publ. Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D. & Cox, D. D. (2019), ‘On the information bottleneck theory of deep learning’, Journal of Statistical Mechanics: Theory and Experiment 2019(12), 124020. Sherstinsky, A. (2020), ‘Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network’, Physica D: Nonlinear Phenomena 404, 132306. Shwartz-Ziv, R. & Tishby, N. (2017), ‘Opening the black box of deep neural networks via information’, arXiv preprint arXiv:1703.00810 . Tan, Y. & Saif, M. (2000), ‘Neural-networks-based nonlinear dynamic modeling for automotive engines’, Neurocomputing 30(1-4), 129–142. BIBLIOGRAPHY 71 Wang, J. (1999), ‘On-line path planning for autonomous mobile robots using recurrent neural networks’, IFAC Proceedings Volumes 32(2), 599–604. Xu, P., Verma, A. & Mayer, R. J. (2007), Neural dynamic optimization for autonomous aerial vehicle trajectory design, in ‘Independent Component Analyses, Wavelets, Unsupervised Nano-Biomimetic Sensors, and Neural Networks V’, Vol. 6576, International Society for Optics and Photonics, p. 65760X. Yang, S. X. & Meng, M. (2000), ‘An efficient neural network approach to dynamic robot motion planning’, Neural networks 13(2), 143–148.