Power Amplifier Behavioral Modeling Through Convolutional Neural Networks

Proyecto Fin de Carrera Ingeniería de Telecomunicación Formato de Publicación de la Escuela Técnica Superior de Ingeniería Autor: F. Javier Payán Somet Tutor: Juan José Murillo Fuentes Dep. Teoría de la Señal y Comunicaciones Escuela Técnica Superior de Ingeniería Universidad de Sevilla Sevilla, 2013 Trabajo Fin de Grado Grado en Ingeniería de las Tecnologías de Telecomunicación Power Amplifier Behavioral Modeling Through Convolutional Neural Networks Autor: Gonzalo Chávez Romero Tutores: Juan Antonio Becerra González, María José Madero Ayora Dpto. Teoría de la Señal y Comunicaciones Escuela Técnica Superior de Ingeniería Universidad de Sevilla Sevilla, 2020 Trabajo Fin de Grado Grado en Ingeniería de las Tecnologías de Telecomunicación Power Amplifier Behavioral Modeling Through Convolutional Neural Networks Autor: Gonzalo Chávez Romero Tutores: Juan Antonio Becerra González Profesor Sustituto Interino María José Madero Ayora Profesora Titular Dpto. Teoría de la Señal y Comunicaciones Escuela Técnica Superior de Ingeniería Universidad de Sevilla Sevilla, 2020 Trabajo Fin de Grado: Power Amplifier Behavioral Modeling Through Convolutional Neural Networks Autor: Gonzalo Chávez Romero Tutores: Juan Antonio Becerra González, María José Madero Ayora El tribunal nombrado para juzgar el trabajo arriba indicado, compuesto por los siguientes profesores: Presidente: Vocal/es: Secretario: acuerdan otorgarle la calificación de: El Secretario del Tribunal Fecha: Agradecimientos En mi primer lugar, me gustaría destacar lo especialemnte priviliegiado que me siento de tener familiares que me han apoyado no solo moralmente durante el transcurso de la carrera, sino también académicamente. Es mucho más fácil afrontar los complicados retos de la carrera sabiendo que los tengo a mi lado y por ello estoy muy agradecido. A continuación, me gustaría destacar la ayuda y compañía de mis compañeros de clase, sobre todo Vicente, Manu, Juan Antonio, Rafa y Luis. Es increíble lo amena que se hace la carrera con personas tan inteligentes y agradables a mi alrededor. Todos y cada uno de ellos han hecho la rutina del día a día realmente cómoda y me llevo muy buenas memorias con cada uno de ellos, como los desayunos con Jota o el día que Vicente vino con un zapato de cada color. Realmente me entristece pensar que es muy poco probable que vuelva a verles día a día y sin duda son la razón por la que echaré de menos este periodo universitario. Por último, me gustaría agradecer al profesor Juan Antonio Becerra su ayuda y motivación durante la carrera. Ahora mismo soy un apasionado por el Machine Learning y probablemente sea gracias a él. Antes de conocerle y ponerme a trabajar en este TFG jamás me habría planteado investigar en el campo de la Teoría de Señal y Comunicaciones, ya que era el que menos me apasionaba de toda la carrera, pero aquí estamos, realizando un TFG sobre predistorsión digital, todo gracias a Juan Antonio, que me hizo ver que aunque mi primera impresión no fuera la mejor, realmente tenía líneas de investigación muy interesantes. I Abstract In this document we present an approach to characteristic modeling of power amplifiers using convolutional neural networks. Radio Frequency Power Amplifiers suffer from distortions when they approach high efficiency levels, and by having a model of how the amplifier is going to distort radiofrequency signals, we can pre-distort the signal before hand with the inverse characteristic to mitigate the distortion the amplifier applies. To do so, we propose a convolutional neural networks. Neural network behave like universal function approximators given enough data. We present that this architecture is fairly successful when modeling nonlinearities on three different amplifiers while keeping its complexity and number of parameters to reasonable levels. Lastly, we replicate results on similar research showing that feeding certain transformations of the inputs to the network leads to better modeling of the amplifier without raising the number of parameters much. Because of the inherent performance of convolutional neural networks, we can augment the input data both memory-wise and order-wise without adding many parameters to the model. III 2Capítulo 1. Introduction provide us with an efficient pre-distortion tool. This application is motivated by recent success of Convolutional Neural Networks on other time-related and forecasting problems, and their simplicity, scalability and paralellization potential compared to Recurrent Neural Networks make them a very attractive option for behavioral modeling. We test the networks performance with a variety of real-world, lab-acquired signals. We also explore which architecture performs the best for each experiment, confirming previous research on related topics. 1.2 Methods and Materials Input and output pairs of signals are obtained using amplifiers at the lab or on online laboratories such as the one provided by Chalmers University of Technology. Neural networks architecture are defined using Python and a variety of machine networks libraries, mostly Pytorch, SKLearn and Optuna and experiments run on a Google Colab environment, which provides us with high-end Graphics Processing Units for quicker computations and optimization loops. 2 Neural Networks Th is section aims to explain what neural networks are and what problems they aim to solve. Neural networks are widely used nowadays for a variety of purposes, it can be either self driving cars, image recognition, natural language processing, stock market prediction and a great amount of different applications are discovered each year for neural networks, but what are they, really? Neural Networks are certainly not the first ever learning algorithm created, but since they have shown to be flexible in a variety of different scenarios, they are raising a great interest in the scientific community. Neural networks’first developments come motivated by neuroscientific research that tried to model the brain behaviour as early as 1943 [ 40 ]. Even though neuroscience is regarded as a source of inspiration for current neural networks, it does not serve as its guide, and modern research on the machine learning field diverges from the neuroscience path. Neural networks have however become an important technology fairly recently, and that has to do with two main factors: the amount of data available to us and the computational power, specifically in current GPUs (Graphic Processing Units). Figure 2.1 Tesla’s Autopilot interpretation of its environment. Here, the input is the different images captured by the cameras and the networks output is represented by the different lines painted over the road identifying each object and path . We can then think of neural networks as universal function approximators, that is, with enough data and computational power, a neural net will be able to find a function that maps from a given input to a desired output, not matter the dimensionality and complexity of the problem at hand. Now, 3 4Capítulo 2. Neural Networks if we come back to one of the aforementioned examples, like the self driving car, we can see when the neural net comes into play: the network maps from a number of inputs (cameras and a number of sensors) to a desired output (steer the driving wheel a number of degrees, activate the brakes, increase acceleration and so on). Of course, the network at hand here becomes fairly complicated, as it has more parts and different layers than the simple one used in this document, but it exemplifies the level of abstraction a neural net can achieve and the vast amount of situation where we can make use of an universal function approximator. 2.1 Overview of Neural Networks The quintessential architecture of neural networks is called Feedforward Neural Net of Multilayer Perceptron. They are called feedforward because the input flows through the different functions of the network to the output, without any feedback of the network outputs to itself (unlike, for example, a Recurrent Neural Net, outside of the scope of this document but widely used nonetheless). As a quick note, we will be using the word tensor throughout the document. Tensor is the abstraction of the matrix or vectors, that is, for example, a vector is a 1-dimensional tensor, and a matrix is a 2 dimensional tensor. Further on, we will talk about 3-dimensional tensors as inputs for our neural networks. The building block of neural networks is called a neuron [ 18 ]. Neurons are small components that take a number of inputs, sum them and apply a defined function. Figure 2.2 Diagram of a single neuron. The inputs are represented by the x vector and the output is represented by the y∗ scalar. Any given neuron can have as many inputs as desired. The w vector represents the weights associated with the neuron, and brepresents the bias of the neuron . For a given neuron, we can express the output as: y∗=g( ∞ ∑ n=1 xnwn+b) 2.1 Overview of Neural Networks 5 Where g(x) is called Activation Function, and they serve a double purpose: to prevent gradient vanishing and gradient exploding, which we’ll explain later in the learning section, and to introduce non-linearity in the network. The parameter b here is called bias, and gives each neuron a constant regulated by the weight of the corresponding connection in case we need to shift the activation function to produce better results. Neurons are arranged in layers inside a neural network, and these layers are connected by synapses governed by weights. The diagram below exemplifies this: Figure 2.3 Diagram of a simple neural network. All the neurons are connected by the weights parameters, the bias bhere is removed for clarity . Where the parameters wl nm are the weights. Each connection on a neural network has a number associated with it called weight. Weights are multiplied by the input on that connection and are the parameters we will adjust during the training process of a neural network. Each element the vector z1represents each input of the corresponding neuron: z1=  x1w1 11 +x2w1 21 +bwb1 x1w1 12 +x2w1 22 +bwb2 x1w1 13 +x2w1 23 +bwb3   And a2represents the vector where each element is the output of each neuron: a2=  g(z1 1) g(z1 2) g(z1 3)  The network above represents a very simple feedforward neural network. More complicated variations exist, where different kind of layers and neurons are used, such as Recurrent Neural Networks using LSTM (Long Short Term Memory) [24] or Convolutional Neural Networks [34]. 6Capítulo 2. Neural Networks 2.2 Forward propagation on Neural Networks When we provide neural networks with an input and it makes the computation necessary to produce and output is called a Forward Pass, because the information is flowing forward from input to output. On this section we are going to explain the computations a neural net follows to produce its outputs. First, the input to every neural network is in the form of a multidimensional matrix or a tensor. In most circumstances the information is going to take the form of a 2D matrix, where each row represents an example and each column represents the information about the abovementioned example we want the network to learn, also called ‘features’. Here’s an example of a dataset of different NBA players where each row represents a different player and each column represent the statistics of each player: Figure 2.4 Simple Dataset of the NBA players. The Dataset contains 31 columns, that is, 31 features for each player. . We could use the above dataset for, for example, train a network to predict the number of 3- pointers a player is going to score given the different information on the dataset. For our example of a forward computation we are going to use however a simple, 2-feature input and a neural net with only one hidden layer and output layer, as the network on Figure 1.3. Given an example input like: x1x2y 1 6 50 2 7 78 3 5 15 Here, x1 and x2 are our input vectors or features and we want to train the network to predict y given x1and x2. Now, we can write the matrix resulting of the first layer computations as: z1=  1 6 2 7 3 5 w1 11 w1 12 w1 13 w1 21 w1 22 w1 23=  1w1 11 +6w1 21 1w1 12 +6w1 22 1w1 13 +6w1 23 2w1 11 +7w1 21 2w1 12 +7w1 22 2w1 13 +7w1 23 3w1 11 +5w2113w1 12 +5w1 22 3w1 13 +5w1 23   Note that Z1 is now a matrix instead of a vector, since we are feeding several examples at the same time. The number of examples (rows) on a forward computation is called batch size. The batch size is one of the many parameters that we have to adjust empirically to find the best results. This 2.2 Forward propagation on Neural Networks 7 parameters are called hyperparameters. Now, if we name each matrix and ignore the bias, we can write: X·W1=Z1 Where Z1 has as many rows as the input matrix and as many columns as the number of neurons on the hidden layer. Then, following this notation, we can write the 4 equations that govern the forward pass on this neural network: Z1=X·W1 A1=g(Z1) Z2=A1·W2 Y∗=g(Z2) Where g() is the activation function present on each neuron. One of the most widely used activation function is the ReLU function: −4−2 0 2 4 −2 0 2 4 Figure 2.5 ReLU activation function. Other activation functions are represented below. What activation function we use is again one of the hyperparameters of the neural network. We may have to try different activation functions for the neurons until we arrive at the best performing one. 8Capítulo 2. Neural Networks −4−2 0 2 4 −1 0 1 2 −4−2 0 2 4 −1 0 1 2 Figure 2.6 Left: Sigmoid activation function. Right: Tanh activation function. −2−1 0 1 2 −2 −1 0 1 2 −4−2 0 2 4 −2 −1 0 1 2 Figure 2.7 Left: Identity activation function. Right: Binary Step activation function. When we first create the neural network, the weights are randomly initialized, since the network doesn’t know yet the importance of each feature. As we train the network, the values of these weights are going to change accordingly to produce better results. To train the network, we need a function to measure the error of a forward pass of the neural network to know how to adjust the weights. This function is called the Cost function (also called Loss function). One Loss function we can use is the Root Mean Squared Error (RMSE) to compute the error between the desired output and the actual output of the net. RMSE has the following formula: RMSE =r∑n i=1(yi−yi∗)2 n That was the simple walk-through of the neural net forward pass, and now we have a way to measure the error of its output. We mentioned before that activation functions introduce non-linearity on the neural networks outputs. To expand on this explanation, we can consider an example where the activation function is the linear function as in Figure 1.7. Given the activation function, we can express the output of the neural network like this: 2.3 Backpropagation and Gradient Descent 9 Z1=W1·X+b1 A2=Z1 Z2=W2·A2+b2 y∗=Z2 Then: y∗=W2·[W1·X+b1]+b2 y∗= [W2·W1]·X+[W2·b1]+b2 Let: W= [W2·W1] b= [W2·b1]+b2 Then the output is: y∗=W·X+b Which is again a linear function. We can say then that if no activation functions are going to be used, we might as well not add more layers to a neural net, since all the layers are going to behave the same way, transforming the data on a linear fashion and not providing any further insight on the data at hand, since the addition of two linear functions is another linear function. Our network couldn’t then approximate any non-linear function, such as the one below, making the network a pretty weak function approximator. By adding nonlinear activation function, we allow the neural network to transform the input data on nonlinear representations to gain insight on the input/output relationship. A really good example of this is the XOR function approximation step-by-step example on [18] 012345 0 0.1 0.2 0.3 0.4 0.3 Figure 2.8 Example of a non-linear function that a neural network with no activation function couldn’t approximate. 2.3 Backpropagation and Gradient Descent Now that we have the difference between the output we want and the output the network gave us, we need to adjust the weights of the network to reduce the error. To calculate how to change the weights to produce outputs with less error we use backpropagation or backprop [ 52 ]. To explain how backprop works, we can plot the RMSE against one of the parameters, for example w1 11 , for the given input X: 10 Capítulo 2. Neural Networks −4−2 0 2 4 0 10 20 30 w1 11 L(w1 11) Figure 2.9 Hypothetical plot of the Loss function and w1 11 . In this graph, we would want the error to be as low as possible when given X or inputs similar to X to the neural network. To do so, we’d need to change the value of w1 11 in the direction where the cost function decreases. To achieve that, we calculate the derivative of the cost function with respect to w1 11 to obtain the slope of the cost function. Now that we know the slope of the cost function, we know in which direction the cost function decreases and increases, so we just need to adjust w1 11 in the direction to decrease the RMSE. This process of updating every weight parameter of a neural network can then be written as [18, 35]: w1+ 11 =w1 11 −εdLoss dw1 11 Where ε is the learning rate parameter. The learning rate is once again one of the hyperparameters of the network. The higher the learning rate, the “faster” the network is going to learn, but if the learning rate is too high, we can overshoot the optimum value of the weight. Figure 2.10 Very good example of how different learning rates affect the learning process. J(θ) represents the Loss and θrepresents the weight parameter. Source [27] . As we explained above, the process of calculating each derivative of every parameter on the neural 2.4 Good practices and classical problems of neural nets 11 net is called backprop. Backprop makes use of the chain rule of calculus, which is used to compute the derivative of functions formed by other functions. In neural networks, a computing graph is built for every parameter on the network (like the one below) and we just need to plug the specific numbers on the derivatives to obtain the slope of the cost function on that weight parameter. Figure 2.11 When calculating the derivatives, we make use of the chain rule. The backprop algorithm simply creates computing nodes describing how to calculate the derivatives. Source [18] . Adjusting weights is how neural networks learn, a good way of thinking about it is that the network is finding the neuron connections and transforming the inputs that produce the best results, that is, the results that lead to the lowest RMSE across all inputs. 2.4 Good practices and classical problems of neural nets The first good practice technique on machine learning is to split the dataset at hand. See, for a given set of examples, we are going to divide them in two groups, a train set to train the neural network and a test set to evaluate the performance of the neural network. We want our network not only to perform well on inputs from the train set, but also to perform well when given previously unseen inputs, that is, we want our network to generalize well. We will consider our model successful when the training loss is low but also when the test error is low as well. We have to be careful however when performing the split because data may be imbalanced, that is, what if our train set is only a subset of too similar examples? When performing the split we need to keep an eye on the variance of the dataset and each train and test subset to make sure that high variance is mantained across all variables on each subset. Cross Validation, however may help prevent this. Using K-Folds Cross Validation, we split the train data on K subsets, train on K-1 sets and use the last subset as test data. We then average the model against each of the folds. Lastly we test the model with the test data to evaluate the performance. 18 Capítulo 2. Neural Networks Figure 2.21 Performance of different models on the image-recognition CIFAR-10 dataset . As we can see, regardless of the regularization or dropout or gradient-improving technique we choose, usually shallow networks outperform very deep networks, mostly due to vanishing gradients. The layers closer to the output are improving their weights while the layers closer to the input are not learning at all. To resolve this, the Residual Block is introduced: Figure 2.22 Residual block, the main component of the ResNET [21] . We are forcing the network to learn the residual mapping from the input to the output instead of transforming the input data through several layers to get the output. This allows the ResNet to learn the most effective depths for a given input. Now we can have networks with hundreds of layers that does not suffer from vanishing/exploding gradient and that is very computationally attractive, since the identity mapping does not introduce extra parameters or computations. Variable-depth networks become very potent, specially in convolution-related tasks, since higher level of abstraction is needed, leading to ResNET to win the ImageNet competition on 2015. Temporal Convolutional Networks (TCN) are popular architecture when working with time-series datasets. They implement several layers of convolutional layers to capture long time dependencies and use residual blocks to prevent vanishing gradient. TCNs are designed for sequence modeling, that is, a TCN outputs: y∗0,...,y∗T=f(x0,...,xT) Where the ground truth yt depends only on x0,...,xt and not on future inputs xt+1,...,xT . TCN prevents leakage from the future, TCN uses 1D convolutional layers where each layer has the same length as the input, and it performs dilated convolution: 2.7 ResNET and Temporal Convolutional Networks 19 F(s) = (x∗df)(s) = k−1 ∑ i=0 f(i)xs−di Where d is the dilation factor, k is the filter size and s−di points to the direction of the past. Using larger dilations allows TCN to look further into the past by expanding each neuron receptive field Figure 2.23 Building blocks of TCN. Left 3 dilated convolutional layers with d=1,2,4 and kernel size k=3 .Middle TCN residual block architecture for TCN. Right Residual connection on TCNs[ 9 ]. TCN are able to train in parallel, unlike RNNs, where past predictions are needed in the training. TCN also allow for variable receptive field by varying the number of convolutional layers, changing the dilation parameter d or increasing the filter size. TCNs outperform most RNNs architectures on a variety of benchmark datasets as shown in [9] 3 Radio Frequency Power Amplifiers Radio Frequency Power Amplifiers (RFPA) are electronic devices that boost a given signal to the according level so it reaches the receiver. An ideal RFPA would take the already modulated signal at a given frequency and output a boosted undistorted signal, that is, we want the amplifier to behave as linearly as possible, but that’s not the case. Amplifiers power consumption often surpasses every other component on a given radio system, so optimizing the efficiency of the amplifier is key to reduce battery consumption or operating costs. Nowadays, circuit design must confront tougher and tougher linearity specifications, to reduce nonlinear distortions as much as possible. However, as we just mentioned, amplifiers are not perfect, and the function that models its behaviour is certainly not linear, several nonlinearities are introduced on the system in the form of distortions. These nonlinearities cause interference on the receiving end. RFPAs are divided into different categories depending on their efficiency: A, B, AB, etc.. . [ 58 ] and there’s a tradeoff between efficiency and linearity on a given amplifier. With linearity, we mean that the output is a linear transformation of the input to the amplifier, such as: y(t) = Gx(t) Where y(t) represents the output of the amplifier, x(t) its input and a is a constant parameter. Figure 3.1 Power inputs and output on an amplifier. Source [48] . On the figure above, Pin and Pout represent the input and output power of the signals through the amplifier, Pdc represents the DC power supply to the amplifier and Pdiss is the power dissipated as heat from the amplifier. We can rewrite the equation above as: 21 22 Capítulo 3. Radio Frequency Power Amplifiers G=Pout Pin And we can write Pout +Pdiss =Pin +Pdc Then G=1+Pdc −Pdiss Pin Thus the total power at the output of the amplifier is limited by the DC supply. We can represent this nonlinearity as: Figure 3.2 Power in the input plotted against the output power and the gain. Source [58] . This effect is called gain compression. The input signal amplitude is affecting the output signal amplitude in a non-linear way. On lower power inputs we can see how both the Gain and the output power follow the linear “ideal” behaviour we want from the RFPA, with this ideal gain being represented by the dashed line. However, at higher powers we can see how both move away from the linear behaviour and enter the saturation zone. At higher power inputs, we won’t get higher power outputs past a certain point. This makes sense, since the amplifier will not be able to create power from nowhere. The maximum power the RFPA will be able to output is going to be close to the power provided to the amplifier as supply, with a small portion of the power dissipated as heat. The thing is, we want the amplifier to operate around this nonlinear zone, since we want to be as efficient as possible, thus making the best use of the power fed to the amplifier. 3.1 Nonlinear distortions To get an idea of how nonlinear system affects the given inputs [ 48 ], let’s work with an example for a bit. Given an typical stimulus to a telecommunications environment, we can write an input baseband signal as: x(t) = A(t)cos[ωct+θ(t)] 3.1 Nonlinear distortions 23 Given a nonlinear system restricted to the third polynomial: yNL =a1x(t−τ1)+a2x(t−τ2)2+a3x(t−τ3)3 The response of a linear system would be: yL=a1A(t−τ1)cos[ωct+θ(t−τ1)−φ1] And the response of the nonlinear system would be: yNL =a1A(t−τ1)cos[ωct+θ(t−τ1)−φ1] +a2A(t−τ2)2cos[ωct+θ(t−τ2)−φ2]2 +a3A(t−τ3)3cos[ωct+θ(t−τ3)−φ3]3 By using the following trigonometric relations: cos(α)2=1 2+1 2cos(2α) cos(α)3=3 4cos(α)+ 1 4cos(3α) We can rewrite the nonlinear response as: yNL =a1A(t−τ1)cos[ωct+θ(t−τ1)−φ1] +1 2a2A(t−τ2)2+1 2a2A(t−τ2)2cos[2ωct+2θ(t−τ2)−2φ2)] +3 4a3A(t−τ3)3cos[ωct+θ(t−τ3)−φ3] +1 4a3A(t−τ3)3cos[3ωct+3θ(t−τ3)−3φ3] Where φ1=ωcτ1,φ2=ωcτ2,φ3=ωcτ3 Here, several interesting terms appeared. We have a number of nonlinear responses on different frequencies. This phenomenon is called spectral regrowth. Though we can filter this frequencies out, other components such as the 1 2a2A(t−τ2)2term introduce the so called DC bias. Lastly, now the amplitude of our baseband signal has been distorted as well, becoming a1A(t−τ1)+ 3 4a3A(t−τ3)3 , as well as the phase of the baseband signal being distorted. This effects are known as AM/AM and AM/PM distortions respectively, meaning that the input amplitude modulation creates output amplitude and phase modulation on the output signal. 24 Capítulo 3. Radio Frequency Power Amplifiers Figure 3.3 AM/AM distortion on a class J amplifier. X axis represents the input power (dBm) while the Y axis represents the AM/AM gain (dB). We can clearly see the nonlinear behaviour of the amplifier, especially towards higher power inputs . Figure 3.4 AM/PM distortion on a class J amplifier. X axis represents the input power (dBm) while the Y axis represents the AM/PM phase shift (dB). Again, the system behaves in a nonlinear fashion more clearly seen on higher input powers . 3.2 Other memory-related distortions As per [ 58 ], memory effects means past inputs of a given signal influence influence the value of the signal in the current timestep. The number of samples from the past that influence the current timestep is known as memory depth. 3.2 Other memory-related distortions 25 Figure 3.5 Past signal influence current-time values on systems with memory [58] . Memory effects can be caused by a variety of reasons, such as charge being built-up on capacitors. On amplifiers, the memory effects affect the rise and fall times, creating different output voltages for the same input voltage. Figure 3.6 Rise and fall times behave differently because of memory effects [58] . Memory effects on amplifiers can be classified on Short-Term Memory Effects (STM) or Long- Term Memory Effects (LTM). STM effects are produced by matching networks, that is, components such as capacitors or inductors built into the amplifier and capacitances inherent to the transistor, where build-up. transport and decay of charge create time delays. LTM effects have affect across timesteps much bigger than the carrier frequency of the input signal. They include thermal effects, inherent to the flow of thermal energy within the components of the amplifier, and can manifest itself as a small change in the gain of the amplifier after the signal has passed. Other LTM effect is the charge trapping, where imperfect components of the amplifier store charge and release it continually. DC supply may also affect the modulation as a LTM effect introducing bias voltage at the modulation frequency affecting the gain of the amplifier. Lastly, if the amplifier contains control features for automatic bias and gain control, the time associated with early mentioned features may introduce LTM effects that can be difficult to correct. 26 Capítulo 3. Radio Frequency Power Amplifiers We can then rewrite this memory dependencies onto our nonlinear system model as: yNL =fNL(u[t],u[t−1],u[t−2],...,u[t−m]) 3.3 Behavioral modeling and Digital Predistortion To predict and quantify the distortions on a given RFPA and design circuits to better mitigate the aforementioned distortions, behavioral models are required. The advantage of building this behavioral models is that we do not need to know exactly how every single component on a complex systems interacts with each other, we could treat the system as a black box and model its input/output relationship through a number of techniques. The performance of a behavioral model resides then in the accuracy of our observations of the signals and the choice of formulation that describes the amplifier itself. There are many options to choose from to model an input-output relationship such as polynomial approximation and Fourier series, but one of the most popular techniques on PA behavioral modeling is by using Volterra series [ 61 ]. Introduced in 1887 by Vito Volterra, they use a combination of linear convolution and nonlinear power series to describe an input/output relationship of a system with fading memory, the general equation of discrete Volterra series is: y(n) = K ∑ k=1 M ∑ i1=0 ... M ∑ ip=0 hp(i1,...,ip) k ∏ j=1 x(n−1) Where hp(i1,...,ip) represents the parameters of the Volterra model, the Volterra kernels, K is the nonlinearity order and M is the memory depth. Given as the input and output of an amplifier: x(t) = ℜ[˜xejω0t] y(t) = ℜ[˜yejω0t] Where ω0 is the carrier frequency and ˜x and ˜y are the envelope of the input and output signal respectively, a discrete Volterra model of K=3is given as: ˜y(n) = M ∑ i=0 ˜ h1(i)˜x(n−i) + M ∑ i1=0 M ∑ i2=0 M ∑ i3=0 ˜ h3(i1,i2,i3)˜x(n−i1)˜x(n−i2)˜x∗(n−i3) +... Where ˜ hp(i1,...,ip) is the complex Volterra kernel and ()∗ represent the conjugate transpose. Even order kernels and the items associated with kernel symmetry are removed on the equation for clarity. With conventional Volterra series, the number of parameters increases drastically with the nonlinearity order and the memory depth. To fix this, several techniques have been proposed to simplify the Volterra model such as [60, 10]. 3.3 Behavioral modeling and Digital Predistortion 27 Figure 3.7 Coefficient exponential growth on Volterra series [58] . The Memory Polynomial Model (MP) the simplest form of the Volterra series, which can be described as a Taylor series with memory: y(n) = K ∑ j=1 M ∑ i=0 ai jx(n−i)|x(n−i)|j−1 Where x(n) is the input signal, ai j are the coefficients of the model, M represents the memory depth of the model, and K represents the nonlinearity polynomial order. Some forms of the MP, with varying complexity, have been proposed in the literature, such as Generalized Memory Polynomial Models (GMP) or orthogonal polynomial models [ 42 ]. Comparison between model is however delicate, with several metrics being proposed for model comparison such as Normalized Mean Squared Error (NMSE), memory effects modeling ratio (MEMR) and weighted error-to-signal power ratio (WESPR), with the NMSE metrics being the most prominently used in research. Although Volterra series and its forms perform very well, it suffers from limitations and tradeoffs we have to choose from including complexity, speed, ease of identification, etc... This, and because of the exponential increase of the polynomials, makes calculating strong nonlinear, highly polynomial order models computationally hard. Often, even-order terms of Volterra series are ignored since their effects fall outside of the frequency band if interest, however, these components may generate bias in the PA bias and supply line and create memory effects[58]. The aim with NNs and their Universal Approximation Theorem is to create an unified model able to tackle all the PAs nonlinearities, dc offset and IQ imbalance included, while keeping the complexity of the model to a reasonable level to allow the computations to be fast on DPD systems. We perform all this modeling for digital-predistortion purposes (from now DPD), where distortion is added to the input signal to cancel the inherent nonlinearities of the PA. We can represent a linear amplifier based on digital pre-distortion as: 34 Capítulo 4. Experimental design code on the cloud, backed by GPUs and with a cell-like structure a lot like Jupyter’s Notebooks [ 31 ]. Training was performed with NVIDIA Tesla K80 GPUs across all models. GPUs are significantly faster when training neural networks, specifically CNNs, because they allow for parallel computation on their CUDA cores of tensor operations. Training on GPUs was on our experience more than two times faster than using CPUs. The libraries we used for the experiment are: import numpy as np import pandas as pd import seaborn as sns %matplotlib inline from sklearn.model_selection import train_test_split from torch.utils.data import Dataset, DataLoader import torch import torch.nn as nn import torch.nn.functional as F import matplotlib.pyplot as plt import glob from sklearn.preprocessing import MinMaxScaler Input data is stored on CSV files with each column representing the components of a given input, such as: Figure 4.2 Example of Pandas Dataframe . Data handling is performed using the Pandas library [ 45 ]. Pandas is a very potent Python library for data analysis and handling. Pandas stores data in a Dataframe object, which looks a lot like a matrix, and allows fast, efficient and complex computation over the data. It allows slicing just like in Numpy or Matlab, indexing and subsetting the dataset and a very robust group by functionality to define our own functions to split and process data. Although Pandas is a Python library, it does most of the heavy-lifting on C. Lastly, Pandas allows writing and reading data to and from a variety of different file types. Pandas is the industry standard in machine learning when it comes to tabular data, data is, 2D data. 35 Figure 4.3 example of Pandas Dataframe with the “All space missions" Dataset. Along with Pandas, we are using the library Seaborn and Pyplot [ 25 , 57 ] for data visualization. Seaborn is built on top of Pyplot and allows fancy graphs of data, although we are only going to use it to plot hyperparameter search data and loss graphs. Figure 4.4 Some of the plots available on Seaborn. To perform the data preprocessing we will need to do, we will use the library scikit-learn [ 47 ], a very powerful Python library with lots of different tools for data analysis and prediction. It comes with some built-in complex models such as Random Forests [ 5 ], Gradient Boosting [ 4 ] and tools for data preprocessing and classification. As mentioned though, we are only going to make use of the component model_selection and the MinMaxScaler to scale our input data between [0,1] , more explanation on this further into the document. Our Deep Learning framework of choice is Pytorch, developed and maintained by Facebook [ 46 ], and there’s a few reasons behind it. First and foremost, is the framework I have already experienced with, having used it before for a few deep learning projects of my own. Also, Pytorch is considerably more popular than its main competitor, Google’s Tensorflow, on the academic field [3]: In the plot above, we can see the ratio between Pytorch papers and papers that use either Tensorflow or Pytorch at the top research conferences on the AI industry. The lines having a positive slope mean that there was a majority of Pytorch papers in each conference. Pytorch has experienced an exponential growth compared to Tensorflow in the past few years and it is for a number of reasons: Pytorch is very similar to Python and Numpy, making it very easy to understand and allowing users to tune a variety of parameters of their models in a familiar way, while Tensorflow behaves a lot like a black-box, where interaction with the model is restricted and calling a method may 36 Capítulo 4. Experimental design Figure 4.5 Mentions of Pytorch on research papers [3] . Figure 4.6 Growth of Pytorch and Tensorflow mentions on research [3] . be doing several operations in the background the user is not aware of. Pytorch allows to create complex models from the ground up with it’s “Layer” system and backpropagation mechanics are easy to understand and customize. Finally, Pytorch seems to be as fast or faster than Tensorflow [53]. To learn a bit more about how Pytorch operates, please check appendix 1. The last library we are going to use in our experiments for hyperparameter search is Optuna. As we mentioned, on a given NN there are a few parameters that have to be chosen empirically to find the best NN configuration for a desired problem, there’s no rule of thumb to choose the optimal number of neurons on a given layer. Some of this hyperparameters are: •Number of neurons on a layer •Depth of the network (number of layers) 37 •Learning rate •Size of the kernel on a CNN •Number of filters on a CNN We have to run experiments with different values of those parameters to choose the best architecture, and Optuna is designed for just that. Optuna samples from given ranges of hyperparameters and searches for the ones with better results given a specific metric. More on Optuna on the second appendix, but here’s a snippet of how Optuna works. We only need to define an objective function with the code to optimize and the possible values and Optuna will run it finding the best match import optuna def objective(trial): x=trial.suggest_uniform('x',-10,10) return (x - 2)** 2 study =optuna.create_study() study.optimize(objective, n_trials=100) study.best_params # E.g. {'x': 2.002108042} Now that we have explained what each library is for, let’s start with the code rundown. After loading the data using Pandas, we have the input data on a Pandas Dataframe called X and the output data in the Dataframe y, which for our first experiment look like this: Figure 4.7 Input and output Dataframes. The .head() method allows us to see the first entries on a given Dataset. Now, since we are going to make use of some transformations of the input data to train our network, we can add the columns like this: 38 Capítulo 4. Experimental design X['mode1']=((np.sqrt((X['real']**2)+(X['imag']**2)))) X['mode2']=((np.sqrt((X['real']**2)+(X['imag']**2)))**2) X['mode3']=((np.sqrt((X['real']**2)+(X['imag']**2)))**3) X['mode4']=((np.sqrt((X['real']**2)+(X['imag']**2)))**4) X['mode5']=((np.sqrt((X['real']**2)+(X['imag']**2)))**5) Here, we are creating a column on the DataFrame X and assigning values to it. We are performing operations with the entire columns here, which means, Pandas is performing the operations elementwise. On that snippet of code, the columns created are the ones corresponding to: X[”mode1”] = |Xin| X[”mode2”] = |Xin|2 X[”mode3”] = |Xin|3 X[”mode4”] = |Xin|4 X[”mode5”] = |Xin|5 And the Dataframe with the extra columns looks like this: Figure 4.8 Dataset of inputs with the extra columns . The next step we are going to perform is data preprocessing. More specifically we are going to re-scale our data to the [0,1] range, column by column. Scikit-learn maxes this operation easy by the MinMaxScaler object: scaler_x =MinMaxScaler() scaler_y =MinMaxScaler() X=scaler_x.fit_transform(X.values) y=scaler_y.fit_transform(y.values) Each value on a given column can be expressed as: 39 Xi−min(X) max(X)−min(X) Where min(X) and max(X) are the minimum and the maximum value on a given column. Performing this operation is good practice in machine learning. We perform this operation to stop the magnitude of a feature from disturbing the error calculation. Imagine that we have a feature on an example whose value is ‘5Kg’, but we decide to write it as ‘5000g’. The vector representing that example on the feature space is going to be governed by the 5000g feature if the rest of the features have small values, and since we are going to use the euclidean distance to calculate the error, the network is going to focus on fixing the high magnitude feature instead if giving equal importance to all the features of the dataset. Here, feature scaling is specially important since as seen on Figure 3.8, the added features are exponentially smaller because they are powers of the I/Q components. With that said, the Dataframe now looks like this: Figure 4.9 Dataset scaled to the range [0, 1] . Next, we define the neural network. We do so by using the torch.nn module from Pytorch and creating a class for it: l_in = 10 class neuralNet(nn.Module): def __init__(self, n_channels, n_hidden, k_size, n_dimensions_in): super(redNeuronal, self).__init__() self.n_channels =n_channels self.k_size =k_size self.l_out =(l_in-k_size)+1 self.conv1 =nn.Conv1d(n_dimensions_in, n_channels, k_size, padding=(k_size-1)) self.fc1 =nn.Linear((self.l_out+2*(k_size-1))*n_channels, n_hidden) self.real =nn.Linear(n_hidden, 1) self.imag =nn.Linear(n_hidden, 1) def forward(self, X): x=F.leaky_relu(self.conv1(X)) x=F.leaky_relu(self.fc1(x.view(-1, (self.l_out+2*(self.k_size-1)) 40 Capítulo 4. Experimental design *self.n_channels))) out_real =F.leaky_relu(self.real(x)) out_imag =F.leaky_relu(self.imag(x)) return out_real, out_imag First we define the layers our network is going to have. Our network is formed by a layer of convolutional neurons, followed by a fully connected layer and it’s going to have two outputs, one for each I and Q component of the output of the power amplifier. The parameters we define on the initialization method are the following: •n_channels:number of kernels we want the network to learn from the input data. •k_size:depth of the network (number of layers) •l_out: length of the output of each kernel. The length of each input l_in is 10, that is, we are taking into account the present sample and 10 past samples to train the neural network. •conv1: the convolutional layer. We use torch.nn.Conv1d to create it, and it requires as inputs the dimensions of the input, the number of kernels to use, the size of each kernel and we add padding to the input so its length is consistent. Padding is just adding a set of ‘0’ at the beginning and end to the input vector so the convolution is consistent. •f c1 , real , imag : these are the fully connected layers we define for the neural network. We need to define the dimensions of the input and output of each one of them. On the first one, the number of dimensions in is going to be: (lout +padding ∗2)·nchannels And the output dimensionality is going to be determined by the paramete n_hidden . The other two layers simply define each output of our neural network, with n_hidden as their input dimensionality and 1 as their output. Here, we could also use one Linear layer instead of real and imag but with 2 as the output dimensionality, the result is exactly the same. We choose as activation function the leaky ReLU function. We do so to prevent the dying ReLU problem [ 38 ], where if the input is too negative, the gradient is going to be 0 and the neuron is going to produce 0 as output for ever, leading to an useless neuron. The leaky ReLU function is the following: f(x) = (αx x <0 x x >0 Where αis a small constant. The plot of the leaky ReLU is the following: 41 −4−2 0 2 4 −4 −2 0 2 4 Figure 4.10 Leaky ReLU plot. The class neuralNet returns then two vectors of shape (batchsize,1) representing the real and imaginary value for each example we feed in. The decisions on this architecture are made empirically. We chose the depth of the architecture by performing several experiments with varying depth and at high depths the network experienced vanishing gradients, so the weights on the layers near the output are not being updated [ 18 ]. We also chose the length of the input data by following [ 56 ], where the memory depth is 3 , being inputs further back less relevant to the behavioral modeling. Length 10 was a number with high enough impact for us to measure as well as keeping the complexity and number of parameters reasonable. As we see, that leaves the number of channels and number of hidden neurons as variables of our architecture. The optimal values of these variables vary from experiment to experiment, so we are using hyperparameter tuning to optimize these values, see appendix 2. The next step then is to define the Dataset class, which is going to help us iterate over the Dataset. We define the class as follows: class _data_(Dataset): def __init__(self, X, y): self.X=X self.y=y def __len__(self): return self.X.shape[0] def __getitem__(self, idx): return self.X[idx], self.y[idx] batch_size = 220 We can find more information about the Dataset class on appendix 1, but we are basically defining here how to return the data and how to calculate the total length of the dataset. This is required by Pytorch to iterate over the data. Here we also define the batch size. We chose the batch size to be 220 since it gave us a big enough set of examples to perform gradient steps without the loss oscilating too much. Now we define the core functions for the experiments. The first function is the training loop for the network, which looks like this: def train_network(network, optimizer, scheduler, X_train, y_train, total_epochs, print_epoch=False): 42 Capítulo 4. Experimental design datos =_data_(X_train, y_train) loader =DataLoader(dataset=datos, batch_size=batch_size, shuffle=True) train_loss =[] for epoch in range(total_epochs): for x, y_true in loader: network.cuda() x, y_true =x.cuda(), y_true.cuda() optimizer.zero_grad() out_real, out_im =network(x.float()) loss =((((out_real-y_true[:, 0].view(-1,1))**2)+( (out_im-y_true[:, 1].view(-1,1))**2)).mean())/2 loss.backward() optimizer.step() train_loss.append(loss.detach().cpu().numpy()) scheduler.step() if print_epoch and epoch%50==0: print(epoch) This function takes as inputs the network to train, an optimizer, an scheduler, the input and output sets and the number of epochs to train the network for. First, it creates the loader to iterate over the input/output sets, more information on loaders on appendix 1. Then, we set the network and input/output tensors on CUDA mode for training on GPUs, clear all the gradients and compute the outputs of the neural network. After that, we calculate the half MSE and store its value on a single-valued tensor called loss. We call the backward method on the tensor, which calculates all the gradients of the network and then call optimizer.step() to update all the weights on the neural network. Then, we save the loss on an Arraylist in case we want to plot it for model evaluation. Lastly, we call scheduler.step() to lower the learning rate of the optimizer. We do this to prevent overshooting the minima of the loss function. Pytorch has scheduler objects which, given an optimizer will gradually reduce the learning rate over the epochs so we get finer adjustments on the later stages of the training loop. The next function we define is the test function. The test function evaluates the network on previously unseen examples to test its generalization capabilities, that is, to see how well a trained network performs: def test_network(network, X_test, y_test): with torch.no_grad(): preds =network(torch.from_numpy(X_test).cuda().float()) preds =np.concatenate( (preds[0].detach().cpu().numpy(), preds[1].detach().cpu().numpy()), axis=1) preds =scaler_y.inverse_transform(preds) y_test =scaler_y.inverse_transform(y_test) y_test =pd.DataFrame(y_test, columns=["real","imag"]) y_test['real_pred']=preds[:, 0] y_test['imag_pred']=preds[:, 1] y_test['error_abs']=np.sqrt(((y_test['real']-y_test['real_pred'])**2)+( (y_test['imag']-y_test['imag_pred'])**2)) y_test['error']=(y_test['error_abs']**2).mean() y_test['norma_abs']=np.sqrt((y_test['real']**2)+(y_test['imag']**2)) y_test['norma']=(y_test['norma_abs']**2).mean() 43 y_test['error/norma']=y_test["error"]/y_test['norma'] y_test['nmse']= 10*np.log10(y_test['error/norma']) return y_test['nmse'].mean() Given a network and an input/output sets, the function first disables the gradients with the line with torch.nograd() . Pytorch keeps track of all the computations on tensors to compute the gradients later, and by setting that clause we are telling Pytorch to ignore the operations made there, since we are testing the network, not training it. Next, we apply the inverse transformation on the data that we scaled between [0,1] to calculate the NMSE. Then, we create a DataFrame to hold the values of errors and absolute values of the NMSE function and make the calculations column-wise. Lastly, we return the total NMSE for that sample. The last helper function we are going the define is called hyperparameter_search , and it looks like this: def hyperparam_search(network, optimizer, scheduler, trial, X_train, y_train): X_train, X_val, y_train, y_val =train_test_split( X_train, y_train, shuffle=False, train_size=0.90) all_nmse =[] for epoch in range(15): train_network(network, optimizer, scheduler, X_train, y_train, 1,False) nmse =test_network(network, X_val, y_val) all_nmse.append(nmse) trial.report(np.mean(all_nmse), epoch) if trial.should_prune(): raise optuna.TrialPruned() if epoch%25==0: print(epoch) print(all_nmse) return np.mean(all_nmse) This function first splits the given X and y sets into a training and validation set. We train the network on a set and for each epoch we test the network on the validation set and add the value to an Arraylist . On this method there’s also defined the trial.report and trial.shouldprune() needed for a trial pruning of the Optuna module. Basically Optuna can stop a trial for its hyperparameter search when it’s not successful enough to save computation time, and it requires both methods to be called on the hyperparameter search loop. The function lastly returns the mean of all the values stored on the Arraylist for hyperparameter selection. Now, onto the main loop of the experiments, which looks like this: columns_for_experiments =[[0,1], [0,1,2], [0,1,3], [0,1,4], [0,1,2,3], [0,1,2,4], [0,1,2,3,4], [0,1,2,4,6], [0,1,2,3,4,5]] for elem in series_no_indexes: print("With columns:") print(elem) data_to_input =np.zeros((X.shape[0]-l_in, len(elem), l_in)) counter = 0 for column in elem: for index in range(X.shape[0]-l_in): data_to_input[index, counter, :] =X[index:index+l_in, column].transpose() counter += 1 X_train, X_test, y_train, y_test =train_test_split(data_to_input, y[:-l_in], shuffle=True, train_size=0.90) 50 Capítulo 5. Experiment Results (19) With no additional inputs. (20) With |X(n)|. (21) With |X(n)|2.(22) With |X(n)|3. (23) With |X(n)|,|X(n)|2. 5.4 Experiment 4 - LTE at 15MHz at -35.5 dBm 51 (24) With |X(n)|,|X(n)|3.(25) With |X(n)|,|X(n)|2,|X(n)|3. (26) With |X(n)|,|X(n)|3,|X(n)|5.(27) With |X(n)|,|X(n)|2,|X(n)|3,|X(n)|4. Performance of the CNN is shown on the table below: Amplitude Terms (In addition to Iand Q) Optimal No. of kernels Optimal No. of neurons on hidden layer NMSE(dB) None 100 20 -48.36 |X(n)|80 30 -49.22 |X(n)|280 10 -48.59 |X(n)|380 40 -48.27 |X(n)|,|X(n)|2100 40 -49.54 |X(n)|,|X(n)|350 40 -49.15 |X(n)|,|X(n)|2,|X(n)|380 10 -50.19 |X(n)|,|X(n)|3,|X(n)|550 10 -50.20 |X(n)|,|X(n)|2,|X(n)|3,|X(n)|460 30 -50.25 5.4 Experiment 4 - LTE at 15MHz at -35.5 dBm Again, LTE signals where generated as in [ 10 ]. The same amplifier as in Experiment 3 is used but now the power is lowered to -35.5 dBm 52 Capítulo 5. Experiment Results (28) With no additional inputs. (29) With |X(n)|. (30) With |X(n)|2.(31) With |X(n)|3. (32) With |X(n)|,|X(n)|2. 5.5 Comments on the experiments 53 (33) With |X(n)|,|X(n)|3.(34) With |X(n)|,|X(n)|2,|X(n)|3. (35) With |X(n)|,|X(n)|3,|X(n)|5.(36) With |X(n)|,|X(n)|2,|X(n)|3,|X(n)|4. Performance of the CNN is shown on the table below: Amplitude Terms (In addition to Iand Q) Optimal No. of kernels Optimal No. of neurons on hidden layer NMSE(dB) None 90 30 -50.64 |X(n)|70 20 -51.21 |X(n)|280 40 -50.95 |X(n)|3120 20 -50.83 |X(n)|,|X(n)|270 20 -52.09 |X(n)|,|X(n)|3120 10 -51.98 |X(n)|,|X(n)|2,|X(n)|3120 10 -52.02 |X(n)|,|X(n)|3,|X(n)|5110 30 -52.23 |X(n)|,|X(n)|2,|X(n)|3,|X(n)|4110 20 -52.76 5.5 Comments on the experiments Results of the presented experiments confirm that adding more terms to the input of the CNN does come with a small increase in performance, as referenced in [ 56 ]. Furthermore, the inclusion of the term |X[n]| allows the network to generate the even-order intermodulation terms, reaching lower NMSE scores, but adding redundant terms to the network such as |X[n]|2 , |X[n]|3 does not improve the networks performance as much. However, the more combinations we add to the input network, 54 Capítulo 5. Experiment Results the better it performs, as we can see in the last rows of the experiments’tables, bringing around -2dB of improvement compared with feeding the network only the I/Q components. We can also observe that an increase in the networks complexity, or the number of parameters and kernels, does not come with an increase in performance. We can see that a higher input dimensionality does not imply a higher complexity, with simpler networks architecture outperforming more complex ones with different input dimensionality. As we can see, hyperparameter selection on deep learning is still an empirical approach, but recent research on AutoML [ 23 ] may help find a more comfortable solution to current machine learning pipelines. 6 Conclusion On this document we observed that CNNs can model a variety of amplifiers and powers regardless of where distortions come from, all while mantaining the number of learning parameters low. The network architectures explored on this document are not very complex, which allows us to fine-tune them to adapt better to each experiment, as we saw on the hyperparameter search section. We also confirmed previous research that a higher number of feature transformations of the input, more specifically adding |X(n)|k terms, greatly increase the accuracy of networks with similar architectures. 6.1 Future lines of investigation More sophisticated architectures of CNNs, such as TCN [ 9 ], which include the use of Residual blocks and several convolutional layers may have greater success on RFPA behavioral modeling comparable or surpassing current numerical approaches. Although they include a higher number of parameters and overall complexity, their ability to model accurately memory effects by causal convolution may come with better overall accuracy. More research will be needed on this topics to compare performances. 55 Appendix A Pytorch In this section we are going to explain a bit what Pytorch is more in depth and how it works. Pytorch is our library of choice for the neural network architecture we proposed and its training loop. The building block of Pytorch is the tensor. A tensor can be described as a multidimensional matrix. Pytorch provides several methods to create tensors, such as the .zeros() method, .randn() method or .empty() method, and also creating tensors from Numpy arrays. x=torch.zeros(5,3) print(x) tensor([[0.,0.,0.], [0.,0.,0.], [0.,0.,0.], [0.,0.,0.], [0.,0.,0.]]) Performing operations on tensors is really simple, it looks a lot like performing operations with Numpy arrays, so it’s very ‘pythonic’: x=torch.zeros(3,3) y=torch.randn(3,3) z=x+y print(z) tensor([[-0.1215,0.0835,-0.7474], [-0.1995,-0.4273,-0.3058], [-0.0596,-0.1303,0.5063]]) When it comes to performance and fast computation, Pytorch allows tensors to be run on CUDA devices. CUDA (Compute Unified Device Architecture) [ 43 ] is a parallel computing platform developed by NVIDIA. CUDA devices are essentially NVIDIA GPUs highly optimized for Deep Learning and tensor operations. CUDA-enabled GPUs offers a set of low-level, high performance instructions and exposes said API to a variety of programming languages. When we want operations to be performed by a CUDA-enabled device, we just have to call .cuda() on a given tensor or model. This will tell Pytorch to move those objects to the GPU as well as all the tensors resulting from their operations. 57 58 Appendix A. Pytorch Since performing operations with CUDA is really straight-forward, we can write code that runs either on GPU or CPU seamlessly, we just need to check if a CUDA device is available to the system and call the .cuda() method on our tensors and models: if torch.cuda.is_available(): device ='cuda' else: device ='cpu' x=torch.randn(4,4, device=device) print(x) tensor([[ 0.4288,-0.2511,-1.3960,-1.4391], [0.7458,-0.0637,0.4903,-0.6889], [-1.0640,0.5303,-2.5384,-0.2693], [-0.6191,0.1749,-0.5994,-0.2116]], device='cuda:0') Pytorch CUDA implementation offers some other low-level operations to the user to fine tune different aspects such as memory allocation and distributed computation across several GPUs. Pytorch’s autograd package is the core package when working with neural networks. Autograd provides automatic differentiation for all operations on tensors. All operations we perform on a tensor are tracked by Pytorch. When we calculate the Loss function and call backward() on the resulting tensor, all the gradients are computed automatically and stored on an attribute called grad on each tensor. We can choose not to track operations by using the detach() method on a tensor or using the sentence with torch.nograd(). with torch.no_grad(): preds =network(torch.from_numpy(X_test).cuda().float()) preds =np.concatenate((preds[0].detach().cpu().numpy(), preds[1].detach().cpu().numpy()), axis=1) Pytorch also stores the operation on the tensor on an attribute called grad_fn that stores the Function reference that created the tensor: x=torch.zeros(2,2, requires_grad=True) y=x+2 print(y) tensor([[2.,2.], [2.,2.]], grad_fn=<AddBackward0>) Autograd is an engine for computing vector-Jacobian products. Given ~y=f(~x) , the gradient of ~y with respect to~xis the Jacobian matrix: J=    δy1 δx1... δy1 δxn . . ..... . . δyn δx1... δym δxn     For a given vector: v= (v1,v2,...,vm)T 59 Autograd valculates vT∗J. If v is the gradient of a scalar function, that is, i=g(~y), then: v= ( δi δy1 ,..., δi δym )T Then the vector-Jacobian dot product would be the gradient of i with respect to would be: JT∗v=    δy1 δx1... δy1 δxn . . ..... . . δyn δx1... δym δxn       δi δy1 . . . δi δym   =   δi δx1 . . . δi δxm    Thus building computing graphs and its derivatives becomes really easy using vector-Jacobian products. More information about specifics on Autograd can be found on [1]. To build neural networks architectures, Pytorch provides the nn.Modules library and a method forward(input) that produces an output. The general training procedure of a neural network goes like this: defineNetwork(); for number of epochs do for batch of data do output = network(input) error = calculateLoss(output, realValues) calculateGradients(error) adjustWeights() end end Algorithm 1: Training loop of neural network To define a network on Pytorch, we have to create a class which inherits from nn.Module and define two methods: the constructor or initialization method were we’ll define the layers of our network, and the forward method, where we’ll define the order of computation of the layers of the neural network. We can take the example of convnet from Pytorch’s documentation: Figure A.1 Convnet general architecture . The network is a 2D CNN for image recognition. Two convolutional layers are needed, along with two fully connected layers and two pooling layers. The pooling layers basically downsize the feature maps to capture more general features. The code to define the neural network above would be: Figures 2.1 Tesla’s Autopilot interpretation of its environment. Here, the input is the different images captured by the cameras and the networks output is represented by the different lines painted over the road identifying each object and path 3 2.2 Diagram of a single neuron. The inputs are represented by the xvector and the output is represented by the y∗scalar. Any given neuron can have as many inputs as desired. The wvector represents the weights associated with the neuron, and brepresents the bias of the neuron 4 2.3 Diagram of a simple neural network. All the neurons are connected by the weights parameters, the bias bhere is removed for clarity 5 2.4 Simple Dataset of the NBA players. The Dataset contains 31 columns, that is, 31 features for each player. 6 2.5 ReLU activation function 7 2.6 Left: Sigmoid activation function. Right: Tanh activation function 8 2.7 Left: Identity activation function. Right: Binary Step activation function 8 2.8 Example of a non-linear function that a neural network with no activation function couldn’t approximate 9 2.9 Hypothetical plot of the Loss function and w1 11 10 2.10 Very good example of how different learning rates affect the learning process. J(θ)represents the Loss and θrepresents the weight parameter. Source [27] 10 2.11 When calculating the derivatives, we make use of the chain rule. The backprop algorithm simply creates computing nodes describing how to calculate the derivatives. Source [18] 11 2.12 K-Folds Cross Validation visualization [47] 12 2.13 Random Dataset plot [7] 12 2.14 Good fit of the dataset [7] 12 2.15 Overfitting model [7] 13 2.16 Dropout visualization [14] 13 2.17 Underfitting model [7] 13 2.18 Momentum on ADAM. ADAM performs larger weight updates when the gradients points to the same directions over several batches [?] 15 2.19 Example input for 1D convolution 15 2.20 Kernel calculating the feature map for the vector input 16 2.21 Performance of different models on the image-recognition CIFAR-10 dataset 18 2.22 Residual block, the main component of the ResNET [21] 18 2.23 Building blocks of TCN. Left 3 dilated convolutional layers with d=1,2,4and kernel size k=3.Middle TCN residual block architecture for TCN. Right Residual connection on TCNs[9] 19 67 68 Figures 3.1 Power inputs and output on an amplifier. Source [48] 21 3.2 Power in the input plotted against the output power and the gain. Source [58] 22 3.3 AM/AM distortion on a class J amplifier. X axis represents the input power (dBm) while the Y axis represents the AM/AM gain (dB). We can clearly see the nonlinear behaviour of the amplifier, especially towards higher power inputs 24 3.4 AM/PM distortion on a class J amplifier. X axis represents the input power (dBm) while the Y axis represents the AM/PM phase shift (dB). Again, the system behaves in a nonlinear fashion more clearly seen on higher input powers 24 3.5 Past signal influence current-time values on systems with memory [58] 25 3.6 Rise and fall times behave differently because of memory effects [58] 25 3.7 Coefficient exponential growth on Volterra series [58] 27 3.8 DPD component with a RFPA 28 3.9 Simple Digital pre-distortion. Source [58] 28 3.10 Polar neural network 29 3.11 Cartesian neural network 29 3.12 Cartesian neural network 30 3.13 Diagram of ARVTDNN 32 4.1 Proposed 1D-CNN architecture for behavioral modeling 33 4.2 Example of Pandas Dataframe 34 4.3 example of Pandas Dataframe with the “All space missions" Dataset 35 4.4 Some of the plots available on Seaborn 35 4.5 Mentions of Pytorch on research papers [3] 36 4.6 Growth of Pytorch and Tensorflow mentions on research [3] 36 4.7 Input and output Dataframes 37 4.8 Dataset of inputs with the extra columns 38 4.9 Dataset scaled to the range [0, 1] 39 4.10 Leaky ReLU plot 41 A.1 Convnet general architecture 59 B.1 Hyperparameter importance [6] 64 B.2 Contour plot [6] 65 B.3 Contour plot [6] 65 Bibliografía [1] Pytorch autograd, https://pytorch.org/tutorials/beginner/blitz/autogradtutorial.html. [2] Google colaboratory, https://research.google.com/colaboratory/faq.html. [3] Deep learning frameworks, https://thegradient.pub/state-of-ml-frameworks-2019-pytorch- dominates-research-tensorflow-dominates-industry/. [4] Gradient boosting, https://towardsdatascience.com/understanding-gradient-boosting-machines- 9be756fe76ab. [5] Random forests, https://towardsdatascience.com/understanding-random-forest-58381e0602d2. [6] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, Optuna: A next-generation hyperparameter optimization framework, Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019. [7] Anas Al-Masri, What are overfitting and underfitting in machine learning?, https://towardsdatascience.com/what-are-overfitting-and-underfitting-in-machine-learning- a96b30864690. [8] Onur Avci, Osama Abdeljaber, Serkan Kiranyaz, and Daniel Inman, Structural damage detection in real time: Implementation of 1d convolutional neural networks for SHM applications, Structural Health Monitoring & Damage Detection, Volume 7, Springer International Publishing, 2017, pp. 49– 54. [9] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. [10] J. A. Becerra, M. J. Madero-Ayora, and C. Crespo-Cadenas, Comparative analysis of greedy pursuits for the order reduction of wideband digital predistorters, IEEE Transactions on Microwave Theory and Techniques 67 (2019), no. 9, 3575–3585. [11] N. Benvenuto, F. Piazza, and A. Uncini, A neural network approach to data predistortion with memory in digital radio systems, Proceedings of ICC ' 93 - IEEE International Conference on Communications, IEEE. [12] James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl, Algorithms for hyper-parameter optimization, Advances in Neural Information Processing Systems 24 (J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, eds.), Curran Associates, Inc., 2011, pp. 2546–2554. [13] Alberto Brihuega, Lauri Anttila, Mahmoud Abdelaziz, Fredrik Tufvesson, and Mikko Valkama, Digital predistortion for multiuser hybrid mimo at mmwaves. 69 70 Bibliografía [14] Amar Budhiraja, Dropout in (deep) machine learning, https://medium.com/@amarbudhiraja/httpsmedium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning- 74334da4bfc5. [15] Changsoo Eun and E. J. Powers, A new volterra predistorter based on the indirect learning architecture, IEEE Transactions on Signal Processing 45 (1997), no. 1, 223–227. [16] Steve C. Cripps, Rf power amplifiers for wireless communications, Artech House Publishers, 2006. [17] F. M. Ghannouchi and O. Hammi, Behavioral modeling and predistortion, IEEE Microwave Magazine 10 (2009), no. 7, 52–64. [18] Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning, MIT Press, 2016, http: // www.deeplearningbook.org. [19] M. T. Hagan and M. B. Menhaj, Training feedforward networks with the marquardt algorithm, IEEE Transactions on Neural Networks 5(1994), no. 6, 989–993. [20] Simon Haykin, Neural networks: A comprehensive foundation, Prentice Hall, 1999. [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition. [22] ,Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. [23] Xin He, Kaiyong Zhao, and Xiaowen Chu, Automl: A survey of the state-of-the-art. [24] Sepp Hochreiter and Jürgen Schmidhuber, Long short-term memory, Neural computation 9 (1997), 1735–80. [25] J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in Science & Engineering 9 (2007), no. 3, 90–95. [26] M. Ibukahla, J. Sombria, F. Castanie, and N. J. Bershad, Neural networks for modeling nonlinear memoryless communication channels, IEEE Transactions on Communications 45 (1997), no. 7, 768–771. [27] Jeremy Jordan, Setting the learning rate of your neural network, https://www.jeremyjordan.me/nnlearning-rate/, https://www.jeremyjordan.me/nn-learning-rate/. [28] Diederik P. Kingma and Jimmy Ba, Adam: A method for stochastic optimization. [29] Serkan Kiranyaz, Onur Avci, Osama Abdeljaber, Turker Ince, Moncef Gabbouj, and Daniel J. Inman, 1d convolutional neural networks and applications: A survey. [30] Serkan Kiranyaz, Turker Ince, and Moncef Gabbouj, Personalized monitoring and advance warning system for cardiac arrhythmias, Scientific Reports 7(2017), no. 1. [31] Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing, Jupyter notebooks – a publishing format for reproducible computational workflows, Positioning and Power in Academic Publishing: Players, Agents and Agendas (F. Loizides and B. Schmidt, eds.), IOS Press, 2016, pp. 87 – 90. [32] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (1998), no. 11, 2278–2324. [33] Yann LeCun, Mnist dataset, http://yann.lecun.com/exdb/mnist/. Bibliografía 71 [34] Yann Lecun and Y. Bengio, Convolutional networks for images, speech, and time-series, 01 1995. [35] Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio, Object recognition with gradientbased learning, Shape, Contour and Grouping in Computer Vision, Springer Berlin Heidelberg, 1999, pp. 319–345. [36] Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar, A system for massively parallel hyperparameter tuning. [37] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, Hyperband: A novel bandit-based approach to hyperparameter optimization. [38] Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Karniadakis, Dying relu and initialization: Theory and numerical examples. [39] David López Bueno, Thi Quynh Anh Pham, Gabriel Montoro López, and Pere Lluís Gilabert Pinal, Linealización digital de transmisores mediante redes neuronales no lineales, Simposium Nacional de la Unión Científica Internacional de Radio. "Proceedings URSI", 2019. [40] Warren S. McCulloch and Walter Pitts, A logical calculus of the ideas immanent in nervous activity, The Bulletin of Mathematical Biophysics 5(1943), no. 4, 115–133. [41] Farouk Mkadem, Behavioural modeling and linearization of rf power amplifier using artificial neural networks, 2010. [42] D. R. Morgan, Z. Ma, J. Kim, M. G. Zierdt, and J. Pastalan, A generalized memory polynomial model for digital predistortion of rf power amplifiers, IEEE Transactions on Signal Processing 54 (2006), no. 10, 3852–3860. [43] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek, Cuda, release: 10.2.89, 2020. [44] M. Ohta and K. Yamashita, A chaotic neural network for reducing the peak-to-average power ratio of multicarrier modulation, Proceedings of the International Joint Conference on Neural Networks, 2003., IEEE. [45] The pandas development team, pandas-dev/pandas: Pandas, February 2020. [46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala, Pytorch: An imperative style, high-performance deep learning library, Advances in Neural Information Processing Systems 32 (H. Wallach, H. Larochelle, A. Beygelzimer, F. d ' Alché-Buc, E. Fox, and R. Garnett, eds.), Curran Associates, Inc., 2019, pp. 8024–8035. [47] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011), 2825–2830. [48] J. Pedro and N. B. Carvalho, Intermodulation distortion in microwave and wireless circuits, 2003. [49] M. Rawat and F. M. Ghannouchi, A mutual distortion and impairment compensator for wideband direct-conversion transmitters using neural networks, IEEE Transactions on Broadcasting 58 (2012), no. 2, 168–177. 72 Bibliografía [50] M. Rawat, K. Rawat, and F. M. Ghannouchi, Adaptive digital predistortion of wireless power amplifiers/transmitters using dynamic real-valued focused time-delay line neural networks, IEEE Transactions on Microwave Theory and Techniques 58 (2010), no. 1, 95–104. [51] Sebastian Ruder, An overview of gradient descent optimization algorithms. [52] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, Learning representations by back-propagating errors, Nature 323 (1986), no. 6088, 533–536. [53] Shaohuai Shi, Qiang Wang, Pengfei Xu, and Xiaowen Chu, Benchmarking state-of-the-art deep learning software tools. [54] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, Going deeper with convolutions, Computer Vision and Pattern Recognition (CVPR), 2015. [55] Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal, Deep complex networks. [56] D. Wang, M. Aziz, M. Helaoui, and F. M. Ghannouchi, Augmented real-valued time-delay neural network for compensation of distortions and impairments in wireless transmitters, IEEE Transactions on Neural Networks and Learning Systems 30 (2019), no. 1, 242–254. [57] Michael Waskom, Olga Botvinnik, Drew O’Kane, Paul Hobson, Saulius Lukauskas, David C Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Julian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesbeck, Antony Lee, and Adel Qalieh, mwaskom/seaborn: v0.8.1 (september 2017), September 2017. [58] J. Wood, Behavioral modeling and linearization of rf power amplifiers, 05 2014. [59] Matthew D Zeiler and Rob Fergus, Visualizing and understanding convolutional networks. [60] A. Zhu, J. C. Pedro, and T. J. Brazil, Dynamic deviation reduction-based volterra behavioral modeling of rf power amplifiers, IEEE Transactions on Microwave Theory and Techniques 54 (2006), no. 12, 4323–4332. [61] A. Zhu, M. Wren, and T. J. Brazil, An efficient volterra-based behavioral model for wideband rf power amplifiers, IEEE MTT-S International Microwave Symposium Digest, 2003, vol. 2, 2003, pp. 787–790 vol.2.