Rényi divergence and kernel methods for testing Gaussianity

Rényi divergence and kernel methods for testing Gaussianity Master’s Thesis In partial fulfilment of the requirements for the Master’s degree in Advanced Telecommunications Technologies Author Aniol Martí Espelt Advisor Jaume Riba Sagarra Barcelona, June 2022 Abstract Gaussianity tests have been used for decades in statistics to determine if a dataset is well modeled by a normal distribution. More recently, they have also found their place in machine learning. The reason of this proliferation is that some parametric methods are only applicable to normal distributions. On the other hand, the number of applications of information theory to statistical signal processing has grown over the years. In this thesis we use information-theoretic metrics to test normality. Focusing on a moderate computational complexity, we define Rényi negentropy and propose a plug-in estimator of it based on Kernel Density Estimation (KDE). In order to evaluate the performance of the estimator, we feed it with normal, uniform and triangular data and we compute its Mean Squared Error (MSE). Finally, we establish the initial steps of two Gaussianity detection problems based on the proposed estimator and U-statistics, respectively. ii Resum Els tests de gaussianitat s’han utilitzat durant dècades en estadística per tal de determinar si un conjunt de dades està ben modelat per una distribució normal. Més recentment, també han trobat el seu lloc en l’aprenentatge automàtic. La raó d’aquesta proliferació és que alguns mètodes paramètrics només són aplicables a distribucions normals. D’altra banda, el nombre d’aplicacions de la teoria de la informació al processament de senyal estadístic ha crescut al llarg dels anys. En aquesta tesi fem servir mètriques procedents de la teoria de la informació per elaborar tests de normalitat. Centrant-nos en aconseguir una complexitat computacional moderada, definim la neguentropia de Rényi i en proposem un estimador basat en la kernel density estimation (KDE). Per avaluar el rendiment de l’estimador, l’alimentem amb dades normals, uniformes i triangulars i en calculem l’error quadràtic mig (MSE). Finalment, establim els passos inicials de dos problemes de detecció de gaussianitat basats en l’estimador proposat anteriorment i en U-statistics, respectivament. iii Agraïments Ja tornem a ser-hi. Els agraïments, l’única part de tota la memòria on ets lliure de dir el que vulguis (de fet, tampoc) sense rigor acadèmic. Aquest cop no he tingut tant temps per preparar-los, tanmateix procuraré no deixar-me ningú. En primer lloc, vull donar les gràcies al director d’aquest treball, en Jaume Riba. A part dels consells que m’ha donat, ha tingut molta paciència, especialment quan a l’abril vaig dir-li que volia canviar el tema del treball. També vull donar les meves més sinceres gràcies al Ferran de Cabrera, que ja està acabant la seva tesi, i al Marc Vilà, que tot just la comença. Ells dos no només han aportat idees pel treball, sinó que també m’han fet passar molt bones estones. No em puc oblidar els companys de la DAT, la DAEC i el CdE. Aquest curs he passat moltíssimes hores amb ells i m’ho he passat realment bé. Companys, no sabeu el greu que em sap acabar i haver de marxar de la Planta 0. Finalment, també vull donar les gràcies als amics que durant aquest quadrimestre han estat allà, tant per les bones com per les dolentes. Doncs apa, ja ens tornarem a veure d’aquí quatre anys als agraïments de la tesi. iv Contents List of Figures vii Notation viii Acronyms ix 1 Introduction 1 1.1 Stateoftheart ............................ 2 1.2 Thesisoutline............................. 2 2 Measures of entropy and information 4 2.1 Shannon entropy and Kullback-Leibler divergence . . . . . . . . 4 2.2 Rényi entropy and divergence . . . . . . . . . . . . . . . . . . . . 5 2.2.1 Specialcases ......................... 6 2.3 Negentropy.............................. 7 3 Kernel Density Estimation 8 3.1 Definition............................... 8 3.2 Bandwidth selection . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 Regularizing effect of kernels . . . . . . . . . . . . . . . . . . . . 9 4 Rényi negentropy estimation 11 4.1 Motivation .............................. 11 4.2 Proposed estimator . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.2.1 Removing the dependence on the variance . . . . . . . . . 14 4.2.2 Performance of the estimator . . . . . . . . . . . . . . . . 16 5 Detection of normality based on Rényi negentropy 22 5.1 Detector interpretation of Ĵ(P, v) . . . . . . . . . . . . . . . . . . 22 5.2 U-statistics detector . . . . . . . . . . . . . . . . . . . . . . . . . 23 6 Conclusions and future work 24 6.1 Futurework.............................. 25 Bibliography 26 A Proof of Equation 2.16 29 B Rényi negentropy of uniform and triangular distributions 30 B.1 Uniform distribution . . . . . . . . . . . . . . . . . . . . . . . . . 30 B.2 Triangular distribution . . . . . . . . . . . . . . . . . . . . . . . . 31 v Contents C Additional plots of bias, variance and density estimates 32 C.1 Normaldata.............................. 32 C.2 Uniformdata ............................. 33 C.3 Triangulardata............................ 34 vi List of Figures 4.1 Plots showing the convex behavior of 𝐽(𝑃,𝑣)............ 16 4.2 MSE of 𝐽(𝑃,𝑣)for normally distributed data at different variances. 17 4.3 NMSE of 𝐽(𝑃,𝑣) for uniformly distributed data at different variances. 18 4.4 Bias of 𝐽(𝑃,𝑣)for uniformly distributed data at different variances. 19 4.5 KDE of uniform data for different variances. . . . . . . . . . . . . 19 4.6 NMSE of 𝐽(𝑃,𝑣)for triangular data at different variances. . . . . 20 4.7 Bias of 𝐽(𝑃,𝑣) for triangularly distributed data at different variances. 20 4.8 KDE of triangular data for different variances. . . . . . . . . . . . 21 C.1 KDE of normal data for different variances. . . . . . . . . . . . . 32 C.2 Plots showing the bias and variance of 𝐽(𝑃,𝑣)for normal data. . 33 C.3 KDE of uniform data for different variances. . . . . . . . . . . . . 33 C.4 Plots showing the bias and variance of 𝐽(𝑃,𝑣)for uniform data. . 34 C.5 KDE of triangular data for different variances. . . . . . . . . . . . 34 C.6 Plots showing the bias and variance of 𝐽(𝑃,𝑣)for triangular data. 35 vii Notation aColumn vector named 𝑎. 𝜃Estimate of 𝜃. 𝑓∗𝑔Convolution between 𝑓and 𝑔. 𝐸[⋅] Expected value. arg min 𝑥∈𝑆 𝑓(𝑥)Value of 𝑥that minimizes 𝑓in the set 𝑆. 𝑋∼𝑃The random variable 𝑋follows a distribution 𝑃. 𝑋∼𝒩(𝜇,𝜎2)Gaussian random variable with mean 𝜇and variance 𝜎2. 𝑋∼𝒰(𝑎,𝑏)Uniformly distributed variable between 𝑎and 𝑏. 𝑋∼𝒯(𝑎,𝑏,𝑐)Triangular variable distributed between 𝑎and 𝑏with mode 𝑐. 𝑝𝑋Probability function of a discrete random variable 𝑋. 𝑓𝑋Probability density function of a continuous random variable 𝑋. erfi(𝑥)Imaginary error function defined as 2 √𝜋∫𝑥 0𝑒𝑡2𝑑𝑡. viii Acronyms AWGN Additive White Gaussian Noise. CLT Central Limit Theorem. ISE Integrated Squared Error. KDE Kernel Density Estimation. MISE Mean Integrated Squared Error. MSE Mean Squared Error. NMSE Normalized Mean Squared Error. PDF Probability Density Function. RKHS Reproducing Kernel Hilbert Spaces. ROC Receiver Operating Characteristic. SPCOM Signal Processing and Communications. ix 2.3. Negentropy 2.3 Negentropy Given that negentropy is a fundamental metric of this thesis, we devote an entire section to its definition. In Chapter 4 we present the main drawbacks of its estimation and we take advantage of them to motivate Rényi negentropy. Negentropy is a measure of distance to normality. It is equal to the KullbackLeibler divergence between 𝑃 and a Gaussian distribution 𝑄 with the same variance as 𝑃 . Formally, given 𝑄∼𝒩(𝜇,𝜎2) and 𝑃 a continuous probability distribution with variance 𝜎2, the negentropy 𝐽1(𝑃)of 𝑃is: 𝐽1(𝑃)=∫+∞ −∞ 𝑝(𝑥)log 𝑝(𝑥) 𝑞(𝑥)𝑑𝑥=ℎ1(𝑄)−ℎ1(𝑃) (2.16) where the last equality is proven in Appendix A. As a particular case of the Kullback-Leibler divergence, negentropy has the following properties: •𝐽1(𝑃)≥0, with equality iff 𝑃Gaussian. •𝐽1(𝛼𝑃)=𝐽1(𝑃),∀𝛼∈ℝ. In order to keep these two properties, in Chapter 4 we propose a definition of Rényi negentropy based on the second order Rényi divergence. However, some authors [11] have proposed an alternative definition based on the difference of second order Rényi entropies 𝐽′ 2(𝑃)=ℎ2(𝑄)−ℎ2(𝑃). (2.17) But taking into account that ℎ2 is not maximized by a Gaussian distribution like in the Shannon case, 𝐽′ 2(𝑃)can be negative. 7 CHAPTER 3 Kernel Density Estimation In Chapter 1 we stated that the source of our data is usually unknown, thus we need a non-parametric method to estimate its PDF. Then, we can plug the estimated PDF into any of the metrics described in Chapter 2 in order to obtain an estimate of that metric. There are several ways to estimate a PDF, but one of the most important is KDE, sometimes also termed the Parzen-Rosenblatt window method [20]. In this chapter we present the fundamentals of this technique and we also link it with the idea of noise regularization often used to improve deep learning models [25]. 3.1 Definition Let 𝑥1,𝑥2,…,𝑥𝑁 be a sequence of i.i.d. samples drawn from some distribution with unknown PDF 𝑓𝑋. Then, the estimation of 𝑓𝑋is given by 𝑓𝑋(𝑥)=1𝑁𝑁 ∑ 𝑖=1𝑔ℎ(𝑥−𝑥𝑖)=1 𝑁ℎ𝑁 ∑ 𝑖=1𝑔(𝑥−𝑥𝑖 ℎ)(3.1) where 𝑁 is the number of samples, 𝑔 is the kernel function and ℎ>0 is a smoothing parameter called the bandwidth. The interesting point of this estimator proposed by Parzen is the capability of reconstructing the PDF given a finite amount of random values, hence giving a non-parametric model. It also acts as a local builder, so outliers have a minor impact on the reconstruction. Regarding the kernel 𝑔 , it must fulfill the following properties in order to perform a proper estimation of 𝑓𝑋: •𝑔(𝑥)≥0,∀𝑥. •∫+∞ −∞𝑔(𝑥)𝑑𝑥=1. •lim 𝑥→∞|𝑥𝑔(𝑥)|=0. The first two requirements ensure that the method of KDE results in a PDF. In addition, kernels are usually chosen to be symmetric: 𝑔(𝑥)=𝑔(−𝑥),∀𝑥. 8 3.2. Bandwidth selection There are several window functions that fulfill these properties and are commonly used: uniform, triangular, biweight, triweight, Epanechnikov, normal, etc. It is reasonable to think that, thanks to its convenient mathematical properties, the normal kernel is often used. Another reason is that there exists a simple expression to obtain the bandwidth ℎ if the true PDF has Gaussian shape and the kernels are also Gaussian. 3.2 Bandwidth selection The selection of ℎ is one of the major constraints of the method. A low bandwidth will result on a spiky estimate (high variance) and a high bandwidth into an oversmoothed estimate. A value with good performance can be obtained by sweeping for a range of ℎ , yet it is not a practical solution in terms of computational time. If normal kernels are used and the underlying density being estimated is Gaussian, the optimal solution is a rule-of-thumb first introduced by Silverman on 1986 [26]: ℎ=1.06⋅𝜎⋅𝑁−1/5 (3.2) where 𝜎is the standard deviation of the samples and 𝑁is the sample size. The parameter ℎ obtained using the Silverman rule is optimal in the Mean Integrated Squared Error (MISE) sense. As we are interested in estimating the PDF over the entire real line [27], we need to consider an error criterion that globally measures the distance between 𝑓𝑋and 𝑓𝑋. The Integrated Squared Error (ISE) 𝐼𝑆𝐸(𝑓𝑋)=∫(𝑓𝑋(𝑥)−𝑓𝑋(𝑥))2𝑑𝑥, (3.3) is the squared distance between the estimate and the target density. The ISE is a random quantity, since it depends directly on the samples. Hence, it is more appropriate to work with its expected value, the MISE: 𝑀𝐼𝑆𝐸(𝑓𝑋)=𝐸[𝐼𝑆𝐸(𝑓𝑋)]=𝐸[∫(𝑓𝑋(𝑥)−𝑓𝑋(𝑥))2𝑑𝑥].(3.4) The Silverman rule is obtained by minimizing Equation 3.4 with 𝑓𝑋 Gaussian and 𝑓𝑋estimated with normal kernels. 3.3 Regularizing effect of kernels It can be proven that the expected value of a PDF obtained by means of KDE is equal to the convolution of the true PDF with the kernel used [27]. Formally, given 9 Chapter 3. Kernel Density Estimation a random variable 𝑋 with unknown density 𝑓𝑋 and a kernel 𝑔ℎ with bandwidth ℎ, the expected value of 𝑓𝑋is 𝐸[𝑓𝑋(𝑥)]=(𝑓𝑋∗𝑔ℎ)(𝑥). (3.5) From probability theory we know that given two independent random variables 𝑈,𝑉the PDF of 𝑇=𝑈+𝑉is given by the convolution 𝑓𝑇(𝑡)=(𝑓𝑈∗𝑓𝑉)(𝑡). (3.6) Taking into account that a kernel fulfills the properties of a PDF, we can see 𝑔ℎ as the PDF of a random variable 𝑌 . From Equations 3.5 and 3.6 it follows that using KDE we are estimating the PDF of 𝑍=𝑋+𝑌 . That is, kernels are contaminating our data. If we use normal kernels, then 𝑌∼𝒩(0,ℎ2) and our data is contaminated with Additive White Gaussian Noise (AWGN). Although this behavior may seem undesirable, adding noise to data has been proven to be favorable, as it has a regularizing effect. For this reason, it is often used in deep learning [25, 28]. In addition, it has also been proven that adding Gaussian noise to data results in the optimality of the differential entropy plug-in estimator [29]. The intrinsic regularization produced by KDE provides us some insights that are exploited in Chapter 5, where we propose a Gaussianity detector based on U-statistics that also takes advantage of a regularizing effect. 10 CHAPTER 4 Rényi negentropy estimation As stated in Chapter 1, the main goal of this thesis is to propose a test for Gaussianity that can be used, for instance, to evaluate preprocessing stages of data compressors. In previous chapters we defined negentropy, a measure of distance to normality, and also KDE, a non-parametric way to estimate a PDF. In this chapter we bring together these ideas and propose an estimator of negentropy, we notice its drawbacks and finally we derive a bound of negentropy based on the Rényi divergence of 𝛼=2. 4.1 Motivation Let 𝑃 be a continuous probability distribution with variance 𝑣 . Then, estimating its negentropy is equivalent to estimating its differential entropy and subtract it to the entropy of a Gaussian distribution with variance 𝑣. From Equations 2.3 and 3.1, the estimate of differential entropy is given by ℎ1(𝑃)=−∫+∞ −∞ 𝑝(𝑥)log 𝑝(𝑥)𝑑𝑥 (4.1) which substituting 𝑝(𝑥)becomes ℎ1(𝑃)=−1 𝑁ℎ∫+∞ −∞ 𝑁 ∑ 𝑖=1𝑔(𝑥−𝑥𝑖 ℎ)log (1 𝑁ℎ𝑁 ∑ 𝑖=1𝑔(𝑥−𝑥𝑖 ℎ))𝑑𝑥. (4.2) At this point, it is difficult to further simplify the expression due to the logarithm inside the integral. However, we can use the Rényi divergence to define an alternative negentropy. Recall that the Rényi divergence is non-negative (being zero iff the two distributions are equal), the logarithm is outside the integral and it acts as an upper bound of the Kullback-Leibler divergence. Hence, it makes sense to define the Rényi negentropy. 4.2 Proposed estimator Based on the motivation given in the previous section, we propose an estimator of the Rényi negentropy. 11 Chapter 4. Rényi negentropy estimation Given 𝑄∼𝒩(𝜇,𝑣) and a continuous probability distribution 𝑃 with variance 𝑣 , we define the Rényi negentropy 𝐽(𝑃)as 𝐽(𝑃)=𝐷(𝑃||𝑄)=log ∫+∞ −∞ 𝑝(𝑥)2 𝑞(𝑥)𝑑𝑥. (4.3) Although the logarithm is now outside the integral, the Rényi negentropy has a limitation that Shannon negentropy does not. As 𝑞(𝑥) is a Gaussian PDF, the integral will only converge if 𝑝(𝑥) decays at least as fast as a Gaussian or if it is 0 for all 𝑥>𝑀,𝑀∈ℝ. Let 𝑥1,𝑥2,…,𝑥𝑁 be a sequence of i.i.d. samples drawn from some distribution 𝑃 with unknown PDF 𝑝 , with mean 𝜇=0 and variance 𝑣 also unknown. Let 𝑄∼𝒩(0,𝑣) . Let 𝑔 be a Gaussian kernel with bandwidth ℎ=√𝑏 . Then, an estimator of the Rényi negentropy is given by: 𝐽(𝑃,𝑣)=log ∫+∞ −∞ 𝑝(𝑥)2 𝑞(𝑥)𝑑𝑥=log ∫+∞ −∞ 1 𝑁2 𝑁−1 ∑ 𝑖=0 𝑁−1 ∑ 𝑗=0 1𝑏𝑔(𝑥−𝑥𝑖 ℎ)𝑔(𝑥−𝑥𝑗 ℎ) 𝑞(𝑥) 𝑑𝑥 =log (√2𝜋𝑣 𝑁2∫+∞ −∞ 𝑁−1 ∑ 𝑖,𝑗=01𝑏𝑔(𝑥−𝑥𝑖 ℎ)𝑔(𝑥−𝑥𝑗 ℎ)𝑒𝑥2 2𝑣 𝑑𝑥) =log ⎛⎜⎜⎝√2𝜋𝑣 𝑁2𝑁−1 ∑ 𝑖,𝑗=0∫+∞ −∞ 1𝑏𝑔(𝑥−𝑥𝑖 ℎ)𝑔(𝑥−𝑥𝑗 ℎ)𝑒𝑥2 2𝑣 𝑑𝑥 ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐼 ⎞⎟⎟⎠. (4.4) Focusing on the integral 𝐼, we now write explicitly the kernel 𝑔: 𝐼=∫+∞ −∞ 1𝑏1 √2𝜋𝑒−(𝑥−𝑥𝑖)2 2𝑏 1 √2𝜋𝑒−(𝑥−𝑥𝑗)2 2𝑏 𝑒𝑥2 2𝑣 =1 2𝜋𝑏∫+∞ −∞ 𝑒−((𝑥−𝑥𝑖)2 2𝑏 +(𝑥−𝑥𝑗)2 2𝑏 −𝑥2 2𝑣)𝑑𝑥 =1 2𝜋𝑏∫+∞ −∞ 𝑒−1 2𝑏((𝑥−𝑥𝑖)2+(𝑥−𝑥𝑗)2−𝑏 𝑣𝑥2)𝑑𝑥 =1 2𝜋𝑏∫+∞ −∞ 𝑒−1 𝑏((1−𝑏 2𝑣)𝑥2−𝑥(𝑥𝑖+𝑥𝑗)+𝑥2 𝑖+𝑥2 𝑗 2)𝑑𝑥 =1 2𝜋𝑏∫+∞ −∞ 𝑒−1− 𝑏 2𝑣 𝑏⎛ ⎜ ⎜ ⎜ ⎝𝑥2−2𝑥(𝑥𝑖+𝑥𝑗) 2−𝑏 𝑣 ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐴 +𝑥2 𝑖+𝑥2 𝑗 2−𝑏 𝑣 ⎞ ⎟ ⎟ ⎟ ⎠𝑑𝑥. (4.5) 12 4.2. Proposed estimator We proceed by completing the square of the exponent 𝐴: 𝐴=𝑥2−2𝑥(𝑥𝑖+𝑥𝑗) 2−𝑏𝑣=(𝑥−𝑥𝑖+𝑥𝑗 2−𝑏𝑣)2−(𝑥𝑖+𝑥𝑗 2−𝑏𝑣)2.(4.6) Substituting back into Equation 4.5: 𝐼= 1 2𝜋𝑏∫+∞ −∞ 𝑒−1− 𝑏 2𝑣 𝑏((𝑥−𝑥𝑖+𝑥𝑗 2−𝑏 𝑣)2+𝑥2 𝑖+𝑥2 𝑗 2−𝑏 𝑣−(𝑥𝑖+𝑥𝑗 2−𝑏 𝑣)2)𝑑𝑥 =1 2𝜋𝑏∫+∞ −∞ 𝑒−1− 𝑏 2𝑣 𝑏(𝑥−𝑥𝑖+𝑥𝑗 2−𝑏 𝑣)2𝑑𝑥 ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐺𝑒−1− 𝑏 2𝑣 𝑏(𝑥2 𝑖+𝑥2 𝑗 2−𝑏 𝑣−(𝑥𝑖+𝑥𝑗 2−𝑏 𝑣)2) ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐵.(4.7) Given that the exponential 𝐵 does not depend on 𝑥 , we can solve the Gaussian integral 𝐺:𝐺= 1 2𝜋𝑏√𝜋𝑏 1−𝑏 2𝑣 =√1 4𝜋𝑏(1−𝑏 2𝑣)(4.8) Regarding the exponential 𝐵 , we do some algebra to obtain a cleaner expression: 𝐵=𝑒−1− 𝑏 2𝑣 𝑏1 2−𝑏 𝑣(𝑥2 𝑖+𝑥2 𝑗−(𝑥𝑖+𝑥𝑗)2 2−𝑏 𝑣)=𝑒−1 2𝑏(𝑥2 𝑖+𝑥2 𝑗−𝑥2 𝑖 2−𝑏 𝑣−𝑥2 𝑗 2−𝑏 𝑣−2𝑥𝑖𝑥𝑗 2−𝑏 𝑣) =𝑒−1 2𝑏(2−𝑏 𝑣)((1−𝑏 𝑣)𝑥2 𝑖+(1−𝑏 𝑣)𝑥2 𝑗−2𝑥𝑖𝑥𝑗)=𝑒−1 2𝑏(2−𝑏 𝑣)((𝑥𝑖−𝑥𝑗)2−𝑏 𝑣(𝑥2 𝑖+𝑥2 𝑗)) =𝑒−(𝑥𝑖−𝑥𝑗)2 2𝑏(2−𝑏 𝑣)𝑒𝑥2 𝑖+𝑥2 𝑗 2𝑣(2−𝑏 𝑣). (4.9) Hence, integral 𝐼is given by 𝐼=𝐺⋅𝐵=√1 4𝜋𝑏(1−𝑏 2𝑣)⋅𝑒−(𝑥𝑖−𝑥𝑗)2 2𝑏(2−𝑏 𝑣)𝑒𝑥2 𝑖+𝑥2 𝑗 2𝑣(2−𝑏 𝑣)(4.10) 13 Chapter 4. Rényi negentropy estimation and substituting it in Equation 4.4, the estimator becomes 𝐽(𝑃,𝑣)=log ⎛⎜⎜⎝√2𝜋𝑣 𝑁2𝑁−1 ∑ 𝑖,𝑗=0√1 4𝜋𝑏(1−𝑏 2𝑣)⋅𝑒−(𝑥𝑖−𝑥𝑗)2 2𝑏(2−𝑏 𝑣)𝑒𝑥2 𝑖+𝑥2 𝑗 2𝑣(2−𝑏 𝑣)⎞⎟⎟⎠ =log ⎛⎜⎜⎝1 𝑁2√𝑣 𝑏(2−𝑏𝑣)𝑁−1 ∑ 𝑖,𝑗=0𝑒−(𝑥𝑖−𝑥𝑗)2 2𝑏(2−𝑏 𝑣)𝑒𝑥2 𝑖+𝑥2 𝑗 2𝑣(2−𝑏 𝑣)⎞⎟⎟⎠.(4.11) As the exponent is symmetric in 𝑖 and 𝑗 , half of the sums are repeated. Then, the estimator can be further simplified by 𝐽(𝑃,𝑣)=log ⎛⎜⎜⎝2 𝑁(𝑁+1)√𝑣 𝑏(2−𝑏𝑣)𝑁−1 ∑ 𝑖=0 𝑁−1 ∑ 𝑗=𝑖 𝑒−(𝑥𝑖−𝑥𝑗)2 2𝑏(2−𝑏 𝑣)𝑒𝑥2 𝑖+𝑥2 𝑗 2𝑣(2−𝑏 𝑣)⎞⎟⎟⎠(4.12) This estimator depends on the unknown parameter 𝑣 , so we cannot implement it. In the following section we propose two methods to overcome this issue. 4.2.1 Removing the dependence on the variance The two proposed approaches to remove the dependence on 𝑣 are the following: 1. Estimate the variance 𝑣with the sample variance. 2. Minimize 𝐽(𝑃,𝑣)with respect to 𝑣. We now briefly describe the two methods from a theoretical perspective. In the next section we compare them in terms of MSE and process time. Estimation with sample variance This first method is the classical approach of estimating the variance 𝑣 through the sample variance 𝑣 . From the hypothesis of Equation 4.3, we know that 𝜇=0 . Hence, the following variance estimator is unbiased: 𝑣=1𝑁𝑁−1 ∑ 𝑖=0 𝑥2 𝑖.(4.13) It seems natural to calculate 𝐽(𝑃,𝑣)|𝑣=𝑣 , but the result would not be the desired one. Taking into account that the PDF 𝑝 is estimated using kernels of bandwidth ℎ , the sample variance is contaminated with the kernel variance [30]. That is, the variance that we need to plug in the expression is 𝑣′=𝑣+𝑏: 𝐽𝑆𝑉(𝑃)= 𝐽(𝑃,𝑣)|𝑣=𝑣+𝑏.(4.14) 14 4.2. Proposed estimator Negentropy minimization The second technique proposed is to minimize 𝐽(𝑃,𝑣) with respect to the variance 𝑣 . First, we motivate this approach by justifying why the best estimation is the minimum value. Then, we verify if this global minimum is unique or if there exist other local minima. From a geometric point of view, the family of all normal distributions forms a statistical manifold [31]. Taking into account that the divergence (hence, the negentropy) is a measure of similarity between distributions and that we are measuring it between the estimated PDF and this manifold, we can think of it as a distance. Then, by definition, the divergence between the estimated PDF and the manifold is the divergence between the PDF and the most similar distribution in the manifold [32]. In order to check if the global minimum is unique we study the convexity of 𝐽(𝑃,𝑣) . If it is convex, the minimum is unique. Applying logarithm properties to Equation 4.11 we obtain 𝐽(𝑃,𝑣)=log (1 𝑁2)+12log (𝑣 2𝑏−𝑏2 𝑣) ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐴+log ⎛⎜⎜⎝𝑁−1 ∑ 𝑖,𝑗=0𝑒−(𝑥𝑖−𝑥𝑗)2 2𝑏(2−𝑏 𝑣)𝑒𝑥2 𝑖+𝑥2 𝑗 2𝑣(2−𝑏 𝑣)⎞⎟⎟⎠.(4.15) The first term is a constant that does not depend on 𝑣 , thus it is both convex and concave. The third term is a log-sum-exp function, hence it is convex [33]. Regarding the second term, it can be rewritten as follows: 𝐴=2log (𝑣)−log (2𝑏𝑣−𝑏2)=2log (𝑣)−log(𝑏)−log(2𝑣−𝑏) (4.16) where the first term is concave and the second and third are convex. Thus, we cannot determine if 𝐴 is convex or not. Given that the behavior of 𝐴 is unknown, we cannot know if 𝐽(𝑃,𝑣)is convex either. Going further in this study is out of the scope of this thesis. However, we provide some insights in order to show that the minimum of 𝐽(𝑃,𝑣)is probably unique. First, we prove that 𝐴 is quasiconvex. A function 𝑓∶ℝ𝑛→ℝ is quasiconvex if dom(𝑓)is convex and the sublevel sets 𝑆𝛼={x∈dom(𝑓)|𝑓(x)≤𝛼} are convex for all 𝛼[33]. Applying this definition to 𝐴we obtain log (𝑣2 2𝑏𝑣−𝑏2)≤𝛼⟹𝑣2≤2𝑏𝛽𝑣−𝛽𝑏2⟹𝑣2−2𝑏𝛽𝑣+𝛽𝑏2≤0. (4.17) 15 Chapter 4. Rényi negentropy estimation Thus, 𝑆𝛽={𝑣∈dom(𝐴)|𝑣2−2𝑏𝛽𝑣+𝛽𝑏2≤0} which are convex for all 𝛽∈ℝ , hence 𝐴 is quasiconvex. Unfortunately, the sum of two quasiconvex functions is not necessarily quasiconvex. The second insight we provide is based on a graphical inspection of 𝐽(𝑃,𝑣) . If we plot it for a fixed value of 𝑏 and for 𝑣∈(1,90) we observe a convex behavior (see Figure 4.1a), but it is not clear if it is monotonically decreasing or not. In order to see that it is not, in Figure 4.1b we show the same curve but for 𝑣∈(20,90) and at tinier scale. There, it is clear that the function exhibits a minimum around 55. 0 20 40 60 80 Variance (v) 0 50 100 150 200 (P, v) (a) 𝐽(𝑃,𝑣)plot for 1<𝑣<90. 20 30 40 50 60 70 80 90 Variance (v) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 (P, v) (b) Zoom of 𝐽(𝑃,𝑣)around 20<𝑣<90. Figure 4.1: Plots showing the convex behavior of 𝐽(𝑃,𝑣). We have justified why the minimization of 𝐽(𝑃,𝑣) gives the optimal value. We have also shown that the minimum should be unique. The remaining question is to see if the variance that minimizes the estimator is the sample variance plus the kernel bandwidth or not. That is, we want to know if the following is true: 𝑣𝑚𝑖𝑛=arg min 𝑣𝐽(𝑃,𝑣)=𝑣+𝑏. (4.18) As there does not exist a closed expression for 𝑣𝑚𝑖𝑛 , in the next section we give an answer to this question by comparing both methods. 4.2.2 Performance of the estimator In order to assess the performance of the estimator we have tested it with normal, uniform and triangular distributed data. This selection has been motivated for two reasons: first, to guarantee that the integral in Equation 4.3 converges, and second, because they allow to find a closed expression for the negentropy that is needed to compute the MSE. The calculations for the uniform and triangular distributions can be found in Appendix B. 16 5.2. U-statistics detector 5.2 U-statistics detector The detector proposed in this section also relies on the concept of Rényi negentropy and KDE, but instead of directly estimating the PDF of the original data we propose to estimate the PDF of the difference between i.i.d. pairs of data. For this reason, we cannot propose an estimator and we must work in a detection framework. Let 𝑈=(𝑢1,…,𝑢𝑁) be a sequence of i.i.d samples drawn from some distribution 𝑃 with unknown PDF 𝑝 . Let 𝑈 be a sequence formed by the pairwise differences: 𝑢𝑖,𝑗=𝑢𝑖−𝑢𝑗, 1≤𝑖≤𝑁−1 𝑖+1≤𝑗≤𝑁 (5.3) Taking into account that 𝑢𝑖 and 𝑢𝑗 are independent, the PDF of 𝑈 that we want to estimate is the autocorrelation of the original PDF 𝑝(𝑥)=𝑝(𝑥)∗𝑝(−𝑥). (5.4) Thanks to the pairwise differences, knowing the mean is not required [36] and we are not constrained to zero mean data as it happened in Section 4.2. In addition, we only need to calculate 𝑁(𝑁−1)/2 differences. The reason is that as we are estimating 𝑝with symmetric kernels the sign of 𝑢𝑖,𝑗is not relevant. Now assume the same detection problem as in Section 5.1 taking 𝑥(𝑛)=𝑈 . We can define the following test: 𝑇(𝑃,𝑣)=log ∫+∞ −∞ 𝑝(𝑥)2 𝑞(𝑥)𝑑𝑥ℋ1 ≷ ℋ0𝛾(5.5) where 𝑝 is the KDE of 𝑝 , 𝑞 is a zero mean normal PDF with variance 𝑣 and 𝛾 is the detection threshold. Notice that this detector is well defined as 𝑝 is Gaussian iff 𝑝is also Gaussian. At this point some observations are needed. First, recall that as a consequence of the Central Limit Theorem (CLT), 𝑈 is more Gaussian than 𝑈 . For instance, if 𝑈 follows a uniform distribution, then 𝑈 follows a triangular one, which is more Gaussian (see Appendix B). This result implies that 𝑇(𝑃,𝑣)≤ 𝐽(𝑃,𝑣) . Consequently, for a given 𝑃𝐷 the value of 𝛾 has to be smaller than that of Equation 5.2. In addition, if we take into account the intrinsic convolution performed by KDE with kernels 𝑔ℎ, the PDF that is being estimated is 𝐸[𝑝(𝑥)]=𝑝(𝑥)∗𝑝(−𝑥)∗𝑔ℎ(𝑥) (5.6) that corresponds to a Gaussianization of data plus a regularization. 23 CHAPTER 6 Conclusions and future work This document is the result of over three months of very intensive work. To properly conclude it, in this last chapter we present a full picture of the thesis and summarize what has been achieved and what is still open. In Chapters 2 and 3 we have presented the main tools used in the present thesis. First we have introduced several information-theoretic metrics until we have reached negentropy, the fundamental measure used in this work to test Gaussianity. Then, we have remarked the need for a universal estimator accounting for ignorance regarding the data source. For this reason, we have introduced KDE as a non-parametric way to estimate the PDF of a random variable from data. We have also connected the ideas presented in these chapters with recent research works. For instance, we have related the intrinsic convolution performed by KDE with noise regularization and the optimality of the plug-in estimator for differential entropy proposed by Goldfeld et al. [29]. In Chapter 4 we have motivated the definition of Rényi negentropy as a way to overcome the computational difficulties that Shannon negentropy estimation exhibits due to the logarithm inside the integral. This motivation is not new, since there already exists literature in which similar reasoning has been followed [11]. Next, we have proposed a plug-in estimator of Rényi negentropy using KDE. This technique depends on data, its variance and the kernel bandwidth, and has a complexity of 𝑂(𝑁2) . Given that it depends on the variance but it is unknown, we propose two different methods to solve this problem. The first approach is to compute the sample variance plus the square of the kernel bandwidth and substitute it into the estimator definition. The second method, which is optimal, consists in minimizing the estimator with respect to the variance in order to obtain the estimated value of the Rényi negentropy. We have compared these two methods by feeding normal, uniform and triangular data to the estimator. The results show that a high number of samples is required in order to have low MSE, as the performance of the estimator is closely related to the quality of the PDF estimate. Taking into account these results, in Chapter 5 we have presented the first steps to develop two different Gaussianity detectors based on Rényi negentropy. The first one is based on a detection interpretation of the estimator proposed in Chapter 4. On the other hand, the second one relies on the concept of U-statistics. We show the idea of convolution appears again but now as a Gaussianizing term. 24 6.1. Future work 6.1 Future work This thesis has contributed in broadening the scopes of Rényi negentropy and has also left the door open to many potential lines of research. In reference to the estimator proposed in Chapter 4, there are some points that could be improved or are not properly closed. Let us enumerate them: • Although the estimator appears to be convex (see Figure 4.1 and the equations above), we have not given a rigorous proof of it. It would be of interest to actually prove it. • The minimization method proposed is robust against outliers. However, the sample variance is not, and it is needed to find the kernel bandwidth with the Silverman rule. Thus, studying adaptive models that do not depend on the sample variance to determine the bandwidth would be important to improve the estimator robustness. • The low number of samples used to obtain the results in Section 4.2.2 does not allow us to confirm if the sample variance and the minimization methods are equivalent or not. Performing simulations with more samples would help to get more insights on this topic. • Proving that, asymptotically, the estimator converges to the Rényi negentropy of the PDF convolved with the kernel. This asymptotical analysis would also help to show if the sample variance and minimization methods are equivalent or not. Besides the lines related with the estimator, in Chapter 5 we just presented a detection problem to test Gaussianity, but it was not solved. Thus, developing the two detectors proposed there and implementing them is also a line of research. Finally, this thesis has focused on the theoretical aspects of a Gaussianity test based on Rényi negentropy. For this reason, we have not paid attention to implementation details such as the optimization technique used in the minimization method. However, these aspects should be taken into account and properly tested in an environment where real data is involved. 25 Bibliography [1] C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. [2] S. Cruces, R. Martín-Clemente, and W. Samek, “Information theory applications in signal processing,” Entropy, vol. 21, no. 7, 2019. [3] J. Riba and F. de Cabrera, “Regularized estimation of information via high dimensional canonical correlation analysis,” arXiv preprint arXiv:2005.02977, 2020. [4] F. de Cabrera, J. Riba, and G. Vázquez, “Entropy-based covariance determinant estimation,” in 2017 IEEE 18th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), 2017, pp. 1–5. [5] A. Martí, “Implementation of FAPEC decorrelation stages for IQ and water column data,” Bachelor’s Thesis. [6] P. Comon, “Independent component analysis, a new concept?” Signal Processing, vol. 36, no. 3, pp. 287–314, 1994. [7] A. Martí et al., “Compression of Multibeam Echosounders Bathymetry and Water Column Data,” Remote Sensing, vol. 14, no. 9, 2022. [8] Q. Wang, S. Kulkarni, and S. Verdú, Universal Estimation of Information Measures for Analog Sources. 2009, isbn: 9781601982308. [9] O. Vasicek, “A test for normality based on sample entropy,” Journal of the Royal Statistical Society. Series B (Methodological), vol. 38, no. 1, pp. 54–59, 1976. [10] A. Hyvärinen and E. Oja, “Independent component analysis: Algorithms and applications,” Neural Networks, vol. 13, no. 4, pp. 411–430, 2000. [11] A. Martins, A. Duarte, J. Dantas, and J. C. Principe, “A new clustering separation measure based on negentropy,” Journal of Control, Automation and Electrical Systems, vol. 26, pp. 28–45, 2015. [12] G. Giannakis and M. Tsatsanis, “Time-domain tests for Gaussianity and time-reversibility,” IEEE Transactions on Signal Processing, vol. 42, no. 12, pp. 3460–3472, 1994. [13] T. W. Anderson and D. A. Darling, “Asymptotic Theory of Certain ”Goodness of Fit” Criteria Based on Stochastic Processes,” The Annals of Mathematical Statistics, vol. 23, no. 2, pp. 193–212, 1952. [14] S. S. Shapiro and M. B. Wilk, “An analysis of variance test for normality (complete samples),” Biometrika, vol. 52, no. 3-4, pp. 591–611, 1965-12. 26 Bibliography [15] S. Fang, M. Skoglund, K. H. Johansson, H. Ishii, and Q. Zhu, “Generic variance bounds on estimation and prediction errors in time series analysis: An entropy perspective,” in 2019 IEEE Information Theory Workshop (ITW), 2019, pp. 1–5. [16] S. Fang, J. Chen, and H. Ishii, Towards Integrating Control and Information Theories: From Information-Theoretic Measures to Control Performance Limitations, 1st. Springer Publishing Company, Incorporated, 2018, isbn: 3319841238. [17] J. Principe, W. Liu, and S. Haykin, Kernel Adaptive Filtering: A Comprehensive Introduction (Adaptive and Cognitive Dynamic Systems: Signal Processing, Learning, Communications and Control). Wiley, 2011, isbn: 9781118211212. [18] S. V. Vaerenbergh, “Kernel Methods for Nonlinear Identification, Equalization and Separation of Signals,” Ph.D. dissertation, 2010-02. [19] F. de Cabrera, J. Riba, and G. Vázquez, “Robust estimation of the magnitude squared coherence based on kernel signal processing,” in 2017 51st Asilomar Conference on Signals, Systems, and Computers, 2017, pp. 890–894. [20] E. Parzen, “On estimation of a probability density function and mode,” The Annals of Mathematical Statistics, vol. 33, no. 3, pp. 1065–1076, 1962. [21] A. Rényi, “On measures of information and entropy,” Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability, vol. I, pp. 547–561, 1960. [22] E. T. Jaynes, “Information Theory and Statistical Mechanics,” Phys. Rev., vol. 106, pp. 620–630, 4 1957. doi: 10.1103/PhysRev.106.620. [23] T. van Erven and P. Harremos, “Rényi Divergence and Kullback-Leibler Divergence,” IEEE Transactions on Information Theory, vol. 60, no. 7, pp. 3797– 3820, 2014. [24] J. Principe, Information Theoretic Learning: Renyi’s Entropy and Kernel Perspectives (Information Science and Statistics). Springer New York, 2010, isbn: 9781441915702. [25] C. M. Bishop, “Training with Noise is Equivalent to Tikhonov Regularization,” Neural Computation, vol. 7, no. 1, pp. 108–116, 1995. [26] B. Silverman, Density Estimation for Statistics and Data Analysis (Chapman & Hall/CRC Monographs on Statistics & Applied Probability). Taylor & Francis, 1986, isbn: 9780412246203. [27] M. Wand and M. Jones, Kernel Smoothing (Chapman & Hall/CRC Monographs on Statistics & Applied Probability). Taylor & Francis, 1994, isbn: 9780412552700. 27 Bibliography [28] H. Noh, T. You, J. Mun, and B. Han, “Regularizing deep neural networks by noise: Its interpretation and optimization,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17, Long Beach, California, USA: Curran Associates Inc., 2017, pp. 5115–5124. [29] Z. Goldfeld, K. Greenewald, J. Weed, and Y. Polyanskiy, “Optimality of the plug-in estimator for differential entropy estimation under gaussian convolutions,” in 2019 IEEE International Symposium on Information Theory (ISIT), 2019, pp. 892–896. [30] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition (Springer Series in Statistics). Springer New York, 2009, isbn: 9780387848587. [31] M. Murray and J. Rice, Differential Geometry and Statistics (Chapman & Hall/CRC Monographs on Statistics & Applied Probability). Taylor & Francis, 1993, isbn: 9780412398605. [32] S. Amari, Information Geometry and Its Applications (Applied Mathematical Sciences). Springer Japan, 2016, isbn: 9784431559788. [33] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004, isbn: 9780511804441. [34] A. J. Lee, U-Statistics: Theory and Practice (Statistics: A Series of Textbooks and Monographs). Taylor & Francis, 1990, isbn: 9780824782535. [35] B. Levy, Principles of Signal Detection and Parameter Estimation. Springer US, 2008, isbn: 9780387765426. [36] M. Vilà and J. Riba, “A Test for Conditional Correlation Between Random Vectors Based on Weighted U-Statistics,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 5792–5796. [37] I. S. Gradshteyn and I. M. Ryzhik, Table of integrals, series, and products. Academic press, 2014, isbn: 978-0123849335. 28 APPENDIX A Proof of Equation 2.16 In order to prove Equation 2.16 we shall prove that the relative entropy between an arbitrary distribution and the Gaussian distribution with the same variance is equal to the difference of differential entropies. Let 𝑄∼𝒩(𝜇,𝜎2) . Let 𝑃 be a continuous probability distribution with variance 𝜎2 . Then, the relative entropy of 𝑃and 𝑄is given by 𝐽1(𝑃)=∫+∞ −∞ 𝑝(𝑥)log 𝑝(𝑥) 𝑞(𝑥)𝑑𝑥=∫+∞ −∞ 𝑝(𝑥)log 𝑝(𝑥)−𝑝(𝑥)log 𝑞(𝑥)𝑑𝑥 =∫+∞ −∞ 𝑝(𝑥)log 𝑝(𝑥)𝑑𝑥−∫+∞ −∞ 𝑝(𝑥)log 𝑞(𝑥)𝑑𝑥. (A.1) We identify the first integral in the last equality as the entropy of 𝑃 multiplied by −1, and the second integral as the cross entropy between 𝑃and 𝑄. It remains to prove that, for 𝑄Gaussian, the cross entropy ℎ1(𝑃,𝑄)=ℎ1(𝑄). −ℎ1(𝑃,𝑄)=∫+∞ −∞ 𝑝(𝑥)log 𝑞(𝑥)𝑑𝑥=∫+∞ −∞ 𝑝(𝑥)log (1 √2𝜋𝜎2𝑒−(𝑥−𝜇)2 2𝜎2)𝑑𝑥 =∫+∞ −∞ 𝑝(𝑥)log 1 √2𝜋𝜎2𝑑𝑥+log(𝑒)∫+∞ −∞ 𝑝(𝑥)(−(𝑥−𝜇)2 2𝜎2)𝑑𝑥 =−12log(2𝜋𝜎2)−log(𝑒) 2𝜎2∫+∞ −∞ 𝑝(𝑥)(𝑥−𝜇)2𝑑𝑥 =−12log(2𝜋𝜎2)−log(𝑒)⋅𝜎2 2𝜎2=−12log(2𝜋𝑒𝜎2)=−ℎ(𝑄) (A.2) where we have used the fact that the last integral is equal to the variance of 𝑃. Finally, substituting in Equation A.1 we obtain the result from Equation 2.16: 𝐽1(𝑃)=−ℎ(𝑃)+ℎ(𝑄)=ℎ(𝑄)−ℎ(𝑃). (A.3)  29 APPENDIX B Rényi negentropy of uniform and triangular distributions The performance of the estimator proposed in Chapter 4 is evaluated in terms of MSE using normal, uniform and triangular distributed data. In order to find the MSE we need to know the theoretical negentropy of these three distributions. Given that the negentropy of a normal distribution is zero by definition, we just derive the theoretical values for the uniform and the triangular distributions. B.1 Uniform distribution Let 𝑈∼𝒰(𝑎,𝑏) with mean 𝜇𝑢 and variance 𝜎2 𝑢 . Let 𝑋∼𝒩(𝜇𝑢,𝜎2 𝑢) . Then, the Rényi negentropy of 𝑈is given by 𝐽(𝑈)=log ∫𝑏 𝑎1 (𝑏−𝑎)2√2𝜋𝜎2 𝑢𝑒(𝑥−𝜇𝑢)2 2𝜎2 𝑢𝑑𝑥 ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐼.(B.1) Taking into account that 𝜇𝑢=12(𝑎+𝑏) (B.2) 𝜎2 𝑢=112(𝑏−𝑎)2(B.3) the integral becomes 𝐼=∫𝑏 𝑎√2𝜋 12(𝑏−𝑎) (𝑏−𝑎)2𝑒6(𝑥−1 2(𝑎+𝑏))2 (𝑏−𝑎)2=∫𝑏 𝑎√𝜋/6 𝑏−𝑎𝑒6(𝑥−1 2(𝑎+𝑏))2 (𝑏−𝑎)2𝑑𝑥. (B.4) This integral can be expressed in terms of the erfi function as follows [37]: 𝐼=𝜋12(erfi (√32)−erfi (√−32))≈1.3286. (B.5) Finally, taking the natural logarithm, 𝐽(𝑈)=log 𝐼≈0.2841. (B.6)  30 B.2. Triangular distribution B.2 Triangular distribution Let 𝑇∼𝒯(−𝑏,𝑏,0) be a triangular random variable with lower limit −𝑏 and upper limit 𝑏 . For simplicity, we also assume that it is symmetric. With these assumptions the PDF is given by𝑓𝑇(𝑥)=𝑏−|𝑥| 𝑏2(B.7) and 𝜇𝑡=0and 𝜎2 𝑡=𝑏2/6. Substituting Equation B.7 into the definition of the Rényi negentropy we obtain 𝐽(𝑇)=log ∫𝑏 −𝑏√𝜋𝑏2 3(𝑏−|𝑥| 𝑏2)2𝑒3𝑥2 𝑏2𝑑𝑥=log ∫𝑏 −𝑏√𝜋3(𝑏−|𝑥|)2 𝑏3𝑒3𝑥2 𝑏2𝑑𝑥. (B.8) Observing that we are integrating an even function on a symmetric interval, we can simplify the previous expression as follows: 𝐽(𝑇)=log ⎛⎜⎜⎝2⋅∫𝑏 0√𝜋3(𝑏−𝑥)2 𝑏3𝑒3𝑥2 𝑏2𝑑𝑥 ⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟ 𝐼 ⎞⎟⎟⎠.(B.9) Like in the uniform distribution, the integral 𝐼 can be expressed by means of the erfi function: 𝐼=136(5𝜋⋅erfi (√3)−2(𝑒3−2)√3𝜋)≈0.5160. (B.10) Finally, multiplying by 2 and taking the logarithm, 𝐽(𝑇)=log (2𝐼)≈0.03153. (B.11)  31 APPENDIX C Additional plots of bias, variance and density estimates We include the estimator bias and variance and also the PDF estimates plots for the normal, uniform and triangular distributions. The MSE plots are omitted as they are all included in the main text. C.1 Normal data 21012 0.0 0.1 0.2 0.3 0.4 0.5 KDE 1 KDE 2 KDE 3 Gaussian PDF (a) KDE of normal data for 𝑣=0.5. 3210123 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 KDE 1 KDE 2 KDE 3 Gaussian PDF (b) KDE of normal data for 𝑣=1. 6420246 0.000 0.025 0.050 0.075 0.100 0.125 0.150 0.175 KDE 1 KDE 2 KDE 3 Gaussian PDF (c) KDE of normal data for 𝑣=5. 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 0.00 0.02 0.04 0.06 0.08 0.10 0.12 KDE 1 KDE 2 KDE 3 Gaussian PDF (d) KDE of normal data for 𝑣=10. Figure C.1: KDE of normal data for variances 𝑣∈{0.5,1,5,10}. 32