Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences

Entropy 2015,17, 2988-3034; doi:10.3390/e17052988 OPEN ACCESS entropy ISSN 1099-4300 www.mdpi.com/journal/entropy Review Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences Andrzej Cichocki 1,2,*, Sergio Cruces 3,* and Shun-ichi Amari 4 1Laboratory for Advanced Brain Signal Processing, Brain Science Institute, RIKEN, 2-1 Hirosawa, Wako, 351-0198 Saitama, Japan 2Systems Research Institute, Intelligent Systems Laboratory, Newelska 6, 01-447 Warsaw, Poland 3Dpto de Teoría de la Señal y Comunicaciones, University of Seville, Camino de los Descubrimientos s/n, 41092 Seville, Spain 4Laboratory for Mathematical Neuroscience, RIKEN BSI, Wako, 351-0198 Saitama, Japan; E-Mail: [email protected] *Authors to whom correspondence should be addressed; E-Mails: [email protected] (A.C.); ser[email protected] (S.C.). Academic Editor: Raúl Alcaraz Martínez Received: 19 December 2014 / Accepted: 5 May 2015 / Published: 8 May 2015 Abstract: This work reviews and extends a family of log-determinant (log-det) divergences for symmetric positive definite (SPD) matrices and discusses their fundamental properties. We show how to use parameterized Alpha-Beta (AB) and Gamma log-det divergences to generate many well-known divergences; in particular, we consider the Stein’s loss, the S-divergence, also called Jensen-Bregman LogDet (JBLD) divergence, Logdet Zero (Bhattacharyya) divergence, Affine Invariant Riemannian Metric (AIRM), and other divergences. Moreover, we establish links and correspondences between log-det divergences and visualise them on an alpha-beta plane for various sets of parameters. We use this unifying framework to interpret and extend existing similarity measures for semidefinite covariance matrices in finite-dimensional Reproducing Kernel Hilbert Spaces (RKHS). This paper also shows how the Alpha-Beta family of log-det divergences relates to the divergences of multivariate and multilinear normal distributions. Closed form formulas are derived for Gamma divergences of two multivariate Gaussian densities; the special cases of the Kullback-Leibler, Bhattacharyya, Rényi, and Cauchy-Schwartz divergences are discussed. Symmetrized versions of log-det divergences are also considered and briefly reviewed. Finally, a class of divergences is extended to multiway divergences for separable covariance (or precision) matrices. Entropy 2015,17 2989 Keywords: Similarity measures; generalized divergences for symmetric positive definite (covariance) matrices; Stein’s loss; Burg’s matrix divergence; Affine Invariant Riemannian Metric (AIRM); Riemannian metric; geodesic distance; Jensen-Bregman LogDet (JBLD); S-divergence; LogDet Zero divergence; Jeffrey’s KL divergence; symmetrized KL Divergence Metric (KLDM); Alpha-Beta Log-Det divergences; Gamma divergences; Hilbert projective metric and their extensions 1. Introduction Divergences or (dis)similarity measures between symmetric positive definite (SPD) matrices underpin many applications, including: Diffusion Tensor Imaging (DTI) segmentation, classification, clustering, pattern recognition, model selection, statistical inference, and data processing problems [1–3]. Furthermore, there is a close connection between divergence and the notions of entropy, information geometry, and statistical mean [2,4–7], while matrix divergences are closely related to the invariant geometrical properties of the manifold of probability distributions [4,8–10]. A wide class of parameterized divergences for positive measures are already well understood and a unification and generalization of their properties can be found in [11–13]. The class of SPD matrices, especially covariance matrices, play a key role in many areas of statistics, signal/image processing, DTI, pattern recognition, and biological and social sciences [14–16]. For example, medical data produced by diffusion tensor magnetic resonance imaging (DTI-MRI) represent the covariance in a Brownian motion model of water diffusion. The diffusion tensors can be represented as SPD matrices, which are used to track the diffusion of water molecules in the human brain, with applications such as the diagnosis of mental disorders [14]. In array processing, covariance matrices capture both the variance and correlation of multidimensional data; this data is often used to estimate (dis)similarity measures, i.e., divergences. This all has led to an increasing interest in divergences for SPD (covariance) matrices [1,5,6,14,17–20]. The main aim of this paper is to review and extend log-determinant (log-det) divergences and to establish a link between log-det divergences and standard divergences, especially the Alpha, Beta, and Gamma divergences. Several forms of the log-det divergence exist in the literature, including the log–determinant αdivergence, Riemannian metric, Stein’s loss, S-divergence, also called the Jensen-Bregman LogDet (JBLD) divergence, and the symmetrized Kullback-Leibler Density Metric (KLDM) or Jeffrey’s KL divergence [5,6,14,17–20]. Despite their numerous applications, common theoretical properties and the relationships between these divergences have not been established. To this end, we propose and parameterize a wide class of log-det divergences that provide robust solutions and/or even improve the accuracy for a noisy data. We next review fundamental properties and provide relationships among the members of this class. The advantages of some selected log-det divergences are also discussed; in particular, we consider the efficiency, simplicity, and resilience to noise or outliers, in addition to simplicity of calculations [14]. The log-det divergences between two SPD matrices have Entropy 2015,17 2990 also been shown to be robust to biases in composition, which can cause problems for other similarity measures. The divergences discussed in this paper are flexible enough to facilitate the generation of several established divergences (for specific values of the tuning parameters). In addition, by adjusting the adaptive tuning parameters, we optimize the cost functions of learning algorithms and estimate desired model parameters in the presence of noise and outliers. In other words, the divergences discussed in this paper are robust with respect to outliers and noise if the tuning parameters, α,β, and γ, are chosen properly. 1.1. Preliminaries We adopt the following notation: SPD matrices will be denoted by P∈Rn×nand Q∈Rn×n, and have positive eigenvalues λi(sorted in descending order); by log(P),det(P) = |P|,tr(P)we denote the logarithm, determinant, and trace of P, respectively. For any real parameter α∈Rand for a positive definite matrix P, the matrix Pαis defined using symmetric eigenvalue decomposition as follows: Pα= (VΛVT)α=V(Λα)VT,(1) where Λis a diagonal matrix of the eigenvalues of P, and V∈Rn×nis the orthogonal matrix of the corresponding eigenvectors. Similarly, we define log(Pα) = log((VΛVT)α) = Vlog(Λα)VT,(2) where log(Λ)is a diagonal matrix of logarithms of the eigenvalues of P. The basic operations for positive definite matrices are provided in Appendix A. The dissimilarity between two SPD matrices is called a metric if the following conditions hold: 1. D(PkQ)≥0, where the equality holds if and only if P=Q(nonnegativity and positive definiteness). 2. D(PkQ) = D(QkP)(symmetry). 3. D(PkZ)≤D(PkQ) + D(QkZ)(subaddivity/triangle inequality). Dissimilarities that only satisfy condition (1) are not metrics and are referred to as (asymmetric) divergences. 2. Basic Alpha-Beta Log-Determinant Divergence For SPD matrices P∈Rn×nand Q∈Rn×n, consider a new dissimilarity measure, namely, the AB log-det divergence, given by D(α,β) AB (PkQ) = 1 αβ log det α(PQ−1)β+β(PQ−1)−α α+β(3) for α6= 0, β 6= 0, α +β6= 0, Entropy 2015,17 2991 where the values of the parameters αand βcan be chosen so as to guarantee the non-negativity of the divergence and it vanishes to zero if and only if P=Q(this issue is addressed later by Theorems 1 and 2). Observe that this is not a symmetric divergence with respect to Pand Q, except when α=β. Note that using the identity log det(P) = tr log(P), the divergence in (3) can be expressed as D(α,β) AB (PkQ) = 1 αβ tr log α(PQ−1)β+β(PQ−1)−α α+β (4) for α6= 0, β 6= 0, α +β6= 0. This divergence is related to the Alpha, Beta, and AB divergences discussed in our previous work, especially Gamma divergences [11–13,21]. Furthermore, the divergence in (4) is related to the AB divergence for SPD matrices [1,12], which is defined by ¯ D(α,β) AB (PkQ) = 1 αβ tr α α+βPα+β+β α+βQα+β−PαQβ(5) for α6= 0, β 6= 0. α +β6= 0. Note that αand βare chosen so that D(α,β) AB (PkQ)is nonnegative and equal to zero if P=Q. Moreover, such divergence functions can be evaluated without computing the inverses of the SPD matrices; instead, they can be evaluated easily by computing (positive) eigenvalues of the matrix PQ−1 or its inverse. Since both matrices Pand Q(and their inverses) are SPD matrices, their eigenvalues are positive. In general, it can be shown that even though PQ−1is nonsymmetric, its eigenvalues are the same as those of the SPD matrix Q−1/2PQ−1/2; hence, its eigenvalues are always positive. Next, consider the eigenvalue decomposition: (PQ−1)β=VΛβV−1,(6) where Vis a nonsingular matrix, and Λβ= diag{λβ 1, λβ 2, . . . , λβ n}is the diagonal matrix with the positive eigenvalues λi>0, i = 1,2, . . . , n, of PQ−1. Then, we can write D(α,β) AB (PkQ) = 1 αβ log det αVΛβV−1+βVΛ−αV−1 α+β =1 αβ log det Vdet αΛβ+βΛ−α α+βdet V−1 =1 αβ log det αΛβ+βΛ−α α+β,(7) which allows us to use simple algebraic manipulations to obtain D(α,β) AB (PkQ) = 1 αβ log n Y i=1 αλβ i+βλ−α i α+β =1 αβ n X i=1 log αλβ i+βλ−α i α+β!, α, β, α +β6= 0.(8) It is straightforward to verify that D(α,β) AB (PkQ) = 0 if P=Q. We will show later that this function is nonnegative for any SPD matrices if the αand βparameters take both positive or negative values. Entropy 2015,17 2992 For the singular values α= 0 and/or β= 0 (also α=−β), the AB log-det divergence in (3) is defined as a limit for α→0and/or β→0. In other words, to avoid indeterminacy or singularity for specific parameter values, the AB log-det divergence can be reformulated or extended by continuity and by applying L’Hôpital’s formula to cover the singular values of αand β. Using L’Hôpital’s rule, the AB log-det divergence can be defined explicitly by D(α,β) AB (PkQ) =                                                1 αβ log det α(PQ−1)β+β(QP−1)α α+βfor α, β 6= 0, α +β6= 0 1 α2tr (QP−1)α−I−αlog det(QP−1)for α6= 0, β = 0 1 β2tr (PQ−1)β−I−βlog det(PQ−1)for α= 0, β 6= 0 1 α2log det(PQ−1)α det(I+ log(PQ−1)α)for α=−β6= 0 1 2tr log2(PQ−1) = 1 2||log(Q−1/2PQ−1/2)||2 Ffor α, β = 0. (9) Equivalently, using standard matrix manipulations, the above formula can be expressed in terms of the eigenvalues of PQ−1,i.e., the generalized eigenvalues computed from λiQvi=Pvi(where vi(i= 1,2, . . . , n) are corresponding generalized eigenvectors) as follows: D(α,β) AB (PkQ) =                                                            1 αβ n X i=1 log αλβ i+βλ−α i α+β!for α, β 6= 0, α +β6= 0 1 α2"n X i=1 λ−α i−log(λ−α i)−n#for α6= 0, β = 0 1 β2"n X i=1 λβ i−log(λβ i)−n#for α= 0, β 6= 0 1 α2"n X i=1 log λα i 1 + log λα i#for α=−β6= 0 1 2 n X i=1 log2(λi)for α, β = 0. (10) Theorem 1. The function D(α,β) AB (PkQ)≥0given by (3) is nonnegative for any SPD matrices with arbitrary positive eigenvalues if α≥0and β≥0or if α < 0and β < 0. It is equal to zero if and only if P=Q. Equivalently, if the values of αand βhave the same sign, the AB log-det divergence is positive independent of the distribution of the eigenvalues of PQ−1and goes to zero if and only if all the Entropy 2015,17 2993 eigenvalues are equal to one. However, if the eigenvalues are sufficiently close to one, the AB log-det divergence is also positive for different signs of αand β. The conditions for positive definiteness are given by the following theorem. Theorem 2. The function D(α,β) AB (PkQ)given by (9) is nonnegative if α > 0and β < 0or if α < 0and β > 0and if all the eigenvalues of PQ−1satisfy the following conditions: λi> β α 1 α+β ∀i, for α > 0and β < 0,(11) and λi< β α 1 α+β ∀i, for α < 0and β > 0.(12) If any of the eigenvalues violate these bounds, the value of the divergence, by definition, is infinite. Moreover, when α→ −βthese bounds simplify to λi> e−1/α ∀i, α =−β > 0,(13) λi< e−1/α ∀i, α =−β < 0.(14) In the limit, when α→0or β→0, the bounds disappear. A visual presentation of these bounds for different values of αand βis shown in Figure 1. Additionally, D(α,β) AB (PkQ)=0only if λi= 1 for all i= 1, . . . , n, i.e., when P=Q. ALPHA BETA 0 0.01 0.14 0.37 0.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 (a) Lower-bounds of λi. ALPHA BETA 2 2.7 7.4 150 8 −2−1.5 −1−0.5 0 0.5 1 1.5 2 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 (b) Upper-bounds of λi. Figure 1. Shaded-contour plots of the bounds of λithat prevent D(α,β) AB (PkQ)from diverging to ∞. The positive lower-bounds are shown in the lower-right quadrant of (a). The finite upper-bounds are shown in the upper-left quadrant of (b). The proofs of these theorems are given in Appendices B, C and D. Figure 2illustrates the typical shapes of the AB log-det divergence for different values of the eigenvalues for various choices of αand β. Entropy 2015,17 2994 (a) −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 0 0.5 1 1.5 ALPHA BETA D 0.5 1 1.5 (b) −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 0 0.5 1 1.5 ALPHA BETA D 0.5 1 1.5 (c) −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 ALPHA BETA D 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 (d) −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 ALPHA BETA D 1.35 1.4 1.45 1.5 1.55 1.6 1.65 1.7 1.75 1.8 1.85 Figure 2. Two-dimensional plots of the AB log-det divergence for different eigenvalues: (a)λ= 0.4, (b)λ= 2.5, (c)λ1= 0.4, λ2= 2.5, (d) 10 eigenvalues uniformly randomly distributed in the range [0.5,2]. In general, the AB log-det divergence is not a metric distance since the triangle inequality is not satisfied for all parameter values. Therefore, we can define the metric distance as the square root of the AB log-det divergence in the special case when α=βas follows: d(α,α) AB (PkQ) = qD(α,α) AB (PkQ).(15) This follows from the fact that D(α,α) AB (PkQ)is symmetric with respect to Pand Q. Later, we will show that measures defined in this manner lead to many important and well-known divergences and metric distances such as the Logdet Zero divergence, Affine Invariant Riemannian metric (AIRM), and square root of Stein’s loss [5,6]. Moreover, new divergences can be generated; specifically, generalized Stein’s loss, the Beta-log-det divergence, and extended Hilbert metrics. From the divergence D(α,β) AB (PkQ), a Riemannian metric and a pair of dually coupled affine connections are introduced in the manifold of positive definite matrices. Let dPbe a small deviation Entropy 2015,17 2995 of P, which belongs to the tangent space of the manifold at P. Calculating D(α,β) AB (P+dPkP)and neglecting higher-order terms yields (see Appendix E) D(α,β) AB (P+dPkP) = 1 2tr[dP P−1dP P−1].(16) This gives a Riemannian metric that is common for all (α, β). Therefore, the Riemannian metric is the same for all AB log-det divergences, although the dual affine connections depend on αand β. The Riemannian metric is also the same as the Fisher information matrix of the manifold of multivariate Gaussian distributions of mean zero and covariance matrix P. Interestingly, note that the Riemannian metric or geodesic distance is obtained from (3) for α=β= 0: dR(PkQ) = q2D(0,0) AB (PkQ) = qtr log2(PQ−1)(17) =klog(PQ−1)kF=klog(Q−1/2PQ−1/2)kF(18) =v u u t n X i=1 log2(λi),(19) where λiare the eigenvalues of PQ−1. This is also known as the AIRM. AIRM takes advantage of several important and useful theoretical properties and is probably one of the most widely used (dis)similarity measure for SPD (covariance) matrices [14,15]. For α=β= 0.5(and for α=β=−0.5), the recently defined and deeply analyzed S-divergence (JBLD) [6,14,15,17] is obtained: DS(PkQ) = D(0.5,0.5) AB (PkQ) = 4 log det 1 2(PQ−1)1/2+ (PQ−1)−1/2 = 4 log det(P)1/2det (PQ−1)1/2+ (PQ−1)−1/2 2det(Q)1/2 det(P)1/2det(Q)1/2 = 4 log det 1 2(P+Q) pdet(P) det(Q) = 4 log det P+Q 2−1 2log det(PQ)= 4 n X i=1 log λi+ 1 2√λi.(20) The S-divergence is not a metric distance. To make it a metric, we take its square root and obtain the LogDet Zero divergence, or Bhattacharyya distance [5,7,18]: dBh(PkQ) = qD(0.5,0.5) AB (PkQ) = 2slog det P+Q 2−1 2log det(PQ) = 2slog det 1 2(P+Q) pdet(P) det(Q).(21) Entropy 2015,17 2996 Moreover, for α= 0, β 6= 0 and α6= 0, β = 0, we obtain divergences which are generalizations of Stein’s loss (called also Burg matrix divergence or simply LogDet divergence): D(0,β) AB (PkQ) = 1 β2tr (PQ−1)β−I−βlog det(PQ−1), β 6= 0.(22) D(α,0) AB (PkQ) = 1 α2tr (QP−1)α−I−αlog det(QP−1), α 6= 0 (23) The divergences in (22) and (23) simplify, respectively, to the standard Stein’s loss if β= 1 and to its dual loss if α= 1. 3. Special Cases of the AB Log-Det Divergence We now illustrate how a suitable choice of the (α, β)parameters simplify the AB log-det divergence into other known divergences such as the Alpha- and Beta-log-det divergences [5,11,18,23] (see Figure 3 and Table 1). (=)αβ . . (1)α= 1 _ 2 1 _ 2 ( =0, 0)βα= 1 1 _ α2[]tr(( ) logdet( )QP Q α_I)αP-1 _1_ (Squared AIRM) _ QP +PQ _1 _1 2 logdet (0,0)αβ == _1 (( ) logdet( )PQ PQI) β] _1 [tr 1 _ β2 1β__ _ PQ +QP _1 _1 2 logdet ( logdet( )QQPP -- 11 )--n tr S-divergence 1 4 logdet logdet( ) 22 [() ] + - PQ PQ (Stein’s Loss) 11 2 1()() logdet 2 -- +PQ QP α α α 11 1()() logdet 1 -- + + ββ β PQ Q P β ( + =1)αβ 11 1log det( ( ) (1 )( ( 1)) 1) [] -- - +- - α αα α ααPQ QP ααα α ( _ ( logdet( )PQ PQI) 1 _1 tr __ _ ( logdet( )QP QPI) 1 _1 tr __ 1 _ 2trlog ( ) 2PQ_1 1 Figure 3. Links between the fundamental, nonsymmetric, AB log-det divergences. On the α-β-plane, important divergences are indicated by points and lines, especially the Stein’s loss and its generalization, the AIRM (Riemannian) distance, S-divergence (JBLD), Alpha-log-det divergence D(α) A, and Beta-log-det divergence D(β) B. Entropy 2015,17 3003 for some selected values of γ > 0and α, β > 0or α, β < 0that can make the kernel positive definite. 6. Similarity Measures for Semidefinite Covariance Matrices in Reproducing Kernel Hilbert Spaces There are many practical applications for which the underlying covariance matrices are symmetric but only positive semidefinite, i.e., their columns do not span the whole space. For instance, in classification problems, assume two classes and a set of observation vectors {x1,...,xT}and {y1,...,yT}in Rm for each class, then we may wish to find a principled way to evaluate the ensemble similarity of the data from their sample similarity. The problem of the modeling of similarity between two ensembles was studied by Zhou and Chellappa in [32]. For this purpose, they proposed several probabilistic divergence measures between positive semidefinite covariance matrices in a Reproducing kernel Hilbert space (RKHS) of finite dimensionality. Their strategy was later extended for image classification problems [33] and formalized for the Log-Hilbert-Schmidt metric between infinite-dimensional RKHS covariance operators [34]. In this section, we propose the unifying framework of the AB log-det divergences to reinterpret and extend the similarity measures obtained in [32,33] for semidefinite covariance matrices in the finite-dimensional RKHS. We shall assume that the nonlinear functions Φx:Rm→Rnand Φy:Rm→Rn(where n>m) respectively map the data from each of the classes into their higher dimensional feature spaces. We implicitly define the feature matrices as Φx= [Φx(x1),...,Φx(xT)],Φy= [Φy(y1),...,Φy(yT)],(47) and the sample covariance matrices of the observations in the feature space as: Cx=ΦxJΦT x/T ∈Rn×n and Cy= ΦyJΦT y/T ∈Rn×n, where J=IT−1 T11Tdenotes the T×Tcentering matrix. In practice, it is common to consider low-rank approximations of sample covariance matrices. For a given basis Vx= (v1,...,vr)∈RT×rof the principal subspace of JΦT xΦxJ, we can define the projection matrix Πx=VxVT xand redefine the covariance matrices as Cx=1 TΦxVxVT xΦT xand Cy=1 TΦyVyVT yΦT y.(48) Assuming the Gaussianity of the data in the feature space, the mean vector and covariance matrix are sufficient statistics and a natural measure of dissimilarity between Φxand Φyshould be a function of the first and second order statistics of the features. Furthermore, in most practical problems the mean value should be ignored due to robustness considerations, and then the comparison reduces to the evaluation of a suitable dissimilarity measure between Cxand Cy. The dimensionality of the feature space nis typically much larger than r, so the rank of the covariance matrices in (48) will be rnand, therefore, both matrices are positive semidefinite. The AB log-det divergence is infinite when the range spaces of the covariance matrices Cxand Cydiffer. This property is useful in applications which require an automatic constraint in the range of the estimates [22], but it will prohibit the practical use of the comparison when the ranges of the covariance matrices differ. The next subsections present two different strategies to address this challenging problem. Entropy 2015,17 3004 6.1. Measuring the Dissimilarity with a Divergence Lower-Bound One possible strategy is to use dissimilarity measures which ignore the contribution to the divergence caused by the rank deficiency of the covariance matrices. This is useful when performing one comparison of the covariances matrices after applying a congruence transformation that aligns their range spaces, and can be implemented by retaining only the finite and non-zero eigenvalues of the matrix pencil (Cx,Cy). Let Irdenote the identity matrix of size rand (·)+the Moore-Penrose pseudoinverse operator. Consider the eigenvalue decomposition of the symmetric matrix (C+ y)1 2Cx(C+ y)1 2=UΛUT(49) where Uis a semi-orthogonal matrix for which the columns are the eigenvectors associated with the positive eigenvalues of the matrix pencil and Λ= diag(λ1, . . . , λr)≡diag Eig+{(C+ y)1 2Cx(C+ y)1 2)}.(50) is a diagonal matrix with the eigenvalues sorted in a descending order. Note that the tall matrix W= (C+ y)1 2U∈Rn×rdiagonalizes the covariance matrices of the two classes WTCxW=Λ(51) WTCyW=Ir(52) and compress them to a common range space. The compression automatically discards the singular and infinite eigenvalues of the matrix pencil (Cx,Cy), while it retains the finite and positive eigenvalues. In this way, the following dissimilarity measures can be obtained: L(α,β) AB (Cx,Cy)≡D(α,β) AB (WTCxWkWTCyW) = D(α,β) AB (ΛkIr),(53) L(α,β) ABS1(Cx,Cy)≡D(α,β) ABS1(WTCxWkWTCyW) = D(α,β) ABS1(ΛkIr).(54) Note, however, that these measures should not be understood as a strict comparison of the original covariance matrices, but rather as an indirect comparison through their respective compressed versions WTCxWand WTCyW. With the help of the kernel trick, the next lemma shows that the evaluation of the dissimilarity measures L(α,β) AB (Cx,Cy)and L(α,β) ABS1(Cx,Cy), does not require the explicit computation of the covariance matrices or of the feature vectors. Lemma 1. Given the Gram matrix or kernel matrix of the input vectors Kxx Kxy Kyx Kyy!= ΦT xΦxΦT xΦy ΦT yΦxΦT yΦy!(55) and the matrices Vxand Vywhich respectively span the principal subspaces of Kxx and Kyy, the positive and finite eigenvalues of the matrix pencil can be expressed by Λ= diag Eig+(VT xKxyK−1 yy Vy)(VT xKxyK−1 yy Vy)T.(56) Entropy 2015,17 3005 Proof. The proof of the lemma relies on the property that for any pair of m×nmatrices Aand B, the non-zero eigenvalues of ABTand of BTAare the same (see [30, pag. 11]). Then, there is an equality between the following matrices of positive eigenvalues Λ≡diag Eig+n(C+ y)1 2Cx(C+ y)1 2)o= diag Eig+CxC+ y.(57) Taking into account the structure of the covariance matrices in (48), such eigenvalues can be explicitly obtained in terms of the kernel matrices Eig+CxC+ y=Eig+(ΦxVxVT xΦT x)((Φ+ y)TVyVT yΦ+ y)(58) =Eig+(VT xΦT x(ΦT y)+VyVT yΦ+ y)(ΦxVx)(59) =Eig+(VT xKxyK−1 yy Vy)(VT xKxyK−1 yy Vy)T.(60) 6.2. Similarity Measures Between Regularized Covariance Descriptors Several authors consider a completely different strategy, which consists in the regularization of the original covariance matrices [32–34]. This way the null the eigenvalues of the covariances Cxand Cy are replaced by a small positive constant ρ > 0, to obtain the “regularized” positive definite matrices ˜ Cx and ˜ Cy, respectively. The modification can be illustrated by comparing the eigendecompositions Cx= (Ux|U⊥ x) Λx0 0 0!(Ux|U⊥ x)T=UxΛxUT x(61) ↓(62) ˜ Cx= (Ux|U⊥ x) Λx0 0ρIn−r!(Ux|U⊥ x)T=Cx+ρU⊥ x(U⊥ x)T.(63) Then, the dissimilarity measure of the data in the feature space can be obtained just by measuring a divergence between the SPD matrices ˜ Cxand ˜ Cy. Again, the idea is to compute the value of the divergence without requiring the evaluation of the feature vectors but by using the available kernels. Using the properties of the trace and the determinants, a practical formula for the log-det Alpha-divergence has been obtained in [32,33] for 0< α < 1. The resulting expression D(α,1−α) AB (˜ Cxk˜ Cy)= 1 α(1 −α)log det(I2r+ρ−1H)−1 (1 −α)log det(ρ−1Λx)−1 αlog det(ρ−1Λy) is a function of the principal eigenvalues of the kernels Λx=VT xKxxVx,Λy=VT xKyyVy,(64) and the matrix H= (α)1 2Wx0 0(1 −α)1 2Wy! Kxx Kxy Kyx Kyy! (α)1 2Wx0 0(1 −α)1 2Wy!T .(65) Entropy 2015,17 3006 where Wx=Vx(Ir−ρΛ−1 x)1 2and Wy=Vy(Ir−ρΛ−1 y)1 2.(66) The evaluation of the divergence outside the interval 0< α < 1, or when β6= 1 −α, is not covered by this formula and, in general, requires knowledge of the eigenvalues of the matrix ˜ C−1 2 y˜ Cx˜ C−1 2 y. However, different analyses are necessary depending on the dimension of the intersection of the range space of both covariance matrices Cxand Cy. In the following, we study the two more general scenarios. Case (A) The range spaces of Cxand Cyare the same. In this case U⊥ y(U⊥ y)T=U⊥ x(U⊥ x)T, and the eigenvalues of the matrix ˜ Cx˜ C−1 y= (Cx+ρU⊥ x(U⊥ x)T)(C+ y+ρ−1U⊥ x(U⊥ x)T)(67) =CxC+ y+U⊥ x(U⊥ x)T(68) coincide with the nonzero eigenvalues of CxC+ yexcept for (n−r)additional eigenvalues which are equal to 1. Then, using the equivalence between (57) and (60), the divergence reduces to the following form D(α,β) AB (˜ Cxk˜ Cy) = L(α,β) AB (Cx,Cy)(69) =D(α,β) AB ((VT xKxyK−1 yy Vy)(VT xKxyK−1 yy Vy)TkIr).(70) Case (B) The range spaces of Cxand Cyare disjoint. In practice, for nrthis is the most probable scenario. In such a case, the rlargest eigenvalues of the matrix ˜ Cx˜ C−1 ydiverge as ρtends to zero. Hence, we can not bound above these eigenvalues and, for this reason, it makes no sense to study the case of sign(α)6= sign(β), so in this section we assume that sign(α) = sign(β). Theorem 3. When range spaces of Cxand Cyare disjoint and for a sufficiently small value of ρ > 0, the AB log-det divergence is closely approximated by the formula D(α,β) AB (˜ Cxk˜ Cy)≈D(α,β) AB (C(ρ) x|ykρIr) + D(β,α) AB (C(ρ) y|xkρIr),(71) where C(ρ) x|y(and respectively C(ρ) y|xby interchanging xand y) denotes the matrix C(ρ) x|y=Λx−ρIr−ρ2Λ−1 y−WT xKxyWyΛ−1 yWT yKyxWx.(72) The proof of the theorem is presented in the Appendix G. The eigenvalues of the matrices C(ρ) x|yand C(ρ) y|x, estimate the rlargest eigenvalues of ˜ C−1 2 y˜ Cx˜ C−1 2 yand of its inverse ˜ C−1 2 x˜ Cy˜ C−1 2 x, respectively. The relative error in the estimation of these eigenvalues is of order O(ρ), i.e., it gradually improves as ρ Entropy 2015,17 3007 tend to zero. The approximation is asymptotically exact, and C(ρ) x|yand C(ρ) x|yconverge respectively to the conditional covariance matrices Cx|y= lim ρ→0C(ρ) x|y=VT xKxxVx−(VT xKxyVy)(VT yKyyVy)−1(VT xKxyVy)T,(73) Cy|x= lim ρ→0C(ρ) y|x=VT yKyyVy−(VT yKyxVx)(VT xKxxVx)−1(VT yKyxVx)T,(74) while ρIconverges to the zero matrix. In the limit, the value of the divergence is not very useful because lim ρ→0D(α,β) AB (˜ Cxk˜ Cy) = ∞,(75) though there are some practical ways to circumvent this limitation. For example, when α= 0 or β= 0, the divergence can be scaled by a suitable power of ρto make it finite (see Section 3.3.1 in [32]). The scaled form of the divergence between the regularized covariance matrices is SD(α,β) AB (˜ Cxk˜ Cy)≡lim ρ→0ρmax{α,β}D(α,β) AB (˜ Cxk˜ Cy).(76) Examples of scaled divergences are the following versions of Stein’s losses SD(0,β) AB (˜ Cxk˜ Cy) = lim ρ→0ρβD(0,β) AB (˜ Cxk˜ Cy) = 1 β2tr (Cx|y)β≥0, β > 0,(77) SD(α,0) AB (˜ Cxk˜ Cy) = lim ρ→0ραD(α,0) AB (˜ Cxk˜ Cy) = 1 α2tr (Cy|x)α≥0, α > 0,(78) as well as the Jeffrey’s KL family of symmetric divergences (cf. Equation (23) in [33]) SD(α,0) ABS1(˜ Cxk˜ Cy) = lim ρ→0ραD(α,0) ABS1(˜ Cxk˜ Cy) = 1 2α2tr((Cx|y)α) + tr((Cy|x)α), α > 0.(79) In other cases, when the scaling is not sufficient to obtain a finite and practical dissimilarity measure, an affine transformation may be used. The idea is to identify the divergent part of D(α,β) AB (˜ Cxk˜ Cy)as ρ→0and use its value as a reference for the evaluation the dissimilarity. For α, β ≥0, the relative AB log-det dissimilarity measure is the limiting value of the affine transformation RD(α,β) AB (˜ Cxk˜ Cy)≡lim ρ→0min{α, β}D(α,β) AB (˜ Cxk˜ Cy)−r αβ log αβρ−(α+β) (α+β)2, α, β > 0.(80) After its extension by continuity (including as special cases α= 0 or β= 0), the function RD(α,β) AB (˜ Cxk˜ Cy) =                    log det(Cx|y) + α βlog det(Cy|x)β > α ≥0 log det(Cx|y) + log det(Cy|x)α=β≥0 log det(Cy|x) + β αlog det(Cx|y)α > β ≥0 (81) provides simple formulas to measure the relative dissimilarity between symmetric positive semidefinite matrices Cxand Cy. However, it should be taken into account that, as a consequence of its relative character, this function is not bounded below and can achieve negative values. Entropy 2015,17 3008 7. Modifications and Generalizations of AB Log-Det Divergences and Gamma Matrix Divergences The divergence (3) discussed in the previous sections can be extended and modified in several ways. It is interesting to note that the positive eigenvalues of PQ−1play a similar role as the ratios (pi/qi) and (qi/pi)when used in the wide class of standard discrete divergences, see for example, [11,12]; hence, we can apply such divergences to formulate a modified log-det divergence as a function of the eigenvalues λi. For example, consider the Itakura-Saito distance defined by DIS(pkq) = X ipi qi + log qi pi−1.(82) It is worth noting that we can generate the large class of divergences or cost functions using Csiszár f-functions [13,24,25]. By replacing pi/qiwith λiand qi/piwith λ−1 i, we obtain the log-det divergence for SPD matrices: DIS(PkQ) = n X i=1 (λi−log(λi)) −n, (83) which is consistent with (24) and (26). As another example, consider the discrete Gamma divergence [11,12] defined by D(α,β) AC (pkq) = 1 β(α+β)log X i pα+β i!+1 α(α+β)log X i qα+β i!−1 αβ ln X i pα iqβ i! =1 αβ(α+β)log X i pα+β i!α X i qα+β i!β X i pα iqβ i!α+β,(84) for α6= 0, β 6= 0, α +β6= 0, which when α= 1 and β→ −1, simplifies to the following form [11]: lim β→−1D(1,β) AC (pkq) = 1 n n X i=1 log qi pi+ log n X i=1 pi qi!−log(n) = log 1 n n X i=1 pi qi n Y i=1 pi qi!1/n .(85) Hence, by substituting pi/qiwith λi, we derive a new Gamma matrix divergence for SPD matrices: D(1,0) CCA(PkQ) = D(1,−1) AC (PkQ) = 1 n n X i=1 log λ−1 i+ log n X i=1 λi!−log(n) = log 1 n n X i=1 λi n Y i=1 λi!1/n = log M1{λi} M0{λi},(86) Entropy 2015,17 3009 where M1denotes the arithmetic mean, and M0denotes the geometric mean. Interestingly, (86) can be expressed equivalently as D(1,0) CCA(PkQ) = log(tr(PQ−1)) −1 nlog det(PQ−1)−log(n).(87) Similarly, using the symmetric Gamma divergence defined in [11,12], D(α,β) ACS (pkq) = 1 αβ log X i pα+β i! X i qα+β i! X i pα iqβ i! X i pβ iqα i!,(88) for α6= 0, β 6= 0, α +β6= 0, for α= 1 and β→ −1and by substituting the ratios pi/qiwith λi, we obtain a new Gamma matrix divergence as follows: D(1,−1) ACS (PkQ) = log ( n X i=1 λi)( n X i=1 λ−1 i)!−log(n)2 = log (1 n n X i=1 λi)(1 n n X i=1 λ−1 i)! = log M1{λi}M1λ−1 i (89) = log M1{λi} M−1{λi},(90) where M−1{λi}denotes the harmonic mean. Note that for n→ ∞, this formulated divergence can be expressed compactly as D(1,−1) ACS (PkQ) = log(E{u}E{u−1}),(91) where ui={λi}and u−1 i={λ−1 i}. The basic means are defined as follows: Mγ(λ) =                                            M−∞ = min{λ1, . . . , λn},γ→ −∞, M−1=n n X i=1 1 λi!−1 ,γ=−1, M0= n Y i=1 λi!1/n , γ = 0, M1=1 n n X i=1 λi,γ= 1, M2= 1 n n X i=1 λ2 i!1/2 ,γ= 2, M∞= max{λ1, . . . , λn},γ→ ∞, (92) with M−∞ ≤M−1≤M0≤M1≤M2≤M∞,(93) Entropy 2015,17 3010 where equality holds only if all λiare equal. By increasing the values of γ, more emphasis is put on large relative errors, i.e., on λiwhose values are far from one. Depending on the value of γ, we obtain the minimum entry of the vector λ(for γ→ −∞), its harmonic mean (γ=−1), the geometric mean (γ= 0), the arithmetic mean (γ= 1), the quadratic mean (γ= 2), and the maximum entry of the vector (γ→ ∞). Exploiting the above inequalities for the means, the divergences in (86) and (90) can be heuristically generalized (defined) as follows: D(γ2,γ1) CCA (PkQ) = log Mγ2{λi} Mγ1{λi},(94) for γ2> γ1. The new divergence in (94) is quite general and flexible, and in extreme cases, it takes the following form: D(∞,−∞) CCA (PkQ) = dH(PkQ) = log M∞{λi} M−∞{λi}= log λmax λmin ,(95) which is, in fact, a well-known Hilbert projective metric [6,26]. The Hilbert projective metric is extremely simple and suitable for big data because it requires only two (minimum and maximum) eigenvalue computations of the matrix PQ−1. The Hilbert projective metric satisfies the following important properties [6,27]: 1. Nonnegativity, dH(PkQ)≥0, and definiteness, dH(PkQ) = 0, if and only if there exists a c > 0such that Q=cP. 2. Invariance to scaling: dH(c1Pkc2Q) = dH(PkQ),(96) for any c1, c2>0. 3. Symmetry: dH(PkQ) = dH(QkP).(97) 4. Invariance under inversion: dH(PkQ) = dH(P−1kQ−1).(98) 5. Invariance under congruence transformations: dH(APATkAQAT) = dH(PkQ),(99) for any invertible matrix A. 6. Invariance under geodesic (Riemannian) transformations (by taking A=P−1/2in (99)): dH(IkP−1/2QP−1/2) = dH(PkQ).(100) 7. Separability of divergence for the Kronecker product of SPD matrices: dH(P1⊗P2kQ1⊗Q2) = dH(P1kQ1) + dH(P2kQ2).(101) Entropy 2015,17 3011 8. Scaling of power of SPD matrices: dH(PωkQω) = |ω|dH(PkQ),(102) for any ω6= 0. Hence, for 0<|ω1| ≤ 1≤ |ω2|we have dH(Pω1kQω1)≤dH(PkQ)≤dH(Pω2kQω2).(103) 9. Scaling under the weighted geometric mean: dH(P#sQkP#uQ) = |s−u|dH(PkQ),(104) for any u, s 6= 0, where P#uQ=P1/2(P−1/2QP−1/2)uP1/2.(105) 10. Triangular inequality: dH(PkQ)≤dH(PkZ) + dH(ZkQ). These properties can easily be derived and verified. For example, property (9) can easily be derived as follows [6,27]: dH(P#sQkP#uQ) = dH(P1/2(P−1/2QP−1/2)sP1/2k(P1/2(P−1/2QP−1/2)uP1/2) =dH((P−1/2QP−1/2)sk(P−1/2QP−1/2)u) =dH((P−1/2QP−1/2)(s−u)kI) =|s−u|dH(PkQ).(106) In Table 2, we summarize and compare some fundamental properties of three important metric distances: the Hilbert projective metric, Riemannian metric, and LogDet Zero (Bhattacharyya) distance. Since some of these properties are new, we refer to [6,27,28]. 7.1. The AB Log-Det Divergence for Noisy and Ill-Conditioned Covariance Matrices In real-world signal processing and machine learning applications, the SPD sampled matrices can be strongly corrupted by noise and extremely ill conditioned. In such cases, the eigenvalues of the generalized eigenvalue (GEVD) problem Pvi=λiQvican be divided into a signal subspace and noise subspace. The signal subspace is usually represented by the largest eigenvalues (and corresponding eigenvectors), and the noise subspace is usually represented by the smallest eigenvalues (and corresponding eigenvectors), which should be rejected; in other words, in the evaluation of log-det divergences, only the eigenvalues that represent the signal subspace should be taken into account. The simplest approach is to find the truncated dominant eigenvalues by applying the suitable threshold τ > 0; equivalently, find an index r≤nfor which λr+1 ≤τand perform a summation. For example, truncation reduces the summation in (8) from 1 to r(instead of 1 to n) [22]. The threshold parameter τcan be selected via cross validation. Entropy 2015,17 3012 Table 2. Comparison of the fundamental properties of three basic metric distances: the Riemannian (geodesic) metric (19), LogDet Zero (Bhattacharyya) divergence (21), and the Hilbert projective metric (95). Matrices P,Q,P1,P2,Q1,Q2,Z∈Rn×nare SPD matrices, A∈Rn×nis nonsingular, and the matrix X∈Rn×rwith r < n is full column rank. The scalars satisfy the following conditions: c > 0,c1, c2>0;0< ω ≤1,s, u 6= 0, ψ=|s−u|. The geometric means are defined by P#uQ=P1/2(P−1/2QP−1/2)uP1/2 and P#Q=P#1/2Q=P1/2(P−1/2QP−1/2)1/2P1/2. The Hadamard product of Pand Q is denoted by P◦Q(cf. with [6]). Riemannian (geodesic) metric LogDet Zero (Bhattacharyya) div. Hilbert projective metric dR(PkQ) = klog(Q−1/2PQ−1/2)kFdBh(PkQ) = 2slog det 1 2(P+Q) pdet(P) det(Q)dH(PkQ) = log λmax{PQ−1} λmin{PQ−1} dR(PkQ) = dR(QkP)dBh(PkQ) = dBh(QkP)dH(PkQ) = dH(QkP) dR(cPkcQ) = dR(PkQ)dBh(cPkcQ) = dBh(PkQ)dH(c1Pkc2Q) = dH(PkQ) dR(APATkAQAT) = dR(PkQ)dBh(APATkAQAT) = dBh(PkQ)dH(APATkAQAT) = dH(PkQ) dR(P−1kQ−1) = dR(PkQ)dBh(P−1kQ−1) = dBh(PkQ)dH(P−1kQ−1) = dH(PkQ) dR(PωkQω)≤ω dR(PkQ)dBh(PωkQω)≤√ω dBh(PkQ)dH(PωkQω)≤ω dH(PkQ) dR(PkP#ωQ) = ω dR(PkQ)dBh(PkP#ωQ)≤√ω dBh(PkQ)dH(PkP#ωQ) = ω dH(PkQ) dR(Z#ωPkZ#ωQ)≤ω dR(PkQ)dBh(Z#ωPkZ#ωQ)≤√ω dBh(PkQ)dH(Z#ωPkZ#ωQ)≤ω dH(PkQ) dR(P#sQkP#uQ) = ψ dR(PkQ)) dBh(P#sQkP#uQ)≤√ψ dBh(PkQ)dH(P#sQkP#uQ) = ψ dH(PkQ) dR(PkP#Q) = dR(QkP#Q)dBh(PkP#Q) = dBh(QkP#Q)dH(PkP#Q) = dH(QkP#Q) dR(XTPX kXTQX)≤dR(PkQ)dBh(XTPX kXTQX)≤dBh(PkQ)dH(XTPX kXTQX)≤dH(PkQ) dR(Z⊗PkZ⊗Q) = √n dR(PkQ)dBh(Z⊗PkZ⊗Q) = √n dBh(PkQ)dH(Z⊗PkZ⊗Q) = dH(PkQ) d2 R(P1⊗P2kQ1⊗Q2) = dBh(P1⊗P2kQ1⊗Q2)dH(P1⊗P2kQ1⊗Q2) =n d2 R(P1kQ1) + n d2 R(P2kQ2)+ ≥dBh(P1◦P2kQ1◦Q2) = dH(P1kQ1) + dH(P2kQ2) 2 log det(P1Q−1 1) log det(P2Q−1 2) Entropy 2015,17 3019 and α+β= 0, which is where the formula is undefined. By enforcing continuity, we can explicitly define the AB log-det divergence on the entire (α, β)-plane as follows: D(α,β) AB (PkQ) =                                                                              1 αβ log det α(PQ−1)β+β(QP−1)α α+β+ for α, β 6= 0, α +β6= 0, 1 α2tr (QP−1)α−I−αlog det(QP−1)for α6= 0, β = 0, 1 β2tr (PQ−1)β−I−βlog det(PQ−1)for α= 0, β 6= 0, 1 α2log det[(PQ−1)−α(I+ log(PQ−1)α)]−1 +for α=−β, 1 2tr log2(PQ−1) = 1 2||log(Q−1/2PQ−1/2)||2 Ffor α, β = 0. (145) C. Eigenvalues Domain for Finite D(α,β) AB (PkQ) In this section, we assume that λi, an eigenvalue of PQ−1, satisfies 0≤λi≤ ∞ for all i= 1, . . . , n. We will determine the bounds of the eigenvalues of PQ−1that prevent the AB log-det divergence from being infinite. First, recall that D(α,β) AB (PkQ) = 1 αβ n X i=1 log "αλβ i+βλ−α i α+β#+ , α, β, α +β6= 0.(146) We assume that 0≤λi≤ ∞for all i. For the divergence to be finite, the arguments of the logarithms in the previous expression must be positive. This happens when αλβ i+βλ−α i α+β>0∀i, (147) which is always true when α, β > 0or when α, β < 0. On the contrary, when sign(αβ) = −1, we have the following two cases. In the first case when α > 0, we initially solve for λα+β iand later for λito obtain λα+β i α+β>−β α(α+β)= β α 1 α+β−→ λi> β α 1 α+β ∀i, for α > 0and β < 0.(148) In the second case when α < 0, we obtain λα+β i α+β<−β α(α+β)= β α 1 α+β−→ λi< β α 1 α+β ∀i, for α < 0and β > 0.(149) Entropy 2015,17 3020 Using sign(αβ) = −1, we can solve for λα+β i, which yields λα+β i α+β> β α 1 α+β∀i. (150) Solving again for λi, we see that λi> β α 1 α+β ∀i, for α > 0and β < 0,(151) and λi< β α 1 α+β ∀i, for α < 0and β > 0.(152) In the limit, when α→ −β6= 0, these bounds simplify to lim α→−β β α 1 α+β =e−1/α ∀i, for β6= 0.(153) On the other hand, when α→0or when β→0, the bounds disappear. The lower-bounds converge to 0, while the upper-bounds converge to ∞, leading to the trivial inequalities 0< λi<∞. This concludes the determination of the domain of the eigenvalues that result in a finite divergence. Outside this domain, we expect D(α,β) AB (PkQ) = ∞. A complete picture of bounds for different values of αand βis shown in Figure 1. D. Proof of the Nonnegativity of D(α,β) AB (PkQ) The AB log-det divergence is separable; it is the sum of the individual divergences of the eigenvalues from unity, i.e., D(α,β) AB (PkQ) = n X i=1 D(α,β) AB (λik1),(154) where D(α,β) AB (λik1) = 1 αβ log "αλβ i+βλ−α i α+β#+ , α, β, α +β6= 0.(155) We prove the nonnegativity of D(α,β) AB (PkQ)by showing that the divergence of each of the eigenvalues D(α,β) AB (λik1) is nonnegative and minimal at λi= 1. First, note that the only critical point of the criterion is obtained when λi= 1. This can be shown by setting the derivative of the criterion equal to zero, i.e., ∂D(α,β) AB (λik1) ∂λi =λα+β i−1 αλα+β+1 i+βλi = 0,(156) and solving for λi. Entropy 2015,17 3021 Next, we show that the sign of the derivative only changes at the critical point λi= 1. If we rewrite ∂D(α,β) AB (λik1) ∂λi = λα+β i−1 α+β! λi αλα+β i+β α+β!−1 ,(157) and observe that the condition for the divergence to be finite enforces αλα+β i+β α+β>0, then it follows that sign (∂D(α,β) AB (λik1) ∂λi)≡sign (λα+β i−1 α+β)=                    −1for λi<1, 0,for λi= 1, +1 for λi>1. (158) Since the derivative is strictly negative for λi<1and strictly positive for λi>1, the critical point at λi= 1 is the global minimum of D(α,β) AB (λik1). From this result, the nonnegativity of the divergence D(α,β) AB (PkQ)≥0easily follows. Moreover, D(α,β) AB (PkQ) = 0 only when λi= 1 for i= 1, . . . , n, which concludes the proof of the Theorems 1 and 2. E. Derivation of the Riemannian Metric We calculate D(α,β) AB (P+dPkP)using the Taylor expansion when dPis small, i.e., (P+dP)P−1=I+dZ,(159) where dZ=dPP−1, α[(P+dP)P−1]β=αI+αβ dZ+αβ(β−1) 2dZdZ+O(|dZ|3). Similar calculations hold for β[(P+dP)P−1]−α, and α[(P+dP)P−1]β+β[(P+dP)P−1]−α= (α+β)I+αβ 2dZdZ, where the first-order term of dZdisappears and the higher-order terms are neglected. Since det I+αβ 2dZdZ= 1 + αβ 2tr(dZdZ),(160) by taking its logarithm, we have D(α,β) AB (P+dPkP) = 1 2tr(dP P−1dP P−1),(161) for any αand β. Entropy 2015,17 3022 F. Proof of the Properties of the AB Log-Det Divergence Next we provide a proof of the properties of the AB log-det divergence. The proof will only be omitted for those properties which can be readily verified from the definition of the divergence. 1. Nonnegativity; given by D(α,β) AB (PkQ)≥0,∀α, β ∈R.(162) The proof of this property is presented in Appendix D. 2. Identity of indiscernibles; given by D(α,β) AB (PkQ)=0 if and only if P=Q.(163) See Appendix D for its proof. 3. Continuity and smoothness of D(α,β) AB (PkQ)as a function of α∈Rand β∈R, including the singular cases when α= 0 or β= 0, and when α=−β(see Figure 2). 4. The divergence can be explicitly expressed in terms of Λ= diag{λ1, λ2, . . . , λn}, the diagonal matrix with the eigenvalues of Q−1P; in the form D(α,β) AB (PkQ) = D(α,β) AB (ΛkI).(164) Proof. From the definition of divergence and taking into account the eigenvalue decomposition PQ−1=VΛ V−1, we can write D(α,β) AB (PkQ) = 1 αβ log det αVΛβV−1+βVΛ−αV−1 α+β =1 αβ log det Vdet αΛβ+βΛ−α α+βdet V−1 =1 αβ log det αΛβ+βΛ−α α+β(165) =D(α,β) AB (ΛkI).(166) 5. Scaling invariance; given by D(α,β) AB (cPkcQ) = D(α,β) AB (PkQ),(167) for any c > 0. 6. For a given αand βand nonzero scaling factor ω6= 0, we have D(ω α, ω β) AB (PkQ) = 1 ω2D(α,β) AB ((Q−1/2PQ−1/2)ωkI).(168) Entropy 2015,17 3023 Proof. From the definition of divergence, we write D(ω α, ω β) AB (PkQ) = 1 (ωα)(ωβ)log det ωα Λωβ +ωβ Λ−ωα (ωα +ωβ)(169) =1 ω2 1 αβ log det α(Λω)β+β(Λω)−α (α+β)(170) =1 ω2D(α,β) AB ((Q−1/2PQ−1/2)ωkI)(171) Hence, the additional inequality D(α,β) AB ((Q−1/2PQ−1/2)ωkI)≤D(ω α, ω β) AB (PkQ)(172) is obtained for |ω| ≤ 1. 7. Dual-invariance under inversion (for ω=−1); given by D(−α,−β) AB (PkQ) = D(α,β) AB (P−1kQ−1).(173) 8. Dual symmetry; given by D(α,β) AB (PkQ) = D(β,α) AB (QkP).(174) 9. Affine invariance (invariance under congruence transformations); given by D(α,β) AB (APATkAQAT) = D(α,β) AB (PkQ),(175) for any nonsingular matrix A∈Rn×n. Proof. D(α,β) AB (APATkAQAT) = 1 αβ log detα((APAT)(AQAT)−1)β+β((APAT)(AQAT)−1)−α α+β =1 αβ log det α(A(PQ−1)A−1)β+β(A(PQ−1)A−1)−α α+β =1 αβ log det(AV) det αΛβ+βΛ−α α+βdet(AV)−1 =1 αβ log det αΛβ+βΛ−α α+β =D(α,β) AB (PkQ)(176) 10. Divergence lower-bound; given by D(α,β) AB (XTPXkXTQX)≤D(α,β) AB (PkQ),(177) for any full-column rank matrix X∈Rn×mwith n≤m. This result has been already proved for some special cases of αand β, especially these that lead to the S-divergence and the Riemannian metric [6]. Next, we present a different argument to prove it for any α, β ∈R. Entropy 2015,17 3024 Proof. As already discussed, the divergence D(α,β) AB (PkQ)depends on the generalized eigenvalues of the matrix pencil (P,Q), which have been denoted by λi, i = 1, . . . , n. Similarly, the presumed lower-bound D(α,β) AB (XTPXkXTQX)is determined by µi, i = 1, . . . , m, the eigenvalues of the matrix pencil (XTPX,XTQX). Assuming that both sets of eigenvalues are arranged in decreasing order, the Cauchy interlacing inequalities [29] provide the following upper and lower-bounds for µjin terms of the eigenvalues of the first matrix pencil, λj≤µj≤λn−m+j.(178) We classify the eigenvalues µjon three sets S− µ,S0 µand S+ µ, according to the sign of (µj−1). By the affine invariance we can write D(α,β) AB (XTPXkXTQX) = D(α,β) AB ((XTQX)−1/2XTPX(XTQX)−1/2kI)(179) =X µj∈S− µ D(α,β) AB (µjk1) + X µj∈S+ µ D(α,β) AB (µjk1),(180) where the eigenvalues µj∈S0 µhave been excluded since for them D(α,β) AB (µjk1) = 0. With the help of (178), the first group of eigenvalues µj∈S− µ(which are smaller than one) are one-to-one mapped with their lower-bounds λj, which we include in the set S− λ. Also those µj∈S+ µ(which are greater than one) are mapped with their upper-bounds λn−m+j, which we group in S+ λ. It is shown in Appendix D that the scalar divergence D(α,α) AB (λk1) is strictly monotone descending for λ < 1, zero for λ= 1 and strictly monotone ascending for λ > 1. This allows one to upperbound (180) as follows X µj∈S− µ D(α,β) AB (µjk1) + X µj∈S+ µ D(α,β) AB (µjk1) ≤X λj∈S− λ D(α,β) AB (λjk1) + X λj∈S+ λ D(α,β) AB (λjk1) ≤ n X j=1 D(α,β) AB (λjk1) (181) =D(α,β) AB (PkQ),(182) obtaining the desired property. 11. Scaling invariance under the Kronecker product; given by D(α,β) AB (Z⊗PkZ⊗Q) = n D(α,β) AB (PkQ),(183) for any symmetric and positive definite matrix Zof rank n. Entropy 2015,17 3025 Proof. This property was obtained in [6] for the S-divergence and the Riemannian metric. With the help of the properties of the Kronecker product of matrices, the desired equality is obtained: D(α,β) AB (Z⊗PkZ⊗Q) = 1 αβ log det α((Z⊗P)(Z⊗Q)−1)β+β((Z⊗Q)(Z⊗P)−1)α α+β =1 αβ log det α(I⊗PQ−1)β+β(I⊗QP−1)α α+β(184) =1 αβ log det I⊗α(PQ−1)β+β(QP−1)α α+β(185) =1 αβ log det α(PQ−1)β+β(QP−1)α α+βn (186) =n D(α,β) AB (PkQ).(187) 12. Double Sided Orthogonal Procrustes property. Consider an orthogonal matrix Ω∈ O(n)and two symmetric positive definite matrices Pand Q, with respective eigenvalue matrices ΛPand ΛQ, which elements are sorted in descending order. The AB log-det divergence between ΩTPΩ and Qis globally minimized when their eigenspaces are aligned, i.e., min Ω∈O(n)D(α,β) AB (ΩTPΩkQ) = D(α,β) AB (ΛPkΛQ).(188) Proof. Let Λdenote the matrix of eigenvalues of ΩTPΩQ−1with its elements sorted in descending order. We start showing that for ∆= log Λ, the function D(α,β) AB (exp ∆kI)is convex. Its Hessian matrix is diagonal and positive definite, i.e., with non-negative diagonal elements ∂2D(α,β) AB (e∆ii k1) ∂∆2 ii >0,(189) where ∂2D(α,β) AB (e∆ii k1) ∂∆2 ii =                              β α+βe−α+β 2∆ii +α α+βeα+β 2∆ii −2for α, β, α +β6= 0 eβ∆ii for α= 0 (1 + α∆ii)−2for α+β= 0 eα∆ii for β= 0. (190) Since f(e∆ii ) = D(α,β) AB (e∆ii k1) is strictly convex and non-negative, we are in the conditions of the Corollary 6.15 in [47]. This result states that for two symmetric positive definite matrices Aand B, which vectors of eigenvalues are respectively denoted by ~ λ↓ A(when sorted in descending order) and ~ λ↑ B(when sorted in ascending order), the function f(~ λ↓ A~ λ↑ B)is submajorized by f(~ λ↓ AB). By choosing A=ΩTPΩ,B=Q−1, and applying the corollary, we obtain D(α,β) AB (ΛPkΛQ) = D(α,β) AB (ΛPΛ−1 QkI)≤D(α,β) AB (ΛkI) = D(α,β) AB (ΩTPΩkQ),(191) Entropy 2015,17 3026 where the equality is only reached when the eigendecompositions of the matrices ΩTPΩ = VΛPVTand Q=VΛQVT, share the same matrix of eigenvectors V. 13. Triangle Inequality-Metric Distance Condition, for α=β∈R; given by qD(α,α) AB (PkQ)≤qD(α,α) AB (PkZ) + qD(α,α) AB (ZkQ).(192) Proof. The proof of this property exploits the recent result that the square root of the S-divergence dBh(PkQ) = pDS(PkQ) = 2slog det 1 2(P+Q) pdet(P) det(Q).(193) is a metric [17]. Given three arbitrary symmetric positive definite matrices P,Q,Z, with common dimensions, consider the following eigenvalue decompositions Q−1 2PQ−1 2=V1Λ1VT 1(194) Q−1 2ZQ−1 2=V2Λ2VT 2,(195) and assume that the diagonal matrices Λ1and Λ2have the eigenvalues sorted in a descending order. For a given value of αin the divergence, we define ω= 2α6= 0 and use properties 6 and 9 (see Equations (168) and (175)) to obtain the equivalence qD(α, α) AB (PkQ) = qD(ω0.5, ω 0.5) AB (PkQ) =r1 ω2D(0.5,0.5) AB ((Q−1/2PQ−1/2)ωkI) =1 2|α|qD(0.5,0.5) AB (Λ2α 1kI) =1 2|α|dBh(Λ2α 1kI),(196) Since the S-divergence satisfies the triangle inequality for diagonal matrices [5,6,17] dBh(Λ2α 1kI)≤dBh(Λ2α 1kΛ2α 2) + dBh(Λ2α 2kI),(197) from (196), this implies that qD(α,α) AB (PkQ)≤qD(α,α) AB (Λ1kΛ2) + qD(α,α) AB (Λ2kI)(198) In similarity with the proof of the metric condition for S-divergence [6], we can use property 12 to bound above the first term in the right-hand-side of the equation by qD(α,α) AB (Λ1kΛ2)≤qD(α,α) AB (V1Λ1VT 1kV2Λ2VT 2) =qD(α,α) AB (Q−1 2PQ−1 2kQ−1 2ZQ−1 2) =qD(α,α) AB (PkZ),(199) Entropy 2015,17 3027 whereas the second term satisfies qD(α,α) AB (Λ2kI) = qD(α,α) AB (V2Λ2VT 2kI) =qD(α,α) AB (Q−1 2PQ−1 2kI) =qD(α,α) AB (ZkQ).(200) After bounding the right-hand-side of (198) with the help of (199) and (200), the divergence satisfies the desired triangle inequality (192) for α6= 0. On the other hand, qD(α,α) AB (PkQ)as α→0converges to the Riemannian metric qD(0,0) AB (PkQ) = lim α→0qD(α,α) AB (PkQ)(201) =klog(Q−1/2PQ−1/2)kF(202) =dR(PkQ),(203) which concludes the proof of the metric condition of qD(α,α) AB (PkQ)for any α∈R. G. Proof of Theorem 3 This theorem assumes that the range spaces of the symmetric positive semidefinite matrices Cxand Cyare disjoint, in the sense that they only intersect at the origin, which is the most probable situation for nr(where nis the size of the matrices while ris their common rank). For ρ > 0the regularized versions ˜ Cxand ˜ Cyof these matrices are full rank. Let ˜ Λ= diag(˜ λ1,...,˜ λn)denote the diagonal matrix representing the neigenvalues of the matrix pencil (˜ Cx,˜ Cy). The AB log-det divergence between the regularized matrices is equal to the divergence between ˜ Λand the identity matrix of size n,i.e., D(α,β) AB (˜ Cxk˜ Cy) = D(α,β) AB ˜ C−1 2 y˜ Cx˜ C−1 2 ykIn=D(α,β) AB ˜ ΛkIn.(204) The positive eigenvalues of the matrix pencil satisfy ˜ Λ≡diag Eig+n(˜ Cy)−1 2˜ Cx(˜ Cy)−1 2)o= diag Eig+n˜ Cx˜ C−1 yo,(205) therefore, the divergence can be directly estimated from the eigenvalues of ˜ Cx˜ C−1 y. In order to simplify this matrix product, we first express ˜ Cxand ˜ C−1 yin term of the auxiliary matrices Tx=Ux(Λx−ρIr)1 2and Ty=Uy(Λy−ρIr)1 2.(206) In this way, they are written as a scaled version of the identity matrix plus a symmetric term: ˜ Cx=Cx+ρU⊥ x(U⊥ x)T =UxΛxUT x+ρ(In−UxUT x) =ρIn+Ux(Λx−ρIr)UT x =ρIn+TxTT x,(207) Entropy 2015,17 3028 and ˜ C−1 y=C+ y+ρ−1U⊥ y(U⊥ y)T =UyΛ−1 yUT y+ρ−1(In−UyUT y) =ρ−1In−ρ−1Uy(Λy+ρIr)Λ−1 yUT y =ρ−1In−ρ−1TyΛ−1 yTT y.(208) Next, using (207) and (208), we expand the product ˜ Cx˜ C−1 y=In+ρ−1TxTT x(In−TyΛ−1 yTT y) + R(209) and approximate the eigenvectors Uy→Uxof the residual matrix Rto obtain the estimate R≡ −Uy(Ir+ρΛ−1 y)UT y≈ −Ux(Ir+ρΛ−1 y)UT x≡ˆ R.(210) Hence, it is not difficult to see that the estimated residual is equal to ˆ R=−Tx(Ir+ρΛ−1 y)T+ x.(211) After substituting (211) in (209) and collecting common terms, we obtain the expansion ˜ Cx˜ C−1 y=In+Txρ−1TT x−ρ−1TT xTyΛ−1 yTT y−(Ir+ρΛ−1 y)T+ x | {z } \ ˜ Cx˜ C−1 y +O(ρ0).(212) Let Eig ≶1{·} denote the arrangement of the ordered eigenvalues of the matrix argument after excluding those that are equal to 1. For convenience, we reformulate the property proved in [30] that for any pair of matrices A,B∈Rm×n, the non-zero eigenvalues of ABTand of BTAare the same, into the following proposition. Proposition 1. For any pair of m×nmatrices Aand B, the eigenvalues of the matrices Im+ABT and In+BTA, which are not equal to 1, coincide. Eig ≶1Im+ABT=Eig ≶1In+BTA(213) Since range spaces of Cxand of Cyonly intersect at the origin, the approximation matrix \ ˜ Cx˜ C−1 y has rdominant eigenvalues of order O(ρ−1)and (n−r)remaining eigenvalues equal to 1. Using Proposition 1, these rdominant eigenvalues are given by Eig≶1\ ˜ Cx˜ C−1 y=Eig≶1Ir+ρ−1TT x−ρ−1TT xTyΛ−1 yTT y−(Ir+ρΛ−1 y)T+ xTx =Eig≶1ρ−1TT xTx−ρ−1TT xTyΛ−1 yTT yTx−ρΛ−1 y.(214) Let ˜ Λmax and ˜ Λmin, respectively denote the diagonal submatrices of ˜ Λwith the rlargest and with the r smallest eigenvalues. From the definitions in (66) and (206), one can recognize that TT xTx=Λx−ρIr,