A new approach to influence analysis in linear models

A New Approach to Influence Analysis in Linear Models Author(s): J. Muñoz-Pichardo, J. Muñoz-Garcia, J. L. Moreno-Rebollo and R. Pino-Mejias Source: Sankhyā: The Indian Journal of Statistics, Series A (1961-2002) , Oct., 1995, Vol. 57, No. 3 (Oct., 1995), pp. 393-409 Published by: Indian Statistical Institute Stable URL: https://www.jstor.org/stable/25051065 JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected]. Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at https://about.jstor.org/terms Indian Statistical Institute is collaborating with JSTOR to digitize, preserve and extend access to Sankhyā: The Indian Journal of Statistics, Series A (1961-2002) This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms Sankhya : The Indian Journal of Statistics 1995, Volume 57, Series A, Pt. 3, pp. 393-409 A NEW APPROACH TO INFLUENCE ANALYSIS IN LINEAR MODELS By J. MU?OZ-PICHARDO J. MU?OZ-GARC?A J.L. MORENO-REBOLLO and R. PINO-MEJIAS Universidad de Sevilla SUMMARY. We propose a new approach to the study of influence in the General Linear Model based on conditional bias. This approach enables us to apply such an analysis to all particular cases of this model. The theoretical foundation, on which this approach is based, does not presuppose a particular hypothesis on the distribution of the variables. Applying the results obtained to the Multiple Linear Regression Model, measures of influence are obtained as already proposed by other authors. Finally we carry out an application of the results on the analysis of covariance. 1. Introduction Influence Analysis arises when one is faced with the need to analyze the sensitivity of the results of a statistical analysis. This need is due to the fact that the conclusions of any analysis are drawn from methods based on sample observations and on asumptions on the underlying models of an experimenta tion. On the whole, in order to measure the influence that one or a set of obser vations has on the statistical analysis, slight disturbances are introduced into the model and the changes produced are quantified. In the literature, the most common disturbance pattern is that of the omission of the observations whose influence is to be studied. Cook (1987) tried to unify the problem under a general formulation which is valid for the different approaches and methods carried out on it. In this formulation, the key questions are the choice of the perturbation pattern and the choice of the model of comparison of the results obtained under the postulated model and the disturbed model. Paper received. August 1993; revised February 1994. AMS (1980) (revised 1985) subject classification. 62J99. Key words and phrases. Conditional bias, general linear model, influence observations. This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms 394MU?OZ PICHARDO, MU?OZ-GARCIA, MORENO-REBOLLO AND PINO-MEJIAS On the other hand, Influence Analysis has developed mainly in the Multiple Linear Regression Model (Belsley, Kuh, Welsch, 1980; Cook and Weisberg, 1982; Cook 1986, 1987; Chatterjee and Hadi, 1986, 1988) and, by extension, in the Multivariate Regression Model (Caroni, 1987; Barret and Ling, 1992). In this paper, we propose a new approach to study the influence on the General Linear Model (Kshirsagar, 1983) based on conditional bias. This approach permits us to apply such an analysis to all particular cases of this model. The theoretical foundation, on which this approach is based, does not presuppose a particular hypothesis on the distribution of the variables. As a particular case, by moving the results obtained from the General Linear Model to the Multiple Linear Regression Model, we obtain measures of influence that have been already proposed by other authors (Belsley et al. (1980); Atkin son (1982); Cook and Weisberg (1982);). Finally we carry out an application of the results on the analysis of covariance. 2. Conditional bias in the general linear model 2.1 The general linear model : Notation. The study of the General Linear Model is of great importance within statistics mainly due to its wide range of applications. It is used as a model for the development of other models, such as the multiple regression, experimental designs, analysis of covariance, etc. The general linear model is defined by Y = Xj3 + <:, E[e] = 0, Var(e) = a2In (GLM) where Y is a random n-vector; X is a known n x p matrix with rank r (r < p < n); ? is a p-vector of unknown parameters and e is an n-vector which represents the non-observable random perturbations. The fitted values vector is denoted by Y = VY, with V = XS~X' = ((u?))??=i,...,n *he prediction matrix, which is symmetric, idempotent, with rank r and unique for any S~ generalized inverse of S = X'X (Kshirsagar, 1983). The residuals vector is represented by e = Y ? Y = Me, where M = In ? V, and the least squares estimators of ? and a2 are denoted by ? and a2, respectively. On the other hand, the estimable linear parametric functions of ? and A/3, where A is a q x p matrix, with rank (A) = q, so that A/3 is the BLUE vector of the components of A?. Finally, given a collection of subindexes / = {?i,..., im} C {1,... ,n}, the m-vector composed of the components of Y subindicated by J is denoted by Y/. Similarly, the matrix formed by the rows of X corresponding to the collection / is denoted by X/, and the submatrix corresponding to V is represented by V/ = X/S~X'7. Likewise, the omission of the observations indexed by / is indicated by the subindex (/). 2.2. Conditional bias. The Decomposition Lemma from Efron and Stein (1981) expresses a statistic T defined on a simple random sample Yi,.. ., Yn as This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms INFLUENCE ANALYSIS IN LINEAR MODELS 395 a finite sum, whose terms are given according to the conditional expectations of the statistic T given the sample observations, which can be considered either individually or together. Taking this lemma as a reference, the following defini tions are given. Definition 2.1. Let Y\)..., Yn be a random sample of the random variable Y, let T ? T(Y\,..., Yn) be a statistic defined on the sample, and let y\,... yn be a realization of the sample. The conditional bias of T given the i-th observations is defined as J[yy,T) = E[T\Yl=yl]-E[T) .(2.1) The conditional bias J[yt;T] can be interpreted as the average effect that the realization of the i-th sample observation has on the statistic T. Therefore, it can be considered as a measure of the influence that such a sample realization has on T. By generalizing this, we can define the conditional bias given a set of obser vations. Definition 2.2. Under the conditions of Defintion 2.1, the conditional bias of T given the set of observations {yn,..., yim } is defined as J[y,l,...,ytm,T} = E[T\y,?=yl?,...,y,m=y,J-E[T} ...(2.2) Therefore, it is considered to be a measure of the joint influence of observa tions {yin...,y?m} on T. 2.3. Conditional bias of the general linear model estimators. We shall now calculate the conditional bias for the BLUE of an estimate linear function of the parameter vector and for the estimator a2. For Y/, the conditional bias is denoted by J\yil,...,yim:T\ = J[yr,T\ = E[T\Y, = yl]-E[T\ .(2.3) Without loss of generality, we assume that these m observations are the last observations. Thus, the following decompositions are obtained X = X(/) X/ ;Y = Y/ ;V: V(/) v0 V0 V/ ;In-V = M(/) Mo M?, M, ...(2.4) where V0 = X(/)S-X'/, M(/) = In_m - V(/), M/ = Im - V, and M0 = -V0. We assume that V/ and M/ are non-singular and that n > r + m. In the following theorem, we obtain the conditional bias for the BLUE of an estimable linear function. This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms 396MU?OZ PICHARDO, MU?OZ-GARCIA, MORENO-REBOLLO AND PINO-MEJIAS Theorem 2.3. Let A? be an estimable linear function, with A being a q x p matrix with rank (A) = q. Then J y/;A3]=AS-X'/[y/-Xi/3] ...(2.5) = AS = AS" Proof. E [A3 I Y7 = y/] - E [a/?] [e [X'7Y, I \j = y/] + E [x'(/)Y(/) | Y, - y,]} - A? x?y/+X'(/)X(/)/j]-A/3 = AS" [X'X? + X'I(y? - X,/?)] - A? = AS'X^y; - Xj?) ?7 Corollary 2.4. For m = 1 and I = {i}, J[y,;A?] = AS-x'i[yi-xi?} ...(2.6) where x? is the i-th row of X. The next lemma is useful to obtain the conditional bias for a2. Lemma 2.5. Given the partitioned matrices (2.4), we obtain MiX(/) = -[Im-V/]X/ ...(2.7) Proof. As V X = X, then M X = 0, with ? being the null matrix. Therefore Mi)X(/) = -M7X; = -[Im - V/]X/ Z7 Theorem 2.6. The conditional bias of d2 given the set of sample obser vations indexed by I is J [yi\?2] = ? {[y/ - X//3)' [Im - V/] [y7 - X,?) - ahr[lm - V,]} n ? r ;..(2.8)' Proof. The unbiased estimator of o2 can be expressed in the following way a2 = 1 n ? r Y'(;)M(/)Y(/) + 2Y/M^Y(/) + Y/M/Y, ...(2.9) Besides E [Y'/M/Y/ | Y, = y/] = y, [I - V,] y?, E [Y'/M^Y^ | Y, = y/] = y'/M()X(/)l3 = -y',[Im - V/JX//3 and E [y'(/)M(/)Y(/) | Y/ = y,j = ?? [y'(/)M(/)Y(/)] = Ea - noctli] - EE ?*.^my.] = *2 Hi Hi "tiki* Et1-***) +E(1-^))x^)2-E k$i ^2 vks(xs?) k$I ls?Is?k (xfc/9) This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms INFLUENCE ANALYSIS IN LINEAR MODELS 397 a2{n-r-tr [lm - V,}} + ?(1 - vkk)(xk?)2 k<$l (l-vuXx^-J^Vkfafl Hi = <r2{n-r-tr[lm-V!]} + Y, je/ (x*/3) jel = <r2{n_r_tr[Im_V/]} + ^ J6/ J] vjk(xk?) H' (x>/9) (1 - %)(Xj/3) 452 M*/3) = a2 {n - r - tr [lm - V,]} + [X//9]' [Im - V,] [X,?} Therefore, the following is obtained E [a2 | Y, - y/] = a2 il - -^ir [Im - V,]} (x>0) + n ? r [yz-X/flX-V/Hyz-X,/?] from which we can deduce the proposed result. Corollary 2.7. For m = 1 arad J = {?}, J [yr,?2] = -i- [i _ v. ] J[2/l _ Xl/?f _ ff2} n ? r l J Z7 ,..(2.10) 2.4. An estimation of the conditional BIAS. In the results previously ob tained we observe that the conditional bias depends on the unknown parameter ? and a2. The following theorem offers an estimation of the conditional bias of an estimator V = V(Y?,..., Yn) of a vector 0 of parameters. Theorem 2.8. Let Y\,..., Yn be a random sample of the random variable Y, whose distribution depends on an unknown parameter 0 G ft ? $tp. Let V = V(Yi,..., Yn) be an unbiased estimator of 0 and let V(i\,... , ?m) be the unbiased estimator obtained from the sample with the omission of the observations {Y?t | k G {i\,... , ?m}}- Then E[v-V(il.im)\Yil=lHl,---,Yim=yim J y?!> ,mm-M ...(2.11) The result follows directly from the fact that V(?lv..,?m) is unbiased and does not depend on Yh,..., Yim. Hence, we denote VV(tl ...lm) by J \yiL,..., yim; VI. In order to estimate the conditional bias either of the BLUE of an estimable linear parametric function for the model (GLM), or of a2, we must obtain the estimators corresponding to the model in which the observations indicated by / are omitted. This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms 398MU?OZ PICHARDO, MU?OZ-GARCIA, MORENO-REBOLLO AND PINO-MEJIAS The resulting linear model with such observations omitted is Y(/)=X(/)/3 + e(/) [GLM(/)J where Y(/), X(/), (/) are the corresponding subvectors or submatrices associated with the set of subindexes /. For the model, the least squares estimators are expressed as 0(0 = S(/)X(/)Y(/) + I1* ~~ Hi/)]^' ue^? arbitrary ...(2.12) with S(7) = X/7vX(/) and H, (/) s(/)s(/), Theorem 2.9. If A? is an estimable linear function for (GLM), it is also for the model [GLM(J)] Proof. We consider the following decomposition X/ x </> v = v{, v(/)J As VX = X, then we obtain V/X/ + V0X(/) = X/. Likewise, if A? is estimable for the model (GLM), there exists a q x n matrix A, such that A = A M7!V0 Im X/A => A = BX/A where B = A Hi) (0 Im MJ1V0 Then A? is an estimable linear function for the model [GLM(I)]. Z7 As a consequence, if A/3 is an estimable linear function for (GLM), then the BLUE for [GLM(J)] is A?yy The following results, Lemma 2.10 and Theorem 2.11, allow us to obtain the estimation of the conditional bias of A?. Lemma 2.10. Let A be a p x q matrix and L be a m x p matrix (m < p), such that L belongs to the row space generated by the matrix A, and the inverse of the matrix [I - L A~L'] exists. Therefore, the matrices { A~ - A~L'[I - LA-L'l^LA" } are generalized inverses of [A - L I/]. Proof. It suffices to base the proof on Pringle and Rayner (1971), where the equalities LA~ A = L and AA'L; = L' verify if L belongs to the row space of A. Theorem 2.11. The least squares estimators of ? in [GLM(I)] can be expressed as ?(I) = S~X'Y - S^X'/M71e/ + (Ijk - H)i/, ueW arbitrary. Proof. From Lemma 2.10, we obtain ...(2.13) '(/) (s - x'jXiy = s- + s-xjMj^/S This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms INFLUENCE ANALYSIS IN LINEAR MODELS 399 on the other hand H(/) = H - S-X'7X/ + S"XJM71X/ - S'X'jMJ1 V/X/ = H Therefore, based on (2.12), we obtain fl{1) = (S" 4S-X'jM^X/S-XX'Y - X'7Y/) 4- (h - H)i/ = S-X'Y - S~X,/M71(Y/ - X/S-X'Y) 4- (I* - H)?> = S-X'Y - S~X'IMjleI 4- (Ifc - H> ?7 The estimation of the conditional bias of A? is obtained in the following. Corollary 2.12. J\yi;A?] = AS-X,/M71e/ ... (2.14) In relation to the unbiased estimator of the parameter a2, we obtain ^2 a(0 1 Y(/) - X(/)3(/) Y(/) - X(/)/J(/) n ? r ? m The relation between (2.9) and (2.15) is stated below, Theorem 2.13. In the model [GLM(I)] 1 ...(2.15) '(/> Proof. n ? r ? m ^ -i ! [(n - r)a2 - e',M^e,} ...(2.16) |Y(/) - X(/)/3(/)j |Y(/) - X(/)/3(/)J = Y'(/)Y(/) - Y'(/)X(/)0(/) = [Y'Y - Y'jYi) - [Y'X - Y'7X/] [S"X'Y - S-X'/M7Ie/ + (I - H>] = Y'(I - V)Y - Y^Y/ + ?X'jMjlei + Y',Xi? - YJ V/M^e/ = Y'(I - V)Y - Y^e, + /3X'/M71e/ - Y',\?M.]le, = Y'(I - V)Y - [Y',[I + V/M/1] - /3X'/M71] e7. = Y'(I - V)Y - [Y'/M71 - /3X'/M71] e7 = Y'(I - V)Y - e/M71e/ (n ? r)d2 - e'/M71e/ from which we can deduce what was previously stated. ?7 Consequently, we obtain the estimation of the conditional bias of a2. Corollary 2.14. J[y,;?2) = 1 m e'?M^ef n ? r ? m ...(2.17) This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms 400mu?oz pichardo, munoz-garcia, moreno-rebollo and pino-mejias 3. Influence measures Corollary 2.12 and 2.14 enable us to propose the following influence measures for the estimators A? and a2 in the (GLM). 3.1 Influence measures on A?. Based on Corollary 2.12, J[yi]A?] = AS-X,/M71e/ G SP can be interpreted as a measure of the influence that the set of observations, subindicated by J, has on A?. And, can be interpreted as a measure of the influence that the i-th observation has on A?. These influence measures on the BLUE of an estimable linear function be long to the ?-dimensional real space. So, in order to characterize the influence, a metric of generalized distance type will be applied, according to the charac terization given by Barnett (1976). Hence, given a symmetric, positive definite matrix Q and a positive scalar c, we define the (Q, c)-norm of a vector X as ll*ll(Q,0 = (Uc)X'QX Considering the matrix Q = (AS~A')_1, with Aagxp matrix such that A? is linearly estimable, and an adequate choice of the scalar c, we can establish the following norms. (I) For c\ = qa2, DJ [A/3] = \\J[y? : A/3]||(Qfi), called ?^-DISTANCE associated with the set of observations yj. Dj[A?] = \\J[y? : A?}\\(QCi), called D/-DISTANCE associated with the set of observations y?. Easily, we obtain the following expressions for DJ [A/3] and Dj[A?] : D}[A?] = 4? [y/ - X//3]'X/S-A' [AS"A']"1 AS"X'7 [yi - X//3] ... (3.1) D/[Aj8) = ^e//M71X/X-A' [AS-A;]_1 AS-X'/M71e/ ... (3.2) In particular, in the study of the influence of a single observation, that is to say I = {i}, the D*-distance and ?),-distance associated with ?/j are, respectively, D\[A?) = -^XiS-A' [AS'A7]"1 AS-x; [yt - xt/3]2 ... (3.3) This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms INFLUENCE ANALYSIS IN LINEAR MODELS 407 320 240 H I 160 H J_I_I-1_I_L J_L D SUED OH ?OO*? 0000*0*0** *00000O*00?0**0%*****0 i-1-1-1-1-I-1-r 0 5 10 15 20 25 30 35 ID Fig. 7. Plot of SMED "?" 6H OH -M -12 H j?i?i?i?i_i_t__? i D SUED O 000 oo $ 0 00? O O ??0?> ? 0 ? * "1-1-1-1-1-1-1-T" 0 5 10 15 20 25 30 35 10 Fig. 8. Plot SMED (deleted case 6) This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms 408MU?OZ PICHARDO, MU?OZ-GARCIA, MORENO-REBOLLO AND PINO-MEJIAS The influence measures, SMED?, on the unbiased estimator of the variance a2, Table 2, are represented in FIG. 7 and FIG. 8. From these we can draw the conclusion that observation number 6 has a considerable greater influence on this estimator than the rest of the observations. However, observations 13 and 15 have very little effect on this estimation, as opposed to that obtained for the influence analysis on [7172]'. Table 2. INFLUENCE MEASURE ON ?2 ? -9.164 T? T864 2? -10.394 31 -2.843 2 -10.605 12 -10.693 22 -10.252 32 -10.163 3 -9.511 13 -3.650 23 -10.661 33 0.514 4 -5.167 14 -10.699 24 -10.454 34 -10.391 5 -10.527 15 -0.535 25 -7.676 35 -4.702 6 302.158 16 -7.076 26 -10.130 36 -4.653 7 -8.926 17 5.536 27 -8.400 37 -4.614 8 -10.505 18 -4.695 28 -8.062 38 -6.174 9 -10.623 19 -9.614 29 -10.653 39 -2.303 10 -10.231 20 -9.519 30 -4.703 40 -10.016 5. Conclusion In conclusion, in this paper, we propose : A new theoretical approach to the study of influence, that it can be ex tended to different statistical models. This approach is based on conditional bias of a given statistic T and does not presuppose hypothesis on the underlying distribution. Influence measures for the statistics of interest in the general linear model. So, we propose D/-distance W/-distance and C/-distance to quantify the in fluence of observations indexed by I on A?. For the linear regression model, the distances coincides with Cook's distance, squared Welseh-Kuh distance and modified Cook's distance, respectively. Nevertheless, these distances can be ap plied to all particular cases of (GLM). Finally, we propose a measure of the influence on a2, SMEDj. REFERENCES Atkinson, A.C. (1982). Regression Diagnostics. Transformations and Constructed Variables. Jour. Roy. Statist. Soc, Ser. B, 44, No. 1, 1-36. BARNETT, V. (1976). The ordering of multivariate data. Jour. Roy. Statist. Soc, Ser. A, 139, Part. 3, 318-344. Barrett, B.E. and Ling, R.F. (1992). General Classes of Influence Measures for Multivariate Regression. Jour. Amer. Statist. Asssoc, 87, No. 417, 184^191. Belsey, D.A., KUH, E. and WELSCH, R.E. (1980). Regression Diagnostics : Identifying Influential Data an Sources of Collinerity, New York : John Wiley & Sons. This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms INFLUENCE ANALYSIS IN LINEAR MODELS 409 Caroni, C. (1987). Residuals and influence in the multivariate linear model. The Statistician, 36, 365-370. CHATTERJEE, S. and Hadi, A.S. (1986). Influential observations, high leverage points and outliers in linear regression. Statistical Science, 1, No. 3, 379-416. -(1988). Sensitivity Analysis in Linear Regression, New York : John Wiley and Sons. Cook, R.D. (1987). Influence assessment. Jour. Appl. Statist., 14, No. 2, 117-132. COOK, R.D. and WEISBERG, S. (1982). Residuals and Influence in Regression, New York : Chapman and Hall. Efron, B. and Stein, C. (1981). The jackknife estimate of variance. Ann. Statist., 9, No. 3, 586-596. FRANE, J., JennricH, R. and SAMPSON, P. (1990). BMDP Statistical Software Manual, Ed. Dixon, W.J. University of California Press, Berkeley. KSHIRSAGAR, A.M. (1983). A Course in Linear Models, New York : Marcel Dekker, Inc. Pringle, R.M. and RAYNER, A.A. (1971). Generalized Inverse Matrices with Applications to Statistics, London : Charles Griffin and Co. Departamento de Estad?stica e Investigaci?n Operativa Facultad de Matem?ticas. Universidad de Sevilla C/. Tarfia s/n. 41012 - SEVILLA ESPA?A. This content downloaded from 150.214.182.24 on Thu, 30 Jun 2022 07:41:56 UTC All use subject to https://about.jstor.org/terms