All about the ⊥ with its applications in the linear statistical models

© 2015 Augustyn Markiewicz and Simo Puntanen licensee De Gruyter Open. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License. Open Math. 2015; 13: 33–50 Open Mathematics Open Access Review Article Augustyn Markiewicz and Simo Puntanen* All about the ?with its applications in the linear statistical models Abstract: For an nmreal matrix Athe matrix A?is defined as a matrix spanning the orthocomplement of the column space of A, when the orthogonality is defined with respect to the standard inner product hx;yi D x0y. In this paper we collect together various properties of the ?operation and its applications in linear statistical models. Results covering the more general inner products are also considered. We also provide a rather extensive list of references. Keywords: Best linear unbiased estimator, Column space, Generalized inverse, Linear statistical model, Orthocomplement, Orthogonal projector MSC: 15A99, 62H12, 62J05 DOI 10.1515/math-2015-0005 Received December 13, 2013; accepted April 14, 2014. 1 Introduction Consider a columnwise partitioned matrix AD.a1W: : : Wam/2Rnm(the set of nmmatrices with real elements). Then the column space of Ais defined as C.A/D fz2RnWzDAt Da1t1CCamtmfor some t2Rmg: The notation C.A/?refers to the orthocomplement of C.A/, i.e., the set of vectors which are orthogonal (with respect to the standard inner product) to every vector of C.A/: C.A/?D fu2RnWu0At D0for all t2Rmg: Thus, because C.A/?D fu2RnWu0At D0for all t2Rmg D fu2RnWA0uD0g; we have C.A/?DN.A0/Dthe null space of A0: Now A?is defined as a matrix whose column space is C.A?/DC.A/?DN.A0/. In view of decomposition RnDC.A/˚C.A/?;where ˚refers to the direct sum, the rank of A?is rank.A?/Dnrank.A/: The set of all matrices A?is denoted as fA?gand hence: Z2 fA?g ” (a) A0ZD0and (b) rank.Z/Dnrank.A/: (1) Augustyn Markiewicz: Department of Mathematical and Statistical Methods, Pozna´ n University of Life Sciences, Wojska Polskiego 28, PL-60637 Pozna´ n, Poland, E-mail: [email protected] *Corresponding Author: Simo Puntanen: School of Information Sciences, FI-33014 University of Tampere, Finland, E-mail: [email protected] 34 A. Markiewicz, S. Puntanen We immediately observe that Z2 fA?g ” A2 fZ?g:Trivially A?is unique only when Ais a nonsingular square matrix in which case A?D0. Notice that A2RnmH) A?2Rns;where snrank.A/: In this paper our purpose is to review various features of the ?operation, the “perp-operation”, say, and in particular, to present several useful applications related to linear statistical models. Results covering the more general inner products are also considered. We believe that our review provides a useful summary of the ?operation and thereby increases the insights and appreciation of this, seemingly simple, operation. 2A?in terms of generalised inverses The generalised inverses offer a very handy tool for explicit expressions of the A?, and in this section we give a short tour into such possibilities. Matrix G2Rmnis a generalised inverse of A2Rnmif AGA DA;(mp1) and it is the Moore–Penrose inverse, denoted as AC, if it also satisfies the following three conditions: (mp2)GAG DG;(mp3).AG/0DAG ;(mp4).GA/0DGA : If Gsatisfies the condition AGA DA, we may denote GDA, or G2 fAg. As the excellent references for the generalised inverses, see Ben-Israel & Greville [10] and Rao & Mitra [46]. In particular, for more of about the Moore (of Moore & Penrose), see Ben-Israel [11]. It is well known that the nullspace N.A/can be expressed as N.Anm/DC.ImAA/ ; where Acan be any generalized inverse of A. Hence we can express C.A/?in terms of A: C.A/?DN.A0/DCŒIn.A0/A0DCŒIn.A/0A0 : (2) The last equality above follows from the fact f.A/0g D f.A0/g:(3) Notice that it is a bit questionable to write .A/0D.A0/because (3) means the equality between two sets. However, for the (unique) Moore–Penrose inverse we always have .AC/0D.A0/C:In light of (2), we have, for example, the following choices for A?(recalling that A2Rnm): In.A0/A0;In.A/0A0;InA.A0A/A0;(4) where we have used the fact A.A0A/2 f.A0/g. By replacing .A0/with .A0/Cin (4) and using .A0/CD.AC/0; .A0/CA0DAAC; we get InAACWD InPAWD QA2 fA?g:(5) It can be shown that if Gsatisfies the conditions (mp1) and (mp3), i.e., G2 fA 13gthen AG is unique and thereby AA 13 DAAC;and hence InAA 13 is one choice for A?. The notations PAand QAin (5) refer to the orthogonal projectors onto C.A/(with respect to the standard inner product) and C.A/?, respectively. Matrix Pis defined as the orthogonal projector onto C.A/if it satisfies the following conditions: PDP0DP2and C.P/DC.A/; (6) All about the ?with its applications in the linear statistical models 35 which can be shown to be equivalent to P.AWA?/D.AW0/; where .AWA?/and .AW0/denote partitioned matrices. The matrix Psatisfying (6) is unique and can be written as PADAACDAA 13 DA.A0A/A0, where the last expression is invariant for any choice of .A0A/; this follows from Rao & Mitra [46, Lemma 2.2.4], which says that for nonnull Aand C, the matrix product ABCis invariant with respect to the choice of the generalized inverse B if and only if C.C/C.B/and C.A0/C.B0/. Notice that AAis not necessarily an orthogonal projector: it is idempotent and it satisfies C.AA/DC.A/but it is not necessarily symmetric. Below is a summary of some of the expressions for A?with obvious extensions to .A0/?in terms of generalised inverses. Theorem 2.1. For A2Rnm, denote QADInAACDInPA. Then (a) In.A0/A02 fA?g; (b) In.A/0A02 fA?g; (c) InPADQA2 fA?g; (d) InAA 13 2 fA?g; (e) ImAA2 f.A0/?g; (f) ImA0.AA0/ADImACADImPA0DQA02 f.A0/?g: Obviously the orthogonal projector QADInAACis often a convenient choice for A?because it is symmetric and idempotent. 3 Some specific formulas Suppose that Zis a choice for A?. Then, for a comformable matrix B, we have ZB 2 fA?g(7) whenever rank.ZB/Drank.Z/. According to Marsaglia & Styan [34, Cor. 6.2] (see Theorem 4.1 below), rank.ZB/Drank.Z/dim C.Z0/\C.B/?; and thereby (7) holds if and only if C.Z0/\C.B/?D f0g:Thus we have the following simple result: Theorem 3.1. Let A2Rnm, and B2Rnq. Then for any A?2Rnnwe have A?B2 fA?g ” CŒ.A?/0\C.B/?D f0g: In particular, choosing QAas A?yields QAB2 fA?g ” C.AWB/DRn; where .AWB/denotes the partioned n.m Cq/ matrix. In the next theorem we take a look at the perps of some particular partitioned matrices. Theorem 3.2. Let A2Rnm, and B2Rnq. Then for any A?we have (a) A?0 0 Iq!28 < : Anm 0qm!?9 = ; ; (b) In B0!A?28 < : AnmBnq 0 Iq!?9 = ; ; (c) In B0!28 < : Bnq Iq!?9 = ; : 36 A. Markiewicz, S. Puntanen Proof. Part (a) is obvious as the orthogonality condition corresponding to (1a) trivially holds and rank A?0 0 Iq!Dnrank.A/Cq: To prove (b), we observe that Œ.A?/0W .A?/0B A B 0 Iq!D0: Moreover, the rank of In B0A?is rank.A?/0W .A?/0BDrankŒ.A?/0Dnrank.A/ ; while rank AnmBnq 0 Iq!? DnCqrank Anm 0!rank Bnq Iq! DnCqrank.A/qDnrank.A/ : Thus (b) is confirmed. Part (c) can be proved in the corresponding way. Theorem 3.3. Consider A2Rnmand B2Rtm. Then for any A?and B?we have A?0 0 B?!28 < : A B!?9 = ; if and only if C.A0/\C.B0/D f0g: Proof. The orthogonality condition (1a) obviously holds while rank A?0 0 B?!DnCtrank.A/rank.B/; rank A B!? DnCtrank.A/rank.B/Cdim C.A0/\C.B0/: Thus the proof is completed. Remark 3.4. It might be a bit tempting to rewrite part (a) of Theorem 3.2 as Anm 0qm!? D A?0 0 Iq!:(8) However, expression like (8) is obviously problematic, and the meaning of the above notation should be clarified. One interpretation for (8) might be to agree that it means that 8 < : Anm 0qm!?9 = ;D( A?0 0 Iq!):(9) In other words, the sets of matrices are identical. However, the statement (9) is incorrect as can be concluded by Theorem 3.5 below. Let us ask the following: which matrices B2Rnpand D2Rqpsatisfy the following: A?Bnp 0 Dqp!28 < : Anm 0qm!?9 = ; ‹ All about the ?with its applications in the linear statistical models 37 We first observe that equation .A?/00 B0D0! Anm 0qm!D 0nm 0qm! holds if and only if C.B/C.A/?. Supposing that C.B/C.A/?, then, in view of Marsaglia & Styan [34, Cor. 19.1], the rank of A?B 0 D is additive on the Schur complement, i.e., rank A?B 0 D!Drank.D/Crank.A?BD0/Drank.D/Crank.A?/Drank.D/Cnrank.A/ : On the other hand, because rank Anm 0qm!? DnCqrank.A/ ; we immediately obtain the following: Theorem 3.5. Consider A2Rnm,B2Rnp, and D2Rqp. Then the relation A?B 0 D!28 < : A 0!?9 = ; holds if and only if C.B/C.A/?and Dhas full row rank. 4 Two rank formulas and a decomposition of orthogonal projector Two particular rank formulas in terms of the orthocomplement are worth special praising due to their numerous applications particularly when dealing with linear statistical models: the rank of the product AnaBamand the rank of the partitioned matrix .AnaWBnb/. Theorem 4.1. The rank of the partitioned matrix .AnaWBnb/can be expressed as rank.AWB/Drank.A/CrankŒ.A?/0BDrank.A/Crank.B0A?/ ; (10) and the rank of the matrix product AnaBamis rank.AB/Drank.A/dim C.A0/\C.B/?:(11) In terms of an arbitrary generalized inverse A, (10) can be expressed as rank.AWB/DrankŒAW.InAA/BDrank.A/CrankŒ.InAA/BDrank.A/CrankŒ.InPA/B : (12) As a reference to (10) and (12) we may mention Marsaglia & Styan [34, Th. 5]. For the references to (11), see, e.g., Marsaglia & Styan [34, Cor. 6.2], Rao [43, p. 28], and Zyskind & Martin [59, p. 1194]. We may also mention that O.M. Baksalary & Trenkler [8] provide several expressions for the ranks of a product of two matrices and of a column-wise partitioned matrix as well as an extensive list of related references. Several applications of (10) and (11) appear in Puntanen, Styan & Isotalo [39, Ch. 5]. One example concerns the decomposition of the column space C.XWVX?/, where X2Rnpand Vis an nn(symmetric) nonnegative definite matrix. Such a situation occurs when we consider the general linear model yDXˇC";denoted as MD fy;Xˇ;Vg;(13) where Xis a known npmodel matrix, the vector yis an observable n-dimensional random vector, ˇis a p1 vector of unknown parameters, and "is an unobservable vector of random errors with expectation E."/D0;and covariance matrix cov."/DV. Then we have the following; see, e.g., Rao [45, Lemma 2.1]. 38 A. Markiewicz, S. Puntanen Theorem 4.2. Consider the linear model MD fy;Xˇ;Vg, defined as in (13). Then C.XWV/DC.XWVX?/DC.X/˚C.VX?/ : Moreover, if the model is correct, in which case it is called consistent, then the observed (realized) value of the random vector ysatisfies y2C.XWV/ : (14) For a discussion concerning the consistency concept, see, e.g., Puntanen & Styan [38], J.K. Baksalary, Rao & Markiewicz [5], Groß [16, p. 314], and Tian et al. [53]. In this paper, we assume that the corresponding consistency holds whatever model we have. When working with linear models, we often need to consider the orthogonal projector onto the column space of the partitioned matrix. Then the following theorem appears to be very convenient in various connections; see, e.g., Puntanen, Styan & Isotalo [39, Th. 8] and Seber & Lee [50, Appendix B]. Theorem 4.3. The orthogonal projector (with respect to the standard inner product)onto the column space C.AnaWBnb/can be decomposed as P.AWB/DPACP.InPA/BDPACPC.AWB/\C.A/?:(15) We complete this section by some remarks on the explicit expression for the intersection of C.A/and C.B/. For a reference, see Rao & Mitra [46, Complement 7, p. 118]. Theorem 4.4. Consider the matrices Anaand Bnband denote QBDInPB. Then C.A/\C.B/DCA.A0B?/?DCA.A0QB/?DCA.IaPA0QB/DCAŒIa.A0QBA/A0QBA: It is obvious that C.A/\C.B/?DCA.A0B/?DCA.IaPA0B/: In particular, if X2Rnpand Vnnis nonnegative definite, then C.X/\C.V/?DCX.X0V/?DCX.X0VX/?DCXŒIp.X0VX/X0VX; and in view of C.M/\C.V/?DC.XWV/?, MŒIn.MVM/MVM2˚.XWV/?; where MDInPX. Notice also that according to Theorem 4.3 we have P.XWV/DPXCPMV and thereby InP.XWV/DMPMV DM.InPMV/2˚.XWV/?: 5 Orthocomplement when the inner product matrix is V 5.1 Vis positive definite Consider now the inner product in Rndefined as hx;yiVDx0Vy ;where Vis a positive definite symmetric matrix. The orthocomplement of C.Anm/with respect to this inner product is C.A/? VD fy2RnWz0A0Vy D0for all z2Rmg: By A? Vwe will denote any matrix whose column space is C.A/? V. Recall that A? Iis shortly denoted as A?. We have C.A/? VD fy2RnWA0Vy D0g D N.A0V/DC.VA/?DC.V1A?/ ; All about the ?with its applications in the linear statistical models 39 where the last equality can be concluded from A0VV1A?D0H) C.V1A?/N.A0V/ ; and rank.V1A?/Drank.A?/Dnrank.A/Ddim C.VA/?: Notice that corresponding to (1), Z2 fA? Vg ” (a) A0VZ D0and (b) rank.Z/Dnrank.A/: Remark 5.1. Obviously we can write V1A?2 fA? Vgand V1A?2 f.VA/?g. Question: Is it correct to write fA? Vg D f.VA/?g‹ It is easy to confirm that the answer is positive. Now we have the following decomposition: RnDC.A/˚C.A/? VDC.A/˚C.V1A?/ ; and hence every y2Rnhas a unique representation as a sum yDAb CV1A?cDyCP y; for some band c. The vector yDAb is the orthogonal projection of yonto C.A/along C.A/? V. The orthogonal projector PAIVis such a matrix which transforms yinto its projection y, i.e., PAIVyDyDAb. Its explicit unique representation is PAIVDA.A0VA/A0V: We may mention that part (a) of Theorem 3.2 holds even if the inner product matrix is V, i.e., A? V0 0 Iq!28 < : Anm 0qm!? V9 = ; : Similarly Theorems 3.3 and 3.5 hold also when all orthocomplements are taken with respect to the inner product matrix V. 5.2 Vis nonnegative definite, possibly singular Let Vbe a singular nonnegative definite matrix. Then ht;uiVDt0Vu is a semi-inner product and the corresponding seminorm (squared) is ktk2 VDt0Vt. For a singular nonnegative definite matrix Vwe can define the matrix A? Vagain as any matrix spanning C.A/? V, and so C.A? V/DC.A/? VDN.A0V/DC.VA/?: As noted by Puntanen, Styan & Isotalo [39, §2.5] for (even) a singular Vwe do have the decomposition RnDC.A/CC.A? V/DC.A/CC.VA/?;(16) but, however, the above decomposition is not necessarily a direct sum. For any nonnegative definite Vwe have, on account of Theorem 4.1, dim C.VA/?Dnrank.VA/DŒn rank.A/ Cdim C.A/\C.V/?; which means that (16) becomes a direct sum decomposition if and only if C.A/\C.V/?D f0g. For the characterization of the generalized orthogonal projector, see Mitra & Rao [36]. Some related considerations appear also in Harville [19, §14.12.i], Rao & Rao [47, p. 81], and Tian & Takane [54, 55]. 40 A. Markiewicz, S. Puntanen 5.3 Some further considerations Consider the linear model MD fy;Xˇ;Vg, defined as in (13), and let Vbe positive definite. Then we have observed that the following sets are identical: (a) C.X/? V1;(b) C.VX?/ ; (c) N.X0V1/ ; (d) C.V1X/?;(e) N.PXIV1/ ; (f) C.InPXIV1/ : For (a), .. . , (f) above, see also Puntanen, Styan & Isotalo [39, §5.13]. When Vis singular, the above considerations become more complicated. A very convenient tool appears to be the following class of matrices: WD fW2RnnWWDVCXUX0;C.W/DC.XWV/g:(17) In (17) Ucan be any ppmatrix as long as C.W/DC.XWV/is satisfied. Of course, Ucan be chosen as 0if C.X/C.V/which happens, for example, when Vis positive definite. The set Wof matrices has an important role in the theory of linear models. Below are listed some useful equivalent statements concerning W: C.X/C.W/ ; (18a) C.XWV/DC.W/ ; (18b) X0WXis invariant for any choice of W;(18c) C.X0WX/DC.X0/for any choice of W;(18d) X.X0WX/X0WXDXfor any choices of the generalized inverses involved. (18e) Moreover, each of these statements is equivalent to C.X/C.W0/, and hence to the statements (18b’)–(18e’) obtained from (18b)–(18e), by setting W0in place of W. As the references to (18), we may mention J.K. Baksalary, Puntanen & Styan [4, Th. 2], J.K. Baksalary & Mathew [3, Th. 2], and Harville [19, p. 468]. According to Puntanen, Styan & Isotalo [39, §5.13] the following now holds. Theorem 5.2. Suppose that Xis an npmatrix, Vis an nnnonnegative definite matrix and W2W, where W is defined as in (17). Then C.VX?/DC.WXWInWW/?; where Wis an arbitrary (but fixed)generalized inverse of W. The column space C.VX?/can be expressed also as C.VX?/DC.W/0XWIn.W/0W0?: Moreover, let Vbe possibly singular and assume that C.X/C.V/. Then C.VX?/DC.VXWInVV/?C.VX/?; where the inclusion becomes equality if and only if Vis positive definite. Remark 5.3. It is of interest to note that the perp symbol ?drops down, so to say, very “nicely” when Vis positive definite: C.VX?/?DC.V1X/ ; but when Vis singular we have to use a much more complicated rule to drop down the ?symbol: C.VX?/?DC.WXWInWW/ ; where W2W. Remark 5.4. Let us next prove the following: If W2W, where Wis defined as in (17), then C.VX?/DC.WX/?”C.XWV/DRn:(19) All about the ?with its applications in the linear statistical models 41 We first observe that C.VX?/DC.WXWInWW/?DC.WX/?\C.InWW/?: Thus we always have C.VX?/C.WX/?;where the equality appears only if dim C.WX/?Drank.VX?/. Now we have rank.VX?/Drank.W/rank.X/ ; dim C.WX/?Dnrank.WX/Dnrank.X/ ; from which our claim (19) follows. For completeness we state the following related result, due to Rao & Mitra [46, p. 140]. Theorem 5.5. Consider the linear model MD fy;Xˇ;Vgand denote WDVCXUX0, where C.W/DC.XWV/, and let Wbe an arbitrary generalized inverse of W. Then C.WX/˚C.X/?DRn;C.WX/?˚C.X/DRn; CŒ.W/0X˚C.X/?DRn;CŒ.W/0X?˚C.X/DRn: 6 Statistical examples 6.1 Centering We would like to start with a simple but at the same time very important orthocomplement in statistics: the set of vectors orthogonal to the vector of ones, that is, C.1n/?, where 1nD.1; 1; : : : ; 1/02Rn. In what follows, we most of the time drop off the subscript from the vector 1n; from the context its dimension should be obvious. The orthogonal projector onto C.1n/is P1D1 n110WD Jand the orthogonal projector onto C.1n/?is In1 n110WD C; Cis the centering matrix. Consider the n2data matrix Upartitioned as UD.xWy/D0 B B B B @ x1y1 x2y2 : : :: : : xnyn 1 C C C C AD0 B B B B @ u0 .1/ u0 .2/ : : : u0 .n/ 1 C C C C A : Here u.i/ Dxi yi2R2represents the ith case or the ith observation in the observation space, and the vectors x;y2Rnrepresent the two variables in the variable space. Let N uDNx Ny2R2denote the mean vector of xand y-variables and Sthe sample covariance matrix: N uD1 nU01nD1 n.u.1/ Cu.2/ CCu.n//D Nx Ny!; SD1 n1U0CU D1 n1 n X iD1 .u.i/ N u/.u.i/ N u/0: Now the following theorem is easy to confirm; for details, see, e.g., Puntanen, Styan & Isotalo [39, Ch. 3]. Theorem 6.1. For conformable matrices, the following statements hold: (a) The vector N N yD Ny1is the orthogonal projection of the variable vector yonto the column space C.1/:N N yD Ny1D Jy DP1y. (b) The centered variable vector Q yis the orthogonal projection of yonto the column space C.1/?:Q yDyJy D Cy D.InP1/y: 48 A. Markiewicz, S. Puntanen Now the estimator Byis the BLUE for Xunder the model Fif and only if Bsatisfies the equation B.XWVX? /D.XW0/ : (34) Substituting (33) into (34) yields B11 B12 B21 B22! X Z RM 0 IqDZ0M!D X Z 0 0 Iq0!:(35) Using (35) Haslett & Puntanen [20, Th. 1] show that all properties of BLUEs and BLUPs in mixed model Lcan be considered using the augmented model F, where both ˇand are fixed parameters. Using the connection between the mixed model Lthe augmented model F, the following result follows from Theorem 6.8 immediately. Theorem 6.12. Consider two mixed models: LiD fy;XˇCZ;Di;Rig;and denote ˙iDZDiZ0CRiand ViDRi0 0 Di; i D1; 2: Then every representation of the BLUE for Xˇunder L1remains the BLUE for Xˇ under L2and every representation of the BLUP for under L1remains the BLUP for under L2if and only if any of the following equivalent conditions holds: (a) Every representation of the BLUE for Xunder F1remains the BLUE for Xunder F2. (b) C.V2X? /C.V1X? /. (c) C R2M D2Z0M!C R1M D1Z0M!: (d) C ˙2M D2Z0M!C ˙1M D1Z0M!: (e) The matrix V2can be expressed as V2DaV1CXN1X0 CV1X? N2.X? /0V1 for some a2Rand matrices N1and N2such that V2is nonnegative definite. Acknowledgement: Thanks go to the referees for helpful remarks. Part of this research was done during the meeting of a Research Group on Mixed and Multivariate Models in the Mathematical Research and Conference Center, Be¸dlewo, Poland, October 2013, supported by the Stefan Banach International Mathematical Center. References [1] Baksalary J.K., An elementary development of the equation characterizing best linear unbiased estimators, Linear Algebra Appl., 2004, 388, 3–6 [2] Baksalary J.K., Mathew T., Linear sufficiency and completeness in an incorrectly specified general Gauss–Markov model, Sankhy¯ a A, 1986, 48, 169–180 [3] Baksalary J.K., Mathew T., Rank invariance criterion and its application to the unified theory of least squares, Linear Algebra Appl., 1990, 127, 393–401 [4] Baksalary J.K., Puntanen S., Styan G.P.H., A property of the dispersion matrix of the best linear unbiased estimator in the general Gauss–Markov model, Sankhy¯ a A, 1990, 52, 279–296 [5] Baksalary J.K., Rao C.R., Markiewicz A., A study of the influence of the “natural restrictions” on estimation problems in the singular Gauss–Markov model, J. Statist. Plann. Inference, 1992, 31, 335–351 [6] Baksalary O.M., Trenkler G., A projector oriented approach to the best linear unbiased estimator, Statist. Papers, 2009, 50, 721–733 [7] Baksalary O.M., Trenkler G., Between OLSE and BLUE, Aust. N. Z. J. Stat., 2011, 53, 289–303 [8] Baksalary O.M., Trenkler G., Rank formulae from the perspective of orthogonal projectors, Linear Multilinear Algebra, 2011, 59, 607–625 [9] Baksalary O.M., Trenkler G., Liski E.P., Let us do the twist again. Statist. Papers, 2013, 54, 1109–1119 [10] Ben-Israel A., Greville T.N.E., Generalized inverses: theory and applications, 2nd Ed., Springer, New York, 2003 [11] Ben-Israel A., The Moore of the Moore–Penrose inverse, Electron. J. Linear Algebra, 9, 150–157, 2002 [12] Bhimasankaram P., Sengupta D., The linear zero functions approach to linear models, Sankhy¯ a B, 1996, 58, 338–351 All about the ?with its applications in the linear statistical models 49 [13] Christensen R., Plane answers to complex questions: the theory of linear models, 4th Ed. Springer, New York, 2011 [14] Davidson R., MacKinnon J.G., Econometric theory and methods, Oxford University Press, New York, 2004 [15] Frisch R., Waugh F.V., Partial time regressions as compared with individual trends, Econometrica, 1933, 1, 387–401 [16] Groß J., The general Gauss–Markov model with possibly singular dispersion matrix, Statist. Papers, 2004, 45, 311–336 [17] Groß J., Puntanen S., Estimation under a general partitioned linear model, Linear Algebra Appl., 2000, 321, 131–144 [18] Groß J., Puntanen S., Extensions of the Frisch–Waugh–Lovell Theorem, Discuss. Math. Probab. Stat., 2005, 25, 39–49 [19] Harville D.A., Matrix algebra from a statistician’s perspective, Springer, New York, 1997 [20] Haslett S.J., Puntanen S., Equality of BLUEs or BLUPs under two linear models using stochastic restrictions, Statist. Papers, 2010, 51, 465–475 [21] Hauke J., Markiewicz A., Puntanen S., Comparing the BLUEs under two linear models, Comm. Statist. Theory Methods, 2012, 41, 2405–2418 [22] Herr D.G., On the history of the use of geometry in the general linear model, Amer. Statist., 1980, 34, 43–47 [23] Isotalo J., Puntanen S., Linear prediction sufficiency for new observations in the general Gauss–Markov model, Comm. Statist. Theory Methods, 2006, 35, 1011–1023 [24] Isotalo J., Puntanen S., Styan G.P.H., A useful matrix decomposition and its statistical applications in linear regression, Comm. Statist. Theory Methods, 2008, 37, 1436–1457 [25] Kala R., Projectors and linear estimation in general linear models, Comm. Statist. Theory Methods, 1981, 10, 849–873 [26] Khatri C.G., A note on a MANOVA model applied to problems in growth curves, Ann. Inst. Statist. Math., 1966, 18, 75–86 [27] Kruskal W., When are Gauss–Markov and least squares estimators identical? A coordinate-free approach, Ann. Math. Statist., 1968, 39, 70–75 [28] LaMotte L.R., A direct derivation of the REML likelihood function, Statist. Papers, 2007, 48, 321–327 [29] Lovell M.C., Seasonal adjustment of economic time series and multiple regression analysis, J. Amer. Statist. Assoc., 1963, 58, 993–1010 [30] Lovell M.C., A simple proof of the FWL Theorem, J. Econ. Educ., 2008, 39, 88–91 [31] Margolis M.S., Perpendicular projections and elementary statistics, Amer. Statist., 1979, 33, 131–135 [32] Markiewicz A., On dependence structures preserving optimality, Statist. Probab. Lett., 2001, 53, 415–419 [33] Markiewicz A., Puntanen S., Styan G.P.H., A note on the interpretation of the equality of OLSE and BLUE, Pakistan J. Statist., 2010, 26, 127–134 [34] Marsaglia G., Styan G.P.H., Equalities and inequalities for ranks of matrices, Linear Multilinear Algebra, 1974, 2, 269–292 [35] Mitra S.K., Moore B.J., Gauss–Markov estimation with an incorrect dispersion matrix, Sankhy¯ a A, 1973, 35, 139–152 [36] Mitra S.K., Rao C.R., Projections under seminorms and generalized Moore–Penrose inverses, Linear Algebra Appl., 1974, 9, 155–167 [37] Puntanen S., Styan G.P.H., The equality of the ordinary least squares estimator and the best linear unbiased estimator (with discussion), Amer. Statist., 1989, 43, 151–161 [Commented by O. Kempthorne on pp. 161–162 and by S.R. Searle on pp. 162–163, Reply by the authors on p. 164] [38] Puntanen S., Styan G.P.H., Reply [to R. Christensen (1990), R.W. Farebrother (1990), and D.A. Harville (1990)] (Letter to the Editor), Amer. Statist., 1990, 44, 192–193 [39] Puntanen S., Styan G.P.H., Isotalo J., Matrix tricks for linear statistical models: our personal top twenty, Springer, Heidelberg, 2011 [40] Rao C.R., Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability: Berkeley, California, 1965/1966, vol. 1, L.M. Le Cam and J. Neyman, eds., University of California Press, Berkeley, 355–372, 1967 [41] Rao C.R., A note on a previous lemma in the theory of least squares and some further results, Sankhy¯ a A, 1968, 30, 259–266 [42] Rao C.R., Unified theory of linear estimation, Sankhy¯ a A 1971, 33, 371–394. [Corrigendum (1972), 34, p. 194 and p. 477] [43] Rao C.R., Linear statistical inference and its applications, 2nd Ed., Wiley, New York, 1973 [44] Rao C.R., Representations of best linear unbiased estimators in the Gauss–Markoff model with a singular dispersion matrix, J. Multivariate Anal., 1973, 3, 276–292 [45] Rao C.R., Projectors, generalized inverses and the BLUE’s, J. R. Stat. Soc. Ser. B Stat. Methodol., 1974, 336, 442–448 [46] Rao C.R., Mitra S.K., Generalized nverse of matrices and its applications, Wiley, New York, 1971 [47] Rao C.R., Rao M.B., Matrix algebra and its applications to statistics and econometrics, World Scientific, River Edge, NJ, 1998 [48] Searle S.R., Casella G., McCulloch C.E., Variance components, Wiley, New York, 1992 [49] Seber G.A.F., The linear hypothesis: a general theory, 2nd Ed., Griffin, London, 1980 [50] Seber G.A.F., Lee A.J., Linear regression analysis, 2nd Ed. Wiley, New York, 2003 [51] Sengupta D., Jammalamadaka S.R., Linear models: an integrated approach, World Scientific, River Edge, NJ., 2003 [52] Tian Y., On equalities for BLUEs under misspecified Gauss–Markov models, Acta Math. Sin. (Engl. Ser.), 2009, 25, 1907–1920 [53] Tian Y., Beisiegel, M., Dagenais E., Haines C., On the natural restrictions in the singular Gauss–Markov model, Statist. Papers, 2008, 49, 553–564 [54] Tian Y., Takane Y., Some properties of projectors associated with the WLSE under a general linear model. J. Multivariate Anal., 2008, 99, 1070–1082 [55] Tian Y., Takane Y., On V-orthogonal projectors associated with a semi-norm, Ann. Inst. Statist. Math., 2009, 61, 517–530 50 A. Markiewicz, S. Puntanen [56] Trenkler G., On the singularity of the sample covariance matrix, J. Stat. Comput. Simul., 1995, 52, 172–173 [57] Watson G.S., Serial correlation in regression analysis, I, Biometrika, 1955, 42, 327–341 [58] Zyskind G., On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models, Ann. Math. Statist., 1967, 38, 1092–1109 [59] Zyskind G., Martin F.B., On best linear estimation and general Gauss–Markov theorem in linear models with arbitrary nonnegative covariance structure, SIAM J. Appl. Math., 1969, 17, 1190–1202