Polarization measurement and inference in many dimensions when subgroups can not be identified

Anderson, Gordon Article Polarization measurement and inference in many dimensions when subgroups can not be identified Economics: The Open-Access, Open-Assessment E-Journal Provided in Cooperation with: Kiel Institute for the World Economy – Leibniz Center for Research on Global Economic Challenges Suggested Citation: Anderson, Gordon (2011) : Polarization measurement and inference in many dimensions when subgroups can not be identified, Economics: The Open-Access, Open-Assessment E-Journal, ISSN 1864-6042, Kiel Institute for the World Economy (IfW), Kiel, Vol. 5, Iss. 2011-11, pp. 1-19, https://doi.org/10.5018/economics-ejournal.ja.2011-11 This Version is available at: https://hdl.handle.net/10419/49713 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by-nc/2.0/de/deed.en Vol. 5, 2011-11 | August 29, 2011 | http://dx.doi.org/10.5018/economics-ejournal.ja.2011-11 Polarization Measurement and Inference in Many Dimensions When Subgroups Can Not Be Identified Gordon Anderson University of Toronto Abstract The most popular general univariate polarization indexes for discrete and continuous variables are extended and combined to describe the extent of polarization between agents in a distribution defined over a collection of many discrete and continuous agent characteristics. A formula for the asymptotic variance of the index is also provided. The implementation of the index is illustrated with an application to Chinese urban household data drawn from six provinces in the years 1987 and 2001 (years spanning the growth and urbanization period subsequent to the economic reforms). The data relates to household adult equivalent log income, adult equivalent living space, which are both continuous variables and the education of the head of household which is a discrete variable. For this data set combining the characteristics changes the view of polarization that would be inferred from considering the indices individually. Special Issue The Measurement of Inequality and Well-Being: New Perspectives JEL C14, C30, I32 Keywords Multivariate polarization measurement Correspondence Gordon Anderson, Department of Economics, University of Toronto, Max Gluskin House, 150 St George St., Toronto, Ontario M5S 3G7, Canada, e-mail: [email protected] Citation Gordon Anderson (2011). Polarization Measurement and Inference in Many Dimensions When Subgroups Can Not Be Identified. Economics: The Open-Access, Open-Assessment E-Journal, Vol. 5, 2011-11. doi:10.5018/economics-ejournal.ja.2011-11. http://dx.doi.org/10.5018/economics-ejournal.ja.2011-11 © Author(s) 2011. Licensed under a Creative Commons License - Attribution-NonCommercial 2.0 Germany 1 Introduction The functionings and capabilities approach to wellbeing measurement (Sen 1992) has given considerable impetus to multidimensional analyses of wellbeing (Grusky and Kanbur 2006). The argument is that individual wellbeing is not just a matter of the incomes they have or could achieve, among other things it depends on individual health and educational status, their political freedoms and environmental factors. In the absence of a well specified wellbeing aggregator of these many sensibilities (i.e. some form of utility function) evaluation of wellbeing has to be evaluated over these many dimensions which of course could be measured discretely or continuously. The multivariate polarization measure presented here is founded upon the notion of polarization within a population distribution f(x) of individual characteristics x into potentially many possible groups which are not identified1 a priori. Imagine for example a population which is a mixture of K classes with respective distributions fk(x) and proportions wk so that the population distribution f(x) may be written as: 1 () () K kk k f xwf  x _________________________ When no class identifier or information on agent membership of the fk(x)’s (i.e. the sub group distributions) is available, all that is observed is f(x), the population distribution. This is the unidentified case for which the polarization measures discussed herein are appropriate. Sometimes, given additional information, the sub distributions can be estimated facilitating calculation of the probability (or partial identification) of group membership for an individual with characteristics x or indeed perfect stratification where group membership is known qith probability 1. For example Anderson, Pittau and Zelli (2011) posit that the incomes of each class are driven by distinct stochastic processes which precipitate distinct log normal distribution specifications for the fk(x)’s permitting, through the employment of semi parametric techniques, estimation of all of the parameters of the mixture distribution. This permits estimation of the extent of polarization 1 Here “identification” refers to known membership of a group rather than a sense of kinship or proximity to other members in a group which is the sense in which it will be used later in defining the polarization measure. www.economics-ejournal.org 1 between any two classes in a similar fashion to when class membership is completely identified. Here no such information is available. Esteban and Ray (1994) and Duclos, Esteban, and Ray (2004) posited a collection of propositions with which such a Polarization measure for the unidentified case should be consistent and proposed a collection of univariate measures appropriate for a variety of circumstances that would reflect such polarization between potentially many groups. The propositions are based upon a so-called Cohesion (or Identification) and Alienation nexus wherein notions of polarization are fostered jointly by an agent’s sense of increasing within-group identity or association and between-group distance or alienation. There have been several proposed univariate polarization indices which focus on an arbitrary number of groups2 in this unidentified case (Esteban and Ray, 1994; Esteban, Gradin and Ray, 1998; Zhang and Kanbur, 2001; Duclos, Esteban and Ray 2004) and a similar number that focus on just two identified groups i.e. when the sub distributions above are observed (Alesina and Spolaore 1997; Foster and Wolfson 1992; Wolfson 1994; Wang and Tsui, 2000). Anderson, 2004 considers tests for various types of polarization between two identified or partially identified groups based upon the anatomy of their respective distributions or the mixture of their respective distributions when the groups were only partially identified. While much work has been done on extending one dimensional wellbeing measures to many dimensions in the context of poverty (Duclos, Sahn and Younger 2006) and inequality measurement (Maassoumi 1986, 1999, Koshevoy and Mosler 1997, Tsui 1995 and Anderson 2008)3 little has been done in extending polarization measures to the many dimensioned case. While Gigliarano and Mosler, (2009) develop a family of multivariate polarization measures based upon measures of between and within group multivariate variation and relative group size which exploit notions of subgroup decomposability and Anderson (2010) and Anderson, Linton and Leo (2011) have developed a trapezoidal measure of polarization which can be applied to two identifiable groups or within a population distribution provided at least two modal points are identified (i.e. the partially identified case), multivariate polarization measures have not been developed for the more general non-identified many group case, nor _________________________ 2 And a fortiori two groups. 3 All however confine themselves to continuous variables. www.economics-ejournal.org 2 for the case where the joint distribution of sensibility indicators is a mixture of discrete and continuous variables.4 An excellent summary of the properties of the univariate indices is to be found in (Esteban and Ray, 2007) wherein the properties of indices are evaluated in terms of their coherence with some basic axioms that reflect three broad notions, 1) When there is only one group there is little polarization, 2) polarization increases when within group inequality is reduced, 3) polarization increases when between group inequality increases. The axioms are formed around a notional univariate density that is a mixture of kernels f(x, a) that are symmetric uni-modal on a compact support of [a,a+2] with E(x) = μ = (a+1) also representing the mean or mode. However these axioms are readily extended to multivariate densities of continuous variables by thinking in terms of a notional multivariate density that is a mixture of multivariate kernels so that x is simply a j dimensioned vector. The kernels are subject to slides (location shifts) g(y) = f(y-x), which may be contemplated in terms of the Euclidean distance5 between vectors y and x, and squeezes (shrinkages) of the form fλ(x) =f({x-[1-λ]μ}/λ)/λ (0 < λ <1) where now μ is a j dimensioned vector of means or modal values of the multidimensional kernel f(x,a) that is symmetric on a compact support of [a,a+2] where a is a j dimensioned vector. Potential indices are evaluated in the context of such changes in terms of the extent to which they satisfy a set of axioms which reflect the following set of ideas. The squeeze of a uni-modal distribution cannot increase polarization and symmetric squeezes of the two kernels cannot reduce polarization. Sliding two kernels away from one another increases polarization and common population scaling preserves the polarization ordering. Polarization indices have to come from a family where if x and y are independently distributed with marginal distributions f(x) and f(y) then the index is the expected value of some function T(f(x),|x-y|) which is increasing in its second argument. Symmetric squeezes of the sub distributions weakly increases polarization. The index should be non-monotonic with respect to outward slides of the sub distributions and flipping the distribution around its support should leave polarization unchanged. Most of these ideas can be _________________________ 4 Furthermore extensions of the stochastic dominance techniques introduced in Anderson (2004), which really explore the anatomy of polarizing distributions, would prove cumbersome in many dimensions because it is not obvious how to define a sensible partition of the distribution across those many dimensions. 5 Other distance metrics (for example Mahalonobis 1936 or Bregman 1967) could be employed. www.economics-ejournal.org 3 contemplated with respect to multivariate densities of continuous variables though there is some difficulty when multivariate densities of discrete variables are contemplated unless slides of the discrete outcome values are permitted and squeezes of the distributions contemplated in terms of transfers of mass between outcome values. Here the most popular general univariate polarization indices for discrete (Esteban and Ray 1994), and continuous (Duclos, Esteban and Ray 2004) variables are combined and extended to describe the extent of polarization between agents in a distribution defined over a collection of many discrete and continuous agent characteristics. The univariate indices have been demonstrated to satisfy the aforementioned axioms. The implementation of the index is illustrated with an application to Chinese urban household data drawn from six provinces in the years 1987 and 2001 (years spanning the growth and urbanization period subsequent to the economic reforms). The data relates to household adult equivalent log income, adult equivalent living space, which are both continuous variables and the education of the head of household which is a discrete variable. 2 The Extension to Many Variables both Discrete and Continuous The multivariate generalization of the Duclos, Esteban, and Ray (2004) (DER) Polarization index is, like DER, based upon the sample equivalents of the population concepts. For scalar continuous x with distribution function F(x) the DER index is given by: () | | () () [1]PfxyxdFydFx      Some intuition for the index may be gained by thinking in terms of f(x)α as the degree of identification or cohesion experienced by an agent with income x (f(x) being and indicator of mass around x) and a(x) as the degree of alienation experienced by a person with income x where: () | | ()ax y x dFy  www.economics-ejournal.org 4 pa(x) = f(x)αa(x), the area of a rectangle with height f(x)α and base a(x), is then the degree of polarization experienced by an agent with x and [1] corresponds to the average polarization experienced across the population of agents. DER, (in Duclos, Esteban and Ray 2004a) demonstrate the estimator of [1] to be asymptotically normally distributed with an asymptotic variance V given by: () 00 var (1 ) ( ) || || ( ) ( ) ( ) 2 || || ( ) ( ) [2] fy y V fy y xdFx y fx dFx y x fx dFx             Their development of the variance formula is sketched in the Appendix. A similar discrete variable index is provided in Esteban and Ray (1994) and is given by: 1 11 || nn iji ij PK xx   j       where πi is the sample weight of the i’th observation and K is a normalizing factor. Development of the polarization index was founded on a set of axioms that such an index should obey, the axioms concern changes (squeezes and slides) in the uni-modal sub distributions in the mixture distribution that is f(x). The resultant index reflects the two primary factors that underlay polarization, the alienation or distance between groups (given by |y-x|) and the association within a group (given by f(x)α). Indeed the intuitive interpretation of Pα as the average value of the areas of all possible trapezoids that can be formed under f(x) whose average height is f(x)α and whose base is |x-y| can be related to the trapezoidal index of polarization employed in Anderson (2010) to study multivariate poverty states and in Anderson, Leo and Linton (2011) to study multivariate convergence issues. Here α is a polarization sensitivity parameter6 chosen by the investigator such that 0.25 ≤ α ≤ 1 with higher values of α corresponding to increased sensitivity. The same axioms can be applied when x is a vector and where ||x-y|| is the Euclidean distance between the vectors.7 _________________________ 6 Note when α = 0 the index is in essence twice the Gini coefficient thus a similar value in the following would provide a multivariate version of a Gini like coefficient and its variance. 7 Anderson, Crawford and Leicester (2011) employ Euclidian distance in developing a nonparametric approach to multivariate welfare rankings. www.economics-ejournal.org 5 Let wi and zi be jointly distributed vectors describing the status of the i’th agent with wi being a k x 1 vector of continuous variables and zi being an h x 1 vector of continuous variables with i =1,..,n being the elements of the sample. The continuous variables all reflect wellbeing positively and for convenience are defined on Rk + and the discrete variables are ordered integers reflecting positive wellbeing in the same fashion.8 The joint density of the w’s for a given configuration of z’s is fz(w|z) and the joint probability of the z’s is p(z) so that the joint density of the w’s and z’s for the i’th agent with continuous characteristics wi and discrete characteristics zi is given by f(wi,zi) = fi(wi |zi )p(zi) which corresponds to her degree of identification. As for the alienation component let xi be the stacked vector wi | zi then the dimension normalized Euclidean distance9 between agents i and j given by ||xi-xj|| is well defined and may be written as: 2 1 () || || Q iq jq q ij xx xx Q    where xiq is the q’th element of the vector xi where Q = k+h. For notational convenience denote the first k continuous components of the vector x as x{c}. Then, retaining the trapezoidal intuition, a multivariate version of [1] is given by: ((|)())|| || ( ) ( )[1] kkw wy x zy RR zx zy PfwzpzyxdFwdF        x wa _________________________ Here summation is over the domain of each element of the z vector and integration is over the domain of each element of the w vector. As in the univariate case the alienation or distance between groups is given by ||y-x|| and the association within 8 For example the continuously measured variables may represent levels of consumption, leisure and housing stock whereas the discretely measured variables may reflect levels of educational, health or freedom status. 9 In DER the columns of X are mean standardized and assumed to reside in the positive orthant which has been followed here, however other distance measures could be equally well employed, for example Mahalonobis distance (Mahalonobis 1936) or Bregman distance (Bregman 1967) would probably better accommodate the variations over the different dimensions (I’m grateful to an anonymous referee for pointing this out). www.economics-ejournal.org 6 a group given by f(w,z)α in exactly the same fashion.10 By employing kernel estimates of the conditional multivariate distributions and sample estimates of the population proportions p(z) the sample equivalents, given n observations on Q variables in an n x Q matrix X with typical element xiq i = 1,.., n, q = 1,..,Q and typical row xi the index can be seen to be:    2 111 2 (( | )()) ( ) Q nn ii i iq jq ijq fw z pz x x PnQ       The multivariate version of [2], the variance of index is given by: 0 () 0 (1 )( ( | ) ( )) || || ( | ) ( ) var |||| ((|)()) (|)() [2] 2|| ||((|)())(|)() z fy z zy fwzPz y xdFwzPz VyfwzPzdFwzPz y x fw zPz dFw zPz                        a  _________________________ Where after ordering the vectors xi on ||xi|| as xi o, the first, second and third terms of the i’th element of the variance vector may be respectively estimated in an obvious fashion as: 10 Note that Esteban and Ray (1994) and DER respectively offer different ranges for α for discrete univariate and continuous univariate distributions this can be accommodated in the present context by considering the association component as f(w|z)αcp(z)αd where αc is the polarization parameter for the continuous components and αd is the polarization parameter associated with the discrete components. www.economics-ejournal.org 7 The results, while obviously specific to these particular data, were salutary with regard to the use of univariate as opposed to multivariate polarization indices. While the individual univariate indices all reflected significant increases in polarization between households over the period of the reforms, when they were combined the polarization result was attenuated. For pair-wise combinations of the variables significant polarization was detected at low levels of polarization sensitivity but at high levels of polarization sensitivity significant depolarization was detected. When all three variables were combined in an index, significant depolarization was detected at all levels of polarization sensitivity. Acknowledgments Many thanks Oliver Linton for help with multivariate Kernel Estimation and to Jean-Yves Duclos, Joan Esteban and two anonymous referees for helpful comments on earlier drafts of this paper. www.economics-ejournal.org 14 References Alesina, A., and E. Spolaore (1997). On the Number and Size of Nations. Quarterly Journal of Economics 112: 1027–1056. http://ideas.repec.org/a/tpr/qjecon/v112y1997i4p1027-56.html Anderson, G.J. (2004). Toward an Empirical Analysis of Polarization. Journal of Econometrics 122: 1–26. http://ideas.repec.org/a/eee/econom/v122y2004i1p1-26.html Anderson, G.J. (2008). The Empirical Assessment of Multidimensional Welfare, Inequality and Poverty: Sample Weighted Generalizations of the Kolmorogorov-Smirnov Two Sample Test for Stochastic Dominance. Journal of Economic Inequality 6: 73–87. Anderson, G.J. (2010). Polarization of the Poor: Multivariate Relative Poverty Measurement Sans Frontiers. Review of Income and Wealth 56: 84–101. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1551556 Anderson, G.J., and Y. Ge (2004). Do Economic Reforms Accelerate Urban Growth: The Case of China. Urban Studies 11: 2197–2210. http://usj.sagepub.com/content/41/11/2197.abstract Anderson, G.J., and Y. Ge (2005). The Size Distribution of Chinese Cities. Regional Science and Urban Economics 35: 756–776. http://ideas.repec.org/a/eee/regeco/v35y2005i6p756-776.html Anderson, G.J., I. Crawford, and A. Leicester (2011). Welfare Rankings from Multivariate Data, a Nonparametric Approach. Journal of Public Economics 95: 247–252. http://ideas.repec.org/a/eee/pubeco/v95y2011i3-4p247-252.html Anderson, G.J., and T.W. Leo (2007). Family Formation and the One Child Policy in Urban China. Mimeo, Department of Economics, University of Toronto. Anderson, G.J., O. Linton, and T.W. Leo (2011). Rich Country-Poor Country Convergence from a Polarization-Cohesion Perspective. Mimeo, Department of Economics, University of Toronto. Anderson, G.J., G. Pittau, and R. Zelli (2011). Partially Identified Poverty Status: A New Approach to Measuring Poverty and the Progress of the Poor. Working Paper tecipa-421, Department of Economics, University of Toronto. http://ideas.repec.org/p/tor/tecipa/tecipa-421.html www.economics-ejournal.org 15 Brady D.S., and H.A. Barber (1948). The Pattern of Food Expenditures. Review of Economics and Statistics 30: 198–206. http://www.jstor.org/stable/1926749 Bregman, L.M. (1967). The Relaxation Method of Finding the Common Points of Convex Sets and its Application to the Solution of Problems in Convex Programming. USSR Computational Mathematics and Mathematical Physics 7: 200–217. Duclos, J.-Y., J. Esteban, and D. Ray (2004). Polarization: Concepts, Measurement, Estimation. Econometrica 72: 1737–1772. http://ideas.repec.org/a/ecm/emetrp/v72y2004i6p1737-1772.html Duclos, J.-Y., J. Esteban, and D. Ray (2004a). Polarization: Concepts, Measurement, Estimation. Bureau for Research in Economic Analysis of Development, Working Paper No. 059, April. Duclos, J.-Y., D. Sahn, and S.D. Younger (2006). Robust Multidimensional Poverty Comparisons. The Economic Journal 116: 943–968. http://ideas.repec.org/a/ecj/econjl/v116y2006i514p943-968.html Esteban, J.-M., and D. Ray. (1994). On the Measurement of Polarization. Econometrica 62: 819–851. http://ideas.repec.org/a/ecm/emetrp/v62y1994i4p819-51.html Esteban, J., and D. Ray (2007). A Comparison of Polarization Measures. UFAE and IAE Working Papers 700.07, Unitat de Fonaments de l’Anàlisi Econòmica (UAB) and Institut d’Anàlisi Econòmica (CSIC). http://ideas.repec.org/p/aub/autbar/700.07.html Esteban, J., C. Gradin, and D. Ray (1998). Extensions of a Measure of Polarization, with an Application to the Income Distribution of Five OECD Countries. Mimeo, Instituto de Analisis Económico. Foster, J., and M. Wolfson (1992). Polarization and the Decline of the Middle Class in Canada and the USA. Mimeo, Vanderbilt University. Gigliarano, C., and K. Mosler (2009). Constructing Indices of Multivariate Polarization. Journal of Economic Inequality 7: 435–460. http://ideas.repec.org/a/kap/jecinq/v7y2009i4p435-460.html Grusky, D., and R. Kanbur (eds.) (2006). Poverty and Inequality. Stanford, CA: Stanford University Press. www.economics-ejournal.org 16 Koshevoy, G.A., and K. Mosler (1997). Multivariate Gini Indices. Journal of Multivariate Analysis 60: 252–276. http://ideas.repec.org/a/eee/jmvana/v60y1997i2p252-276.html Maasoumi, E. (1986). The Measurement and Decomposition of Multidimensional Inequality. Econometrica 54: 771–779. http://ideas.repec.org/a/ecm/emetrp/v54y1986i4p991-97.html Maasoumi, E. (1999). Multidimensioned Approaches to Welfare Analysis, chapter 15. In J. Silber (ed.), Handbook of Income Inequality Measurement. Norwell, MA: Kluwer Academic Publishers. Mahalanobis, P.C. (1936). On the Generalised Distance in Statistics. Proceedings of the National Institute of Sciences of India 2 (1): 49–55. Sen, A.K. (1992). Inequality Reexamined. Cambridge, MA: Harvard University Press. Silverman, B.W. (1986). Density Estimation for Statistics and Data Analysis. Chapman Hall. Tsui, K.-Y. (1995). Multidimensional Generalizations of the Relative and Absolute Inequality Indices: The Atkinson-Kolm-Sen Approach. Journal of Economic Theory 67: 251–265. http://www.sciencedirect.com/science/article/pii/S0022053185710733 Wang, Y. Q., and K. Y. Tsui (2000). Polarization Orderings and New Classes of Polarization Indices. Journal of Public Economic Theory 2: 349–363. http://onlinelibrary.wiley.com/doi/10.1111/1097-3923.00042/pdf Wolfson, M.C. (1994). When Inequalities Diverge. American Economic Review 84: 353–358. http://ideas.repec.org/a/aea/aecrev/v84y1994i2p353-58.html Zhang, X., and R. Kanbur (2001). What Difference Do Polarization Measures Make? An Application to China. Journal of Development Studies 37: 85–98. http://ideas.repec.org/a/taf/jdevst/v37y2001i3p85-98.html www.economics-ejournal.org 17 Appendix The derivation of the variance of the estimator follows DER (2004a) where P is rewritten in terms of a(y) and pa(y) where: () | | () () () () a a y y x dF x and p y f y a y     Noting that a(y) may be written in terms of the population mean μ and first partial moment μ* as: () (2 () 1) 2 *() () *() () y ay y Fy y where ydF y and y xdF x         DER(2004a) show that, for large n, P may be decomposed across its sources of sampling variability into:     () () (() ())() () ()(() ()) () [1] ()( )() PF PF fy fy aydFy f yayaydFy A pydFFy            Assuming that the second order moments of all the components of the right hand terms are finite and that h, the window width of the kernel estimator, goes to 0 as n gets large, each term on the right hand side can be shown to be o(n-0.5) and each respectively may be written as: www.economics-ejournal.org 18 www.economics-ejournal.org 19   1 1 1 (() ); 1() () 2 () () 2 () () () 2 ( 2 *()) (); 1() () ) ii n ai i n ii iyy n ii i py P n fydFyy y fydFy yfydFy n fy yFy ydFy fy ay P n                         [A2] Applying the law of large numbers to [A1} and noting that root n times [A1] has expectation 0. Invoking the central limit theorem and collecting and re-arranging the terms yields the variance formula [2]. Please note: You are most sincerely encouraged to participate in the open assessment of this article. You can do so by either recommending the article or by posting your comments. Please go to: http://dx.doi.org/10.5018/economics-ejournal.ja.2011-11 The Editor © Author(s) 2011. Licensed under a Creative Commons License - Attribution-NonCommercial 2.0 Germany