PanelWhiz: Efficient Data Extraction of Complex Panel Data Sets – An Example Using the German SOEP

Haisken-DeNew, John P.; Hahn, Markus H. Article PanelWhiz: Efficient Data Extraction of Complex Panel Data Sets – An Example Using the German SOEP Schmollers Jahrbuch – Journal of Applied Social Science Studies. Zeitschrift für Wirtschafts- und Sozialwissenschaften Provided in Cooperation with: Duncker & Humblot, Berlin Suggested Citation: Haisken-DeNew, John P.; Hahn, Markus H. (2010) : PanelWhiz: Efficient Data Extraction of Complex Panel Data Sets – An Example Using the German SOEP, Schmollers Jahrbuch – Journal of Applied Social Science Studies. Zeitschrift für Wirtschafts- und Sozialwissenschaften, ISSN 1865-5742, Duncker & Humblot, Berlin, Vol. 130, Iss. 4, pp. 643-654, https://doi.org/10.3790/schm.130.4.643 This Version is available at: https://hdl.handle.net/10419/292316 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ PanelWhiz: Efficient Data Extraction of Complex Panel Data Sets – An Example Using the German SOEP By John P. Haisken-DeNew and Markus H. Hahn 1. Introduction Applied social scientists have forever been faced with different data interfaces for different data sets. In most cases, an interface is not even available, forcing the researcher to address data files by name, and extract the information required by hand. However, the specific structure of panel data can be very complex and vary dramatically as described in Haisken-DeNew (2001). Some panel data sets provide many files per year (“wide format”), differing by their population, or level of aggregation etc., creating many obstacles for researchers. If one wants to put together variables across time (“long format”), this is typically much more difficult, but ultimately the format which is required for estimation. PanelWhiz is a collection of subroutines that allows researchers to use an intuitive “common” graphical interface for accessing many panel datasets directly within the statistical package Stata/SE 10 or better (http://www.stata. com), whereby the researcher does not select individual variables, but rather vectors of variables (items) with one mouse click. This allows for an efficient method of selecting information for a data set retrieval, especially if the panel data set contains many waves (years) of information. With one mouse-click, data can be automatically retrieved, with merging and matching done automatically. With the PanelWhiz system, the user can open data files by clicking on a browse page. The idea behind the tool is that because of the intrinsically longitudinal nature of the data, one is typically not interested in retrieving a variable in a single wave, but rather in retrieving the variable for several waves, i.e. an item. For all data sets, a variable renaming algorithm (where necessary) is used to ensure time consistent variable names (See Haisken-DeNew, 2001 for more information on this). Thus, if one opens a data file and one finds a variable of interest, one clicks on the variable and information for the entire item (vector of variables) is also collected and added to a PanelWhiz “project”. Straightforwardly, the object is to collect items and save them into the data “project”, Schmollers Jahrbuch 130 (2010), 643–654 Duncker & Humblot, Berlin Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 644 John P. Haisken-DeNew and Markus H. Hahn allowing an automatic data retrieval. The data are extracted in “long” format allowing easy further data cleaning or direct estimation using Stata’s panel “xt” commands. PanelWhiz, since appearing in 2006, now has several hundred registered users, using the common interface to access many different datasets, such as the German SOEP, British BHPS, the Australian HILDA, the German IAB Establishment Panel, the American CPS, etc. Recently, support has been extended to the American PSID. This paper describes using PanelWhiz for the German SOEP, providing specific examples for this panel data set. However, due to the generalized nature of PanelWhiz, the interface is almost completely identical for all other supported datasets and thus this paper can also be used as a general reference for other supported data sets. 2. Overview of PanelWhiz 2.1 Getting Started with PanelWhiz PanelWhiz is described in detail on the PanelWhiz Website at http://www. panelwhiz.eu and has been presented several times at international data conferences, the UK Stata Users Group and the German Stata Users Group. Due to the extensive size of PanelWhiz, the user downloads only a very small startup Stata Add-On program from the PanelWhiz Website. This small startup program then downloads automatically the component parts required for the full installation over the internet. All component parts are stored in compressed format, and as such, typically require only one-tenth of the usual download time and are automatically expanded locally on the user’s hard disk. PanelWhiz is installed as a collection of Stata Add-On programs and loads every time Stata is started. For example in the following Screen 4 Shot 1, one can select by mouse click the desired data set to be supported. 2.2 Some Details Using the German SOEP In the example (Screen Shot 2), the German SOEP has been selected. One can select an already existing project, or create a new one from scratch. In this example, we will examine an existing project zufr.soep. Because it has been already saved, PanelWhiz keeps a note of the last 10 saved projects and allows easily loading by simply clicking on the link indicating the project name. Here we have indeed opened the PanelWhiz project zufr.soep, and have a heads-up display indicating the contents of the project and the possible pro- Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 PanelWhiz 645 ject commands as seen in Screen Shot 3. The top area displays the possible project commands and the bottom area the contents of the project. Here the project already contains 4 items (vectors of variables). The item labels are clickable, linked to a keyword thesaurus. Screen Shot 1: Select Data Set Screen Shot 2: Open a Project Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 646 John P. Haisken-DeNew and Markus H. Hahn Screen Shot 3: Project Page The variables underlying the 4 items in the project are listed below (Screen Shot 4). Each variable listed under the item displays the associated wave/year. In the SOEP, the “a” wave is 1984, the “b” wave is 1985 and so on. The “z” wave is 2009. Thus for the item IK622, in 1984 the underlying variable is ap6801 and in 2009 it is zp15701. One can update the project using the automatic “Update” function on the project page. When a new data distribution becomes available, for each item, the newest variable is added automatically to the relevant item. The retrieval can be run again, having now the most recent information. 2.2.1 Items and Specials Assuming that one would like to add any items to the project, one can choose between two types of concepts: “items” or “specials”. Items are vectors of variables that have a standard time dimension associated with them, i.e. one variable for each year. SOEP examples of these files would be ap.dta,apgen.dta, ah.dta,ahgen.dta etc. “Specials” have a non-standard time dimension, i.e. they may have one observation per person and be time invariant, or may already be in long format, with person-year observations as the unit of analysis. 1 Here we will first examine the page associated with items. Schmollers Jahrbuch 130 (2010) 4 1To learn more about the long or wide data format, see http://www.stata.com/help. cgi?reshape. OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 PanelWhiz 647 Screen Shot 4: Start Page Clicking on a year like [a - 1984], will load a browse page allowing one to click on all variables/items associated with the year 1984. Alternatively, one can click on a special file, such as [bioimmig] where the data is already in long format. In contrast, the information from the special file [bioparen]is time invariant and contains only one entry per person. PanelWhiz knows how to extract and merge the information from all of these kinds of files (Screen Shot 5). For both items and specials, all associated items have been scanned and the contents of the item labels have been catalogued into a thesaurus of keywords. Thus, if one were interested in all items or specials regarding the topic of “occupation”, one would click on the [o] of the keyword index, to examine all keywords starting with the letter “o”. Technically speaking, PanelWhiz works because the item correspondence information (for each item, that vector of variables over all 26 years which are available currently for SOEP) is injected into each of the relevant variables as a Stata variable characteristic. PanelWhiz reads this information from a variable in one particular file/wave and automatically knows where to find the corresponding information in all other files/waves. 2.2.2 Item Browse Page To find an item available say for the year 2009, we click on the year [z- 2009]; and receive the following browse page (Screen Shot 6). The browse page contains all variables/items from all files in the year 2009. Alternatively, Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 Screen Shot 5: Items and Specials 648 John P. Haisken-DeNew and Markus H. Hahn Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 PanelWhiz 649 one can jump only to specific variables/items in a particular file. Further, the variables for the German SOEP are sorted using the same hierarchical categorisation scheme as in SOEPinfo. See Haisken-DeNew and Frick (2005) for more information on SOEPinfo. Screen Shot 6: Top Level Item Browse Page In this example, we select the clickable button [ALL FILES] from wave “z” and get the following browse page (Screen Shot 7).All variables are listed in the order they naturally exist in the respective physical data files. The example shows that for the SOEP variable erwtyp09, there is a PanelWhiz item IK2264 associated with it. By actually clicking on IK2264, one would select the entire item (potentially all underlying variables from wave “a” through “z”). One can also examine the changing nature of the item over time. The variable erwtyp09 contains value labels. Thus we can click on “i” to the left of the item name and label. There is a ready-made HTML page showing all labelled values for all variables of the entire item. This will be especially useful information for data cleaning requirements. Just because a variable has been coded one way in one year/wave, it does not mean it will remain so over all time. Screen Shot 8 illustrates this example. Jumping from wave 1984 to 1985, there have been some additional outcome values added. These changes are colour coded in grey. By clicking on “N” to the left of the variable label, one can view the item “notes”, giving an indication of the variable names of variables belonging to the item over all years (Screen Shot 9). Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11 650 John P. Haisken-DeNew and Markus H. Hahn Screen Shot 7: Look at Items Screen Shot 8: Start Page Screen Shot 9: Item Notes All words appearing in an item label have been added to a keyword thesaurus. Each keyword is linked to all items in the entire dataset containing the keyword (see Screen Shot 10). Schmollers Jahrbuch 130 (2010) 4 OPEN ACCESS | Licensed under | https://creativecommons.org/about/cclicenses/ DOI https://doi.org/10.3790/schm.130.4.643 | Generated on 2023-01-16 13:36:11