Coding of free-text answers on the most important political problem in Germany: Methods of coding and documentation of results

Naujoks, Tabea; Nestler, Wiebke; Brümmer, Martin Research Report Coding of free-text answers on the most important political problem in Germany: Methods of coding and documentation of results SOEP Survey Papers, No. 476 Provided in Cooperation with: German Institute for Economic Research (DIW Berlin) Suggested Citation: Naujoks, Tabea; Nestler, Wiebke; Brümmer, Martin (2017) : Coding of freetext answers on the most important political problem in Germany: Methods of coding and documentation of results, SOEP Survey Papers, No. 476, Deutsches Institut für Wirtschaftsforschung (DIW), Berlin This Version is available at: https://hdl.handle.net/10419/172791 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by-sa/4.0/ SOEP Survey Papers Series D – Variable Descriptions and Coding The German Socio-Economic Panel study Coding of Free-Text Answers on the Most Important Political Problem in Germany – Methods of Coding and Documentation of Results 476 SOEP — The German Socio-Economic Panel study at DIW Berlin 2017 Tabea Naujoks, Wiebke Nestler, Martin Brümmer Running since 1984, the German Socio-Economic Panel study (SOEP) is a wide-ranging representative longitudinal study of private households, located at the German Institute for Economic Research, DIW Berlin. The aim of the SOEP Survey Papers Series is to thoroughly document the survey’s data collection and data processing. The SOEP Survey Papers is comprised of the following series: Series A – Survey Instruments (Erhebungsinstrumente) Series B – Survey Reports (Methodenberichte) Series C – Data Documentation (Datendokumentationen) Series D – Variable Descriptions and Coding Series E – SOEPmonitors Series F – SOEP Newsletters Series G – General Issues and Teaching Materials The SOEP Survey Papers are available at http://www.diw.de/soepsurveypapers Editors: Dr. Jan Goebel, DIW Berlin Prof. Dr. Martin Kroh, DIW Berlin and Humboldt Universität Berlin Prof. Dr. Carsten Schröder, DIW Berlin and Freie Universität Berlin Prof. Dr. Jürgen Schupp, DIW Berlin and Freie Universität Berlin Please cite this paper as follows: Tabea Naujoks, Wiebke Nestler, Martin Brümmer. 2017. Coding of Free-Text Answers on the Most Important Political Problem in Germany - Methods of Coding and Documentation of Results. SOEP Survey Papers 476: Series D. Berlin: DIW/SOEP This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. © 2017 by SOEP ISSN: 2193-5580 (online) DIW Berlin German Socio-Economic Panel (SOEP) Mohrenstr. 58 10117 Berlin Germany [email protected] Coding of Free-Text Answers on the Most Important Political Problem in Germany – Methods of Coding and Documentation of Results Tabea Naujoks, Wiebke Nestler, Martin Brümmer Berlin, 2017 0 Introduction What is the most important political problem? In both 2015 and 2017, survey respondents (more than 1,000 individuals answered this open-ended question. This paper documents the subsequent coding process of the resulting free-text answers. This documentation might be helpful for coding the answers in similar surveys in the future. In the first part, the data used will be described. In the second part, the coding scheme as well as its development will be presented. The coding process as well as the coding tool used will form the subject of the third section. Results of the coding in the form of descriptive frequencies will be presented in the fourth part of the paper. In addition, various possibilities of evaluation are discussed and compared. In the concluding discussion section, we will once again consider the specific features of the coding process which should be taken into account for further usage of the resulting data set. 1 Data 1.1 Data collection In January 2017, a representative telephone survey on the relevance of selected political objectives in 1,016 eligible voters was carried out by Kantar Public.1 The main objective of the survey was to assess the importance of different policy objectives. In addition to a list of predefined policy objectives, respondents were also asked what they considered to be ‘the most important political problem’ at the moment. The interviewees were able to name keywords on the telephone, which were recorded by the interviewers. In addition, a small part of the respondents of the German Socio-Economic Panel Study (SOEP) who participated in a citizens’ dialogue with the chancellor (or belonged to the control group) answered the same question in 2015, both before as well as after the dialogue. All in all, 289 answers from 216 unique respondents are available.2 The data set discussed here pools all text answers from 2015 and 2017, resulting in a sample of 1,305 responses. 1.2 Data characteristics At the first survey in 2015, an average of 21.1 characters (range: 4-97) and 2.3 words (range: 1-17) were recorded; at the follow-up 2015 an average of 15.7 characters (range: 4-55) and 1.5 words (range: 1-9). In the 2017 survey, an average of 27.1 characters (range: 3-214) and 3.9 words (range: 1-38) were used.3 These differences in the number of recorded characters and words likely reflect differences in the willingness of the telephone interviewers 2015 and 2017 to record more or less words. * These two authors contributed equally to the work. 1 Giesselmann, M., Brümmer, M., Kroh, M., Siegel, N. A., & Wagner, G. G. (2017). Fluchtzuwanderung ganz oben auf der Liste der dringenden politischen Prioritäten. Wirtschaftsdienst, 97(3), 192-200. 2 Wagner, G. G., Bruemmer, M., Glemser, A., Rohrer, J. M., & Schupp, J. (2017). Dimensions of Quality of Life in Germany: Measured by Plain Text Responses in a Representative Survey (SOEP). SOEPpaper No. 893, Berlin. 3 Giesselmann, M., Brümmer, M., Kroh, M., Siegel, N. A., & Wagner, G. G. (2017). Fluchtzuwanderung ganz oben auf der Liste der dringenden politischen Prioritäten. Wirtschaftsdienst, 97(3), 192-200. SOEP Survey Paper 476 1 2 Development of the coding scheme The coding scheme was developed prior to the coding process by Tabea Naujoks and Wiebke Nestler. A list of categories was collected based on theoretical considerations and summarized in a smaller number of supercategories. This first coding scheme was examined using a small subsample of the respondents’ answers and was afterwards modified accordingly. The modification process involved the renaming and exchange of categories, supercategories, as well as the change of descriptions of single categories wherever the existing coding scheme was not able to represent the respondents’ answers adequately. For a summarising list of the final supercategories, categories and their descriptions please refer to Table 1. In the following paragraphs, we would like to address some central decisions of the final version of the coding scheme and its categories that were made during the development phase of the scheme. First of all, a relatively broad definition of the term ‘refugees’ provides the basis for the category ‘refugees ambiguous’, ‘refugees negative’ and ‘refugees positive’. Individuals who fall into these categories are the following: asylum seekers, foreigners, immigrants, refugees, immigrants and migrants. Since a SOEP Survey Paper 476 2 differentiated use of those more precise and accurate terms could not be noticed and guaranteed in the respondents’ answers, a differentiation in the coding scheme did not seem appropriate. The supercategory ‘refugees’ was structured in three different subcategories, one of them covering all negative comments on refugees in Germany (‘refugees negative’) and one reflecting a positive attitude towards refugees in Germany (‘refugees positive’). A third category was introduced to summarize all comments that were not clearly identifiable as either negative or positive (‘refugees ambiguous’). This third category was chosen whenever the short answer given by a respondent did not allow a clear assessment of the interviewees attitude towards refugees. This also included statements that are connoted with a clear evaluative judgement (like ‘refugees flood’) in everyday use. For statements like the example ‘refugees flood’ we decided against a coding based on the everyday connotation of the given answer, since one could not tell whether the connotation was intended by the respondent or not and was therefore largely in the eye of the coder and dependent on her sensitivity to handling of language in this context. Additionally, one could not tell if the possibly negative evaluative judgement implied by the statement referred to the refugees themselves or the situation in general. That is why we tried to minimize the room for interpretation of the coders by suggesting a conservative way of coding for answers addressing the topic of migration and refugees whenever there was doubt about the evaluative judgement contained in a statement. This approach was chosen to reduce the bias of codings, leading to an ambiguous category that contains a broad variety of statements. Therefore, we would like to point out, that although this third category was formerly named and conceptualized as a neutral category, it should not be treated as neutral. 3 Coding instruction and process of coding 3.1 Coding Instructions and Process of Coding The respondents’ answers were coded by three independent coders, Laura Lükemann (Universität Bielefeld), Tabea Naujoks (Freie Universität Berlin) and Wiebke Nestler (Universität Leipzig). Although the interviewees were asked for the most important political problem, some of them reported more than one political problem. To capture the problems reported as granular as possible, we decided to code up to five entries of the mentioned political problems (no respondent reported more than five entries). The coders had the instruction to name the number of problems, and afterwards to separately encode the different entries. If an interviewee mentioned several entries, coders were instructed to separately assign them to the corresponding categories. The following example is meant to clarify this process of coding: The entry ‘Refugee crisis, terrorism, contact with Turkey’ covers three political problems. The first problem ‘refugee crisis’ is assigned to the first category ‘refugees ambiguous’, the second entry ‘terrorism’ is assigned to category 6, ‘terrorism’, and the last entry ‘contact with turkey’ is assigned to ‘foreign relations’ (category 12). It may be argued that if a person mentions the refugee crisis and terrorism as the most important political problems, the person considers the existence of refugees in Germany as a problem, and the entry would hence belong in the second category (‘refugees negative’). Since a combined examination of several entries is harder to control and reproduce, we decided upon a more transparent, conservative and verifiable procedure. SOEP Survey Paper 476 3 3.2 Development of the coding tool In order to ease the coding process and allow for quick and consistent coding, the coding instructions were implemented by Martin Bruemmer in a coding tool using the Python Tk library.4 The tool loads commaseparated values (CSV) files that contain respondents’ IDs and their respective answer. It then displays each answer to the coder, allowing them to choose the number of topics mentioned in the answer (Figure 1 A). For each topic identified, the tool then shows all categories available for coding (Figure 1 B). After categories have been selected for each topic identified in the answer, the next answer is presented. Coding is finished when all responses in the loaded CSV file have been coded. The results are then saved to another CSV file named after the coder and the file coded. In addition to the respondents’ IDs and answers, the result CSVs contain the number of topics identified as well as one column for each topic containing the number of the category selected. The source code is available on github5. Figure 1. User interface of the coding tool The tool was built for Windows 10 and Windows 7 using py2exe6 and for Mac OS X using py2app7. These libraries only work when the built itself is run on the operating system it is built for. Thus, to supply all coders with version compatible with their systems, the tool had to be built three times on three different systems. This lack in cross-platform compatibility constitutes a major drawback of using Python to develop user interfaces. Electron8 apps are currently the state-of-the-art in local cross-platform applications but come with a relatively 4 https://docs.python.org/3/library/tk.html 5 https://github.com/der-bruemmer/political-text-codingtool 6 http://www.py2exe.org/ 7 https://pypi.python.org/pypi/py2app/ 8 https://electron.atom.io/ SOEP Survey Paper 476 4 large overhead in development time and system resources required. The online survey framework formR9 provides an easily configurable alternative, allowing ratings to be collected in a web application. However, for the purpose of this study, responses were deemed too sensitive for the web-based approach. 3.3 Harmonization and Indices of Coding The codings of the three coders were merged together and anonymized by an independent person. As shown in table 2, 78 % of the codings of the first entry coincide. In 20 % of the codings, two coders agreed while the third coder diverged. In those cases, the majority principle was used to harmonize the coding. If there were three different codes (2 %), one of the three codes was chosen randomly. Two measures of “inter-rater reliability”, the percentage of agreement and Krippendorff’s alpha, were used to investigate the agreement between the coders regarding the number of topics mentioned in one statement and the categorization of the entries themselves. Both measures were calculated for the entirety of codes (i.e. combining 2015 and 2017). A separate consideration of the years was not implemented, since there were no substantial reasons to do so for the analysis of the inter-rater reliability. The agreement on the categorization of the single entries was calculated using only those cases in which all of the three coders agreed on the topic count (n = 1,203), since it did not seem appropriate to calculate the agreement on the second entry if, for example, one could not be sure if both coders coded the same piece of information. Furthermore, eight cases with no codings resulting from errors in the reading of data were removed, leaving a total of n = 1,195. The percentage of agreement was computed by calculating the average pairwise agreement among all possible coder pairs across all observations. The coders agreed to 95 % on the number of topics that were mentioned in one statement (n = 1,305). The agreement was 88 % for the first entry (n = 1195), 81 % for the second entry (n = 102) and 79 % for the third entry (n = 13). After that, a significant drop of the percentage of agreement was observable, with only 22 % for the fourth entry (n = 3) and 33 % for the fifth entry (n = 1). Since the percentage of agreement does not take into account the possibility of random agreement between coders, Krippendorff’s alpha was additionally calculated. Ratings showed an agreement of α = .753 (n = 1,305) on the number of topics mentioned in one statement. The agreement amounted to α = .847 (n = 1,195) for the first entry, α = .787 (n = 102) for the second entry and α = .77 (n = 13) for the third entry. Inter-rater reliability dropped significantly below the acceptable level of .70 for the fourth (α = .07, n = 3) and fifth rating (α = -.25, n = 1). Both percentage of agreement10 and Krippendorff’s alpha11 showed an acceptable reliability for the number of topics and for the rating of the first three entries. Since the fourth and fifth entry are not to be 9 Arslan, R.C., & Tata, C.S. (2017). formr.org survey software (Version v0.16.5). Zenodo. http://doi.org/10.5281/zenodo.398836 SOEP Survey Paper 476 5