Researcher effects in survey‐based research: Insights from research in South Sudan

Tomiak, Kerstin Article — Published Version Researcher effects in survey‐based research: Insights from research in South Sudan Development Policy Review Provided in Cooperation with: John Wiley & Sons Suggested Citation: Tomiak, Kerstin (2022) : Researcher effects in survey‐based research: Insights from research in South Sudan, Development Policy Review, ISSN 1467-7679, Wiley, Hoboken, NJ, Vol. 40, Iss. 6, pp. 1-15, https://doi.org/10.1111/dpr.12629 This Version is available at: https://hdl.handle.net/10419/266755 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by-nc/4.0/ Dev Policy Rev. 2022;40:e12629. | 1 of 15 https://doi.org/10.1111/dpr.12629 wileyonlinelibrary.com/journal/dpr Received: 26 October 2021 | Accepted: 21 February 2022 DOI: 10.1111/dpr.12629 ARTICLE Researcher effects in surveybased research: Insights from research in South Sudan Kerstin Tomiak This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes. © 2022 The Author. Development Policy Review published by John Wiley & Sons Ltd on behalf of ODI. CODE University of Applied Science, Science, Technology, and Societyprogramme, Berlin, Germany Correspondence Kerstin Tomiak, CODE University of Applied Science, Science, Technology, and Societyprogramme, Berlin, Germany. Email: kerstin.[email protected]c.il Summary Motivation: In policyrelevant and applied research in international development, the evidencebased turn has led to increasing donor demand for evidence that is neutral, objective, and valuefree. Rather than this positivist understanding, the article argues for reflexivity and acknowledgement of positionality to help overcome potential researcher effects. Purpose: Drawing on the example of survey research in South Sudan, the article argues that social relations between the researcher, surveyors, and participants shape the research process and hence knowledge creation. It examines why survey research conducted under similar circumstances led to distinctive data sets. Methods and approach: The argument is based on comparing survey data gathered by two groups of locally hired surveyors in South Sudan and subsequent semistructured interviews with them. Findings: The data show that the researcher's positionality, broadly conceived, influenced data collection. The way the locally hired surveyors perceived the lead researcher and the economically challenging environment of South Sudan— a country severely embroiled in violent conflict— affected the data that were gathered with a social survey, and consequently affected knowledge production. Policy implications: The article contributes to the literature that argues that researcher effects also occur in quantitative research. It shows that in policyrelevant research reflexivity is necessary to strengthen research results. Researchers working in conflictaffected, impoverished environments, and donors requesting evaluation and measurement, should be encouraged to take positionality into account and to ask questions about research practices. KEYWORDS evaluation, interview effects, positivism, quantitative research, reflexivity, social surveys, South Sudan 2 of 15 | TOMIAK 1 | INTRODUCTION This article examines whether and to what extent knowledge produced for policy decisions in conflictridden and impoverished regions is influenced by research practices. Conducting research in such environments matters. Often, such research is concerned with issues related to violent conflict and the associated problems of poverty and human suffering. It is frequently policyrelevant and done to plan and evaluate projects that are conducted to end poverty, improve health care, or contribute to peacebuilding. Thus, such research is directly affecting people's lives. The question this article is concerned with, whether research results are influenced by research practices, does not arise because of the environment in which it takes place or the people and issues with which it is concerned. Research in conflictaffected and lowincome countries often presents particular challenges, including logistical matters and issues with obtaining access to participants. Further, in recent years, the dangers of onthe- ground research in conflictaffected environments have increased, which has resulted in a move towards remote datagathering (Perera, 2017a). These are serious challenges; still, the question of the accuracy of research results arises because of how policyrelevant research with regards to peacebuilding and international development is conducted, particularly because of its frequent grounding in positivism. Positivist methodological approaches understand data to be observable, scientific inquiry as being marked by objectivity, and the rigorous application of method as eliminating subjectivity. This understanding of reality as an independently existing entity from which facts can be gathered, together with policymakers' wishes for objective and clearly organized and presented answers to the question of whether a policy works, potentially leads to flawed research results. Scholars working in critical theory have shed light on the numerous ways in which social relations can play out in the research process and influence results (Behl, 2017; Gallagher, 2016; Mahé, 2019), and in critical and mostly qualitative research, reflexivity and awareness of researcher effects are thus close to standard practice (Alveson & Skoeldberg, 2009; Attia & Edge, 2017; Patnaik, 2013). Reflexivity, the constant “examining how one's positionality, perspective, backgrounds, and insights influence all aspects of the study” (Vagle, 2018, p. 14), has been named “a crucial strategy in the process of generating knowledge by means of qualitative research” (Berger, 2015, p. 219). Still, reflexivity and considerations of the researcher's positionality are, for the most part, absent from quantitative research, with some notable exceptions, for example Louise Ryan's and Anne Golden's thoughts on the value of reflexivity when conducting survey research (Ryan & Golden, 2006) and the work by Brady West and Annelies Blom on interviewer effects on survey data collection processes (West & Blom, 2016). Drawing on this, this article explores the questions of whether and how researcher positionality affects data gathering through a discussion of my own attempts to gather information on media development in South Sudan. The examination of survey research conducted in the country's capital, Juba, and subsequent semistructured interviews with the locally hired surveyors, shed light on the underresearched area of the relationship between international researchers and locally hired surveyors in a conflictaffected, impoverished country, and these relationships' effects on research results. The article argues that much of the applied research conducted in international development and peacebuilding rests on a positivist worldview and that this prevents the acknowledgement of researcher effects, potentially resulting in flawed conclusions. A move away from positivism, on the other hand, would strengthen, not weaken, research results; a reflexive approach and awareness of researcher positionality are vital not only in qualitative but also in quantitative research. My aim is to show the importance of acknowledging context, including the environment, power, and social relations, regardless of whether the data is qualitative or quantitative, and the methods used are based on interviews, survey research or field experiment. I argue that the understanding of research as an independent or neutral datagathering exercise is not valid; researcher effects are present both in qualitative and in quantitative research. To explicate the argument that data are not gathered in an independently existing reality and that data (and thus, knowledge) are instead produced by the actors participating in the research and their relationships and dynamics, the article proceeds as follows. Following this introduction, Section 2 provides an overview of the principles underpinning research in international development, showing its grounding in positivism. This is followed | 3 of 15 TOMIAK in Section 3 by a brief exploration of the social survey as a preferred method and the problems when conducting social surveys in conflictaffected and impoverished environments. In Section 4, the case study— data gathering in South Sudan— is introduced, followed in Section 5 by a description of the two surveys and subsequent interviews with the surveyors and an elaboration of the recruitment of the studentsurveyors. Section 6 analyses the data and presents the insights from the questionnaires and interviews. In Section 7, I elaborate on the research environment as part of the research context, before offering some thoughts in Section 8 on how to manage researcher effects. The conclusion pulls the argument together. Based on the insight gained from comparing the questionnaires and analysing the interviews it argues that the researcher's positionality, how the researcher is perceived by the locally hired surveyors, and the research environment all affect research results. This is a significant insight into the practice of survey research that spotlights the idea that research grounded in positivism and its belief in and demand for valuefree, neutral evidence is problematic. While drawing only on South Sudan, these insights have value beyond this case. South Sudan is severely affected by violent conflict and poverty, and this environment, together with how local surveyors perceived the researcher, affected the data and research results. Such a problematic environment, however, is not unique to South Sudan but is present in many countries in which international development and peacebuilding take place. The article shows that omitting the researcher and potential researcher effects from a study does not prevent such effects. Rather, reflexivity and awareness of positionality would be a better strategy to help manage researcher effects. Thus, the article argues for the need to consider positionality and reflexivity in policyrelevant research conducted in povertyridden and conflictaffected regions in order to better understand the issues under investigation. 2 | A GROUNDING IN POSITIVISM The argument put forward here is that research in general and data collection more specifically are happening in social relations and are consequently shaped by them. This holds true not just for qualitative but also for quantitative research, and this insight is significant for research conducted in countries affected by violent conflict and poverty. In such environments, a researcher might be unaware of social, economic, and political circumstances that have the ability to affect data gathering; it is all the more essential to consider any influences and their potential impact on a study. A penchant for quantitative evidence and social surveys can be spotted in international development and policyrelevant research (Sushant et al., 2018), even if qualitative methods such as focus groups and interviews are mentioned in manuals and publications (Gudda, 2011; Nuguti, 2015; Wholey et al., 2010; Zarinpoush, 2006). Field experiments are also conducted (John, 2017). What these methods have in common in the practice of policyrelevant research is a grounding in positivism and the common notion that “evaluations must be independent, impartial and credible” and that they constitute “an objective assessment of a subject free from undue influences that distort or bias the conduct or findings” (United Nations, 2019, p. 3). This is an example of explicit positivist language, and the positivist underpinning of policyrelevant research is again revealed. In its evaluation policy, the United States Agency for International Development (USAID) points out that “evaluations will be undertaken in a manner that ensures credibility, unbiasedness, transparency, and the generation of highquality information and knowledge” (USAID, 2016, p. 9). USAID further asks for replication and “evaluation findings that are based on facts, evidence, and data” (USAID, 2016, p. 10). Similar language can, for example, be found in the Austrian Development Agency's guidelines (Austrian Development Agency, 2020) and in many Monitoring & Evaluation (M&E) manuals. Organizations, donors, and evaluation experts form a professional network: an epistemic community (Haas, 1992) or an epistemic system, which can arise by design or evolution (Goldman, 2011; Whitcomb, 2011). The verification, and hence sustainability, of certain practices among members are features of an epistemic system and make for strong professional practices and beliefs. The watershed moment that brought positivism into the epistemic community of donors, policymakers, and M&E experts working on peacebuilding and international 4 of 15 | TOMIAK development was most probably the evidencebased turn. It introduced a research agenda with a focus on “defining, managing and measuring” results (Hatton & Schroeder, 2007, p. 426), a predilection for empirical evidence (Sutcliffe & Court, 2006), and the understanding that evidence needs to be technical, neutral, and valuefree (Eyben, 2015). This trend was a departure from the ideologically driven international development of earlier years, particularly the Washington Consensus with its understanding that projects foremost needed to subscribe to liberal economic values. Dean Karlan and Jacob Appel state that “conversations about poverty alleviation and development are [today] much more focused on evidence than they were before” (Karlan & Appel, 2016, p. 2), and Nicole Stremlau notes how “international development donors have become increasingly preoccupied with how ‘evidence’ can ground policymaking” (Stremlau, 2014, p. 1). Empirical evidence is essential for donor decisions (Sutcliffe & Court, 2006, p. 2). The increasing demand for empirical evidence has been linked to two major concerns in international development: (1) that most development agencies report outputs, not outcomes; and (2) that related to and as a consequence of this, there is very little reliable evidence about the actual results of projects and interventions (Bamberger et al., 2010, p. 614). Whether development projects and policy decisions indeed contribute to democratization, poverty eradication, and reduction of illiteracy (among other issues), and to what extent, can thus still be seen as open to debate. In general, there is discussion in the international development community and among academics and practitioners regarding the evidencebased framework, with some being very critical of it and others seeing benefits in the approach (Eyben & Guijt, 2015, chap. 1). For example, it has been said that “quantitatively oriented development researchers, if not the development community as a whole, have responded enthusiastically to the evaluation challenge” (Bamberger et al., 2010, p. 615). Much of the research in the field of international development and on policyrelevant questions, though, as stated above, by no means all, has since the introduction of the evidencebased turn been done using quantitative instruments such as social surveys, and a need for “objective” data is clearly phrased by donor agencies, as shown above. 3 | SOCIAL SURVEYS AS A PREFERRED METHOD For decades, since the beginning of the last century, researchers have found surveys to be a useful instrument for data collection (Bulmer, 2001). They produce quantitative descriptions of people's lives and aim to make experiences comparable through different contexts (Balnaves & Caputi, 2001; Fowler, 1984). They are “a feasible and relatively economical means of collecting social data” (Bulmer, 1993, p. 8). Also, numerical data is understood to be objective (Merry, 2016; Porter, 1995), and “quantification…offers concrete, numerical information that allows for easy comparison and ranking of countries, schools, job applicants, teachers, and much else” (Merry, 2016, p. 1). Furthermore, quantitative research is “most of the time…concerned with cause – with questions about whether some intervention works” (Turner, 2007, p. 121). Hence, it offers answers to the questions asked by development agencies and donors as to whether a programme or project works. Besides all these advantages, there are also wellknown problems with surveys. The data collected may be of poor quality, as Morten Jerven (2013) described in the context of fieldwork in Africa, showing how poor quantitative data misled the understanding of economic development. Others described problematic selfreporting (Magee & Doces, 2015) and meagre selfassessment. For example, the actual number of saplings planted in India was significantly lower than the demand for saplings that farmers had reported in a survey. The authors of this study concluded that procedural and more longterm engagement with the surveyed population would have led to better survey results, where “better” means describing actual needs (Sushant et al., 2018). This gap between reported and actual behaviour is a wellknown issue for survey research (Dauti, 2021). With regard to media interventions, Jessica Noske- Turner depicted less ideal research and evaluation practices with “some…added a survey or some statistics…, but it was rare for quantitative data to be statistically representative or reliable, with commonly reported issues in the quality of sampling and lack of trust in local | 5 of 15 TOMIAK ratings agencies and research institutes” (Noske- Turner, 2017, p. 10). Correct sampling can be problematic in conflictedaffected countries where there is often no reliable census, making the representative modelling of the target audience impossible or nearly so. Other challenges for survey research include bias (Fowler, 1984; Weisberg & Bowen, 1977) and “accuracy limitations” (Zarkovich, 1993, p. 103), such as misunderstandings and confusion about the questions asked and concepts used on a questionnaire. Such misunderstandings are persistent when questions are oversimplified. Surveys have been critiqued for introducing concepts that might be removed from local reality, although there have been attempts to overcome this particular problem (Firchow, 2018). In general, and despite the wellknown problems, there is a tendency among policymakers to value simplified, reductionist knowledge over complex, contextual knowledge (Perera, 2017b). This tendency might be understandable given the need for accountability and transparency; however, omitting the context can lead to erroneous research results. The lack of a discussion of how knowledge is produced and not gathered and the researcher's separation from the presumed objective reality, as well as the consequent lack of researcher reflexivity, are all wellknown points of critique with regard to research underpinned by positivism (Alveson & Skoeldberg, 2009). The failure to consider researcher effects in positivist research is all the more puzzling as the enumerator effect is well known in survey research. How enumerators or survey interviewers influence survey results has been described, for instance in relation to religion (Blaydes & Gillum, 2013), public health (Davis et al., 2010), and, in more general terms, in the literature on methodology (Singh, 1990; West & Blom, 2016). The question of whether interviewer or enumerator effects matter has also recently been posed in international development (Pietrelli et al., 2021). This rich literature is concerned with the effects that surveyor characteristics might have on the surveyed, thus showing how research is shaped by social relationships. However, the thus far underresearched relationship between the researcher and the surveyors can also have an impact. In countries affected by conflict or experiencing its aftermath, and in the context of international development, this relationship is often between an international researcher and locally hired surveyors, which can further affect the research results. This section has shown the positivist grounding of policyrelevant research and reviewed the potential problems of such an approach, particularly when it comes to social surveys. The following section looks further at survey research in South Sudan, shedding further light on the problems with positivist research in international development and peacebuilding. 4 | DATA GATHERING IN SOUTH SUDAN The following description of data gathering for quantitative research in South Sudan shows how the researcher's positionality, as seen by the locally hired surveyors, created unexpected glitches that resulted in two dissimilar data sets delivered by the same questionnaire. This inability to repeat survey research and gather approximately the same data shows that reflexivity is essential, and that positionality needs to be considered, when collecting quantitative data. Positionality “refers to the perspective, orientation and situatedness of the researcher visà- vis the researchees” (Henry et al., 2009, p. 468). It concerns the individual or the team conducting the research as well as the researcher's self and their position towards the research and the research participants. A researcher's gender, class, and ethnicity can all play an important role; thus, positionality is about asking and acknowledging the questions, “who am I?” and “how is who I am influencing aspects of the research?” (Bourke, 2014, p. 2; Tomiak, 2019, p. 98). In positivist research, such considerations are seldom present, and while, as mentioned above, enumerator effects are well known, the management of these effects is mainly limited to the idea of matching interviewers with the respondents' sociodemographic features (Davis et al., 2010). In policyrelevant research in conflictaffected and divided countries, such as South Sudan, this rarely happens, and the relationship between lead researcher and locally hired surveyors creates an additional field for potential researcher effects. 6 of 15 | TOMIAK The conclusions presented here are based on a comparison of a social survey conducted twice in Juba in 2014 and 2015, and subsequent semistructured interviews with the locally hired surveyors. The circumstances of the two surveys were roughly similar, with the exception being the social distance of the international researcher— me— to the locally hired surveyors. This was not planned initially; the survey was originally part of a wider research project that investigated the effects of Westernled media development in a postconflict country (Tomiak, 2022). When the survey was conducted the first time, in December 2014, it delivered problematic data for the planned analysis, which led to the decision to conduct the survey a second time. This was then done in July 2015. The second survey delivered different results which fitted the planned analysis for the question on media interventions. Still, together the two surveys gave rise to the question of why the results were different and the meaning of this for policyrelevant research in conflictaffected and impoverished environments more generally. Subsequent semistructured interviews with the studentsurveyors led to the conclusion that the distance to, knowledge of, and, thus, position of the researcher towards the surveyors, together with the research environment— South Sudan— were the main reasons why the two surveys produced different data. At the same time, the research conducted— surveys and interviews— provided insights into the social reality of South Sudan that were potentially meaningful for possible future media projects. In the next section, the two surveys and the interviews subsequently carried out are described in more detail. 5 | THE TWO SURVEYS AND SUBSEQUENT INTERVIEWS WITH THE SURVEYORS A comparison of the data produced by survey one and survey two, the recruitment process, and the statements of the locally hired surveyors led to the conclusion that the researcher's positionality led to the tremendously different data sets and thus influenced the research. The survey research was conducted as part of a larger project on media development. In particular, the aim was to find out about people's media usage, the preference for one or another of the radio stations active in Juba, and if these preferences were connected to opinion about the South Sudanese government and attitudes towards different ethnic groups in the country. The questionnaire thus consisted of questions that would have been used in a baseline study, asking about time spent listening to broadcasts, but also asked respondents to rank specific statements about attitudes towards the government and ethnic groups on a Likert scale. Prior to the research, the questionnaire had been discussed extensively with a panel of experts— six South Sudanese nationals, among them journalists, teachers, and employees at international and local nongovernmental organizations (NGOs). Following this, the questionnaire had been pretested. A testrun was conducted, with two South Sudanese acquaintances voluntarily distributing 50 questionnaires. Paper questionnaires were used. For the actual surveys, both times sample size had been calculated with a margin error of 5%, a confidence level of 94%, and a standard derivation of 0.5. The biggest problem was population size: the last census in southern and northern Sudan, prior to the research, had been conducted in 2008 before the country became independent. The validity of this census is questionable at best; results were said to be increased or decreased due to logistic challenges but also for political reasons (Afrol, 2009; Fick, 2009; Thompkins, 2009). In this census, the population of Juba was given as 368,436 inhabitants (National Bureau of Statistics, 2008). In the years between this census and the start of the research project described here, the country had conducted a referendum on secession from Sudan, declared its independence, and experienced a new outbreak of violent conflict. It could reasonably be assumed that the population number given in 2008 was no longer valid, if it had ever been true to begin with. Thus, determining population size was sheer guesswork, a common problem for surveys conducted in post/conflict countries. Population size was decided at 1.5 million people, and the sample size was calculated ( Z - s c o r e ) 2 × StdDev × (1- StdDev) / (margin of error) at 354. For each of the two surveys, 500 questionnaires were distributed to allow for data cleaning and potential loss of questionnaires. Twenty South Sudanese students from Juba University were employed for each survey, tasked with distributing 25 questionnaires using a snowball | 7 of 15 TOMIAK system for data collection to protect the surveyors' safety. Due to the country's problematic security situation, discussions of rather sensitive topics, like the media or the government, either with strangers or in public, was not advisable. Up to this point, methodological choices were similar for what would become survey one and survey two. The first survey was conducted in December 2014. When the questionnaires came back, they showed various issues that rendered the data collected invalid: multiple boxes had been ticked where only one answer was permitted, categories had been included, and comments were scribbled in the margins. Because of this, the survey was repeated under similar circumstances in July 2015. At that point, it delivered different data and did not show any of the issues experienced with survey one. Despite similar methodological choices, the outcome of the two surveys was remarkably different. This led to the question about why these differences had occurred, and to investigate this question further, the data, the questionnaires, and field notes were compared to examine the similarities and differences between the practice of conducting the surveys. Also, interviews were conducted with the studentsurveyors. Twenty students from the Political Science Department had been working on survey one and 20 students from the Mass Communication Department were employed for survey two. All 40 students were asked to participate in semistructured interviews. Interview questions centred around three points: (1) the studentsurveyors' experience in conducting the survey, how they contacted respondents and decided whom to contact, and their practices of filling in the questionnaire; (2) the recruitment process and training given, how they heard of the opportunity and put their name down for it, and how they experienced the training; and (3) their perception of me as the international researcher and employer. While the direction of the interviews was preordained, within this framework studentsurveyors were encouraged to describe their experiences in depth and in their own words. Using prompts and probes, the aim was to encourage a conversation to gain an understanding of the studentsurveyors' experience of the recruitment, training, and surveying processes and practices. Of the 40 studentsurveyors who worked on the two surveys, 25 agreed to participate in formal, recorded interviews; 12 of these had worked on survey one, 13 on survey two. The interviews, which lasted on average 60 to 90 minutes, were audio recorded on an iPhone and later transcribed. Another 12 studentsurveyors, eight from survey one and four from survey two, agreed to share their experiences, but only in rather informal exchanges that were not recorded. These chats also lasted on average 60 minutes and extensive notes were taken. The interviews and notes were analysed using thematic analysis. As the most remarkable difference between the two surveys was the recruitment process, showing the importance of the lead researcher's positionality, I will detail this in the next section, before turning to the analysis of data and the findings. 5.1 | Recruitment of the studentsurveyors Survey one was conducted in December 2014. Twenty fourthyear students from Juba University's Political Science Department were employed. A personal contact at the university allowed for an easy recruitment process: an acquaintance who worked in a senior position at Juba University. They arranged for a group of students to work as surveyors, and a date was set for me to meet with them, explain the research, and provide some training. On the day, my acquaintance accompanied me to the classroom, introduced me to the students, and asked them “to help my friend and colleague Ms. Kerstin with her very important research” (field notes, December 2, 2014). The research aims and how the questionnaire should be completed were explained during a 90- minute training session. The brevity of this training was a concern, but I was assured that research training was part of a degree programme at Juba University, and some of the students had worked as surveyors before. Contact information was provided in case of additional questions, and a time for collecting the completed questionnaires and payment was arranged. As mentioned, this survey delivered highly problematic data, with scribbles in the margin and additional categories. Thus, the survey was repeated in July 2015. 8 of 15 | TOMIAK For the second survey a more informal recruitment process took place. In July 2015 I had spent roughly 10 months in South Sudan, and during this time, ties to the university had strengthened; I had taught a class at the Department of Mass Communication. For the second recruitment process, to keep circumstances comparable, students were again approached through a faculty member; but this time, they had seen me on university premises and knew that I was teaching there. While the students approached were not my students, I had met many of them on campus. When their lecturer approached them with the offer to do some survey work for me, and when I entered their classroom to provide training, the students already knew who I was. This survey delivered quality data: the students had followed my instructions to the letter. As the recruitment process and knowledge of me were the only significant differences between the two surveys, this seems to have been the main, or at least an essential, reason for the different data collected. To examine this premise— that the difference in distance, and thus, the researcher's positionality, had affected the datagathering processes— the interviews described above were conducted with the studentsurveyors. 6 | INSIGHTS FROM THE QUESTIONNAIRES AND INTERVIEWS The two sets of data, consisting of the questionnaires (the data collected and survey results) and the interviews (conducted with the studentsurveyors after the data gathering), provided three insights: (1) that the way the surveyors perceived me had a significant impact on the datagathering process; (2) that this perception of me, together with the environment where the research took place and the situation of the locally hired surveyors played an important role; and (3) that reflexivity and acknowledgement of the researcher's positionality does not weaken research results, but is in fact necessary to identify and manage potential researcher effects. The two sets of questionnaires on the question of media development delivered significantly different data, with the data from the first survey being problematic for the planned analysis. The reason for this significantly different data was that the studentsurveyors' perception of me, the lead researcher, differed in a substantial way. The interviews showed that the students who worked on survey one had seen me as an influential person with the ability to provide further, and potentially more permanent, employment. This (incorrect) assumption, together with the problematic economic situation in South Sudan, made one set of the studentsurveyors want to stand out as surveyors in the hope of recommending themselves for future work opportunities. The second set of students, who had a clearer idea of who I was, did not feel this need. This is explored in more detail in this section, which looks at the questionnaires as visual representations of the datagathering process, at the recruitment and training processes of the two surveys and the interview data of the two sets of studentsurveyors. In the questionnaires of the first survey, comments had been scribbled in the margins, additional categories had been added, and multiple boxes had been ticked on questions for which only one answer was required. Thus, a total of 17 subcategories of the independent variable had been created for the question on which radio station respondents listened to the most. With these 17 subcategories, with several having only one or two instances, statistical power was drastically reduced; a meaningful analysis of the initial research question regarding democratization and the peacebuilding effects of the media was close to impossible. Datareduction methods would have been an option to make the data behave, and the idea of cleaning unruly data is common in quantitative analysis. Advice is given on how to detect and tame nasty data (McClelland, 2000). Data cleaning refers to removing unusable, incomprehensive, or simply unnecessary data from a data set. This process, however, typically starts when the data are already entered into an analysis software such as SPSS. In this way, the data are “cleaned” on being transferred from the questionnaires to the computer, becoming a more ordered version of themselves. Instead of this, questionnaires filled out by hand can also be understood not just as containers of information but as visual representations of a thought process. As such, paper questionnaires can reveal additional information. Visual inspection of the questionnaires from the first survey gave rise to the suspicion that some kind of negotiation process had been ongoing between the surveyors and the participants. The questionnaires showed that in | 15 of 15 TOMIAK USAID. (2016, October). Evaluation: Learning from experience: USAID Evaluation Policy. January 2011 updated October 2016. h t t p s : / / 2 0 1 2 - 2 0 1 7 . u s a i d . g o v / s i t e s / d e f a u l t / f i l e s / d o c u m e n t s / 1 8 7 0 / U S A I D E v a l u a t i o n P o l i c y . p d f Vagle, M. D. (2018). Crafting phenomenologial research. Routledge. Webb, E. J., Campbell, D. T., Schwartz, R. D., & Sechrest, L. (1966). Unobtrusive measures: Nonreactive research in the social science. Rand McNally & Company. Weisberg, H. F., & Bowen, B. D. (1977). An introduction to survey research and data analysis. W.H. Freeman & Co. West, B. T., & Blom, A. G. (2016). Explaining interviewer effects: A research synthesis. Journal of Survey Statistics and Methodology, 5(2), 175– 211. https://doi.org/10.1093/jssam/ smw024 Whitcomb, D. (2011). Introduction. In A. I. Goldmann & D. Whitcomb (Eds.), Social epistemology: Essential readings (pp. 11– 37). Oxford University Press. Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (2010). Handbook of practical programme evaluation. Jossey- Bass. Zarinpoush, F. (2006). Project evaluation guide for nonprofit organizations: Fundamental methods and steps for conducting project evaluation. Imagine Canada. Zarkovich, S. S. (1993). Some problems of sampling work in underdeveloped countries. In M. Bulmer & D. P. Warwick (Eds.), Social research in developing countries: Surveys and censuses in the Third World (pp. 101– 108). UCL Press. How to cite this article: Tomiak, K. (2022). Researcher effects in surveybased research: Insights from research in South Sudan. Development Policy Review, 40, e12629. https://doi.org/10.1111/dpr.12629