Research Perspectives : Reconsidering the Role of Research Method Guidelines for Interpretive, Mixed Methods, and Design Science Research

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC-ND 4.0 https://creativecommons.org/licenses/by-nc-nd/4.0/ Research Perspectives : Reconsidering the Role of Research Method Guidelines for Interpretive, Mixed Methods, and Design Science Research © 2020 the Authors Accepted version (Final draft) Siponen, Mikko; Soliman, Wael; Holtkamp, Philipp Siponen, M., Soliman, W., & Holtkamp, P. (2021). Research Perspectives : Reconsidering the Role of Research Method Guidelines for Interpretive, Mixed Methods, and Design Science Research. Journal of the Association for Information Systems, 22(4), Article 1. https://doi.org/10.17705/1jais.00692 2021 Reconsidering the Role of Research Method Guidelines for Interpretive, Mixed Methods, and Design Science Research Mikko Siponen1, Wael Soliman2 and Philipp Holtkamp3 1University of Jyväskylä, Faculty of Information Technology, Finland, mikko.t.sipone[email protected] 2University of Jyväskylä, Faculty of Information Technology, Finland, w[email protected] 3University of Vaasa, Innovation Office, PO Box 700, FI-65101 Vaasa, Finland, [email protected] Abstract Information systems (IS) scholars have proposed guidelines for interpretive, mixed methods, and design science research in IS. Because many of these guidelines are also suggested for evaluating what good or rigorous research is, they may be used as a checklist in the review process. In this paper, we raise the question to what extent do research guidelines for interpretive, mixed methods, and design science research offer such evidence that they can be used to evaluate the quality of research. We argue that scholars can use these guidelines to evaluate what good research is if there is compelling evidence that they lead to certain good research outcomes. We use three well-known guidelines as examples and contend that they seem not to offer evidence such that we can use them to evaluate the quality of research. Instead, the “evidence” is often an authority argument, popularity, or examples demonstrating the applicability of the guidelines. If many research method principles we regard as authoritative in IS are largely based on speculation and opinion, we should take these guidelines less seriously in evaluating the quality of research. Our proposal does not render the guidelines useless. If the guidelines cannot offer cause-and-effect evidence for the usefulness of their principles, we propose seeing the guidelines as idealizations for pedagogical purposes, which means that reviewers cannot use these guidelines as checklists to evaluate what good research is. While our examples are from interpretive, mixed methods, and design science research, we urge the IS community to ponder to what extent other research method guidelines offer such evidence that they can be used to evaluate the quality of research. Keywords: research guidelines, interpretive research, design science, mixed methods, theory of scientific methodology 1 Introduction In the past, the mainstream methodology in Information Systems (IS) was statistical (Orlikowski & Baroudi, 1991). To increase the publication opportunities in the alternative research genres, such as interpretive, mixed methods, and design science, research method guidelines were introduced in these areas (Hevner, March, Park, & Ram, 2004; Klein & Myers, 1999; Venkatesh, Brown, & Bala, 2013). For example, guidelines for interpretive field studies were motivated by interpretive studies not being “widely accepted” (Klein & Myers, 1999, p. 67). To give another example, the most frequently cited mixed methods guideline was motivated because “there is a dearth of mixed methods research in information systems” (Venkatesh et al., 2013). These guidelines can be credited for increasing the visibility of interpretive, mixed methods, and design science research in IS. However, sometimes good things come with a price. One possible side effect is that the guidelines can also prevent the publication of good research, if reviewers regard them as legislating what is acceptable (or rigorous or high quality) and what is not acceptable IS research from a methodological viewpoint. How real is this problem? Many authoritative sources, including the former editor-in-chief (EIC) of MISQ and the EIC of EJIS, imply that generally this is the case: If editors do not signal their a prioris about a paper to the reviewers, then the natural inclination of the reviewers is to judge the paper against very high methodological standards first, which leads them to subsequently have a different, less favorable view of the overall contribution of the paper. (Straub, 2008, p. ix) In our experience, methodological rigor is a prerequisite for publication in IS journals. Increasingly, we observe, editors are willing to jettison strong theory for papers that have good empirical contributions and potential theoretical implications [...]. But we have seen no similar looseness over method. (Rowe & Markus in Hovorka et al., 2019, p. 1362) Moreover, Grover and Lyytinen (2015) note how IS authors “produce knowledge that seeks to get through reviewers looking to check boxes on theory and method” (p. 275). These reports imply that (1) reviewers in top IS journals often use method guidelines (standards) in the review process; and that (2) method guidelines can influence the review process (in top IS journals). In fact, “check boxes on method,” “judge the paper against very high methodological standards,” and “methodological rigor is a prerequisite for publication” imply that not meeting methodology guidelines alone may lead to rejection by top IS journals. Related concerns can be found in IS literature. Fitzgerald (2008) reports that during the doctoral consortium of the International Conference on Software Engineering, “research method was mentioned just once (and that was by a student) and the focus was much more on the actual content of the research.” He states that when he attended the doctoral consortium of the European Conference on Information Systems (ECIS), “more than 50% of the time involved discussions of research method issues. However, I do not necessarily think that this was time well-spent.” The reasons for check-box compliance are complex. On one hand, readers (e.g., IS authors, reviewers, and editors) may require strict compliance with some method guideline, irrespective of what the guideline said. On the other, the research method guidelines may lay down principles in a legislative or normative manner. Consider, for example, guidelines for interpretive (Klein & Myers, 1999), design science (Hevner et al., 2004), and mixed methods (Venkatesh et al., 2013). All caution against rote or mechanistic use of methodological principles. However, all three guidelines are introduced not only for conducting research but also for evaluating what good or rigorous research is. This implies for IS readers that research that does not meet the evaluation criteria is not good or not rigorous. In this provocative paper, we debate to what extent research method guidelines offer such evidence that they can be used to evaluate good, high-quality, or rigorous research. We focus on the interpretive, mixed methods, and design science research genres (see Section 2.2). However, many of our claimssuch as whether there is evidence that allows one to evaluate the quality of researchmay be useful in scrutinizing the research method guidelines in other genres. Asking these questions is important. Despite all good intentions, reviewers’ strict reading of these guidelines may unduly block some important research that does not meet them. For example, a highly cited mixed methods guideline recommends that “IS researchers should employ a mixed methods approach only when they intend to provide a holistic understanding of a phenomenon for which extant research is fragmented, inconclusive, and equivocal” (Venkatesh et al., 2013, p. 36). This suggests that, in other situations, scholars should not use mixed methods. Reviewers who follow the guideline to the letter should not then allow mixed methods research (per Venkatesh et al., 2013), if the extant research is not fragmented. But then, reviewers may block important mixed methods research in the area where existing research is not fragmented or inconclusive. Moreover, if a deviation from the guidelines increases the risk of rejection by the top IS journals (as implied by Straub, 2008; Rowe & Markus in Hovorka et al., 2019), then research settings that do not meet the guideline may be avoided as too risky. Furthermore, if IS scholars mainly “produce knowledge that seeks to get through reviewers looking to check boxes on [...] method” (Grover & Lyytinen, 2015, p. 275), then there is a risk that research education may be trivialized as a checklist approach so that the research meets the guidelines of good or rigorous research. Finally, and importantly, even if statements about check-box compliance and methodological rigor as prerequisites for publication (Grover & Lyytinen, 2015; Rowe & Markus in Hovorka et al., 2019; Straub, 2008) do not apply widely among the top (e.g., Basket 6) IS journals, we must guard against the risk that IS research is not widely and unduly blocked in the future because the research does not meet some guidelines of “rigor,” which lack demonstration of cause and effect. Klein and Myers (1999) emphasize that “[u]ltimately the quality (and status) of interpretive research within IS will benefit from a lively debate about its standards” (p. 68). This point is important. Scientific ideas should not be accepted dogmatically simply because they are published in top journals, are written by famous scholars, or are highly cited, but because they can withstand serious scrutiny from the scientific community. Such scrutiny can also reveal the weaknesses of the scientific ideas (Laudan, 1977). Moreover, scientists should be allowed to suggest scientific ideas be rejected, in light of evidence, or because of a lack of evidence. Unfortunately, not only such a lively debate but also a critical review of these research method guidelines seem to be missing (in our top journals). The research method guidelines we review in this paper are widely used (as indicated by the citations), but researchers have not yet seriously scrutinized these guidelines. We start this lively debate in Section 2 by asking fundamental methodological questions about these guidelines. In Section 2.1, we introduce several concepts, and in Section 2.2, we point out that the guidelines (we review) follow a legitimization strategy, which typically outlines four types of evidence. We advance the interpretation that none of the types of evidence allows one to make normative claims for how to conduct or evaluate good, rigorous, or high-quality research. In Section 3, we discuss the theory of scientific methodology, and its implications for research method guidelines for interpretive, mixed methods, or design science research. We end by presenting a naturalistic approach to research method guidelines in IS, which regards them as either scientific hypotheses with evidence or idealizations. The naturalistic approach requires evidence that each principle leads to a specific outcome. We are skeptical whether such evidence can be provided in settings such as interpretive, mixed methods, or design science research, which allows us to say that certain research principles promote good research outcomes better than others. Alternatively, we suggest that research method guidelines for interpretive and design science and in which situations to use mixed methods should be given the status of idealizations and may have various benefits for educational purposes. Having said that, it is debatable whether those research method guidelines, offering no causal evidence for how their principles “cause” good outcomes, can be used to evaluate the quality of research, for which purpose these guidelines are also proposed. The usefulness of such research guidelines may lie elsewhere, including pedagogical purposes. 2 Methodological Guidelines for Interpretive, Design Science, and Mixed Methods Research We first explain research method guidelines (RMGs) and research method principles (RMPs), and then we review three RMGs for research in IS, and why we selected them. We point out that these guidelines are also outlined as criteria for good or rigorous research. Finally, we review what evidence these RMGs provide to back up their claim that they can be used as guidelines for how to conduct and evaluate good research. 2.1 RMGs and RMPs In the philosophy of science, RMPs and RMGs belong to the “theory of scientific methodology” (Laudan, 1981a, p. 3). There is no common definition for RMPs and RMGs in the philosophy of science. Roughly speaking, RMPs are concerned with “how scientific theories in general are appraised and validated” (Laudan, 1981a, p. 3). What philosophers propose as RMPs in the philosophy of science varies. We characterize RMPs as any principles that provide normative guidance for conducting or evaluating good research (or both). In IS, RMPs can vary from requiring some tests, such as p values, and sample sizes (in statistical research) to requiring certain steps to conceptualize a construct (Polites, Roberts, & Thatcher, 2012) or procedures to validate it (Mackenzie, Podsakoff, & Podsakoff, 2011). RMPs may also propose normative statements about when using a research method is, or is not, acceptable (Venkatesh et al., 2013). An RMG consists of one or more RMPs; thus, broadly speaking, an RMG is a collection of RMPs. For example, Klein and Myers (1999) suggest seven principles for interpretive research (see Table 1). Each principle can be called an RMP, according to our terminology, while the seven principles altogether form an RMG. 2.2 Guidelines for Conducting and Evaluating Good or Rigorous Research It is important to separate the descriptive and prescriptive functions of RMGs. Descriptive use is when articles or books, for example, characterize RMGs, or give examples of their use, without necessarily imposing certain practices as preferred or required for evaluating quality. Prescriptive function is when RMGs are proposed or imposed for conducting or evaluating research. When should RMPs be rationally imposed or required? Requiring or imposing RMGs is clearly rational, for example, when there is undisputed evidence that RMGs are necessary to achieve some specific good research outcomes. Further, linking the quality of research outcomes to RMPs typically assumes causality between them. If, for example, design science or interpretive RMPs are required to be followed (for good science), then there is an assumed causal relationship between RMPs and outcomes. The point is, why do we require an RMP if we do not have compelling evidence for its causal role in good research outcomes? Similarly, reviewers who impose the guidelines for evaluating what is acceptable science seem to implicitly assume cause and outcome (effect) relationships, where RMPs have some causal influence on good or acceptable research outcomes. Our concern is that reviewers may impose research method guidelines for conducting and evaluating research, when these imposed guidelines lack evidence for the need for RMPs that “cause” good outcomes. Therefore, we want to draw the attention of the IS community to ask to what extent RMGs contain evidence that allows one to use the RMG to evaluate how good research is conducted and evaluated. At the same time, many RMGs contain other evidence, such as evidence of their use, which some may wrongly confuse as causal evidence of the research quality outcomes. Introducing these issues and concerns requires specific examples from actual RMGs. Accordingly, we selected guidelines in three different genres: interpretive studies, design science research, and mixed methods research. In these areas, we found several observations. First, we found influential and potentially authoritative RMPs, which can be inferred by IS readers as normative. Second, it is questionable whether these RMGs contain necessary evidence that closely ties each RMP to good outcomes. Third, these RMGs contain other evidence, which can be wrongly confused with demonstrating evidence of the effects of RMPs on good outcomes. Finally, IS scholars working in these alternative genres do not generally or discuss this evidence (the second and third points). We selected three domains (interpretive studies, design science research, and mixed methods research) as examples to illustrate that these issues are relevant not only in one genre but also in different genres. For selecting one guideline in each genre, we chose influential RMGs. For influence, we looked for citations for lack of a better measure. Although citations do not demonstrate the quality of the study, they may demonstrate influence. Accordingly, we reviewed RMGs for interpretive field studies (Klein & Myers, 1999), design science research (Hevner et al., 2004), and mixed methods research (Venkatesh et al., 2013). These RMGs were published in Management Information Systems Quarterly (Hevner et al., 2004; Klein & Myers, 1999) or Information Systems Research (Venkatesh et al., 2013), and are highly cited. Although we selected these guidelines, most questions we ask go beyond them. For example, our discussion is simply that we believe our proposed principles are consistent with a considerable part of the philosophical base of literature on interpretivism and hence an improvement over the status quo” (Klein & Myers, 1999, p. 68). Two observations are necessary. First, even if we accept that an RMG can be justified or acceptable when it is consistent with “a considerable part of” something, the interpretive RMG does not (try to) show that its principles “are consistent with a considerable part of the philosophical base of literature on interpretivism.” To clarify, making this claim requires supporting evidence, for example, based on a review of all philosophical literature on interpretivism, and then illustrating that the proposed guidelines are consistent with said philosophy. However, they “decided to concentrate on the hermeneutic philosophers, especially Gadamer and Ricoeur” (Klein & Myers 1999 p. 70). They admit that “the complete literature of interpretive philosophy comprises so many varied philosophical positions that it is unlikely to yield one consistent set of principles for doing interpretive research” (ibid., p. 70). One may also question to what extent the interpretive RMG (Klein & Myers, 1999) is consistent with the philosophical literature on interpretivism. For example, “the most fundamental principle” is the hermeneutical circle (ibid., p. 71). Stegmüller (1977 p. 8) provides several examples of how the understanding process does not follow a circle but a “hermeneutic spiral” or a dilemma. To give another example, one can ask to what extent the philosophy underlying interpretive research supports the idea of presenting a priori prefixed quality evaluation principles. For example, according to Salmon (2003, p. 722), famous interpretivists such as Dilthey, Collingwood, Winch, and Geertz either reject or at least limit causal explanations in social sciences. At the same time, using RMGs as evaluation criteria or evaluation criteria for good research seems to require assuming some causality (see Section 3.3.2). Second, the justification for guidelines is consistency with one or a set of writers. Scientific writings often report that the findings are consistent with those of other studies. In some circumstances, this is reasonable. Having said that, readers must understand that using the consistency argument is problematic for scientific justification. Consider the following wellknown thesis: The Earth is flat. Then, consider the following argument: The Earth is flat because this view is consistent with Carpenter’s (1885) flat Earth theory. It is true that this argument is consistent with Carpenter’s (1885) view. However, who would accept this as evidence that the Earth is flat? In scientific research, it is not a good justification practice for researchers to base their arguments on saying that their opinion is consistent with some other opinions. Justifying claims by stating consistency with a previous study does not require the presentation of evidence for or against the claim. Proposing a principle for conducting and evaluating high-quality research, be it interpretive, mixed methods, or design science, should require the presentation of available evidence for and against each principle. This evidence should not be replaced with someone’s opinion (without evidence) and references that are consistent with these opinions. 2.3.2 Evidence of Applicability The RMGs we reviewed select two or three articles they call exemplars. The interpretive RMG uses “three published examples of interpretive field research from the IS research literature…in order to demonstrate how authors, reviewers, and editors can apply the principles” (Klein & Myers, 1999, p. 79). Similarly, the design science guideline states: Following Klein and Myers (1999) treatise on the conduct and evaluation of interpretive research in IS, we use the proposed guidelines to assess recent exemplar papers published in the IS literature in order to illustrate how authors, reviewers, and editors can apply them consistently. (Hevner et al., 2004, p. 78) In turn, the mixed methods RMG “illustrate[s] the applicability of our guidelines using two exemplars of mixed methods research from the IS literature” (Venkatesh et al., 2013, p. 23). They note that their goal for these exemplars is to “demonstrate how our guidelines can be used to understand and apply the process of conducting and validating mixed methods research in IS” (ibid., p. 45). We interpret these examples as examples of how reviewers, editors, and authors can apply the RMPs. These RMGs are proposed for conducting and evaluating good or rigorous research. Let us presume for the sake of the argument that the guidelines possess the capability of pointing out what good or rigorous research is. For reviewers and editors to judge whether the research is good or rigorous, based on the guideline, should the RMG also point out cases that do not meet the guidelines? Similarly, if we want authors to “understand and apply the process of conducting and validating” (Venkatesh et al., 2013) high-quality or rigorous methods research in IS, then should we also explain and justify why certain practices are not rigorous and not high quality? One could use these deviations to show, for example, what important complications result. However, the RMGs we reviewed do not present such deviations, which do not meet the principles of high quality or rigor. The RMGs we reviewed purposefully avoid criticizing any paper. For example, the mixed methods RMG notes that “the purpose of this discussion is not to critique the application of the mixed methods approach in these papers” (Venkatesh et al., 2013). Similarly, the design science RMG notes: “Our goal is not to perform a critical evaluation of the quality of the research contributions, but rather to illuminate the design-science guidelines” (Hevner et al., 2004, p. 90). Importantly, these RMGs do not explicitly say that these exemplars are used to validate, justify, or test the guidelines. For example, for the mixed methods RMG, the two cases illustrate “[t]he applicability of these guidelines” (Venkatesh et al., 2013). For the design science RMG, the exemplars “demonstrate the application of these guidelines” (Hevner et al., 2004, pp. 75–76). The applicability evidence presented should not be confused with the quality of the research or a demonstration of cause and effect. We illustrate this point with a simple provocative example. Let us presume that one is diagnosed with cancer, and the treatment advice is to walk one mile every day. If one can do that, we may agree that it demonstrates that the advice was doable (for this person at least). However, that a person is able to walk for one mile is not evidence that this is a good cancer treatment. Moreover, a high number of citations may indicate the influence or impact of the article. Having said that, a high number of citations is not evidence that the guidelines can be used to evaluate what good research is. Generally speaking, how many times a paper is cited should not be conflated with evidence of the outcome. The number of citations is not evidence that the claim is true or justified. We may agree that the theory the Earth is flat is widely known. However, despite being widely known, it is hardly true. Moreover, the popularity of a claim or wide acceptance is not evidence per se that the claim is true 2 . To summarize, the RMGs we reviewed (Hevner et al., 2004; Klein & Myers, 1999; Venkatesh et al., 2013) demonstrate the applicability or application of the guidelines. These RMGs provide some evidence that an RMP has been used or demonstrates how an RMP can be used. (Whether the examples demonstrate the usability of the RMGs is not the focus in this paper because we are not evaluating, for instance, the pedagogical usefulness of these examples.) We want to emphasize that these applicability examples do not constitute evidence of cause and effect or good outcomes. Scholars cannot use these examples to claim that these guidelines are appropriate for evaluating what good (interpretive, mixed methods, or design science) research is. 2 In addition, it is not clear to what extent a high number of citations demonstrates, for instance, popularity in the cases of these reviewed guidelines. For example, if there are few guidelines for interpretive research or mixed methods research in the top IS journals, and the reviewers expect adherence to the guidelines, then is the authors’ best bet to cite these guidelines whether they like the guidelines or not? 2.3.3 Potential Sources of Confusion in RMG Writings In this section, we emphasize some potential inconsistencies in RMG writings. Most notably, although these guidelines propose lists of RMPs and evaluation criteria, the authors of the RMGs also warn readers against 1) using the principles automatically, or 2) using the principles as bureaucratic rules, or 3) treating the guidelines as legislative. If reviewers are using these guidelines as normative checklists, do the reviewers simply misunderstand the RMGs? The issue is more complex. In many cases, the RMGs can be read normatively, and there are potential inconsistencies. For example, some recommendations should be revised if they are not intended to be legislative. To start, the design science RMG aims “to inform the community of IS researchers and practitioners of how to conduct, evaluate, and present design-science research…by developing a set of guidelines for conducting and evaluating good design-science research” (Hevner et al., 2004, p. 77). Scholars repeat this message elsewhere: “it is vital that we as a research community provide clear and consistent…guidelines…for the design and execution of high quality design science research projects…to establish the credibility of IS design science research” (Hevner, 2007, p. 87). At the same time, the guideline emphasizes that “guidelines should be addressed in some manner for design science research to be complete” (Hevner et al. 2004 p. 82). If it is vital to have “clear and consistent…guidelines,” then how can we say that they can be “addressed in some manner”? Allowing “some manner” seems to risk having clear and consistent guidelines. These are not the only unclarified issues. The design science RMG also “advised against mandatory or rote use of the guidelines” (ibid, p. 82) and maintains that “[r]esearchers, reviewers, and editors must use their creative skills and judgment to determine when, where, and how to apply each of the guidelines in a specific research project” (Hevner et al., 2004, p. 82). However, this recommendation seems to conflict with the call for “clear and consistent…guidelines.” Finally, how can the guidelines be used “for conducting and evaluating good design-science research (ibid., p. 77) if in every project, scholars have to use their creative skills and judgment regarding when and how to apply each guideline? Readers of the interpretive RMG face similar riddles. Klein and Myers (1999) caution that: Principles are not like bureaucratic rules of conduct, because the application of one or more of them still requires considerable creative thought…[I]t is incumbent upon authors, reviewers, and editors to exercise their judgment and discretion in deciding whether, how, and which of the principles should be applied and appropriated in any given research project. (p. 71) This indicates flexibility in the application of the RMG. However, the interpretive RMG also notes that “while we believe that none of our principles should be left out arbitrarily, researchers need to work out themselves how (and which of) the principles apply in any particular situation” (Klein & Myers, 1999, p. 78). The dilemma is that if “researchers need to work out themselves how (and which of) the principles apply in any particular situation,” then how do we know that they are not “left out arbitrarily,” which was banned? There seems to be one more fundamental conflict. Recall that the set of interpretive principles is a response to how “interpretive field research should be conducted and how its quality can be assessed” (Klein & Myers, 1999, p. 67). How can the RMG constitute an adequate response to how research “should be conducted and how its quality can be assessed” (p. 67) if applying the RMG “require[s] considerable creative thought” (p. 78), and “researchers need to work out themselves how (and which of) the principles apply in any particular situation” (p. 78)? Mixed methods guidelines note that “these guidelines could be seen as legislative,” but this is not their intention (Venkatesh et al., 2013, p. 48). This indicates flexibility in the application of the RMG. However, many RMPs in the mixed methods RMGs are not presented as nonlegislative. For instance, each RMP is accompanied by a message for authors dictating what to do during research, as well as another message for reviewers and editors dictating what to do during evaluation (Table 1). Furthermore, many of these principles cannot be presented as nonlegislative without changing the meaning of the original message. As an example, let’s consider the first guideline of the mixed methods research (Venkatesh et al., 2013): “IS researchers should employ a mixed-methods approach only when they intend to provide a holistic understanding of a phenomenon for which extant research is fragmented, inconclusive, and equivocal” (p. 36). This principle rewritten in a non-legislative tone would sound something like this: “IS research can employ mixed method research whenever they find it offering new or useful information. Therefore those situations where mixed methods may be used cannot and should not be predicted a priori.” If the aim of the RMG is not to be legislative, then many RMPs in the mixed methods guidelines (Venkatesh et al., 2013) should be rewritten to reflect their non-binding nature. 3 RMPs between Classical and Contemporary Thought In this section, we review some of the philosophy of science underlying RMPs. Due to space limitations, this review is incomplete and omits numerous other ideas of the philosophers we discuss. Nonetheless, the views we present are helpful to further reflect the challenges facing the RMPs in interpretive, mixed methods, and design science research in IS. In the first part, we present a brief overview of several classic attempts to establish infallible research methods, and how this task turned out to be difficult. Then, we introduce several contemporary perspectives that view research methods as hypothetical and revisable; a philosophical position known as normative naturalism (Laudan, 1990). We conclude this section by pointing to some of the key challenges of applying this naturalist view to interpretive, design science, and mixed methods IS research. 3.1 A Classical View: Strict Adherence to the Scientific Method 3.1.1 The Scientific Method Gives Us Infallible Knowledge A key historical figure advocating for the infallible scientific method was Aristotle. He believed that critical thinking can separate with absolute certainty scientific knowledge from opinion and superstition (Laudan, 1983). For Aristotle, scientific knowledge was infallible (Oddie, 2016). Later, scientific research cast serious doubts on this view. Often, at any point in time, when we look at the research, older theories often appear naïve or wrong (Laudan, 1981a, 1981b; Niiniluoto, 1999). In other words, the scientific knowledge (the best science at the time) in the past was often fallible and was replaced or corrected by later research (Laudan, 1981a, 1981b; Niiniluoto, 1999). As a result, a historical review of the sciences does not support the conclusion that scientific knowledge has been infallible (Laudan, 1981a, 1981b; Niiniluoto, 1999). Quite the contrary, reviewing the sciences suggests that scientific knowledge has been fallible. Then, scientific knowledge cannot be separated from opinion and superstition with absolute certainty. Philosophers realized that scientific knowledge is not certain and infallible. At the same time, many natural sciences were seemingly highly progressive (Laudan, 1983). This raised the question, why were they successful? Given that Aristotelian infallible critical thinking was not the scientific method that explained the success of science, philosophers such as Comte, Jevons, Helmholtz, and Mach suggested other candidates for the scientific method (Laudan, 1983). However, these philosophers could not agree on what this scientific method was (Laudan, 1983). Even more problematic was that Duhem (1906) showed that successful scientists either did not use or violated the proposed RMPs. As researchers in science continued to make breakthroughs in physics and medicine, interest in understanding this success was high (Laudan, 1983). Vienna Circle logical positivists (e.g., Schlick, Neurath, and Carnap) suggested that the scientific method not only explains the success of science but also can differentiate science from pseudo-science (Siponen & Tsohou, 2018). For example, Schlick (1932) put forward the verification method. Vienna Circle Logical positivists’ ambition to establish an objective or infallible method for separating science from nonsense attracted a lot of criticism, which ultimately clarified that perhaps no method is truly objective or can give infallible knowledge. We discuss some of the major criticism below. 3.1.2 Challenges Facing the Absolute Method The Vienna Circle logical positivism’s infallible method project experienced attacks not only from its opponents but also from the Vienna Circle. That is, criticism of the Vienna Circle logical positivists’ absolute RMPs first emerged as a self-critique by the Vienna Circle members themselves. Neurath, and later Carnap, objected to absolute RMPs (Hempel, 1935). For example, Carnap (1932) notes that no methodological norm provided “objective validity,” because norms cannot “be empirically verified or deduced from empirical propositions; indeed (norms) cannot be affirmed at all” (p. 237). In other words, Carnap deems the acceptance of any RMP a matter of taste (Laudan, 1996). This does not mean that Carnap lacks RMP preferences. Instead, Carnap regards justifying that one RMP is ultimately better than another as impossible (Carnap, 1935). Therefore, for Carnap, RMPs are “proposals, which no one was obligated to accept” (Laudan, 1996, p. 15). Similarly, a father of logical empiricism, Reichenbach (1938) regards the aims of science as “volitional bifurcations.” Laudan’s (1996) interpretation is that for Reichenbach these bifurcations include the choice of fundamental methodological norms, and (per Laudan’s interpretation) Reichenbach admits that they ultimately are “a matter of personal taste and preference” (p. 15). In addition to the self-critique by Vienna Circle logical positivists, louder attacks came from philosophers outside the circle. We briefly discuss these criticisms next. (1) Quine and the argument against verification. Quine’s (1951) second dogma points out that verification (e.g., verification by observation) cannot test a single statement or hypothesis isolated from its underlying assumptions. That is, any test or observation, no matter how simple and obvious it may sound, is always associated with numerous underlying presuppositions that are not empirically testable and must be assumed. Quine (1951) maintains that when a claim is tested, a complex web of assumptions and presuppositions is also tested; thus, he concludes that any hypothesis can be accepted by revising the underlying assumptions. Generally, Quine’s critique applies to any test or RMP. (2) Kuhn and the argument that fundamental method decisions are irrational. One major task for logical empiricists and Popper, around 1930 to 1950, was to articulate the logic of theory acceptance, including generic methodological rules under which scientific theories are or should be rationally accepted, rejected, falsified, confirmed, or disconfirmed (Laudan, 1996). For example, Popper (1934/1959), who called himself a critical rationalist, and not a positivist, proposed the idea that scientific theories are rejected (falsified) if the observation does not match the prediction 3 . The Popperian and logical empiricist program endeavors not only to outline the rationality under which theories are evaluated but also how the theories can be compared rationally (Laudan, 1996). Soon, philosophers of science reported that physicists did not follow the Popperian rationality of rejecting the theory in the case of negative observation (Lakatos, 1970; Laudan, 1978). Furthermore, Kuhn, who regarded himself as “an ex-physicist” (Kuhn, 1959, p. 225), famously shocked the philosophy of science (Hoyningen-Huene, 1992, p. 491) with a number of issues. For instance, he argued that different paradigms in one scientific discipline tend to have radically different methodological norms for assessing theories (Kuhn, 1962). In addition, according to Kuhn (1962), the worldview and languages of each paradigm are so different that 3 “A theory which is not refutable by any conceivable event is nonscientific” (Popper, 1934/1959, p. 347). (2) Concerns about establishing cause-and-effect relationships. The second issue concerns normative naturalism (Laudan, 1987), and its implications for cause and effect. This issue is also important for any IS guideline, which suggests that certain RMPs can be used to evaluate good research. For Laudan, “[i]f one’s goal is y, then one ought to do x” (p. 24), and “all methodological rules…can be re-cast as (a) contingent statement of…[the] connections between means and ends” (p. 25). We argue the following: If RMPs are normative, for example, then they claim to say what is good design science or when mixed methods research can be used. Generally speaking, this claim implicitly assumes cause and outcome (effect) relationships. Are there such causal relationships? Is this even possible? To clarify this point, let us consider the three commonly recognized causal capacities (or types of causation): 1) deterministic, 2) random, and 3) probabilistic (Dupré & Cartwright, 1988). A deterministic causation “is one which, under specifiable circumstances, always produces its effect” (Dupré & Cartwright, 1988). Basically, deterministic causality requires the existence of true, 100% exception-less laws. In psychological and social phenomena, deterministic laws have been difficult to find (Dupré & Cartwright, 1988). It would be highly questionable to claim that the RMPs we reviewed are deterministic, that is, that they cause the effect, namely, the outcome of good research, with no exceptions in every case. This leaves us with two options: The causation is probabilistic or random. Dupré and Cartwright (1988) state: Probabilistic capacity also operates only sometimes, but the strength of the tendency to produce the effect is nomologically fixed…[I]f there are genuinely random capacities, it is obvious that there are capacities that cannot be reduced to quantitative probabilities. (p. 522) None of these guidelines show probabilistic evidence, that, for example, with an 85% likelihood, employing a certain RMP leads to better results than following some other principle. This is not a criticism of these guidelines: We doubt to what extent producing such evidence is possible. Finally, random causality suggests that an action “sometimes produces its effect and sometimes does not, but nature does not determine how often or how regularly it does so” (Dupré & Cartwright, 1988, p. 522). Thus, if there are no deterministic or probabilistic causes between the RMPs and the outcomes because the phenomenon is indeterministic, then no philosophizing or tests can establish it. This does not imply that the interpretive, mixed methods, or design science research is “unscientific.” Instead, it means that we should doubt the idea that certain RMPs produce or guarantee (i.e., cause) good research outcomes. 4 Discussion Nothing that we say in this paper suggests that interpretive, design science, or mixed methods research is not scientific in principle. Furthermore, although we outline many fundamental concerns with the three RMGs, it does not mean that we regard them as useless. However, the RMGs we reviewed are not proposed as “useful sources” for mixed methods, interpretive, or design science research. Their ambition is much higher. They are proposed as normative guidelines for how good or rigorous research should be conducted and evaluated. One of our arguments is that the RMGs we reviewed cannot—and therefore, should not—be used for this specific purpose, which they are advanced, namely, to evaluate the quality of research. Next, we discuss a number of key issues highlighted by our analysis of the three RMGs. 4.1 Requiring Clear and Consistent Guidelines May Hinder New Ideas Hevner (2007) notes that “it is vital that we as a research community provide clear and consistent…guidelines…for the design and execution of high quality design science research projects…to establish the credibility of IS design science research” (p. 87). Having said that, the design science RMG seems to conflict with this idea as this RMG requires subjective case-bycase applications (Hevner et al., 2004, p. 82). The potential problem with the requirement for clear consistent guidelines is that it may force design science research into a uniform format that does not allow for variety. Klein and Myers (1999) note that “the complete literature of interpretive philosophy comprises so many varied philosophical positions that it is unlikely to yield one consistent set of principles for doing interpretive research” (p. 70). A concern is whether clear consistent guidelines prohibit creativity and out-of-the-box thinking, which can hinder scientific innovation. As we note, Kuhn (1977) claims that the emergence of new scientific ideas requires methodological disagreements. Feyerabend (1975) claims that new scientific ideas require breaking established methodological rules. Although not every major scientific discovery may require breaking existing method principles, some or many may require it. 4.2 Inappropriate Judgments and Check-Box Compliance Klein and Myers (1999) note, “it is better to have some principles than none at all, since the absence of any criteria increases the risk that interpretive work will continue to be judged inappropriately” (p. 68). We call this the absence argument. Let us clarify that the title of the Klein and Myers (1999) guideline is for interpretive research, and in the abstract, they hint at providing guidelines for evaluating interpretive research. Then, the guidelines are an improvement of the status quo of interpretive research in IS (p. 68). Later, they note that their principles apply mostly to the hermeneutical type of interpretive research (p. 68). However, if IS reviewers or IS authors then apply Klein and Myers’ (1999) “guideline for interpretive research” to a non-hermeneutical type of interpretive research, not following “hermeneutic philosophers, especially Gadamer and Ricoeur” (p. 70), then, in such cases, “interpretive work will continue to be judged inappropriately.” If IS authors and reviewers apply Klein and Myers’ (1999) RMG for evaluating non-hermeneutical research, then this is not a problem of Klein and Myers’ (1999) RMG. Nevertheless, it is relevant to note the possibility that even the introduction of a perfect RMG for one type of interpretive research can lead to judging other types of interpretive research inappropriately. This is the case if the guideline is only for one type of interpretive research, and/or it is applied to other types of interpretive research. More generally, there is also a possible risk that the RMGs we reviewed pave the way for several inappropriate judgments (using Klein and Myers’ [1999] term). The idea that by reading an article about a set of principles researchers and reviewers can conduct and evaluate (good) research in the area can be misleading, and can lead to check-box compliance. First, these RMGs may create a situation in which reviewers or editors, without any hands-on expertise in interpretive, design science, or mixed methods research, take on the role of gatekeepers by simply using these guidelines as a checklist. Second, if paper acceptance in leading IS journals requires compliance with these principles to meet “methodological rigor” (see Rowe & Markus in Hovorka, 2019, p. 1362; Straub, 2008), then what is required for publication is not expertise in interpretive or design science research, but instead, how to demonstrate compliance with the RMGs. If this is true, then it may lead to a situation in which the required methodological principle is complying with the guidelines and not the mastery of, say, interpretive research. Consequently, we may have IS scholars whose primary “training” for interpretive research, mixed methods research, or design science research is reading the RMGs, with a focus on strict compliance to maximize paper acceptance. We are afraid that this may already be the case to some extent, which Rowe and Markus in Hovorka (2019, p. 1362), Straub (2008, p. ix), and Grover and Lyytinen (2015, p. 275) actually imply. Finally, what happens to creative and unexpected research and any settings or circumstances that do not fit the guidelines? For instance, the mixed methods guideline outlines “seven purposes for mixed methods research that [the authors] adapted from prior research” (Venkatesh et al., 2013, p. 5), indicating that only these seven purposes are possible. Moreover, “if researchers fail to provide and explain meta-inferences, the very objective of conducting a mixed methods research study is not achieved” (p. 19). Or consider, “IS researchers should employ a mixed-methods approach only when they intend to provide a holistic understanding of a phenomenon for which extant research is fragmented, inconclusive, and equivocal” (p. 36). These are examples of legislative claims that may limit the use of mixed methods research. Moreover, there could be a risk that noncompliance with the guidelines is viewed as a “lack of methodological rigor” or a “flaw,” thus leading to rejection. If the reviewers’ “challenge is to find the fatal flaws” (Saunders, 2005, p. iii), then we are afraid that a good source is to find a setting or circumstances where one cannot have perfect compliance with a guideline (as mentioned, many sources suggest that this is the case in IS; Grover & Lyytinen, 2015, p. 275; Rowe & Markus in Hovorka, 2019, p. 1362; Straub, 2008, p. ix). It could be that many important cases and settings are those complex and dynamic cases that do not match the established guidelines. 4.3 Do the RMGs meet the standards they impose? The RMGs reviewed propose “validation” or “evaluation” guidelines and ask authors to validate their research with different “rigorous” tests. At the same time, readers can hardly find such “rigorous” evidence supporting the RMPs in these articles. This lack of rigorous evidence makes readers wonder whether the RMG articles meet the standards they set for empirical research. The short answer is no. For example, the design science RMG suggests comparisons with rival approaches, but the authors do not consider doing this for their guidelines (Hevner et al., 2004). To give another example, the mixed methods RMG asks for a rigorous test for mixed methods research. However, they do not provide evidence that, for instance, the following normative recommendation meets any rigorous test: “IS researchers should employ a mixed methods approach only when they intend to provide a holistic understanding of a phenomenon for which extant research is fragmented, inconclusive, and equivocal” (Venkatesh et al., 2013, p. 36). The interpretive guideline reports that their RMPs do not meet their own interpretive guidelines, because their study is conceptual (Klein & Myers, 1999, p. 70). We disagree that evaluating good research outcomes is purely a conceptual problem. The key reason is the following. A conceptual problem is primarily a logical or mathematical riddle. We argue that if an RMP claims it should be used to evaluate what good research is, then this is a claim for cause and outcome. This question is also “empirical” and not only a matter of logic: Can a certain RMP promote good research outcomes better than some other RMPs? 4.4 The “Authority” and “Consistent with” Arguments The RMGs we reviewed justify the individual principles or the whole RMG with an authority argument (such as “in our opinion” or “based on our insights”) or “consistency with” arguments. An authority argument is, for example, when one declares an opinion without presenting the available evidence to justify it. Readers may wonder whether the RMPs, especially when they are used to regulate research, are too important to be a matter of authority or opinion. If RMGs contain rules or principles regulating what a “rigorous” or “good” practice is, and reviewers may require authors to follow them, then should such principles not only be testable claims of how the RMPs are empirically successful in achieving the specific goals? Should they also present evidence for (and against) each RMP? The “consistent with” argument is a questionable approach to justification when the underlying evidence is not presented. 4.5 Should we regard the RMGs and RMPs as mere idealizations? As noted, all the RMGs analyzed were introduced to increase publication opportunities in alternative genres. However, by doing this important task, all the RMGs ended up doing much more, as they were also introduced to evaluate the quality of research. It is not clear why increasing publication opportunities requires an evaluation guideline. One reason might be scientific legitimacy of a young discipline such as IS. For example, in the first ICIS, Keen (1980) worries that the legitimacy of IS, and one reason bleak was “that we have no clear criteria for evaluating our research” (ibid, p. 10). Later, Straub, Nanyang, and Evaristo (1994) argue that a lack of explicit evaluation criteria is responsible for the nearly 90% manuscript rejection rate in top IS journals. The philosophy of science, however, suggests that the issue of evaluation criteria is not that straightforward. Kuhn claims that the decisions about which RMPs are used by a research community are irrational, so that a shift in such norms is comparable to a religious “conversion experience” (Kuhn, 1962). Polanyi (1946) denies the whole business of proposing a set of principles and claims that research method competence is tacit. Laudan’s (1986, 1996) approach subjects RMPs to scientific study. According to this view (Laudan, 1986), research on “methodology is the study of how to conduct inquiry effectively” (p. 349). According to Laudan (1987), the key question for scholars examining research methodologies is understanding that “methodology rules are…statements about instrumentalities, about effective means for realizing cherished ends” (p. 24). Given that RMGs are similar to other theories or sets of propositions in science, RMPs are testable to the same extent as any other theories, hypotheses, and propositions in science (Laudan, 1986). We do not find such tests reported in the RMGs we reviewed. Ultimately, if we cannot show any evidence that these principles lead to better outcomes than their competitors, then why should we require them? We suggest that if RMPs or RMGs cannot offer evidence for their claims, then they should have a less normative effect on our journal review process. We propose the RMGs should be regarded as idealizations. This does not mean that RMGs are useless; instead, they can have a pedagogical function, for example, in PhD training. However, if RMGs are idealizations, and they, for example, simplify how research methods are really used, or sometimes require things that are irrelevant, unnecessary, or harmful in some specific situations, they cannot be used to assess how good, rigorous, or highquality research is conducted or evaluated. 4.6 Future Work We focused on only three different guidelines. As mentioned, Klein and Myers (1999) call for “a lively debate” (p. 68). To facilitate this debate, and to further increase our understanding, future research is needed to examine other RMGs. We do not want to restrict the debate or future research, but we offer some possible directions. Future work could focus on finding to what extent the other guidelines in different interpretive, mixed methods, and design science guidelines use the reported legitimations strategy (see Section 2.3). Such studies should also examine to what extent other RMGs also claim to offer evaluation criteria for good, rigorous, or high-quality studies. Finally, such studies could examine on what evidence the guidelines for good research are based. Future critical studies on methods can also focus on guidelines for quantitative (statistical) research from the philosophical viewpoint. Let us use statistical significance as a concrete example. Winter (2008) argues that for “behavioural IS research, statistical significance is established as a clear and common measure of its results’ rigour” (p. 470). This claim is questionable. For example, the American Statistical Association (ASA) announced this warning for authors who employ statistical significance tests uncritically: “Statistical significance is not equivalent to scientific, human, or economic significance” (Wasserstein & Lazar, 2016). In medicine, “[t]he non-equivalence of statistical significance and clinical importance has long been recognised” (Altman & Bland, 1995, p. 311), and “statistical significance at any level does not prove medical, scientific, or commercial importance” (Ziliak, 2010, p. 325). Moreover, future work could also include a conversation on how we can assess research quality without normative RMGs. Finally, future work could examine how we can empirically identify RMPs that are based on compelling evidence for the RMP and outcomes. 5 Conclusions IS scholars have put forward RMGs for conducting and evaluating interpretive, design science, and mixed methods research. Although these RMGs require rigorous testing and validation, they do not seem to meet these requirements. The evidence for the guidelines consists of the authors’ opinions and claims that the principles are consistent with some previously published research or views. The guidelines give examples of the applicability of the principles, showing how scholars can apply the principles or how a study has applied them. Neither focus counts as evidence for conducting and evaluating good or high-quality research. If scholars use the method guidelines for evaluation (as intended by their authors), then such RMGs should not be based on the authority argument, usability, or a demonstration that the principle is consistent with some papers. We contend that the RMGs we reviewed do not contain evidence for assessing good research outcomes. If this argument is accepted, then the guidelines should be rejected for the purpose of evaluating the research quality. Moreover, there could be a risk that the RMGs may contribute to check-box compliance, where research not meeting them is regarded as lacking methodological rigor. RMGs that mandate in which circumstance to use the method may also limit research and new idea development. For example, why we should limit a priori in which circumstances mixed methods can and cannot be used? It is also important to ask why IS journals need to present method guidelines when there are many research method journals 7 . What value do the IS guidelines add? There is also a risk that they are trivializing the complexity of research and downplay tacit knowledge. Finally, the philosophy of science regarding the scientific method offers important lessons for IS. One option is regarding each RMP as a tentative and revisable principle that enjoys evidence for or against. This view means that RMGs are similar to hypotheses in science and have limited generalizability. This option, however, seems to be difficult to achieve in the IS context of mixed, design science, and interpretive methods. Alternatively, the Kuhnian “normal science” regards 7 For example, Ethnography, Organizational Research Methods, Psychological Methods, Qualitative Research, Qualitative Research journal, Action Research, International Journal of Action Research, International Journal of Qualitative Methods, Journal of Organizational Ethnography, Journal of Phenomenological Psychology, and Qualitative and Multi-Method Research. RMGs as dogmatic and irrational conventions. We propose deeming RMGs idealizations, which can be useful for pedagogical purposes for novices. If our proposal to regard RMGs as idealizations is accepted, then we should reject the guidelines for the specific research quality evaluation purpose for which they are proposed. The RMGs we reviewed are introduced to assess how good, rigorous, or high-quality research is conducted or evaluated. Idealized guidelines cannot be used to assess these outcomes. This does not mean that interpretive, design science, or mixed methods research is nonscientific or the IS guidelines are useless, but instead, that RMGs should have less weight in evaluating what is an acceptable use of a method. Acknowledgements We thank the senior editor and reviewers of Journal of the Association for Information Systems for their helpful comments on the early versions of this paper. References Altman, D., & Bland, J. (1995). Statistics notes: Absence of evidence is not evidence of absence. BMJ (Formerly the British Medical Journal), 19(311(7003):485). Bechtel, W. (2009). Generalization and discovery by assuming conserved mechanisms: Cross‐species research on circadian oscillators. Philosophy of Science, 76, 762–773. Carnap, R. (1932). Ueberwindung der Metaphysik durch logische Analyse der Sprache. Erkenntnis 2. Carnap, R. (1935). Philosophy and Logical Syntax. Carpenter, W. (1885). A Hundred Proofs the Earth Is Not a Globe. Retrieved from https://www.gutenberg.org/files/55387/55387-h/55387-h.htm Diesing, P. (1991). How Does Social Science Work? Reflections on Practice. University of Pittsburgh Press. Doppelt, G. (1990). The Naturalist Conception of Methodological Standards in Science: A Critique. Philosophy of Science, 57(1), 1–19. Duhem, P. (1906). The Aim and Structure of Physical Theory (Translated by P. Wiener).