A model-based approach for specifying changes in replications of empirical studies in computer Science

Computing (2023) 105:1189–1213 https://doi.org/10.1007/s00607-022-01133-x REGULAR PAPER A model-based approach for specifying changes in replications of empirical studies in computer Science Margarita Cruz1,2 ·Beatriz Bernárdez1,2 ·Amador Durán1,2 · Cathy Guevara-Vega3·Antonio Ruiz-Cortés1,2 Received: 30 July 2022 / Accepted: 15 November 2022 / Published online: 3 December 2022 © The Author(s) 2022 Abstract The need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also replicability itself. Unfortunately, the lack of proposals for systematic reporting of changes in replications undermines these desirable objectives. The main goal of the work presented in this article is to provide researchers in Computer Science with a systematic tool-supported approach for the specification and reporting of changes in the replications of their empirical studies. Applying Design Science Research, we have developed and validated a composite artifact consisting of (i) a metamodel that formalizes all the relevant concepts related to replications and their changes; (ii) templates and linguistic patterns that facilitate their reporting; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study that includes 9 families of empirical studies not only from Computer Science, but also from an area as different as Agrobiology , to check the external validity of our approach. The 9 families encompass 23 replication studies and a total of 92 replica- BMargarita Cruz [email protected] Beatriz Bernárdez [email protected] Amador Durán [email protected] Cathy Guevara-Vega cgue[email protected] Antonio Ruiz-Cortés [email protected] 1I3US Research Institute, Universidad de Sevilla, Sevilla, Spain 2SCORE Lab, Universidad de Sevilla, Sevilla, Spain 3eCIER Research Group, Universidad Técnica del Norte, Ibarra, Ecuador 1190 M. Cruz et al. tion changes, for which we have analyzed the suitability of our proposal. The multiple case study revealed some initial limitations of our approach related to threats to experimental validity and context variables. After several improvement iterations on the artifact, all of the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity and their corresponding visualization. Our proposal for the specification of replication changes seems to fit the needs not only of replications in Computer Science, but also in other research areas. Nevertheless, further research is needed to improve it and disseminate its use among the research community. Keywords Replication changes ·Templates ·Linguistic patterns ·Families of experiments ·Threats to validity Mathematics Subject Classification 68-02 1 Introduction As in most research areas, empirical studies, especially controlled experiments, can be used in Computer Science to rigorously evaluate technologies, methods, and tools and help guide further research by revealing existing problems and difficulties [17]. However, for their results to be generalizable, reported experiments must be replicated in different contexts, at different times, and under different conditions [15] by means of so-called families of experiments.1As Basili et al introduced in 1999 [8], a family of experiments consists of abaseline or original study, followed up by a set of replications that answer the same research questions as the original study. Later, Santos et al in 2018 [58], proposed the following premises to consider a series of experiments as a family, namely: (i) access to the raw data is guaranteed; (ii) researchers know the exact setup of each experiment; and (iii) at least three experiments evaluate the effects of at least two different technologies (or methods or tools) on the same response variable. Nowadays, it is widely accepted in the research community that the knowledge obtained from a family of experiments is more robust and reliable than that obtained from a single isolated experiment, the results of which can only be considered as preliminary [65, 69]. Let us consider for a moment a novice researcher in Computer Science who decides to replicate an original study or a previous replication of an original study. According to Carver [16], she needs to carefully review the entire family of previous studies, in order to acquire the necessary knowledge to properly adapt the experimental settings, improve its design—if possible—to increase its experimental validity, or avoid making the same mistakes than in previous studies. To alleviate this situation, several initiatives have recently emerged, the most relevant being Open Science [43], which promulgates 1 The concept of family of experiments can be understood as an specialization of the more general concept of family of empirical studies. Since experiments are the most common type of empirical study in Computer Science and its related disciplines, in this article we will use mainly the term family of experiments,but most of the proposal is also applicable to other types of replicable empirical studies such as, for example, case studies or surveys. A model-based approach for specifying changes… 1191 thatmakingavailablethe datasets,theanalyses, andapreprint versionofanexperiment and its related software, provides valuable knowledge allowing not only that any interested party may audit it, but also that others build directly upon the previous work through reuse and repurposing [5]. A complementary approach to increase the visibility and reproducibility of empirical studies in Computer Science is promoting the availability of artifacts such as laboratory or replication packages [8,67]. According to [65], replication packages should include not only datasets, analyses, and experimental material, but also guidelines to conduct a replication and a summary of the evolution of the experiment across the family [68], promoting traceability among replications. Despiteall theseefforts,Shepperdet al.[63] statedthat althoughthere isa consensus onreplications beingessential toconsolidate thefindings of empiricalstudies, thereis a need for better reporting of both original and replicated studies. To report a replication of a controlled experiment, Computer Science researchers usually use as a reference some seminal works such as those by Wohlin et al. [73], Jedlitschka et al. [33], or Juristo and Moreno [35], complementing them with Carver’s replications guidelines [16]. The recommendations in the aforementioned works are meant for controlled experiments in general, but not for the specification of the changes that often arise during replications to address threats to experimental validity and, therefore, improve the original study and the validity of its results [14]. To the best of our knowledge, there is a lack of specific proposals to specify the changes introduced between replications of the same family of experiments. In this situation, researchers choose either to report the experimental setting of the replication without highlighting the changes from the original study or previous replications [24,31,44,59], or they report the changes in an ad-hoc manner, describing them in narrative text [1,4,50,51]. This lack of detail in the specification of changes leads to some problems in carrying out new replications, since the replicator has difficulties not only in deciding which aspects of the experimental setup are best suited to the new environment, but also in avoiding mistakes made in previous experiments that threatened their validity [5,16]. On the other hand, a proper knowledge of the changes allows a better meta–analysis of the family, since they impact on issues such as the definition of moderator variables, the definition of the aggregated family design, or the choice of the most appropriate analysis, among others [10]. The proposal described in this article aims to specify replication changes in a structured, systematic way, identifying aspects such as the rationale for each change or its effect on experimental validity, which can be quantified and used to visualize the evolution of the entire family of experiments, thus supporting decision making for new replications [16]. By specifying changes explicitly, our proposal helps to decrease not only the so-called tacit knowledge [64], but also experimenter bias [60], that occurs when replicators have to interact with the original experimenters to request missing information about the family, specifically their changes. In this work, we focus on the specifications of replications and their corresponding changes of empirical studies in general and controlled experiments in particular. For that purpose, we have adopted Design Science Research (DSR ) as our research methodology, creating and evaluating an artifact that is designed to solve an identified problem [71]. In our case, the developed artifact consists of (i) a metamodel devel- 1192 M. Cruz et al. oped by an iterative and incremental process that represents the relevant concepts about replications and their changes; (ii) templates for reporting the information included in the metamodel, thus promoting reusability, avoiding redundancies, and tracing the effects of changes on experimental validity across the family of experiments; and, (iii) CÆSAR, a proof-of-concept model-driven software tool developed to provide an initial evaluation and support for our proposal, including the aforementioned templates and the automatic visualization of the effect of the changes on experimental validity across the family of experiments. The rest of the paper is organized as follows. Section 2 summarizes a brief state- of-the-art on families of experiments, replications, and changes in Computer Science. Section 3 presents the three components of the developed artifact, i.e. the metamodel, the templates, and the tool support. Section 4 reports a multiple case study carried out to evaluate the artifact, encompassing 9 families of experiments from Computer Science and Agrobiology . Related work is commented in Sect. 5 and, finally, concluding remarks and future work are presented in Sect. 6. 2 Background According to [2], a replication can be defined as the repetition of an experiment, either as closely following the original experiment as possible, or with deliberate changes to the original experiment’s settings, in order to achieve, or ensure, greater validity in the carried out research. Despite the importance of replications, and although their practice has increased in recent years [18, 20], the number of replications in Computer Science in general, and in Software Engineeringin particular, remains low [68]. Among the causes influencing this situation are (i) the tacit knowledge not explicit in replication reports [64]; (ii) the lack or incompleteness of laboratory packages [68]; (iii) the lack of agreement on common terminology and criteria for reporting replications [16]; and, last but not least, (iv) the effort and resources needed to carry out an experiment [20]. Several taxonomies have been proposed to classify the different types of replications. There is a wide agreement to use the term internal replication for those replications carried out by the same experimenters at the same site than the original study, whereas external replication is used when the experimental team and site are different from the original ones. With respect to the process to carry out a family of experiments, once the original experiment concludes, it is advisable to carry out internal replications to confirm preliminary results and adjust experimental settings. Then, external replications can be carried out for generalizing the provisional results from internal replications [13]. Nevertheless, there is a lack of agreement in the used terminology in other replication taxonomies. For example, according to the degree to which the original experiment protocol is followed, Shull et al. [65] classify replications as exact and conceptual,but Juristo and Vegas [38]use closed and differentiated instead. Basili et al refer to strict replications when the original study is duplicated as accurately as possible [8], whereas Gómez et al. [28], classify them as literal, operational and conceptual depending on the changes carried out and their purpose. Other taxonomies such as those proposed in [2] and [6] use some of the terms commented above. In [42], de Magalhães et al A model-based approach for specifying changes… 1193 compare different taxonomies, concluding that any attempt to establish a replication typology must be done with care since, as stated in [27], there are authors who use the same term for different types of replications and conversely, use different terms to refer to the same type of replication. In our opinion, the main underlying concept behind the different proposed taxonomies are the changes between the original study and its replications. In particular, in human-oriented experiments such as those common in Software Engineering, each change in the experimental settings, even if the protocol of the original experiment is followed, can eventually produce different results [23]. As a consequence, specifying the changes and their motivation allows the comparison of results and increase knowledge by analyzing the conditions under which the results were obtained, thus encouraging further replication. Last but not less important, and although several authors [16,63] have reported their relevance, only a few replications report their related changes when published. When included, changes are reported in narrative text, e.g. [1,32,61], or in a simple, non-standard tabular form, with one row per change including a few properties only, such as elements affected by the change (population, experimental design, etc.) or the situation after the change, e.g. [4,50,51]. 3 Model-based proposal for specifying replication changes In this section, the model-based proposal for the specification of replication changes in empiricalstudies, encompassing ametamodel, sometemplates, anda proof-of-concept software tool, is presented. 3.1 Metamodel for replication changes Figure 1shows the proposed metamodel for replication changes using a UML class diagram. As can be seen, the classes at the top model the necessary context information for understanding replication changes, whereas the classes at the bottom model the structure and properties of the changes themselves. The enumerations on the right of the diagram correspond to the categorical domains of some class attributes. Withrespecttothereplicationcontext,twotypesofempiricalstudiesareconsidered, original studies and replications. Both of them are identified by an acronym,take place in a given site at a given date, and usually have a report that can be accessed using a URL. In the case of original studies, a specification of their goal—that should include at least the cause and effect constructs and the population under study, for example using a well-known template such as GQM [7]—and their description are also included, since they are supposed to be shared by its replications, thus providing a context which is needed for understanding related replications and their changes. For replications, their type (internal,external), their purposes (confirm results,generalize results,orovercome limitations), and their changes are recorded together with their base study, i.e. the study they replicate, which can be an original study or a previous replication, as modeled by the abstract superclass EmpiricalStudy. 1194 M. Cruz et al. Fig. 1 Metamodel for replication changes (UML class diagram) Regarding replication changes, they are identified by a descriptive name and must describe narratively what changes from the base study (base_situation) to the replication (replication_situation). The purpose of the changes must also be recorded, as well as their impacts (if any) on experimental validity (modeled by the ChangeImpact class), following the validity taxonomy described by Wohlin et al in [73] and modeled by the Validity enumeration. As commented by Navarro [46]in[46], construct validity is basically a question of whether the treatment, metrics, and measurement instruments and procedures actually reflect the cause and effect constructs. Internal validity is concerned with the effect being caused only by the treatment and not by other confounding factors, which must be controlled in case they are identified. External validity relates to the generalizability of the findings, i.e., to what extent are the results to be seen in “real life” as they are in the study. Finally, conclusion validity is concerned with the choice of statistical analyses to draw the correct conclusions about the relationships between the treatment and the measures. For each impact of a change, its rationale, i.e. why the change affects a given validity type, must be recorded together with its effect, using a 7-point linear scale modeled by the Effect enumeration. Note that associating a value in an linear scale from substantially (-3), moderately (-2), and slightly (-1) decreases to their positive counterparts, including a does not affect (0) central point, allows to have an idea on how the changes in the different experiments in a family increase or decrease original A model-based approach for specifying changes… 1195 study’s experimental validity and it can be easily visualized graphically, as it will be shown in Sect. 3.3. 3.1.1 Change dimensions For classifying the different types of changes in replications, we have followed and expanded the dimensions for experimental configuration proposed by Gómez et al in [28], resulting in the classification described below. Operationalization change In a controlled experiment, the cause construct is operationalized as one or more treatments, whereas the effect construct is operationalized as one or more metrics, which are measured following measurement procedures.Any change related to any of the above items (modeled as the OperationalizationElement enumeration), e.g. changing the duration of a treatment, should be an instance of the OperationalizationChange class. Population change This class encompasses any change related to the experimental subjects or objects, e.g. changing the experience level or the average age of the experimental subjects with respect to those in the base study. Protocol change As defined by Juristo and Gómez in [34], the experimental protocol is the setup of the experimental design,experimental material,experimental guides, measuring instruments, and data analysis techniques, which are the elements modeled by the ProtocolElement enumeration in our metamodel. Any change affecting one of those elements, such as changing the tasks that subjects have to perform or using Bayesian instead of frequentist statistics, should be an instance of the ProtocolChange class. Experimenter change These are changes related to the experimenters and their roles in the replication when compared to the base study. As modeled by the Experimenter- Role enumeration, the roles considered are designer,trainer,monitor,measurer and analyst. Context change This class, not present in [28], models any change related to the context in which the replication is carried out compared to the context in which the base study took place. For example, in human-oriented experiments using students as subjects, changing the time of year when the study is conducted from before final exams to after final exams is a change of this kind. In technology-oriented experiments, moving from running the software on a real machine to running it on a virtual machine would also be a context change. 3.2 Templates for replications changes Templates havebeensuccessfullyapplied inComputerScience,SoftwareEngineering, and related areas. For example, the well-known GQM template proposed in 1994 by Basili et al. [7], the Wieringa’s proposal for the specification of the problem under study in DSR , and many others, including templates for requirements [21,22], process performance indicators [52,53], or metamorphic testing relations [62]. 1196 M. Cruz et al. Fig. 2 Template for replication context information Fig. 3 Template for replication changes (one for each change in the replication) Templates help visualizing information in a standard form, which can be easily adopted by practitioners, especially by novice researchers. In order to improve its usability, we have augmented templates with linguistic patterns (L-patterns) when possible, in a similar manner than in [21, 22, 52, 53]. L-patterns are pre–written, parametrized sentences that can be used for filling in some template fields in a easier, standard way. Driven by the metamodel in Fig. 1, our proposal includes two templates, A model-based approach for specifying changes… 1197 Fig. 4 Example of context template in CÆSAR for the replication reported in [11] Fig. 5 Example of change template in CÆSAR for one of the changes in [11] one for the replication context shown in Fig. 2, and another for replication changes shown in Fig. 3. In the notation used in the templates in Figs. 2and 3, placeholders are depicted between angle brackets (<…>), single options are enclosed by curly brackets and separated by vertical bars ({…|…}), multiple options are indicated by a plus symbol after the closing bracket ({…|…}+), and elements in square brackets ([…]) are considered as optional. 3.3 Proof-of-concept software tool: CÆSAR Applying a model-driven development approach, we have built a proof-of-concept tool called CÆSAR (ChAngE SpecificAtion for Replications) using the Grails framework 1204 M. Cruz et al. Table 1 continued Case study Family of exp. #Repl #Chng Originally reported as Artifact evolution Additional findings TOE Testing 3 3 Narrative No relevant evolution of the artifact as a consequence of This type of experiments could be properly specified using an evolved version of the metamodel and templates, although the specification of change dimensions was not always obvious due to differences with HOEstudies. this case study. SPL 1 2 Narrative Total 2 4 5 Agrobio Soil 2 16 Narrative A clear need of contextual information related to the original study of a family of experiments is detected. This results in new attributes in the OriginalStudy superclass and it is the motivation for the quantification of effects on validity and its visualization in the CÆSAR tool in further iterations. Without contextual information, replications are almost impossible to understand. Specification of change dimensions was not obvious. Some differences in experimental terminology were detected. Positive feedback with respect to the the CÆSAR tool. Harvest 1 1 Narrative Olive 1 11 Tabular Diet 1 1 Narrative Total 4 5 29 Total 92392 A model-based approach for specifying changes… 1205 Fig. 9 Metamodel evolution for modeling replications based on replications Fig. 10 Metamodel evolution for modeling more than one validity effect per change visualize the evolution of the experimental validity across a family of experiments (see Figs. 6and 7), supporting decisions in further replications. It is worth mentioning that in most of the reported replications the purposes of the changes were rarely stated, making it difficult to identify their effects on validity and the modified dimensions. 4.2.2 TOE–case analysis The main goal of this case study was to confirm the results of the previous one and to consolidate the evolutive changes on the artifact. Using the evolved templates, we were able to specify all the replications changes, although it took us some time to agree on the dimensions modified by some changes. Overall, considering that most of the limitations of the initial artifact had been identified in the first case study, the participating researchers found our approach satisfactory, providing very positive feedback. 4.2.3 Agrobio–case analysis After validating our approach for the specification of changes in Computer Science experiments, the main goal of the third case study was to test it in a different research area with users other than ourselves. One of the first issues identified during this case study was the need of a context for replication changes to be understood due to their high specificity. Clearly, that context had to be provided by the original study since 1206 M. Cruz et al. Fig. 11 Specification of the Soil–2018 replication [45]intheCÆSAR tool it was shared by all its replications. As shown in Fig. 9, in the early version of the metamodel, only the goal of the original study was registered whereas in the evolved metamodel, a description is added to provide such a context. See the description of the original study of the Soil–2018 replication in Fig. 11 for a clear example. In families of experiments in Agrobiology , it is a common practice to change the growing medium of the plants from Petri dishes to culture chambers, to greenhouses, and finally, to natural soils. It is also common to repeat the same experiment in different seasons to confirm results. These kinds of change, that we were not able to classify in any of the change dimensions by Gómez et al. [28], were the reason to expand their proposal with the context2dimension. Including this new change dimension, we were also able to classify some changes in the HOE–Case in which the experiments were held with students at different moments of the academic course, e.g., at the beginning of the course, during examination period, and that we had not been able to completely specify before. 2 The name was decided because this kind of change usually affects context variables in experimental design. An alternative name, environment dimension, was also discussed among us but was finally discarded in favor of context. A model-based approach for specifying changes… 1207 Regarding the user experience of the participating researchers, they found the templates and L-patterns very useful for reporting their replications, helping them to focus on the essential information. Those who agreed to use CÆSAR, despite being a proof- of-concept tool, found it very useful for generating reports of their experiments and showed a great interest to use it in the future. Nevertheless, some differences in the terminology used in the templates were detected. Particularly, the concept of change dimension was foreign to them, something that it seemed reasonable considering that it is a recent concept proposed by an Computer Science researcher [28]. Another foreign concepts to them were those of experimental validity and validity threats, which was very surprising to us considering the importance they are given in our discipline. After several meetings, we found out that they handle a list of usual experimental risks with their corresponding actions to mitigate them, but they use the term avoiding experimental errors instead of minimizing validity threats as we do in Computer Science. On the other hand, they preferred the term repetition to replication. 4.3 Threats to validity Runeson et al. [56] provide a detailed description of threats to the validity of case studies, which are a specific version of those enumerated in [73] but adapted to the special characteristics of case studies. In this section, such threats and the actions performed to mitigate them in our multiple case study are described. 4.3.1 Construct validity This validity is concerned with the degree to which the operationalization actually reflects what is to be the investigated. According to [74], using multiple sources of evidence during data collection increases construct validity, as it has been our case. On the other hand, it is also recommended in [74] that a draft of the case study report be reviewed by key informants. In our case, the participant researches have not only followed the evolution of the artifact across the whole evaluation process, but they have also reviewed a draft report of our multiple case study. 4.3.2 Internal validity This validity is concerned when causal relations are examined. The main hypothesis in our study is that the use of the proposed artifact is useful for the specification and reporting of replications, not only in Computer Science, but also in other areas of knowledge. The only potential threat to the internal validity is that the differences in the used terminology could affect the usage of the artifact by participant researchers, especially for those from research areas other than Computer Science. To mitigate this threat, we explained the use of the templates and assisted the Agrobiology researchers during the Agrobio–Case, apart from investigating the differences and similarities between their terminology and that used in Computer Science, in order to improve our communication with them. 1208 M. Cruz et al. 4.3.3 External validity This validity is concerned with the generalization of results in cases that have common characteristics. The fact that the templates were firstly used by ourselves could limit the generalization of the findings. In order to overcome this threat, we invited other researchers from our own research area and from other research areas to participate in the study, thus reducing the potential bias. Nevertheless, further research is needed to validate the artifact in more replications, belonging to the Computer Science discipline or to other different disciplines. In this sense, the CÆSAR tool is available to the scientific community, so other researchers can report their replication changes using it and provide feedback for further improvements. 4.3.4 Reliability This aspect is concerned with obtaining similar results when the study is carried out by other researchers. Since the specification of all replications with their corresponding changes is available at Zenodo [19], the study can be repeated by other researchers which can then compare their results with ours. 5 Related work Regarding replications in Computer Science, several literature reviews show the relevance of the topic [12, 18, 20, 42]. These reviews are systematic mapping studies dealing with general aspects such as types of replications, conceptual frameworks, or addressed research topics. However, to the best of our knowledge, there is a lack of literature reviews on more specific aspects of replications such as the reporting of changes. Despite of the recommendations by Carver [16], [63] Shepperd et al, Juristo and Vegas [37, 38], Gómez et al. [28], and Baldassarre et al. [6] to report replication changes, no concrete proposals have been provided on how to specify them in Computer Science. The works that are most related to ours are [2], [66], and [70], who propose a tabular form to summarize the experiments that compose a family. In [2], replications are reported including their motivation, their changes (in unstructured narrative text), the confirmation or non-confirmation of results in previous experiments, other characteristics such as subjects, tasks, and materials, and whether hypotheses or research questions changed from previous experiments. In [66], a tabular replication summary is used, including information on, among others, experimental design, laboratory package, material preparation, replication operation, data analysis, and experimenter evaluation. Finally, in [70], the specification of each experiment is summarized including elements such as factors, treatments, response variables, design, experimental objects, and participants. After reviewing these proposals, we consider that all of them provide an in-depth overview of the configuration of each experiment in a family. However, to identify replication changes, table content corresponding to the different replications must be compared, whereas in our proposal, apart from being model-based and having tool A model-based approach for specifying changes… 1209 support, each change is specified separately in a explicit manner, thus reducing tacit knowledge as recommended in [41]. In a recent survey closely related to the approach presented in this article, 137 articles published between 2013 and 2018 reporting at least one replication were analyzed [18]. Most of the replication changes were defined in controlled experiments, confirming that this is the most frequently applied empirical strategy in the research area [29]. In general, most changes—also referred to as adjustments or differences—were described with respect to an original experiment using natural language or tabular forms, only about 25–30% of the studies reported the purpose of the changes and their effect on experimental validity. 6 Conclusions and future work In the context of replications of empirical studies in Computer Science, and applying DSR , we have developed and evaluated a composite artifact to systematically specify andreportreplicationchanges.We havedevelopedtheartifactfollowingamodel-based approach, generating (i) a metamodel formalizing all the relevant concepts related to replication changes; (ii) templates for reporting replication contexts and replication changes using the information in the metamodel together with L-patterns to facilitate their use; and (iii) a proof-of-concept tool based on the metamodel and supporting the managementof theproposedtemplates. Wehavealso evaluatedtheartifactbymeans of a multiple case study in which 92 replications changes corresponding to 23 replication studies in 9 families of experiments in the areas of Computer Science and Agrobiology were specified by ourselves and by other researchers and subsequently analyzed. The evaluation revealed some initial limitations of our approach, but they were overcome after several improvement iterations. One of the most relevant improvements was the quantitative determination of the effect of changes on experimental validity, which allows the visualization of the evolution of the validity of a family of experiments thus supporting decision-making. Our model-based proposal for the specification of replication changes seems to fit the needs not only of Computer Science, but also of other research areas such as Agrobiology , obtaining very positive feedback from the participant researchers from both disciplines. As future work, our aim in the middle term is to apply our templates not only to report changes of already conducted replications, but to use them during replication design, as a means of analyzing and documenting the purpose and effects of replication changes before they are performed. This design-oriented approach needs a more advanced version of the CÆSAR tool, including a virtual assistant, probably a chatbot, that guides the researcher in the process and suggests, for example, potential effects of changes on experimental validity depending on the modified dimension or on other criteria that might be identified from experience. In the short term, we want to provide Computer Science researchers with a consolidated L ATEX package for using the proposed templates in their articles when they need to report some replication changes, thus disseminating our artifact in the community. 1210 M. Cruz et al. Acknowledgements We want to thank Dr. Carmen Florido, Dr. Aránzazu García, Dr. Rocío Abia, and Dr. Eddy Plasquy for their contribution to the multiple case study by specifying their experiments and for their kind willingness to provide us with all the information required. Funding Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work has been partially supported by grants RTI2018-101204-B-C21, RTI2018-101204-B-C22, PID2021- 126227NB-C21, PID2021-126227NB-C22 funded by MCIN/AEI/10.13039/501100011033 and “ERDF a way of making Europe”; grants PYC20 RE 084 US, P18-FR-2895, US-1264651, US-1381595 funded by Junta de Andalucía/ERDF,UE; and FPU19/00666 funded by MCIN/AEI/10.13039/501100011033 and by “ESF Investing in your future”. Open Access This articleislicensedunderaCreative CommonsAttribution4.0InternationalLicense,which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. A online resources The specification of the replications and their corresponding changes selected in the multiple case study are available at the Zenodo repository [19]. The CÆSAR tool is available at http://score.us.es/caesar/. References 1. Albayrak Ö, Carver JC (2014) Investigation of individual factors impacting the effectiveness of requirements inspections: a replicated experiment. Empir Softw Eng 19(1):241–266. https://doi.org/10.1007/ s10664-012-9221-0 2. Almqvist JPF (2006) Replication of controlled experiments in empirical software engineering - a survey. Master’s thesis, master’s thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden 3. Aranda A (2016) Empirical study of the influence of analyst experience and domain knowledge on the effectiveness of requirements education. PhD thesis, Polytechnic University of Madrid, https://doi. org/10.20868/UPM.thesis.40566 4. Assar S, Borg M, Pfahl D (2016) Using text clustering to predict defect resolution time: a conceptual replication and an evaluation of prediction accuracy. Empir Softw Eng 21(4):1437–1475. https://doi. org/10.1007/s10664-015-9391-7 5. Association for Computing Machinery (2020) Artifact review and badging. https://www.acm.org/ publications/policies/artifact-review-and-badging-current 6. Baldassarre MT, Carver J, Dieste O, et al (2014) Replication types: towards a shared taxonomy. In: Proceedings of EASE’14, pp 18:1-18:4, https://doi.org/10.1145/2601248.2601299 7. Basili VR, Caldiera G, Rombach HD (1994) Goal question metrics paradigm. Encycl Softw Eng, Wiley, pp 528–532 8. Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473. https://doi.org/10.1109/32.799939 9. Bernárdez B, Durán A, Parejo JA, et al (2014) A controlled experiment to evaluate the effects of mindfulness in software engineering. In: Proceedings of ESEM’14, pp 17–27, https://doi.org/10.1145/ 2652524.2652539 A model-based approach for specifying changes… 1211 10. Bernárdez B, Durán AD, Parejo JA et al (2020) Effects of mindfulness on conceptual modeling performance: a series of experiments. IEEE Trans Software Eng 48(2):432–452. https://doi.org/10.1109/ TSE.2020.2991699 11. Bernárdez B, Durán A, Parejo JA et al (2018) An experimental replication on the effect of the practice of mindfulness in conceptual modeling performance. J Syst Softw 136:153–172. https://doi.org/10. 1016/j.jss.2016.06.104 12. Bezerra RM, da Silva FQ, Santana AM, et al (2015) Replication of empirical studies in software engineering: An update of a systematic mapping study. In: Proceedings of ESEM’15, IEEE, pp 1–4, https://doi.org/10.1109/ESEM.2015.7321213 13. Brooks A, Daly J, Miller J, et al (1996) Replication of experimental results in software engineering. Technical Report ISERN-96-10, University of Strathclyde, Glasgow, UK 14. Brooks A, Roper M, Wood M, et al (2008) Replication’s role in software engineering, Springer, pp 365–379. https://doi.org/10.1007/978-1-84800-044-5_14 15. Campbell DT, Stanley JC (2015) Experimental and quasi-experimental designs for research. Ravenio books 16. Carver JC (2010) Towards reporting guidelines for experimental replications: a proposal. In: Proceedings of the 1st international workshop on replication in empirical software engineering, pp 1–4 17. Ciolkowski M, Shull F, Biffl S (2002) A family of experiments to investigate the influence of context on the effect of inspection techniques. In: Proceedings of EASE’02, pp 48–60 18. Cruz M, Bernárdez B, Durán A et al (2019) Replication of studies in empirical software engineering: a systematic mapping study, from 2013 to 2018. IEEE Access 8:26,773-26,791. https://doi.org/10.1109/ ACCESS.2019.2952191 19. Cruz M, Bernárdez B, Durán A et al (2021). Supplemental material: instantiation of the proposed templates in the multiple case study using CÆSAR LaTeX template. https://doi.org/10.5281/zenodo. 6631976 20. Da Silva FQ, Suassuna M, França ACC et al (2014) Replication of empirical studies in software engineering research: a systematic mapping study. Empir Softw Eng 19(3):501–557. https://doi.org/ 10.1007/s10664-012-9227-7 21. Durán A, Bernárdez B, Ruiz-Cortés A, et al (1999) A requirements elicitation approach based in templates and patterns. In: Proceedings of WER’99 22. Durán A, Corchuelo R, Ruiz-Cortès A, et al (2002) Supporting requirements verification using XSLT. In: Proceedings of RE’02, pp 165–172, https://doi.org/10.1109/ICRE.2002.1048519 23. Fernández DM, Graziotin D, Wagner S, et al (2020) Open science in software engineering, Springer International Publishing, pp 477–501. https://doi.org/10.1007/978-3-030-32489-6_17 24. Fernández-Sáez AM, Genero M, Caivano D et al (2016) Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments. Empir Softw Eng 21(1):212–259. https://doi.org/10.1007/s10664-014-9354-4 25. Gamma E, Helm R, Johnson R, et al (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley 26. García A, Rodríguez-Juan E, Rodríguez-Gutiérrez G et al (2016) Extraction of phenolic compounds from virgin olive oil by deep eutectic solvents (DESs). Food Chem 197:554–561. https://doi.org/10. 1016/j.foodchem.2015.10.131 27. Gómez OS, Juristo N, Vegas S (2010) Replications types in experimental disciplines. In: Proceedings of ESEM’10, pp 71–75, https://doi.org/10.1145/1852786.1852790 28. Gómez OS, Juristo N, Vegas S (2014) Understanding replication of experiments in software engineering: a classification. Inf Softw Technol 56(8):1033–1048. https://doi.org/10.1016/j.infsof.2014. 04.004 29. Guevara-Vega C, Bernárdez B, Durán A, et al (2021) Empirical strategies in software engineering research: A literature survey. In: II International conference on information systems and software technologies (ICI2ST 2021). IEEE Press, Quito, Ecuador 30. Carvajal de la Haza S (2016) Copper extraction by brown mustard (brassica juncea) plants during vegetative growth in an artificially contaminated soil and effect of rhamnolipid application. http://hdl. handle.net/11441/50282, (Final Degree Project). Universidad de Sevilla 31. Herbold S, Trautsch A, Grabowski J (2017) Global versus local models for cross-project defect prediction. Empir Softw Eng 22(4):1866–1902. https://doi.org/10.1007/s10664-016-9468-y 1212 M. Cruz et al. 32. Itkonen J, Mäntylä MV (2014) Are test cases needed? replicated comparison between exploratory and test-case-based software testing. Empir Softw Eng 19(2):303–342. https://doi.org/10.1007/s10664- 013-9266-8 33. Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to advanced empirical software engineering. Springer, pp 201–228, https://doi.org/10.1007/ 978-1-84800-044-5_8 34. Juristo N, Gómez OS (2012) Replication of software engineering experiments. In: Empirical software engineering and verification. Springer, pp 60–88, https://doi.org/10.1007/978-3-642-25231-0_2 35. Juristo N, Moreno AM (2013) Basics of software engineering experimentation. Springer Science and Business Media, Berlin. https://doi.org/10.1007/978-1-4757-3304-4 36. Juristo N, Vegas S (2003) Functional testing, structural testing and code reading: what fault type do they each detect? In: Empirical methods and studies in software engineering. Springer, pp 208–232, https://doi.org/10.1007/978-3-540-45143-3_12 37. Juristo N, Vegas S (2009) Using differences among replications of software engineering experiments to gain knowledge. In: Proceedings of ESEM’09, IEEE, pp 356–366, https://doi.org/10.1109/ESEM. 2009.5314236 38. Juristo N, Vegas S (2011) The role of non-exact replications in software engineering experiments. Empir Softw Eng 16(3):295–324. https://doi.org/10.1007/s10664-010-9141-9 39. Juristo N, Vegas S, Solari M, et al (2012) Comparing the effectiveness of equivalence partitioning, branch testing and code reading by stepwise abstraction applied by subjects. In: Proceedings of international conference on software testing, verification and validation, pp 330–339, https://doi.org/10. 1109/ICST.2012.113 40. Juristo N, Vegas S, Solari M et al (2013) A process for managing interaction between experimenters to get useful similar replications. Inf Softw Technol 55(2):215–225. https://doi.org/10.1016/j.infsof. 2012.07.016 41. Kitchenham B (2008) The role of replications in empirical software engineering a word of warning. Empir Softw Eng 13(2):219–221. https://doi.org/10.1007/s10664-008-9061-0 42. de Magalhães CV, da Silva FQ, Santos RE et al (2015) Investigations about replication of empirical studies in software engineering: a systematic mapping study. Inf Softw Technol 64:76–101. https:// doi.org/10.1016/j.infsof.2015.02.001 43. MendezD, Graziotin D, Wagner S, et al (2020)Openscienceinsoftware engineering. In:Contemporary empirical methods in software engineering. Springer, pp 477–501, https://doi.org/10.1007/978-3-030- 32489-6_17 44. Mondal M, Rahman MS, Roy CK et al (2018) Is cloned code really stable? Empir Softw Eng 23(2):693– 770. https://doi.org/10.1007/s10664-017-9528-y 45. Márquez M (2018) Copper extraction by brown mustard (brassica juncea) plants during vegetative growth in an artificially contaminated soil and effect of rhamnolipid application. https://hdl.handle. net/11441/132481, (Final Degree Project). Universidad de Sevilla 46. Navarro D (2018) Learning statistics with R: A tutorial for psychology students and other beginners (version 0.6). https://learningstatisticswithr.com/ 47. Pacheco YM, López S, Bermúdez B et al (2008) A meal rich in oleic acid beneficially modulates postprandial sicam-1 and svcam-1 in normotensive and hypertensive hypertriglyceridemic subjects. J Nutr Biochem 19(3):200–205. https://doi.org/10.1016/j.jnutbio.2007.03.002 48. Parejo JA, Sánchez AB, Segura S et al (2016) Multi-objective test case prioritization in highly configurable systems: a case study. J Syst Softw 122:287–310. https://doi.org/10.1016/j.jss.2016.09.045 49. Plasquy E, Florido MC, Sola-Guirado RR et al (2021) Effects of a harvesting and conservation method for small producers on the quality of the produced olive oil. Agriculture 11. https://doi.org/10.3390/ agriculture11050417 50. Reimanis D, Izurieta C, Luhr R, et al (2014) A replication case study to measure the architectural quality of a commercial system. In: Proceedings of ESEM’14, ACM, pp 1–8, https://doi.org/10.1145/ 2652524.2652581 51. Riaz M, King J, Slankas J et al (2017) Identifying the implied: findings from three differentiated replications on the use of security requirements templates. Empir Softw Eng 22(4):2127–2178. https:// doi.org/10.1007/s10664-016-9481-1 52. del Río-Ortega A, Resinas M, Durán A, et al (2012) Defining process performance indicators by using templates and patterns. In: Proceedings of BPM’12, Springer, pp 223–228, https://doi.org/10.1007/ 978-3-642-32885-5_18 A model-based approach for specifying changes… 1213 53. del Río-Ortega A, Resinas M, Durán A et al (2016) Using templates and linguistic patterns to define process performance indicators. Enterp Inf Syst 10(2):159–192. https://doi.org/10.1080/17517575. 2013.867543 54. del Río P (2019) Effect of rhamnolipid JBR–425 on the development of brassica juncea in urban garden soils in Sevilla. https://hdl.handle.net/11441/132478, (Final Degree Project) Universidad de Sevilla 55. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131. https://doi.org/10.1007/s10664-008-9102-8 56. Runeson P, Höst M, Rainer A, et al (2012) Case study research in software engineering. In: Guidelines and examples. Wiley Online Library, pp 109–126, https://doi.org/10.1002/9781118181034.ch8 57. Sánchez AB, Segura S, Ruiz-Cortés A (2014) A comparison of test case prioritization criteria for software product lines. In: Proceedings of ICST’14, pp 41–50, https://doi.org/10.1109/ICST.2014.15 58. Santos A, Gómez OS, Juristo N (2018) Analyzing families of experiments in software engineering: a systematic mapping study. IEEE Trans Softw Eng 46(5):566–583. https://doi.org/10.1109/TSE.2018. 2864633 59. Santos AR, do Carmo Machado I, de Almeida ES et al (2019) Comparing the influence of using feature-oriented programming and conditional compilation on comprehending feature-oriented software. Empir Softw Eng 24(3):1226–1258. https://doi.org/10.1007/s10664-018-9658-x 60. dos Santos DA, de Almeida ES, Ahmed I (2022) Investigating replication challenges through multiple replications of an experiment. Inf Softw Technol. https://doi.org/10.1016/j.infsof.2022.106870 61. Scanniello G, Marcus A, Pascale D (2015) Link analysis algorithms for static concept location: an empirical assessment. Empir Softw Eng 20(6):1666–1720. https://doi.org/10.1007/s10664-014-9327- 7 62. Segura S, Durán A, Troya J, et al (2017) A template-based approach to describing metamorphic relations. In: Proceedings of international workshop on metamorphic testing, pp 3–9, https://doi.org/ 10.1109/MET.2017.3 63. Shepperd M, Ajienka N, Counsell S (2018) The role and value of replication in empirical software engineering results. Inf Softw Technol 99:120–132. https://doi.org/10.1016/j.infsof.2018.01.006 64. Shull F, Basili V, Carver J, et al (2002) Replicating software engineering experiments: addressing the tacit knowledge problem. In: Proceedings of ISESE’02, pp 7–16, https://doi.org/10.1109/ISESE.2002. 1166920 65. Shull F, Carver JC, Vegas S et al (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218. https://doi.org/10.1007/s10664-008-9060-1 66. Solari M (2013) Identifying experimental incidents in software engineering replications. In: Proceedings of ESEM’13, IEEE, pp 213–222, https://doi.org/10.1109/ESEM.2013.26 67. Solari M, Vegas S (2006) Classifying and analysing replication packages for software engineering experimentation. In: Proceedings of WSESE’06 Amsterdam, Paises Bajos, pp 19–24 68. Solari M, Vegas S, Juristo N (2017) Content and structure of laboratory packages for software engineering experiments. Inf Softw Technol 97:64–79. https://doi.org/10.1016/j.infsof.2017.12.016 69. Spence JR, Stanley DJ (2016) Prediction interval: what to expect when you’re expecting—a replication. PloS One 11(9):e0162,874. https://doi.org/10.1371/journal.pone.0162874 70. Vegas S, Riofrío P, Marcos E et al (2020) On (mis) perceptions of testing effectiveness: an empirical study. Empir Softw Eng 25:2844–2896. https://doi.org/10.1007/s10664-020-09805-y 71. Von Alan RH, March ST, Park J et al (2004) Design science in information systems research. MIS Q 28(1):75–105. https://doi.org/10.2307/25148625 72. Wieringa RJ (2014) Design science methodology for information systems and software engineering. Springer, Berlin 73. Wohlin C, Runeson P, Höst M et al (2012) Experimentation in software engineering: an introduction. Springer, Berlin 74. Yin RK et al (2003) Case study research. Design and methods. Thousand Oaks, Sage Publications Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.