Methods as data: Investigating how reporting in methods sections affects perceived replicability of models

Methods as data - Investigating how reporting in methods sections affects perceived replicability of models Dr. Anna Korzeniowskaaand Dr. Louise Saulb,c a. eLife Sciences Publications Ltd, Cambridge CB2 1AW ; b. University of Southampton, Southampton, SO17 1BJ; c. Careers and Skills for Data-driven Research (CaSDaR) Overview Background and introduction Scoping review Survey Future work Background Project Development •Initially focused on Alzheimers (Potential Fabrication in Research Images Threatens Key Theory of Alzheimer’s Disease, 2022); high representation of transgenic mouse lines. •Broadened search to include Experimental Autoimmune Encephalitis (Huntemann et al., 2022; Ransohoff, 2012; Baker & Amor, 2014) •Switched focus by identifying research areas that involved the highest number of procedures under ASPA; looking at maximum impact/reach Introduction •Acknowledgement that current preclinical solid tumour models are currently not reported with sufficient detail (Gengenbacher et al., 2017) •Errington et al., 2021 described barriers encountered when assessing replicability of preclinical cancer models •There have been issues in translation of preclinical trials to clinical settings (Wartha et al., 2014), however these have focused on ‘human aspect’ (Llovera & Liesz, 2016) •We were interested in the replicability of methods as opposed to reproducibility of claims –we felt this was an ethical concern Introduction •The potential for replicability and reproducibility of an experiment is vital for trustworthy research and to verify claims made when publishing the research •Detailed methods sections are essential for replicability •If results of a study cannot replicated - the results are put into question •No established method of assessing replicability during peer review Introduction Introduction •Moral imperative to consider replicability of animal studies •Unsuccessful replications = waste of animals, unnecessary suffering •Protocols lacking information = doubts around refinement and reduction •Inability to follow previously tested protocols - unnecessary piloting/ starting from scratch •Conversely, ensuring good animal welfare linked to better replicability (Smith and Lilley, 2019) Introduction Previous work in this area •There has been work to address steps that could be taken to promote reproducibility in pre-clinical experiments: •Identification of factors to include/mitigate in pre-clinical experiments (Wilson et al, 2023) •The publishing of the Gold Standard Publication Checklist (Hooijmans et al., 2010), to support systematic reviews of preclinical work •Generation of the MDAR framework to promote reporting of preclinical models (Macleod et al., 2021) •ARRIVE reporting guidelines (du Sert et al., 2020) help ensure transparency and make methods sections more replicable. Our Objectives ●Use a scoping review to identify experimental factors/details currently included in methods sections of manuscripts describing Breast or Prostate mouse models ●Combine with experimental factors/details identified from previously mentioned sources ●Survey researchers to assess how inclusion of factors affects perceived replicability of methods ●Apply weighting factors based upon results ●Systematic review of literature to determine proportion of manuscripts that could be replicated based upon the data in the methods section Distribution of the Survey - Preliminary Results Demographic data of respondents: Average number of years working with experimental animals: 14 years (one non-respondent) 2 respondents had previously worked with breast or prostate models of cancer 5 respondents had considered attempting to replicate another researcher’s work 3 respondents were early careers researchers, 1 was a senior career researcher, another a mid career researcher 4 respondents worked at a Higher Education Institution, 1 respondent worked at a private company Systematic review To answer the question: "What is the distribution of perceived replication in peer-reviewed studies involving breast or prostate cancer animal models published in the last five years?" Inclusion criteria: ●Peer-reviewed ●Non-transgenic mice ●Only using information in the materials/methods section ●Use of a reasonable level of scrutiny Use of Systematic Review Facility to manage the process Barriers to assessing perceived replicability in literature Method Description Barriers E -utilities Entrez programming utilities, (E-utilities) which allow search and retrieval of information from the PubMed database with the use of a public API to retrieve numbers of manuscripts from the database which use specific keywords identified from the survey. The data in the PubMed database is unstructured, meaning that it is difficult to determine if the entries returned have the keywords present in the methods or elsewhere in the text. This presents an issue as we are focusing solely on the methods sections. Manual search A systematic review of the literature by creating an exclusion criteria of specific literature, searching the PubMed database, and manually collecting data from the methods sections of the returned manuscripts. This will need to be completed by more than 2 named researchers; bias minimisation steps will be required throughout the process and independent and random verification required (n=158, https://doi.org/10.7554/eLife.71601) Future work ●Analysis of manuscripts to assess proportion of manuscripts supporting replicability ●Works to be done by 2 independent researchers, with the third individual verifying the work by random allocation ●Qualitative research - discussion about ranking of factors and decide if there are additional factors to include and barriers to replicability Q&A/Feedback