Automating License-Aware Full-Text Retrieval for Systematic Reviews: An End-To-End Scalable System to Reduce Reviewer Workload

AUTOMATING LICENSE-AWARE FULL-TEXT RETRIEVAL FOR SYSTEMATIC REVIEWS: AN END-TO-END SCALABLE SYSTEM TO REDUCE REVIEWER WORKLOAD D. Zhuk*,†,1, E. Sandner†,2,4, I. Jakovljevic2, A. Simniceanu3, L. Fontana3, A. Henriques2, A. Wagner2, C. Gütl4 1University of Vienna, 1010 Vienna, Austria 2CERN, 1211 Geneva, Switzerland 3WHO, 1211 Geneva, Switzerland 4Graz University of Technology, 8010 Graz, Austria Abstract Systematic reviews are widely regarded as the most rigorous method for synthesizing scientific evidence, yet they remain highly labour-intensive. Full-text retrieval is a monotonous, repetitive, and time-consuming task that requires reviewers to locate and validate large numbers of articles. Existing tools only partially address this step, with limited support for automated, open-source, and legally compliant retrieval across heterogeneous repositories. To address this gap, a license-aware, open-source system was developed to automate full-text retrieval, extraction, and validation as part of the NeutrinoReview project. The system integrates open APIs (Unpaywall, PubMed, EuropePMC, Crossref) with a prioritized lookup strategy, browser-based PDF downloading, text extraction, and metadata-based validation. Performance was evaluated across 500 articles from five major scholarly repositories (PubMed, PMC, EuropePMC, IEEE Xplore, ACM Digital Library). Results show consistently high combined extraction rates (CER ≥ 0.800) and average processing times of 7–9 seconds per article. In realistic review scenarios, the system achieves a PDF retrieval rate of 82.68% and reduces manual retrieval workload by approximately 80%, corresponding to time savings of more than 3 hours in median sized SRs. These findings demonstrate the feasibility of automating a critical step in SR workflows, improving reproducibility and scalability while freeing researchers to focus on evidence synthesis. INTRODUCTION To gain a comprehensive understanding of a subject area, fragmented knowledge must be organized into structured information. Systematic review (SR) is a synthesis of identified and critically assessed evidence for topic understanding. This process is considered more rigorous and robust than a literature review as it follows a strict methodological framework, typically accompanied by predefined inclusion criteria [1]. Generally, SR consists of several phases that may vary depending on the methodology applied: •Project Initiation and Data Retrieval – defining the research question, setting inclusion and exclusion criteria, and retrieving bibliographic metadata from relevant scholarly repositories. •Screening – applying eligibility criteria to titles, abstracts, and subsequently full-text articles to ensure only relevant studies are included. •Data Extraction – gathering relevant methodological details, outcomes, and contextual information from the included articles for further analysis and synthesis. Full-text retrieval is the key part of the screening stage of SR, where reviewers must examine the complete text of articles to determine their eligibility [2]. Without access to the full-text data, important methodological details, outcomes, or context may remain hidden, leading to biased evidence synthesis. In particular, this paper addresses the following research question: How can an open-source, legally compliant solution be developed to automate full-text retrieval, extraction, and validation across multiple scholarly repositories, reducing reviewer workload and improving reproducibility in SRs? BACKGROUND AND RELATED WORK It is worth noting that overall performing a high-quality SR requires a lot of manual work and remains time-consuming, especially when following specific guidelines to be built upon (e.g., Preferred Reporting Items for SRs and Meta-Analyses) [3]. Conducting SR may take from 6 to 18 months [4]. In particular, full-text retrieval often represents a critical bottleneck – while bibliographic records are typically retrievable through Application programming interfaces (API) or structured search interfaces, access to corresponding full-text articles can be fragmented, license-restricted, or entirely unavailable without manual intervention. For instance, one of the most common problems include but are not limited to: •Publisher paywalls and subscription barriers as many articles remain inaccessible without institutional access or individual payments. •Licensing and copyright restrictions as even when access is granted, text-mining or bulk retrieval may be legally constrained. •Heterogeneous platforms and formats as full texts are dispersed across multiple sources with inconsistent metadata, formats, and access protocols. ___________________________________________ * a1244642[email protected]ie.ac.at † Both authors contributed equally to this work https://doi.org/10.5281/zenodo.17220241 Figure 1: Conceptual architecture of full-text retrieval module for systematic review automation tools; Blue represents the Open-Source API Querying Component, Green represents PDF Full-text Data Extraction, and Red represents PDF Full-text Validation. •Incomplete or unreliable linking as bibliographic metadata often lacks stable IDs or direct links to fulltext sources, requiring manual searches. •Limited API support as only some repositories (e.g., PubMed) provide open APIs, while others restrict programmatic access. As a result, reviewers often spend substantial time locating, downloading, and verifying eligible resources and corresponding articles – an effort that detracts from the analytical phase of the review process. It is reported that resource-intensive task such as full-text retrieval can consume a minimum of 8,000 minutes of researcher time, depending on the screening approach used [5]. Despite recognition of the challenges described, existing tools (e.g., ASReview, Cadmus, Covidence) provide only partial solutions, with limited support for automated, open-source, and legally compliant full-text retrieval [6]. It is therefore worth emphasizing that the burden of effort in SR is still skewed toward labour-intensive steps, where retrieval, extraction, and validation of full-texts are performed manually, even when upstream bibliographic searches are automated. The imbalance overall not only slows down review production but also risks inconsistencies across respective research projects consuming valuable time resources [7]. Consequently, those persistent limitations highlight the need for new approaches that integrate metadata search with transparent full-text access. Such approaches will not only reduce manual workload but also promote consistency and reproducibility SR teams. CONCEPTUAL DESIGN The NeutrinoReview prototype1 already supports automated bibliographic metadata retrieval from major sources such as PubMed, MEDLINE, and EuropePMC, as well as from user-supplied datasets. The metadata is stored in a structured database, providing a foundation for scalable and reproducible review workflows. However, the current implementation stops short of delivering full-text retrieval capabilities, leaving reviewers to perform this step manually. Figure 1 illustrates the proposed system for full-text retrieval, featuring end-to-end, license-aware architecture designed for seamless integration into the NeutrinoReview project. It is organized into three key components, each addressing a critical step in the retrieval process, which are described subsequently. Open-Source API Querying The first component takes bibliographic inputs (DOI, PMID/PMCID, title, authors) and attempts to discover legally retrievable full-text artefacts (i.e., PDF URLs, XML structure) and accompanying license. It follows a prioritized lookup strategy designed to maximize accuracy and reproducibility: •If DOI is present, the system queries Unpaywall [8] to obtain candidate open-access PDF URLs and license information. Unpaywall is preferred because it aggregates open-access locations and returns explicit license metadata when available. •If only PMID/PMCID is supplied (or DOI lookup fails), the system queries PubMed/PubMed Central (PMC)/BioC endpoints to recover structured XML and any license statements embedded in repository metadata. When the result contains DOI, the DOI is rechecked against Unpaywall as a secondary source. •As a final lookup, a metadata-to-DOI lookup against Crossref is attempted using title and author strings (also in case if DOI/PMID/PMCID lookups fail); any discovered DOI is then checked with Unpaywall. All license strings returned by external services are normalized into a compact decision set used by downstream ___________________________________________ 1) https://gitlab.cern.ch/caimira/caimira-wp4/neutrinoreview https://doi.org/10.5281/zenodo.17220241 logic: permissive for text mining and storing (open) or not permissive (unavailable). The component logs each service query, timestamps, raw service responses, and the normalization rationale to create an auditable tracking record. PDF Full-text Data Extraction The second component is responsible for acquiring the canonical article file (i.e., PDF structure) from PDF URLs when available, converting that file into extractable text, and returning a normalized textual representation suitable for downstream parsing, validation, and screening. It is implemented by two cooperating routines: a browser-based downloader used as a fallback and primary PDF retrieval and text-extraction function. In operation, the extractor first prepares conservative browser-like HTTP headers and prefers structured or direct access. If that fails, it performs an HTTP GET and validates the response Content-Type before opening the bytes. When publishers serve PDF files dynamically or require JavaScript, the extractor falls back to a headless Chromium downloader that polls a temporary download directory for a completed .pdf file. Once valid PDF stream is obtained, the extractor iterates pages to collect page-level text and returns a single concatenated text (with page breaks preserved). Such failures as non-PDF responses, network timeouts, corrupted files, or images result in a None return and are recorded with standardized diagnostics. PDF Full-text Validation The third component verifies that the full-text extracted from a retrieved PDF corresponds to the expected bibliographic metadata and meets minimum quality criteria before the document is further processed. This validation is performed by two routines: a TF-IDF/cosine similarity scorer and a validator that applies heuristic thresholds. The validator lowercases the extracted full-text data and uses the first 10,000 characters as the primary search window since titles, authors, and abstracts typically appear near the start. A conservative regex attempts to extract an “abstract” block from the text; pairwise similarities are then computed between the supplied title and the document start, the supplied abstract and the extracted abstract, and the supplied authors and the document start. Then, the validator returns a compact diagnostic object containing the three similarity scores and a boolean flag of validity; by default, a record is accepted if abstract similarity is greater than 0.20 or authors similarity is greater than 0.40 or title similarity is greater than 0.30. These thresholds are set up empirically by testing different ranges are tested for abstract, author, and title similarities, including 0.2 to 0.4, 0.3 to 0.5, 0.6 to 0.8, and 0.7 to 1.0. Given the low dimensionality of abstracts and article metadata, selected thresholds yielded the best results and are sufficient for robust validation. If no text is available or the checks fail decisively the function signals invalidity (i.e., False); all similarity scores are logged for tracking and threshold tuning. Solution Outcome The outcomes of the system are machine-readable tables that encode for each article: its licensing status, the canonical PDF link, PDF structure, XML structure, and PDF validation outcomes (if any). These outputs can be directly consumed by SR pipelines for automated or manual fulltext screening, data extraction, or critical appraisal. Moreover, the solution is extensible, ensuring that new retrieval methods, content sources, or document validation formats can be incorporated without substantial redesign. Crucially, the approach adheres to applicable legal restrictions and does not depend on paid content providers. EVALUATION METHODOLOGY For evaluation, the proposed system was applied to multiple scholarly sources – PubMed, PMC, EuropePMC, IEEE Xplore, and ACM Digital Library – with a focus on medical literature. These tests are conducted in order to demonstrate the feasibility of significantly reducing reviewer workload while maintaining reproducibility and scalability. Each source is queried with domain-specific search strings designed to capture representative subsets of research articles relevant to airborne transmission, respiratory particle dynamics, or open-access retrieval architectures. From each source, the first 100 articles are returned by the queries selected, resulting in 500 articles in total. The query configurations are as follows: • PubMed – sorted by Best Match, bibliographic metadata is PMID, in Summary (text) format. • PMC – sorted by Default order, bibliographic metadata is PMCID, in PMCID list format. • EuropePMC – sorted by Relevance, bibliographic metadata is PMCID, in ID list format. • IEEEXplore – sorted by Relevance, bibliographic metadata is DOI, in Plain text format. • ACM Digital Library – sorted by Recency, bibliographic metadata is DOI, in ACM Ref format. Search strings for each of the sources can be found in Appendix 1. The system performance is quantified using extraction and validation outcomes per 100-article from each source. Table 1 lists the considered metrics along with their definitions. Table 1: Metrics Overview Metrics Name and Description Metrics Abbreviation and Equation Open Articles – Count of articles determined to have an open license 𝑂𝐴 = 𝑁𝑂𝐴 PDF Retrieval Rate – Fraction of open articles with a canonical PDF link successfully retrieved 𝑃𝑅𝑅 = 𝑁𝑃𝐷𝐹 𝑁𝑂𝐴 PDF Extraction Rate – Fraction of open articles from which PDF structure is extracted 𝑃𝐸𝑅 = 𝑁𝑃𝐸𝑅 𝑁𝑂𝐴 https://doi.org/10.5281/zenodo.17220241 XML Extraction Rate – Fraction of open articles with XML structure available 𝑋𝐸𝑅 = 𝑁𝑋𝐸𝑅 𝑁𝑂𝐴 Combined Extraction Rate – Fraction of open articles with either PDF or XML structure available 𝐶𝐸𝑅 = 𝑁𝑃𝐸𝑅 + 𝑁𝑋𝐸𝑅 − (𝑁𝑃𝐸𝑅 ∩ 𝑁𝑋𝐸𝑅) 𝑁𝑂𝐴 Total Processing Time – Wall-clock time statistics, total runtime (in seconds) 𝑇𝑃𝑇 = 𝑇𝑝𝑟𝑜𝑐 EVALUATION RESULTS The system’s ability to retrieve and extract full-text varied significantly across repositories, reflecting differences in openness, metadata availability, and source infrastructure, as detailed in Table 2. Table 2: Benchmark Evaluation Results Name OA PRR PER XER CER TPT PubMed 90 0.589 0.478 1.000 1.000 707.971 PMC 100 0.620 0.410 1.000 1.000 926.470 EPMC 93 0.925 0.871 1.000 1.000 912.000 IEEE 10 1.000 0.800 0.000 0.800 246.39 ACM 57 1.000 0.860 0.000 0.860 841.220 In terms of availability, open-source articles coverage is the highest for PMC (100/100) and EuropePMC (93/100), reflecting their open mandates. The proposed solution also performes well on PubMed (90/100), while coverage on ACM Digital Library (57/100) and especially IEEEXplore (10/100) is much more restricted. In terms of retrieval, the system achieves its strongest performance on EuropePMC, with PRR (0.925) and PER (0.871), complemented by perfect XML coverage. On ACM and IEEE perfect PRR (1.000) is reached, but the absence of XML fallback limits CER to 0.860 and 0.800, respectively. PubMed and PMC sources provide complete coverage through XML, though their respective PER scores are less reliable. Processing times are generally consistent, averaging 7-9 seconds per article. IEEEXplore shows the fastest total runtime due to its small OA sample, whereas PMC required slightly longer because of additional fallback operations. Overall, the system demonstrates strong performance across all sources, with CER never falling below 0.800, ensuring that full-text data is consistently available either in PDF or XML format. DISCUSSION The impact of automated full-text retrieval in systematic reviews becomes evident when considering its potential to reduce reviewer workload in real-world scenarios. An analysis of 195 systematic reviews showed that between 0 and 4,385 studies (mean = 63) were included at the title and abstract screening stage and therefore had to be retrieved in full text [9]. When automation is not available, reviewers must perform this step manually, and retrieving a single full text is estimated to take an average of 4 minutes [5]. Consequently, manually retrieving full texts for a systematic review with the mean number of included studies (63) requires about 4 h 12 min, whereas the most exhaustive case (4,285 studies) would demand approximately 292 h 20 min. By contrast, the proposed solution achieves an average PRR of 82.68%, implying that only 17.32% of articles require manual retrieval. The average processing time for 100 studies is 726.81 seconds, corresponding to 7.3 seconds per article. Applying this solution to a systematic review requiring 63 full texts, about 52 can be retrieved automatically, while 11 must be retrieved manually. The system’s processing time amounts to 6 min 20 s, with an additional 44 min of manual work, yielding a total retrieval time of 50 min 20 s. This corresponds to a workload reduction of 3 h 21 min 40 s for a mean-sized systematic review. Based on the same assumptions, for a systematic review with 4,385 records, the system’s processing time would be 7 h 21 min 10 s without any parallelization of the retrieval mechanism, whereas manual retrieval would require about 232 h 22 min 50 s. In this extremely large case, the system reduces the workload by 232 h 22 min 50 s. Consequently, the system can reduce the time required for full-text retrieval by 80%. LIMITATIONS Each component of the solution proposed has practical constraints that may influence performance. Firstly, coverage depends heavily on source policies: repositories with restrictive access models (e.g., IEEEXplore, ACM Digital Library) yield fewer open articles, which reduces overall retrieval opportunities despite high PDF success rates when links are available. Secondly, PDF extraction remains fragile in cases of scanned documents, image-only pages, or publisher-specific encodings, where structured XML is not available as a fallback. Thirdly, metadata inconsistencies (e.g., variant author strings, missing abstracts) can lower validation scores and may exclude possible usable texts. Moreover, processing speed, while generally acceptable, is influenced by network conditions and the need for browser-based fallback routines, which may not scale well at very large volumes. Finally, the system is designed to operate within legal boundaries of open-access content – paywalled or license-restricted materials remain inaccessible by design, which can limit completeness for certain research domains. https://doi.org/10.5281/zenodo.17220241 FUTURE WORK Future development of the system will focus on three main directions. Improving robustness of PDF extraction by integrating Optical character recognition (OCR) pipelines for scanned or image-only documents, and experimenting with hybrid approaches that combine parsing with Machine Learning-based text recovery. Expanding source coverage by incorporating additional APIs and institutional repositories, thereby improving completeness in restricted domains. Refining validation by training domain-adaptive similarity models that go beyond heuristics, enabling more accurate alignment of metadata and full-text data. Additionally, efforts will be made to optimize processing speed and resource efficiency, ensuring the system remains scalable for large SR projects. Continuous feedback and more real-world testing will guide those iterative improvements. CONCLUSIONS Thus, in this paper, a license-aware, open-source solution for automated full-text retrieval, extraction, and validation across multiple major scholarly repositories is presented. Benchmarking against PubMed, PMC, EuropePMC, IEEEXplore, and ACM Digital Library demonstrates that the system consistently achieves high combined extraction rates (CER is greater than 0.800), ensuring reliable availability of either PDF or XML structures. The results confirm both the feasibility and scalability of automating a critical bottleneck in systematic reviews, reducing manual reviewer workload while maintaining reproducibility. At the same time, differences across repositories highlight the continued challenges of restricted access and heterogeneous infrastructures. By providing extensible components and transparent diagnostics, the system lays a foundation for future improvements, including expanded coverage, more robust extraction methods, and tighter integration with SR pipelines. ACKNOWLEDGEMENTS This research was conducted as part of the joint CERN– WHO ARIA2 project, which funds Elias Sandner’s PhD studies and within which this paper was prepared. We also gratefully acknowledge the OpenWebSearch.EU3 project and its members for their valuable support with this publication. REFERENCES [1] R. Randles and A. Finnegan, “Guidelines for writing a systematic review”, Nurse Education Today, vol. 125, p. 105803, June 2023. doi:10.1016/j.nedt.2023.105803 [2] L. Schmidt et al., “Data extraction methods for systematic review (semi)automation: Update of a living systematic review”, F1000Research, vol. 10, article 401 (version 3), Apr. 2025. doi:10.12688/f1000research.51117.3 [3] F.M. Delgado-Chaves et al., “Transforming literature screening: The emerging role of large language models in systematic reviews”, Proc. Natl. Acad. Sci. U.S.A., vol. 122, e2411962122, Jan. 2025. doi:10.1073/pnas.2411962122 [4] V. Phillips and E. Barker, “Systematic reviews: Structure, form and content”, Journal of Perioperative Practice, vol. 31, p. 349-353, Jan. 2025. doi:10.1177/1750458921994693 [5] I. Shemilt et al., “Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews”, Syst. Rev., vol. 5, article 140, Aug. 2016. doi:10.1186/s13643-016-0315-4 [6] L. Affengruber et al., “An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review”, BMC Med Res Methodol, vol. 24, article 210, Sept. 2024. doi:10.1186/s12874-02402320-4 [7] K.E.K. Chai et al., “Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews”, Syst. Rev., vol. 10, article 93, Apr. 2021. doi:10.1186/s13643-021-01635-3 [8] Unpaywall API, https://unpaywall.org/products/api [9] R. Borah et al., “Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry”, BMJ Open 2017, vol. 7, article e012545, Oct. 2016. doi:10.1136/bmjopen-2016012545 ___________________________________________ 2) https://partnersplatform.who.int/tools/aria 3) https://openwebsearch.eu https://doi.org/10.5281/zenodo.17220241 APPENDIX Source Name Search String PubMed (airborne[tiab] OR aerosol*[tiab] OR "airborne transmission"[tiab] OR "air transmission"[tiab] OR inhalation[tiab]) AND (risk[tiab] OR "risk assessment"[tiab] OR exposure[tiab] OR hazard*[tiab]) AND (model[tiab] OR models[tiab] OR modelling[tiab] OR modeling[tiab] OR "mathematical model"[tiab] OR "computational model"[tiab] OR simulation[tiab] OR simulations[tiab]) PMC (droplet* OR particle* OR aerosol*) AND (size OR diameter OR "particle size" OR "droplet size" OR volume* OR cm OR centimetre OR centimeter OR µm OR micron OR "micrometer" OR "micro-meter") AND ("expiratory activity" OR "expiratory activities" OR "respiratory activity" OR "respiratory activities" OR breath* OR speak* OR talk* OR shout* OR sing* OR cough* OR sneez*) EuropePMC ((droplet* OR particle* OR aerosol*) AND (size OR diameter OR "particle size" OR "droplet size" OR volume* OR cm OR centimetre OR centimeter OR µm OR micron OR micrometer) AND ("expiratory activity" OR "expiratory activities" OR "respiratory activity" OR "respiratory activities" OR breath* OR speak* OR talk* OR shout* OR sing* OR cough* OR sneez*)) IEEEXplore (open OR "open-source" OR "open access") AND (search* OR retrieval OR discovery OR "information retrieval") AND (architecture* OR framework* OR system* OR platform* OR infrastructure* OR toolkit*) ACM Digital Library (open OR "open-source" OR "open access") AND (search* OR retrieval OR discovery OR "information retrieval") AND (architecture* OR framework* OR system* OR platform* OR infrastructure* OR toolkit*) Appendix 1: Search strings for benchmark evaluation https://doi.org/10.5281/zenodo.17220241