scieee AI-readable full text Open interactive document viewer

rplaywright: Bridging Playwright to R for Reproducible Web Automation in Research

Siregar, Erika

Abstract

Dynamic web content plays an important role in research, from accessing public dashboards to automating testing workflows. However, R users face challenges when working with JavaScript-heavy websites, as most available tools rely on static HTML scraping or browser automation approaches that can be difficult to integrate smoothly into R workflows.To address these challenges, rplaywright (https://erikaris.github.io/rplaywright/) was developed to integrate the R environment with Microsoft’s Playwright, a browser automation framework used across multiple programming languages. The package enables automated interaction with Chromium, Firefox, and WebKit browsers, operating in either headless or visible modes. It supports tasks such as navigating web pages, scrolling through dynamically loaded content, completing online forms, and extracting information from sites that update dynamically. Features including automatic waiting for page components, handling of infinite scrolling, and reliable interaction with dynamic interfaces facilitate reproducible workflows for research involving web resources.rplaywright facilitates the seamless integration of dynamic web content into R-based research workflows, supporting scalable and reproducible data collection from interactive web portals. The project originated as part of the rOpenSci Champions Program, emphasising open development and community-driven technical mentorship. The package is suitable for workflows involving interactive data portals, social media platforms, and dynamic dashboards, enabling researchers to programmatically navigate pages, interact with filters, and extract underlying data for analysis. This poster will present the architecture of rplaywright, highlight key design considerations, demonstrate integration with popular R packages for streamlined analysis, and explore how cross-language tools can extend R’s capabilities to enhance research outcomes.Acknowledgements This project was initiated through my participation in the rOpenSci Champions Program 2024 (https://ropensci.org/blog/2024/02/15/champions-program-champions-2024/). I thank the rOpenSci team for their guidance and feedback throughout the development process. No external funding was received for this work.

Full text

Static website Scrape Static Website rplaywright can also scrape static websites using methods such as locator, get_by_alt_text, get_by_label, get_by_placeholder, get_by_role, get_by_test_id, get_by_text, and get_by_title. Scan QR Code for example. no-wait no result wait for response has result rplaywright rplaywright R R 11 11 cl a sses classes 186 186 methods methods rplaywright rplaywright R R rplaywright rplaywright R R rplaywright : Bridging Playwright to R for Reproducible Web Automation in Research Erika Siregar University of Sheffield [email protected] rplaywright is an R package that bridges R with the Node.js Playwright library. It enables R users to perform simplified web testing and automation. The package was developed through the rOpenSci Champions Program. RVest RPlaywright Without Browser ✅ Cross browser: Chromium, Firefox, Webkit ✅ Static, dynamic, lazily loaded web page ✅ Static web page only ✅ Wait for request, response, locator, event ✅ No wait for mechanism ✅ Cookie injection ✅ No cookie injection ✅ Script injection ✅ No script injection ✅ Script evaluation, e.g. scroll to bottom to load more ✅ No script evaluation ✅ Multiple browser context ✅ No browser context ✅ Multiple page ✅ Single page ✅ Traditional scraping tools often struggle with dynamic, JavaScript-heavy websites. rplaywright solves this by waiting for page elements and events and interacting directly with the browser Document Object Model (DOM), bringing modern web automation to R. How rplaywright works: Limitations: Node.js Dependency – rplaywright relies on a Node.js-based API, which increases installation size. ✅ Browser Engines – rplaywright downloads dedicated builds (its own versions) of Chromium, Firefox, and WebKit. This ensures reproducibility but adds to the package footprint. ✅ Workflow Integration – rplaywright outputs are not yet dataframes, so a short conversion is needed before using %>% or |>. Once structured, they integrate smoothly with packages like dplyr, tidyr, and ggplot2. ✅ Coverage : Contribute: Future Works: Expanded API Coverage – rplaywright does not yet implement all classes and methods of the Node.js Playwright API. Ongoing development and contributions will extend its functionality. ✅ Enhanced R Integration – Compatibility with both magrittr pipe (%>%) and native pipe (|>) is planned to improve workflow fluency. Additional work is also needed to handle methods with identical names by introducing clearer input differentiation. ✅ Use Cases: *For demonstration purposes only; always respect robots.txt when web scraping. Fill Form rplaywright can automate form interactions, such as registration, login, or surveys. Methods include locator, blur, check, clear, click, dblclick, fill, focus, hover, press, select_option, and more. Scan QR Code for example. Fill Form Scraping X post Scrap X Posts rplaywright can handle complex dynamic websites, interacting with the DOM and network. In this example, a cookie is injected into the browser context, and posts are captured from network responses while scrolling to the bottom. Scan QR Code for example. WHAT WHY