scieee Open visual document viewer

Testing Research Software

Eisty, Nasir; Carver, Jeffrey; Kanewala, Upulee

Publisher
Zenodo
DOI
10.5281/zenodo.17258788
Repository
zenodo.org
Language
en

Abstract

This poster describes the work presented in a published paper [1], which reports the results of a survey of research software developers. The focus of the study was to understand how research software developers design test cases, handle output challenges, use metrics, execute tests, and select tools. Specifically, the study explores the following research questions; RQ1: What are the characteristics of the research software testing process? RQ2: What challenges do developers face throughout the testing process of research software? RQ3: How do research software developers design test inputs? RQ4: What specific challenges do research software developers face when determining the expected output of test cases? RQ5: What metrics do research software developers use to measure software quality and test quality of tests? RQ6: How do research software developers execute their tests? RQ7: What testing tools do research software developers use and what are their limitations? RQ8: What features should a testing tool specifically developed for research software contain? RQ9: Are demographic characteristics of research software projects related to the testing process? The poster will contain graphs reporting the detailed results related to these questions. Our overall findings show that research software testing varies widely. The primary challenges faced by research developers include test case design, evaluating test quality, and evaluating the correctness of test outputs. Our findings highlight the need to allocate more resources to software testing and provide more education and training to research software developers on software testing. References Eisty, N.U., Kanewala, U. & Carver, J.C. Testing research software: an in-depth survey of practices, methods, and tools. Empir Software Eng 30, 81 (2025). https://doi.org/10.1007/s10664-025-10620-6

Full text

Tes ing Resea ch So wa e Nasi Eis y, Uni e si y o Tennessee, Knox ille, TN Je ey Ca e , Uni e si y o Alabama, Tuscaloosa, AL Upulee Kanewala, Uni e si y o No h Flo ida, Jackson ille, FL Con ex Re e ence Conclusion Recommenda ions Objec i es Key Resul s ❖Challenges: ➢Tes case design ➢E alua ing es quali y ➢Co ec ness o ou pu s ❖App oaches: ➢62% manual inpu design ➢Ou pu s o en lack o acle ➢Non-de e minism complica es alida ion ❖Me ics: ➢S a emen co e age mos used ➢Many don’ ack quali y me ics ❖Execu ion: ➢Mos ly on push o end o cycle ➢Few use p io i iza ion ❖Tools: ➢Gi Hub Ac ions, Py es , Jenkins, Codeco ➢No s anda d se ❖Explo e es ing p ac ices o esea ch so wa e ❖Unde s and challenges: es design, ou pu s, me ics, execu ion, ools ❖In es iga e demog aphic in luences (domain, eam size, oles) ❖Resea ch so wa e is c i ical o science and inno a ion ❖Ensu ing co ec ness is essen ial bu challenging ❖Indus ial es ing ools a e o en inadequa e ❖Resea ch so wa e es ing is less sys ema ic and au oma ed han indus y. ❖O acle p oblem and es case design a e bigges hu dles. ❖Mo e esou ces, specialized ools, and educa ion needed. ❖Funde s/manage s: alloca e mo e es ing esou ces ❖De elope s: adop documen a ion + sys ema ic es ing ❖Tool builde s: c ea e domain-awa e and easy o use ools ❖Educa o s: in eg a e es ing in o esea ch cu icula Me hod ❖Su ey o 131 esea ch so wa e de elope s ❖Topics: es inpu design, expec ed ou pu s, me ics, ools, limi a ions ❖Rec ui men ia US-RSE, UK-RSE, IDEAS-ECP, BSSW mailing lis s Eis y, N.U., Kanewala, U. & Ca e , J.C. Tes ing esea ch so wa e: an in-dep h su ey o p ac ices, me hods, and ools. Empi So wa e Eng 30, 81 (2025). h ps://doi.o g/10.1007/s10664-025-10620-6 Demog aphics Cha ac e is ics o he Tes ing P ocess RSE? Deg ee ype? Domain Sou ce o es ing knowledge #FTEs % ime on es ing Ranking o Tes ing Challenges In luence o Demog aphics (Sys ema ic s FTEs) Scan o ead he pape