Raise the MAST (DOI) – Assessing Preprints for Data DOIs

Raise the MAST (DOI) Assessing Preprints for Data DOIs Sophie Jo Miller (STScI) Acknowledgements Special thanks to the STScI Metrics Office (Jenny Novacescu, Chris Wilkinson, and Achu Joy Usha), DSMO (John F. Wu), and MAST (Sarah Weissman, Scott Fleming) for their comments, input, meeting notes, recommendations, and all the work that preprint review requires, and to Felix Stoehr (ESO), who guided STScI through initial planning for the preprint process. Thanks also to Erin Miller for comments, edits, and confirmation that this makes sense to someone who doesn’t review preprints. Background image of SMACS 0723-73 compiled with JWST NIRCam data from program 2736. Image: NASA, ESA, CSA, STScI (https://science.nasa.gov/asset/webb/webbs-first-deep-field-nircam-image/). Dataset available at https://www.doi.org/10.17909/7cpc-1z46 This poster has made use of the Astrophysics Data System, funded by NASA under Cooperative Agreement 80NSSC21M00561. Citations Stoehr, F., Grothkopf, U., Meakins, S., Bishop, M., Uchida, A., Testi, L., Iono, D., Tatematsu, K., & Wootten, A. (2015). ALMA Cycle 0 Publication Statistics. The Messenger, 162, 30-35. https://doi.org/10.48550/arXiv.1601.04499 JWST Mission Office. (2024, March 15.) Citing JWST data. JWST User Documentation. https://jwstdocs.stsci.edu/accessing-jwst-data/citing-jwst-data Since the launch of the James Webb Space Telescope, the Space Telescope Science Institute Metrics Office has partnered with MAST (Mikulski Archive for Space Telescopes) to identify preprints that use JWST data. Preprints are evaluated to determine whether the data is cited using a DOI, as required by the JWST Mission Office (2024). If not, authors are sent instructions for including data DOIs before the refereed publication goes to print. Preprints are identified through an integration of our paper review system (PaperTrack) with arXiv. The Metrics Office reviews preprints throughout the year, which enables us to identify potential science papers before finalizing our annual report. Contacting authors increases the inclusion of data DOIs. As of 2024, the rate of data DOIs in JWST papers was higher when we were able to contact the author during the preprint stage (47%) than when we were unable to contact the author (40%). Origins with ALMA Use of ALMA data requires a publication acknowledgement. Felix Stoehr (ESO) developed a system where preprints that use ALMA data are reviewed for the required acknowledgement. If the acknowledgement isn’t present, the author is asked to add it before the refereed version is published. This system has been highly successful; 98% of ALMA publications include the required acknowledgement (Stoehr et al., 2015), as a result. The Major Difference ALMA only asks for an acknowledgement (author can cut-and-paste into their paper). STScI asks for a data DOI (author must create before adding to their paper). Preprints vs. Refereed Publications Preprints are JWST-only; refereed publications are reviewed for Hubble, Webb, and Roman keywords. Preprints are only reviewed for science; non-science preprints are ignored. oRefereed non-science papers are further categorized as data influenced, supermention, mention, engineering, or instrument. Preprints identified as “science” are only reviewed for DOIs; refereed publications are reviewed for DOIs, instruments, program IDs, and datasets. The Preprint Process and PaperTrack Our paper review system, PaperTrack, pulls in arXiv preprints via ADS. Preprints are filtered through a simple keyword set (JWST, Webb Space Telescope, and Webb Telescope). Filtered preprints are reviewed with two questions in mind: Does this paper use JWST data for science? If yes, does this paper include a MAST data DOI? Science vs. Non-Science Science papers are categorized as such, allowing us to provide preliminary estimates of JWST science papers throughout the year. Non-science papers are ignored. We expect to see them again once the refereed paper is published as part of our standard paper review (where papers are reviewed and categorized for all our flagship missions). Refereed Publication Review Once the refereed publication is available, papers are fed back into PaperTrack based on an extensive keyword set (which includes name variations and instruments for all flagship missions). Preprints are matched with refereed publications through ADS (which combines bibcodes for both the preprint and the refereed publication into a single record). If a paper was marked “science” as a preprint, we see that category on the refereed publication record. Compliant (DOI) Non-Compliant (No DOI) Future Plans LLM Automation We are currently incorporating partial automation in the preprint process. Preprints are downloaded from arXiv as PDFs, then converted to plain text. Plain text files are parsed for JWST-related keywords + ~3 surrounding sentences (for context) to create “text snippets” for ranking. Text snippets are ranked by the LLM in order of most to least relevant to the “science or non-science” question. A JWST science score is returned, based on those rankings. If the JWST science score is high enough, the process runs again, using DOIspecific keywords to identify and rank snippets to return a DOI score. The rest of the process stays the same. All science papers are manually reviewed to confirm that they use JWST data, to check DOI compliance, and to identify author contact information. Ongoing Work Continued evaluation of the preprint process; it may reach a point where the results no longer justify the time investment. LLM Automation is expected to be fully integrated by mid-2026, which will help us keep up with the increasing number of JWST publications every year while (hopefully) decreasing the time investment. No plans as of now to incorporate this into Roman bibliographic work; however, we are considering how the process can be applied to future flagship missions. Is Preprint Evaluation Worth It? It depends. While it only improves data DOI inclusion by around 7%, it also encourages researchers build a habit around data citation. For example, we are starting to see DOIs for HST-only data, which may be a result of encouraging this habit with JWST. Ultimately, the benefits go beyond raw numbers. 21 17% 99 83% Preprint Available, Not Tracked No DOI No Preprint, 46 DOI No Preprint, 31 No DOI Preprint Status Unclear, 99 DOI Preprint, 21 Preprint, 538 Preprint, 475 No DOI DOI All JWST Papers, 2022-2024 31 40% 46 60% Preprint Not Available, Not Tracked Statistical Breakdown For the 2022-2024 reporting period, there were 1,210 JWST Science Papers total. The rate of compliance is slightly higher among papers where we were able to contact the author (47%) vs. papers where a preprint was unavailable for review (40%). This accounts for 1,090 papers. There are an additional 120 papers that fall into a third category—Preprint Available, Not Tracked. Of these, 21 include a DOI. These preprints were reviewed and a DOI was found, so the author was not contacted (and not tracked, which happens when an author email is input). The remaining 99 papers do not have a DOI and the author was not contacted, despite an arXiv preprint being available. This can happen for several reasons, including preprints being published too close to the refereed publication (no time to review), preprints not being found via keyword search, and human or PaperTrack error. 475 47% 538 53% Preprint Reviewed, Tracked