Open comments on the "Open Letter: Establishing a national research software award "

Open comments on the “Open Letter: Establishing a national research software award” Teresa Gomez-Diaz (University Gustave Eiffel-LIGM-CNRS, France), Tomas Recio (University Antonio de Nebrija, Spain) Contact: [email protected], [email protected] This work is licensed under the Creative Commons Attribution 4.0 International License (CC-BY 4.0) http://creativecommons.org/licenses/by/4.0/ January 11th, 2024, V1 1 Foreword The goal of this document is to openly contribute with our comments to the Open Letter [1] Establishing a national research software award published in Open Research Europe, that has received two Referee reports [2, 3]. We hope that they can contribute to enrich the discussion on the involved subjects. Moreover, we think that these suggestions could be also useful for the authors to react to the Referees’ comments when they discuss and ask for more precision regarding both the concept of research software presented in the Open Letter, and the proposed evaluation criteria and steps. 2 Introduction As the authors of the references [11, 12, 14, 15, 13, 16, 17], we would like to express our acknowledgment of this interesting French initiative that will undoubtedly help to build an Open Science ecosystem in which research software finds its right place as a research production. This award will help to convey best dissemination practices [11, 12] such as, for example, to put a free/open source license in the software before its dissemination. It will also help to increase the visibility of research software produced within French (research) institutions and establish research software models to be followed, which will help to install and expand an Open Science culture. It will also establish a model to be followed, which is already in its beginning pace with the Australian award mentioned at the end of the Open Letter [1]. On the other hand, we would also like to offer some comments on this Open Letter. We hope that they can contribute to enrich the discussion on the involved subjects. Moreover, we think that these suggestions could also be useful for the authors to react to the Referees’ comments when they discuss and ask for more precision regarding both the concept of research software presented in the Open Letter, and the proposed evaluation criteria and steps. 3 Comments Definition of Research Software In the Defining research software section of [1], it is stated: research software is software designed, maintained, and/or used by scientists and/or research institutions. It is developed to meet a specific need of science, hence it results from research work and/or enables scientific work, which is notably valued by publications before/on/around/with the software. 1 This definition is unclear as it seems to us that it does not delimit a precise subset of software, as software used by scientists and/or research institutions include many different layers of software at the very instant a scientist opens a computer or a phone for its every day work. Would every word processor, used to write scientific papers, qualify as research software? A much more precise definition would be something like: research software is software designed and developed by scientists to meet a specific need of science, hence it results from research work and/or enables scientific work, which is notably valued by the publication of the scientific results obtained with the software. This means that the software that is qualified as research software is a product of the research work, coinciding thus with the research software definition provided in section 2.1 of [14]: research software is a well identified set of code that has been written by a (again, well identified) research team. It is software that has been built and used to produce a result published or disseminated in some article or scientific contribution. Each research software encloses a set (of files) that contains the source code and the compiled code. It can also include other elements as the documentation, specifications, use cases, a test suite, examples of input data and corresponding output data, and even preparatory material. The authors of the Open Letter should clarify the precise formula proposed for the research software concept and put it in perspective with the definition provided in [14]. We would like to remind here that much of the software developed by scientists or by research institutions is simply not maintained, then this qualification should be avoided in the definition, otherwise it will be gambling to define an almost empty set. As not every software produced by scientists is disseminated as free/open source software, or simply disseminated, this should not be a characteristic that enters in the definition of research software, but in the software that is considered within the award context. As remarked by the Referee [3], this point could benefit of more insight about how it was dealt with the software without a license. Evaluation criteria The Referee [3] comments: * Establishing a national award: I see opportunities for facilitating the evaluation of activities related to the production of research software as a research activity, which could be important for a more fair evaluation of research activities in the context of open science. This point is also remarked by the Referee [2]: The Evaluation Criteria section (page 6) could benefit from more explicit information. Which also coincides with the point: * Evaluation criteria of [3]. We also agree with both Referees, and this presentation of the French national research software award will gain in interest if the evaluation criteria could be disclosed. Among other benefits, it will provide a more transparent context that will help potential candidates to improve the conditions to present new research software to the future editions of the award. In particular we would like to compare these award criteria with the CDUR evaluation protocols proposed in section 4 of [14]. CDUR stands for Citation, Dissemination, Use and Research, and proposes these four steps in order to evaluate different aspects of the research software. We submit here a precise question to the Open Letter authors: how the three considered levels “strong support, support, do not support” are related to the basic/good/excellent levels that we do study in the section 4.3 CDUR: a use case step-by-step of [14]. Standing on the shoulders of Giants As this French research software award shows, the situation of research software in France is in a privileged status, where many important points related to this research production are in discussion at the Ministry level. But to arrive to this situation has taken different and previous stages that have not occur in other countries or that are currently in construction, which also shows the French idiosyncratic situation. Among all the pre-existing work, we would like to mention the French DevLOG network1that gathers the software development community. It was launched in 2009 with a website and a mailing list, and 1https://www.devlog.cnrs.fr/ 2 it organised the JDEV2011, the 1`ere ´edition des Journ´ees nationales du DEVeloppement logiciel2in Toulouse (see also the other JDEV editions3). This network was launched by some developers, members of the research community that were already part of the team of the French PLUME project (20072013) [15]. The platform where this project published software descriptions was closed in 2022, which is surely the reason for not to mention it in the Open Letter, nevertheless, descriptions of the project goals, and of the work realized by the PLUME team can be found in [14, 15]. Readers interested to have more insight about this project can consult [5, 6, 7, 8, 9], and [10] (work related to years 2007-2013), and the video [4]. Among all the publications produced within the PLUME project we would like to highlight [13], and the already mentioned [11], others are available at the mentioned link [10]. Data award A possible new version of this Open Letter could also mention the Data award also organized by the French Minister of Higher Education and Research in 20224. Again, we would be interested to have more knowledge about the review criteria process that has been used in this Data award, as we would like to compare it with our proposed evaluation protocols for research data [16, 17]. Note The present comments are also submitted to the Comment section of the Open Letter [1]. Acknowledgements We would like to thank the authors of the Open Letter for such an inspiring work. References [1] Blanc Catala I, Di Cosmo R, Giraud M et al., (2023). Establishing a national research software award [version 1; peer review: 2 approved] Open Research Europe 2023, 3:185 https://doi.org/10.12688/openreseurope.16069.1 [2] Ram K, (2023). Peer Review Report For: Establishing a national research software award [version 1; peer review: 2 approved]. Open Research Europe 2023, 3:185 https://doi.org/10.21956/openreseurope. 17349.r35884 [3] Gonz´alez Barahona JM, (2023). Peer Review Report For: Establishing a national research software award [version 1; peer review: 2 approved]. Open Research Europe 2023, 3:185 https://doi.org/10.21956/ openreseurope.17349.r35882 [4] Archimbaud JL, (2007). Video de la pr´esentation du projet Plume au CC-IN2P3, Lyon. Webcast CCIN2P3. https://webcast.in2p3.fr/video/presentation-du-projet-plume [5] Archimbaud JL, Romier G, (2009). La place des logiciels libres dans l’Enseignement Sup´erieur et la Recherche (ESR), ´etat des lieux `a travers PLUME. Et que font les autres ? https://hal.science/ hal-00559446 [6] Archimbaud JL, Romier G, (2010). Bilan `a trois ans de PLUME : les services rendus aux laboratoires de recherche et aux universit´es. https://hal.science/hal-00559460 [7] Archimbaud JL, (2011). PLUME : Promouvoir les Logiciels Utiles Maˆıtris´es et ´ Economiques dans l’Enseignement Sup´erieur et la Recherche. 10e colloque national en calcul des structures, May 2011, Giens, France. https://hal.science/hal-00592935 [8] Archimbaud JL, (2012). La place du logiciel libre au CNRS : t´emoignage d’un ing´enieur en informatique au CNRS depuis 1984. https://hal.science/hal-00850874 2https://archive-devlog.cnrs.fr/jdev2011 3https://www.devlog.cnrs.fr/jdev/ 4https://www.enseignementsup-recherche.gouv.fr/fr/prix-science-ouverte-des-donnees-de-la-recherche-86179, https://www.ouvrirlascience.fr/remise-des-prix-science-ouverte-des-donnees-de-la-recherche-2023/ 3 [9] Gomez-Diaz T, (2007-2013). Le th`eme PLUME Patrimoine logiciel d’un laboratoire, Zenodo Community, https://zenodo.org/communities/plume-patrimoine-logiciel-laboratoire/ [10] Gomez-Diaz T, (2007-2024). Page web with presentations, articles, posters and other productions since 2007, Laboratoire d’informatique Gaspard-MOnge (LIGM), http://igm.univ-mlv.fr/~teresa/ logicielsLIGM/ [11] Gomez-Diaz T, (2010). Diffuser un logiciel de laboratoire : recommandations juridiques et administratives, Publication for the French PLUME project (2007-2013), https://zenodo.org/record/7096216 [12] Gomez-Diaz T, (2014). Free software, Open source software, licenses. A short presentation including a procedure for research software and data dissemination, Zenodo, Preprint, Septembre 2014, http://zenodo.org/record/11709/. Presented at the Workshop on open licenses: Data licencing and policies, EGI Conference 2015, Lisbon, May 2015, https://indico.egi.eu/event/2452/sessions/ 1522/#20150520. Spanish version: Software libre, software de c´odigo abierto, licencias. Donde se propone un procedimiento de distribuci´on de software y datos de investigaci´on, Zenodo, Preprint, Septembre 2015, https://zenodo.org/record/31547/ [13] Gomez-Diaz T, (2015). Article vs. Logiciel : questions juridiques et de politique scientifique dans la production de logiciels. 1024 - Bulletin de la soci´et´e informatique de France, N. 5, 2015, http://www.societe-informatique-de-france.fr/wp-content/uploads/2015/04/ 1024-5-gomez-diaz.pdf, also available at https://hal.science/hal-01158010. First version published for the French PLUME project (2007-2013) in 2011, https://zenodo.org/record/7063154 [14] Gomez-Diaz T and Recio T, (2019). On the evaluation of research software: the CDUR procedure [version 2; peer review: 2 approved]. F1000Research 2019, 8:1353, https://doi.org/10.12688/ f1000research.19994.2 [15] Gomez-Diaz T, (2019). Le Projet PLUME et le paysage actuel des logiciels de la recherche dans la science ouverte. Zenodo, Preprint, 2019. https://zenodo.org/record/2591474 [16] Gomez-Diaz T and Recio T, (2022). Research Software vs. Research Data I: Towards a Research Data definition in the Open Science context, F1000Research 2022, 11:118, https://f1000research.com/ articles/11-118/v2 [17] Gomez-Diaz T and Recio T, (2022). Research Software vs. Research Data II: Protocols for Research Data dissemination and evaluation in the Open Science context, F1000Research 2022, 11:117, https: //f1000research.com/articles/11-117/v2 4