Customising FAIR EVA: Lessons learned on building a FAIR assessment plugin tailored to your repository.

Customising FAIR EVA: Lessons learned on building a FAIR assessment plugin tailored to your repository Implementation Story 1 Implementation Story - #3 Authors FIDELIS authors ٩Alessandra Polimeno, Data Archiving and Networked Services (DANS) Participants co-authors ٩David Schimmel, RWTH Aachen University ٩Gregor Žibert, Slovenian Social Science Data Archives ٩Piotr Krajewski, Gdańsk University of Technology ٩Vladimir Otasevic. Belgrade University Computer Centre (RCUB) ٩Marija Ristic, Belgrade University Computer Centre (RCUB) Lead Expert of the Support Offer: ٩Fernando Aguilar Gómez, Spanish National Research Council Related TTRAM element(s) Curation, Quality & Compliance; Access, Reuse; Interoperability Support offer Support for the Adoption of Solutions | Develop your own FAIR EVA (Evaluator, Validator & Advisor) plugin Abstract The FAIR EVA support offer introduced repository managers and technical staff to the FAIR EVA evaluator, validator and advisor tool and guided them through the development of a custom plugin tailored to their repository. Participants learned to adapt the code to integrate additional metadata schemas. Although the plugin development took considerable technical effort, it deepened participants’ understanding of their metadata practices and provided a solid starting point for the integration of the tool into their repository. 2 Implementation Story - #3  Introduction The FAIR EVA support offer consisted of a series of webinars and hands-on workshops aimed at repository managers, administrators, technical staff, and data stewards. There were a total of four sessions that took place over the course of two months. The main goal was to familiarise participants with FAIR EVA, a tool that evaluates, validates, and advises on the FAIR (Findable, Accessible, Interoperable, Reusable) compliance of digital objects stored in repositories or data portals. FAIR EVA not only assesses FAIRness but also offers guidance on improving both data and metadata. The first session demonstrated the tool’s basic function in which the user is given a report on how well a dataset complies with various FAIR Data principles. For example, the tool checks whether the data uses a proper format, whether the metadata is machine-actionable, or whether Persistent Identifiers (PIDs) are provided by an accepted authority. In later sessions, customisation options were demonstrated and replicated by the participants. There are varying levels of customisation options, with the most extensive options becoming available in the development of a plugin tailored to the repository’s domain. The support offer took place from October 2 to November 27, 2025. users Participants 4 participants joined this support offer with teams of various sizes, culminating in a total of 5 individuals supported. The following organisations were represented: ٩Slovenian Social Science Data Archives (ADP) | Slovenia | Social sciences; Engineering and technology; Humanities | ADP hoped to deepen their understanding of how the FAIR EVA tool functions, to systematically and transparently assess the FAIRness of digital objects and metadata within the Dataverse repository. The practical training helps to develop concrete technical skills on installation, configuration, and plugin development guided by the support team helping with any technical issues. ٩RWTH Aachen IT Center | Germany | Engineering and technology; Discipline-agnostic | RWTH Aachen IT Center aims to make research data FAIR, implementing various standards such as Fair Digital Object or Fair Data Point. However, a consistent tool to assess the FAIRification of our services based on sets of criteria is currently lacking. Therefore, developing their own FAIR EVA plugin may help improve the research support process in the German scientific landscape. ٩Belgrade University Computer Centre (RCUB) | Serbia | Engineering and technology | RCUB is the technical support team for research repositories in Serbia. Learning more about FAIR assessment of repository data and possible customisation opportunities can have a useful impact on these repositories. ٩Gdańsk University of Technology | Poland | Engineering and technology | Gdańsk participated in the support offer in an observing role, not completing the full scope of work but gathering knowledge of the tool and experimenting with applying it to their services.  Approach taken The solution offered consisted of the development of a plugin with guidance and mentoring from the main FAIR EVA developer. Not every discipline has the same idea on how to best apply the FAIR Principles, and may have different views on how research data is best managed. The development of a customised plugin can incorporate these differences by changing the weights of the FAIR indicators as specified by the Research Data Alliance. The plugin can also specify and take into account which metadata schemas, controlled vocabularies and ontologies, default and optional properties and repository services or policies are available to the user. These customisation options allow for a flexible solution that can be adapted to any domain or repository, thus enabling the automatic assessment of the repository’s FAIRness and finding where it could be improved. The sessions started with a demonstration of the tool, alongside a look under the hood. The GitHub repository that holds the code is openly available to everyone, and the participants were instructed to adapt the existing code for their customised plugin. Most participants worked on the inclusion and mapping of metadata schemas that are used in their repository but not by default offered by the tool, such as DDI. We found that the addition of these metadata schemas significantly improved the scores of the FAIR indicators and made the generated report more useful. 3 Implementation Story - #3 alarm-exclamation Challenges and lessons learned Throughout the sessions there was room for the exchange of experiences and feedback, but time was reserved explicitly in the last session, where participants were asked to present their results and discuss challenges and achievements. The most common challenge was that fully customising the tool to a repository takes time and resources that are not always widely available. The developer working on the customisation needs technical skills as well as knowledge of metadata schemes and standards. Especially when a repository makes use of a heterogeneous set of metadata schemes, creating a mapping can be time-consuming and error-prone. ٮSlovenian Social Science Data Archives: Adapting different metadata formats and schemas requires substantial development effort: Our experience showed that cross-schema mappings are complex, and ensuring accuracy is challenging. Small inconsistencies or missing elements can significantly affect FAIR indicator scores. For small archives without dedicated IT staff, integrating or customising FAIR assessment tools becomes particularly difficult. While initial plugin tweaks are manageable, long-term maintenance, troubleshooting, and support for additional metadata formats demand ongoing technical expertise that small teams often cannot sustain. ٮClear, detailed explanations of missing metadata elements were extremely valuable: The tool’s transparency, showing precisely which fields were incomplete or absent improved our understanding of FAIR expectations. Despite technical limitations, the process was a highly informative learning experience: Working with various metadata formats, examining schema compatibility, and observing how FAIR indicators react to metadata quality offered meaningful insights into our systems and workflows. This hands-on exploration helped deepen our understanding of both strengths and gaps in our current metadata practices. ٮRWTH Aachen IT Center: The main obstacles for us were: 1) Our service Coscine is a collaborative research data and metadata management platform for researchers of all fields. Therefore, the need to be able to adapt to a wide range of user-defined metadata profiles for which the tool did not seem to be developed out of the box. 2) The multi-layered implementation of FAIR Digital Objects (FDO) made gathering and accessing all relevant metadata and the access to the ground entities difficult. We learned how to adapt the tool to generate a report based on a shared subset of properties for our metadata records but are also more informed now about design requirements to make more FAIR maturity indicators automatically testable. On the other hand, we also confirmed that by the design of our service, several indicators are, by design, satisfied. ٮGdánsk University of Technology: Bridge of Knowledge is not a standalone repository but a multi-module platform linking datasets with publications, projects, and researcher profiles. This created both challenges and opportunities. The main technical work focused on funding validation: the plugin follows links from dataset pages to project records within the platform, extracting funder name, award number, and award title to verify provenance. For NCN-funded datasets (Polish National Science Centre), it checks CC BY 4.0 or CC 0 license compliance. Building this crossreferencing caused the plugin to be sensitive to page structure changes. DataCite XML and HTML sources sometimes contain funding in different formats, so we implemented fallback extraction. Testing revealed real issues - datasets with incomplete funding fields or mismatches between declared and linked project information. ٮBelgrade University Computer Centre (RCUB): We manage and provide IT support for numerous institutional and thematic repositories. Identifying a scalable solution capable of offering customizable configurations that accommodate different metadata schemas was therefore of particular importance to us. The main challenge we faced concerned the diversity of metadata schemas. Specifically, we considered whether it would be more appropriate to harmonize export schemas across different repository OAI interfaces or to customize the developed plugins individually for each repository. We would like to see further development of this tool, particularly in terms of providing comprehensive summaries and aggregated statistics at the level of the entire repository. shield-check Impact All participants started the creation of their own plugin, or at least a proof of concept, as customising the tool to work with all metadata schemas and controlled vocabularies takes some effort to define. For a smaller repository it became clear during the process that the resources needed to develop a production-ready plugin were currently disposable. Another participant noted that their repository may use FAIR EVA not only for the assessment of data but for other things in their service as well. 4 Implementation Story - #3 ✅ Key messages and advice For Trustworthy Digital Repositories it is essential to have a way to assess and improve the FAIRness of the (meta)data that it houses. Developing a plugin of the FAIR EVA tool that is tailored to the repository’s services is a great way to get this ongoing evaluation started. Some key messages to take into consideration when adopting this approach: ŴSlovenian Social Science Data Archives: Assess your technical capacity early and choose solutions that align with your available resources. If your organisation lacks dedicated IT staff, be realistic about how much customisation and long-term maintenance you can support, and prioritise tools that work well out of the box rather than those requiring ongoing development. Use evaluation tools that offer transparent, indicator-level feedback. Clear explanations of low scores make it much easier to learn, improve metadata quality, and build internal understanding without significant technical effort. ŴRWTH Aachen IT Center: When developing FDOs, it is important to design the metadata schemata such that all required data and metadata objects are accessible without requiring additional knowledge that cannot be accessed via the metadata records themselves. When working with partially restricted digital objects, it might be a worthwhile use case to use a FAIR evaluation tool to encourage users to make their data more publicly available by demonstrating the impact on the FAIR evaluation of their FDOs. ŴGdánsk University of Technology: FAIR EVA’s plugin architecture allows implementing custom indicators beyond standard RDA metrics. We added a domain-specific test for funding validation that would not be possible with generic tools. The framework supports multiple metadata sources – for instance, our plugin combines Schema.org JSON-LD extraction with DataCite XML content negotiation, using whichever provides better data. The customisation effort is substantial but results in assessment tailored to your repository’s actual metadata practices. ŴBelgrade University Computer Centre (RCUB): During the sessions, we successfully developed two plugins for our repositories. Thanks to the clear guidelines, detailed video tutorials, and continuous support provided by the organizers, we experienced no difficulties in deploying FAIR EVA via Docker. At an early stage, prior to customization, we observed that certain indicators did not accurately reflect the actual state of deposited records. After examining the available configurable parameters in more detail, we identified the necessary adjustments in the FAIR EVA configuration. We see significant potential for further integration of this tool, enabling repositories to present and reuse FAIR EVA assessment results. Explore other FIDELIS training and support opportunities! link eden-fidelis.eu bluesky fidelis-project.bsky.social linkedin company/fidelis-project-eu youtube @FIDELIS-PROJECT communities/fidelis-project