Jupyter4NFDI - Proposal for the Integration Phase of Base4NFDI

Jupyter4NFDI A central JupyterHub providing access to various software stacks and computing and data resources across the NFDI consortia Proposal for the Integration Phase of Base4NFDI Submitted: April 23rd, 2025 On behalf of: WG Research Software Engineering, Section Common Infrastructures Base4NFDI proposal form – 11/2024 – Integration page 2 of 42 1 General Information •Name of proposed Basic Service (in English) A central JupyterHub for the NFDI •Acronym of the proposed Basic Service Juypter4NFDI •Service "subtitle" explaining key functionality A central JupyterHub providing access to various software stacks and computing and data resources across the NFDI consortia •Corresponding NFDI Section Common Infrastructures •Lead institution Forschungszentrum Jülich GmbH, 52425 Jülich •Name of lead institution principal investigator Björn Hagemeier <[email protected]> Base4NFDI proposal form – 11/2024 – Integration page 3 of 42 •Participating institutions Principal Investigator Institution, location Contact E-mail Member in [consortium] Funding requeste d [yes | no] Björn Hagemeier Forschungszentrum Jülich GmbH, Jülich Supercomputing Centre (JSC) b.hagemeier @fz-juelich.de Text+ Yes Arnim Bleier GESIS – Leibniz-Institute for the Social Sciences arnim.bleier @gesis.org NFDI4DS Yes Bernd Flemisch University of Stuttgart (US) bernd.flemisch @iws.unistuttgart.de NFDI4ING Yes Matthias Lieber Technische Universität Dresden (TUD), Center for Information Services and High Performance Computing matthias.lieber @tu-dresden.de NFDI4DS Yes Klaus Reuter Max Planck Computing and Data Facility (MPCDF) klaus.reuter @mpcdf.mpg.de FAIRmat No George Dogaru Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) george.dogaru @gwdg.de Text+ Yes Daniel Mietchen FIZ Karlsruhe – Leibniz-Institut für Informationsinfrastruktur Daniel.mietchen@ fiz-karlsruhe.de MaRDI Yes Sheeba Samuel Technische Universität Chemnitz (TUC) sheeba.samuel@i nformatik.tuchemnitz.de No No Table 1: List of participating institutions Base4NFDI proposal form – 11/2024 – Integration page 4 of 42 •Initialisation Phase The initialisation phase started on June 1st, 2024 and will last until September 30th, 2025. •Planned duration of the integration phase We ask for an immediate start of the integration phase on October 1st, 2025 and envision a two year duration until September 30th, 2027. •Statement on efforts required by consortia for integration of service The central service should be immediately usable by most consortia. However, some aspects may be considered when tailoring the service for their specific requirements, primarily the integration of consortia-specific libraries and software packages. The platform has been enabled to include these requirements during the initialisation phase, where a feature to support the execution of custom Docker images and Binder-ready software repositories in the Jupyter environment has been introduced. This will be the baseline of support. Similarly, the enabling of custom software stacks on specific resources, e.g. HPC software modules, is also supported. Depending on prior usage of such environments within the consortium, such a setup would have to be transferred to the resources. Secondly, depending on the consortium-specific entitlements to the usage of specific resources, an integration with the community AAI beyond mere authentication may be required. Jupyter4NFDI is ready to support various scenarios in this area in collaboration with IAM4NFDI. •Summary of the proposal in English and German Deutsch: Jupyter ist ein wichtiger Pfeiler, um nachvollziehbare und reproduzierbare wissenschaftliche Ergebnisse zu erreichen. Ebenso bietet die Plattform Zugang zu diversen Rechenund Datenressourcen, die über einen einheitlichen Einstiegspunkt, den JupyterHub, für die wissenschaftliche Gemeinschaft in Deutschland und darüber hinaus erreichbar sind. Während der Initialisierungsphase des Services wurden mehrere vorbereitende Ziele erreicht, die nun weitergehend in den NFDI-Konsortien, einzelnen Anwendungsfällen und wissenschaftlichen Communities integriert werden können. Ebenso können weitere externe Ressourcen hinzugefügt werden. Beides soll durch Aufrufe für Inkubatorprojekte erreicht werden, die im Projektverlauf gemeinsam mit den Interessenten durchgeführt werden. Des Weiteren werden wir den Dienst als Reaktion auf Anfragen aus den Inkubatorprojekten weiterentwickeln. Dieser Ansatz hat sich bereits in der Anfangsphase bewährt und zu Funktionen wie der repo2docker-Integration, der Verwendung benutzerdefinierter Docker-Images und Freigabelinks für Jupyter-Konfigurationen geführt. Auf dieser soliden Basis werden wir Inhalte generieren, die im Rahmen eigener Trainings von Nutzenden und Multiplikatoren in den Communities verwendet werden. Die Inhalte befassen sich mit den FAIR-Prinzipien und wie diese Base4NFDI proposal form – 11/2024 – Integration page 5 of 42 mit Hilfe von Jupyter umgesetzt werden können. Dabei werden auch fachspezifische Informationen und bewährte Methoden aus den Konsortien integriert. English: Jupyter is an important pillar for achieving comprehensible and reproducible scientific results. The platform also provides access to various computing and data resources that can be accessed via a single entry point, the JupyterHub, for the scientific community in Germany and beyond. During the initialisation phase of the service, several preparatory goals were achieved, which can now be further integrated in the NFDI consortia, individual use cases and scientific communities. Likewise, further external resources can be integrated. Both will be achieved through calls for incubator projects, which will be carried out together with interested parties during the course of the project. Furthermore, we will continue to develop the service in response to requests from researchers, consortia and, particularly, the incubator projects. This approach has already proven successful in the initial phase and has led to features such as the repo2docker integration, the use of custom Docker images and shareable links for Jupyter configurations. On this sound basis, we will generate content that can be used for end-user and multiplier training in the communities. The content addresses Jupyter itself, the FAIR principles and how they can be implemented using Jupyter. Domain-specific information and best practices from the consortia will also be integrated. 2 Summary of Initialisation Phase Results 2.1 Change in Background and Motivation since the Start of the Initialisation Phase The TEC evaluation of our proposal for the Initialisation Phase mentioned two major concerns revolving around policies for resource access and key performance indicators to assess effectiveness and utility of the basic service. We will try to respond to these in the following. It is important to distinguish access to the Jupyter service and access to the resources behind it. There is a tendency to confuse these two, and users who have access to the service, e.g. by being able to log in, may falsely believe that this should grant them access to all resources connected to the service. We agree that the access policy and how to apply for more resources must be clearly defined and communicated. A dedicated section within the service’s documentation has been created to address this situation and will be extended as the service develops and more resources will be added to it.1 The definition of appropriate KPIs is part of this proposal as requested by the template and will be dealt with in section 3.1.5. 1 Resource access and available systems https://nfdi-jupyter.de/features/#1-systems-available Base4NFDI proposal form – 11/2024 – Integration page 6 of 42 Furthermore, all consortia were asked about their current needs with respect to the service, their intention to pursue an own development, and general comments about the Jupyter4NFDI service and approach. This provided some valuable insights about consortia requirements that we would like to respond to in this proposal. We also received feedback from consortia after submission of the proposal for the initialisation phase. Several of them indicated requirements from the areas of data access, custom containers, education and training, and FAIR principles. Further requirements mentioned by individual responses referred to IAM, federation, resource access, parallelisation, PIDs, workflows, legal framework and privacy. This is also supported by the results of the survey that we conducted between December 2024 and January 2025, cf. requirements analysis D1.4.1 in section 2.2.1. Some of these aspects are already covered in the existing basic service today, whereas others will receive attention during the integration phase. For example, custom containers can already be used within the service today, offering great flexibility to users. A generic data access mechanism is under development. 2.2 Results of Initialisation Phase The central service2 has been put in place already in July 2024. Initially, it provides access to IaaS Cloud resources at Jülich Supercomputing Centre (JSC), co-located with the central hub. By popular demand, the hub configuration has been extended with the following features: •Share links3 allow to reference ready-to-use Jupyter configurations. •Custom Docker images4 allow to provide and use specific software environments that would otherwise not be covered by the default settings. •The Repo2Docker5 functionality facilitates building Docker images directly from GitHub, GitLab, Zenodo or other sources. An additional environment for the preparation of new resource integrations or feature evaluations has been made available6. Among other things, it currently provides access to resources at Max Planck Computing and Data Facility (MPCDF). Similarly, cloud resources from our partner TU Dresden have been made available and the integration of High-Performance Computing (HPC) resources of the Verein für Nationales Hochleistungsrechnen (NHR)7 is in preparation. 2 Central JupyterHub https://hub.nfdi-jupyter.de/ 3 Share link documentation https://nfdi-jupyter.de/users/misc/#share-button 4 Custom Docker images https://nfdi-jupyter.de/users/jupyterlab/customdockerimage/ 5 Repo2Docker https://nfdi-jupyter.de/users/jupyterlab/repo2docker/ 6 Staging environment https://hub-staging.nfdi-jupyter.de/ 7 Verein für Nationales Hochleistungsrechnen https://www.nhr-verein.de/ Base4NFDI proposal form – 11/2024 – Integration page 7 of 42 Towards the end of 2024, we have published a survey targeted at potential stakeholders of our infrastructure. Despite the fact that Jupyter is an established service in many communities, our intention is to have the best possible coverage of relevant use cases. The project website8 has been put in place, providing information about the infrastructure. The domain had previously been used to host an inventory of available Jupyter services within the NFDI, which is still available as part of the site. Based on input from various user groups, the basic service appears to be already usable for their use cases. Usage statistics of the service from the initial release until recently indicate a growing interest in using the service9. 2.2.1 Interim Report on Requirements for Finalisation of the Initialisation Phase a) Requirements Analysis (D1.4.1) Due: 3 months after start date Percent finished: 100% Status: report / finished Overview of relevant outcomes: A survey was opened in December 2024 and ran until January 9, 2025. 75 participants from 53 German research institutions provided their answers. The survey responses revealed awareness of and interest in Jupyter4NFDI across diverse institutions. Participants emphasized the platform’s value in lowering technical barriers, enabling reproducibility, and offering a reliable alternative to local setups. The results highlight the broad relevance of Jupyter4NFDI and support further expansion and user-driven development during the upcoming integration phase. The results of this survey have been made available in two forms. First of all, we have a summary document that is hosted as a GitHub page10. On this GitHub page, we link to our very own Jupyter service with a launch button, leading to an interactive execution of the summary generation. This can be used for a more thorough analysis or query the survey data about new questions. b) Software Evaluation (D1.4.2) Due: 6 months after start date Percent finished: 100% 8 Jupyter4NFDI project web site https://nfdi-jupyter.de/ 9 Jupyter4NFDI usage statistics https://zenodo.org/records/15187218 10 Survey result summary https://gesiscss.github.io/Jupyter4NFDI_survey_results/Jupyter4NFDI%20Survey %20Summary.html Base4NFDI proposal form – 11/2024 – Integration page 8 of 42 Status: finished Overview of relevant outcomes: The central part of the software stack for the Jupyter4NFDI service is a quite natural choice, as Jupyter is taken for granted. There are multiple ways in which a central JupyterHub can be deployed. One of them is a Kubernetes (K8S) environment. Whereas this is not essential for the end users, it is still useful during the operation of a large-scale JupyterHub, because this deployment model provides for some level of fault-tolerance without the intervention of administrators, leading to higher uptimes and a more reliable service. The two central pieces of software, of which rudimentary versions were available at the onset of the project, are the JupyterHub Outpost11 and the corresponding JupyterHub OutpostSpawner12. These components provide for a flexible solution involving multiple types of resources that can all be accessed from the single hub. Outlook: There are individual software solutions appearing regularly that share some functionality with Jupyter. In some cases, these can even be integrated with the central service to complement the overall functionality. At other times, they are simply comparable solutions. We will monitor the landscape to identify opportunities. Notable candidates for this are VSCode, R-Studio, MATLAB and XPra, to name a few. c) Service Design (D1.4.3) Due: 6 months after start date Percent finished: 100 Status: report / finished Overview of relevant outcomes: The service design has been documented as part of the project web site and user documentation. Relevant links can be found in the appendix (OpenProject status report). Outlook: Additional requirements from NFDI consortia or other users will be monitored and, if need be, will influence the service design and further development of the service. d) Service Prototype (D1.4.4) Due: 12 months after start date Percent finished: 100% Status: finished 11 JupyterHub Outpost: https://zenodo.org/records/14616149 12 JupyterHub OutpostSpawner: https://zenodo.org/records/14616171 Base4NFDI proposal form – 11/2024 – Integration page 9 of 42 Overview of relevant outcomes: the central hub has been deployed13 and documentation for users and interested resource providers is available at our central website14. Outlook: maintenance and further development of the service will be the subject of WP1 described in section 5.2.1. e) Service Piloting and User Testing (D1.4.5) Due: 12 months after start date Percent finished: 50 Status: running Overview of relevant outcomes: The service has been tested in several consortia regarding its applicability for end users. At the same time, we are in the process of onboarding additional external resources. More details can be found in the attached OpenProject report. Outlook: For an accepted service like this, there is a natural feedback loop with users establishing requirements towards the service. In addition to the regular consultation hours for external resource providers, we will also offer separate consultation hours for end users of our service in the future. The idea is to have a low entry barrier for users and maybe discuss prepared topics arising from specific requests that may be of broader interest. 2.2.2 Other During the first nine months of the project, members of the project team attended several workshops and events to present their ideas and the current status of the project. On July 15, Jupyter4NFDI was presented within an NFDI Talk15. The project was also represented at the 1st Base4NFDI User Conference in November, and at the Base4NFDI Roadshow in December. Project representatives also visited consortia meetings of non-participating consortia. For example, Jupyter4NFDI was presented at the MaRDI Service Marketplace during the MaRDI annual meeting in Kaiserslautern in November 2024 and the Text+ Show and Tell series of talks in January 2025. 2.3 Update on Technical Readiness Level (TRL) of the Proposed Basic Service During preparations for this proposal, the TRL levels of several key components have been reassessed based on their stability, functionality, and integration into the production environment. •JupyterHub @ JSC and OutpostSpawner has been upgraded from TRL 7 to TRL 8. The system has demonstrated its stability beyond the prototype phase and has proven to be a reliable solution for the project's evolving needs. 13 Central JupyterHub https://hub.nfdi-jupyter.de/ 14 Juypter4NFDI web site: https://nfdi-jupyter.de/ 15 NFDI Talk Jupyter4NFDI: https://zenodo.org/records/14162496 Base4NFDI proposal form – 11/2024 – Integration page 16 of 42 funding for physical resources, primarily storage and servers, and therefore we consider a central basic service favourable in the competition for available resources and their integration. Furthermore, an evaluation of the legal challenges of the central service led to the insight that a central service is also preferable over many fragmented services, because if users turn to consortia-specific, local, or even external (potentially untrusted) services instead of Jupyter4NFDI, this could bring more problems and complications. Also, a fragmented system leads to inconsistencies, less interoperability, and potential security risks. 4 Support Actions from Base4NFDI / NFDI Sections, and Integrating NFDI Consortia / Efforts (max 1 page) Support from Work package / Description of contribution Contact person Basic Service Base4NFDI Support organising workshop/training, promotion André Giesler, [email protected] Base4NFDI Training TA 4 Development of training material Antje Manske Base4NFDI User research and evaluation for Jupyter4NFDI products and prototypes Tianying Chen, GESIS, [email protected] Brigitte Mathiak, GESIS, brigitte.ma[email protected]g EduTrain/DALIA Hosting of domain specific training materials Sonja Herres-Pawlis RDMTraining4NFDI Integration of training material with Jupyter4NFDI, e.g., direct usage links, trainings and workshops on the hub Mareike Wohltmann nfdi.software Integration of software with Jupyter4NFDI, e.g., direct usage links Marc Hanisch Knowledge Graph Infrastructure FAIRJupyter Knowledge Graph, use of Jupyter notebooks for exploring NFDI knowledge graphs Daniel Mietchen Table 2: Support needed from Base4NFDI / Service Stewards / Section Support from Involved effort Consortium (contact) NFDI4DS Users of JupyterHub and Sonja Schimmler, Base4NFDI proposal form – 11/2024 – Integration page 17 of 42 connected resources. Provider of ecosystem of re-usable materials. [email protected] ofer.de Text+ Content provider, maintenance of consortium’s Fluffy Import. George Dogaru NFDI4ING Shared task “Jup4ING”. See LoC for details. Peter Pelz, [email protected] MaRDI Assessment of computational reproducibility of mathematical software. Daniel Mietchen Table 3: Contributions required from the integrating consortia 5 Work Programme (ca. 6 pages) 5.1 Overview of Work Packages The project is divided into five work packages only partially overlapping with work packages from the initialisation phase. WP1 will be largely dedicated to the maintenance and development of the basic service, comparable to WP1 of the initialisation phase. The resource manager interface has been defined and its maintenance and updates will also be covered by WP1. The new WP2 will deal with the integration into the AAI landscape. Part of this will already happen towards the end of the still ongoing initialisation phase, but we do expect a more fine grained authorisation infrastructure for resource access to be required by some communities. WP3 will be about the integration of additional resources to the basic service. In WP4, we will create user-oriented training material as plain documentation as well as exemplary setups or Jupyter Notebooks for various user groups. WP5 remains dedicated to dissemination and the integration into the wider Jupyter ecosystem. We provide details about the work packages in the following sections. Work package Deliverables (D) and milestones (M) Responsible partner and participants 1. Maintenance and development of service D1.1 Persistent import API D1.2 BinderHub instance as part of the MyBinder federation D1.3 FAIRJupyter deployed M1.1 All components for Content integration are available M1.2 Parameterised execution of JupyterLab JSC GESIS, FIZ Base4NFDI proposal form – 11/2024 – Integration page 18 of 42 M1.3 Provide public available KPIs 2. AAI integration M2.1 Integration of IAM infrastructure proxy M2.2. Integration of consortium specific permission structures D2.1 Documentation of entitlements and permissions D2.2 Updated documentation of entitlements and permissions GWDG JSC, TUD 3. Integration of additional resources D3.1 Extended partner tutorials D3.2 Hands-on support in onboarding external resources M3.1 Integration template repository MPCDF TUD, US 4. Content integration D4.1 Use cases featuring fully computationally reproducible content D4.2 Domain-independent training materials D4.3 Domain-specific adaptation of the materials M4.1 Initial version of the materials available M4.2 Final version of the materials available GESIS FIZ, GWDG, TUD, US, TUC 5. Community engagement and communication M5.1 First call for incubator projects D5.1 Jupyter4NFDI training material for RDMTraining4NFDI M5.2 First workshop “Make your work reproducible with Jupyter4NFDI” D5.2 Presentation material for roadshows, consortia meetings, and others US FIZ, GESIS, GWDG Table 4: Overall work programme with work packages, deliverables, milestones, and responsible partner. A Gantt chart of the milestones andn deliverables is provided in section 5.2.6. 5.2 Detailed Work Programme 5.2.1 WP1: Maintenance and Development of Service WP lead: JSC Contributors: FIZ, GESIS The primary objective of this work package is to ensure the technical maintenance and development of the core service infrastructure, which includes the central JupyterHub and the JupyterHub Outposts. This work package is crucial for maintaining operational stability and Base4NFDI proposal form – 11/2024 – Integration page 19 of 42 adapting to evolving requirements (D1.1, M1.1). It involves addressing technical issues, implementing updates, and supporting the JupyterHub Outpost interface. WP1 will focus on several key aspects, beginning with the regular monitoring and optimisation of service performance to ensure reliability. Any technical issues that arise will be promptly addressed to minimise downtime. Additionally, audits of the service's health will be conducted regularly, and fixes will be implemented as necessary. We will also build on the already existing KPIs29 and basic performance metrics that have been established. Through integration and collaboration with IAM4NFDI, we aim to improve these indicators further. The KPIs will be made publicly available in order to increase transparency and to track the growing use and anchoring of Jupyter4NFDI in the NFDI communities (M1.3). A significant component of WP1 is the maintenance and improvement of the JupyterHub Outpost interface and installation, which plays a crucial role in managing infrastructure resources. The work will ensure that the interface remains functional and up to date, adapting to any changes in underlying infrastructure or APIs. This includes incorporating improvements that streamline resource allocation and enhance user experience. Another focus for WP1 is applying software patches and updates to keep dependencies secure and compatible. As new features become necessary to align with project goals, they will be integrated thoughtfully to enhance functionality and usability. One of these new features will be the parameterised start of JupyterLab. This will allow users to configure variables and add them to a specific JupyterLab configuration (M1.2). Alongside technical updates, WP1 will emphasise maintaining comprehensive documentation of the service architecture, configurations, workflows, and changes. Another significant aspect of WP1 is the provision of a dedicated staging instance. This environment is designed to allow external resource providers to thoroughly test the integration of their resources before transitioning to the production environment of Jupyter4NFDI. The staging instance ensures that new integrations are seamless, stable, and fully compatible with the existing infrastructure. By offering a safe and controlled space for testing, this initiative minimises potential disruptions in the production environment and supports a smooth onboarding process for new resource providers. WP1 anticipates challenges, such as managing service downtime during updates and adapting to infrastructure changes. These will be mitigated by scheduling updates during off-peak hours, implementing fallback mechanisms, and fostering regular collaboration with infrastructure teams. Alongside the maintenance and development of the core services of Jupyter4NFDI, the work package will also ensure that complementary services are integrated into the overall Jupyter4NFDI 29 Jupyter4NFDI usage statistics https://doi.org/10.5281/zenodo.15187218 Base4NFDI proposal form – 11/2024 – Integration page 20 of 42 landscape. This includes making a BinderHub instance available as part of the MyBinder federation (D1.2), or integrating the reproducibility assessment workflows of FAIRJupyter into Jupyter4NFDI (D1.3). We will continuously monitor the evolving ecosystem of tools and services, and integrate the ones that provide clear benefits to our users. By focusing on these areas, WP1 will provide the foundational technical support required to maintain the stability and functionality of the service while ensuring it evolves in line with project objectives. 5.2.2 WP2: AAI Integration WP lead: GWDG Contributors: JSC, TUD The work on integrating Jupyter4NFDI with IAM4NFDI is ongoing and happens in coordination with the IAM4NFDI project. The main goal is to extend the reach of Jupyter4NFDI so that it is usable via all community AAIs which are part of IAM4NFDI, so basically the availability across all NFDI consortia is intended. The central Jupyter4NFDI service, which was made available in a short time after the start of the project’s initialisation phase, is already accessible via the Helmholtz AAI. Its role is to integrate resources from participating institutions and enable users to run Jupyter notebooks on infrastructure (JupyterHubs) available to them according to their access rights. At a technical level, the communication required to delegate work to the available hubs is realised by JupyterHub Outpost, which is well documented, including the options, adaptations, and customisations relevant for JupyterHub providers in order to communicate with the central hub and with the (current) AAI. This allows for a fine-grained control over the access to resources according to information received from the AAI. When the NFDI Infra Proxy becomes available, the required adaptations to the service will be made in order for this to be used instead of the current AAI. In the time before this switch (and if necessary, also after), the focus will be on specifying requirements for the NFDI AAI based on the use cases that need to be or to remain supported. This can include adaptations to, and requirements for, the infrastructure proxy (M2.1), but also the community AAIs (M2.2). The metadata supplied by the NFDI AAI can for example be used to allow customized access to resources depending on affiliation to a consortium or to a dedicated user group. Determining and refining appropriate AAI metadata will happen in cooperation with the IAM4NFDI service. It will also be possible to leverage the metadata in order to gain better insight into how different consortia or groups use the service and to derive relevant KPIs from it. Base4NFDI proposal form – 11/2024 – Integration page 21 of 42 At a conceptual level, the work will involve translating the AAI concepts in appropriate options for the service, in formulating requirements of the service for the AAI, in helping to harmonise the options of the central service and the existing JupyterHubs and their AAIs. Another outcome of the work package will be a documentation of the AAI options and aspects of the service (D2.1, D2.2). 5.2.3 WP3: Integration of Additional Resources WP lead: JSC Contributors: MPCDF, TUD, US In the project’s initialisation phase, we established foundational steps, including the development of reference templates and initial documentation for resource APIs that facilitated essential external resource integration. Building on this groundwork, WP3 will focus on collaborating with external partners to integrate their resources into the Jupyter4NFDI ecosystem. This work includes guiding partners through testing on the staging instance to ensure compatibility and stability before moving to production. WP3 will also capture lessons learned and potential challenges to further refine the integration process, ultimately enhancing the overall service. In particular, we plan to offer a variety of channels that provide extended partner-centric tutorials (D3.1) and updated integration templates (M3.1) enabling resource partners to set up their service infrastructure with a high degree of self-sufficiency as well as supporting hands-on sessions (D3.2) from the Jupyter4NFDI team. Our next-phase documentation (D3.1) will incorporate feedback from early adopters, offer deeper technical guidance, and present best practices learned from real-world integrations, addressing common pitfalls and advanced scenarios. These refinements will accommodate a wider variety of environments, orchestration frameworks, and networking scenarios, ensuring that future partners have a smoother and more scalable path for integration. Ultimately, it all comes together in an integration template repository (M3.1) which will offer readyto-run solutions for resource partners. To achieve this, we will identify different computing infrastructures commonly employed at institutions, including variations in hardware architectures (HPC, Research Cloud, VMs, etc.), software stacks, orchestration frameworks (Kubernetes, OpenStack, Slurm, etc.), networking setups, and storage solutions. We will build our guidelines with this heterogeneous scientific computing landscape in mind. Especially with the prospect of an ever-growing community of JupyterHub Outpost providers, and the fact that we have a limited number of in-person support resources, this will ensure that our support efforts can truly scale with our integration pursuits. This will also reduce the time required for new partners to become operational, thereby accelerating overall adoption. Through these detailed templates, common troubleshooting steps, and robust testing guidelines, this milestone will empower both current and Base4NFDI proposal form – 11/2024 – Integration page 22 of 42 future partners to set up JupyterHub Outposts effectively, thereby driving sustainable growth of the Jupyter4NFDI ecosystem. In addition to the enhanced documentation and templates (D3.1, M3.1), we will provide direct hands-on assistance (D3.2) for new resource providers integrating their infrastructure into the Jupyter4NFDI ecosystem. This deliverable will include interactive support channels such as scheduled consultation sessions, on-demand troubleshooting, and collaborative testing on staging environments, to help institutions address challenges unique to their HPC, cloud, or specialised computing setups. By offering live guidance beyond self-service materials, D3.2 ensures that complex technical issues and edge cases can be resolved more quickly, accelerating resource integration. Moreover, insights gathered during these one-on-one engagements will continuously feed back into our documentation, templates, and best practices, further refining the overall integration experience for future partners. Additionally, WP3 will actively explore opportunities to expand the ecosystem by identifying and engaging with potential new resource providers. Examples include collaborations with platforms such as de.NBI Cloud or Multi-Cloud infrastructures as well as centres of the NHR alliance, broadening the range of integrated resources available to the Jupyter4NFDI community. This effort will ensure that the service remains dynamic and continues to grow in alignment with project goals. 5.2.4 WP4: Content Integration WP lead: GESIS Contributors: FIZ, GWDG, TUD, US, TUC With this work package, we will establish an ecosystem that supports creating, sharing and reusing computationally reproducible notebook content, along with the accompanying data, for the Jupyter4NFDI service. To this end, we will provide readily available practical training resources, templates, and use case studies to facilitate the efficient adoption and integration of the service by users across diverse domains and consortia. The core objective is to enable users to develop and disseminate computationally reusable Jupyter notebook content and data, making it available for the service. This includes identifying the necessary metadata (CPU, RAM, storage requirements, supported programming languages) to select and configure the right node with the appropriate resources and transferring notebooks and data to the node. The use cases will concentrate on standard repository platforms like GitHub, GitLab, and DataVerse. This content will be usable along with the necessary data within the Jupyter4NFDI ecosystem through three mechanisms deployed as part of WP1: 1) the Binder launch capability, 2) the JupyterHub Repo2Docker import capability, and 3) the FAIRJupyter reproducibility assessment workflow (Samuel & Mietchen, 2024a,b). These three mechanisms are interoperable with respect to the provided content but address distinct needs: exploratory lightweight launching of an interactive session (Binder), long-term content Base4NFDI proposal form – 11/2024 – Integration page 23 of 42 development (Repo2Docker import) as well as running and documenting batch verification of reproducibility (FAIRJupyter). To bootstrap such an ecosystem, we will create use cases (D.4.1) by identifying and implementing representative examples of notebook-centred content. This will include, e.g., use cases from the social sciences (GESIS), artificial intelligence (TUD with ScaDS.AI) and biomedicine (FIZ/ FAIRJupyter), demonstrating the potential of Jupyter for reproducible research within the NFDI. Based on these use cases, we will develop domain-independent training materials (D.4.2). The materials will take the form of step-by-step guides on preparing and publishing content using the Jupyter4NFDI infrastructure and lead to a template for a short workshop on ‘How to Bring Your Work to Jupyter4NFDI‘ that will be usable across domains. As part of WP5, the materials will then be presented to different consortia through workshops and other formats. They will be adaptable for both self-study and instructor-led sessions. Furthermore, we will tailor the training materials to different audiences, including students and instructors, following the established Train-the-Trainer pattern. The materials for instructors will also be integrated into more general workshops, such as introductions to Python or R programming that are available through platforms like The Carpentries. Given our experience with trainings on reproducible research and Jupyter (e.g. GESIS Training Workshop 2023 ‘Workflows for Reproducible Research with R & Git’30), we believe that leveraging synergies between domain-specific programming education and Jupyter notebook training maximises relevance and impact. To that end, we will also develop domain-specific training materials (D.4.3) for selected fields of study. This work package aligns closely with other Base4NFDI initiatives focused on training materials, user story development, and persona creation. By building on existing resources and approaches, safeguard compatibility and integration with broader NFDI efforts, such as the Software Marketplace, are safeguarded. .The materials developed in this work package will lower the barrier for researchers to adopt Jupyter4NFDI for their computational analysis and promote best practices in reproducible research across the NFDI community. Feedback from the community is collected and used to improve the materials. 5.2.5 WP5: Community Engagement and Communication WP lead: US Contributors: FIZ, GESIS, GWDG, JSC, TUD 30 https://github.com/jobreu/reproducible-research-gesis-2023 Base4NFDI proposal form – 11/2024 – Integration page 24 of 42 This work package will ensure effective interaction between our team and the diverse research communities within the NFDI, as represented by the disciplinary consortia. We will foster a collaborative environment, promoting the adoption of Jupyter-based workflows and gathering user feedback to refine the service continuously. Key activities include outreach, organisation of trainings, and tailored communication strategies to align with the needs of various scientific domains. We will further formalise and enhance the measures that started in the initialisation phase. This includes our weekly Onboarding Open Hour, primarily targeted at service providers potentially interested in connecting to the central hub. A comparable open hour for end users of the service has just been established. In the future, we will contribute to the outreach activities organised by Base4NFDI, such as the roadshows and the user conferences, aggregating and condensing the material developed in WPs 1-4 (D5.2). We will also continue bilateral exchange with the disciplinary consortia by presenting and discussing, for example, at their annual meetings. As outlined in Section 3.1, we will conduct incubator projects targeting content and service providers as well as end users. WP5 will take the coordinating role of announcing the incubator cycles which are planned semi-annually, collecting information from potential participants, assigning each proposed incubator to the best-suited work package, as well as collecting and disseminating individual incubator results (M5.1). In addition to these activities, we will focus on dedicated training for domain scientists and prospective users, again based on the materials developed in WPs 1-4. The materials will also be provided to the consortia and the basic service RDMTraining4NFDI, e.g., via the knowledge base DALIA of the section EduTrain (D5.1). We will offer to conduct trainings ourselves and participate in activities offered by the consortia. To this end, we rely on our Base4NFDI service stewards to establish connections to relevant interested communities. Based on the expertise and experience of the respective principal investigators, we will also frequently offer longer workshops “Make your work reproducible with Jupyter4NFDI” (M5.2). There, participants bring in an individual research process which they would like to render reproducible. Base4NFDI proposal form – 11/2024 – Integration page 25 of 42 5.2.6 Gantt chart JupyterHub OutpostSpawner and Outpost M1.1 Completed These two components make up the two ends of the integration layer between JupyterHub and generic external resources. A lightweight communication layer allows for almost arbitrary deployments of Jupyter Notebook services at the external resources. IAM4NFDI M1.1 Comleted This central service within Base4NFDI can be taken for granted. Coincidentally, it is base on the same software stack that has already been in use at JSC for years, such that we are confident to be able to do the integration. Comprehensive summary Jupyter4NFDI has not been developed as a greenfield project. There has been a precursor running at JSC for several years, from which we also inherited some of the versatility of the central hub regarding the connection to backend resources of various types. Based on this solid foundation, we further extended the ability to integrate various external resources. This will include, but is not limited to HPC resources available within NHR, generic Cloud computing resources, e.g. from an NFDI Multi-Cloud. The service stack has been crafted in such a way, that it will be scalable and cater for serving many simultaneous users, certainly in the hundreds. [1] https://zenodo.org/records/12699382 1.3. Jupyter4NFDI / D1.4.3 (TA1): Documentation of service design ID 12369 TYP Task STATUS Completed AUFWAND VERBLEIBENDER AUFWAND % ABGESCHLOSSEN 100 Beschreibung From TA1 instructions on Zenodo Mapping & status table Task Milestones & Deliverables of DMP4NFDI Status Outcome Deployment of central JupyterHub M1.1, D1.1 completed The Jupyter4NFDI service documentation has been created [1] 4/922.04.2025 14:53 Arbeitspakete Resource Manager M2.1, D2.1 completed The JupyterHub Outpost architecture has been documented [2] Comprehensive summary Comprehensive service documentation for the Jupyter4NFDI basic service was created as part of the initialisation phase [1]. The documentation contains a general explanation of the service and is aimed at both users of the service and service providers who want to connect external resources to the central Jupyter Hub. Understanding the JupyterHub Outpost architecture will help set up and manage Outpost instances effectively, while ensuring your resources remain secure. The JupyterHub Outpost architecture has been described in a new chapter of the Jupyter4NFDI service documentation [2]. [1] https://nfdi-jupyter.de/ [2] https://nfdi-jupyter.de/providers/architecture/ 1.4. Jupyter4NFDI / D1.4.4 (TA1): Service prototype ID 13135 TYP Task STATUS Completed AUFWAND VERBLEIBENDER AUFWAND % ABGESCHLOSSEN 100 Beschreibung TA1 instructions on Zenodo Mapping & status table Task Milestones & Deliverables of DMP4NFDI Status Outcome Deployment of central JupyterHub M1.1, D1.1 completed The service prototype has been made available at [1]Ê Resource Manager M2.1, D2.1 completed JSC resources have been made available in central Jupyter4NFDI hub [1]. Blueprint for deploying a JupyterHub Outpost (aka external Resources) via Helm Chart [3] Integration of the AAI M3.1, D3.1 completed Authentication is already possible via Helmholtz Login [1]. Continuing IAM4NFDI incubator project for connecting service to IAM Infra Proxy [2] has started. 5/922.04.2025 14:53 Arbeitspakete Comprehensive summary The service prototype has been made available at https://hub.nfdi-jupyter.de/ along with documentation for users and interested resource providers. Authentication is currently possible via Helmholtz Login, which is an integral part of IAM4NFDI. We are waiting for the IAM4NFDI infrastructure proxy to become Êavailable and will connect to it. To this end, we have applied as an incubator project to the IAM4NFDI project. Helmholtz Login does not limit authentication to only users from the Helmholtz Association, but also allows for using all eduGAIN (excluding Russian and Belarussian) and social IdPs. [1] https://hub.nfdi-jupyter.de/ [2] https://incubators.nfdi-aai.de/ [3] https://gitlab.jsc.fz-juelich.de/jupyterjsc/denbi-deployments/-/tree/jupyter-outpost 1.5. Jupyter4NFDI / D1.4.5 (TA1): Service piloting and user testing ID 13136 TYP Task STATUS In progress AUFWAND VERBLEIBENDER AUFWAND % ABGESCHLOSSEN 50 Beschreibung TA1 instructions on Zenodo Mapping & status table Task Milestones & Deliverables of Jupyter4NFDI Status Outcome Deployment of central JupyterHub, Resource Manager M1.1, M2.1 completed Central JupyterHub deployed and resources have been made available (see D1.4.4). Communities of NFDI consortia have started pilot testing and usage of the available resources [2], [3], [4], [5], [6], [7], [8], [9] Connecting external resources M4.1 Initial external resources have been made available in staging environment of central hub [1]. 6/922.04.2025 14:53 Arbeitspakete Comprehensive summary Consortia integration In Text+, one of the participating consortia, the Jupyter4NFDI service is being used regularly for at least one use case: The Fluffy Import [5]. This involves a notebook-based application [6] for - among other things - importing data in TextGridRep [7], a repository of great relevance for the humanities, which is optimised for TEI-XML data and allows the publication of text and images. The notebook-based aproach replaces the initial import and publication system, which became hard to maintain and update. The NFDI JupyterHub is the recommended platform for running the application, and a link to a preconfigured environment based on a Docker image is provided in the readme of the code repository. The service has also recently been used in the workshop "TEI-Dokumente in TextGrid Repository veröffentlichen und archivieren: neue Features und fluffiger Import Workflow" [8], where it proved very useful and worked as expected. Furthermore, the service is being showcased and recommended repeatedly in the bi-weekly meeting "Text+ Show and Tell" [9], where it also was the main topic of the kickoff meeting. Other projects have manifested interest and are expected to use the service. • NFDI4Ing Together with NFDI4Ing we were able to show applicability of our generic solution for at least one use case of the 11th Society of Petroleum Engineers Comparative Solution Project [2]. At the time ofo setting up this scenario, we needed to import a dataset to Jupyter4NFDI at JSC to make it available within Jupyter Labs there. Since then, we have developed a method to generically import data from external repositories, such that a migration of data prior to execution will not be needed anymore provided that it is available from another service. As can be seen in the repository, it directly links to an executable Jupyter environment in Jupyter4NFDI. • FlowR [3] Validation with user groups The group of Prof. Martin Pfleiderer at the University of Music Franz Liszt Weimar was able to immediately make use of the central Jupyter4NFDI service for their use case involving an opensource software toolbox for sheet music annotation, statistical analysis, visualization, and pattern search operations. This is actually our expectation for many standard use cases that do not have particular demands. As a matter of fact, because Jupyter4NFDI allows for using custom images, we believe to be able to accomodate a wide range of user demands even if they have very specific software requirements. •MaRDI •NFDI4Biodiversity (Seeger) •Fluffy Import (Text+) •4DS 7/922.04.2025 14:53 Arbeitspakete Connecting external resources Besides validating the service applicability with end users, we have validated our approach to integrate external resources with several resource providers. The natural choice for resources were those at JSC for which a precursor already existed with the Jupyter JSC offering. Through this integration, we were able to provide access to the JSC Cloud computing resources. Furthermore, Cloud resources at MPCDF and de.NBI Cloud have been integrated in the staging environment [1] with resources at MPCDF ready to be added to the production environment. Another type of resources is under way provided by TU Dresden. Following the template of JSC's approach to integrating HPC resources in JupyterHub, resources at TU Dresden, which in turn will serve as a template for the integration of other NHR resources. Further integrations are planned with the University of Stuttgart, where a Kubernetes cluster is currently set up. Demonstration of general applicability Without consideration of individual use cases or user groups, the number of unique users per period (day, week, month) is increasing on average, as can be seen by the numbers we gathered between 2024-06-26 and 2025-04-09 [10]. References • [1] JupyterHub staging environment https://hub-staging.nfdi-jupyter.de/ • [2] https://github.com/Simulation-Benchmarks/11thSPE-CSP • [3] https://github.com/Code-Inspect/binder-flowR • [4] https://www.hfm-weimar.de/en/working/professors-teachers/lehredetail?tx_jobase_pi3%5Baction%5D=orgadetail&tx_jobase_pi3%5Bcontroller%5D=Elements&t x_jobase_pi3%5BjoOrgaDetail%5D=106&tx_jobase_pi3%5BjoRefererId%5D=220&cHash=ae2d 8/922.04.2025 14:53 Arbeitspakete f9e057049ac2a5c4cb507a9ddaac • [5] https://marketplace.sshopencloud.eu/workflow/iqJ7B6 • [6] https://gitlab.gwdg.de/textplus/textplus-io/nb-actions • [7] https://textgridrep.org • [8] https://dhd2025.dig-hum.de/?page_id=8 • [9] https://events.gwdg.de/category/286/ • [10] https://zenodo.org/records/15187218 1.6. Jupyter4NFDI / (TA1): Presentation of final results in section meeting ID 13389 TYP Task STATUS To be scheduled AUFWAND VERBLEIBENDER AUFWAND % ABGESCHLOSSEN 0 Beschreibung Presentation of final results in section meeting TA1 instructions Should be done, but is not a hard requirement When done provide link to minutes and slides. 9/922.04.2025 14:53 Arbeitspakete