EOSC SYNERGY HANDBOOK: A Handbook targeted at Computer Centre Management, Administrators, and Users of the Infrastructure

EOSC SYNERGY HANDBOOK A Handbook targeted at Computer Centre Management, Administrators, and Users of the Infrastructure Marcus Hardt and Viet Tran for the EOSC-Synergy WP2 Team Creative Commons "CC BY-SA 3.0" Table of contents 31. Introduction 52. Thematic Services 52.1 Thematic Services Challenges 72.2 Thematic service technology choices 83. Architecture of the EOSC 83.1 AAI 83.2 Monitoring 83.3 Accounting 93.4 Information Provider 93.5 EOSC-Exchange / Marketplace 104. Resources Integration (for managers) 104.1 Why should computer centres join the EOSC ecosystem 114.2 Policies and concept for access to resources 154.3 How to join the infrastructure 185. EOSC Synergy Services and Tools 195.1 Overview 215.2 Services and tools for Cloud computation on EGI Federated Cloud 235.3 Services and tools for authentication 245.4 Services and tools for Cloud storages 265.5 Services and tools for HPC 265.6 Platform for Software and Services for Quality Assurance 275.7 Services for online training 285.8 Monitoring and Accounting Services Table of contents - 2/28 - Creative Commons "CC BY-SA 3.0" 1. Introduction In this handbook we describe how to integrate computing and storage infrastructure in such a way as to comply with current guidelines and best practices of the European Open Science Cloud (EOSC). We also describe how it may be used, operated and extended, with a focus as well on the system administrator's perspective. We present a set of example applications that were adapted to profit from the services supported by the infrastucture. This includes the services used, the general architecture of EOSC, the tools we chose to support the applications, and how computer centres can join their own resources into the federated cloud. Our initial starting point was: We have 10 demanding Thematic Services that need infrastructure resource (cpu/gpu, storage, network, accounting, and monitoring) to provide their services and or results to the users We have an architecture for the European Open Science Cloud (EOSC) with definitions of the core services / federating core, ... We have an EOSC Marketplace that provides a large choice (320+) of different solutions and tools, many of which are in an unknown state. We have a team of experts in distributed computing that (happen to) operate the first working prototype of an EOSC infrastructure: EGI Federated Cloud The challenge: Bring users and infrastructures together - in a scalable way that avoids vertical solutions. The EOSC-Synergy way to address this was the introduction of tools that provide a natural separation between the different roles and requirements on the infrastructure. Users are supported by the Community Manager, or by their Community Developers. These two representatives of the Community can request changes to the infrastructure services. Site administrators are responsible for the operation of the physical machines that provide CPU cycles and storage. They are in contact with Community Administrators that are in contact with Community managers to request the capacity in which these services are provided. Details are described in our Deliverable [D2.2]. All tools and solutions used are provided by open source software, exclusively. This ensures a long term perspective for a sustainable infrastructure. Furthermore, the open approach taken guarantees that custom extensions can always be implemented by third parties to create tools that bridge gaps that may be identified. One example for this are the three different tools that may be used to access the infrastructure. Two different web-tools and one Command Line Interface (CLI) tool, address the whole bandwidth of user experience. These tools Infrastructure Manager (IM) and its dashboard, the Openstack Dashboard and the fedcloudclient are described together with many others in section 4. Federated is not just distributed. The federated nature of the infrastructure brings several challenges that need to be addressed in order to build a sustainable solution. One particular challenge stems from the fact that our users are identified by entirely different legal entities in different continents. Making use of recently developed modern Identity and Access Management solutions (often called AAI) allows offering services to reliably identified users without the attached cost of user management. Just as our users, are the computer centres that provide capacity to the cloud are federated across different legal entities in different countries, most of which are currently situated in Europe. One essential concept for addressing this are Virtual Organisations (VOs). The VO Management is delegated to a user community. Community managers negotiate quotas for their VOs with individual resource centres. The infrastructure provides • • • • 1. Introduction - 3/28 - Creative Commons "CC BY-SA 3.0" usage statistics (accounting) and monitoring on the granularity of the VO. VOs are probably the most cost efficient and scalable way of addressing federated users on federated infrastructures at large. VOs as implemented on the EGI Federated Cloud provide a balance between the freedom in the authorisation decision and a strict governance on technology and policies (which software, which regulations, who is responsible). Without VOs, furthering endeavours such as the European Open Science Cloud EOSC do not seem feasible. The overall organisation of this handbook is as follows. We start with an introduction of the EOSC-Synergy supported Thematic Services (TS) and their requirements. Each Thematic Service is described in section 2. Then we describe the general components of the EOSC architecture in Section 3. The more technical Sections 4 and 5 describe how to integrate new resource centres (I.e. CPU, GPU, or storage hardware to provide cloud services) into the federated cloud, followed by a list of the components and tools used within the EOSC-Synergy Project. We close with the summary in Section 6. 1. Introduction - 4/28 - Creative Commons "CC BY-SA 3.0" 2. Thematic Services EOSC-Synergy is supporting ten different thematic services in four scientific areas (Earth Observation, Biomedicine, Astrophysics and Climate Change see here for a detailed list). Each service corresponds to a separate Community, each of which has different requirements, software tools, and access patterns. All of these thematic services require access to infrastructure services such as CPU, Storage and Network. The way in which the infrastructure is allocated, accessed, and used is different between each thematic service, though. Many services provide access to research data and benefit from a wider availability. As such they need to act as clients and servers simultaneously. Often, the underlying user management (also called AAI) is shared for both roles. The thematic services serve as examples, since they addressed a larger number of issues than many other services. This gives us the chance to either prove that EOSC-Synergy is ready to access federated datasets, in clusters distributed across Europe, or to develop additional tools for the ecosystem in case they are still missing. During the project lifetime the communities have progressed towards best practices for the adoption of common EOSC guidelines, tools, interfaces and services. This includes strengthening the communities in increasing the capacity, performance, reliability and/or functionality of these thematic services through their integration in EOSC. This was especially important to increase the number of users of these thematic services substantially. A detailed report will be published by EOSC-Synergy’s Work Package 4 (WP4), that describes requirements and solutions of the thematic services in detail. [WP4-IS] 2.1 Thematic Services Challenges Here we provide a short overview about the specific challenges faced by each thematic service. These challenges regard access to Computing and Storage specifically. For more information about each specific thematic service, we provide references to a publication that describes the service in more detail. The following information have originally been collected in the paper “A 2. Thematic Services - 5/28 - Creative Commons "CC BY-SA 3.0" survey of the European Open Science Cloud services for expanding the capacity and capabilities of multidisciplinary scientific applications” by Ignacio Blanquer et. al. [WP4-IS] Table 1: The Thematic services with their challenges that need to be addressed in EOSC Synergy Thematic Service Limitations and needs WORSICA - Improve download speed and number of concurrent downloads of satellite images. - Increase storage of the images needed for the algorithm. - Increase computational resources: GPU and RAM to speedup the image processing. - Seamless authentication and authorization for end users. SAPS - Need for a larger-scale deployment: computing, storage and data access. - Scalability and standardisation of services - Integrated and widely supported AAI GCore - Overcome limited access to data repository due to network bandwidth restrictions. - Infrastructure resources for processing and reprocessing large data sets. - Data delivery volume. Increasing size of files to be delivered to users. SCIPION - Insufficient Cloud resources for the workflow: GPUs, CPUs and RAM - Need of a Resource Management able to optimize the use of cloud resources. - Storage limitations and data transfer performance: 1-3 TB raw data. - Distributed and shared file system. OpenEBench - Need to work on heterogeneous systems to reach Life Sciences Communities - Need to efficiently store processed data and workflows in a FAIR manner. LAGO - Limitations on data preprocessing. - Needs data storage that copes with FAIR, curation and harvesting; - Need for computing power for simulations, together with optimal scheduling. SDS-WAS - Lack of services needed for Data storage and curation. - Lack of computing power for data analysis on-demand. - Lack of reliability of data sources, especially about observations UMSA - Long-term data storage is required, together with appropriate data curation. - Tracking provenance of the secondary (derived) datasets. - Need for reimplementing UMSA algorithms to deal with sparse data. MSWSS - Needs data protection measures because of the usage of confidential data. - The data has to be stored in a private storage only. - Implement security policies to protect VMs. O3AS - Requires larger storage resources, specially improving data availability - Fast handling of big data 2.1 Thematic Services Challenges - 6/28 - Creative Commons "CC BY-SA 3.0" 2.2 Thematic service technology choices To address the identified challenges, WP4 of EOSC-Synergy undertook an analysis of the Services offered via the EOSC Marketplace. More than 320 services are available. In [WP4-IS] these services are organised into six categories, out of which “Access physical & eInfrastructures” is the one we are interested in. Table 2 shows those services chosen by each service to address the needs within the different categories. More details are available in the corresponding WP4 Deliverable [D4.3]. Table 2: The solutions used by thematic services in the different domains. (from [D4.3]) Service AAI Workload Mng. Resource Mng. Data Storage WORSICA EGI Check in ArcCE, Batch (SLURM) IM (TOSCA) Nextcloud, Datavers G-Core CAS User/pwd & EGI Check in GCore+ K8s IM / EC3 ElasticSearch SAPS EGI Check in K8s IM / EC3 OpenStack Swift Scipion EGI Check in Batch (SLURM) IM / EC3 Local + EGI DataHub OpenEBench Life Sciences AAI WfExS + NextFlow OpenNebula Local + B2SHARE LAGO eduTEAMS + EGI Check-in Batch (SLURM) Local clusters + IM / EC3 EGI DataHub ONEDATA SDS-WAS B2ACCESS Batch (SLURM) Local clusters B2HANDLE / B2SAFE UMSA EGI Check in & Lifescience AAI Batch (SLURM) in IM/ EC3 (in Galaxy) IM / EC3 Local + S3 MSWSS EGI Check in Batch (SLURM) in EC3 (in Galaxy) IM / EC3 Local + Dataverse O3AS EGI Check in Batch (SLURM) & K8s cluster Local + WebDAV 2.2 Thematic service technology choices - 7/28 - Creative Commons "CC BY-SA 3.0" 3. Architecture of the EOSC 3.1 AAI The integration with the Authentication and Authorisation Infrastructure (AAI) is the fundamental step, because this is the mechanism that delivers the identities (unique identifiers and group memberships) to the infrastructure. This is essential in order for the users to access the services in the first place. Furthermore, AAI is the logical “point” where users may be informed about the policies (like privacy notices, AUPs) and where access rights to services may be enforced. In collaboration with EOSC-Hub and EGI we have recorded a procedure [AAI-Integration-Procedure]. There, services are first integrated with a demonstration instance of the EGI Check-in service for initial testing. Successfully tested services will then be moved to the production instance of EGI Check-in. This process is documented and supported in various procedures of EGI [EGIProc-09]. This integration will allow users to access all services, using their home-Identity, their community identities supported by EGI (six at the time of writing) or one of six social identity providers. In addition, Virtual Organisations (VOs) have been created [EGIProc-14] to support the thematic services and facilitate their resource allocation requests. For conducting trainings we envisage collaboration with the EGI training VO. 3.2 Monitoring Every production service requires a monitoring system, so the desired efficiency and accountability can be verified at any time. After defining what should be monitored, it is easy to create a test and display its results. For EOSC-Synergy, Nagios core service will be used. One of the key aspects of Nagios is to get the information about a particular test. With Nagios we can monitor various kinds of hosts, services and actions. From the simplest check on a web service (ping) to a more complex test on the service itself, such as testing outputs giving some input on APIs, websites using selenium or even testing if the login runs smoothly. This is all examples, since plugins can be written to all types of tests to check even the small functionality, using different programming languages. In addition, contacts and alerts will be used to notify about problems detected by the Nagios service. This allows us to efficiently react to problems in a timely manner. 3.3 Accounting The EOSC Accounting service collects, stores, aggregates, and displays usage information of HTC compute, storage space, cloud VM and data set resources. Resource Centres that are providing compute or storage to the EOSC infrastructure have to implement a collector (a stand-alone script or program, or a built-in function of their resource system), that gathers accounting metrics formatted into a standardised record format. These metrics are then transferred via a messaging service to the Accounting Repository, which stores and processes the data to produce aggregations that are then sent to the Accounting Portal for display. The Accounting Portal retrieves topology information on how resource centres relate to national infrastructures and regions from the configuration management database (CMDB) and community affiliation from the AAI service to properly organise the 3. Architecture of the EOSC - 8/28 - Creative Commons "CC BY-SA 3.0" accounting data. Information related to groups or VOs should also contain information about scientific disciplines to allow the portal to properly classify the resource usage. The Accounting Portal already is integrated with several other tools, such as GOCDB and REBUS for topology and geographical data, Check-In for the AAI, the Operations Portal for VO and scientific discipline information. It also uses X.509 certificates to map users to institutions and these to countries. 3.4 Information Provider The information system collects data from the resource providers in a research infrastructure and makes it available for workload orchestration. In the EGI e-Infrastructure this is a fundamental service both for the HTC or Grid technology and for the EGI FedCloud. There are different solutions for each type of service within those technologies, e.g. both ARC and HTCondorCE have ad hoc provider implementations for gathering the information. For Federated Cloud, we use a unique implementation, coined as cloud-info-provider, to fetch data from the supported Cloud Management Frameworks (CMFs), notably OpenStack and OpenNebula. The cloud-info-provider component leverages the APIs exposed by those CMFs to get key information about the Cloud resource provider, such as the projects and images that any given VO is allowed to use. Consequently, the data collected by the information providers is essential for the operation of the different workload management services available through EOSC, such as the INDIGO PaaS Orchestrator or DIRAC4EGI. 3.5 EOSC-Exchange / Marketplace 320+ Service, out of which the thematic service chose the most useful ones. Interoperability Framework Federation • • • 3.4 Information Provider - 9/28 - Creative Commons "CC BY-SA 3.0" guaranteed to the end-user. All necessary steps and procedures are documented in detail at the EGI Cloud Compute webpage. This integration can be grouped into three categories: Organisational prerequisites: Join your national grid initiative (NGI) to obtain an entry in the GOCDB. Ensure you can support the relevant policies. Technical prerequisites: Integration of cloud stacks into EGI FedCloud follows a well-defined path, with infrastructure services such as accounting, monitoring, authentication and authorisation, etc. These configurations make your site discoverable and usable by the communities you wish to support, and allow EGI to support you in operational and technical matters. Have a cluster of compute nodes available on which OpenStack will be installed. Install the required additional tools and services for monitoring, authentication, accounting, networking, ... Allocation policies: Make decisions regarding which Virtual Organisations (VOs) you want to support. These decisions may be updated at any time. Direct your existing and/or local user communities to setup a Virtual Organisation. 4.3.2 Infrastructure users: How to join as a user or community Allocation of resources (CPU/GPU hours, storage) is done at the Virtual Organisation (VO) Level. Users therefore must either be a member of an existing VO, or create one. Once a VO is supported at one or more sites, users can start using resources. While this is generally well described in the corresponding EGI Federated Cloud documentation. we give a general overview here. Also, the list of services and tools in section 5 will provide useful support to users at all levels. The infrastructure provides multiple interfaces designed for different knowledge levels of users and for the different types of services. The cloud infrastructure provides access - you guessed it - to cloud resources. More specifically, these are OpenStack. resources, installed at multiple computer centres (distributed across Europe) called “sites”. One way to use these cloud instances is to use the OpenStack web interface (horizon) at every site. Since these are non-trivial to discover, the EOSC-Synergy dashboard. was developed for simpler discovery. The web interface offers access to all functionality of OpenStack, which includes computing, blockand object storage, and networking. Images available for running are provisioned via the AppDB. made available to the VO by a VO administrator. More user-friendly access (everybody who knows the horizon web-interface knows there is room for improvement) is available via the Infrastructure Manager Dashboard. After initial configuration (documentation is available, including youtube videos. readthedocs. and moodle. users can easily deploy pre-configured VM infrastructures, including dynamic SLURM, hadoop or kubernetes clusters. This may be understood as a starting point for the exploration of the infrastructure. All of it is available in a more technical way for automation via the commandline fedcloudclient. and REST interfaces, described in Section 5. Persistent storage is available via the EGI Datahub. which allows multiple useful access patterns, including mounted filesystems and object storage. This versatile cloud infrastructure may be used for a variety of use-cases. The spectrum of favourable patterns includes medium scale HPC (including GPU usage), on one end of the spectrum, via Portals that serve results of queries to large databases and conduct HPC analyses on request, all the way to traditional server hosting on the other end. For any service in this environment, users will authenticate via EGI Check-In, to which they are redirected automatically. EGI Check-In offers to either authenticate via your home-organisation (e.g. the university you work at), or via a “Community-AAI” such as eduTEAMS, ORCID, GitHub, B2Access, Umbrella or Facebook. It is important to choose the correct one, because your VO membership information may come from the chosen community. What may be a bit confusing is that EGI is also a CommunityAAI. To find your VO memberships, you need to choose different identities to log in (e.g. google at first, university later). To avoid confusion, it is important to remember the choices made. 1. • • 2. • • 3. • • 4.3.2 Infrastructure users: How to join as a user or community - 16/28 - Creative Commons "CC BY-SA 3.0" Further features include features include: Global accounting that aggregates and allows visualisation of usage information across the whole federation. Monitoring of Availability and Reliability of the providers to ensure SLAs are met. Since the opening of the EGI Federated Cloud, the following usage models have emerged: Service hosting: the EGI Federated Cloud can be used to host any IT service as web servers, databases, etc. Cloud features, as elasticity, can help users to provide better performance and reliable services. Examples: NBIS Web Services, Peachnote analysis platform. Compute and data intensive applications: for those applications needing a considerable amount of resources in terms of computation and/or memory and/or intensive I/O. Ad-hoc computing environments can be created in the EGI cloud providers to satisfy extremely intensive HW resource requirements. Examples: VERCE platform, The Genetics of Salmonella Infections, The Chipster Platform. Datasets repository: the EGI Cloud can be used to store and manage large datasets exploiting the large amount of disk storage available in the Federation. Disposable and testing environments: environments for training or testing new developments. Example: Training infrastructure. All these tools may be used and combined to develop individual solutions that may be tailored perfectly for each use case. • • • • • • • • • • • • • • • • 4.3.2 Infrastructure users: How to join as a user or community - 17/28 - Creative Commons "CC BY-SA 3.0" 5. EOSC Synergy Services and Tools Within EOSC Synergy, we have used, integrated and developed several tools. Some tools existed already, others were extended or developed from scratch. All of the tools have in common that they proved to be useful for the integration of the thematic services supported by the project. This chapter presents these useful tools. Since all services are open source, their interfaces are open and may be used with several different tools. The tools presented here present a subset of possible solutions. They are not meant to be exclusive. All tools may benefit from combining them with others. 5. EOSC Synergy Services and Tools - 18/28 - Creative Commons "CC BY-SA 3.0" 5.1 Overview This table shows an overview about the tools presented in this handbook 5.1 Overview - 19/28 - Creative Commons "CC BY-SA 3.0" HPC Cloud Compute Storage AAI Q/A Training Generic udocker ✓ ✓ SLURM ✓ ✓ Infrastructure Manager ✓ ✓ Fedcloud Client ✓ ✓ ✓ Dynamic DNS ✓ ✓ ✓ EOSC Performance ✓ ✓ ✓ Cinder ✓ Swift ✓ RClone ✓ EGI DataHub ✓ B2Share ✓ Core AAI/IAM ✓ oidc-agent ✓ mytoken ✓ ssh-oidc ✓ flaat ✓ Vault ✓ Pipeline as a Service ✓ Quality Badges ✓ Learn@Synergy ✓ Online Training Platform ✓ Video Conferencing Tool ✓ Cloud Storage ✓ Training Infra Mgmt ✓ Jupyter Notebooks ✓ Hackathon as a Service ✓ Service Management ✓ ✓ ✓ ✓ ✓ ✓ Monitoring ✓ ✓ ✓ ✓ ✓ 5.1 Overview - 20/28 - Creative Commons "CC BY-SA 3.0" 5.2 Services and tools for Cloud computation on EGI Federated Cloud 5.2.1 Infrastructure Manager Infrastructure Manager is a tool that eases the access and the usability of cloud infrastructures by automating Virtual Machines Instances (VMI) selection, deployment, configuration, software installation, monitoring and update of Virtual Appliances. It supports APIs from a large number of virtual platforms, making user applications cloud-agnostic. Infrastructure Manager is intensively used by Thematic services in EOSC-Synergy. The service was significantly improved during the project based on feedback from users. In addition, several new recipes for EOSC-Synergy services were developed and added to the dashboard. Links: Infrastructure Manager Dashboard Service Service at EOSC Marketplace Documentation Training and Tutorial 5.2.2 Openstack Dashboard OpenStack Horizon is a web-based graphical interface that users can access to manage OpenStack compute, storage and networking services. It allows service administrators to use this Dashboard to launch virtual machine instances, storage volumes or even manage their networks. On top of this, EOSC-Synergy developed a dashboard that allows accessing all participating sites from one dashboard. The dashboard becomes the central web-based GUI interface for managing resources on all OpenStack sites in the project. Links: EOSC-Synergy dashboard Documentation Training and Tutorial 5.2.3 FedCloud client The FedCloud client is a command-line client designed for interacting with OpenStack services in the EGI infrastructure. The client can access various EGI services and perform many tasks for users. It includes managing access tokens, listing services, and command execution on OpenStack services located in the EGI Cloud infrastructure. The client was developed during the EOSC-Synergy project and has become the official client for EGI Cloud infrastructure. FedCloud client is designed for using in shell scripts or Python programs. That enables sophisticated ways to automate tasks interfacing the cloud infrastructure. Complex tasks like listing all virtual machines owned by a user on all OpenStack sites in EGI Cloud infrastructure can be easily completed by simple scripts using FedCloud client. HPC Cloud Compute Storage AAI Q/A Training Generic Accounting ✓ ✓ ✓ ✓ ✓ • • • • • • • 5.2 Services and tools for Cloud computation on EGI Federated Cloud - 21/28 - Creative Commons "CC BY-SA 3.0" Links: Package repository EOSC Marketplace Documentation Training and tutorial 5.2.4 Dynamic DNS The Dynamic DNS service provides a dynamic Domain Name System (DNS) service for EGI Cloud infrastructure. Users can register their own meaningful and memorable host names, using a list of provided domains (e.g. fedcloud.eu, eosc-synergy.eu) and assign to public IPs of their servers hosted in EGI Federated Cloud. Simple login using EGI Check-in allows registering your own hostnames. By using Dynamic DNS you can host services in EGI with meaningful service names and freely move their virtual machines (VMs) between sites without modifying configurations (federated approach). The hostnames also enable the services to get SSL certificates for improving security and privacy. Some domains dedicated for EOSC-Synergy were added to the service for supporting Thematic services: o3as.fedcloud.eu, repository.fedcloud.eu, vm.fedcloud.eosc-synergy.eu, worsica.fedcloud.eosc-synergy.eu. The Dynamic DNS service is also integrated with Infrastructure Manager for deploying thematic services with registered hostnames from Dynamic DNS. Links: Service EOSC Marketplace Documentation Training and tutorial 5.2.5 EOSC Performance EOSC-Performance is a search-and-compare platform where you can upload and search through results from multiple benchmarks. By comparing the data acquired from benchmarks, you can evaluate and decide which computing infrastructure provider would give the best performance for your applications. The service is developed and supported by the EOSC-Synergy project. For computing infrastructure providers, they can also submit new entries so users can find their services. The interface to the platform can be done through a web based Graphical User Interface (GUI) or through an API in case you want to automate or integrate the data with your project. Links: Service GUI EOSC Marketplace Open-API self-documentation Training Video Documentation API Documentation • • • • • • • • • • • • • • 5.2.4 Dynamic DNS - 22/28 - Creative Commons "CC BY-SA 3.0" 5.3 Services and tools for authentication 5.3.1 AAI / IAM Services The cloud infrastructure relies on services that provide the Authentication and Authentication Infrastructure (AAI). More specifically, so-called “Community AAIs” as defined in the AARC Blueprint Architecture (BPA) are required for users to log into the EOSC services. This is fully in line with the EOSC Architecture. The EGI Federated Cloud uses EGI Checkin (https:// aai.egi.eu) as its infrastructure proxy. This enables a large number of communities to use the Cloud. Within EOSC Synergy, we have successfully used the EGI, the GEANT eduTEAMS (https://eduteams.org), and the EUDAT B2access (https:// b2access.eudat.eu) services for our users. 5.3.2 oidc-agent oidc-agent is a set of tools to manage OpenID Connect tokens and make them easily usable from the command line. It follows the ssh design, so users can handle OIDC tokens in a similar way as they would do with ssh keys. If users are using or designing an API which relies on OIDC authentication like accessing OpenStack sites with the FedCloud client mentioned above, these tools will come really handy to them. Links: Repository Documentation 5.3.3 mytoken OIDC tokens are a very handy and secure way to handle user identification and authorisation between systems, especially in a federated environment such as EOSC. However, their short life is a problem on tasks where the execution time can be longer than the expiration time of the token. Such tasks are not rare in a scientific community such as EOSC. To solve these issues, mytoken was developed to provide OIDC Access Tokens for example to long-running compute jobs. Mytoken is a web service to obtain OpenID Connect Access Tokens in an easy but secure way for extended periods of time and across multiple devices. Mytoken focuses on integration with the command line through a command line client but also offers a web interface for users who prefer managing their tokens with a browser. If you like oidc-agent and you need to execute long lived tasks on cloud or HPC, this tool is definitely for you. Links: Service demo instance Documentation 5.3.4 ssh-oidc ssh-oidc consists of a set of tools that allows (you guessed it) ssh with OIDC. This tool allows you to authenticate and log in to remote machines using your institution (or any other organisation) credentials instead of using a secret key or password. Focused on usability, ssh-oidc is divided around several tools and libraries to mimic a subset of the popular ssh capabilities. The two main components are an SSH client wrapper: mccli designed to run on clients computers and the service motley-cue for mapping OIDC identities to local identities (to be run on the server where users are planned to log in). Links: General repository mccli manual motley-cue manual Putty extension • • • • • • • • 5.3 Services and tools for authentication - 23/28 - Creative Commons "CC BY-SA 3.0" 5.3.5 Python API for AAI in services: flaat Flaat is a simple python library that allows a straightforward implementation of REST interfaces that are well integrated with the AAI. By using decorators, individual functions can be protected, so they may only be accessed by authorised users. Authorisation may be limited to VO and Group Membership as well as to the Assurance of a users identity. Links: Github Repository Documentation 5.3.6 Vault Secrets Manager Applications in EGI Infrastructure may need different secrets (credentials, tokens, passwords, etc.) during deployments and operations. The secrets are often stored as clear texts in configuration files or code repositories that expose security risks. Furthermore, the secrets stored in files are static and difficult to change/rotate. The secret management service for EGI Infrastructure is developed to solve the issues. Links: Documentation 5.4 Services and tools for Cloud storages 5.4.1 OpenStack Cinder If your cloud is hosted on an infrastructure managed by OpenStack, this type of storage will be the easiest you can access and use. It is available via OpenStack dashboard and will look just like a harddrive in your VM. However, note with this solution only the users and services with access to your VM can access the storage folder. If you need to provide access to data to external users but you do not want to provide VM access, you probably have to look for another alternative. However, you can still use Cinder to extend your VM storage or combine it with other solutions providing the interface you like the most (e.g. Nextcloud). 5.4.2 OpenStack Swift Another solution by OpenStack. If you need fast data access with infinite scalability (no need to reshape volumes) you probably should look into Object Storage technology. Swift (OpenStack) is the storage alternative to Cinder and probably one of your better options to work with object storage. There are multiple ways to access Swift storage, however they might not be so intuitive. If you decide to use rclone to synchronise your filesystem with Swift, the tool EGI Swift Finder can implement all the discovery and configuration for you. 5.4.3 RClone Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols. Rclone mounts any local, cloud or virtual filesystem as a disk on Windows, macOS, linux and FreeBSD, and also serves these over SFTP, HTTP, WebDAV, FTP and DLNA. Authentication and Authorisation will depend on the protocol you choose. • • • 5.3.5 Python API for AAI in services: flaat - 24/28 - Creative Commons "CC BY-SA 3.0" To facilitate the use of RClone, which was developed in a different context, a utility program “EGI Swift Finder” that sets up the environment for the EOSC context was developed. Links: RClone homepage EGI Swift Finder Documentation for RClone in the EGI context 5.4.4 Nextcloud Nextcloud is a suite of client-server software for creating and using file hosting services. Software is free and open-source making anyone allowed to install and operate it on their own private server devices. Manage and access your files knowing your data is in your data centre, on a server managed by you or your team, rather than floating somewhere in the cloud. It is simple to install and deploy, for example in one of your hosts at the Federated Cloud. Nextcloud is designed to be accessed via the web interface and WebDAV. Authentication via EGI Check In currently only works for the web interface. To use WebDAV and other protocols, currently passwords or OAuth2 Tokens have to be created in the web interface, before they can be used on the commandline. Links: Nextcloud Homepage 5.4.5 EGI DataHub A data management solution trying to provide High-performance with unified data access across globally distributed environments. If you have a very distributed cluster that your services need to access, this is probably an option for you. The data organisation and sharing is similar to a filesystem, users organise their data in virtual volumes called spaces and share access between groups. To access your data you have multiple options such as web interface, CLI (command-line interface) or an API. Authentication and authorisation are based on OpenID Connect and SAML, supporting as well the usage of tokens at API level. Links: EGI DataHub OneData Documentation 5.4.6 B2Share B2SHARE is a user-friendly, reliable and trustworthy way for researchers, scientific communities and citizen scientists to store, publish and share research data in a FAIR way. B2SHARE is a solution that facilitates research data storage, guarantees longterm persistence of data and allows data, results or ideas to be shared worldwide. B2SHARE supports community domains with metadata extensions, access rules and publishing workflows. EUDAT offers communities and organisations customised instances and/or access to repositories supporting large datasets. To manage your data there is a web interface and HTTP API. Authentication and authorisation are based on password or OIDC, using access tokens in the case of the API. Note that EUDAT encourages FAIR principles, so double check the privacy of your data (e.g. Metadata is always publicly available). Links: B2Share technical documentation EUDAT service catalogue entry • • • • • • • • 5.4.4 Nextcloud - 25/28 - Creative Commons "CC BY-SA 3.0"