EOSC Monitoring: Architecture and Interoperability Guidelines

eoscfuture.eu intern EOSC Monitoring: Architecture and Interoperability Guidelines Version 1.0 September 2023 eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 1 EOSC Monitoring: Architecture and Interoperability Guidelines Lead by National Infrastructures for Research and Technology – GRNET SA Authored by Koumantaros, Konstantinos, Themis Zamani, Kagkelidis Konstantinos, Thermolia Chrysa (GRNET). Emir Imamagic; Daniel Vrcic; Katarina Zailac (SRCE) Cyril L'Orphelin (CNRS) Dissemination Level of the Document Public Abstract Monitoring is the key service needed to gain insights into an infrastructure. It needs to be continuous and ondemand to quickly detect, correlate, and analyse data for a fast reaction to anomalous behaviour. The challenge of this type of monitoring is how to quickly identify and correlate problems before they affect end-users and ultimately the productivity of the organisation. Management teams can monitor the availability and reliability of the services from a high level view down to individual system metrics and monitor the conformance of multiple SLAs. The EOSC Monitoring service combines two operational monitoring services: the EOSC-CORE and the EOSC-Exchange Monitoring Services, Respectively monitoring the EOSC-Core services (EOSC Core Monitoring) and the services onboarded to the Marketplace (EOSC-Exchange Monitoring). The EOSC Monitoring services were implemented adopting the ARGO technology. This document describes the current architecture of the EOSC Monitoring and provides guidelines for five (5) integration Options available. eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 2 Version History Version Date Authors/Contributors Description V0.1 01/01/2022 Koumantaros, Konstantinos, Themis Zamani, Kagkelidis Konstantinos, Thermolia Chrysa (GRNET). Emir Imamagic; Daniel Vrcic; Katarina Zailac (SRCE) Cyril L'Orphelin (CNRS) Initiation – Proposed ToC – First draft V0.2 15/01/2022 Koumantaros, Konstantinos, Themis Zamani, Kagkelidis Konstantinos, Thermolia Chrysa (GRNET). Emir Imamagic; Daniel Vrcic; Katarina Zailac (SRCE) Cyril L'Orphelin (CNRS) 1st draft V0.4 30/01/2022 Koumantaros, Konstantinos, Themis Zamani, Kagkelidis Konstantinos, Thermolia Chrysa (GRNET). Emir Imamagic; Daniel Vrcic; Katarina Zailac (SRCE) Cyril L'Orphelin (CNRS) 2nd draft V0.6 1/3/2022 Koumantaros Konstantinos, Themis Zamani, Kagkelidis Konstantinos 1st revision / review V0.8 1/7/2022 Koumantaros Konstantinos (GRNET) Apply new template. V0.9 01/09/2022 Koumantaros, Konstantinos, Themis Zamani, Kagkelidis Konstantinos, Thermolia Chrysa (GRNET). Emir Imamagic; Daniel Vrcic; Katarina Zailac (SRCE) Cyril L'Orphelin (CNRS) Initial Version Published to Zenodo. V1.0 01/09/2023 Kostas Koumantaros (GRNET) Revised Version with feedback received. Copyright Notice This work by Parties of the EOSC Future Consortium is licensed under a Creative Commons Attribution 4.0 International License The EOSC Future project is cofunded by the European Union Horizon Programme call INFRAEOSC-03-2020, Grant Agreement number 101017536. eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 3 Table of Contents 1 Glossary ..................................................................................................................................... 4 2 Introduction ............................................................................................................................... 5 2.1 Licensing Information: ............................................................................................ 5 2.2 Intended Audience .................................................................................................. 5 2.3 Description and Main Features ................................................................................. 5 3 Response to Community Need .................................................................................................... 6 4 High-level Architecture ............................................................................................................... 6 4.1 Definitions ............................................................................................................. 7 4.2 How the EOSC Monitoring service checks the status of a Service ................................ 9 5 Related Guidelines ...................................................................................................................... 9 6 Adopted Standards .................................................................................................................... 9 7 Integration Options .................................................................................................................. 10 7.1 Integration Option 1: Monitor an Onboarded Service ............................................... 10 7.2 Integration Option 2: Monitor an Infrastructure. ...................................................... 11 7.3 Integration Option 3: Integrate External Monitoring service. .................................. 12 The monitoring team assists the Provider to create the necessary profiles: .................................... 12 Supported monitoring Engine and Operating System (Nagios on Centos 7): .............................. 12 Other monitoring systems: ............................................................................................................ 13 7.4 Integration Option 4: Combine Results of existing ARGO Tenants. ........................... 14 7.5 Integration Option 5: Third-party services exploiting EOSC Monitoring data .............. 15 7.5.1 Example: Exploit results for the EOSC Messaging Service: ........................................................... 15 Example ............................................................................................................................................. 17 API call examples for status reports .............................................................................................. 17 Detailed documentation: https://argoeu.github.io/api/v3/status/ ............................................... 17 Example ............................................................................................................................................. 17 eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 4 1 Glossary EOSC Future project Glossary is incorporated by reference: https://wiki.eoscfuture.eu/x/JQCK. eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 5 2 Introduction 2.1 Licensing Information: For software licensing information, please visit: https://github.com/ARGOeu/argomonitoring/blob/master/LICENSE 2.2 Intended Audience Technical experts of service and resource providers that would like their services and/or resources to be interoperable or integrate with EOSC-Core Services. 2.3 Description and Main Features Monitoring is the key service needed to gain insights into an infrastructure. It needs to be continuous and on-demand to quickly detect, correlate, and analyse data for a fast reaction to anomalous behaviour. The challenge of this type of monitoring is how to quickly identify and correlate problems before they affect end-users and ultimately the productivity of the organisation. Management teams can monitor the availability and reliability of the services from a high-level view down to individual system metrics and monitor the conformance of multiple SLAs. The key functionalities offered by the EOSC Monitoring service are: • Monitoring of: o EOSC-Core services o EOSC-Exchange Services (see Integration Options section) • Reporting availability and reliability, • Visualisation of the services status, • Provide dashboard interfaces that can be target towards both Providers and End Users (e.g. Researchers), • Sending real-time alerts to Providers (Service operators) and EOSC-Core service operators to varying levels of complexity (for example, for the purposes of alerting operators to availability issues, or to alert the EOSC Provider Onboarding Team of issues with Resource Profiles). The dashboard design should enable easy access and visualisation of data for end-users. APIs are supported to allow third parties to gather monitoring data from the system . The EOSC Monitoring service was designed to: • Support multiple entry points (different types of systems can work together), • Being easily interoperable with other monitoring systems, • Operate the different components of the systems in High Availability, • Support for Multiple Tenants, Configurations, Metrics and profiles to add flexibility and ease of customisation. The EOSC Monitoring service combines two operational monitoring services: the EOSCCORE and the EOSC-Exchange Monitoring Services, respectively monitoring the EOSCCore services (EOSC Core Monitoring) and the services onboarded to the Marketplace (EOSC-Exchange Monitoring). eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 6 The EOSC Monitoring services were implemented adopting the ARGO technology. 3 Response to Community Need Two needs have been identified: 1. Services operated by Providers that do not have easy access to monitoring solutions, and would benefit from integrating with a federated monitoring service. 2. Providing Service metadata in a Resource Catalogue alone may not be sufficient to convince a user that the service is reliable and properly supported. Publishing a monitoring dashboard for a service can increase the confidence of a potential user of a service, that the service/resource is actively supported and available for use. 4 High-level Architecture The EOSC Monitoring service collects status (metrics) results from one or more monitoring engine(s) deployed across distributed infrastructure and delivers daily and/or monthly availability (A) and reliability (R) results for monitored services. Status results and A/R metrics are presented through a Web UI, with the ability for a user to drill-down from the availability of a site to individual test results that contributed to the computed figure. Figure 1. High level architecture of a Monitoring service eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 7 The main components of the EOSC Monitoring service are depicted in the high-level architecture diagram and described in Figure 1. Monitoring Engine(s): this component executes the service checks against the distributed infrastructure and delivers the metric data (probe check results) to the Messaging Service. Sources of Truth: The Monitoring system supports a number of connector plugins that are able to fetch topology, metrics and factors from various sources such as the CMDB and the EOSC Resource Catalogue. A Metric and Profile Management Component allows checks (probes) to be defined and associated with specific Services. Each combination of checks and service types forms a profile. Messaging: The monitoring system uses a Pub/Sub Messaging Service to connect its components. Computations & Analytics: Computational jobs are defined for ingesting data, calculating status and availability/reliability and a management service automatically configures, deploys and executes those jobs on a distributed processing engine for stateful computations. This component analyses the monitoring results and sends notifications based on a set of rules, to inform the Service Providers about the status of their services. WEB API: RESTful HTTP API service that provides access to status and availability/reliability results. It supports token-based authentication and authorisation with established roles. Results are provided in JSON Format. WEB UI: The Web UI is the component to present the information about the status of the services. The global information from the primary and heterogeneous data sources is retrieved by means of the different plugins. The collected information is structured and organised within configuration files in the service and, finally, made available to the web application without the need for any further computations. This modular architecture is conceived in order to make it easy to add new data sources and to use cached information if a primary source is unavailable. The resulting data is exposed through a RESTful web service interface. The following table defines key terms that help explain how the EOSC Monitoring service works and the types of information a Provider should supply/define to start to use the service. 4.1 Definitions Tenant The tenant is an isolated instance of the EOSC Monitoring service that relies on common components and provides the user with its own environment. The EOSC Monitor provides default UI and POEM URLs in following form: • UI: https://monitoring.eosc-portal.eu & monitoring-core.eoscportal.eu. • POEM: https://eosc.poem.argo.grnet.gr & eosccore.poem.argo.grnet.gr Custom URLs can also be used, in such cases the customer is responsible for providing valid certificates and DNS aliases. eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 8 The EOSC Monitoring service requires following topology information in order to monitor services: • the services and service endpoints they are running, • the way they are organised (e.g. groups of sites, groups of services), • the service actors (owners, admins, contact points). Topology EOSC Monitoring service requires the following topology information in order to monitor services: • the services and service endpoints they are running • the way they are organised (e.g. groups of sites, groups of services), • the service actors (owners, admins, contact points). The topology can be further extended with attributes needed for individual probes (e.g., service port or URL, path to be used in case of storage services, e.g.). The topology sources currently supported are: • EOSC Resource Catalogue • EGI Configuration Database (GOCDB) • EUDAT DPMT • JSON feed in the predefined format. Metric A metric is a small piece of software that checks specific functionality of a given service. For example, a metric such as Portal-WebCheck runs on a site and checks if the HTTP connection responds correctly or not. Probe A probe is a small piece of software that implements single or multiple tests. The probe must comply with the guidelines for monitoring probes. Registry of probes and metrics EOSC Monitoring Service provides a registry of probes and metrics. New probes and metrics can be added to the registry with the support of the EOSC Monitoring team. Metric Profile A Metric Profile is used to associate a Service with the corresponding metrics. eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 15 This integration option can be enabled by following these steps: The Provider should submit a request on the EOSC helpdesk describing: • Tenants to be used in the combined report; • Services and metrics; • Aggregation profile. The monitoring team creates a new tenant that will host the combined report. This tenant acts as a host tenant for the combined results and will rely on the data of the other tenants as input for the computations of the availability, reliability and status results. After the previous steps are completed, the monitoring of the service starts and the EOSC Monitoring Computation and Analytics component calculates availability and reliability of the services, and creates a report. The User can have a look at the A/R and status results from the combined reports from the UI. 7.5 Integration Option 5: Third-party services exploiting EOSC Monitoring data This option covers the scenario according to which the Provider needs to use the results of the EOSC Monitoring Service in an external service/dashboard. The customer can access the following information via an API: • A/R information about the service and its service components; • Status information about the service and its service components; • The topology and grouping of the service. This integration option can be enabled by following these steps: The user who wants to gain access to this type of monitoring information will get a token with readonly access to the A/R and status results. The user via the EOSC helpdesk may send his request to the monitoring team by sending: • The name of the service that requires the information; • An email to create the user able to access to the monitoring information; • The type of information (A/R results, status results, or both) The monitoring team will provide the required token and information, guidance on how to retrieve the information. 7.5.1 Example: Exploit results for the EOSC Messaging Service: In this example we are going to present how the user can get the availability, the reliability values and the status of the EOSC Messaging Service (endpoint: https://msg.argo.grnet.gr ) of the Organisation GRNET. The Monitoring Service Monitoring Service is checking the services at regular intervals. It actually runs explicit tests (checks) in order to assess the status of the service. The result of the checks decides on the status of the service. In order to display status information, it uses reports where it keeps all the necessary information. eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 16 At the same time, it produces useful conclusions about the monitoring item via the monitoring analytics engine. One very useful conclusion is to decide if the item is available for usage and if it is considered as reliable. To succeed this, availability/reliability values (hourly, daily, monthly) are calculated. These different types of information are also encapsulated in a report. The EOSC monitoring service monitors the Messaging Service and it performs the following checks • cert_validity_check: a metric that checks the validity of the certificate used by the service • ams_check: a metric that checks a list of functionalities provided by the messaging service. Based on the explanation provided above, the information about the service follows: Definition Value Description GROUP GRNET A collection of services SERVICE AMS The type of one of the services of the collection SERVICE endpoint msg.argo.grnet.gr(AMS) is defined as the combination of a hostname and Service Type. (a Service Type of AMS listening on port/s <ams-port/s> on the host msg.argo.grnet.gr is a service endpoint) Grouping used in the report SERVICEGROUPS the way the services are organized (e.g. in groups of sites, in groups of services) in the monitoring engine A/R report Default The place where the A/R results are provided. Status report Default The place where status results are provided. This is the configuration that the user will have to use to use the api calls. API call examples for A/R reports The api authenticates the user using the api-key within the x-api-key header. Users can specify time granularity (monthly or daily) for retrieved results and also format using the Accept header. Depending on the form of the request the user can request a group, service or service endpoint. Detailed documentation: https://argoeu.github.io/api/v3/results/ eoscfuture.eu EOSC Monitoring: Architecture and Interoperability Guidelines 17 Example For the AMS the corresponding api call to get the A/R of the service group GRNET is: Request for A/R results for service group GRNET $ curl -X GET -H "Accept: application/json" -H "Content-Type: application/json" -H "x-api-key: secret-token" https://api.argo.grnet.gr/api/v3/results/Default/SERVICEGROUPS/GRNET?start_time=2021-0805T00:00:00Z&end_time=2021-08-05T23:59:59Z API call examples for status reports The api authenticates the user using the api-key within the x-api-key header. Users can specify time granularity (monthly or daily) for retrieved results and also format using the Accept header. Depending on the form of the request the user can request a group, service or service endpoint. Detailed documentation: https://argoeu.github.io/api/v3/status/ Example For the EOSC Messaging Service the corresponding api call to get the status of the service group GRNET is: Request for status results for service group GRNET $ curl -X GET -H "Accept: application/json" -H "Content-Type: application/json" -H "x-api-key: secret-token" https://api.argo.grnet.gr/api/v3/status/Default/SERVICEGROUPS/GRNET?start_time=2021-0805T00:00:00Z&end_time=2021-08-05T23:59:59Z