GN5-1 CARNET OAV Architecture Analysis – Update

© GÉANT Association on behalf of the GN5-1 project. The research leading to these results has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101100680 (GN5-1). Co-funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. 31-12-2024 CARNET OAV Architecture Analysis – Update Grant Agreement No.: 101100680 Work Package: WP6 Task Item: Task 4 Nature of Document: White Paper Dissemination Level: PU (Public) Document ID: GN5-1-25-12W13E Authors: Lidija Jakovčić (CARNET), Damir Regvart (CARNET), Ivana Golub (PSNC), Maria Isabel Gandia Carriedo (CSUC/RedIRIS) Abstract This document is an update of a previous published analysis of the mapping of the CARNET architecture to the TM Forum’s Open Digital Architecture aiming to provide a standardised view of the components and implementations of orchestration, automation and virtualisation within Research and Education organisations connected to the NRENs. CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E ii Table of Contents Executive Summary 1 1 Introduction 2 2 Architecture Analysis 6 2.1 High-Level OAV Approach 6 2.2 Mapping to ODA Functional Architecture 6 2.2.1 Engagement Management 7 2.2.2 Party Management 9 2.2.3 Core Commerce Management 11 2.2.4 Production 12 2.2.5 Intelligence Management 14 3 Conclusions 15 References 16 Glossary 18 Table of Figures Figure 1.1: CARNET infrastructure 3 Figure 1.2: The TM Forum ODA functional architecture 4 Figure 2.1: CARNET’s NMS components mapped to the TM Forum ODA 7 Figure 2.2: CARNET NOC Portal 8 Figure 2.3: NOC dashboard with Grafana – visualisation overview 9 Figure 2.4: Institution information and related links in the Membership DB 10 Figure 2.5: NetBox dashboard 11 CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 1 Executive Summary Analysing National Research and Education Network (NREN) architectures from an orchestration, automation and virtualisation (OAV) point of view using a common reference architecture helps align efforts between the NRENs and find similarities in the way different functionalities and components are implemented, which in turn facilitates potential collaboration and future interoperability between organisations. The TM Forum Open Digital Architecture (ODA) is being used as a reference model to map several architectures in the GÉANT project. Mapping the architectures to a single blueprint makes it easier to cross-compare them, and helps the organisations working on digital transformation. A reference architecture like ODA supports the transition from traditional work organisation based on manual work and procedures to a model where services/products are automatically provisioned and orchestrated. This document presents an updated version of the mapping of the components implementing the Croatian Academic and Research NETwork’s (CARNET) network observability and other main components into the ODA functional blocks. It highlights the evolution and updates made to these components over the past three years. Since the publication of the first version, several components have been enhanced and updated to align with needs, ensuring improved functionality and scalability. CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 2 1 Introduction The purpose of this document is to update the previous analysis of the components of CARNET’s network management system architecture, focusing on the orchestration, automation and virtualisation (OAV) aspects of its implementation. The NREN community can use this analysis to compare their own components and approaches with those of CARNET, and may find similarities which inspire them to work together on their journey towards OAV. This mapping is part of a set of mappings of different NREN management architectures against ODA, thus providing a common reference point and the means for a cross-NREN comparative analysis of components and approaches. The Croatian Academic and Research NETwork – CARNET – is a public institution operating within the Ministry of Science and Education in the field of Information and Communication Technology and its application in education [CAN]. The CARNET network is a private network connecting both the academic, scientific and research community of the Republic of Croatia, and the primary and secondary education system institutions. CARNET also connects government sites and almost all health institutions in Croatia – hospitals, health centres, Croatian health insurance fund, and others. The CARNET network covers more than 5,000 locations, enabling a highly distributed community, especially when considering that this includes over 718 islands on the Croatian coast. CARNET’s mission is to provide reliable information flows for both the remote online education and the exchange of health-related data. The main activities carried out by CARNET are focused on three key areas: internet service provisioning, information society development, and education for the new era [CNW]. CARNET was founded in 1991, and was the first internet service provider in Croatia. Services (products in TM Forum ODA) such as helpdesk, file storage and online education courses were provided from the very beginning. The CARNET Computer Emergency Response Team (CERT) was also established in the early years, transforming in 2007 into the Croatian National CERT. The CARNET network has grown constantly over the years to keep pace with increasing user demands. Today, the CARNET IP/MPLS core has 4 regional PoPs, 23 county network PoPs and 2 data centre sites. In order to support a high-quality education experience, CARNET not only helps education providers to broadcast courses and events online, but also instituted its own “train the trainers” (later “e-Mentors”) programme that is in charge of building a community of CARNET trainers. The e-learning focus started in 2004 when the CARNET education centre Edupoint decided to strategically move to online delivery. Online courses are offered via the established E-learning Academy. Starting in 2005, schools were connected to the CARNET network, and hosting services were offered to both primary and secondary schools. The schools on the islands have been connected via the eIslands project since 2007, with CARNET providing broadband internet and video conferencing systems. This is when the CARNET Portal for Schools for teachers, students, parents, and schools was first launched. Most of the schools today use Loomen, the CARNET content management system (CMS) based on Moodle [PSS]. In addition, CARNET hosts a portal for the exchange of school textbooks, and has also been involved in the creation of the first Croatian online encyclopedia. Via the CERT, content filtering is also provided for more than 555,000 students in primary and secondary schools. CARNET is also responsible for the State Final Examination Portal, first established in 2008. Today, the Edu.hr portal provides a central access point to all CARNET education services [EDU]. The latest addition to the educational services family managed by CARNET is the MOOC courses portal [MOOC]. Introduction CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 3 In addition to e-Education services, CARNET also provides services for the Research and Education (R&E) community, such as the CARNET Cruncher service that provides its members remote access to supercomputing facilities. In 2007 CARNET established an integrated help desk for all CARNET users by merging the Customer Support Helpdesk (previously managed by the University Computing Centre) and the Helpdesk for Customer Support for the primary/secondary education system. CARNET is also in charge of all activities related to the Croatian top-level domain .hr, including the administration of free secondary and tertiary domains. Payable domains are registered using authorised registrars. To help CARNET system engineers with their daily activities, CARNET provides a number of “sys” services [SYS]. This includes services such as sys.backup (for secure remote data archiving), sys.monitor (for remote web monitoring using Icinga2) and sys.virtual (a virtual provider for small organisations, managed by CARNET). The CARNET Trusted Certificate Service (TCS) is provided via a portal and is provided to all CARNET members [CERT]. The e-Schools program, one of the largest projects for the digitalisation of the primary and secondary education system, has been successfully completed [ICTS]. This initiative involved designing school networks across over 2,400 locations, installing active network equipment, establishing functional wireless networks, delivering over 85,000 laptops for school staff, and facilitating the development of new e-services. The current focus is now on planning and implementing the e-Universities project. This project will upgrade the existing network equipment in all universities in Croatia. The equipment will include next-generation firewalls, routers, switches, and wireless access points with a centralised cloud management platform. There is also the continuous upgrade and maintaining of e-Schools project services and the Croatian Science and Education Cloud [ZOO] project services. Figure 1.1: CARNET infrastructure Introduction CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 4 CARNET network has made connection to the global network through the pan-European research network GÉANT, at a current connection speed of 100 Gb/s. The link to other Internet service providers in Croatia is realised through the Internet exchange in Croatia, CIX. Within Croatia, CARNET network connects larger and smaller Croatian cities at the mainland and a number of places on the islands. Major university centres (Dubrovnik, Osijek, Pula, Rijeka, Slavonski Brod, Split, Varaždin, Zadar, Zagreb) are connected with speed connections ranging from 2 Gb/s to 100 Gb/s, while smaller centers are secured at speeds of 1 Gb/s to 10 Gb/s. In Zagreb and Split, faculties and scientific institutions are connected through an infrastructure that supports speeds up to 10 Gb/s. CARNET network also connects institutions from the primary and secondary education system at access speeds of 4 Mb/s to 10 Gb/s. In Rijeka, Zagreb, Split and Pula, CARNET connects the locations of the institutions to the CARNET network with its own built infrastructure. CARNET’s OAV architecture analysis has been conducted using the TM Forum Open Digital Architecture (ODA) functional blocks as a reference point. The TM Forum ODA is promoted as a blueprint for new digital industry architectures and the rationale for its selection as a reference model by the Network eAcademy team of the Network Development work package (WP6) of the GN5-1 project is given in GN4-3 Deliverable D6.6 Transforming Services with Orchestration and Automation [D6.6]. The whole set of ODA documentation provides common terminology, a minimum set of core design principles, and groups of decoupled functionalities. Together they define the requirements for the implementation of an agile model-driven service management architecture that incorporates orchestration and automated operations, as well as virtualised or hybrid environments. The main idea behind ODA is that of decoupling and integration of components which enables independent choice of solutions for each component, while at the same time maintaining a unified overall approach that supports the full end-to-end service lifecycle (including interoperability). The high-level ODA functional architecture maps the main components by their capabilities into the ODA function blocks (see Figure 1.2). Figure 1.2: The TM Forum ODA functional architecture Introduction CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 5 In a nutshell: • The Engagement Management functional block focuses on the engagement with the end-users (people and systems) that can interact via multiple channels. • The Party Management functional block handles the processes that are related to all parties that interact with the organisation and defines their roles and relationships. • The Intelligence Management functional block is in charge of the implementation of data analytics processes, and, based on the analysis, provides closed control loops for full automation wherever possible. • The Core Commerce Management functional block focuses on the placement of products and services to the customers and manages the product lifecycle. • The Production functional block manages the delivery and lifecycle of all customer-facing and resourcefacing services that can be based on different technologies or might be a combination of multiple operational domains, including multi-domain services provided with the cooperation of other parties. CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 6 2 Architecture Analysis 2.1 High-Level OAV Approach In recent years, CARNET has seen huge growth in the number of connected locations, to the point where its network now serves over 5,000 sites. This number includes many schools that need to be managed, with over 1,300 LANs, more than 8,000 LAN switches, and 35,000 Wi-Fi Access Points. However, the number of engineers available to work in this field is extremely small when considering the highly distributed scenario, and number of end-sites and devices. In addition, there is a plethora of educational services that need to be provided by CARNET on top of the network that are put into place [DMS]. As a result, CARNET as an organisation is facing scalability challenges that can only be addressed by the implementation of OAV principles. The main principle that drives the evolution of service management in the NREN is the identification of repeatable tasks and their automation using scripting. This eliminates the need for all network engineers to acquire the expertise to handle each specific vendor equipment – which is of great interest to the NREN because its network is built using equipment sourced from multiple manufacturers. To support this evolution, network engineers developed a Network Operations Centre (NOC) portal, which is continuously improved to keep up with the growth of the network and any changes that occur. This portal offers access to a wide array of tools, including internal resources and custom open-source solutions, facilitating the daily activities of network engineers. Furthermore, CARNET has developed LAN Cloud Management [LANCLOUD], an in-house network management solution for school networks (which are predominantly wireless). Using this solution, the network provisioning and monitoring can be done remotely, via an API. This provides a vendor-agnostic perspective to school admins, and a remote global admin access to the CARNET Network Operations Centre (NOC) [ESC] [NMM]. The admin tools in this case include topology visualisation, network behaviour analysis, security with unified threat management, mobile device management, etc. CARNET also developed a self-care portal for school admins. The portal enables efficient and proactive troubleshooting via passive and active monitoring. The main aim of the solution is to move all of the complexity into the private CARNET cloud that will provide security, centralised management, and granular visibility of each client, user, and application. The technical support for this solution is using a three-tier system. The first tier is the e-School technician, the second tier is the CARNET Helpdesk, and the third tier is the CARNET NOC. The e-services that are provided include AAI@EduHR, LMS, CMS, e-Class register, CARNET fi (simplified authentication), GDPR360 (for user master data management), Mobile Device Management MDM, and Enterprise Resource Planning. 2.2 Mapping to ODA Functional Architecture When put into context of the TM Forum ODA functional representation, CARNET’s Network Management System (NMS) architecture components can be represented as in Figure 2.1 below. The white boxes in the diagram represent CARNET’s network management system architecture components, and their placement within the ODA functional blocks is defined based on their main functionalities [ODA]. If the components provide complex multifaceted functionalities, they are broken down into sub-components which are placed in the respective functional block (e.g. membership database). Architecture Analysis CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 7 Figure 2.1: CARNET’s NMS components mapped to the TM Forum ODA 2.2.1 Engagement Management A number of web portals offer access to the different services provided by CARNET. Information about most of the main services related to connectivity, internet service, training, collaboration, and security services is made available via the main CARNET web page. It then branches to different access portals depending on the service type, such as https://domene.hr/, https://certifikati.carnet.hr/, and https://meduza.carnet.hr/. In addition to the main website, there are a number of social media channels that are used by the NREN to highlight its activities and available services. For CARNET member institutions that have already subscribed to the main connectivity services, the Membership portal provides all institution-specific information. Through the portal, institutions can view the status of their services, order additional services, access service monitoring tools, report and track problems (by creating a problem ticket), access all agreements and service-related documentation, and provide feedback. The schools’ and school-related educational services are provided via a second set of portals including the government-supported e-Citizens system, on whose behalf CARNET manages the mToken application for generating one-time passwords. The educational portals that can be accessed by students, parents, teachers and schools include e-Dnevnik [DNE], Loomen [LOO], e-Lektire [ELE], Skole [SKOLE], e-Laboratorij [LAB], Nikola Tesla distance learning [TESLA], CMS, Sigma, Delta, and Edutorij. System admins and technicians in institutions that collaborate with CARNET can also access the sys portal that provides them with training resources, tools and services used in their daily activities. For CARNET employees, a special set of management portals has been developed. Architecture Analysis CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 14 2.2.5 Intelligence Management Within the Intelligence Management functional block there are a number of CARNET components that focus on gathering different types of information from multiple sources and visualising it using customised visualisation tools. These tools include: • Monitoring with Zenoss (SNMP, alarm trigger, performance, link/interface threshold) • Monitoring with LibreNMS (still under development) • NetFlow information (NFSEN) • Network weather map • Grafana with Influx DB integration for performance management A number of different performance-reporting tools are used for capacity tracking and planning, where the focus is on identifying both under-utilisation and over-utilisation of links. CARNET has developed a predictive, threshold-based system for link monitoring. If the institution or backbone link is utilising more bandwidth than a learned baseline, an alarm is shown in the NOC console. In the e-Schools realm, the CARNET Delta service focuses on learning analytics and educational data mining [DELTA] [CDE]. The goal of this service is to improve the decision-making and policy-making processes by supporting the CARNET NOC with relevant information, as well as improve overall learning outcomes. The learning analytics architecture is designed to store logs from all e-School-related services in an Oracle-based data lake (centralised repositories that store structured and unstructured data at any scale). The data lake is then accessed by an enterprise data warehouse system that performs business intelligence (DW/BI). The results from the analytics are presented through the service portal. In the future development of the e-Schools project, the service will be augmented with machine learning and related artificial intelligence algorithms for deep insight analytics and predictions. CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 15 3 Conclusions CARNET provides a large number of services and manages related requirements from its vast community of users. Network users demand an efficient and effective approach to service management that can scale easily. Having faced a rapid growth in locations and traffic, CARNET has leaned heavily on automation and orchestration opportunities to be able to produce solutions that are highly reliable and scalable in times of great demand. CARNET engineers have embraced the golden rule of automation – by focusing on automating repetitive daily tasks, engineers become more efficient, freeing them up to focus on more complex issues. The analysis of the components used for service management in all ODA functional blocks show that, for most of the network and service management activities, CARNET has decided to either use a large set of tools developed in-house or customise existing open-source solutions. The main components that provide functionalities across several ODA functional domains are NetBox and the CARNET NOC portal. The OKD solution for hosting services provides an agile platform for rapid application development, while the network devices’ automated configuration scripts are accelerating the process of network management in the backbone and school LANs. The e-Schools project and the ongoing e-Universities project are significantly contributing to the digital transformation of Croatia's research and education landscape into a fully digital platform. These initiatives lay the foundation for a technologically advanced platform that facilitates efficient knowledge exchange and enhances access to education. The results of the previous architecture analysis [COAA] provided valuable insights into the existing framework, identified areas for improvement and optimisation shaping the development of these initiatives. CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 16 References [AAI] AAI@EduHr Authentication and authorisation infrastructure https://www.aaiedu.hr/ [COAA] CARNET OAV Architecture Analysis, https://resources.geant.org/wpcontent/uploads/2022/02/GN4-3_White-Paper_CARnet-OAV-ArchitectureAnalysis.pdf [CAN] About CARNET, https://www.carnet.hr/en/about-carnet/ [CDE] Natasha Glavor, Education – Community Discussion Learning Analytics in CARnet, GEANT TF-EDU, November 2020, https://wiki.geant.org/download/attachments/154996732/CARNET_delta-TFEDU.pdf?version=1&modificationDate=1607000688099&api=v2 [CERT] https://certifikati.carnet.hr/ [CNW] https://en.wikipedia.org/wiki/CARNet [D6.6] GN4-3 D6.6 Transforming Services with Orchestration and Automation, November 2020, https://resources.geant.org/wp-content/uploads/2022/02/D6.6Transforming_Services_with_Orchestration_and_Automation.pdf [DELTA] https://delta.e-skole.hr/ 1 [DMS] Gordana Jugo, Igor Balaban, Marijana Pezelj, Nina Begicevic Redjep. Development of a Model to Assess the Digitally Mature Schools in Croatia. 11th IFIP World Conference on Computers in Education (WCCE), Jul 2017, Dublin, Ireland. pp.169-178, ff10.1007/978-3-319-74310-3_19ff. ffhal01762871f https://hal.inria.fr/hal-01762871/document [DNE] https://ocjene.skole.hr/ [EDU] http://www.edu.hr [ELE] https://lektire.skole.hr/ [ESC] Damir Regvart, Bojan Schmidt, E-Schools in Croatia, SIG-NOC meeting April 2019, https://geant.app.box.com/s/fji5tdbv2dhxlfed137kl7mj806mmi16 [ICT] Vuk, Branka, Vedran Mornar, and Ivica Boticki. "ICT in School Education in Croatia." ICT in Education and Implications for the Belt and Road Initiative. Springer, Singapore, 2020. 37-52. https://www.researchgate.net/publication/346316478_ICT_in_School_Education_ in_Croatia [ICTS] Adequate ICT Infrastructure in eSchools project, https://www.carnet.hr/en/thee-schools-project-successfully-digitizes-all-schools-in-croatia/ [LAB] https://e-laboratorij.carnet.hr/ [LANCLOUD] https://ctrl.e-skole.hr 2 [LOO] https://loomen.carnet.hr/ [MOOC] https://mooc.carnet.hr/ [NMM] Darko Paric, Bojan Schmidt, Lightning talk, Workshop on Network Management and Monitoring, October 2019. 1 Access to this site is restricted. It is only available within IP addresses in Croatia. 2 Access to this site is restricted. It is only available within IP addresses in Croatia. Conclusions CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 17 https://wiki.geant.org/display/PUB/Workshop+on+Network+Management+and+M onitoring?preview=/131629403/131635133/lightning_talks-1.pdf [ODA] TM Forum, GB998 Open Digital Architecture (ODA) Concepts & Principles v2.1.0, March 2021, https://www.tmforum.org/resources/reference/gb998-open-digital-architectureoda-concepts-principles-v2-1-0/ [OKD] https://www.okd.io/ [PSS] Gordana Jugo, Mirta Janesh, Implementing Online Courses in Primary and Secondary Schools in Croatia - The Pilot Project, IADIS 2008. http://www.iadisportal.org/e-learning-2008-proceedings [RTS] Open-source enterprise grade ticketing system, Request Tracker, https://bestpractical.com/ [SKOLE] https://www.skole.hr/ [SYS] https://sysportal.carnet.hr/ [TESLA] https://tesla.carnet.hr/ [TNC] Damir Regvart, Lidija Jakovčić, Silvije Milisic, CARNET OAV, OAV BoF, TNC’19 [ZOO] Marić, Ivan, et al. “Dani e-infrastrukture Srce DEI 2019: povezivanje, komunikacija, suradnja. Uvodna konferencija projekta HR-ZOO: knjiga sažetaka.” Dani e-infrastrukture-Srce DEI 2019: povezivanje, komunikacija, suradnja. Sveučilište u Zagrebu, Sveučilišni računski centar-Srce, 2019. https://repozitorij.srce.unizg.hr/islandora/object/srce:386/datastream/FILE0/dow nload <Title> Document ID: <Doc Property: Keywords> 18 Glossary AAI Authentication and Authorisation Infrastructure API Application Programming Interface BI Business Intelligence CARNET The Croatian Academic and Research NETwork CEST Central European Standard Time CERT Computer Emergency Response Team CLI Command Line Interface CMS Content Management System CPE Customer Premise Equipment DB Database DDoS Distributed Denial of Service DNS Domain Name System DW Data Warehouse GDPR General Data Protection Regulation IdP Identity Provider IP Internet Protocol IX Internet Exchange LAN Local Area Network LMS Learning Management System MDM Mobile Device Management ML Machine Learning MOOC Massive Open Online Courses MPLS Multiprotocol Label Switching NMS Network Management System NOC Network Operations Centre NREN National Research and Education Network OAV Orchestration, Automation and Virtualisation ODA Open Digital Architecture PoP Point of Presence OKD Origin Kubernetes Distribution QoS Quality of Service SDN Software Defined Network SIM Subscriber Identity Module SLA Service Level Agreement SMS Short Message Service SNMP Simple Network Management Protocol SSH Secure Shell R&E Research and Education TCS Trusted Certificate Service URL Uniform Resource Locator VLAN Virtual LAN Glossary CARNET Orchestration, Automation and Virtualisation Architecture Analysis – Update Document ID: GN5-1-25-12W13E 19 VLE Virtual Learning Environment VRF Virtual Routing and Forwarding VoIP Voice over IP WP Work Package