ENHANCING THE TESTING OF CERN'S LINUX SOFTWARE BUILDING SERVICE AND IMPROVING MONITORING

ENHANCING THE TESTING OF CERN‘S LINUX SOFTWARE BUILDING SERVICE AND IMPROVING MONITORING AUGUST 2024 AUTHOR: Zeynep Öztürk SUPERVISOR(S): Marta Vila Fernandes Alex Iribarren CERN OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 PROJECT SPECIFICATION Linux Software Building Service – Koji, Linux Operating System, GitLab CI-CD Pipelines, Grafana, OpenSearch, Monitoring. 2 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 ABSTRACT Selected to work as OpenLab Summer Student at CERN in the “Enhancing the Testing of CERN’s Linux Software Building Service And Improving Monitoring” project. This report contains the work done over two months on two independent projects. In the first project, a Koji tag structure was created for the missing operating systems supported at CERN by the Linux team in the kojitest environment. Improvements to the Koji-tester pipeline were made to ensure that the most common Koji operations run successfully. It is quite an important task to maintain the quality of the Linux software building service, ensuring that upgrades with each new Koji release are smooth and reliable. After the creation of a new OpenSearch indexes, the dashboards developed in Opensearch needed to be rebuilt. The Linux team decided to move it to Grafana, a very reliable monitor tool. There are twenty-seven Opensearch dashboards that need to be reviewed and moved to Grafana. We migrated 2 out of 27 to monitor Nomad jobs and https://linuxsoft.cern.ch/ accesses. 3 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 TABLE OF CONTENTS 1. INTRODUCTION 05 2. PROJECT OVERVIEW 06 2.1 Enhancing the Testing of CERN’s Linux Building Service 2.2 Migration of Dashboards from OpenSearch to Grafana CONCLUSION 15 FUTURE WORKS 15 REFERENCES 16 4 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 1. INTRODUCTION Koji is a system that helps us to build and tracking the RPMs and images. It also facilitates the organization of packages through tags, which are essentially the names given to the package repositories. CERN’s Linux software building service uses Koji, which requires secure and reliable upgrades to ensure smooth functionality of the service. In order to effectively test and develop the Koji system, the CERN Linux team developed a GitLab project called Koji-Tester. The aim of the first project was to improve the pipeline of the Koji-Tester project by defining= Koji tag structures for different operating systems supported at CERN, in the Kojitest environment. In addition, monitoring is a very important task nowadays, and the Linux team uses OpenSearch to keep the logs of Nomad jobs and access to linuxsoft.cern.ch. Due to changes in the OpenSearch indexes, all the visualizations had to be reviewed and migrated to Grafana, a more flexible monitoring tool. 5 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 2. PROJECT OVERVIEW a. Enhancing the Testing of CERN’s Linux Building Service 2.a.1 Overview of Koji and Koji-Tester Koji is a system that builds and tracks RPMs and images securely, [1]. Like any software, Koji may need to be upgraded. Koji-Tester is a pipeline with different tests that helps us ensure that koji.cern.ch (production) and kojitest.cern.ch (testing) are working fine, especially during upgrades, [2]. We always upgrade the kojitest environment and keep it for at least one week before we upgrade the koji production environment. I got Admin privileges to access the Kojitest environment in this project. 2.a.2 Objectives and Goals The main goal of this project is to ensure that the Koji system works safely and smoothly for building RPMs, cloud images, and Docker images for different operating systems supported at CERN during any upgrade. Table 1 shows the operating systems supported by the CERN Linux team and their deprecation dates. Operating System Deprecation Date Supported by the Linux team SLC 6: Scientific Linux CERN 6 30.11.2020 no CC7: CERN CentOS 7 30.06.2024 no CS8: CentOS Stream 8 30.09.2023 no CS9: CentOS Stream 9 30.06.2023 no RHEL7: Red Hat Enterprise Linux 7 30.06.2024 no RHEL8: Red Hat Enterprise Linux 8 - yes RHEL9: Red Hat Enterprise Linux 9 - yes ALMA8: AlmaLinux 8 - yes ALMA9: AlmaLinux 9 - yes Table 1. Supported Operating Systems 6 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 The objective is that the Koji-Tester pipeline run each test successfully for all operating systems supported at CERN. Additionally, a pipeline schedule should be created to trigger the Koji-Tester pipeline daily on GitLab. The tests included: RPM building Cloud images building  CS8 Docker image building Alma 8 Docker image building Alma 9 Docker image building Driver Update Disks building Figure 1 shows the failed tests when we triggered the Koji-Tester pipeline at the beginning of the project. Figure 1. Triggered Pipeline at The Beginning To achieve this goal, new Koji tags for supported OS should be created in the Kojitest environment. A tag is the repository name that will contain the packages associated with it. 7 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 2.a.3 Creating New Koji Tag structure The Koji commands can be run on Lxplus or Aiadm. During the project, we connected Lxplus via SSH, as shown in Figure 2. Figure 2. Lxplus connection via SSH Then, the tag structure should be created in the Kojitest environment. There is a profile argument to switch between Koji and Kojitest environments as shown in the Figure 3. First, we need to define the Kojitest profile, then we can run the Koji commands to create tags. The tag structure can change depending on the operating system. Figure 3. Koji Profile To achieve the goal of this project, we needed to define the Koji tag structure. There was already a tag structure defined for AlmaLinux 9. We started by defining the tag structure for AlmaLinux 8 because it is similar. Figure 4. AlmaLinux 9 Tag 8 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 Figure 4 is an example of a AlmaLinux 9 tag, with the inheritance tag structure and the external repositories defined. These repositories can be defined in the Kojitest environment through Koji commands. We need to check the local repositories names belonging to each distribution. These repositories are an upstream copy, and they can differ between OS. When we compare Figure 5 and Figure 6, we can see that there is no CRB repository for AlmaLinux 8 like it is defined for AlmaLinux 9. The tagging processes should be made carefully, [3]. Figure 5. AlmaLinux 9 CERN Local Repository 9 OPENLAB SUMMER STUDENT PROJECT REPORT CERN openlab Report // 2024 5. REFERENCES [1] Koji Documentation https://docs.pagure.org/koji/HOWTO/ [2] Koji-Tester Project Repository https://gitlab.cern.ch/linuxsupport/testing/koji-tester [3] Repository of Alma 8 https://linuxsoft.cern.ch/cern/alma/8/ [4] OpenSearch Dashboards https://opensearch.org/docs/latest/about/ [5] Grafana Documentation https://grafana.com/docs/grafana/latest/ 16