Monitoring Evaluation for Large-Scale Environments and Optimizing Data System Health

Monitoring Evaluation for Large-Scale Environments and Optimizing Data System Health Anastasiia Petrovych1and Apostolos Karvelas2 1Ukrainian Catholic University, anastasiyapetro[email protected] 2CERN, [email protected] The LHCb experiment at the Large Hadron Collider investigates particle physics by analyzing heavy quarks in high-energy proton collisions and has a high-throughput data acquisition system. The experiment’s online computer cluster is encountering performance anomalies, and to address this, an automated monitoring system utilizing the Kubeflow pipeline is being considered as a solution to optimize resource usage and minimize downtime. Introduction The LHCb experiment 1plays a significant role in discovering particle physics using heavy quarks such as b and c quarks. It has been designed to resist proton collisions of 14 TeV center-of-mass energy up to an instantaneous luminosity of 2×1033 cm−2s−1.Instantaneous luminosity measures how tightly particles are packed into a space, the LHC’s proton beam in the given case. A higher rate means a greater likelihood of particle collision, resulting in a desired interaction. The data acquisition system is designed to perform a full readout and reconstruction of the 40 MHz rate without any low-level hardware trigger, with an expected collected throughput of ∼32 Tb/s. The online computer cluster is built using 170 servers, which are interconnected via a 200 Gb/s InfiniBand-based network. InfiniBand, developed by the InfiniBand Trade Association (IBTA), is an industry-standard communications specification defining a switched fabric architecture for connecting servers, communication infrastructure equipment, storage, and embedded systems in data centers. It is known for its high performance, low latency, and scalability, delivering up to 400 gigabits per second throughput. Highly scalable, InfiniBand can support tens of thousands of nodes within a single subnet. As a result, the data from the LHCb subdetectors are fractured throughout all the servers and must be assembled before the event reconstruction. Servers play a crucial role in the LHCb experiment because they are used to collect, preprocess, and store data for further discoveries. Thus, a dedicated team of DevOps and MLOps manages the infrastructure. Furthermore, it is their primary responsibility to keep the servers operational despite various anomalies in their performance. Detecting them in a large fleet of servers and responding quickly are evident challenges for the team. Therefore, they have set up Prometheus to collect various metrics and logs to monitor servers’ performance. However, diagnosing related problems is complex and time-consuming, as the volume of data is enormous. There is a growing consensus that it should be automated to save resources and detect potential issues in time. One possible solution is an automated pipeline using Kubeflow, an open-source platform for building and deploying portable, scalable Machine Learning workflows based on Docker containers. Fig. 1. Map of the CERN accelerator complex Cern accelerator complex Method Kubeflow - a platform for Machine Learning and MLOps on Kubernetes, was introduced by Google as an open-source tool. In Kubeflow, various software components correspond to key stages of the machine learning lifecycle, including development, training, serving, and automated processes. Kubernetes accelerates the ML lifecycle by offering flexibility, scalability, and portability for all the steps. It allows scaling Machine Learning workloads based on demand, optimizes resource allocation, ensures portability across different environments and cloud platforms, and provides fault tolerance and self-healing capabilities. In the given case, the primary purpose of this pipeline 2is to create an anomaly detection system performed on the daily collected data and an alert system for all the team members via email with detailed information. It includes nine components, divided into four main stages: data collection, data processing, model testing, and anomaly detection. Anastasiia Petrovych | August 20, 2024 | 1 - 5 Fig. 2. Anomaly detection pipeline 1. Data collection. The pipeline begins with data collection from the previous day using Prometheus and server logs. Prometheus is an open-source software application designed for event monitoring and alerting in dynamic cloud environments, particularly for containerized applications and services. It gathers performance metrics such as CPU utilization, disk I/O, memory and network activity, and provides an overview of the system health. Server logs capture the server status, indicating whether it is running, stopped, not ready, or ready. Based on their specific states, these statuses are then converted into binary values of 0 or 1. These metrics are then stored in datasets, which are subsequently used for model training and testing. 2. Data processing. This step includes the three main components: handling missing values and choosing important features from the correlation matrix, dataset splitting, and feature scaling. 2.1. Feature selection and handling of missing values. The dataset consists of various system performance metrics collected over time, which are essential for monitoring and analyzing the health and performance of computing resources. The dataset includes 19 features - mostly numerical, with one boolean feature - recorded across different time intervals. Key attributes of the dataset include a timestamp, hostname, CPU usage (idle, system, user), disk I/O time, hlt2 status, kernel vmstat pgpgin/pgpgout, memory available percent, network bytes (received and sent), network drops/errors, swap used percent, system load over last 5 minutes, and system uptime. A detailed explanation of these metrics is provided in Appendix 4. Subsequently, a subset of relevant features was selected for model construction using the correlation matrix. A correlation matrix computes correlation coefficients between features and a factor of interest and helps identify features with high-potential associations. For training, variables such as timestamp, hostname, hlt2, and up were exFig. 3. Correlation matrix cluded. 2.2. Dataset splitting. Data splitting involves dividing a dataset into distinct subsets, such as training, validation, and test sets. This process is essential for training models, tuning parameters, and evaluating performance. In this Kubeflow pipeline, the dataset is divided into 80% for training and 20% for validation. 2.3. Feature scaling. Feature scaling is a method used to standardize the independent features in the data within a fixed range. It is carried out during the data pre-processing to address highly varying magnitudes, values, or units. If feature scaling is not applied, a machine learning algorithm tends to assign greater weight to higher values and consider smaller values lower than their actual value. The two main scalers used in this pipeline were Standard and MinMax. Standard scaler works by normalizing the distribution of each feature to ensure that they have a mean of zero and a standard deviation of one. This approach guarantees that all features are at the same scale, preventing any one feature from overpowering the learning process because of its larger size. Mathematically, this transformation is expressed z=x−µ σ where x represents the original feature value, µ is the mean of the feature, σis the standard deviation, and z is the standardized feature value. A standard scaler does not change the shape of the distributions of each feature; it only shifts and scales them, preserving the relative relationships between values. Min-Max scaler shrinks the range of the data such that it is now between 0 and 1, following the formula for each feature xi−min(x) max(x)−min(x) Anastasiia Petrovych | August 20, 2024 | 2 - 5 Fig. 4. Standard Scaler Feature Scaling with scikit-learn This scaler performs better for cases where the Standard scaler might not work well, for example if the distribution is not Gaussian or the standard deviation is very small. Fig. 5. Min-Max Scaler Feature Scaling with scikit-learn 3. Model testing. The collected dataset does not have labels, so the pipeline uses unsupervised learning. Unsupervised learning utilizes Machine Learning algorithms to analyze and cluster unlabeled data sets. They discover hidden patterns or data groupings without human intervention. In this Kubeflow pipeline, a Sequential model and Variational Autoencoder (VAE) were used. ASequential model is a linear stack of layers where each layer has exactly one input tensor and output tensors. It is a simple and powerful model that creates deep neural networks where each layer feeds into the next in a sequence. This model is particularly well-suited for cases where the network architecture is straightforward, with a clear flow from the input through hidden layers to the output. AVariational Autoencoder (VAE) is a type of neural network that learns to reproduce input data and maps it to a latent space. The term "variational" refers to variational inference, a method closely related to variational Bayes. Variational inference can be seen as an extension of the Fig. 6. Sequential model architecture Keras Models(CNN): Functional Vs. Sequential expectation-maximization (EM) algorithm, which is traditionally used to optimize latent variable models when direct maximization of p(x)is not possible. However, unlike EM, variational inference operates within a Bayesian framework, where the goal is to learn distributions of parameters rather than point estimates. An autoencoder is a neural network designed to learn to reproduce its input. The important feature of an autoencoder is its use of fewer hidden units than input units, creating a bottleneck that forces the network to learn a compact data representation. Fig. 7. VAE architecture Variational autoencoder After model prediction, the mean absolute and squared errors were applied to test its accuracy. Mean absolute error (MAE) evaluates the absolute distance between predicted and dataset values, taking the average of all observations. It can be expressed as MAE = 1 nPn i=1   yreal i−ypred i   Mean squared error (MSE) squares the distance to make results positive. Higher errors weigh more in the metric than lower ones due to the nature of the power function. MSE = 1 nPn i=1 yreal i−ypred i2 In some cases, the fast convergence of MSE can be advantageous, but its vulnerability to outlier impact reduces its suitability for datasets that include anomalies. As a result, MAE’s resilience to outliers ensures a more unbiased training process, resulting in better-generalized models. 4. Anomaly detection. Based on the model prediction, the reconstruction loss is calculated, the difference between an original input and a model output. Higher values of the reconstruction error indicate a higher probability of detecting an anomaly. The threshold as 99.9 percentile was chosen Anastasiia Petrovych | August 20, 2024 | 3 - 5 for classifying it, meaning that only 0.1 percent of the data is above this value and is detected as anomalies. After this process, an automated email is sent to LHCb team members with a detailed description of the servers’ issues for the last 24 hours.8. Fig. 8. Example of anomaly detection report Conclusions The Kubeflow pipeline brings full automation and high scalability to the anomaly detection process, helping to prevent downtimes and improper server activity. This system involves the whole process of anomaly detection based on the collected data. As a result, the following pipeline reduces the resources spent on manual intervention, repetitive tasks, and inefficient processes. This work aimed to automate manual monitoring of LHCb servers to diagnose anomalies in their performance. These servers play a crucial role in the experiment as they collect, preprocess, and store data for further discoveries. Their downtime badly affects results, so constant monitoring is essential to find anomalies and respond quickly. The solution was implemented with Kubeflow – an opensource tool for MLOps developed by Google. This platform enabled the creation of a fully automatic pipeline for daily anomaly detection, which notifies the team via email in case of alerts. Kubeflow provides auto-scaling and optimized resource allocation, ensuring a stable operation under any load. The pipeline comprises nine components, which are organized into four stages. Initially, data is collected from Prometheus and server logs and then preprocessed to train the machine learning model. Key features are selected and scaled for effective training, and the data is split into training and validation sets. Using a Sequential model or Variational Autoencoder, an unsupervised model is trained and tested for accuracy. The model identifies anomalies based on reconstruction loss, with values exceeding a 99.9 The anomalies detected by the pipeline are reported to the team via email, which contains a detailed description of the servers’ issues. As a result, the DevOps and MLOps team receives daily notifications about the servers’ status and problems for quick response and resolution, significantly speeding up the manual processes. Appendix Dataset Description. •timestamp: The date and time when the metrics were recorded. •hostname: The name or identifier of the machine or server from which the metrics were collected. •cpu_usage_idle: The percentage of CPU time spent in the idle state. A higher value indicates that the CPU was less active during the recording period. •cpu_usage_system: The percentage of CPU time used by system processes, which are typically low-level operations performed by the operating system. •cpu_usage_user: The percentage of CPU time used by user-level processes. This includes all the operations requested by applications running on the system. •diskio_io_time: The total time spent on I/O operations for disk drives, typically measured in milliseconds. This metric reflects the activity level of the disk subsystem. •hlt2: Indicates server’s status as 1 or 0. •kernel_vmstat_pgpgin: The number of kilobytes the kernel has paged in from disk to memory. This indicates the rate of data being read from disk into memory. •kernel_vmstat_pgpgout: The number of kilobytes the kernel has paged out from memory to disk. This reflects the rate at which data is written from memory to disk. •mem_available_percent: The percentage of physical memory available on the system. This metric helps in understanding the memory utilization level. •net_bytes_recv: The total number of bytes received over the network interface. This metric is useful for analyzing network input bandwidth. Anastasiia Petrovych | August 20, 2024 | 4 - 5 •net_bytes_sent: The total number of bytes sent over the network interface. This metric helps in understanding the network output bandwidth. •net_drop_in: The number of incoming packets that were dropped. This can indicate issues with network congestion or buffer overflows. •net_drop_out: The number of outgoing packets that were dropped. This can be a sign of network congestion or hardware issues. •net_err_in: The number of errors that occurred while receiving packets. High values could indicate issues with the network interface or connection quality. •net_err_out: The number of errors that occurred while sending packets. High values may suggest problems with network transmission. •swap_used_percent: The percentage of swap memory currently in use. Swap memory is used when physical memory (RAM) is fully utilized, so high values can indicate memory overload. •system_load5: The system load averaged over the last 5 minutes. This metric provides a quick view of system workload and helps in identifying potential bottlenecks. •up: Refers to the uptime of the system, indicating how long the system has been running without a reboot. Bibliography 1. A new data-centre for the LHCb experiment 2. Kubeflow documentation 3. Prometheus overview 4. AutoEncoder (AE) and Variational AutoEncoder (VAE) 5. Sequential anomaly detection based on temporaldifference learning: Principles, models and case studies Anastasiia Petrovych | August 20, 2024 | 5 - 5