The Preliminary Study of Federated Learning for Enhancing the HPC Job Scheduling Policy

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE The Preliminary Study of Federated Learning for Enhancing the HPC Job Scheduling Policy Weicheng Huang National Center for Highperformance Computing National Applied Research Laboratories Hsinchu, Taiwan, R.O.C. [email protected] Le An DigiARC National Institute of Advanced Industrial Science and Technology Tokyo, Japan [email protected] Truong Thao Nguyen DigiARC National Institute of Advanced Industrial Science and Technology Tokyo, Japan [email protected]p Hsui-Mei Chou National Center for Highperformance Computing National Applied Research Laboratories Hsinchu, Taiwan, R.O.C. [email protected]w Te-Min Chen National Center for Highperformance Computing National Applied Research Laboratories Hsinchu, Taiwan, R.O.C. gv[email protected] Jason H. Haga DigiARC National Institute of Advanced Industrial Science and Technology Tokyo, Japan [email protected] Yusuke Tanimura DigiARC National Institute of Advanced Industrial Science and Technology Tokyo, Japan [email protected].jp Abstract— With joint Federated Learning Platform initiated, the optimization of job queuing system configurations of HPC facilities is used to demonstrate the usability of such approach for the international collaboration. The progress of this attempted collaboration is presented in this paper from different technical aspects, including platform, data management, time series data analysis, etc. Also, a preliminary result using CIFA10 dataset was demonstrated. Keywords— Machine Learning, Federated Learning, Scheduler INTRODUCTION Data, the fuel of modern scientific and engineering development, needs to be shared in order to generate greater value. Especially when trying to take advantage of the power of Machine Learning (ML), the amount of sensible data plays a crucial role in the success of the ML model. On the other hand, the sensitivity and privacy issues of the data have to be managed for a safer data sharing. Among various technologies to tackle this issue, the federated learning (FL) is a promising technology which shared no raw data but the ML model tuning parameters. An attempt of an international federated learning platform, which is at its infant stage, was initiated. In order to promote this initiative of FL platform, a demonstrative application; optimization of HPC job scheduler, is also under the investigation. The work load and resource management from both AIST and NCHC will be shared via the FL mechanism instead of the exchange of raw data. Thus, the concern of data privacy for both sites can be preserved, to a certain degree. Currently, the FL platform as well as possible technologies attempted has been studied, tested and partially implemented. In the paper, the progress of this attempt is shared and hopefully further shade lights on the future direction of the work. I. TECHNOLOGY The technologies involving with the work are itemized and elaborated with their progress in the following. A. Federated Learning Platform The FL platform, as illustrated in Figure 1, is intended to be built across international borders to facilitate AI/ML application with power of shared data but without really sharing the data itself. Although there are arguments regarding the security of data through FL platform, nevertheless, it imposes additional difficulties and thus extra layer of protection for the data involved in the training. Various FL platforms have been studied and with the NVFLARE from Nvidia[1][2][3] as the final choice. B. Data and ML model training The data of the application involves data from different computers managed by different authorities to serve diverse • National Science and Technology Council, Taiwan, R. O. C. • National Applied Research Laboratories, Taiwan, R. O. C. deman supply Figure 2: Illustration of supply and demand of HPC Service Figure 1: Illustration of Federated Learning Platform communities. The heterogeneity is built in to such data which will be used to train ML model via Federated Learning platform. Regarding the current identified demonstrative application, the data from the ABCI of AIST and Taiwania III (T3) of NCHC will be loaded for the FL training distributed across two major supercomputing centers that are more than a thousand of miles apart. The configuration of the queues of HPC job can be adjusted to reduce the gap between the “supply” and “demand” in the Figure 2 to improve the utilization of the computing power. While cleaning the data stated above, existing well organized time series data is used to test, tune the technologies involved in this work. The Western European Power Consumption dataset from the Kaggle[4] is adopted for such purpose. The data is first decomposed into three components; 1). Trend , 2), Seasonal and 3) Remainder, for detailed investigation. The technology is applied to the Austria Power Consumption data from the Western European Power Consumption dataset to demonstrated the efforts of data decomposition. The same data is also used to test the capability of XGBoost that is used to analyze the time series data. In addition to conventional data cleaning processes, the data augmentation technologies [5][6][7] are studied in case more data is needed to facilitate the training. C. Optimization The purpose of the current demonstrative application on the FL is to enhance the utilization of HPC system. Therefore, an optimization procedure, the Particle Swarm Optimization (PSO), is used to work with the data from the Slurm simulator[10][11][12] to suggest better configurations of queues if exist. The procedure of PSO adopted with matching components from the Slurm simulator will also be presented during the presentation. The parameters for the PSO are currently under investigation. II. DEMONSTRATION With joint efforts invested into the technologies above, a demonstration of a FL job was performed during SC23 located in Denver. Colorado in Nov. 2023. The FL platform was formed by a server node joint with two client sites. The FL server was provided by NCHC while the two clients were provided by AIST and NCHC. The platform architecture is illustrated in Figure 3. The CIFA10 dataset [13] was distributed across the two clients for the FL training run. The result from the FL training is compared with that of centralized training to demonstrated the feasibility of the FL. III. FUTURE WORK This work is aimed to build a persistent Federated Learning Platform to facilitate data sharing type application across different parties without sharing the raw data. The technologies that can be used to facilitate the demonstrative application are studied and tested that will be used as the foundation of the application and adopted onto the FL platform. However, what we have now is a preliminary result with test data used. To tackle the actual application, various modifications are needed to ensure the success of the application on the FL platform. As the platform is forming with initial success, the authors would like to call for international participation of this effort for both the participation in the platform architecture itself and the demonstrative application, as well as other applications. ACKNOWLEDGMENT Current research is supported by National Science and Technology Council of Taiwan, with project “Secure and Distributed Data Cloud for AI Platform between Taiwan and Japan”, NSTC-107-2923-E-492-002-MY4, and National Applied Research Laboratories Innovation Project “Federated Learning Framework for International Collaboration – using scheduling optimization of computing facility as example”. REFERENCES [1] NVIDIA Flare, https://developer.nvidia.com/flare [2] NVIDIA Flare, https://nvflare.readthedocs.io/en/main/index.html [3] NVIDIA Flare, https://github.com/NVIDIA/NVFlare [4] “Western Europe Power Consumption” dataset from Kaggle, https://www.kaggle.com/datasets/francoisraucent/western-europepower-consumption/code [5] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, XueWang, Huan Xu, “Time Series Data Augmentation for Deep Learning: A Survey,” Proceedings of the Thirtieth International Conference on Artificial Intelligence (IJCAI-21), 2021. [6] Kasun Bandara, Hansika Hewamalage, Yuan-Hao Liu, Yanfei Kang, and Christoph Bergmeir, “Improving the accuracy of global forecasting models using time series data augmentation,” arXiv preprint arXiv:2008.02663, 2020. [7] Hailin Hu, MingJian Tang, and Chengcheng Bai. Datsing, “Data augmented time series forecasting with adversarial domain adaptation,” Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2061–2064, 2020. [8] XGBoost, https://xgboost.readthedocs.io/en/stable/python/index.html [9] Marie Roehm, “Top 7 Open-Source Frameworks for Federated Learning,” https://www.apheris.com/resources/blog/top-7-open-sourceframeworks-for-federated-learning [10] BSC Slurm Simulator, https://github.com/BSC-RM/slurm_simulator [11] Marco D’Amico, Ana Jokanovic, Julita Corbalan, “SLURM Simulator improvements and evaluation,” Barcelona Supercomputing Center, https://slurm.schedmd.com/SLUG18/slurm_sim.pdf, SLUG’18, 2018. [12] Slurm Simulator software package, https://github.com/ubccr-slurmsimulator/slurm_simulator [13] CIFA10 dataset, https://www.kaggle.com/datasets/hungvtm/cifa10-trainval-test?select=cifar10_train_val_test Figure 3: FL platform architecture utilized for the demo.