Building a hybrid network topology in an HPC Infiniband-based cluster

Building a hybrid network topology in an HPC Infiniband-based cluster Gabriel Gomez-Lopez, Jesus Escudero-Sahuquillo, Pedro Javier Garcia and Francisco Jose Quiles May 2023 1 Extended Abstract In recent years, the fastest-growing applications and services, among the ones requiring high-performance computing (HPC), have been those related to Deep Learning and Artificial Intelligence [1]. The systems supporting this applications and services handle large amounts of data that must be processed in a short period of time. As a result, these systems require ever-increasing computing and storage capacity. The computational capacity required to solve the complex problems presented by these applications is enormous, hence HPC system designers and researchers aim at increasing computational capacity year after year. Currently, the world’s most powerful supercomputer, according to the Top500 list, reaches 1.194 PetaFLOPs, and it is the first supercomputer to reach the ExaFlop. In order to reach high computational capacity, today’s HPC systems consist of thousands of processing nodes working in parallel that communicate with each other using a fast interconnection network. Indee, one of the most critical requirements for maximum performance at HPC clusters is to provide high bandwidth and low latency to the communication between these nodes to facilitate cooperation. This means that, in an HPC system, the interconnection network must be fast enough to accomplish with these bandwidth and latency requirements, otherwise becoming the bottleneck of the entire HPC system. Therefore, the interconnection network must be designed cautiously to fulfill the communication requirements of HPC applications. Among the most challenging design aspects of interconnection networks, we can underscore the network topology and routing algorithms. Note that there are other design challenges, such as switch micro-architecture, flow control, congestion management or power saving, which are out of the scope of this paper. Over the years, topologies have evolved from simpler networks (bus, ring) to more complex but more efficient ones (3D Tori, Fat-Trees, DragonFlies, Slimflies, etc.). Network topologies are usually classified into the following types: direct, indirect, hierarchical and hybrid [2]. A direct network has orthogonal structures, in which the nodes are organized in several dimensions. Each node has at least a direct connection to a neighbor in each dimension. These networks 1 are easy to implement if only 2 or 3 dimensions are used, but more dimensions increase the complexity and network diameter, thereby making them not scalable in terms of performance. By contrast, indirect topologies were proposed as an alternative to improve the performance of direct ones. Nodes in indirect networks are interconnected using switches organized in different stages. As the number of nodes increases, indirect networks offer smaller network diameter than direct networks, but the complexity of switches (i.e., the port counts) and the number of required components also increase, so the network cost can be prohibitive. Hence, hybrid topologies were proposed to overcome the direct and indirect networks drawbacks. The hybrid networks are easy to implement, offer scalability when the number of nodes to interconnect is large at a reduced cost, and also maintain a reduced network diameter. One of these topologies is the KNS [3, 4], which combines an n-dimensional direct network with small indirect subnetworks connecting the nodes in each dimension. By combining the advantages of direct and indirect networks, KNS topologies are able to interconnect a large number of nodes while maintaining low latency, higher bandwidth, and higher fault tolerance, while keeping a lower cost compared to that of indirect networks. However, as far as we know, the KNS topology had not been implemented yet in a real cluster infrastructure. In this work, we describe the process to build a KNS topology in a real HPC cluster. Note that the KNS topology has its own specific routing algorithm: Hybrid DOR, which is an adaptation of the Dimension Order Routing (DOR) algorithm, used in direct networks. We have used the CELLIA cluster (Cluster for the Evaluation of Low-Latency Interconnection Architectures) to build the KNS topology using the Hybrid-DOR routing. The CELLIA cluster interconnection network is based on the InfiniBand technology [5], implemented using Mellanox/NVIDIA products (i.e., hardware, drivers and control software). Note that the Infiniband technology is very flexible as it allows the implementation of any topology, provided that it is possible to reproduce its connection pattern and routing algorithm in a systematic way in the InfiniBand subnet manager (SM). More precisely, the SM functions are to assign identifiers to the network devices, to discover the network topology that connect these devices, and to populate the switch routing tables according to the routing algorithm. In InfiniBand networks, the SM behavior is implemented in a dedicated software entity, called OpenSM. The OpenSM software is open-source and is included in the OpenFabrics Software (OFS) [6]. We have implemented the KNS topology discovering and the hybrid-DOR routing in the OpenSM software. In order to evaluate the KNS implementation, we have run different HPC applications and benchmarks (e.g., HPCC, Graph500, Netgauge and GPCNeT) in the CELLIA cluster. We have compared the KNS topology performance with that obtained using a Fat-tree network configuration [7] using deterministic routing. From the obtained results, we can conclude that the KNS topology behaves as expected and also outperforms the Fat-Tree performance under certain traffic patterns. 2 Acknowledgments This work has been jointly funded by the BBVA foundation and Becas Leonardo (call 2020) with the grant IN[20] TIC TIC 0042, by the Ministry of Science, Innovation and Universities and the European Commission (FEDER funds) under the project RTI2018-098156-B-C52 (MCIU/FEDER), by the Junta de Comunidades de Castilla-La Mancha under the project SBPLY/17/180501/000498, and by the Universidad de Castilla-La Mancha under the project 2023-GRIN34056. References [1] X. Chen and X. Lin, Big Data Deep Learning: Challenges and Perspectives, vol. 2. 2014. [2] J. Duato, S. Yalamanchili, and N. Lionel, Interconnection Networks: An Engineering Approach. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2002. [3] R. Pe˜naranda, C. G´omez, M. E. G´omez, P. L´opez, and J. Duato, “A new family of hybrid topologies for large-scale interconnection networks,” in 2012 IEEE 11th International Symposium on Network Computing and Applications, pp. 220–227, 2012. [4] R. Pe˜naranda, C. G´omez, M. E. G´omez, P. L´opez, and J. Duato, The k-ary ndirect s-indirect family of topologies for large-scale interconnection networks, vol. 72. Mar 2016. [5] InfiniBand Trade Association, “InfiniBand Architecture Specification Volume 1, Release 1.3,” 2015. [6] OpenFabrics Alliance, “The OpenFabrics Alliance.” https://www. openfabrics.org. [Online; last access May 29, 2023]. [7] G. Rodriguez, C. Minkenberg, R. Beivide, R. Luijten, J. Labarta, and M. Valero, Oblivious routing schemes in extended generalized Fat Tree networks. 01 2009. 3