Experimental and numerical analysis for potential heat reuse in liquid cooled data centres

Applied Thermal Engineering 234 (2023) 121260 Available online 2 August 2023 1359-4311/© 2023 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/bync-nd/4.0/). Research Paper Experimental and numerical analysis of the thermal behaviour of a single-phase immersion-cooled data centre Paolo Taddeo a , Joaquim Romaní a , * , Jon Summers b , Jonas Gustafsson b , Ingrid Martorell c , Jaume Salom a a Catalonia Institute for Energy Research (IREC), Thermal Energy and Building Performance Group, Jardins de les Dones de Negre, 1, 08930, Sant Adri` a De Bes` os, Barcelona, Spain b RISE ICE, RISE Research Institutes of Sweden, Bj¨ orkskataleden 112, 973 47 Luleå, Sweden c Sustainable Energy, Machinery and Buildings (SEMB) Research Group, INSPIRES Research Centre, Universitat de Lleida, Pere de Cabrera s/n, 25001 Lleida, Spain ARTICLE INFO Keywords: Data centre Single-phase cooling Energy model Simulation Immersion cooling ABSTRACT Server power densities are foreseen to increase, and conventional air-cooling systems will struggle to cope with thermal demand. Single-phase immersion systems are a promising alternative to operate very intensive workload such as high-performance computing, cryptocurrencies mining or research activities. However, few companies deal with this kind of system and there is a lack of energy models that can reproduce an accurate analysis of the system behaviour. This study addresses the experimentation, data collection, and model validation of a singlephase immersion cooling system where 54 open compute project servers, each with a peak power of 400 Watts that are submerged and operated in a dielectric coolant. Results show the evolution of the thermal profile of the system under static and dynamic workloads, and it provides a correlation of server energy use under various system temperatures. The energy model is presented, validated against real data, and exploited to investigate the system response to different cooling conditions. In conclusion, the study demonstrates the validation of the energy model and supports the basis for further investigation. 1. Introduction 1.1. The rise of IT racks power density Large volumes of data are stored every day and the trend is expected to continue to grow exponentially, mainly due to the ongoing global digitalization of developed and developing countries. According to a report by Reinsel et al. [1], the amount of data stored and processed will exceed the 33 Zettabytes (33⋅10 21 bytes) of 2018 to 175 Zettabytes in 2025. The facilities where data are stored and processed are called data centres (DC) and they are spreading all over the world, as does the data. Information Technology (IT) equipment are usually placed in racks that could reach power densities higher than 50 kW per rack, depending on the workloads assigned and the equipment capabilities. According to Bizo et al. [2], average density remains under 10 kW per rack but it tends to be higher in data centres with capacities above 5 MW. A likely explanation is that hyper-scale facilities deal with cloud and internet companies that operate cabinets fully populated with high-end, energy demanding servers. The power density of IT racks highly affects the data centre infrastructure. Its layout and cooling architecture are specifically designed to match the operational requirements of different configurations. Almost the entire power feed to the IT equipment is converted into excess heat [3]. This must be removed for proper operation of the equipment, as it has been demonstrate the electronic equipment rate of failure increases exponentially beyond 75 ◦C [4]. The energy requirement of the cooling system can represent up to 40% of the overall facility energy consumption [5] and it is commonly provided by air-cooling systems. The low cooling efficiency of air systems, due to undesired air recirculation, has been improved by different strategies such as hot/ cold aisles containment, variable speed fans, an increase of the air inlet temperatures and free cooling [6]. However, Moore’s law [7] theorizes a continuous increase in IT equipment energy density that will make it hard to cover the entire critical IT thermal management by means of only air cooling systems. To overcome this bottleneck, an enhanced heat transfer between the coolant medium and the IT equipment surface is required [8]. * Corresponding author. E-mail address: [email protected] (J. Romaní). Contents lists available at ScienceDirect Applied Thermal Engineering journal homepage: www.elsevier.com/locate/apthermeng https://doi.org/10.1016/j.applthermaleng.2023.121260 Received 10 May 2023; Received in revised form 30 June 2023; Accepted 29 July 2023 Applied Thermal Engineering 234 (2023) 121260 2 1.2. Modelling of liquid cooling techniques The interest on liquid cooling technologies for data centres increased due to larger heat transfer capacity. On one side, this achieves lower and more uniform temperatures on the electronic components. On the other side, the cooling system can operate at higher temperatures, increasing energy efficiency, among other advantages such as lower noise operation. Liquid cooling technologies can be classified according to the interaction between the heat transfer media and the electronic or by the thermophysical state of the heat transfer fluid. On indirect liquid cooling the heat transfer between the electronics, mainly the CPU, and the heat transfer fluid is done through a cold plate. This acts as heat sink for the electronic components and is coupled to the cooling circuit, hence the heat transfer fluid is never in direct contact with the electronics. Indirect cooling is commonly referred to as on-chip cooling. On direct liquid cooling, the heat transfer fluid is in contact with the electronics. The most common systems consist of immersion systems, in which the servers are submerged in a basin a heat transfer fluid. Alternatively, the heat transfer fluid can be sprayed onto the electronics. Direct cooling technologies require dielectric fluids that avoid short circuits. Finally, single-phase technologies use heat transfer fluids only in liquid form while two-phase technologies exploit the latent heat of low-temperature boiling fluids. The raise of liquid cooling technologies is recent. Fulpagare and Bhargav review on thermal management technologies in data centres from 2015 barely contains some references on liquid cooling [9]. In 2016 Kheirabadi and Groulz design review on cooling of server electronics technologies [8] included immersion cooling, although centred on two-phase immersion cooling focusing on experimental studies on heat transfer properties. Similarly, the review on innovative cooling and thermal management technologies from Nadjahi et al. in 2018 [10] included single-phase and two-phase immersion cooling, although these still presented a minor part of the review. On 2019 Kunkoro et al. [11] presented a first review specific for immersion cooling technologies, including the different methods (single or two phase), type of fluid, and immersion mode (indirect or direct). Yuan et al. [12] reviewed in 2021 the phase change application to data centres, including heat pipes, integrated vapor compression, thermosyphon system, cold storage and two phase immersion cooling. A more generic review on immersion cooling technologies, including applications for computers and data centres, was published by Pambudi et al. [13] in 2022. The literature showcases mostly experimental research, although modelling studies are also present. Regarding on-chip cooling, Carb´ o et al. [14] validated a TRNSYS model with the experimental results with a water-cooled data centre test bed to characterize the thermal behaviour and the liquid cooling capacity of an on-chip cooling solution. The model was used to explore the waste heat recovery capabilities at different operation conditions with real DC workloads. Similarly, Zhang et al. [15] modelled a liquid on-chip cooling, using MATLAB to model the heat transfer off the on-chip and TRNSYS to model the behaviour of the whole system. The study focused on optimizing the primary and secondary flow to minimize the energy consumption while ensuring chip operation at safe temperature. The same approach was later used to determine chip temperature sensitivity to the primary and secondary circuit [16] and then to determine the adequate working pairs of inlet temperature and flow rates of both circuits [17]. Modelling studies on two-phase on-chip cooling have also been developed. Lamaison et al. [18] presented experimental results of various different on-chip two-phase cooling cycles as well as the numerical modelling and validation of a new simulation tool. Zhang et al. [19] used a validated computational fluid dynamics (CFD) model to analyse the temperatures of different configuration of heat pipe cooled servers. Nevertheless, Kanbur et al. [20] concluded that data centres that host high-density cabinets are the most suitable to be coupled with more efficient cooling solutions such as immersion cooling. Thermodynamic and thermo-economic assessment on experimentally tested single-phase Nomenclature Equation parameters A Area [m 2 ] C Thermal capacitance [kJ/K] cp Specific heat [kJ/kg/K] dt Time differential ε Effectiveness h Heat transfer coefficient [W/m 2 ⋅K] m˙Mass flow rate [kg/h] q˙Thermal power [W] U Thermal transmittance [W/m 2 ⋅K] T Temperature [K] t Time [s] V Volume [m 3 ] Acronyms AC Alternate Current CDU Cooling Distribution Unit CFD Computational Fluid Dynamics CPU Central processing unit DB Database DC Data centre and Direct Current DIMM Dual in-line memory module GPU Graphic processing unit HPC High performing computing IPMI Intelligent Platform Management Interface IT Information Technology KPI Key Performance Indicators MAE Mean Average Error OCP Open Compute Project ODE Ordinary differential equations OU Open Unit PLC Programmable Logical Controller PSU Power supply unit PUE Power Usage Effectiveness RMSE Root Mean Square Error SCADA Supervisory Control and Data Acquisition TPU Tensor Processing Unit TRNSYS Transient System Simulation Tool Subscripts avg Average env Environment equip Equipment hx Heat exchanger in Individual In Inlet IT Information Technology oil Dielectric Oil Out Outlet rack Rack Tk Tank tot Total water Water P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 3 and two-phase immersion cooling systems showcased that immersion cooling systems become more feasible for the server power rates above 5 kW. However, single-phase immersion cooling modelling has mainly consisted of computational fluid dynamics (CFD) to study the design of immersion bath, heat sinks, dielectric fluids, and operation conditions. Gandhi et al. [21] used CFD to study the impact of inlet temperature and flow rate into CPU temperatures. In the same line, Cheng et al. [22] (2020) analysed the circulating speed and different materials for the heat sinks into temperature level and the uniformity of the electronics. A similar study was performed by Niazmand et al. [23] that studied the impact on pumping power and CPU temperature of different dielectric oil enhanced with nanoparticles which have been calculated theoretically. Other studies focused on tank configurations and operation. Huang et al. [24] studied the heat transfer efficiency of buoyancy and pump driven single-phase immersion cooling. Muneeshwaran et al. [25] focused on the inlet/outlet configurations and bypass effect of the immersion cooling bath and Shrigondekar et al. [26] performed a similar analysis focusing on the bypass effect. Finally, two-phase immersion cooling also have been modelled with CFD, an example being Sun et al. [27] that used Euler multiphase flow method to simulate subcooled flow boiling, analysing the impact of flow velocity and different fluids onto the temperature level and uniformity. Research on immersion cooling modelling includes few research on control and decision making tools. Lionello et al. [28] derived a general lumped-parameter grey box model of a proof-of-concept natural convection immersion cooling unit achieving great accuracy on cooling water outlet temperature. Luo et al. [29], developed a decision support system for waste heat recovery using a case study 530 kW date centre with 400 servers immersed in an engineered dielectric fluid. The results showed a 68% recovery of waste heat from the IT equipment and a 10% improvement in the data centre’s power usage effectiveness (PUE). Despite these examples, to the authors knowledge there is a lack of research on single-phase immersion cooling models for design, operation optimization or control purposes, as also pointed by other authors [28]. 1.3. Necessity, aim and structure of the study The previous section highlighted the lack of single-phase immersion cooling models useful to study system integration, operation or control. The paper objective is to validate a liquid cooling model with experimental data and to use it for resilience stress testing with different boundary conditions. Section 2 presents the experimental setup where both static and dynamic tests have been performed changing the IT workloads to monitor the dielectric coolant tank temperature evolution, the servers power usage, and the heat reuse potentiality. Section 3 describes the energy model built in a transient simulations software (TRNSYS) as well as the main variables and parameters involved. Section 4 reports the results of the experimental test and the validation of the energy model against real data. In Section 5, a resilience test is performed to assess the response of the system under different tank setpoint temperatures and water inlet temperatures. Finally, Section 6 concludes the study and highlights further work expected to be developed. 2. The experimental setup The system under investigation is a SUBMER PODXL [30]. This immersion tank is designed to comply with the Open Compute Project (OCP) specifications [31] with the so-called OU (open unit) of 48 mm in addition to the standard server U measurements of 44.5 mm per U. The immersion tank contains 18 OCP cubbies, each housing three 2OU (Open Unit) servers connected to a shared 12 V direct current connector. This totals 18 cubbies of 3 servers each giving a total of fifty-four servers. In the experimental set-up the servers used each contain dual socket Intel Xeon E5-2678 version 3, 2.5 GHz microprocessors. The system has two 3OU power shelves rated for a peak of 27 kW and convert 3 phase 400 V alternating current (AC) to feed the two 12VDC busbars at the bottom of the tank. Each busbar has nine cubbies connected, one on the left and one on the right of the immersion tank. The immersed power shelves each take 10 3 kW Power Supply Units (PSUs) in an N +1 configuration that provides power to the servers while the cooling water inlet and outlet are connected to a cooling distribution unit (CDU) to cool down the tank’s liquid coolant via an internal heat exchanger. The system has a nominal cooling capacity of 50 kW, and the manufacturer recommendation is to operate it coupled with dry cooling towers. The tests are performed in a controlled environment. The immersion system is connected to the experimental liquid immersion cooling testbed that is a facility dedicated to testing innovative technologies for data centres and it is physically located in the Infrastructure and Cloud research & Test Environment (ICE) at the Research Institutes of Sweden (RISE) [32] (see Fig. 1). The experimental setup comprises a monitoring system and a control module that monitors the main physical quantities involved in the system process and allows operating the system under different conditions. Monitoring capabilities include a set of temperature and flow rate sensors and a dedicated Supervisory Control and Data Acquisition (SCADA) system. Moreover, using the servers’ Intelligent Platform Management Interface (IPMI) it is possible to monitor the temperatures of servers’ central processing units (CPU) and the electrical power consumption. All the measured quantities are first collected by a data acquisition system, details of which can be found in Gustafsson et al [33] with sampling time of five seconds and then sent to Zabbix, an opensource software that offers monitoring and storage services. Then, data are stored in KairosDB that acts as long-term storage database. Main variables monitored in the system are reported in Table 1. The control of the system is managed by Python scripts that allow changing remotely the computational load of each CPU of each server. In the specific, the computational load is changed by utilizing the stress-ng tool, see [34], a Linux-based stress test installed in each server. Fig. 2 shows a schematic of the system. The left side of the basin hosts the cooling distribution unit (CDU) where the oilwater heat exchanger and the oil pump are located. The other space in the tank is left for servers and power shelves. The oil flows in a Tconfiguration [25,26], it is released from the bottom of the basin and rises while cooling down the servers, thus becoming hotter. Finally, the hot oil is collected at the top sides of the tank and circulated to the CDU and cooled again. For the experimental tests, the water-cooling loop of the test bench has been carefully flushed to avoid the presence of air in the pipes. The volumetric flow rate was controlled at a constant value of 2.7 m 3 /h and the inlet water temperature was manually controlled by opening and closing the chilled water valve of the liquid cooling test bench. 3. Energy system modelling The software TRNSYS [35] is a flexible graphically based environment and it is used to simulate the transient behaviour of the system. The IT equipment are modelled as a unique lumped mass (Type 963) that can be characterized by the initial temperature, capacitance, and the heat transfer coefficient to their surrounding environment, which is the inside of the dielectric oil tank. This unique lumped capacitance represents all the different servers submerged in the oil bath and the model does not distinguish between the individual component such as boards, chips. The main parameters to characterize the IT equipment through the lumped mass model are the thermal capacitance of the IT equipment (CITequip), the heat transfer area of the IT equipment (AITequip) and the heat transfer coefficient from the IT equipment to the dielectric oil in the tank (hITequip). Following the Landauer theory [3], the entire electrical energy provided to the IT equipment is converted into heat gains (˙ qtotITequip ). IT equipment are immersed into a dielectric oil tank, P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 4 modelled as a Flat Bottom Storage Tank (Type 531), transferring the heat gains to the dielectric oil is achieved with miscellaneous heat input to the “type”, thus heating up the oil. The tank is characterized by its volume (VTank), its capacitance (CTank), the number of nodes in which thermal stratification is modelled, the inlet/outlet positions of the ports, and the fluid properties (main properties of the fluid are summarized in Table 2). The supplier only provides the values on the nominal operation temperature. Studies on other dielectric oil highlighted the temperature dependency of the properties although the change in the usual operation range is small [23,36]. Therefore, the present study considered constant fluid properties. The dielectric oil circulates from the storage tank to the water-oil heat exchanger, which is modelled as a counter flow heat exchanger (Type 5), with a constant speed pump (Type 114) that operates in an ON-OFF mode. According to the collected experimental data, the volumetric flow rate of the oil pump is constant at 7.5 m 3 /h when the pump is active. To account for oscillation effects generated by the oil pump operation inside the oil tank, a variable heat transfer coefficient has been programmed into the model. To compare the experimental outputs with the model outputs, experimental inputs were recorded and then used in the dynamic simulations. To insert the input data in the model, a Data Reader component (Type 9) is used. Fig. 3 shows the implementation of the model in TRNSYS while Table 3 reports the TRNSYS types used in the simulation. The implementation is faithful to the experimental setup, but it presents an important simplification regarding the tank geometry. Standard TRNSYS libraries do not include any component that could model a rectangular storage tank where both horizontal and vertical stratifications are numerically solved. The equations that describe the main heat and mass transfer phenomena of the system modelled are reported. Considering first the IT equipment modelling, Eq.1 represents the sum of heat fluxes entering the lumped capacitance where ˙ q tot_ITequip is the total heat power generated in the IT equipment, ˙ q in_ITequip are the individual contributions generated by the servers’ electrical power consumption that is entirely converted into heat. ˙qtotITequip =∑˙qinITequip (1) The temperature variation of the IT equipment can be characterized with the convective heat transfer (Eq. (2)) that can be reformulated as linear constant coefficient ODE (Eq. (3)). dTITequip dt = ˙qtotITequip CITequip (2) dTITequip dt =aT +b(3) Setting the constant of Eq. (3) as: Fig. 1. Single-phase immersion system tested: inside view (a), unit view (b). Table 1 List of parameters measured in the data centre. Variable Units Servers power consumption W CPU temperature ◦C DIMM temperature ◦C Water inlet temperature ◦C Water outlet temperature ◦C Oil inlet temperature ◦C Oil outlet temperature ◦C Water flow rate m 3 /h Oil flow rate m 3 /h Pump ON_OFF signal [0–1] Fig. 2. Single-phase immersion cooling simplified front view scheme. Table 2 Physical and chemical properties of the dielectric fluid, data from the manufacturer [37]. Physical and chemical properties Units Value Density at 20 ◦C kg/m 3 796 Kinematic Viscosity at 40 ◦C cSt 5.1 Thermal conductivity at 40 ◦C W/mK 0.14 Specific heat at 40 ◦C kJ/kgK 2.26 P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 5 a= − hITequip ⋅AITequip CITequip (4) b= ˙qtotITequip +hITequip ⋅AITequip⋅Tenv CITequip (5) The general solution for the constant “a” different from zero is: TITequip =C⋅eat −b a(6) Setting TITequip =TITequipinitial at t =0 the following solution is obtained: TITequipfinal =TITequipinitial ⋅eat +b a⋅(eat −1)(7) Then, it is possible to estimate the average temperature of the lumped IT equipment modelled with Eq.8 and the losses to the environment with Eq. (9). TITequipavg =TITequipfinal +TITequipinitial 2(8) ˙qITequiqskin=hITequip⋅AITequip⋅(TITequipavg −Tenv)(9) Finally, the energy stored inside the IT equipment that results it the temperature increase of the systems is given by Eq. (10): ˙qITequipstored = C⋅(TITequipfinal +TITequipinitial ) dt (10) With the same mathematical formulation, Eq. (11) describes the temperature variation of the tank where TTank is the average tank temperature, ˙ qInTank and ˙ qOutTank are the heat input and output of the tank, CTank is the capacitance of the tank. This ODE as the same solution as the previous equation. dTTank dt =(˙qInTank −˙qOutTank ) CTank (11) Finally, Eq. (12) calculates the heat transferred by the heat exchanger ˙ q HX , based on its effectiveness, ε , where ˙ m oil and ˙ m water are oil and water mass flow rates, cp oil and cp water are oil and water specific heats, and T in_oil and T in_water are the inlet temperatures of the two streams. ˙qHX =MIN(˙moil⋅cpoil,˙mwater⋅cpwater)⋅(Tinoil −Tinwater )⋅ ε (12) 4. Experimental and validation results Two tests are performed to investigate the behaviour of the system and to collect data to ensure robustness of the energy model. The first is the validation of the thermo-physical characteristics in static conditions and the second is to validate the dynamic regime under a synthetic IT variable workload. Fig. 4 presents the static experimental test (left side) and the dynamic one (right side). The static test starts with the servers in idle mode with an approximate power consumption of 3.4 kW and a basin temperature of 25 ◦C. In the first stage of the test, the basin slowly heats up due to the small amount of power released by the servers. Process temperatures increase and the oil pump acts in a basic ON-OFF fashion (automatic PLC control programmed for oil recirculation), since the basin temperature is well under the set point of 45 ◦C. Applying a fast step change in the IT load up to the 50% of the IT power installed, the temperatures suddenly increase. Notice how the temperature oscillates with the oscillations generated by the ON-OFF pumping process. Oil temperatures, and consequently water temperatures, heat up until the set point of 45 ◦C plus 2 ◦C of upper dead band limit. Then, the internal control of the PLC of the oil pump sets the pump in constant regime to maintain the set point. Due to the thermal inertia of the system, temperatures tend to further increase before the cooling starts to be effective. This is important to consider in the control of the system since the highest temperatures in the servers must be sufficiently low enough to promote proper operation. On the other hand, to investigate the dynamic behaviour of data centre operations, a synthetic IT workload has Fig. 3. Scheme diagram of the TRNSYS model for the immersion-cooled data centre. Table 3 TRNSYS types used. Name TRNSYS Type Main Parameters Storage tank filled with dielectric oil 539 Tank Oil Volume =860 [Liters] Number of Tank Nodes =1 [-] Fluid Specific Heat =2.26 [kJ/kg.K] Lumped Mass 963 Capacitance =50 [kJ/K] Servers number =54 [Units] Surface area =0.55 * Servers number [m 2 ] Initial temperature =65 ◦C Oil pump 114 Rated volumetric flow rate =7.5 [m 3 /h] Heat Exchanger: Counter Flow 5 Overall heat transfer coefficient = 24000 [kJ/h.K] Specific heat of source side fluid = 4.19 [kJ/kg.K] Specific heat of load side fluid =2.26 [kJ/kg.K] P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 6 been applied ranging the power input between 20% and 50% of the rated IT load installed. The two variables considered for the model validation are the average temperature of the IT equipment and the cooling water exiting the basin. The average rack temperature has been considered as a weighted average of CPUs and DIMMs temperature. To reproduce the same boundary conditions applied to the experimental test, data such as the IT power load and the water inlet temperature and flow rates are used as input of the energy model. The model presents an uncertainty on the heat transfer coefficient between the oil and the rack. This latter is hard to evaluate experimentally since it evolves dynamically depending on the real time fluid dynamics of the system. Through the model calibration process, the static and dynamic heat transfer coefficients have been estimated of 140 and 160 kJ/hr⋅m 2 ⋅K, respectively. Fig. 4 shows the comparison between the real and simulated outputs for the two tests conducted. The thermal inertia of the system is perfectly captured in the static validation proving that the modelling strategy is effective. The dynamic comparison shows that the model presents smoother outlet temperatures compared to real data but in general it follows the real trend. Before the model can be used with confidence, a proper assessment of the magnitude of the errors that may result from using it should be performed. Validation of models involves, in its simplest formulation, the process of comparing simulated and observed values. Validation criteria have been separated into two groups, the summary measures, and the difference measures. The indicators proposed are the index of agreement, formulated by Willmott et al. [38], the R squared value, the mean average error (MAE) and the root mean squared error (RMSE). Their respective values are reported in Table 4 for both the static and dynamic tests. The statistical analyses have been performed on both data sets chosen for the validation, the average IT equipment temperature, and the cooling water outlet. Results show a good level of accuracy, and the model is considered validated. From the data analysed, the dependence of the servers’ power consumption with the oil temperature and thus the servers’ temperature is clear to see. Fig. 5 shows the experimental data collected during the heating up process of the static test and their fitted curves. For the same workload applied to the servers, a 20 ◦C increase of the average server’s temperature led to a 5% increase in the power consumption. Polynomial correlations have been obtained and they are reported in the graphs. The correlations are normalized on the rated IT power of the considered servers (Intel®Xeon®CPU E5-2678 v3, 2.50Ghz), and they are validated for the typical range of operating temperature of such systems. The right plot in Fig. 5 shows power consumption in comparison to the oil temperature. This fitting is less precise due to the fluctuation generated by the oil pump activations which generate deviations. As demonstrated by the indicators used for the validation, the model predicts the system status in an accurate way, and it can be used to extrapolate results for different operating conditions. 5. Resilience stress testing for critical operating temperatures 5.1. Motivation In Mediterranean and central European zones where outside temperatures can exceed 40 ◦C it is relevant to investigate how the system will react to a cooling system fault or to an extremely hot weather event where dry cooler towers are typically employed and cannot ensure thermal setpoints. Analysis from such a modelled scenario is useful to provide an indication of the response time that an operator has available before any servers reach critical temperature levels. 5.2. Simulation testing methodology and model adaptation The proposed methodology to evaluate the response to an extreme weather event is a step test methodology where the water inlet temperature suddenly changes its value. To investigate the operation of a realistic data centre, some modifications have been applied to the validated model. Simulations consider that the workload profile engages a Fig. 4. Static IT (a) and dynamic IT workload (b) experimental tests. Table 4 Indicators for model validation. Test Validation Summary measures Difference measures Static IT equipment average temperature Index of Agreement =0.996 MAE =0.566 ◦C R squared =0.986 RMSE = 1.883 ◦C Dynamic Index of Agreement =0.938 MAE =1.819 ◦C R squared =0.857 RMSE = 2.195 ◦C Static Water outlet temperature Index of Agreement =0.998 MAE =0.676 ◦C R squared =0.993 RMSE = 1.069 ◦C Dynamic Index of Agreement =0.963 MAE =1.661 ◦C R squared =0.867 RMSE = 2.720 ◦C P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 7 percentage of the rated IT power of 50 kW, with 54 immersed servers, and that their CPU temperatures affect the rated power consumption by the correlation reported in Fig. 5. A constant HPC profile of 75% of the rated IT power has been applied to the model. HPC workload is typically CPU intensive since they perform many scientific calculations. They do not have real-time requirements, and they are usually allocated in job queues. The PLC control of the oil pump has been programmed with an upper dead band of 2 ◦C, in line with the real system. The water flow rate is kept constant at the maximum volumetric flow rate recommended for this basin (11 m 3 /h) and the initial inlet water temperature is at 30 ◦C. The step test comprehends a first stage of simulation initialization and variables stabilization where it is possible to see the data centre operation under high load operation and optimal cooling conditions. Then, the temperature step test starts, and the water inlet temperature increases from the recommended 30 ◦C up to 37.5 ◦C, 40 ◦C and 42.5 ◦C and the system responds to this sudden change. Simulations are performed using a parametric approach on the tank temperature setpoint for four different values in the range of recommended operation. Table 5 resumes the stress tests performed and the time delay until servers’ CPUs reach the critical shutdown temperature of 90 ◦C. Fig. 6 shows the step tests temperature profiles of servers, oil, and water. Temperature fluctuations are directly related to the oil pump activation, whose signal is not reported in the graphs to facilitate easier reading of the charts. Before the step test, the programmed control can maintain the desired tank temperature turning on and off the oil pump with lesser or higher frequencies depending on the setpoint. Notice how the pump needs to turn on more often for a lower tank setpoint at the same water conditions and IT workload. Once the step test is applied, the system struggles to maintain the current conditions and a logarithmic growth of all system temperatures ensues. Due to the higher temperature difference between the stabilized and the initial temperature of the dielectric coolant in the different cases, the temperature ramps up faster when operating at lower setpoints. These results show that, running the system at lower setpoints will delay system critical overheating in case of a cooling system fault or extreme outdoor conditions. However, they also show that, when operating with the water temperature recommended by the manufacturer, the system can guarantee the cooling process also running extremely high IT loads and high tank setpoints. Indeed, before the step test server’s temperature remains below the critical limit of 90 ◦C. Notice that in the 40 ◦C tank set point simulation, the pump signal does not cycle but it is constantly ON and the setpoint it is not achieved. This means that in the real-life operation this configuration will lead to conflict in the PLC control of the oil pump and the system will react sending warning messages to the user since it realizes that the setpoint conditions are not achieved. Operating at higher tank setpoint temperatures enables cooling water to be supplied at higher temperatures within the range of safe operation. This will be valuable when the facility presents a heat recovery system that can exploit the low-grade waste heat generated by the servers and extracted by the cooling water loop. However, the experimental tests highlighted that the power consumption of the servers increases for the same workload depending on the liquid coolant temperature. It is also to be considered that the oil pump activates less for higher setpoints thus reducing its power consumption and increasing its lifespan. 6. Conclusions and further work Single-phase immersion cooling is a technology adopted to cool down servers in a more effective way. It also allows for an increased power density of data centres and potential improvements of their energy efficiency. Server power-densities are increasing, and server invasive liquid cooling technologies are gaining traction for compute intensive applications. However, the mainstream data centre industry has to overcome barriers before such immersion systems are adopted in a relevant way. This study presents the results of an experimental test where a single-phase immersion cooling system is operated to investigate the response of the system to static and dynamic workloads. The experimental data is then used to validate a white-box energy model built in a transient simulation software (TRNSYS) and the model is used Fig. 5. Fitted curves of power consumption correlation with servers’ temperature (a) and oil temperature (b). Grey dots represent real measurements and the purple line the fitting curve. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.) Table 5 Step tests simulation resume. Tank SP Final inlet water temperature Time delay until critical servers’ temperature 37.5 ◦C – 40 ◦C 40 ◦C – 42.5 ◦C <30 Minutes 37.5 ◦C – 45 ◦C 40 ◦C – 42.5 ◦C <30 Minutes 37.5 ◦C – 50 ◦C 40 ◦C – 42.5 ◦C <15 Minutes 37.5 ◦C – 55 ◦C 40 ◦C – 42.5 ◦C <5 Minutes P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 8 to extrapolate system operations to different boundary conditions, further use to study the dynamic behaviour. The comparison with the experimental data validates the simulation assumptions. The IT equipment has been modelled as a lumped capacity with two distinctive heat transfer coefficients, one on static conditions, with the coolant pump OFF, and one dynamic, with the coolant pump ON. The dynamic behaviour of the model was adjusted against the experimental data of the step test and the validated with an IT load profile, achieving good results both in summary measures (R-squared and index of agreement) and difference measures (mean average error and root mean square error). The study provides useful correlations to estimate, for the same applied IT workload, the increment in the power consumption with respect to the average temperature of the IT equipment and oil temperature. Despite these correlations are specific for the experimental setup presented (combination of server types and dielectric oil), the curves are useful to estimate the change on power consumption. The validated model was used to perform stress test to investigate the response of the system to a sudden change in the water-cooling conditions. The test concerns a step change where the inlet water is abruptly raised emulating a cooling system fault or an extreme climate condition. Results show that, with an inlet water temperature of 42.5 ◦C the system will have components that reach critical levels of equipment and oil temperatures regardless of the initial tank setpoint. Operating at lower temperature setpoints will give the operator more time to re-schedule job queues for batch applications, attempting to reduce the power consumption. On the other hand, working at high setpoints will lead to a fast shut down of the servers and the loss of their respective jobs. Further work will include the modelling and simulation of a complete data centre system with different power and cooling systems, including waste heat recovery. The evaluation of typical data centre KPIs (Key Performance Indicators) will be performed together with the impact of the energy systems, including the power grid, fuel supply, and thermal supply. Yearly simulations to capture seasonal effects will be conducted for different typical data centre clustering locations. CRediT authorship contribution statement Paolo Taddeo: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. Joaquim Romaní: Conceptualization, Methodology, Formal analysis, Investigation, Writing – review & editing, Supervision, Funding acquisition. Jon Summers: Investigation, Writing – review & editing, Supervision, Fig. 6. Step test results for the range of tank setpoint temperatures: (a) 40 ◦C, (b) 45 ◦C, (c) 50 ◦C, (d) 55 ◦C. In the legend Tw_out is the water outlet temperature, Tequip is the IT equipment temperature, and To_out is the coolant oil outlet temperature. P. Taddeo et al. Applied Thermal Engineering 234 (2023) 121260 9 Project administration, Funding acquisition. Jonas Gustafsson: Investigation, Writing – review & editing. Ingrid Martorell: Methodology, Investigation, Supervision. Jaume Salom: Conceptualization, Methodology, Investigation, Writing – review & editing, Supervision, Project administration, Funding acquisition. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability Data will be made available on request. Acknowledgments This work has received funding from the European Union H2020 Framework Programme under Grant Agreement no. 857801 (WEDISTRICT). IREC authors would like to thank Generalitat de Catalunya for the project grant given to their research group (2021 SGR 01403). Ingrid Martorell would like to thank Generalitat de Catalunya for the project grant given to her research group (2021 SGR 01370). References [1] D. Reinsel, J. Gantz, J. Rydning, The Digitization of the World - From Edge to Core, Fram. Int. Data Corp., no. November, 2018, p. US44413318. [2] D. Bizo et al., Uptime Institute Global Data Center Survey 2021 - Growth stretches an evolving sector, 2021. [3] R. Landauer, Irreversibility and Heat Generation in the Computing Process, IBM J. Res. Dev., 5(July) (1961) 191, 183. [4] U.S. Deparment of Defense, Reliability predicion of electronic equipment, MILHDBK-2. Springfiled, 1974. [5] A. Capozzoli, G. Primiceri, Cooling systems in data centers: State of art and emerging technologies, Energy Procedia 83 (2015) 484–493, https://doi.org/ 10.1016/j.egypro.2015.12.168. [6] M.M. Nirendra Lal Shrestha, Thomas Oppelt, Thorsten Urbaneck, ` Oscar C` amara, Toni Herena, Eduard Or´ o, Francisco Diaz, Jaume Salom, Hans Trapman, Gilbert de Nijis, Joris van Dorp, Catalogue of advanced technical concepts, 2016. [7] G.M. Moore, Cramming more components onto integrated circuits With unit cost, Electronics 38 (8) (1965) 114. [8] A.C. Kheirabadi, D. Groulx, Cooling of server electronics: A design review of existing technology, Appl. Therm. Eng. 105 (March) (2016) 622–638, https://doi. org/10.1016/j.applthermaleng.2016.03.056. [9] Y. Fulpagare, A. Bhargav, Advances in data center thermal management, Renew. Sustain. Energy Rev. 43 (Mar. 2015) 981–996, https://doi.org/10.1016/J. RSER.2014.11.056. [10] C. Nadjahi, H. Louahlia, S. Lemasson, A review of thermal management and innovative cooling strategies for data center, Sustain. Comput. Informatics Syst. 19 (Sep. 2018) 14–28, https://doi.org/10.1016/J.SUSCOM.2018.05.002. [11] I.W. Kuncoro, N.A. Pambudi, M.K. Biddinika, I. Widiastuti, M. Hijriawan, K. M. Wibowo, Immersion cooling as the next technology for data center cooling: A review, J. Phys. Conf. Ser. 1402 (4) (2019) pp, https://doi.org/10.1088/17426596/1402/4/044057. [12] X. Yuan, et al., Phase change cooling in data centers: A review, Energy Build. 236 (Apr. 2021), 110764, https://doi.org/10.1016/J.ENBUILD.2021.110764. [13] N.A. Pambudi, A. Sarifudin, R.A. Firdaus, D.K. Ulfa, I.M. Gandidi, R. Romadhon, The immersion cooling technology: Current and future development in energy saving, Alexandria Eng. J. 61 (12) (Dec. 2022) 9509–9527, https://doi.org/ 10.1016/J.AEJ.2022.02.059. [14] A. Carb´ o, E. Or´ o, J. Salom, M. Canuto, M. Mac´ Ias, J. Guitart, Experimental and numerical analysis for potential heat reuse in liquid cooled data centres, Energy Convers. Manag. 112 (Mar. 2016) 135–145, https://doi.org/10.1016/J. ENCONMAN.2016.01.003. [15] J. Zhang, et al., Optimal thermal management on server cooling system to achieve minimal energy consumption based on air-cooled chiller, Energy Rep. 8 (Dec. 2022) 154–161, https://doi.org/10.1016/J.EGYR.2022.10.237. [16] W. He, et al., Performance optimization of server water cooling system based on minimum energy consumption analysis, Appl. Energy 303 (Dec. 2021), 117620, https://doi.org/10.1016/J.APENERGY.2021.117620. [17] W. He, et al., Optimal thermal management of server cooling system based cooling tower under different ambient temperatures, Appl. Therm. Eng. 207 (May 2022), 118176, https://doi.org/10.1016/J.APPLTHERMALENG.2022.118176. [18] N. Lamaison, J.B. Marcinichen, J.R. Thome, Recent advances in on-chip cooling systems: Experimental evaluation and dynamic modeling, in: Proc. 15th Int. Heat Transf. Conf. IHTC 2014, no. January 2016, 2014, doi: 10.1615/ihtc15.kn.000006. [19] Y. Zhang, C. Li, M. Pan, Design and performance research of integrated indirect liquid cooling system for rack server, Int. J. Therm. Sci. 184 (Feb. 2023), 107951, https://doi.org/10.1016/J.IJTHERMALSCI.2022.107951. [20] B.B. Kanbur, C. Wu, S. Fan, F. Duan, System-level experimental investigations of the direct immersion cooling data center units with thermodynamic and thermoeconomic assessments, Energy 217 (2021), 119373, https://doi.org/ 10.1016/j.energy.2020.119373. [21] D. Gandhi et al., Computational Analysis for Thermal Optimization of Server for Single Phase Immersion Cooling, ASME 2019 Int. Tech. Conf. Exhib. Packag. Integr. Electron. Photonic Microsystems, InterPACK 2019, Dec. 2019, doi: 10.1115/IPACK2019-6587. [22] C.C. Cheng, P.C. Chang, H.C. Li, F.I. Hsu, Design of a single-phase immersion cooling system through experimental and numerical analysis, Int. J. Heat Mass Transf. 160 (Oct. 2020), 120203, https://doi.org/10.1016/J. IJHEATMASSTRANSFER.2020.120203. [23] A. Niazmand, P. Murthy, S. Saini, P. Shahi, P. Bansode, D. Agonafer, Numerical Analysis of Oil Immersion Cooling of a Server Using Mineral Oil and Al 2 O 3 Nanofluid, ASME 2020 Int. Tech. Conf. Exhib. Packag. Integr. Electron. Photonic Microsystems, InterPACK 2020, Dec. 2020, doi: 10.1115/IPACK2020-2662. [24] Y. Huang, J. Ge, Y. Chen, C. Zhang, Natural and forced convection heat transfer characteristics of single-phase immersion cooling systems for data centers, Int. J. Heat Mass Transf. 207 (Jun. 2023), 124023, https://doi.org/10.1016/J. IJHEATMASSTRANSFER.2023.124023. [25] M. Muneeshwaran, Y.C. Lin, C.C. Wang, Performance analysis of single-phase immersion cooling system of data center using FC-40 dielectric fluid, Int. Commun. Heat Mass Transf. 145 (Jun. 2023), 106843, https://doi.org/10.1016/J. ICHEATMASSTRANSFER.2023.106843. [26] H. Shrigondekar, Y.C. Lin, C.C. Wang, Investigations on performance of singlephase immersion cooling system, Int. J. Heat Mass Transf. 206 (Jun. 2023), 123961, https://doi.org/10.1016/J.IJHEATMASSTRANSFER.2023.123961. [27] X. Sun, Z. Han, X. Li, Simulation study on cooling effect of two-phase liquidimmersion cabinet in data center, Appl. Therm. Eng. 207 (May 2022), 118142, https://doi.org/10.1016/J.APPLTHERMALENG.2022.118142. [28] M. Lionello, M. Rampazzo, A. Beghi, D. Varagnolo, M. Vesterlund, Graph-based modelling and simulation of liquid immersion cooling systems, Energy 207 (2020), 118238, https://doi.org/10.1016/j.energy.2020.118238. [29] Y. Luo, J. Andresen, H. Clarke, M. Rajendra, M. Maroto-Valer, A decision support system for waste heat recovery and energy efficiency improvement in data centres, Appl. Energy 250 (January) (2019) 1217–1224, https://doi.org/10.1016/j. apenergy.2019.05.029. [30] “SUBMER SmartPod.” https://submer.com/smartpod/ (accessed Jun. 15, 2023). [31] “Open Computer Project.” https://www.opencompute.org/ (accessed Jun. 15, 2023). [32] “ICE DataCentre.” https://www.ri.se/en/ice-datacenter (accessed Jun. 15, 2023). [33] J. Gustafsson, S. Fredriksson, D. Olsson, A demonstration of monitoring and measuring data centers for energy eficiency using opensource tools, in: Proc. Ninth Int. Conf. Futur. Energy Syst. 2018 Jun 12, 2018, pp. 506-512. [34] “Wiki Ubuntu stress-ng.” https://wiki.ubuntu.com/Kernel/Reference/stress-ng (accessed Jun. 15, 2023). [35] TESS, “TRNSYS 18.” [Online]. Available: https://www.trnsys.com/. [36] Q. Luo, C. Wang, H. Wen, L. Liu, Research and optimization of thermophysical properties of sic oil-based nanofluids for data center immersion cooling, Int. Commun. Heat Mass Transf. 131 (Feb. 2022), 105863, https://doi.org/10.1016/J. ICHEATMASSTRANSFER.2021.105863. [37] “SUBMER SmartCoolant.” https://submer.com/smart-coolant-liquid/ (accessed Jun. 21, 2023). [38] C. J. Willmott, M. Robeson, K. Matsuura, Short Communication A refined index of model performance, vol. 2094, no. September 2011, 2012, pp. 2088–2094, doi: 10.1002/joc.2419. P. Taddeo et al.