Productive Efficiency of Energy-Aware Data Centers

energies Article Productive Efficiency of Energy-Aware Data Centers Damián Fernández-Cerero 1,* , Alejandro Fernández-Montes 1and Francisco Velasco 2 1Department of Computer Languages and Systems, University of Seville, 41012 Sevilla, Spain; [email protected] 2Department of Applied Economy I, University of Seville, 41018 Sevilla, Spain; [email protected] *Correspondence: damiancerer[email protected]; Tel.: +34-954-559-531 Received: 17 July 2018; Accepted: 1 August 2018; Published: 8 August 2018 Abstract: Information technologies must be made aware of the sustainability of cost reduction. Data centers may reach energy consumption levels comparable to many industrial facilities and small-sized towns. Therefore, innovative and transparent energy policies should be applied to improve energy consumption and deliver the best performance. This paper compares, analyzes and evaluates various energy efficiency policies, which shut down underutilized machines, on an extensive set of data-center environments. Data envelopment analysis (DEA) is then conducted for the detection of the best energy efficiency policy and data-center characterization for each case. This analysis evaluates energy consumption and performance indicators for natural DEA and constant returns to scale (CRS). We identify the best energy policies and scheduling strategies for high and low data-center demands and for medium-sized and large data-centers; moreover, this work enables data-center managers to detect inefficiencies and to implement further corrective actions. Keywords: data envelopment analysis; return-to-scale; cloud computing; efficiency; energy policies 1. Introduction Data centers, which constitute the computational muscle for cloud computing, can be compared in energy consumption to many industrial facilities and towns. The latest trends show that these infrastructures represent approximately 2% of global energy consumption [ 1 ], with a 5% annual growth rate [2]. The data envelopment analysis mathematical model enables the management organizational divisions to measure the performance of an organization by providing the relative efficiency of each organizational unit. This relative efficiency measurement can be applied to a set of decision-making units, also known as DMUs, or for productive efficiency. The productive efficiency, also called technical efficiency, involves a collection of inputs (the resources needed for the production) and outputs (the production achieved). To this end, DEA constructs an “efficiency frontier” which places the relative performance of all units so these can be contrasted. This method is notably well-suited for the examination of the behavior of complex relations, even unknown, between numerous inputs and outputs, where the decisions made are affected by a level of uncertainty [ 3 ]. Moreover, DEA has been used both in private [4,5] and in public contexts [6–9]. Many initiatives have emerged looking for the decrease of the consumption of energy and the CO 2 trace of data-centers, especially those of a medium and large size. These facilities are composed of thousands and even tens of thousands of machines. A substantial part of these initiatives focuses on the improvement of the Power Usage Effectiveness (PUE) , that is the amount of energy consumed in non-computational tasks, such as power supply, cooling and networking components. This accounts for more than half of the energy consumption of an Internet data-center (IDC). Several strategies are proposed to significantly improve energy efficiency in large-scale clusters [ 10 ]: cooling and temperature management [ 11 , 12 ]; power proportionality for CPU and Energies 2018,11, 2053 ; doi:10.3390/en11082053 www.mdpi.com/journal/energies Energies 2018,11, 2053 2 of 17 memory hardware components [ 13 , 14 ]; fewer energy-hungry and non-mechanical hard disks [ 15 ]; and new proposals for energy distribution [16]. On the other hand, almost 50% of energy is consumed by computational servers to satisfy the incoming workload. The job arrival is not stable over time, but usually presents correlative low and high periods, such as those present in day/night and weekday/weekend workload patterns. Such scenarios present a huge opportunity for the improvement of energy efficiency through proper scheduling and through the application of low-energy consumption modes to servers, since keeping servers in an idle state is extremely energy-inefficient. Many energy-aware schedulers, which aim to raise server usage, have been proposed in order to free up the maximum amount of machines so that they may put into hibernation [ 17 – 19 ]. In addition to these schedulers, several energy-conservation strategies may be applied in virtualized environments, such as the consolidation and migration of virtual machines [20,21]. Other strategies focus on the reduction of energy consumption in specific scenarios, such as those of distributed file systems [22,23]. The most aggressive approach involves the shut-down of underutilized servers in order to minimize energy consumption. Several shut-down policies have been proposed for grid computing environments in [ 24 ]. This strategy is yet to be widely implemented in working data-centers since a natural reticence to worsening QoS is usually present in data-center operators [25]. The innovation of the research presented in this paper involves the utilization of data envelopment analysis (DEA) as a mathematical technique to compare the efficiency regarding the consumption of energy and the performance of various workload scenarios, scheduling models and energy efficiency policies. This efficiency analysis enables data-center operators to make appropriate decisions about the number of machines, the scheduling solution and the shut-down strategy that must be applied so that data-centers run optimally. The final goal is the maximization of the productive efficiency, which is computed as the amount of energy consumed to serve a workload with a determined performance. The major contributions of this paper can be summarized as follows: 1. Extensive empirical experimentation and analysis of various cloud-computing scenarios with a trustworthy and detailed simulation tool. 2. Impact analysis in terms of the energy consumption and performance of several energy efficiency policies, which shut-down idle machines by means of data envelopment analysis. 3. DEA-conducted analysis of the performance impact and energy consumption of a set of scheduling models for large-scale data-centers. 4. Empirical determination and proposal of corrective actions to achieve optimal efficiency. The work is organized as follows. In Section 2, the authors introduce the current literature for the utilization of DEA presented for various areas, as well as the DEA model employed in this work. In Section 3, we briefly explain the set of energy efficiency policies that shut down idle servers. The scheduling models considered are explained in Section 4. In Section 5, the tool used for the simulation, the experimental environment, the energy model and DEA inputs/outputs are presented. Natural constant returns to scale (CRS) DEA results are described and analyzed in Section 6. Finally, we summarize this paper and present conclusions in Section 7. 2. Data Envelopment Analysis Model Data envelopment Analysis (DEA) is a method that analyzes the connections between the outputs and inputs required in a production process in order to establish the efficiency frontiers [ 26 ]. This non-parametric technique was first described for the determination of the efficiency of DMUs by [ 27 ] and was formally defined by [ 28 ]. DEA has been proposed to measure the efficiency in various areas of operations research and management science [ 29 – 32 ]. Moreover, it has been applied to measure the environmental performance by other authors [ 33 – 40 ], who describe the gains of this method in the field of environmental management, which is a matter of undoubted relevance Energies 2018,11, 2053 3 of 17 for the valuation of the sustainable development ability and pathway [ 41 ]. A critical feature of DEA for environmental analysis is the inclusion of desirable and undesirable outputs along with its own production variables, which cannot be isolated in an environmental analysis model of these features [42] . In this way, ref. [ 36 ] have refined a non-radial and radial model of DEA for environmental measurements. This approach separates the outputs into desirable and undesirable and presents two concepts: natural and managerial disposability. In this work, we employ the DEA radial approach for environmental assessments proposed by [ 37 ]. It should be borne in mind that a main feature of this approach is the utilization of DEA-RAM (range-adjusted measure), first proposed by [ 43 ] to treat in a unified manner the analysis of managerial and natural disposability. 2.1. Natural Disposability Natural disposability refers to a DMU that improves its efficiency by decreasing its inputs in order to decrease its undesirable outputs, as well as to increase the desirable outputs. In Model (1), each j-th DMU j= 1, . . . , n , considers inputs Xj= (x1j , . . . , xmj)T for the production of desirable outputs Gj= (g1j , . . . , gsj)T and undesirable outputs Bj= (b1j , . . . , bhj)T . Furthermore, dx i , i= 1, . . . , m , dg r , r= 1, . . . , s and db f , f= 1, . . . , h are all slack variables which are related to inputs, desirable and undesirable outputs, respectively. λ= (λ1 ,..., λn)T are structural or intensity variables, which are unknown and are used for the connection of the input and output vectors by means of a convex combination. R is the range resolute through the lower and upper limits of inputs, desirable outputs and undesirable outputs, denoted by: Rx i= (m+s+h)−1(max{xij/j=1, . . . , n} − min{xij/j=1, . . . , n}) Rg r= (m+s+h)−1(max{grj/j=1, . . . , n} − min{grj/j=1, . . . , n})and Rb f= (m+s+h)−1(max{bf j/j=1, . . . , n} − min{bf j/j=1, . . . , n}) The natural efficiency of the k-th policy is computed by the following CRS and radial VRS model (see [37] for a better understanding): max ξ+e( m ∑ i=1 Rx idx i+ s ∑ r=1 Rg rdg i+ h ∑ f=1 Rb fdb i) s.t.∑n j=1xijλj+dx i=xik ,i=1, . . . , m, ∑n j=1grjλj−dg r−ξgrk =grk ,r=1, . . . , s, ∑n j=1bf jλj+db f+ξbf k =bf k ,f=1, . . . , h, dx i≥0 ,i=1, . . . , m, dg r≥0 ,r=1, . . . , s, db f≥0 , f=1, . . . , h, ξUnrestricted (1) where the unrestricted parameter ξ denotes an unknown inefficiency rate expressing the gap between the efficiency frontier and an empirical group of undesirable and desirable outputs. The parameter e takes the value of 0.0001 in this work to minimize the influence of slack variables. If the restriction ∑n j=1λj=1 is added to Model (1), then the obtained model is a VRS (Model (1∗)). The first restriction in equation systems ((1), (1 ∗ )) explores the values of λj to create a composite unit , considering inputs such as: ∑n j=1xijλj=−dx i+xik , i= 1, . . . , m . The values of the inputs can be decreased when the positive slack variables dx i are present. This may unquestionably vary the given rates, which implies that the system presents some inefficiencies. Energies 2018,11, 2053 4 of 17 In the same way, the second restriction, ∑n j=1grjλj=dg r+ξgrk +grk , r= 1, . . . , s , indicates that the desirable outputs can be maintained or increased by making an increase of the slack variable dg r and a radial expansion ξgrk. The third restriction, ∑n j=1bf jλj=−db f−ξbf k +bf k , f= 1, . . . , h , shows the decrease of the inputs, and then, we could reduce the undesirable outputs both in their slack variables and radially. The objective function considers that two origins of inefficiency may be established. A k-policy can be considered efficient when the following two conditions are met: (a) ξ= 0 ; (b) dx i= 0, dg r= 0, db f= 0. In this case, the k-policy belongs to the efficiency frontier, since it fulfills the constraints present in equation systems ((1), (1 ∗ )), and consequently, the objective function takes a value of zero. Otherwise, the value of the objective function for non-efficient policies is greater than zero, due to possible displacements in the slack variables and radial movements. The natural efficiency is then computed by: θ∗=1−"ξ∗+e( m ∑ i=1 Rx idx∗ i+ s ∑ r=1 Rg rdg∗ i+ h ∑ f=1 Rb fdb∗ i)# The value of this unified efficiency measure ranges between zero and one. If the k-policy is efficient, then the objective function of equation systems ((1), (1 ∗ )) is zero, and hence, the efficiency score equals θ∗=1. Slack variables resulting in the optimality of the models represented in equation systems ((1), (1∗)) show the level of inefficiency. 2.2. Managerial Disposability The managerial efficiency of the k-th policy is evaluated by the following CRS and VRS radial model [37]: s.t.∑n j=1xijλj−dx i=xik ,i=1, . . . , m, ∑n j=1grjλj−dg r−ξgrk =grk ,r=1, . . . , s, ∑n j=1bf jλj+db f+ξbf k =bf k ,f=1, . . . , h, dx i≥0 ,i=1, . . . , m, dg r≥0 ,r=1, . . . , s, db f≥0 , f=1, . . . , h, ξUnrestricted (2) Similarly, if the restriction ∑n j=1λj= 1 is added to Model (2), then the obtained model is a VRS (Model (2 ∗ )). In this model (2), increasing the inputs is allowed since new technologies that emit less CO2emissions to the atmosphere can be used. By using the VRS models, we can obtain the returns to scale (RTS) and damage to scale (DTS) (see [ 37 ] for a better understanding). It is clear that for the natural efficiency, the returns to scale have to be increasing, and for managerial efficiency, the damages to scale have to be decreasing. Otherwise, the technical units are not working well and should correct the imbalances, using the information of the efficient units to which they have to be similar (peers). 3. Energy Policies for Data Centers at a Glance The following set of energy efficiency policies for shutting down underutilized machines have been developed in this work as an evolution of those presented in [ 24 ], which have been adapted to the more complex reality of the cloud-computing paradigm: •Never: prevents any shut-down process. •Always: shuts down every server running in an idle state. •Load: shuts down machines when data-center load pressure fails to reach a given threshold. • Margin: assures that a determined number of machines are turned on and available before shutting down any machine. Energies 2018,11, 2053 5 of 17 • Random: shuts down machines randomly by means of a Bernoulli distribution with parameter 0.5. • Exponential: shuts down machines when the probability of one incoming task negatively impacting on the data-center performance is lower than a given threshold. This probability is computed by means of the exponential distribution. • Gamma: shuts down machines when the probability of incoming tasks oversubscribing to the available resources in a particular time period is lower than a given threshold; this probability is computed by means of the Gamma distribution. 4. Scheduling Models for Data Centers at a Glance Cluster schedulers constitute a core part of cloud computing systems, since they are responsible for optimal task assignation to computing nodes. Several degrees of parallelism have been added to overcome the limitations present in central monolithic scheduling approaches when complex and heterogeneous systems with a high number of incoming jobs are considered. The following scheduling models are studied in this work: • Monolithic: A centralized and single scheduler is responsible for scheduling all tasks in the workload in this model [ 44 ]. This scheduling approach may be the perfect choice when real-time responses are not required [ 45 , 46 ], since the omniscient algorithm performs high-quality task assignations by considering all restrictions and features of the data-center [ 47 – 50 ] at the cost of longer latency [ 46 ]. The scheduling process of a monolithic scheduler, such as that given by Google Borg [51], is illustrated in Figure 1. B1 B2 S1 S2 M1 M9 M2 M10 M3 M11 M4 M12 M5 M13 M6 M14 M7 M15 M8 M16 Scheduler Scheduling logic Figure 1. Monolithic scheduler architecture. M, worker node; S, service task; B, batch task [52]. • Two-level: This model achieves a higher level of parallelism by splitting the resource allocation and the task placement: a central manager blocks the whole cluster every time a scheduler makes a decision to offer computing resources to schedulers; and a set of parallel application-level schedulers performs the scheduling logic against the resources offered. This strategy enables the Energies 2018,11, 2053 6 of 17 development of sub-optimal scheduling logic for each application, since the state of the data-center is not shared with the central manager, nor with the application schedulers. The workflow of the Two-level schedulers [53,54] is represented in Figure 2. B1 B2 S1 S2 M1 M9 M2 M10 M3 M11 M4 M12 M5 M13 M6 M14 M7 M15 M8 M16 Resource Scheduler R1 R2 M1 M9 M2 M10 M1 M9 M2 M10 M7 M15 M8 M16 M7 M15 M8 M16 SA2 SA1 C1 C2 O1 O2 Figure 2. Two-level scheduler architecture. C, commit; O, resource offer; SA-, scheduler agent [52]. • Shared-state schedulers: On the other hand, in shared-state schedulers, such as Omega [ 55 ], the state of the data-center is available to all the schedulers. The central manager coordinates all the simultaneous parallel schedulers, which perform the scheduling logic against an out-of-date copy of the state of the data-center. The scheduling decisions are then committed to the central manager, which strives to apply these decisions. The utilization of stale views of the cluster by the schedulers can result in conflicts, since the chosen resources may not longer be available. In such a scenario, the local view of the state of the data-center stored in the scheduler is refreshed before the repetition of the scheduling process. The workflow of the shared-state scheduling model is represented in Figure 3. Energies 2018,11, 2053 7 of 17 B1 B2 S1 S2 M1 M9 M2 M10 M3 M11 M4 M12 M5 M13 M6 M14 M7 M15 M8 M16 U1 U2 SA2 SA1 C1 C2 Cluster state Figure 3. Shared-state scheduler architecture. U, cluster state update [52]. 5. Methodology In these next sections, the experimental environment designed for the implementation of the natural CRS DEA analysis is presented. The workflow followed in this work is shown in Figure 4. Selection of inputs/ outputs Generation of theoretical DMUs (combination of parameters) Simulation of Decisionmaking Units (DMUs) DEA analysis with simulation results Figure 4. Methodology workflow employed in this work. DEA, data envelopment analysis. 5.1. Simulation Tool The SCORE simulator [ 52 ] is employed in this work, since simulation is the best alternative in scenarios where the implementation of the proposed strategies on real large-scale data-centers remains unfeasible. This simulator provides us with the tools for the development and application of the energy policies described in Section 3and the scheduling models presented in Section 4on realistic large-scale cloud computing systems. Energies 2018,11, 2053 8 of 17 5.2. Environment and DMU Definition Following the trends presented in [ 56 , 57 ], two utilization environments have been simulated in this paper for seven days of operation: • the low-utilization scenario, which represents highly over-provisioned infrastructures and achieves an average utilization of approximately 30%. • the high-utilization scenario, which represents facilities of a more efficient nature that use approximately 65% of available resources on average. These scenarios are applied to three data-center sizes: (a) Small: composed of 1000 computing servers; (b) Medium: composed of 5000 computing servers; and (c) Large: composed of 10,000 computing servers. Each server is equipped with four CPU cores and 8 GB of RAM. Decision-making units (DMUs) are defined by the following elements: (a) an energy efficiency policy; (b) a scheduling model; and (c) a workload scenario. 5.3. Energy Model The following states are presented for each resource in the energy model applied in this work: (a) Idle: when the machine is not executing tasks; and (b) Busy: otherwise. Let ti idle represent the time the i -th resource is idle, and let ti busy denote the time during which the machine is computing tasks. In the same way, Pi idle and Pi busy represent the power required for the machines to run in these states, respectively. The time a machine spends on executing a job may be defined as follows: tij busy =max t∈TasksiCt(3) where Tasksij represents the tasks of the j -th job assigned to Mi and Ct denotes the completion time of the t-th task of the j-th job. In the same way, the total time a machine is executing tasks and the total time it is in an idle state may be defined as follows: ti busy = j ∑ j=1 tij busy (4) ti idle =ti total −ti busy (5) where ti total represents the total operation time. Therefore, we can express the energy consumption as follows: m ∑ i=1 (Pi busy ∗ti busy +Pi idle ∗ti idle)(6) The considered power states, transitions and values for the energetic model are shown in Figure 5. Energies 2018,11, 2053 9 of 17 ON [150W * nCores} HIBERNATED 10 W POWERING ON [160W * nCores] HIBERNATING [160W * nCores] 30s 10s THibernated -> On TOn -> Hibernated IDLE [70W * nCores} Figure 5. Machine power states [52]. 5.4. DEA Inputs and Outputs The inputs and outputs considered in DEA analysis and representative experimentation values are shown in Tables 1and 2, respectively. One hundred and eight DMUs were analyzed, which were the result of the combination of all energy policies, scheduling models, data-center sizes and workload types described in Sections 3,4and 5.2, respectively. However, for clarity, a subset of the most interesting eighteen DMUs i shown in this paper. Each environment presents the following inputs and outputs: • Inputs: Two inputs are considered in this work: (a) the number of machines in the data-center (D.C.), as shown in Section 5.2; and (b) the number of shut-down operations performed. These inputs may be reduced or kept equal. • Outputs: One desirable output and two undesirable outputs are considered in this paper: (a) the time used to perform tasks’ operations. The longer the time, the less idle the data-center. This good input can be maximized or kept equal; (b) the energy consumption of the data-center. The lower the energy consumption, the more efficient the data-center. This bad input may be reduced or kept equal; and (c) the average time jobs spend in a queue until they are scheduled. The shorter the time, the more performant the system is. This bad input may be reduced or kept equal. Table 1. DEA inputs and outputs. Action column arrows mean whether the input/output value may be decreased (down arrow), increased (up arrow) or kept equal. Parameter Description Action Inputs Data-center size Number of machines in the data-center ↓Ö #shut-downs Number of shut-down operations ↓Ö Outputs Computation time Total amount of useful task computation ↑Ö Energy consumption Total data-center energy consumption ↓Ö Queue time Average time until jobs are fully scheduled ↓Ö Energies 2018,11, 2053 16 of 17 27. Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978,2, 429–444. [CrossRef] 28. Banker, R.D.; Charnes, A.; Cooper, W.W. Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis. Manag. Sci. 1984,30, 1078–1092. [CrossRef] 29. Fernández-Montes, A.; Velasco, F.; Ortega, J. Evaluating decision-making performance in a grid-computing environment using DEA. Expert Syst. Appl. 2012,39, 12061–12070. [CrossRef] 30. Campos, M.; Fernández-Montes, A.; Gavilan, J.; Velasco, F. Public resource usage in health systems: A data envelopment analysis of the efficiency of health systems of autonomous communities in Spain. Public Health 2016,138, 33–40. [CrossRef] [PubMed] 31. Fernández-Serrano, J.; Berbegal, V.; Velasco, F.; Expósito, A. Efficient entrepreneurial culture: A cross-country analysis of developed countries. Int. Entrep. Manag. J. 2017,14, 105–127. [CrossRef] 32. Exposito, A.; Velasco, F. Municipal solid-waste recycling market and the European 2020 Horizon Strategy: A regional efficiency analysis in Spain. J. Clean. Prod. 2018,172, 938–948. [CrossRef] 33. Scheel, H. Undesirable outputs in efficiency valuations. Eur. J Oper. Res. 2001,132, 400–410. [CrossRef] 34. Färe, R.; Grosskopf, S.; Hernandez-Sancho, F. Environmental performance: An index number approach. Resour. Energy Econ. 2004,26, 343–352. [CrossRef] 35. Zhou, P.; Ang, B.W.; Poh, K.L. Measuring environmental performance under different environmental DEA technologies. Energy Econ. 2008,30, 1–14. [CrossRef] 36. Sueyoshi, T.; Goto, M. Returns to scale and damages to scale on US fossil fuel power plants: Radial and non-radial approaches for DEA environmental assessment. Energy Econ. 2012,34, 2240–2259. [CrossRef] 37. Sueyoshi, T.; Goto, M. DEA radial measurement for environmental assessment: A comparative study between Japanese chemical and pharmaceutical firms. Appl. Energy 2014,115, 502–513. [CrossRef] 38. Halkos, G.E.; Tzeremes, N.G. Measuring the effect of Kyoto protocol agreement on countries’ environmental efficiency in CO 2 emissions: An application of conditional full frontiers. J. Prod. Anal. 2014 ,41, 367–382. [CrossRef] 39. Sanz-Díaz, M.T.; Velasco-Morente, F.; Yñiguez, R.; Díaz-Calleja, E. An analysis of Spain’s global and environmental efficiency from a European Union perspective. Energy Policy 2017,104, 183–193. [CrossRef] 40. Vlontzos, G.; Pardalos, P. Assess and prognosticate green house gas emissions from agricultural production of EU countries, by implementing, DEA Window analysis and artificial neural networks. Renew. Sustain. Energy Rev. 2017,76, 155–162. [CrossRef] 41. Yu, S.H.; Gao, Y.; Shiue, Y.C. A Comprehensive Evaluation of Sustainable Development Ability and Pathway for Major Cities in China. Sustainability 2017,9, 1483. [CrossRef] 42. Dios-Palomares, R.; Alcaide, D.; Pérrez, J.D.; Bello, M.J.; Prieto, A.; Zúniga, C.A. The Environmental Efficiency using Data Envelopment Analysis: Empirical methods and evidences. In The stated of the Art for Bieconomic and Climate Change; Editorial Universitaria UNAN Leon, Ed.; Red de Bioeconomia y Cambio Climático: Cordoba, Spain, 2017; p. 48. 43. Cooper, W.W.; Park, K.S.; Pastor, J.T. RAM: A range adjusted measure of inefficiency for use with additive models, and relations to other models and measures in DEA. J. Prod. Anal. 1999,11, 5–42. [CrossRef] 44. Delimitrou, C.; Kozyrakis, C. Paragon: QoS-aware scheduling for heterogeneous datacenters. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, Houston, TX, USA, 16–20 March 2013; pp. 77–88. 45. Isard, M.; Prabhakaran, V.; Currey, J.; Wieder, U.; Talwar, K.; Goldberg, A. Quincy: Fair scheduling for distributed computing clusters. In Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, Big Sky, MT, USA, 11–14 October 2009; pp. 261–276. 46. Delimitrou, C.; Sanchez, D.; Kozyrakis, C. Tarcil: Reconciling scheduling speed and quality in large shared clusters. In Proceedings of the Sixth ACM Symposium on Cloud Computing, Kohala Coast, HI, USA, 27–29 August 2015; pp. 97–110. 47. Grandl, R.; Ananthanarayanan, G.; Kandula, S.; Rao, S.; Akella, A. Multi-resource packing for cluster schedulers. ACM SIGCOMM Comput. Commun. Rev. 2015,44, 455–466. [CrossRef] 48. Zaharia, M.; Borthakur, D.; Sen Sarma, J.; Elmeleegy, K.; Shenker, S.; Stoica, I. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European Conference on Computer systems, Paris, France, 13–16 April 2010; pp. 265–278. Energies 2018,11, 2053 17 of 17 49. Delimitrou, C.; Kozyrakis, C. Quasar: Resource-efficient and QoS-aware cluster management. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, Salt Lake City, UT, USA, 1–5 March 2014; pp. 127–144. 50. Zhang, X.; Tune, E.; Hagmann, R.; Jnagal, R.; Gokhale, V.; Wilkes, J. CPI 2: CPU performance isolation for shared compute clusters. In Proceedings of the 8th ACM European Conference on Computer Systems, Prague, The Czech Republic, 15–17 April 2013; pp. 379–391. 51. Verma, A.; Pedrosa, L.; Korupolu, M.; Oppenheimer, D.; Tune, E.; Wilkes, J. Large-scale cluster management at Google with Borg. In Proceedings of the Tenth European Conference on Computer Systems, Bordeaux, France, 21–24 April 2015; p. 18. [CrossRef] 52. Fernández-Cerero, D.; Fernández-Montes, A.; Jakóbik, A.; Kołodziej, J.; Toro, M. SCORE: Simulator for cloud optimization of resources and energy consumption. Simul. Model. Pract. Th. 2018,82, 160–173. [CrossRef] 53. Hindman, B.; Konwinski, A.; Zaharia, M.; Ghodsi, A.; Joseph, A.D.; Katz, R.H.; Shenker, S.; Stoica, I. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, Boston, MA, USA, 30 March–1 April 2011; pp. 295–308. 54. Vavilapalli, V.K.; Murthy, A.C.; Douglas, C.; Agarwal, S.; Konar, M.; Evans, R.; Graves, T.; Lowe, J.; Shah, H.; Seth, S.; et al. Apache hadoop yarn: Yet another resource negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing, Santa Clara, CA, USA, 1–3 October 2013; p. 5. 55. Schwarzkopf, M.; Konwinski, A.; Abd-El-Malek, M.; Wilkes, J. Omega: Flexible, scalable schedulers for large compute clusters. In Proceedings of the 8th ACM European Conference on Computer Systems, Prague, The Czech Republic, 15–17 April 2013; pp. 351–364. 56. Armbrust, M.; Fox, A.; Griffith, R.; Joseph, A.D.; Katz, R.; Konwinski, A.; Lee, G.; Patterson, D.; Rabkin, A.; Stoica, I.; et al. A view of cloud computing. Commun. ACM 2010,53, 50–58. [CrossRef] 57. Ruth, S. Reducing ICT-related carbon emissions: An exemplar for global energy policy? IETE Tech. Rev. 2011,28, 207–211. [CrossRef] c 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).