Efficient Computing

Data Center Efficiency Measurements

We design each element of our Google data centers to operate at optimal efficiency, from the servers, storage, and networking equipment to facility power and cooling infrastructure. Continuous measurement of power usage by all these elements lets us monitor the health, operating cost, and relative efficiency of our data centers. As we outline below, our approach has led to state-of-the-art efficiency; our efforts have cut our electricity consumption and significantly reduced both our operating costs and environmental footprint. As a result, we believe our data centers are the world's most efficient. 

Such a strong claim demands evidence, especially in light of recent criticism of companies "gaming the numbers." On this page we will explain our measurements in detail to ensure that they are realistic and accurate. It is worth noting that we only show data for facilities with an actual IT load above 5MW, to eliminate any inaccuracies that can occur when measuring small values. This section is aimed at data center experts, but we have tried to make it accessible to a general technical audience as well.

To assess the efficiency of our data centers we use the Power Usage Effectiveness (PUE) metric¹. PUE puts a focus on maximizing the power devoted to the equipment running applications and minimizing the power consumed by support functions like cooling and power distribution.  PUE is defined as the ratio of the total power consumed by a data center to the power consumed by the IT equipment that populate the facility:

For example, a PUE of 2.0 indicates that for every watt of IT power, an additional watt is consumed to cool and distribute power to the IT equipment.

We measure PUE in the spirit it was intended and follow its definition strictly.  Our numbers include all power-consuming items in the facility, with one single exception (power used by the office area), and are measured throughout the year and not just during favorable seasons.  Similarly, we count only the servers, storage and networking equipment as IT equipment power. For example, electrical losses in a server's power cord would be counted as overhead, not as IT power. Similarly, we measure total utility power at the utility side of the substation so that losses in substation transformers are included in our PUE. (For more details see the section on measurement methodology further below.) We strongly encourage all data center owners and operators to adhere to Green Grid PUE measurement standards, so we can all drive efficiency forward with meaningful comparisons.

The total data center power consumption is comprised of three major categories: IT power, data center power distribution losses, i.e., losses associated with delivering power to the IT equipment from the electric grid, and the power consumed to operate the data center cooling system.  The intent of the PUE metric is to highlight the fraction of the total data center power consumption devoted to IT equipment.  The ideal PUE value is 1.0, corresponding to a data center where all of the electrical grid power supplied to a data center is devoted to IT equipment and no power is used for cooling and power distribution.  A PUE <1 would be possible with on-site generation from waste heat, but currently this is commercially impractical to implement.

PUE Benchmark Data

Several technical papers have provided estimates and actual measured PUE data for data centers^2-4.  In their report to the U.S. Congress⁵, the US EPA provided a set of efficiency improvement scenarios which predicted future data center energy consumption and associated PUE values to 2011.  The EPA listed four categories of data center efficiency improvements, with increasing cost and complexity, summarized in Table 1.  The report estimates that in 2006, the typical enterprise data center had a PUE of 2.0 or higher.  It is expected that equipment efficiency improvements alone, with current practices, could result in a 2011 PUE of 1.9.  Data centers combining these efficiency gains with better operational practices are expected to reach a PUE of 1.7.  Data centers with advanced efficiency solutions are projected to reach a PUE of 1.3.  Beyond that, the EPA predicted that "state-of-the-art" data centers, employing exotic energy-efficient power and cooling technologies such as liquid cooling and combined heat-and-power energy generation solutions, could reach a PUE of 1.2. We're happy to report that today, on average for all Google designed data centers, we meet the EPA's most optimistic scenario for 2011.

Scenario	PUE
Current Trends	1.9
Improved Operations	1.7
Best Practices	1.3
State-of-the-Art	1.2

Our Results

Figure 1 summarizes the PUE results from all Google-designed data centers with an IT load of at least 5MW and time-in-operation of at least 6 months. The trailing twelve-month (TTM), energy-weighted average PUE for all of these facilities is 1.19, exceeding the EPA's 2011 goal for state-of-the-art data center efficiency. We achieve this result without the use of any exotic techniques the EPA report assumes are necessary to reach this level of efficiency. Individual facilities can return even lower PUE values, with at least one facility returning a TTM PUE of 1.14. Our lowest published quarterly average PUE of an individual facility is 1.11, meaning that all electrical losses and cooling overhead combined used just 11% of the IT load. We believe that each of these data centers ranks among the most efficient large data centers currently in operation today.

Why is there variation among the data? Power and cooling architectures differ between facilities, and the facilities themselves are located in different climates, which influences the PUE performance. Also, the data centers shown in Figure 1 were built at various times since 2005, and over time our designs have evolved and become more efficient. For example, Data Center A is the oldest facility of the group and has one of the highest average PUE values. Data Centers E and F are two of the newest and have among the lowest values. In addition, PUE values are impacted by seasonal weather patterns, and thus the PUE during cooler quarters tends to be lower than in warmer ones. Figure 2 plots daily PUE of data center F through commissioning and into operation. The graph shows early amplified day-to-day variations and elevated PUE typical of bring-up activities, and smoothing as it transitions into operation.

Q3 2009 Performance

Q2 2009 Performance

Q1 2009 Performance

Q4 2008 Performance

Q3 2008 Performance

Measurement Methodology

The PUE of a data center is not a static value. Varying server and storage utilization, the fraction of design IT power actually in use, environmental conditions, and other variables strongly influence PUE. Thus, we use multiple on-line power meters in our data centers to characterize power consumption and PUE over time. These meters permit detailed power and energy metering of the cooling infrastructure and IT equipment separately, allowing for a very accurate PUE determination.  Our facilities contain dozens or even hundreds of power meters to ensure that all of the power-consuming elements are accounted for in our PUE calculation, in accordance with the metric definition⁶. Only the office space energy is excluded from our PUE calculations. Figure 3 shows a simplified power distribution schematic for our data centers.

Equation for PUE for Our Data Centers

• E_US1 Energy consumption for type 1 unit substations feeding the cooling plant, lighting, and some network equipment
• E_US2 Energy consumption for type 2 unit substations feeding servers, network, storage, and CRACs
• E_TX Medium and high voltage transformer losses
• E_HV High voltage cable losses
• E_LV Low voltage cable losses
• E_CRAC CRAC energy consumption
• E_UPS Energy loss at UPSes which feed servers, network, and storage equipment
• E_Net1 Network room energy fed from type 1 unit substitution

Error Analysis

To ensure our PUE calculations are accurate, we performed an uncertainty analysis using the root sum of the squares (RSS) method.  Our uncertainty analysis shows that the overall uncertainty in the PUE calculations is less than 2% (99.7% confidence interval).  Our power meters are highly accurate (ANSI C12.20 0.2 compliant) so that measurement errors have a negligible impact on overall PUE uncertainty.  The contribution to the overall uncertainty for each term described above is outlined in the table below.

Term	Overall Contribution to Uncertainty
EUS1	4%
EUS2	9%
ETX	10%
ECRAC	70%
EUPS	<1%
EHV	2%
ELV	5%
ENet1	<1%

Conclusions and Further Work

While we've made a lot of progress when it comes to data center efficiency, we're still learning. As we review our real-time efficiency performance we might, for example, notice that one of our data centers is not performing consistently with others of similar size and locale. So we'll take a closer look at optimizing that facility. Are we using fans to cool spaces that don't need to be cooled? Is the thermostat at the right set-point? Can we reduce the time the chillers need to run while keeping the machines operational? Are there lessons that are applicable from better performing data centers to existing facilities? Using the least amount of power to do the most amount of computing is the right thing to do for both the environment and our bottom line. So far our efficiency efforts have saved hundreds of millions of kWhrs of electricity, cutting our operating expenses by tens of millions of dollars and averting the emission of tens of thousands of tons of CO2.

Please check back for more details and updated PUE data for these data centers and new ones as they come online.

References

^[1] The Green Grid, 2007, “The Green Grid Data Center Power Efficiency Metrics: PUE and DCiE,” Technical Committee White Paper, http://www.thegreengrid.org/gg_content/TGG_Data_Center_Power_Efficiency_Metrics_PUE_and_DCiE.pdf.

^[2] Malone, C., Belady, C., 2006, “Metrics to Characterize Data Center & IT Equipment Energy Use,” Proceedings of 2006 Digital Power Forum, Richardson, TX.

^[3] Greenberg, S., Mills, E., Tschudi, W., Rumsey, P., Myatt, B., 2006, “Best Practices for Data Centers: Lessons Learned from Benchmarking 22 Data Centers,” ACEEE Summer Study on Energy Efficiency in Buildings, http://eetd.lbl.gov/emills/PUBS/PDF/ACEEE-datacenters.pdf.

^[4] Accenture and Silicon Valley Leadership Group, 2008, “Data Center Energy Forecast Report,” https://microsite.accenture.com/svlgreport/Documents/pdf/SVLG_Report.pdf.

^[5] Environmental Protection Agency, ENERGY STAR Program, 2007, “Report to Congress on Server and Data Energy Efficiency,” http://www.energystar.gov/ia/partners/prod_development/downloads/EPA_Datacenter_Report_Congress_Final1.pdf.

^[6] The Green Grid, 2008, “The Green Grid Metrics: Data Center Infrastructure Efficiency (DCiE) Detailed Analysis,” Technical Committee White Paper, http://www.thegreengrid.org/gg_content/White_Paper_14_-_DCiE_Detailed_Analysis_072208.pdf.

^[7] Barroso, L. and Hölzle, U., 2007, “The Case for Energy-Proportional Computing,” IEEE Computer, Vol. 40, No. 12, December 2007.