Data Centre Cooling – Simplified

Data Centre Cooling

This post discusses cooling (air conditioning) of your data centre.

What is a Data Centre?

Data centres can be small or massive. In simple terms they are rooms (or data halls) where you put your IT equipment.

As the importance, to you, of your data centre grows, you start to consider:

  • UPS
  • Air conditioning
  • Failure
  • Supply quality

Air Conditioning

As you group more and more equipment into a room, the amount of heat rises and so does the temperature.

The IT equipment has recommended operating temperatures for optimum operation and for extended life. These recommendations dictate the temperature and humidity the air conditioning system needs to maintain in your room (See also ASHRAE conditions).

At this point, it’s simple, you need cooling. Often (historically) the entire room was kept at c. 24 Deg C.

ASHRAE

Recent design guidance by ASHRAE has provided a set of conditions that are acceptable for data centre equipment. These conditions allow for your room(s) to be maintained at higher temperatures.

In essence this just saves you a lot of air conditioning. The upper end of acceptable conditions is now 27 Deg C (supply air onto racks).

In summary ASHRAE have come up with the following requirements; a stable environment of (18°C to 27°C) with moisture between (5.5°C) dew point and 60% relative humidity with (15°C) dew point.

Delivery of cold air

The physical constraints of your room often dictate the method of getting cold air into the space. Air can be supplied via under floor plenum or ceiling located ductwork and diffusers.

Some systems employ cooling banks within the racks themselves, either DX or chilled water, with fans drawing air through each cooling bank. These systems are not discussed in this post.

High Level Supply – General

High level supply is normally non preferred. This is because typically, the air is delivered to the room at say 10 to 12oC with a large portion of the room being fully treated.

 High Level Supply – Full Mixing

Typically the air mixes with the entire room load and a mix condition is achieved at the rack equipment. This method is energy intensive.

High level Supply

High Level Supply – Hot Isle/Cold Isle

With this method, the aim is to supply cold air to the inlet side of the racks, with air being discharged to the warm isle side. The warm air is then removed via a return air duct. This method is an improvement on full mixing in terms of energy consumption.

 Under Floor Supply

This is the preferred method of providing cooling for many data halls. This is for three main reasons, namely; it keeps overhead areas free for wiring, cable trays etc;  allows for supply air at low level, local to racks, thus avoiding mixing of supply air with room air, as occurs with high level supply and allows installation of racks without constant need to modify ductwork or grilles.

Under floor supply has many issues that need resolving including:

  • Limitation of supply air being delivered under the floor, with typical un-ducted throws to circa max 20m.
  • Floor leakage
  • Poor air flow control with difficulty experienced in commissioning floor grilles for correct air flow
  • Poor air flow for partially occupied data rooms resulting in multiple AHUs being run to achieve required air flows.

Central Plant

So, we need to cool the air in the room due to the heat load from the racks. This means air conditioning.

There are many new systems around at the moment professing to be the best solution for air conditioning your room. Most are really plugging the low air conditioning PUE (Power required for air conditioning vs IT power load). In general the quoted PUE’s are all very similar.

Don’t be ‘sucked in’ to quickly, when looking at these low air conditioning PUE solutions. Basically the low PUE’s are being achieved purely because they can provide a lot of cooling by using low temperature outside air. Further these manufactures are selling a product, a packaged product and decanting some of their parts is not in their sales interest. No slight intended here, there making a product for sale, we can buy it or not.

Traditionally the large data centres provided cooling by central plant chillers serving in room (or corridor Located) close control air conditioning units (CRAC Units).

CHW CRAC

Chillers were used as these, combined with a cooling tower, had the highest COP (Co-efficient of performance). Chilled water made at the chillers was then pumped to the Crac units, where the return air was drawn over the chilled water cooling coils to be cooled before being supplied back to the underfloor plenum. The only thing missing in this scenario is not utilising the free cooling when the outside air was cold enough to provide ‘free’ cooling (instead of the chillers). With the new ASHRAE conditions allowing for warmer data hall supply air temperatures, allows for even more free cooling availability from the outside air. Thus, this type of system has lost some favour. See later for free cooling chillers.

The main systems under consideration at present are:

  1. Direct Outside Air System, With Adiabatic Cooling + Air Cooled Chiller
  2. Indirect Outside Air System, With Adiabatic Cooling + Air Cooled Chiller
  3. Enhanced Crac unit, with free cooling cycle added in.

All systems purport very low PUE’s.

Simplified Review.

  1. Direct Outside Air Systems

Example Manufacturer: Trane

Trane unit

These systems, are in simple terms, a large air handling unit comprising:

  • Return air connection
  • Outside air connection, with motorised damper
  • Mixing plenum
  • Filters
  • Spill air fans
  • Air to Air heat exchanger with enhanced cooling due to wetted surface
  • Adiabatic cooler
  • Cooling system (coils, compressors and the like)
  • Supply air fans

Advantages

  • Utilises free cooling by using outside air, when low outside air temperatures.
  • Utilises free cooling by removal of latent heat when outside air wet bulb less than indoor wet bulb.
  • For heat transfer uses air directly to space, thus avoids minor loss of efficiency, when using an indirect air heat exchanger.

Disadvantages

Very large unit. Needs careful positioning. Significant air flows required for scavenged air fans, spill air and outside air intake.

Large fan power losses due to:

  •            spill air fans
  •            scarification fans
  •            Main supply air fan overcoming resistance of ductwork, heat exchanger and the like
  • Loss of control of air temperatures and humidity due to possible leaky outside air dampers and the like.
  • In outside air mode, relies purely on filters to remove any external contaminants. This needs consideration in industrial areas, bush fire areas, high insect areas and the like.
  • High water usage, thus on site N+1 water supply required.
  • Room humidity control likely to be difficult, with air direct from outside.
  • As the filters are subject to outside air, regular cleaning will be required to avoid energy loss due to fans overcoming dirty filter pressure drop.

In summary these units are designed to maximise the new ASHRAE acceptable conditions in a data hall. As these new conditions, in simple terms, are warmer, cooling energy is saved by 1, having to do less cooling, 2) ability to use free cooling for a large proportion of the year. The disadvantages are that all the energy saved is then significantly eroded due to having to supply large quantities of air to the data hall, against significant internal resistance.

  1. Indirect Outside Air System

Example Manufacturer: Munters (Oasis)

Munters Section

Munters Oasis

These systems are similar to the direct outside air units, except as the name suggests, no outside air enters the room.

Room air is continually recirculated through the unit and cooled via outside air which is drawn over the Munters heat exchanger. To increase free cooling, Munters also spray the heat exchanger to increase heat loss.

Advantages

  • Utilises free cooling by using outside air, when low outside air temperatures.
  • Utilises free cooling by removal of latent heat when outside air wet bulb less than indoor wet bulb (likely not as efficient as the direct outside air system).
  • Air filters are circulating room air only, thus less dirt resulting in less fan energy
  • Reliance on modulating dampers for outside air control (and possible damper leakage) does not occur (no dampers).
  • No Issues with quality of outside air E.g. Brushfires

Disadvantages

Very large unit. Needs careful positioning. Significant air flows required for scavenged air/spill air fans.

Large fan power losses due to:

  •               Scarification/spill air fans
  •               Main supply air fan overcoming resistance of ductwork, heat               exchanger and the like
  • No outside air supply to the data hall. A supplementary unit is required for this.
  • High water usage, thus on site N+1 water supply required.
  • Room humidity control likely to be difficult, as no moisture is added to the room. Thus for low humidity control a supplementary AHU with active moisture input will be required.

In summary these units are, again, designed to maximise the new ASHRAE acceptable conditions in a data hall. As these new conditions in simple terms are warmer, cooling energy is saved by 1, having to do less cooling, 2) ability to use free cooling for a large proportion of the year. The disadvantages are that all the energy saved is then significantly eroded due to having to supply large quantities of air to the data hall, against significant internal resistance. These units are theoretically less efficient that the direct outside air systems (heat exchanger efficiency loss), but counteract this loss with less fan power, due to filtration reduction, damper leakage etc.

Improvements – Direct and Indirect Systems

The direct and indirect systems are fairly new and are step, as a result of the Ashrae conditions, to provide a neat packaged air conditioning system. Fit and forget.

Improvement Ideas

Consider using water cooled chillers for supply of chilled water to the cooling coils. With modern chillers it is likely that much better COP’s can be achieved than with the manufacturers DX systems. Further modern chillers can be supplied with free cooling systems using a heat exchanger and the outside air to pre cool the chilled water prior to compressor operation. In addition the chiller refrigerant can be air cooled, without the compressor, with low outside air temperatures.

During modulating outside air mode, with the direct outside air systems, consider passive one way flow relief dampers in the walls of the data hall. This will avoid the fan power associated with dragging the entire air backup the AHU only to be discharged by mechanical fans to the atmosphere.

Consider a passive thermal floor with an underfloor supply air system. The slab is sitting atop earth at say 17 oC. With the correct concrete selection heat loss to the ground can be encouraged, thus providing free cooling.

Consider and active thermal slab. Here chilled water pipes are embedded during the slab pore. Chilled water is delivered to the slab to cool it. This provides a significant amount of cooling and reduces the amount of air being delivered by the fans.

  1. Enhanced CRAC unit System

Example Manufacturer: Liebert

DSE unit DSE with free cooling

These systems, in simple terms are packaged DX split systems.

Their main benefit, on the energy front, is that they are located in the room (or close to) and thus have small fans to deliver the air to and from the room. This feature alone greatly assists this technology as compared to the large fan power associated with the Direct and Indirect air systems, discussed above.

Refrigerant is a great way to transfer heat, with refrigerants able to transport per unit volume circa twice as much energy as water and circa 40 times as much heat as air. Thus keeping air and to a degree water movement out of the ‘picture’ saves a lot of energy.

The ‘enhancement’ is by getting some free cooling of the refrigerant from the outside air, without compressor power. Liebert call it the “EconoPhase economizer”.

DX (Direct Expansion) cooling uses a refrigerant to take heat out of the room air and then get rid of it to the outside air. The refrigerant is maintained, in simple terms, in two states; a vapour and a liquid.

The refrigerant in liquid phase picks up the room heat, adding energy, which causes the refrigerant to expand into a gas and a higher pressure, this gas then has the heat removed by blowing air over it (within a finned tube) and the refrigerant reverts back to a liquid. Assisting this process is a compressor which provides the compression of the refrigerant to make it easy for heat to be picked up from the room air and an expansion valve with creates a pressure change allowing for control to the liquid state.

The Liebert ‘EconoPhase economizer’ is basically a set of tubes, with a liquid refrigerant pump, that pumps the liquid refrigerant back to the indoor unit, by passing in effect the compressor. The colder the outside air, thus the colder the refrigerant liquid is, the less compressor work required, thus less power consumed as the liquid refrigerant pump uses a lot less energy than a compressor.

Refer to Liebert Technical Manual, for detailed description of operation: http://www.emersonnetworkpower.com/documentation/en-US/Products/PrecisionCooling/LargeRoomCooling/Documents/SL-18920.pdf

Advantages

  • Known technology
  • Avoids massive air flow movement with associated fan power costs
  • Can be dual cooled (chilled water coils and DX coils)
  • No direct outside air
  • Close control (temperature and humidity control)
  • Low spatial requirements (when combined with a chilled water system – removes dry air cooler).
  • No water consumption

Disadvantages

On the assumption that the quoted PUE’s are correct, there are few disadvantages with this system.

  • Low fan static ability
  • Need for outside air via separate unit.

Improvements – Enhanced CRAC unit System

Improvement Ideas

Where dry air coolers are used for heat rejection, consider a micro mist spray to increase the heat transfer process.

Client Review

Before considering new air conditioning for your data hall the following should be considered:

Undertake report to verify the best cooling solution. The report should consider:

Systems

  • Direct Air Systems
  • Indirect Air Systems
  • CRAC DX Splits
  • Chilled Water vs DX

Equipment

Review of major plant items and their efficiencies

  • Fan Motor Efficiency
  • VAV fans and turn down ability
  • Compressor size and numbers and are they infinitely controllable to match cooling load.
  • Damper Air Tightness
  • Duel circuit cooling system or not.

Passive Free Cooling

  • Slab Cooling
  • Air Intake to avoid SOL air Temperatures
  • Passive Spill Air
  • Location of plant to avoid supply and return air ductwork air flow pressure loss
  • Roof, wall thermal loads (make zero).
  • Air tight (make air tight)

Water

  • Water usage and water costs

Spatial

  • Size and weight of plant
  • Requirements to avoid heat short circuiting
  • Chillers to avoid large DX heat rejection plant

Reliability

  • Proof of reliability
  • Proof of failure rates
  • Proof of maintenance ability

Risks

  • Water loss
  • Damper Leakage
  • Bush Fires
  • Refrigerant loss
  • Water Treatment
  • Filter cleaning
  • Heat Exchanger Cleaning

Controls

  • Fully packaged controls with high level interface
  • No controls (relies on separate BMS)
  • Open protocol language, match your requirements

Installation

  • Time to install
  • Ease of replacement
  • Spare parts

Life Cycle Costing

  • Capital Cost
  • Energy Consumption
  • Verified efficiency data for your location
  • Replacement cost
  • Maintenance costs

Practicalities

So you have spent all this money, got a ‘you bute’ data hall full of air conditioning and you’re not happy. Why? Some of the items below could be why:

My data hall is too hot: This could be down to many things (wrong heat loads, fabric heat gains not accounted for, incorrect unit sizing and so on). Discounting all of these it is likely you have not got hot isle containment, thus the return air is getting up to say 39 oC and thus so is your entire data hall. To solve this retrospectively you will need to install hot isle containment. If this is not possible you will need to throw away energy efficiency and supply air at say 18 oC.

I have to run all my air handling units to get the air flow out of the floor grilles: Another classic. This can be down to many reasons (leaking floor tiles, leaking supply air system and floor void, incorrect fan selection, floor air supply flow patterns and turbulence). Discounting all of these it’s likely that you have designed your data hall for a future full occupancy scenario and you have one rack sitting in the room. So the cooling load only equates to a partial unit. Operation of this unit alone will not create enough pressure in the floor plenum to get the required air out of the required floor grilles. There are air flow management solutions (not discussed in this post).

I lose all my air conditioning on a power interruption: You probably forgot to think about a few things, including; auto change over switching on each AHU or a whole MSSB (Mechanical Services Switch Board) or the AHU controls, on loss of power and then on power resumption, don’t know what to do and require a manual restart.

My air conditioning PUE doesn’t seem right: This is important as you have ‘spent up’ significantly having a report written to select your air conditioning and the manufacturers have quoted you all sorts of amazing figures. So, get the electricity and water consumption to each AHU monitored and logged against IT load. Also monitor Stevenson screen air temperature and humidity as well as actual air intake conditions.

Sign up your supplier for a performance guarantee. Make them responsible for ensuring their AC units are working correctly, with sensors calibrated, VSD fans ramping up and down correctly etc.

Commissioning & Monitoring

I’ve ended this post with commissioning and monitoring. All the above can be compromised by lack of commissioning an don going monitoring. It is strongly recommend a commissioning consultant is brought on board to ensure required commissioning is undertaken and proven.

The consultant would liaise between the designer, the equipment manufacturers, the installers and client to set up a required commissioning plan and ensure that each commissioning test is undertaken.

Continuously monitoring of the energy used by the racks and the air conditioning energy, together with the rack supply air temperature is vital. Depending on the equipment in each rack, more energy may be consumed by the racks due to ASHRAE high air on temperatures (27 oC). Typically this is due to the equipment fans operation at higher speeds and for longer. So all that thought and design and free cooling calculations by using warmer supply air temperatures to get greater efficiencies from the cooling system may in reality not be achieved. It is recommended cooler supply air temperatures are used in the first instance (say 24 oC) and a slow temperature increase undertaken, with energy verification at each step rise in supply air temperature occur. If the power usage goes down, try another supply air temperature increase and so on.

Also monitor your equipment failure rate at each temperature increase. Equipment failure increases with temperature.

Further information provided on request.

Author: Jorgen Knox

Jorgen Knox PIC

Original Post Date: 19/01/2015

Contact: e: jorgenk@knoxadv.com.au, t: 02 800 33 100, w: KAE, LI

Blog: https://engineeringbyjorgen.wordpress.com/