Data Centers

The฀following฀article฀was฀published฀in฀ASHRAE฀Journal,฀April฀2005.฀©฀Copyright฀2005฀American฀Society฀of฀Heating,฀Refrigerating฀and฀Air-
Conditioning฀Engineers,฀Inc.฀It฀is฀presented฀for฀educational฀purposes฀only.฀This฀article฀may฀not฀be฀copied฀and/or฀distributed฀electronically฀or฀
in฀paper฀form฀without฀permission฀of฀ASHRAE.

Staying On-line:
Data Center
Commissioning
By Mark Hydeman, P.E., Member ASHRAE; Reinhard Seidl, P.E., Member ASHRAE; and Charles Shalley, P.E.

D
ata฀centers฀require฀tight฀coordina- to฀think฀in฀terms฀of฀system฀failures,฀and฀ coordination฀required฀in฀the฀design฀of฀the฀
tion฀ between฀ the฀ trades,฀ special฀ more฀sophisticated฀techniques฀are฀required฀ electrical,฀mechanical฀and฀control฀systems฀
consideration฀ for฀ redundancy฀ to฀analyze฀critical฀links.฀In฀our฀previous฀ and฀ summarizes฀ the฀ experience฀ of฀ two฀
and฀ reliability,฀ and฀ carefully฀ scripted฀ example,฀consider฀the฀failure฀of฀an฀elec- firms฀(one฀mechanical฀and฀one฀electrical)฀
commissioning฀ of฀ systems.฀ Consider,฀ trical฀panel฀that฀serves฀two฀or฀all฀three฀of฀ that฀ have฀ collaborated฀ in฀ the฀ design฀ and฀
for฀ example,฀ the฀ issue฀ of฀ redundancy.฀ the฀ chillers.฀Although฀ the฀ chillers฀ were฀ commissioning฀of฀many฀data฀centers.
For฀ non-critical฀ commercial฀ facilities,฀ designed฀for฀N+1฀redundancy,฀a฀failure฀of฀
the฀ design฀ for฀ redundancy฀ typically฀ is฀ the฀electrical฀feed฀to฀more฀than฀one฀chiller฀ A Commissioning Process
approached฀on฀an฀equipment฀basis.฀ can฀shut฀down฀the฀data฀center.฀ This฀ article฀ cannot฀ comprehensively฀
For฀ example,฀ in฀ a฀ chilled฀ water฀ plant฀ As฀ compared฀ to฀ the฀ office฀ building,฀ cover฀all฀of฀the฀steps฀in฀commissioning฀a฀
with฀a฀1,000฀ton฀(3517฀kW)฀load,฀three฀500฀ the฀risk฀of฀failure฀in฀data฀centers฀is฀com- data฀center,฀but฀it฀highlights฀the฀four฀key฀
ton฀(1760฀kW)฀chillers฀may฀be฀provided฀to฀ pounded฀by฀several฀issues:฀1)฀with฀high฀ steps฀ and฀ provides฀ examples฀ from฀ real฀
meet฀the฀requirement฀of฀N+1฀redundancy.฀ density฀loads,฀the฀time฀for฀recovery฀from฀ projects฀for฀each.฀These฀steps฀are:
With฀this฀design,฀in฀theory,฀any฀chiller฀can฀ equipment฀ or฀ system฀ failure฀ is฀ much฀ •฀ Design฀review;
be฀pulled฀out฀of฀service฀and฀the฀plant฀can฀ shorter;฀ 2)฀ the฀ cost฀ of฀ failure฀ is฀ much฀ •฀ Preparation฀for฀functional฀testing;
continue฀ to฀ serve฀ the฀ design฀ load.฀With฀ higher;฀and,฀3)฀as฀loads฀may฀cluster฀within฀ About the Authors
an฀ office฀ building,฀ this฀ simplistic฀ view฀ any฀group฀of฀racks,฀redundancy฀must฀be฀
Mark Hydeman, P.E., is a principal at Taylor En-
of฀redundancy฀may฀be฀sufficient.฀Failure฀ provided฀in฀the฀“downstream”฀distribu- gineering in Alameda, Calif. He is a correspond-
to฀ meet฀ the฀ design฀ load฀ merely฀ causes฀ tion฀system฀as฀well฀as฀the฀central฀plant. ing member of ASHRAE Technical Committee
an฀inconvenience฀to฀tenants;฀it฀does฀not฀ This฀article฀presents฀a฀process฀that฀can฀be฀ 9.9, Mission Critical Facilities, Technology Spaces
and Electronic Equipment.Reinhard Seidl, P.E.,
compromise฀the฀core฀business. used฀to฀review฀and฀test฀the฀system฀design฀to฀
is a senior engineer at Taylor Engineering.
With฀ data฀ centers฀ and฀ other฀ mission฀ achieve฀a฀high฀degree฀of฀system฀reliability.฀ Charles Shalley, P.E., is a principal emeritus at
critical฀facilities,฀engineers฀are฀challenged฀ It฀also฀emphasizes฀the฀importance฀of฀close฀ the Engineering Enterprise in Alameda, Calif.

60 ASHRAE Journal ashrae.org April 2005

With data centers and other mission critical facili-
ties, engineers are challenged to think in terms of
system failures….
•฀ Implementation฀of฀the฀functional฀tests;฀ equipment฀ to฀ prevent฀ overloading฀ the฀ of฀possible฀fault฀propagation฀modes฀such฀
and generator฀when฀it฀comes฀on-line.฀ as฀ Monte฀ Carlo฀ (www.geocities.com/
•฀ Review฀of฀trends฀and฀tests. This฀ is฀ complicated฀ by฀ the฀ internal฀ CollegePark/Quad/2435/history.html)฀
Obviously,฀ these฀ steps฀ are฀ part฀ of฀ lockout฀ of฀ mechanical฀ cooling฀ equip- or฀Fault฀Tree฀Analysis฀(http://reliability.
an฀ iterative฀ process฀ that฀ must฀ react฀ to฀ ment,฀which฀might฀prevent฀or฀delay฀re- sandia.gov/Reliability/Fault_Tree_Anal-
problems฀uncovered฀in฀the฀field.฀In฀our฀ start฀of฀a฀chiller฀or฀direct฀expansion฀(DX)฀ ysis/fault_tree_analysis.html).฀
experience,฀no฀script฀can฀cover฀all฀of฀the฀ unit฀if฀it฀fails฀on฀loss฀of฀power.฀Often฀the฀ Most฀mechanical฀and฀electrical฀design-
contingences฀that฀include฀field฀installa- settings฀on฀these฀internal฀delays฀(or฀the฀ ers฀do฀not฀perform฀these฀kinds฀of฀analyses,฀
tion,฀control฀sequences,฀equipment฀inter- number฀of฀restarts)฀have฀to฀be฀adjusted฀ either฀because฀their฀fees฀in฀a฀competitive฀
nal฀controls฀and฀configuration,฀unit฀delays฀ for฀fail-safe฀operation. marketplace฀ do฀ not฀ allow฀ it,฀ or฀ because฀
and฀unanticipated฀issues฀uncovered฀in฀the฀ Many฀branches฀of฀engineering,฀includ- they฀do฀not฀have฀the฀required฀experience฀to฀
commissioning฀process.฀ ing฀ aerospace,฀ materials฀ science฀ and฀ make฀use฀of฀statistical฀tools.฀Another฀bar-
industrial,฀ provide฀ formal฀ training฀ in฀ rier฀is฀the฀lack฀of฀historical฀statistical฀data฀
Design Review the฀field฀of฀failure฀analysis.฀A฀brief฀Web฀ on฀equipment฀failure,฀which฀is฀a฀required฀
Whether฀acting฀as฀a฀third-party฀com- search฀on฀“failure฀analysis”฀or฀“reliability฀ input฀to฀some฀of฀these฀programs.
missioning฀ agent฀ or฀ as฀ the฀ engineers฀ analysis”฀results฀in฀a฀range฀of฀university฀ A฀simple฀failure฀analysis฀can฀be฀done฀by฀
of฀record,฀we฀have฀found฀a฀peer฀review฀ programs,฀ magazines,฀ consulting฀ firms฀ hand฀or฀implemented฀in฀a฀spreadsheet.฀The฀
(either฀internal฀or฀external)฀by฀someone฀ and฀software.฀Unfortunately฀in฀the฀HVAC฀ process฀needs฀to฀be฀thorough฀and฀methodi-
other฀ than฀ the฀ actual฀ designer฀ to฀ be฀ an฀ industry,฀we฀have฀little฀formal฀training,฀ cal.฀Using฀the฀example฀central฀plant฀shown฀
important฀part฀of฀the฀design฀process.฀ guidance฀or฀tools฀geared฀for฀this฀work.฀ in฀Figure฀1,฀we฀can฀see฀what฀happens฀if฀the฀
Having฀outside฀eyes฀take฀a฀fresh฀look฀ Chapter฀ 17,฀ “Data฀ Processing฀ and฀ control฀panel฀CP-1฀fails.฀CP-1฀controls฀the฀
at฀ the฀ design฀ often฀ uncovers฀ design฀ Electronic฀ Office฀Areas,”฀ of฀ the฀ 2003฀ primary฀chilled฀water฀pump฀PCHP-1.฀With฀
contingencies฀that฀the฀designer฀had฀not฀ ASHRAE฀ Handbook—HVAC฀Applica- failure฀of฀this฀panel฀the฀pump฀also฀fails.฀
considered.฀ In฀ addition฀ to฀ the฀ normal฀ tions฀simply฀states,฀ Chiller฀CH-1,฀in฀turn,฀relies฀on฀PCHP-1฀
items฀ under฀ a฀ peer฀ review,฀ a฀ review฀ of฀ “System฀reliability฀is฀so฀vital฀that฀the฀ and฀detects฀its฀failure฀as฀a฀loss฀of฀flow.฀
a฀ data฀ center฀ must฀ carefully฀ consider฀ potential฀cost฀of฀system฀failure฀may฀ Consequently,฀CH-1฀shuts฀down.฀
failure฀modes,฀operation฀at฀part฀load฀and฀ justify฀redundant฀systems,฀capacity,฀ Chillers฀1,฀2฀and฀3฀run฀lead/lag/standby฀
coordination฀of฀controls฀as฀discussed฀in฀ and/or฀ components.฀The฀ designer฀ with฀each฀chiller฀having฀50%฀of฀total฀ca-
the฀following฀paragraphs. should฀ identify฀ potential฀ points฀ of฀ pacity.฀As฀a฀result฀of฀CH-1฀shutting฀down,฀
Failure฀Modes:฀Both฀the฀designer฀and฀ failure฀ that฀ could฀ cause฀ the฀ system฀ the฀ remaining฀ chillers฀ CH-2฀ and฀ CH-3฀
reviewer฀need฀to฀consider฀what฀will฀happen฀ to฀interrupt฀critical฀data฀processing฀ will฀run฀and฀can฀still฀provide฀100%฀total฀
on฀failure฀of฀any฀piece฀of฀equipment฀or฀sup- applications฀and฀should฀provide฀re- capacity.฀The฀central฀plant,฀therefore,฀is฀
port฀system.฀Data฀center฀support฀systems฀ dundant฀or฀back-up฀systems.”฀ able฀ to฀ provide฀ the฀ design฀ capacity฀ on฀
are฀complex฀and฀interwoven:฀the฀mechani- Although฀ the฀ statement฀ is฀ true,฀ the฀ failure฀of฀control฀panel฀CP-1.฀
cal,฀electrical฀and฀control฀systems฀must฀be฀ means฀for฀analyzing฀the฀modes฀of฀failure฀ A฀ failure฀ of฀ CP-3,฀ on฀ the฀ other฀ hand,฀
reviewed฀as฀a฀whole฀since฀failure฀in฀any฀one฀ are฀not฀presented฀in฀the฀ASHRAE฀hand- would฀shut฀down฀both฀of฀the฀condenser฀wa-
may฀cascade฀to฀failures฀in฀the฀others.฀ books฀or฀guidelines.* ter฀pumps฀(CWP-1฀and฀CWP-2).฀The฀lack฀
For฀ instance,฀ loss฀ of฀ utility฀ power฀ in฀ A฀ rigorous฀ analysis฀ of฀ system฀ reli- of฀condenser฀water฀flow฀would฀prevent฀any฀
a฀data฀center฀will฀cause฀the฀emergency฀ ability฀ requires฀ statistical฀ modeling฀ of฀ of฀the฀three฀chillers฀from฀running,฀leaving฀
generator฀systems฀to฀come฀on-line.฀The฀ components฀and฀systems฀combined฀with฀ the฀plant฀with฀0%฀capacity.฀This฀clearly฀is฀
mechanical฀design฀must฀consider฀not฀only฀ a฀method฀of฀stepping฀through฀the฀array฀ a฀design฀flaw฀that฀needs฀to฀be฀fixed.
the฀ continuation฀ of฀ cooling฀ throughout฀ A฀ spreadsheet฀ can฀ be฀ automated฀ to฀
*ASHRAE TC 9.9, Mission Critical Facilities, Tech-
the฀ process฀ but฀ also฀ the฀ power-off฀ and฀ nology Spaces and Electronic Equipment, is mak- simulate฀failure฀of฀each฀piece฀of฀equip-
restart฀ sequences฀ of฀ the฀ mechanical฀ ing a major revision to this Handbook chapter. ment฀in฀turn.฀As฀each฀piece฀of฀equipment฀

April 2005 ASHRAE Journal 61

is฀failed฀in฀the฀automation,฀the฀software฀also฀should฀note฀the฀ quence฀requires฀the฀generators฀to฀communicate฀to฀the฀mechani-
failure฀of฀dependent฀equipment฀(e.g.,฀a฀chiller฀that฀relies฀on฀the฀ cal฀control฀system฀the฀loss฀of฀power฀and฀subsequent฀readiness฀of฀
operation฀of฀a฀condenser฀water฀pump),฀then฀test฀if฀the฀remaining฀ the฀generators฀to฀accept฀load.฀The฀mechanical฀and฀electrical฀bid฀
equipment฀has฀sufficient฀capacity฀to฀supply฀100%฀of฀the฀load.฀ documents฀must฀be฀reviewed฀to฀ensure฀that฀all฀interconnections฀
The฀simulation฀should฀flag฀any฀piece฀of฀equipment฀that฀results฀ and฀associated฀sequences฀are฀fully฀specified,฀communication฀
in฀less฀than฀100%฀design฀capacity฀when฀failed.฀This฀discovery฀ protocols฀are฀matched฀on฀each฀end,฀and฀that฀the฀scope฀of฀work฀
should฀lead฀to฀the฀redesign฀of฀the฀system฀(such฀as฀the฀condenser฀ for฀each฀contractor฀is฀clearly฀outlined.฀Even฀within฀a฀trade,฀care฀
water฀pumps฀in฀the฀earlier฀example).฀ must฀be฀taken฀to฀coordinate฀the฀passage฀of฀information฀between฀
While฀using฀common฀sense฀to฀analyze฀failures฀is฀possible,฀ equipment฀from฀various฀manufacturers.฀
some฀weak฀links฀may฀be฀overlooked.฀This฀simplistic฀analysis฀ Another฀ issue฀ to฀ consider฀ is฀ provision฀ of฀ uninterruptible฀
neither฀accounts฀for฀the฀timed฀lockout฀for฀chiller฀restart฀dis- power฀supply฀(UPS)฀power฀to฀the฀control฀system฀panels.฀This฀
cussed฀ previously,฀ nor฀ does฀ it฀ cover฀ the฀ event฀ of฀ coincident฀ generally฀can฀be฀provided฀at฀a฀low฀incremental฀cost฀because฀the฀
failures฀of฀more฀than฀one฀piece฀of฀equipment.฀ control฀panels฀are฀low฀wattage฀devices.฀Uninterrupted฀power฀to฀
Part-Load฀Operation:฀With฀data฀centers,฀consideration฀of฀ the฀control฀panels฀can฀greatly฀improve฀the฀stability฀of฀systems฀
how฀the฀equipment฀unloads฀is฀important.฀Although฀the฀computer฀ during฀the฀power฀down฀restart฀sequences.
equipment฀within฀the฀center฀tends฀to฀run฀close฀to฀full฀load,฀most฀
centers฀are฀phased฀in฀with฀racks฀being฀installed฀in฀groups฀over฀ Preparation for Functional Testing
time.฀Furthermore,฀many฀of฀the฀assumptions฀for฀the฀expected฀ Functional฀Test฀Scripts:฀Functional฀test฀scripts฀like฀the฀example฀
load฀density฀change฀shortly฀after฀the฀data฀center฀becomes฀op- shown฀in฀Figure฀2฀should฀be฀developed฀for฀each฀major฀piece฀of฀
erational,฀or฀sometimes฀during฀con- equipment,฀all฀control฀reset฀sequences,฀
struction.฀Because฀of฀this,฀most฀data฀ CP-1 CP-2 CP-3
all฀ equipment฀ staging,฀ and฀ all฀ an-
CT-2 CT-1
centers฀ are฀ built฀ either฀ with฀ future฀ ticipated฀failure฀scenarios.฀Note฀these฀
capacity฀ already฀ installed,฀ or฀ with฀ scripts฀supplement฀and฀don’t฀replace฀
provisions฀for฀future฀capacity฀to฀be฀ the฀ system฀ prefunctional฀ tests฀ (like฀
added฀in฀later฀construction฀phases.฀ CH-1 hydronic฀pressure฀testing฀of฀the฀chilled฀
During฀startup,฀systems฀usually฀are฀ PCHP-1 water฀ piping),฀ contractor฀ startup฀ or฀
running฀at฀part-load. control฀system฀commissioning฀(such฀
The฀design฀of฀the฀equipment฀and฀ CH-2 as฀the฀point-to-point฀verification฀and฀
systems฀supporting฀the฀data฀center฀ PCHP-2 testing฀of฀sequences).
must฀ take฀ into฀ account฀ part-load฀ ฀The฀example฀in฀Figure฀2฀is฀a฀test฀
CH-3
operation฀during฀the฀initial฀startup,฀ SCHP-1
script฀ for฀ secondary฀ chilled฀ water฀
PCHP-3
and฀ (as฀ appropriate)฀ uninterrupted฀ pumps.฀These฀pumps฀are฀controlled฀
operation฀ as฀ subsequence฀ phases฀ SCHP-2 with฀ variable฀ speed฀ drives฀ and฀ are฀
are฀ built฀ out.฀ For฀ initial฀ startup฀ of฀ designed฀ to฀ operate฀ in฀ parallel฀ for฀
the฀data฀center฀and฀each฀subsequent฀ CWP-1 better฀redundancy.
phase฀ of฀ build-out,฀ the฀ cooling฀ Load฀Banks:฀As฀previously฀dis-
systems฀must฀be฀evaluated฀for฀their฀ CWP-2 cussed,฀the฀data฀center฀generally฀will฀
ability฀to฀stay฀on-line฀and฀the฀provi- Figure฀1:฀Schematic฀of฀example฀control฀plant. not฀ be฀ fully฀ loaded฀ during฀ system฀
sion฀of฀redundancy.฀ startup฀and฀commissioning.฀In฀most฀
For฀pumps,฀fans฀and฀compressors฀(chillers฀and฀DX฀units)฀the฀ cases,฀the฀computer฀systems฀are฀installed฀into฀the฀racks฀during฀
review฀should฀ensure฀proper฀unloading฀controls฀are฀specified,฀and฀ the฀final฀phases฀of฀construction฀or฀just฀after฀commissioning,฀but฀
that฀the฀part-load฀operation฀is฀well฀away฀from฀the฀surge฀regions.฀To฀ provide฀little฀or฀no฀heat฀load฀for฀system฀testing.฀Load฀banks฀typi-
prevent฀temperature฀fluctuations฀and฀premature฀equipment฀failure,฀ cally฀are฀used฀to฀introduce฀heat฀loads฀and฀to฀allow฀simultaneous฀
compression฀cooling฀should฀operate฀at฀the฀lowest฀anticipated฀load฀ testing฀of฀both฀electrical฀and฀cooling฀systems.฀
levels฀without฀excessive฀cycling.฀ Renting฀ and฀ operating฀ load฀ banks฀ is฀ costly,฀ and฀ introduce฀
For฀cooling฀towers,฀the฀reviewer฀should฀ensure฀that฀the฀tower฀ risk—if฀cooling฀fails,฀load฀banks฀can฀quickly฀overheat฀a฀space฀
cells฀are฀designed฀for฀the฀highest฀and฀lowest฀anticipated฀flows฀ and฀potentially฀trigger฀sprinkler฀systems.฀This฀means฀that฀the฀
with฀proper฀coverage฀of฀the฀fill.฀Variable฀speed฀fans฀or฀tower฀ time฀ for฀ using฀ load฀ banks฀ and฀ associated฀ operators฀ must฀ be฀
bypass฀should฀be฀considered฀to฀keep฀the฀condenser฀water฀tem- minimized.฀For฀large฀projects,฀a฀sufficient฀number฀of฀load฀banks฀
perature฀stable฀under฀low฀loads. must฀be฀reserved฀in฀advance฀to฀ensure฀availability.
Controls:฀Successful฀control฀of฀data฀center฀systems฀requires฀ The฀mechanical,฀electrical฀and฀control฀systems฀must฀all฀be฀
careful฀coordination฀of฀the฀control฀design฀for฀the฀mechanical฀ ready฀to฀run฀when฀load฀banks฀arrive฀and฀the฀functional฀tests฀
and฀electrical฀equipment.฀A฀successful฀power฀down฀restart฀se- are฀to฀be฀run:฀

62 ASHRAE Journal ashrae.org April 2005

 A. Chilled Water System
1. Secondary chilled water pumps and associated VFDs:
a. Operation: Both pumps are designed to operate continuously. The control system shall modulate the speed of both pumps through the VFDs to control
the differential pressure at the end of the piping system to a constant pressure setpoint.
b. Failure tests:
(1) Disconnect the differential pressure transmitter.
(a) Confirm that both pumps continue to operate at a constant speed (speed shall be per the last signal given).
(b) Confirm that the BAS is in alarm.
(c) Confirm that the owner’s representative(s) have been paged.
(d) Confirm proper operation after the sensor has been reconnected.
(2) Disable the entire BAS (panels) in the building.
(a) Confirm that both pumps continue to operate at a constant speed (speed shall be per the last signal given).
(b) Confirm that the BAS is in alarm.
(c) Confirm that the owner’s representative(s) have been paged.
(d) Confirm proper operation re-enabling the BAS in the building.
(3) Disable the BAS control panel serving the pump VFDs.
(a) Confirm that both pumps continue to operate at a constant speed (speed shall be per the last signal given).
(b) Confirm that the BAS is in alarm.
(c) Confirm that the owner’s representative(s) have been paged.
(d) Confirm proper operation after re-enabling the control panel.

Figure฀2:฀Functional฀test฀script฀for฀secondary฀chilled฀water฀pumps.

•฀ The฀support฀systems,฀generators,฀UPS฀and฀power฀distribution฀ ฀ ฀ stable฀operation.฀This฀should฀be฀done฀for฀every฀piece฀of฀฀
systems฀must฀be฀complete.฀ ฀ ฀ equipment;
•฀ Control฀ systems฀ must฀ be฀ programmed฀ and฀ ready฀ to฀ trend฀ ฀ ·฀ Automatic฀transfer฀switch฀(ATS)฀and฀UPS฀operation;
equipment฀operation.฀ ฀ ·฀ Generator฀operation;฀and
•฀ Chillers฀and฀associated฀hydronic฀systems,฀as฀well฀as฀com- ฀ ·฀ Complete฀power-down฀and฀automatic฀restart.
puter฀ room฀ air฀ conditioner฀ (CRAC)฀ units,฀ package฀ units,฀ •฀ Startup,฀prefunctional฀testing฀and฀functional฀testing฀of฀each฀
DX฀systems฀and฀any฀other฀installed฀equipment฀must฀have฀ system.฀The฀mechanical฀and฀controls฀contractors฀need฀to฀care-
completed฀startup฀and฀be฀ready฀for฀operation.฀Chillers฀will฀ fully฀coordinate฀the฀testing฀of฀their฀systems฀with฀the฀electrical฀
not฀stay฀on-line฀without฀significant฀load,฀so฀they฀can฀be฀run฀ contractors฀to฀ensure฀uninterrupted฀power฀is฀available฀during฀
only฀once฀load฀banks฀are฀started. their฀tests.฀All฀three฀trades฀are฀typically฀testing฀their฀systems฀
Coordination฀of฀the฀Trades:฀Preparation฀for฀these฀tests฀is฀a฀ simultaneously.
multidisciplinary฀effort฀that฀requires฀input฀from฀both฀the฀design฀ •฀ Contingency฀ time฀ reserved฀ for฀ correction฀ of฀ errors฀ and฀
and฀ construction฀ teams.฀The฀ functional฀ tests฀ cover฀ not฀ only฀ retesting.
individual฀pieces฀of฀equipment฀but฀also฀the฀entire฀integrated฀
mechanical,฀ electrical฀ and฀ control฀ systems.฀ Designers฀ and฀ Execution of Functional Tests
contractors฀of฀these฀systems฀should฀provide฀input฀to฀the฀script฀ During฀the฀development฀of฀the฀functional฀test฀scripts฀a฀com-
well฀in฀advance฀of฀the฀scheduled฀commissioning฀dates.฀ missioning฀log฀should฀be฀created฀from฀the฀script฀that฀can฀be฀used฀
We฀ have฀ found฀ that฀ holding฀ regular฀ commissioning฀ meet- to฀record฀the฀event.฀An฀example฀log฀is฀shown฀in฀Figure฀3.฀This฀
ings฀for฀several฀months฀before฀actual฀commissioning฀serves฀to฀ log฀should฀record฀the฀events฀that฀occur฀during฀testing฀in฀parallel฀
raise฀awareness฀of฀critical฀issues฀in฀all฀of฀the฀team฀members.฀ with฀automatic฀monitoring฀and฀trend฀logs฀from฀control฀systems.฀
Engineers,฀owners,฀contractors฀and฀equipment฀vendors฀can฀use฀ Recorded฀information฀should฀include:฀date฀and฀time฀of฀testing,฀the฀
these฀meetings฀to฀agree฀on฀sequence฀of฀events,฀coordination฀ participants,฀and฀the฀expected฀and฀actual฀outcomes฀for฀each฀test.฀
of฀ schedules฀ and฀ responsibilities฀ of฀ key฀ players.฀ Scheduling฀ During฀testing,฀frequently฀unexpected฀results฀occur฀due฀to฀
considerations฀should฀include฀these฀milestones: system฀attributes฀that฀were฀overlooked฀during฀design฀or฀intro-
•฀ A฀date฀for฀power฀to฀all฀the฀mechanical฀and฀control฀systems; duced฀during฀construction.฀One฀such฀example฀is฀given฀in฀Figure฀
•฀ A฀final฀date฀for฀precommissioning฀of฀all฀systems,฀which฀includes฀ 3.฀Test฀1.b.3฀illustrates฀the฀test฀of฀dual฀secondary฀chilled฀water฀
the฀typical฀startup฀for฀each฀piece฀of฀equipment฀and฀testing฀of฀ pumps฀ with฀ variable฀ speed฀ drives.฀According฀ to฀ the฀ control฀
all฀control฀system฀wiring,฀I/O฀point฀status,฀programming,฀con- sequences,฀failing฀the฀control฀panel฀of฀the฀variable฀speed฀drives฀
figuration฀of฀alarms฀and฀configuration฀of฀trending.฀The฀trends฀ should฀result฀in฀maximum฀speed฀from฀both฀drives฀and฀a฀control฀
and฀alarms฀must฀be฀active฀during฀the฀functional฀tests. system฀alarm฀to฀the฀operator.฀
•฀ Arrival฀of฀load฀banks฀and฀duration฀of฀testing.฀Expected฀time฀ Note฀that฀in฀this฀case,฀redundancy฀was฀designed฀not฀by฀provid-
to฀perform฀all฀testing,฀including: ing฀separate฀control฀panels฀to฀both฀drives,฀but฀by฀allowing฀drives฀
฀ ·฀ Part-load฀and฀full-load฀(where฀possible)฀operation; to฀run฀to฀maximum฀speed฀after฀loss฀of฀the฀control฀signal.฀As฀the฀
฀ ·฀ Sequenced฀ failure฀ of฀ equipment,฀ restart฀ and฀ return฀ to฀฀ actual฀test฀log฀shows,฀both฀drives฀initially฀went฀to฀zero฀speed฀and฀

April 2005 ASHRAE Journal 63

 Project: Present:
Eng:
Date:
Test No. Pass/Fail Time Description Variables Result
1.b.3 Fail June 28 12:04 Disable the BAS control panel serving the pump VFDs.
(a) Confirm that both pumps continue at constant speed. speed= 0%
Fail (b) Confirm that the BAS is in alarm. msg= <none>
(c) Confirm owner’s representative(s) have been paged.
(d) Confirm operation after BAS has been reconnected.
Fail Note that฀
with eight different speeds, using three binary inputs at the drive. Two solutions exist:
(a) Never allow drives to be disabled during normal operation. Upon failure of panel, drive will no longer be enabled.
Program drive to run to 60 Hz in this case. Drives should never be intentionally disabled during normal usage (there is no
staging sequence), so the only disable command will occur on service. For service, the mechanic will switch local discon-
nect to “off” position. No drive disable through BAS is therefore ever required.
(b) Run a separate set of wires to each drive, using the second of three inputs on the drive, a failure condition can now
be sensed in addition to an intentional disable command. The drive will then go to zero speed upon disable, 27 Hz on 0%
speed input, and 60 Hz upon power failure at panel. This option is more elegant, but adds little effective functionality.
Fail June 29 14:20 After resetting of drive interface, retest. Drives now go into error condition. Need to reprogram.
Fail July 1 11:55 Drives remain in constant speed. However, after test (13:55), drives run to 0% speed before recovery. Need to adjust
control program. See also test 5.b.9.

Figure฀3:฀Example฀of฀commissioning฀log฀used฀to฀record฀events฀that฀occurred฀during฀testing.

had฀to฀be฀reprogrammed฀twice฀on฀separate฀days฀to฀provide฀the฀ the฀functional฀test.฀The฀loss฀of฀a฀chiller฀for฀example฀may฀have฀
desired฀response.฀Even฀then,฀the฀control฀system฀had฀to฀undergo฀ been฀caused฀by฀the฀loss฀of฀its฀chilled฀or฀condenser฀water฀pump.฀
some฀reprogramming฀to฀ensure฀the฀signal฀sent฀to฀the฀drives฀after฀ Since฀all฀three฀fail฀at฀once,฀the฀root฀cause฀may฀only฀become฀clear฀
restoration฀of฀the฀control฀panel฀did฀not฀disrupt฀operation.฀ through฀review฀of฀the฀control฀system฀trends฀and฀the฀error฀logs฀
This฀ type฀ of฀ unanticipated฀ failure฀ underlines฀ the฀ need฀ to฀ from฀the฀chiller฀control฀panel฀and฀the฀variable฀speed฀drives.
test฀every฀piece฀of฀critical฀equipment.฀The฀iterative฀process฀of฀ Figure฀4฀depicts฀the฀chilled฀water฀equipment฀status฀and฀data฀
testing,฀ noting฀ failures,฀ adjustment฀ and฀ retesting฀ is฀ required฀center฀temperatures฀during฀functional฀testing฀of฀the฀generators฀
for฀reliable฀operation฀of฀critical฀systems.฀Sufficient฀time฀in฀the฀ and฀ UPS฀ systems.฀This฀ figure฀ shows฀ two฀ stages฀ of฀ generator฀
schedule฀ needs฀ to฀ be฀ allocated฀ to฀ allow฀ correction฀ of฀ errors฀testing:฀the฀first฀at฀1,400฀kW฀and฀the฀second฀2,000฀kW฀(the฀full฀
uncovered฀in฀testing,฀and฀subsequent฀retesting.฀In฀some฀cases,฀ build-out฀capacity฀of฀the฀data฀center฀in฀this฀phase฀of฀construction).฀
equipment฀may฀be฀operated฀in฀override฀mode,฀allowing฀other฀ This฀data฀center฀had฀not฀been฀fully฀built฀out,฀so฀complete฀testing฀at฀
tests฀to฀proceed฀on฀schedule.฀In฀other฀cases,฀deficiencies฀may฀ full฀load฀was฀not฀possible.฀Some฀of฀the฀electrical฀and฀mechanical฀
need฀to฀be฀corrected฀on฀site฀before฀tests฀can฀proceed.฀ equipment฀would฀not฀be฀installed฀until฀later฀phases฀of฀build-out.฀
It฀is฀important฀to฀have฀the฀right฀people฀on฀hand฀for฀every฀ As฀a฀result,฀only฀the฀installed฀capacity฀could฀be฀tested.
test฀ so฀ they฀ can฀ witness฀ any฀ failures฀ and฀ participate฀ in฀ the฀ For฀each฀stage฀of฀testing,฀there฀were฀two฀power฀outages฀for฀the฀
remediation฀and฀retesting.฀This฀includes฀the฀technicians฀from฀ chillers:฀when฀the฀power฀failed฀until฀the฀generator฀was฀brought฀
the฀ associated฀ trades,฀ project฀ managers,฀ and฀ the฀ appropriate฀ on-line฀and฀again฀when฀the฀power฀was฀restored฀and฀the฀generator฀
owner’s฀representatives.฀This฀deployment฀needs฀to฀be฀carefully฀ was฀switched฀off.฀Figure฀4฀shows฀progressive฀improvement฀in฀
coordinated฀and฀scheduled฀well฀in฀advance.฀ chiller฀restart฀and฀temperature฀control฀through฀these฀four฀events.฀
For฀larger฀equipment฀such฀as฀chillers฀or฀UPS฀systems,฀factory฀ This฀is฀due฀to฀changes฀in฀the฀control฀system฀programming฀(timing฀
representatives฀should฀also฀be฀on฀site฀for฀the฀tests.฀You฀often฀dis- sequences)฀and฀tuning฀of฀the฀chiller฀restart฀delays.฀By฀the฀second฀
cover฀during฀the฀tests฀that฀reprogramming฀or฀reconfiguration฀of฀ tests,฀the฀chillers฀are฀barely฀off-line฀throughout฀the฀event.฀
a฀piece฀of฀equipment฀is฀required฀to฀pass฀the฀test.฀For฀example,฀in฀ In฀this฀example,฀the฀generator฀start฀sequence฀after฀a฀power฀
one฀project฀the฀control฀contractor฀had฀landed฀the฀remote฀chilled฀ outage฀could฀only฀be฀tested฀at฀part-load฀(only฀the฀first฀phase฀
water฀reset฀and฀demand฀limit฀signals฀on฀the฀chiller’s฀control฀panel,฀ of฀ data฀ center฀ build-out).฀The฀ expected฀ recovery฀ time฀ at฀
but฀the฀chiller฀had฀not฀been฀programmed฀to฀look฀at฀the฀remote฀ full฀ build-out฀ and฀ load฀ was฀ calculated฀ using฀ test฀ results฀ to฀
control฀inputs.฀Having฀the฀right฀people฀on฀site฀during฀the฀tests฀can฀ ensure฀that฀room฀temperatures฀will฀not฀rise฀excessively.฀The฀
significantly฀reduce฀the฀time฀for฀remediation฀and฀retesting. temperature฀rise฀in฀the฀data฀center,฀as฀a฀result฀of฀the฀tested฀
loads,฀is฀around฀6°F฀(3°C)฀per฀minute.฀From฀this,฀the฀expected฀
Review of Trends and Tests rise฀in฀temperature฀under฀full฀future฀load฀is฀calculated฀to฀be฀
After฀functional฀testing฀is฀complete,฀the฀control฀system฀trend฀ around฀8°F฀(4.5°C)฀per฀minute,฀resulting฀in฀a฀maximum฀al-
logs฀and฀error฀logs฀should฀be฀thoroughly฀analyzed.฀Many฀inter- lowable฀ downtime฀ of฀ two฀ to฀ three฀ minutes฀ for฀ mechanical฀
related฀actions฀are฀only฀clear฀through฀review฀of฀trends฀since฀only฀a฀ equipment.฀During฀the฀tests฀conducted,฀control฀systems฀and฀
small฀fraction฀of฀system฀settings฀can฀be฀observed฀real-time฀during฀ chiller฀ controls฀ were฀ tuned฀ to฀ produce฀ automatic฀ restart฀ of฀

64 ASHRAE Journal ashrae.org April 2005

 100 100

80 80
Max. Temp. Data Center

Avg. Temp. Data Center D

C
A B
60 60

Temperature (°F), Speed Percentage

sCHW Return, °F
Temperature, °F

sCHW Supply, °F
40 40

sCHW VFD Speed Percentage

20 20

CH-2 Off CH-2 Off

For 35 For 15
Min. Min.
Switch to Generator Switch to Generator
13:50 Using 1,400 kW 16:02 Using 2,000 kW
0 0
CH-2

CH-1
CH-1 Off CH-1 Off
For 8 Min. For 1 Min.

–20 –20
10:00 10:36 11:12 11:48 12:24 13:00 13:36 14:12 14:48 15:24 16:00 16:36 17:12
July 1, 2004

Figure฀4:฀Trend฀report฀of฀functional฀testing฀of฀the฀generators฀and฀UPS฀systems.

all฀ systems฀ within฀ two฀ minutes฀ after฀ power฀ failure,฀ with฀ a฀ •฀ Examination฀and฀testing฀of฀the฀system฀redundancy;
complete฀recovery฀to฀design฀conditions฀(pull-down)฀of฀about฀ •฀ Strong฀ collaboration฀ between฀ mechanical,฀ electrical฀ and฀
10฀minutes. controls฀trades;฀
Documenting฀these฀results,฀and฀providing฀clear฀illustrations฀ •฀ Clear฀and฀early฀communication฀with฀the฀construction฀team฀
of฀errors฀encountered,฀provides฀not฀just฀a฀working฀system,฀but฀ to฀ schedule฀ and฀ coordinate฀ the฀ commissioning฀ tests฀ with฀
also฀valuable฀information฀for฀the฀facilities฀staff฀in฀running฀and฀ equipment฀delivery,฀startup฀and฀control฀system฀installation;
maintaining฀the฀plant.฀It฀also฀provides฀better฀information฀than฀ •฀ Testing฀of฀all฀major฀components฀and฀control฀sequences฀to฀
as-builts฀for฀future฀expansions฀and฀additional฀work. ensure฀that฀the฀desired฀system฀reaction฀actually฀occurs;฀
•฀ Careful฀interpretation฀of฀testing฀results฀to฀determine฀which฀
Conclusion systems฀need฀to฀be฀redesigned฀or฀retuned;฀and
In฀this฀article,฀we฀presented฀a฀process฀for฀the฀commission- •฀ Clear฀documentation฀of฀the฀testing฀results฀with฀punch฀lists฀
ing฀ of฀ the฀ mechanical,฀ electrical฀ and฀ control฀ systems฀ of฀ a฀ of฀items฀that฀need฀to฀be฀fixed.
data฀center.฀Although฀the฀process฀is฀similar฀from฀project฀to฀ With฀ careful฀ preparation฀ and฀ diligent฀ teamwork,฀ you฀ can฀
project฀each฀design฀is฀unique฀and฀poses฀its฀own฀challenges.฀ create฀a฀project฀that฀meets฀or฀exceeds฀the฀needs฀of฀your฀clients.฀
As฀discussed฀in฀this฀article,฀the฀design฀and฀commissioning฀of฀ For฀those฀that฀are฀interested฀in฀learning฀more฀about฀data฀center฀
data฀centers฀requires฀a฀high฀level฀of฀coordination฀between฀the฀ design,฀ we฀ recommend฀ that฀ you฀ keep฀ abreast฀ of฀ASHRAE฀
trades,฀not฀just฀the฀mechanical฀and฀electrical฀engineers฀but฀ Technical฀Committee฀9.9,฀Mission฀Critical฀Facilities.฀They฀have฀
also฀the฀commissioning฀agent฀(if฀applicable),฀manufacturer’s฀ just฀completed฀two฀publications:฀“Thermal฀Guideline฀for฀Data฀
representatives฀ and฀ contractors.฀ Reliability฀ is฀ possible฀ but฀ Processing฀ Environments”฀ and฀ “Datacom฀ Equipment฀ Power฀
you฀have฀to฀be฀methodical฀in฀both฀design฀and฀testing.฀Keys฀ Trends฀and฀Cooling฀Applications”฀and฀are฀working฀on฀design฀
to฀achieving฀this฀goal฀include: guidelines฀for฀these฀facilities.

April 2005 ASHRAE Journal 65