[EcoStruxure™ Reference Design 109]
7392 kW, Tier III, NAM, Chilled Water,
Liquid-Cooled AI Clusters
Design Overview
Data Center IT Capacity
7392 kW Introduction
High-density AI clusters and liquid cooling bring new challenges to data
Target Availability center design. Schneider Electric’s data center reference designs help
Tier III shorten the planning process by providing validated, proven, and
documented data center physical infrastructure designs to address such
Annualized PUE at 100% Load challenges. This design focuses on the deployment of high-density AI
San Francisco, CA: 1.12 clusters, specifically NVIDIA’s GB200 NVL72 racks, in a single IT room. The
Dallas, TX: 1.15 IT room is purpose-built and optimized for three NVIDIA 1152 GPU DGX
SuperPOD GB200 clusters using Motivair liquid-to-liquid CDUs and fluid
Racks and Density coolers with adiabatic assist. Facility power and cooling design are
Total Racks: 96 optimized for capital cost, efficiency, and reliability.
Rack Power Density: Reference Design 109 includes information on four technical areas: facility
Networking racks up to 22 kW/rack power, facility cooling, IT space and lifecycle software. These areas
AI racks up to 132 kW/rack represent the integrated systems required to meet the design’s
specifications provided in this overview document.
Data Center Overall Space
37,774 ft2
Regional Voltage and Frequency
480V, 60Hz
About this Design
• IT space and power distribution
designed to accommodate AI clusters
with density up to 132 kW per rack
• Design optimized to support liquid-
cooled racks, with liquid-to-liquid
coolant distribution units (CDUs) and
fluid coolers with adiabatic assist
• Chilled water systems optimized for
high water temperatures using Uniflair
FWCV fan walls and Uniflair XRAF air-
cooled packaged chillers
• Redundant design for increased
availability and concurrent
maintainability
Public
[EcoStruxure™ Reference Design 109] 2
Facility Power
Facility Power Block Diagram The facility power system supplies power to all components within the data center.
In this concurrently maintainable electrical design, power to the IT rooms is
supplied through four 3 MW powertrains. The four powertrains provide 3+1
distributed redundant UPS power to the IT space, backed up by diesel generators.
Each powertrain consists of a 4000-amp QED-2 main switchboard feeding two
1500 kW Galaxy VX UPS with 5 minutes of runtime in parallel and a 4000-amp
QED-2 distribution section. At this loading, the Galaxy VX UPSs’ overload
capacity can manage NVIDIA’s NVL72 racks’ electrical design point (EDP) power
peak. Downstream, these powertrains feed Hyper 1.3 MVA 480V:415V/240V
power distribution units (PDUs), that power the IT racks with 3+1 redundancy.
Separately, two 1.5 MW powertrains feed the fan walls, chillers, and fluid coolers
with 2N redundant power. They also feed a 200 kW Galaxy VL UPS that provides
critical power to the liquid-to-liquid coolant distribution units (CDUs) and facility
water system pumps.
The facility power system is designed to support integrated peripheral devices like
fire panels, access control systems, and environmental monitoring and control
devices. Power meters in the electrical path monitor power quality and allow for
predictive maintenance & diagnostics of the system. These meters also integrate
with EcoStruxureTM Power Monitoring Expert.
Every component in this design is built and tested to the applicable IEEE or UL
standards.
Further design details, such as dimensions, schematics, and equipment lists are
available in the engineering package.
Facility Power Attributes
Design Options Name Value Unit
This reference design can be modified as Total facility peak power (IT and cooling) 10,500 kW
follows without a significant effect on the
Total amps (IT main bus, each) 4000 A
design’s performance attributes:
Input voltage (IT main bus) 480 V
• Provision for load bank
• Change UPS battery type & runtime Switchboard kAIC (IT main bus) 65 kA
• Add/remove/change standby Generator redundancy (IT main bus) Distributed redundant
generators: IT power path 3+1
o Location & tank size IT space UPS capacity, per powertrain 3000 kW
IT space UPS redundancy Distributed redundant
IT space UPS runtime @ rated load 5 minutes
IT space UPS output voltage 480 V
Total amps (Facility cooling bus, each) 1600 A
Input voltage (Facility cooling bus) 480 V
Switchboard kAIC (Facility cooling bus) 35 kA
Generator redundancy (Facility cooling
2N
bus)
Facility cooling UPS capacity 200 kW
Facility cooling UPS redundancy 2N
Facility cooling UPS runtime @ rated
5 minutes
load
Public
Document Number RD109DS Revision 8
[EcoStruxure™ Reference Design 109] 3
Facility Cooling
Facility Cooling Block Diagrams
The facility cooling design features a dual path piping system optimized for data
center efficiency. A chilled water loop integrates Uniflair XRAF chillers, with free
cooling capabilities, to deliver 73°F chilled water to fan walls in N+1
configuration. This lower temperature water loop handles the air-cooling needs
of the data center. A separate high temperature water loop supplies 99°F water
to Motivair liquid-to-liquid CDUs used to cool NVIDIA’s GB200 NVL72 liquid
cooled racks. Outdoor heat rejection for this loop is supported by Guntner GFD
V-Shape Vario fluid coolers. With supply water temperatures approaching
100°F, fluid coolers become useful heat rejection units in a wide range of
climates. They can also help reduce energy and capex cost compared to
chillers. For ambient temperatures above 90°F, the fluid coolers use adiabatic
assist to provide sufficient cooling for the data center.
An integrated thermal storage system provides 5 minutes of continuous cooling,
in case of power outage, to allow the chillers and fluid coolers to restart. The
CDUs and facility pumps are on UPS power. More information on fan wall and
CDU cooling architecture is detailed in the IT room section of this document.
This design is instrumented to work with EcoStruxureTM IT Expert and AVEVA
Unified Operations Center.
Further design details such as dimensions, schematics, and equipment lists are
available in the engineering package.
Facility Cooling Attributes
Similar piping architecture for chilled water loop Name Value Unit
and fluid cooler loop 9051 (Dallas)
Total max cooling capacity kW
9369 (San Francisco)
Input voltage 480 V
Heat rejection medium Water
Chiller redundancy N+1
Fluid cooler with
Design Options Outdoor heat exchange adiabatic assist &
packaged chiller
Chiller CW supply temperature 73 °F
This reference design can be modified as
follows without a significant effect on the Chiller CW return temperature 91 °F
design’s performance attributes: Fluid cooler CW supply
99 °F
temperature
• Change storage tank size Fluid cooler CW return temperature 117 °F
• Integrate heat exchanger on FWS Combined* storage tank size 15,002 gallon
with additional Uniflair XRAF chillers Ride-through time 5 minutes
to enable 100% dry cooling Outdoor ambient temperature
11.1 to 110.5 °F
range
Economizer type Water-side
*Summation of both facility water systems
Public
Document Number RD109DS Revision 8
[EcoStruxure™ Reference Design 109] 4
IT Room
IT Room Diagrams The IT room features forty-eight 132 kW liquid cooled IT racks, modeled after
NVIDIA’s GB200 NVL72, with forty-eight 22 kW air-cooled networking racks
organized into three pods. Each pod consists of two rows of eight 132 kW IT
racks and eight 22 kW networking racks. The liquid cooled racks remove 87%
of the heat via liquid while 13% requires air. Each pod is deployed with
Prefabricated Modular EcoStruxureTM Pod Data Center to provide a 6 ft. wide
ducted hot aisle containment for proper airflow, busway and cabling support, and
TCS piping. Ducted hot aisles and a common ceiling plenum return hot air to
the fan walls for cooling,
Six Uniflair FWCV chilled water fan walls supply conditioned air to the IT room
in an N+1 configuration. Three Motivair MCDUs provide precise liquid cooling to
each pod with N+1 redundancy. Redundant piping systems across the IT room
provides an alternate path for chilled water in case of cooling equipment failure
or maintenance.
The 22 kW networking racks are NetShelter SX racks configured with 1+1 60 A
NetShelter Rack PDU Advanced rack-mount power distribution units (rPDUs).
The 132 kW AI racks are NetShelter Open Architecture Rack MGX configured
with 6+2 33 kW NetShelter Open Architecture Power Shelves. Each row is
powered by four Hyper Solutions High Density 1300 kVA PDUs and two Hyper
Solutions High Density 2x800 A RPPs, providing A-, B-, C-, and D-side power to
the row. The 132 kW AI racks have a pair of 60 A feeds coming from each of the
800 A panel in the RPPs, for a total of eight power feeds per rack from the RPPs
– one for each power shelf. The 22 kW networking racks have a pair of 60 A
feeds from two of the four 800 A RPPs for 2N redundant feeds. The Hyper
Solutions High Density 2x800A RPP includes PowerLogic HDPM for branch
circuit power monitoring, PowerLogic ION900T for metering each 800 A RPP
with transient capture, and PowerPacT B-Frame circuit breakers that can be
configured with shunt trip units for those feeding liquid cooled racks.
IT Room Attributes
Name Value Unit
IT load 7392 kW
Supply voltage to IT 415 V
Rack power feed redundancy Redundant
Number of 132 kW liquid-cooled racks 48 racks
Number of 22 kW networking racks 48 racks
IT floor space 5486 ft2
Design Options CRAC/CRAH type Fan wall
This reference design can be modified as CRAC/CRAH redundancy N+1
follows without a significant effect on the CRAC/CRAH supply air temperature 82 °F
design’s performance attributes: CW supply temperature 73 °F
CW return temperature 91 °F
• Use Uniflair FXCV fan walls Containment type Ducted hot aisle
• CRAHs can be selected instead of CDU type L2L
fan walls
CDU redundancy N+1
• Variations in AI cluster
CDU CW supply temperature 99 °F
configuration
CDU CW return temperature 117 °F
• Use SE ACDUs for liquid cooling
TCS loop supply temperature 104 °F
TCS loop return temperature 129 °F
Public
Document Number RD109DS Revision 8
[EcoStruxure™ Reference Design 109] 5
Lifecycle Software
High-density AI clusters push the limits of data center facility infrastructure, so it’s
critical to leverage advanced planning and operation tools to ensure safe and
reliable operations.
Planning & Design
Electrical Safety and Reliability: Due to the high amount of power supplied to an
AI cluster, design specifications such as available fault current, arc flash hazards
and breaker selectivity must be analyzed in the design phase. Applications like
Ecodial and ETAP simulate the electrical design and reduce the chance of costly
mistakes or even worse, injury.
Cooling: AI clusters are pushing the limits of what can be done with air-cooling.
Modeling the IT space with computational fluid dynamics (CFD) helps spot issues
including high pressure areas, rack recirculation, and hot spots. This is especially
true when retrofitting an existing data center with an AI cluster. Schneider
Electric’s EcoStruxureTM IT Design CFD can quickly model airflow, allowing rapid
iteration to find the best design and layout.
Operations
EcoStruxureTM is Schneider Electric’s open, interoperable, integrated Internet of
Things (IOT)-enabled system architecture and platform. It consists of three
layers: connected products, edge control, and applications, analytics, and
services.
EcoStruxureTM Data Center is a combination of three domains of EcoStruxureTM:
Power, Building, and IT. Each domain is focused on a subsystem of the data
center: power, cooling, and IT. These three domains combined will reduce risks,
increase efficiencies, and speed operations across the entire facility.
• EcoStruxureTM Power monitors power quality, generates alerts, while
protecting and controlling the electrical distribution the electrical
distribution system of the data center from the MV level to the LV level.
It uses any device for monitoring and alerting, uses predictive analytics
for increased safety, availability, and efficiency, while lowering
maintenance costs.
• EcoStruxureTM Building controls cooling effectively while driving
reliability, efficiency, and safety of building management, security, and
fire systems. It performs data analytics on assets, energy use, and
operational performance.
• EcoStruxureTM IT makes IT infrastructure more reliable and efficient
while simplifying management by offering complete visibility, alerting
Visit EcoStruxureTM for Data Center and modelling tools. It receives data, generates alerts, predictive
for more details. analytics, and system advice on any device to optimize availability and
efficiency in the IT space.
There are several options for supervisory visibility and control. AVEVA Unified
Operations Center can provide visibility at a site or across an entire enterprise.
Public
Document Number RD109DS Revision 8
[EcoStruxure™ Reference Design 109] 6
Design Attributes
OVERVIEW Value Unit
Target availability III Tier
1.12 San Francisco, CA
Annualized PUE at 100% load
1.15 Dallas, TX
Data center IT capacity 7392 kW
Data center overall space 37,774 ft2
Maximum rack power density 132 kW/rack
FACILITY POWER Value Unit
Total facility peak power (IT and cooling) 10500 kW
Total amps (IT main bus, each) 4000 A
Input voltage (IT main bus) 480 V
Switchboard kAIC 65 kA
Generator redundancy (IT main bus) Distributed redundant
IT Power path 3+1
IT space UPS capacity, per powertrain 3000 kW
IT space UPS redundancy Distributed redundant
IT space UPS runtime @ rated load 5 minutes
IT space UPS output voltage 480 V
Total amps (facility cooling bus, each) 1600 A
Input voltage (facility cooling bus) 480 V
Switchboard kAIC (facility cooling bus) 35 kA
Generator redundancy (facility cooling
2N
bus)
FACILITY COOLING Value Unit
Total max cooling capacity 9051 (Dallas), 9369 (San Francisco) kW
Input voltage 480 V
Heat rejection medium Water
Chiller redundancy N+1
Outdoor heat exchange Fluid cooler with adiabatic assist & packaged chiller
Chiller CW supply temperature 73 °F
Chiller CW return temperature 91 °F
Fluid cooler CW supply temperature 99 °F
Fluid cooler CW return temperature 117 °F
Combined* storage tank size 15,002 gallon
Ride-through time 5 minutes
Outdoor ambient temperature range 11.1 to 110.5 °F
Economizer type Water-side
*Summation of both facility water systems
Public
Document Number RD109DS Revision 8
[EcoStruxure™ Reference Design 109] 7
Design Attributes continued
IT SPACE Value Unit
IT load 7392 kW
Supply voltage to IT 415 V
Maximum rack power density 132 kW/rack
Number of racks 96 racks
IT floor space 5486 ft2
Rack power feed redundancy Redundant
CRAC/CRAH redundancy N+1
Containment type Ducted hot aisle
CDU Type L2L
CDU redundancy N+1
CRAC/CRAH supply air temperature 82 °F
CW supply temperature 73 °F
CW return temperature 91 °F
CDU CW supply temperature 99 °F
CDU CW return temperature 117 °F
TCS loop supply temperature 104 °F
TCS loop return supply temperature 129 °F
Public
Document Number RD109DS Revision 8
[EcoStruxure™ Reference Design 109] 8
Schneider Electric Life-Cycle Services
Life Cycle Services
Plan
1 Team of over 7,000 trained specialists covering every
phase and system in the data center
What are my options?
Install Standardized, documented, and validated methodology
How do I install and commission? 2 leveraging automation tools and repeatable processes
developed over 45 years
Operate
How do I operate and maintain?
3 Complete portfolio of services to solve your technical or
business challenge, simplify your life, and reduce costs
Optimize
How do I optimize?
Renew
How do I renew my solution?
Get more information for this design:
Engineering Package
Every reference design is built with technical documentation for engineers
and project managers. This includes engineering schematics (CAD, PDF),
floor layouts, equipment lists containing all the components used in the
design and 3D images showing real world illustrations of our reference
3D spatial views Floor layouts designs.
Documentation is available in multiple formats to suit the needs of both
engineers and managers working on data center projects. The engineering
package of this design can be downloaded here.
One-line schematics Bill of materials
Public
Click here to download the Engineering Package
Document Number RD109DS Revision 8
Email referencedesigns@[Link] for further assistance
