Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts

Company ProprietaryKarl Kortkamp, SE, SGI

New Mexico Computing Applications Center(NMCAC)

System Diagrams&

Physical Layouts


New Mexico Computing Applications Center HPC for New Mexico


Overall Top-level Diagram

10 Gig-E

EncantoAltix ICE System

14,336 Compute Cores1,792 Nodes

28 Racks

UNMAltix ICE Exemplar176 Compute Cores

22 Nodes1 Rack

NMSUAltix ICE Exemplar176 Compute Cores

22 Nodes1 Rack

New Mexico TechAltix ICE Exemplar176 Compute Cores

22 Nodes1 Rack


Main System “Encanto” Overview

Major Components

• 172 TFLOPS Altix ICE 8200 Cluster

• 1792 Nodes, 3,584 Sockets, 14,336 Cores

• 28.7 TB memory (2GB/core)

• 172 TB Parallel File Subsystem (Lustre)

• 20 OSS Service Nodes

• 20 TB NAS & Archive Storage Subsystem

• 1 System Admin Controller

• 28 Rack Leader Controllers

• 1 Batch Service Node

• 5 Login Service Nodes

• Lambda Rail Connectivity

• 36 Racks – 28 Compute – 5 Storage – 3 Support


Encanto Top-level Diagram

33 IB Lines

1Lamba Rail

Fiber

120

Lambda RailGateway

5

Login ServiceNodes

IB DDRGig-E 10 Gig-E

Login ServiceNodesLogin Service

NodesLogin ServiceNodesLogin Service

Nodes

Qty 5

11System Admin

ControllerBatchNode

20

Lustre-Based ParallelStorage Sub-system

172 TB Usable

5

2

4

NAS CUBE / ArchiveStorage Sub-systemNAS: 20 TB Usable

Arch: 40TB Uncompressed

10

Altix ICE 8200 Cluster14,336 Compute Cores

1,792 Nodes28 Racks


NMCAC Qwest Fiber Connections to New Mexico “GigaPop”

505 MarquetteAlbuquerque, NM

“Lambda Rail”Point of Presence

4100 Sara RoadRio Rancho, NM

“Encanto”Supercomputer

Existing National Lamba RailFiber Ring


1 3

Encanto Lambda-Rail Router DiagramLamba Rail

Fiber


10

Future Expansion

21

21

21

21

21

Gateway/Router Cisco 6509 10-GigE Router

5

Login Service Node

Login Service Node

Login Service Node

Login Service Node

Login Service Node


1,792 Nodes28 Racks

5

1

1

1

1

1


Login Node

Login Node

Login Node

Login Node

Login Node

NMCAC Switch• Owned by NMCAC• Router between Fibre & Encanto• Redundant AC Power • Routing module to 5 Login Nodes• Load-leveling Module

NMCAC Switch• Owned by NMCAC• Redundant DC Power

Cisco

6509 Sw

itch

Qw

es

t Ge

om

ax

Qw

es

t G

eo

ma

x

Cis

co 6

509

Sw

itch

Cis

co 6

513

Sw

itch

UNM Switch• Owned by UNM• Connected to NLR• Connects to NMSU• Connects to NMTECH• Redundant DC Power

UNM

NMTECH

NMSU

NationalLamba Rail

Encanto External Network Diagram

505 Marquette Site 4100 Sara Road Site


…

Encanto Lustre-Based Storage System

10 Lustre Storage Units

Lustre Storage Unit

1

1

20

SASIB DDRGig-EHeartbeat FC4


1,792 Nodes28 Racks

Meta-data Servers

IS220

6015B

6015B

IS4500OST

XE310 Node

OSS1

OSS1

XE310 Node

IS4500OST

XE310 Node

OSS1

OSS1

XE310 Node

IS4500OST

XE310 Node

OSS1

OSS1

XE310 Node

IS4500OST

XE310 Node

OSS1

OSS1

XE310 Node

Place OSS XE310 servers on topof the following ICE racks

(5, 6, 7, 8, 9, 10, 11, 12, 24, 25)


Object StorageServers

Encanto Lustre Storage Unit

IS4500Dual-controller 2GB Cache

FC FC

1 1

Object StorageTargets

12x ( 4 + 1 ) LUNs + 4 Hot SparesAcross 4 FC Loops

FC RAIDDrive Tray

Loop

FC RAIDDrive Tray

Loop

FC RAIDDrive Tray

Loop

FC RAIDDrive Tray

Loop

Altix XE 310 Node2x3 GHz quad-core

16GB memory

FC

1

IB

Altix XE 310 Node2x3 GHz quad-core

16GB memory

FC

1

IB

1 1

64x 400GB 15K rpmFibre Channel drives


OSSs

Lustre Storage Cabling Details

OSTs

IS4500Dual-controller

FC FC

FC RAIDDrive Tray

FC4

FC RAIDDrive Tray

FC4

FC RAIDDrive Tray

FC4

FC RAIDDrive Tray

FC4

Altix XE 310Node

FC

1

IB

Altix XE 310Node

FC

IS4500Dual-controller

FC FC

FC RAIDDrive Tray

FC4

FC RAIDDrive Tray

FC4

FC RAIDDrive Tray

FC4

FC RAIDDrive Tray

FC4

Altix XE 310Node

FC

Altix XE 310Node

FC

1

IB

1

IB

1

IB

IB DDRHeartbeat FC4Gig-E


Altix 450

Combined NAS & Archive Storage Subsystem

Altix 450 Blade EnclosureAtempo Archive Server

SL500Dual-LT03 Tape Drives

FC FC

FC FC

Control Line Data Lines

TapeCartridgeTape

CartridgeTapeCartridgeTape


Cartridge

TapeCartridgeTape



Cartridge

TapeCartridgeTape



Cartridge

TapeCartridgeTape



Cartridge

Altix 450 Blade EnclosureNAS Head Node

IS4000RAID-controller

FC FC

FC FC

4

IB

NUMAlink

1 2 11

SATA RAIDDrive Tray

FC

SATA RAIDDrive Tray

FC

SATA RAIDDrive Tray

FC


Altix ICERack014

(Front)

Altix ICERack013

(Front)

Altix ICERack015

(Front)

Altix ICERack016

(Front)

Altix ICERack006

(Front)

Altix ICERack005

(Front)

Altix ICERack007

Altix ICERack008

(Front)

Altix ICERack010

(Front)

Altix ICERack009

Altix ICERack011

(Front)

Altix ICERack012

(Front)

(Front)

ArchiveRack

{Front)028

Altix ICERack

{Front)027

Altix ICERack

{Front)026

Altix ICERack

{Front)025

Altix ICERack

{Front)020

Altix ICERack

{Front)019

Altix ICERack

{Front)018

Altix ICERack

{Front)017

Altix ICERack

{Front)024

Altix ICERack

{Front)023

Altix ICERack

{Front)022

Altix ICERack

{Front)021

Altix ICERack

(Front)Storage

Rack001

(Front)Storage

Rack002

(Front)Storage

Rack003

(Front)Storage

Rack004

(Front)Storage

Rack005

Encanto Floor Plan

{Front)SWITCH

Rack

Altix ICERack002

(Front)

Altix ICERack001

(Front)

Altix ICERack003

(Front)

Altix ICERack004

(Front)

ADMINRack

(Front)

Lustre IB connections Chilled Water & CondensateIndicates overhead cable trellis

XE310

XE310

XE310 XE310 XE310XE310XE310

XE310

XE310XE310


Encanto Compute Rack (28)

Rack Leader Ctrl – 6015B

Lustre OSS I/O Service Nodes(XE310 servers) are placed on top of the following ICE racks

(5, 6, 7, 8, 9, 10, 11, 12, 24, 25)

Altix ICE 8200 Blade Enclosure (16 blades)




Altix ICERack

(Front)


Encanto Admin Rack

Sys Admin Ctrl – 6015BLogin Node – 6015B

Batch Node – 6015B

Login Node – 6015BLogin Node – 6015BLogin Node – 6015BLogin Node – 6015B

Administrator Console

Altix 450 Blade Enclosure

IS4000 Drive Tray (16 drives)


NAS CUBE & Archive

Main Cluster Service Nodes

IS4000 Controller

Altix 450 Blade Enclosure

ADMINRack

(Front)


MDS Node – 6015BMDS Node – 6015B

IS220 (12 drives)

Encanto Switch Rack

{Front)Support

ICERack

Sys Admin Ctrl (Test) – 6015BLogin Node (Test) – 6015B

Rack Leader Ctrl (Test) – 6015B

Administrator Console (Test)

Lustre Meta Data Server

Test ICE SystemAltix ICE 8200 Blade Enclosure

(Test) (16 blades)

Cisco 6509 10-GigE Switch


Encanto Luster Storage Rack (5)

IS4500 Controller









IS4500 Controller Lustre Storage Unit

StorageRack

(Front)

Lustre Storage Unit


Encanto Archive Library Cartridge Rack

(Front)

ArchiveRack

STK SL500

Cartridge Expansion Unit

Archive & BackupCartridge Storage Units


Major Components

• 2.1 TFLOPS Altix ICE 8200 Cluster

• 22 Nodes, 44 Sockets, 176 Cores

• 352 TB memory (2GB/core)

• 16 TB NAS Storage Subsystem

• 1 System Admin Controller

• 1 Rack Leader Controller

• 1 Login Service Node

• Lambda Rail Connectivity

• 1 Consolidated Rack

Exemplar System Overview


Exemplar Top-level Diagram

33 IB Lines

1Lamba Rail

Fiber

Lambda RailGateway

33


Login ServiceNode

System AdminController

4

Altix ICE 8200 Cluster176 Compute Cores

22 Nodes1 Rack

2

NASStorage Subsystem

16 TB Raw


Altix 450NAS Head Node

Exemplar NAS Storage Subsystem

4

IB

IS4000RAID-controller

FC FC

FC FC

11

SATA RAIDDrive Tray

FC

SATA RAIDDrive Tray

FC


Exemplar System Rack

Rack Leader Controller – 6015B


Administrator Console

Altix 450

NAS Storage CUBEIS4000 Controller and 16 Drives


Sys Admin Controller – 6015BLogin Node – 6015B

Drive Tray (16 Drives)

Altix ICE Cluster

Documents

Company Proprietary Karl Kortkamp, SE, SGI New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts