Project Olympus 3U PCIe Expansion Server Deep Dive

Raymond Miles, VP Architecture, ZT SystemsMark D. Chubb, Director Architecture ZT Systems

Executive Track

ZT Systems OverviewProject Olympus Building BlocksZT 3U PCIe Expansion Server ProductsChassisFlexible Riser Card OptionsPower Distribution BoardSlide Rails

Agenda

ZT Systems Overview

A solution design and manufacturing partner with a record of success serving the world’s largest data center customers

Leading provider of optimized servers and storage servers for hyperscale data centers

Founded in 1994 HQ in Secaucus, NJ; global sites & capabilities Private company

Built for Hyperscale Maximum design flexibility Continuity of Supply Industry leading achievement to aggressive delivery SLAs Full range of flexible service and support options Broad range of component options

ZT OCP Engagement• History

• Engaged since 2012• Gold Member since 2014• Engineering projects• Solution Provider• Sponsored 8 summits

• Products• 2U Open Storage (2014)• 1U XPO200 (2017)• XPO200-3UN PCIe Expansion System (2019)

Project Olympus HW Building Blocks• Motherboards: Flexible expansion

• Power Supply: 1KW three-phase

• Universal PMDU: AC power cord adaption

• Rack: 19” EIA-310 standard

• Rack Manager: Restful API or Redfish via Ethernet

Project Olympus is an

OCP Accepted™

Specification

Intel® SKU

3U PCIe Expansion Server3 x Load Balanced

1kW PSUs

Power Distribution

Board(PDB)

6 x FHFL, Double-wide 300W x16 PCIe Cards

+1 x FHHL Single-wide 75W x16 PCIe Card

OR

Project Olympus Rack Fully Compatible

2.5”/3.5” SSD or HDD

6 x 60mm Dual Rotor Fans

(room for 8 Fans)

Supports Slide Rail Kit for smoother travel

PSUs Blind mate into Rack PMDU

12 x FHFL, Single-wide 75W x16 PCIe Cards

+1 x FHHL Single-wide 75W x16 PCIe Card

Supports Project Olympus motherboards

SERVER

AMD® SKU

5 x FHFL, Double-wide 300W x16 PCIe Cards

+1 x FHHL Single-wide 75W x16 PCIe Card

ZT Solution: 3U PCIe Expansion Server(AMD® EPYC™ Based; ZT-XPO200-3UA)

Feature Qty DescriptionChassis 1 3U, 19” EIA310-D Compliant supporting Project Olympus PMDU connections

Motherboard 1 Project Olympus 2-Slot AMD® EPYC™ Motherboard (OCP leveraged) from ZT SystemsProcessor 2 AMD® EPYC™ 7551 (Naples), 180W, 32C, 2GHzMemory 16

(32max)32GB DDR4, DR (1DPC), 2667 R-DIMM; Total System Memory: 512GB

PCIe Riser 3 & 5 2 2-Slot Active PCIe x16 Riser Card with 48-lane PCIe SwitchPCIe Riser 4 1 2-Slot Passive PCIe x32 Riser CardGPU Card 4 (5 max) AMD® MI25 FHFL, Double-Wide, 300W PCIe x16 CardNetwork Card 1 10G Single Port SFP+ PCIe2.0 x8 5GT/sHDD/SSD 5 (8 max) M.2 960GB NVMe SSD, PCIe x4 110mmM.2 Riser 1 (2 max) 2-Slot M.2 Riser (in PCIe Slot 1)Security 1 TPM2.0 SPI ModuleSystem Fans 6 60mmx56mm Dual Rotor FanPower Supply 3 Project Olympus 1020W 3-Phase, non-LES PSUPower Distribution 1 PDB and Cable Harnesses to support 12V Power to MB, Risers, GPUs, and System Fans

Items in Orange are being provided to OCP by ZT Systems

ZT Solution: 3U PCIe Expansion Server(Intel® Xeon®-SP Based; ZT-XPO200-3UN)

Feature Qty DescriptionChassis 1 3U, 19” EIA310-D Compliant supporting Project Olympus PMDU connections

Motherboard 1 Project Olympus 2-Slot Intel® Xeon®-SP MotherboardProcessor 2 Intel® Xeon® Platinum 8168 (Skylake), 205W, 24C, 2.7GHzMemory 12

(24max)32GB DDR4, DR (1DPC), 2667 R-DIMM; Total System Memory: 384GB

PCIe Riser 3 & 5 2 4-Slot Active PCIe x16 Riser Card with 96-lane PCIe SwitchPCIe Riser 4 1 5-Slot Active PCIe x16 Riser Card with 96-lane PCIe Switch & Sliver to Edge Conn x8 CablesGPU Card 12 Nvidia® P4 Low Profile, Single-Wide, 75W PCIe x16 CardNetwork Card 1 10G Single Port SFP+ PCIe2.0 x8 5GT/sHDD/SSD 1 (4 max) M.2 960GB NVMe SSD, PCIe x4 110mmSecurity 1 TPM2.0 SPI ModuleSystem Fans 6 60mmx56mm Dual Rotor FanPower Supply 3 Project Olympus 1020W 3-Phase, non-LES PSUPower Distribution 1 PDB and Cable Harnesses to support 12V Power to MB, Risers, GPUs, and System Fans

Items in Orange are being provided to OCP by ZT Systems

3U PCIe Expansion System Options

ZT Systems: ZT-XPO200-3UA ZT Systems: ZT-XPO200-3UN

®

ZT Systems in process of submitting these 3U Server Design Specifications to OCP Community!

System Block Diagram

PCIe x16 SLOT #3

PCIe x16 SLOT #5

PCIe x32 SLOT #4

CPU1 CPU0

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

CHA

CHB

CHC

CHD

CHE

CHF

CHG

CHH

PEX8747

M.2 Module #3

M.2 Module #4

Oculink x8

BMC

BMCFlash #1 32 MB

BMCFlash #2 32 MB

DDR4512 MB

Debug Connector

VGADE-15

RJ45 BCM546121GBe PHY

NCSIConnector

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

DIMMDIMM

CHA

CHB

CHC

CHD

CHE

CHF

CHG

CHH

GMI

GMI

GMI

GMI

BIOSFlash

32 MB

BIOSFlash

32 MB

TPM2.0

Internal USB 2.0

InternalUSB 2.0 Header

GL850GUSB2.0

HUB

SATA0 x1

SATA1 x1

SATA2 x1

SATA3 x1

BMC5781010G Controller

Front PanelSFP+

Front PanelSFP+

Front PanelUSB 3.0

Front PanelUSB 3.0

PCIE x8 SLOT #1

PCIE x8 SLOT #2

M.2 Module #1

M.2 Module #2

Mini SASHD x4 #1

Mini SASHD x4 #2

PCIe x16 SLOT #2

PCIe x16 SLOT #1

PCIe x16 SLOT #2

PCIe x16 SLOT #1

33MHz

480 Mb/s

480 Mb/s

480 Mb/s

480 M

b/s

USB2A

USB2B

LPC

SMBUS

PCIe

NCSI

SPI

DDR4

UART

RGMII

VGA

P2[15:0]

P0[15:0]

P1[15:0] P3[3:0]

P3[7:4]

P3[15:8]

USB1(3)

GMI1

GMI0

GMI3

GMI2

GMI3

GMI2

GMI1

GMI0

SPI

USB1(3)

LPCSMBUSP1[12]

P1[11:8]

P1[7:4]

P3[15:12]P3[11:8]P2[15:0]P0[15:8]

P0[7:0]

P1[0]P1[1]P1[2]P1[3]

P3[7:0]USB0(0)USB0(1)

10 Gb/s

Gen3 x8

10 Gb/s

5 Gb/s

5 Gb/s

6 Gb/s

6 Gb/s

6 Gb/s

6 Gb/s

Gen2 x1

Gen3 x8

Gen3 x8

Gen3 x16

Gen3 x4

Gen3 x4

Gen3 x4 or 4 SATA 6 Gb/s

Gen3 x4

Gen3 x16

Gen3 x16

Gen3 x16

Port 0

Port 8 Port 16

Gen3 x16

Gen3 x16

PEX8747

PCIe x16 SLOT #2

PCIe x16 SLOT #1

Port 0

Port 8 Port 16

Gen3 x16

Gen3 x16

2-Slot Active PCIe x16 Riser Card Layout 2 x PCIe x16 slots supporting (Slots #1-2, Bottom to Top):

• 2 x FHFL Double-Wide PCIe cards

• Requires PWRBRK_N (pin B30) signal for High Power PCIe Card throttling

Broadcom® 48-Lane PCIe Gen3 Switch (PEX8747)

• Switch Bifurcated into 2 x16 Ports (1 x16 Port to each x16 Slot)

• Switch is located on Backside as required by Chassis Thermals (Heatsink is required)

• Use of all 48-lanes and requires 15.5W

Differential Clock Buffer on board for additional Clock Fanout

I2C MUX used to avoid Address contention (see I2C block diagram)

8 Layer Stackup, Mid-Loss Material (0.062”)

Enclosed in a Riser Cage for easy Serviceability and Retention

12.68”

4.66”

PCIe x16 slot 2

PCIe x16 slot 1

Topside

2-Slot Passive PCIe x32 Riser Card Layout For Project Olympus AMD® EPYC™ MB Only (Middle Riser)

Uses 2 x HSEC8 connectors in-line for x32 Lanes

1 x PCIe x16 slot supporting (Slot #1, Bottom):• 1 x FHHL Single-Wide PCIe card

• Designated for the Network Card

1 x PCIe x16 slot supporting (Slot #2, Top):• 1 x FHFL Single-Wide PCIe Card OR

• 1 x FHFL Double-Wide PCIe Card

• All x16 PCIe lanes sourced from PCIe Riser Slot #4 (No Switch Required)

• Requires PWRBRK_N (pin B30) signal for High Power PCIe Card throttling

I2C MUX used to avoid Address contention (see I2C block diagram)

8 Layer Stackup, Mid-Loss Material (0.062”)

5.91”

3.62”PCIe x16 slot 2

PCIe x16 slot 1

2x2 Pwr

Other 3U Server PCIe Riser Card Options

TX: TOPRX: BOT

Port0

TX: L3/BOTRX: L8

Port4

Port12

Port20 Port16

Port8PEX8796

Slot 4

Slot 3

Slot 2

Slot 1 TX: L3/BOTRX: L8

TX: L3/BOTRX: L8

TX: L3/BOTRX: L8

TX: TOPRX: L3/BOT

JTAGPCIe x16

PCIe x8PCIe x8

TX: BOTRX: TOP/L4

Slot 2

Slot 1

TX: TOPRX: L4

TX: TOPRX: L4

TX: BOTRX: TOP/L6

Slot 3

Slot 2

Slot 1

TX: BOTRX: L6

TX: BOTRX: L6

TX: BOTRX: TOP

JTAG PCIe x16TX: BOTRX: TOP/L10

Port0(not used)

TX: L3/BOTRX: L8

Port4

Port12

Port20 Port16

Port8PEX8796

Slot 4

Slot 3

Slot 2

Slot 1

TX: L3/BOTRX: L8

TX: L3/BOTRX: L8

TX: L3/BOTRX: L8

TX: L3/BOTRX: TOP

Slot 5

JTAG PCIe x16

3-Slot Active PCIe x16 Riser Card

2-Slot Passive PCIe x16 Riser Card 4-Slot Active PCIe x16 Riser Card

5-Slot Active PCIe x16 Riser Card

Provides PCIe Expansion System Flexibility!

Middle PCIe Riser Card connectionsSliver Connector to PCIe x8 Edge Connector Cable• Provides additional PCIe Lane connections from unused PCIe

Slots

• 2 Cables provide PCIe x16 Lanes to Riser PCIe Slot or Riser PCIe Switch

• Supports Middle Riser Options:

• 2-Slot Passive x16 Riser Card

• 3-Slot Active x16 Riser Card

• 5-Slot Active x16 Riser Card

• Typically used on the Intel® Xeon®-SP System SKU

• ZT Partnered with Tyco Electronics® on design

PCIe I2C Block Diagram

BMCAST2520

I2C

PCA9548A0xE0

I2C_0

I2C_1

I2C_2

I2C_3

I2C_4

PCA9617

PCA9617

PCA9617

PCA9617

I2C_1

I2C_2

I2C_1

I2C_2

Riser 1, PCIe x8

Riser 2, PCIe x8

P3V3 P1V8_PCH_STBY

0

0

0

0

2.2K 2.2K

0

P3V3_STBY

4.7K

334.7K

I2C_5 0

P3V3_STBY

4.7K

0

P3V3_STBY

4.7K

I2C_6

P3V3_STBY

I2C_1

* 0 based

Bus Naming:1-based pin 0-based int

P3V3_STBY

NI4.7K

0I2C I2C_0

I2C_1I2C_2I2C_3

0

P3V3_STBY

ISOLATOR

ISOLATOR

4.7K

P3V3

4.7K

Slot2,PCIe x16

Slot1,PCIe x16

PCA9548A0xE4

I2C_4

I2C_5 PCA9617

4.7K

0ISOLATOR

PEX87470x70

CLK Buffer0xD8

P3V30

4.7K

Riser 3

ISOLATORI2C_6

I2C_7

P3V3 P1V8

4.7K 4.7K

P3V3_STBY P3V3_STBY

4.7K

0

0

0

0

Tem p sensor0xF8(Topside)

FRU0xAC

22 2222

Tem p sensor0x98(Backside)

P3V3_STBY

Voltag e Monitor0x90

4.7K

22

P3V3_STBY

NI4.7K

0I2C I2C_0

I2C_1I2C_2I2C_3

0

P3V3_STBY

ISOLATOR

ISOLATOR

4.7K

P3V3

4.7K

Slot2,PCIe x16

Slot1,PCIe x16

PCA9548A0xE4

I2C_4

I2C_5 PCA9617

4.7K

0ISOLATOR

PEX87470x70

CLK Buffer0xD8

P3V30

4.7K

Riser 5

ISOLATORI2C_6

I2C_7

P3V3 P1V8

4.7K 4.7K

P3V3_STBY P3V3_STBY

4.7K

0

0

0

0

Tem p sensor0xF8(Topside)

FRU0xAC

22 2222

Tem p sensor0x98(Backside)

P3V3_STBY

Voltag e Monitor0x90

4.7K

22

I2C_7 NA

Riser 1, PCIe x8

Riser 2, PCIe x8

I2C I2C_0I2C_1I2C_2I2C_3

0

P3V3_STBY

4.7K

ISOLATORISOLATOR

P3V3_SL OT2

4.7K

Slot2,PCIe x16Slot1,PCIe x16

PCA9548A0xE4

I2C_4I2C_5

Temp sensor(EMC1413)

0xF8

FRU(24C64)0xAC

0 0

I2C_6

I2C_74.7K

P3V3_STBY

P3V3_STBY

4.7K

0

P3V3_SL OT1

I2C

4.7K

Riser4(x32)

P3V3_STBY

NI4.7K

0

Notes:• M.2 Slots labeled #1-4 in MB Silkscreen.• M.2 I2C Addresses (VPD: 0xA6; Mgmt: 0x3A)

0

P1V8_P0

4.7K

0

M.2 S0M.2 S1

0

0

P1V8_P0

4.7K

M.2 S3M.2 S2

TCA9548A0xE2

* 0 based

I2CI2C_0

I2C_1

I2C_2

I2C_3

I2C_[7:4] NA

Slot 3.4Slot 3.2

Slot 4.2

Slot 4.1

Slot 5.4Slot 5.2

Physical Slot #’s

Physical Slot #’s

Physical Slot #’s

All PCIe Connectors connect to the BMC I2C for PCIe Card telemetry.

• Use the PCIe designated pins for I2C (B5 & B6).

• To avoid I2C Address contention between PCIe Cards, an I2C MUX is needed on each riser

• The design must add appropriate voltage translation/isolation between different I2C voltage domains.

PCIe Clock BlockDiagramDifferential Clock Buffers used on the Risers to

fanout additional PCIe Clocks.

The Clock Buffers are set to PLL Bypass Mode by default (shown to provide best margins)

Included resistor stuffing Options to change the PLLBandwidth Mode if needed to adjust Clock Jitter performance:

• PLL High BW Mode

• PLL Low BW Mode

• PLL Bypass Mode (default)

Power Distribution Board Layout

PDB supports single 12V Power Domain from 3 x Project Olympus PSUs

• Uses Passive Droop Sharing

• No V_Sense or I_Share signals used

Power Delivery Requirements

• Hot Swap Controller for Power Monitoring and Isolation

• 12V Current Sense Ckts used for each branch

• Supports Power to entire System (MB, Risers, PCIe Cards)

• Fan Power also comes from PDB (none from MB)

Provides passthrough of PMDU signals between MB and RM

6 Layer Stackup (0.093”), 2oz copper planes

Different power harnesses can be used for different High-Power PCIe Cards

Power Harness for 2 x AMD® MI25 GPUs

MB Signal

PSU1

PSU2

PSU3

PSU1 Signal

PSU2 Signal

PSU3 Signal

MB

PWR

LED Signal

Spare PWR

PCIe Cards

PCIe Cards

PCIe Cards

Riser5 PWR

Riser4 PWR(Hi PWR)

Riser4 PWR(2-Slot)

Riser3 PWR

Fan PWR

7.48”

4.49”

PDB Power DistributionBlock Diagram

Power Distribution Board MB

Hot Swap Controller

(ADM1278)

P12V

_PSU

_SW

PSU10xB0

P12V_PSU 2x12

Riser 3 (2-Slot)

2x6

Riser 5 (2-Slot)

2x6

Riser 4 (2-Slot x32)

2x2

Fan Power(6 x 60mm)

2x12

PSU20xB0 2x12

PSU30xB0 2x12

Rev 1.2

GPUTwo 2x4

GPUTwo 2x4

GPUTwo 2x4

GPUTwo 2x4

Note: Motherboard has its own Hot Swap Controller & Switch Ckt.

1U Fan Power& HDD

PWR Conn

PWR Conn

HSC/SW

2x4

1U Fan PowerConn Not Used

Sense

Sense

Sense

Sense

Hot Swap Controller

(ADM1172)

P12V

_STB

Y

Gen5 RiserP3V3_CB VR

P3V3_STBY VR

P3V3_RISER4SW

PDB Misc Signal Block Diagram

1U

Co

nn

PSU2(0xB0)

C2020 MB PDBBMC

PSU1(0xB0)

PSU1_PRESENT_N

PSU1_PS_ON_NPSU1_ALERT_N

BMC_NODE_ID_[5:0]

I2C_PSU1_SCL/SDA

SMBUS MUXPCA9546A

(0xE0)I2C_PSU2_SCL/SDA

PSU1_ALERT_N

PSU1_BLADE_EN_N

PMDU

PSU1_THROTTLEBLADE_EN#BLADE_THROTTLENODE_ID[5:0]

BLADE_PRESENT#

PMBUS

PSU_ALERT#PS_ON#

PSU1_THROTTLE_N

GPIOF4

GPIOF3

Throttle Logic

PSU1_BLADE_EN_N

BMC_NODE_ID_[5:0]GPIX[5:0]

GPIOJ0

CPLD 1K

MM2

SCL/SDA

SC0/SD0SC1/SD1

PMBUS

PSU_ALERT#PS_ON#

PSU2_PS_ON_NPSU2_ALERT_N

BLADE_EN#BLADE_THROTTLENODE_ID[5:0]

PSU2_PRESENT_NBLADE_PRESENT#

PSU3(0xB0)

I2C_PSU3_SCL/SDAPMBUS

PSU_ALERT#PS_ON#

PSU3_PS_ON_NPSU3_ALERT_N

BLADE_EN#BLADE_THROTTLENODE_ID[5:0]NC

PSU3_PRESENT_NBLADE_PRESENT#

Rev 1.4

I2C I/OExpanderPCA9555

(0x40) NC

IO00

SCL/SDA

IO01PSU2_BLADE_EN_N

PSU3_BLADE_EN_N

IO03IO12IO02IO13IO10IO11

PSU3_PS_ON_BUF_NPSU2_PS_ON_BUF_N

NC

NC

NI

Jum

per

FRU0xA6

TempSensor0xF8

NIPS

U L

ED C

able

PSU_LED0PSU_LED1

PSU_LED0PSU_LED1

PSU_LED0PSU_LED1

P3V3_STBY

PSU1_G_LED

PSU2_G_LED

PSU3_G_LED

PSU1_Y_LED

PSU2_Y_LED

PSU3_Y_LED

RESET#

PSU3_Y_LED

PSU3_G_LED

PSU2_Y_LEDPSU2_G_LED

PSU1_Y_LEDPSU1_G_LED

EN_HSC_IN

RST_I2C_MUX_N

2U

Co

nn

MM8MM3

CPLD

MB HSCSN1402039 (0x80)

PDB HSCADM1278 (0x26)

PSU2_PS_ON_NPL3C

HSC_P12V_FAULT_N

HSC_P12V_CSOUT Comparator Circuit

PDB_CURRENT_ALERT_N

SMB_BMC_MM2_SCL/SDA

SMB_BMC_MM8_SCL/SDASMB_BMC_MM3_SCL/SDA

PL5A

Note: Cabled LEDs mount directly to front panel of Chassis.

SC2/SD2SC3/SD3

Not UsedNot Used

IO14IO05

IO04

IO15IO06IO16

IO17IO07

RISER5_OC_CLEARRISER5_OC_ALERT_NFAN_OC_CLEARFAN_OC_ALERT_N

RISER4_OC_CLEARRISER4_OC_ALERT_N

RISER3_OC_CLEARRISER3_OC_ALERT_N

Sen

seSe

nse

Sen

seSe

nse

PSU3_BLADE_EN_R_NPSU2_BLADE_EN_R_N

NI

NI

PSU2_ALERT_NGPIOM5

OD

OD

OD

Provides passthrough of PMDU signals between MB and RM

3U Server Slide Rails

Supports Slide Rails for smoother travel during servicing

Due to the weight of C2310 Server (90lbs), standard Project Olympus T-Pin Slide Rails will not allow the Chassis to pull in and out smoothly.

Width of Chassis prevents thicker rail slides (only 67% pull-out)

Top cover is split into two sections (Front and Rear).

Front Top Cover section allows access to most of the System hardware, leaving only the System Fans, PDB and Power Cables under the Rear Top Cover without in-Rack access.

Using Slide Rails from King Slide®

Plans for OCP Recognition of Project Olympus 3U Server

3U Server Base Spec submitted by Microsoft® (Siamak Tavallaei) – Targeting OCP Acceptance by Mid 2019

3U AMD® EPYC™ Server Product Spec & Design Collateral submitted by ZT – Targeting OCP Acceptance by Q3 20193U Intel® Xeon®-SP Server Product Spec & Design Collateral submitted by ZT– Targeting OCP Acceptance by Q3 2019

Both Products to be Available on Open Compute Marketplace 2H 2019

Visit us at Booth A17Or www.ztsystems.com/ocp