Introduction to MicroTCA - Pages - … 4-5, 2012 4 PICMG-ese Term Definition PICMG PCI Industrial Computer Manufacturers Group, 250 corporations ATCA Advanced Telecommunications Computing

June 4-5, 2012 1

Introduction to MicroTCA

Ray Larsen

SLAC MicroTCA Review

June 4-5, 2012

SLAC MicroTCA Standards Review

June 4-5, 2012 2

Outline

I. MicroTCA for Physics Basics

II. Physics System Requirements

III. COTS & Lab Developments

IV. xTCA Market Growth Projections

V. Summary Conclusions


June 4-5, 2012 3

I. MicroTCA Basics


June 4-5, 2012 4

PICMG-ese Term Definition

PICMG PCI Industrial Computer Manufacturers Group, 250 corporations

ATCA Advanced Telecommunications Computing Architecture large board

Carrier ATCA or µTCA board that supports smaller standard board

Shelf Crate, ATCA (large) or µTCA (small)

RTM/µRTM Rear/Micro Rear Transition Module

AMC Advanced Mezzanine Card mounting on ATCA Carrier, µTCA shelf

Micro/µTCA Crate designed to support AMCs directly

MCH MicroTCA Carrier Hub switch module for µTCA shelf

PU, CU Power Unit (Module), Cooling Unit (fan or fan tray)

IPMI Intelligent Platform Management Interface

Shelf Mgr Shelf board hosting IPMI controller (BMC, MMC controllers)

Wide, High High (vertical module height), Wide (front panel width)

xTCA ACTA and /or MicroTCA standard platforms


June 4-5, 2012 5

MicroTCA Basics

• First a word about ATCA – ATCA board, shelf is first modular computer architecture with

completely serial multi-Gbps backplane, dual star or mesh

– Board designed with standard Mezzanine card (AMC) for

ease in upgrading as technology evolves (Moore’s Law)

– ATCA managed system (IPMI), standard diagnostics detect

problems at ATCA, AMC levels; take remedial action to evade

machine/system interruption

– Target Availability for ATCA shelf (crate) is 0.99999 or greater

– In Telecom automated load switchover is common in systems

of many identical processor blades

• Less practical in accelerator, but less required for subsystems with high

redundancy


June 4-5, 2012 6 6


ATCA Shelf Carrier

ATCA Carrier w/ 4 AMCs ATCA Dual Star 14-Slot Shelf

June 4-5, 2012 7 7


ATCA Fabric: Dual Star & Mesh

June 4-5, 2012 8

ATCA System Features, Extensions

• ATCA Board and RTM – Designed for ATCA board hot swap from front

– Designed so all or most I/O from rear via RTM

– IPM system isolates problem board, calls maintenance

– Main board or AMC swapped while RTM remains in place

– IPM returns to service with no shelf-wide interruption

• Extensions for Physics - RTM – PICMG 3.0 RTM had no management features, standard

connectors, keying

– Physics Committee designed PICMG3.8 with IPM power

management, high density fabric I/O connectors, JTAG


June 4-5, 2012 9 9


PICMG3.8 RTM

Interface Standard

Rear View

120 - IO Channels (3x40)

IPMI, Power Connector (blue)

2 Mechanical Keys

Courtesy M. Huffer et al, SLAC

June 4-5, 2012 10

IPMI System Basics


June 4-5, 2012 11


• IPMI ~1990

pre-dates

ATCA for

use in

Mainframe

computers,

PCs etc.

– Architecture,

protocols,

chip sets

developed by

Intel, HP,

NEC, DELL

June 4-5, 2012 12

MicroTCA Shelf

• ATCA Carrier Board Transitions to AMC Shelf – Segment of ATCA committee developed AMC shelf to lower

infrastructure costs

– Key was 1-wide multi-tongue MCH combining star switching,

shelf manager and IPMI communications

– MicroTCA shelves developed: 1U, 2U, 3U, horizontal and

vertical card orientation; no single shelf format

– Double wide AMC had space for RTM but undeveloped


June 4-5, 2012 13

MicroTCA Infrastructure for Physics

• Physics TC declared 2-wide w/ RTM essential – 3X larger clean analog design space than 1-wide AMC alone

– Rear I/O, hot-swap options

– Technical Committee design included:

• 2-wide AMC, mirror image RTM

• AMC extended IPMI management to RTM for power, temperature, user

diagnostics, hot swap

• 3x30-pair fabric balanced line data connectors (same style as ATCA)

• Separate new connector for all RTM power and IPMI control

• Mechanical & E-keying, JTAG

• Hot swap handles, indicators, for both AMC, RTM

– Major Goal: Higher level of interoperability of lab-industry products

by defining I/Os of generic AMCs to support multiple applications


June 4-5, 2012 14

AMC-RTM Mechanics-Front/Rear Panels


Module sizes

June 4-5, 2012 15

MTCA.4 Mechanical Details


June 4-5, 2012 16


June 4-5, 2012 17

MTCA.4 Full Payload Shelf

• RFQ developed for Industry – Infrastructure costs least when all payload slots filled

– Hot swap dictates all components be plug-modular including

power and cooling

– Features:

• 12 payload slots mid-size front panel 2-wide modules

• Push-pull cooling bottom to top, front to back with redundant controllable

hot swappable fans

• Redundant intelligent power modules of 1 KW nominal (smaller if not

needed)

• Full function 4-tongue MCHs with Shelf Manager

• Dual star backplane with special layer for timing, triggering, point-to-point

• Designation of spare lines (extended options region) for switched point-to-

point low jitter timing (~1 psec)


June 4-5, 2012 18

MTCA.4 AMC-RTM-Shelf Concept


text

FAN TRAY-HOTSWAP

FRONT END MODULE

ADVANCED

MEZZANINE CARD (AMC)

INPUT/OUTPUT SIGNAL

CONDITIONING

REAR TRANSITION MODULE

(RTM)

I/

O

I/

O

B

P

C

O

N

N

B

A

C

K

PL

A

N

E

AIR IN

AIR OUT

FAN TRAY-HOT SWAP

US

ER

DE

FIN

ED

I/O

CO

NN

EC

TO

RS

June 4-5, 2012 19 19


12 Payload Slots

2-Wide w/ RTM

Dual Star

Dual MCHs

Dual PM, CU

June 4-5, 2012 20

Point-to-

Point Links

8 x M-LVDS:

Trigger

MTCA.4 Backplane Extensions

12 Slot MTCA.4

Backplane

June 4-5, 2012 21

MTCA.4 Clock, Trigger & Interlocks

June 4-5, 2012 22

IPMI Extension to RTM

• When adding RTM, AMC supplies all power,

management – 80W max was spit to 50W AMC, 30W RTM

– New connector pins sized accordingly

• Extension IPMI to RTM required protocol

compatibility w/ ATCA – ATCA committee worked on parallel effort for managed

RTM, but no standard connectors as done for MTCA.4

– Physics committee assured protocol compliance

– Following slides show implementation


June 4-5, 2012 23

IPMI Extensions


Additional RTM control signals for MTCA.4

Backplane

MCH 1..2

AMC 1..12

CU 1..2

Presence

Enable

PM 1..4

Power On

PP (+12V)

MP (+3.3V)

Presence

Enable

Presence

PP (+12V)

MP (+3.3V)

µ RTM 1..12

PP (+12V)

MP (+3.3V)

Presence

MMC

Rear cooling Backplane

MCH 1..2

AMC 1..12

CU 1..2

Presence Presence

Enable Enable

PM 1..4

Power On Power On

PP (+12V) PP (+12V)

MP (+3.3V) MP (+3.3V)

Presence

Enable

Presence Presence

Enable Enable

PP (+12V)

MP (+3.3V)

PP (+12V) PP (+12V)

MP (+3.3V) MP (+3.3V)

µ RTM 1..12

PP (+12V)

MP (+3.3V)

Presence

MMC

Rear fans

EMMC

Front fans

MP (+3.3V)

Enable

June 4-5, 2012 24

IPMI Extensions 2


IPMB-L Connects the MCMC on

the MCH to the MMC on

the AMC Modules

Radial architecture

IPMB-0 Connects the MCMC on

the MCH to the EMMC on

the PM and CU

Bused architecture

I2C-bus Connects the AMC to the

µRTM

The µRTM is treated as

managed FRU of the AMC

Management extensions in MTCA.4

June 4-5, 2012 25

II. Physics System Requirements


June 4-5, 2012 26

Physics Requirements 1

• Control Systems – MTCA.4 optimized for controls requirements

– Basic Infrastructure:

• Powered shelf star, dual star backplane

• MCH, Processor, IPMI

• Timing Generator/Receiver

– Applications AMCs – Generic- 3 Types:

• Fast multichannel high resolution low noise ADC-DAC

• FPGA programmable Processor

• Industry Pack Carrier

– Each leverages multiple applications with single design

– Strategy: Develop min. 2 suppliers each generic AMC


June 4-5, 2012 27


• Applications Specific RTMs – Generics can serve multiple requirements via RTMs

– Application specific RTMs vary in type of design skill

• GHz Low Level RF, high performance wide dynamic range

• Low speed <60 MHz 12 bit analog interlock fault detection

• industrial monitoring and control -- vacuum, temperature,

pressure, voltage, current

– RTMs also can be adaptable, programmable

– Strategy: Build RTMs to standard generic AMC

interfaces; develop COTS sources when quantities

justify


June 4-5, 2012 28


• AMC-RTM Examples


Generic AMC Typical RTMs

Fast 16 bit ADC-DAC LLRF Feedback

Beam Position Monitor

Cavity BPM

Toroid Precision Charge

FPGA Medium Performance Fast-Slow12 bit ADC

Interlocks , FPGA

RF Interlocks

Beam Loss Monitor

Beam Length Monitor

3- Industry Pack Carrier Motor Controllers

Wire scanners

Vacuum monitor/control

Temperature monitor/control

June 4-5, 2012 29

III. COTS & Lab Developments


June 4-5, 2012 30

Shelves

• 3 Models – 6 Payload slot development unit half-rack 6U

• Produced early while main production unit being designed

• Power and cooling non-standard, incompatible

• New units do have “Physics” backplane timing layer

• Schroff, ELMA

– 12 Payload slot production rack-mount 8U

• Fully redundant dual star, MCH, power & cooling units

• Schroff, ELMA, PT (Performance Tech)

– 6 Payload slot production rack-mount 2U

• Fully redundant dual star production unit, 3 slots w/RTMs

• PowerBridge to Schroff specification (First prototype delivered)


June 4-5, 2012 31

Shelves


6-Slot Development

Non-Redundant

Schroff, Elma 12-Slot Fully Redundant

Schroff, Elma

12 Slot Fully Redundant 40GbE

PT Performance Tech

6 Slot Non redundant

3 slots w/ RTMs PowerBridge

June 4-5, 2012 32

MCH Carrier Manager and Data Hub

• Standard MCHs – Support all hub switch, IPMI operations in Physics

backplane except radial extended options clock lines

which SLAC currently does not need

– Currently evaluated by SLAC are NAT, Vadatech

– Some firmware differences were resolved to meet

MTCA.4 standards

– NAT currently developing 1 psec jitter switch card for

radial lines for DESY

– Vadatech will also produce this option in future


June 4-5, 2012 33

MCH Units


N.A.T MCH

Manages 12 AMCs, 2 cooling units,

1-4 power modules

Supports PCIe, Serial Rapid IO

Vadatech UTC002

Currently using in LLRF

System tests

Fully featured, compliant,

preferred front panel

network I/O

June 4-5, 2012 34

Processors

• All are single wide totally compatible with

ATCA carriers

• Processors evaluated to date are: – AdLink AMC 1000 Core 2 Duo

– Kontron AM4020M i7

• Other processors available – Concurrent, Radisys, GE, etc.

• On order – Vadatech AMC720 i7 (delayed due to silicon late

delivery)


June 4-5, 2012 35

Timing Interim Solution: PMC Adapter

• Existing solution for LLRF test system and lab prototyping

uses MRF (Micro Research Finland) EVR PMC adapter.

– Not a permanent solution as all cabling is via external coaxes and

SMA or Lemo connectors; timing backplane bypassed; however

works fine for test purposes

– New highly efficient solutions discussed in following slides


MRF EVR on

Vadatech PMC

Adapter

June 4-5, 2012 36

Timing Modules

• DESY-Stockholm Generator-Receiver

– Single wide unit operational for over a year at DESY

– SLAC has a unit but not implemented

– Not compatible with SLAC’s MRF (Micro-Research

Finland) EVR (Event Receiver) timing network protocol

• Gen 2 Prototype with RTM – 2nd Generation 2-wide with RTM nearing completion

– Generator will be optional plug-in daughter-card

– Additional I/O Tx-Rx options via RTM

– To be produced commercially


June 4-5, 2012 37

Timing AMC-µRTM 1&2-Wide MTCA.4

New Developments in I&C Standards for

Physics

• Both single & double-wide MTCA.4 compliant

• Double-wide RTM allows rear expansion to multiple receivers

• Accesses all parallel, point-to-point serial switched lines

• SW Committee looking into standardizing protocols

MTCA.4 Backplane

June 4-5, 2012 38

Timing Modules 2

• International Technologies (I-Tech) Receiver

EVRX – EVR & MTCA.4 Compatible single wide in prototype stage

– Will also be configurable as generator

– To be shown at xTCA Workshop June 9-10 2012 Berkeley

• 3


EVRX PC Board

Top View

June 4-5, 2012 39

I-Tech Timing EVRX

• EVRX 3D Models – Courtesy International

Technologies


June 4-5, 2012 40

Example Timing Solution - BPMs

• Current solution: 4 network cables per BPM – BPMs are rackmount Pizza boxes with 4 long timing & data network

cables per box, expensive, consume large rack space

– Cables are primary trigger, calibration triggers, clock, data

• MicroTCA Solution: 1 EVR cable to BPM shelf – BPM RTM contains filters and calibrate circuits 4 input Ch.

– Single parallel bus trigger ahead of beam from timing module starts all

units sampling ringing filtered input signals, processing data in FPGA,

delivering to processor on demand

– Each BPM digitizer FPGA when ready delivers 2 consecutive timed

calibrate signals via RTM

– After 3 cycles unit is ready for next pulse

– Large network cable plant eliminated; rack space minimized to 1 shelf


June 4-5, 2012 41

Lab-Industry 1: Fast ADC RTM & ADC-DAC


Physics

Courtesy A. Young, SLAC & Struck Co.

Includes IPMI Extension to RF Chassis,

Rear clock, trigger in & DAC out

June 4-5, 2012 42

Example 2: Generic FPGA & Fast-Slow 12

bit ADCs RTM for interlocks


Physics

Courtesy D. Brown, SLAC & TEWS Co.

June 4-5, 2012 43

V. xTCA Market Growth

Projections


June 4-5, 2012 44

Survey Data

• Limitations – Survey 1 for total ATCA market commissioned by

industry leader, Emerson

– Survey 2 looked at only blade sales which appear to be

about half the total market

– Market segmentation shows 80%+ total market going to

Telecom

– MicroTCA is represented partly in remaining 20%

– Total of $800M/yr projected for 2012

– Close to entire VME market which is 80% military


June 4-5, 2012 45

Survey 1: ATCA Total Market Forecast


Note:

Stated target of

PICMG for

ATCA was

eventual

penetration of

global market of

10% of $100B

Note - Current

ATCA market is

approx same as

total VME market

June 4-5, 2012 46

Survey 1: ATCA Adoption Potential in

Adjacent Markets

• Markinetics also believes the broad ATCA software

ecosystem and developer community must also evolve in

parallel in order for ATCA to achieve greater acceptance

across multiple industries, including a number of mission-

critical enterprise sectors.

• For instance, one specific opportunity on which some ATCA

suppliers have been attempting to capitalize is the

widespread reallocation of military and defense-related

spending toward network-centric cyber-security initiatives,

with ATCA as a common, open-standard platform option.


June 4-5, 2012 47

Survey 1: Forecast Analysis

• Looking ahead, the growth trajectory for ATCA adoption certainly will be

affected over the next five years, Markinetics believes, by a number of

important factors, including:

– 1. Improving macro- and micro-economic conditions and the gradual return

to an investment (rather than a cost-cutting) posture among the world’s

leading telecommunications services providers

– 2. Broader acceptance across various information-critical industries

adjacent to the traditional telecom sector

– 3. A far more engaged software developer community focused on enabling

robust ATCA-based solutions

– 4. Incremental advances in ATCA-compliant technology (e.g., engineering,

design, reduced energy consumption, thermal management, etc.)

– 5. A unique capability and capacity to adequately provide a range of support

services that complement the hardware product


June 4-5, 2012 48

Survey 2: Market Segmentation


Note:

Segmentation

in rough

agreement

with Survey 1

Note:

Current

telecom

market over

80% of total

June 4-5, 2012 49

VI. Summary & Conclusions


June 4-5, 2012 50

Summary

• Summaries of COTS and Lab initiatives,

latter principally in RTM sections, shown in

following sllides.


June 4-5, 2012 51

MTCA.4 Shelves 6/7 & 12 Slot


Item- Shelves Mfgr Type Available

6-slot Schroff 1-Star backplane. √ Non redundant

7 slot Schroff 1-Star backplane.

√ Standard PS

Removable fan

6-slot ELMA 1-Star backplane. √ Non redundant

12-slot Schroff 2-Star redundant √ Managed PU, CU

12-slot ELMA 2-Star redundant √ Managed PU, CU

12-slot Perf Tech 2-Star redundant 6/12 for β test

6-slot Power

Bridge

1 Star non

redundant

2U, 3 RTM slots

12 Slot VT811 Vadatech 2-Star redundant 6/12 release

June 4-5, 2012 52

MTCA.4 Power Modules


Item- Power

Modules

Mfgr Type Available

Puma µBlade Puma 900W Unreliable, RF Noise

6-pack mod. Schroff 300W Devmt √ Incompatible w/12 slot

6-pack mod. Elma 300W Devmt √ Incompatible w/12 slot

12-pack mod. Schroff 900W Due 8/12

6-Pack mod Power-

Bridge

600W Prototype delivered

12 Pack mod Performan

ce Tech

1000W, 1200W

modules

Beta unit 6.12

12-pack mod. Vadatech 1000W W/ AC in Due 7/12

12-pak mod. Vadatech 1200W, -DC in Due 7/12

12-pak mod Wiener 900W AC in Due 7/12

June 4-5, 2012 53

MCH Modules


Item-

Processors

Mfgr Type Available

UT002 Vadatech Full featured √

PCIe Gen3 NAT Full featured √

? Performance Tech PT Full featured

√ Beta test

unit 6/12

? Kontron Full featured? √

June 4-5, 2012 54

MTCA.4 Processors


Item-

Processors

Mfgr Type Available

AMC121 Performance Tech PT Intel core 2 duo √ Beta test

unit 6/12

ASLP11 GE Intel core 2 duo √

AMC1000 Adlink Intel core 2 duo Unreliable

AM31x Concurrent Tech Intel i7 √

AM930 Concurrent Tech 2-wide i7

PCIeGen3

9/12

AMC 720 Vadatech Intel i7 On Order

Radisys Inteli7 √

June 4-5, 2012 55

MTCA.4 Generic AMC Modules


Item- AMCs Mfgr Type Available

Digitizer 8300 Struck 125MSPS 16 bit 10 CH √

Digitizer 720 Vadatech 125MSPS 16 bit 10 CH Q3 12

Digitizers SP Devices 1.6-7 GSPS 8-14 bits

No RTM

Q4 12

IP Adapter Hytek 3-IP adapter w/RTM √

TAMC220 TEWS 3-IP adapter w/RTM Intfce √

TAMC651 TEWS FPGA Processor Spartan 6

w/ RTM Intfce

√

June 4-5, 2012 56

Application RTM Adapters 2


Item-

RTMs

Lab/

Co.

Type Mates

With

Available

LLRF SLAC Passive DC, IPMI Extend SIS 8300 √

Interlock SLAC Fast/Slow 12 bit ADCs TEWS651

BPM SLAC Filter, Calibration Interface SIS8300 1st proto in

test

LLRF DESY Downconverter 1.3-3.9 GHz SIS8300 √

Sensors

Interface

XFEL Avalanche photo diode

pulse stretcher

SIS 8300 √

TAMC002 TEWS 3-Industry Pack RTM TAMC651 √

? Hytek 3-Industry Pack RTM ? √

ADC/DAC DESY 8ch 16bit ADC, 8ch 16bit DAC DAMC2 1st proto in

test

ADC DESY Coupler interlock DAMC2 1st proto

June 4-5, 2012 57

Application RTM Adapters 2


Item-

RTMs

Lab/

Co.

Type Mates

With

Available

Digital IO DESY Machine protection system DAMC2 1st proto in

production

ADC DESY Coupler interlock DAMC2 1st proto test

ADC DESY Beam loss monitors DAMC2 Development

ADC DESY Toroid protection / readout DAMC2 Development

UCL Clock & trigger control for exp. DAMC2 1st proto test

Interface DESY Wire scanner DAMC2 Development

BPM DESY Low charge button/stripline SIS8300 1st proto test

June 4-5, 2012 58

Why MicroTCA?

• New standards essential as technology develops, legacy

systems no longer cost-effective or supportable – NIM (1966), CAMAC (1976), FASTBUS (1986), VMEp (1998)

• Last decade revolution in analog, digital, communication

technologies accelerates need for change – Programmable FPGA’s obsolete discrete logic components,

– Multilayer board design enables Gigabit backplanes >10 GHz BW

– Integrated SERDES communications obsolete parallel bus backplanes

– LVDS balanced multi Gbps backplanes minimize discrete switch blades

– Intelligent platform management diagnoses enterprise wide problems to

board level, initiates corrective action

– Redundant architectures => shelf Availability of 5 nines or better (0,99999)


June 4-5, 2012 59

Summary of Performance Advantages

• Excellent analog performance

• Modular partition AMC-RTM allows planned upgrades FPGA,

processor parts (Moore's Law) separate from analog and

interface module

• Partition interface enables different design skills to work on

complete system

• System based on high performance scalable state-of-art

communication technology only (PCIe and Ethernet)

• Common interfaces and operating systems for control systems

• High availability architecture and remote management/

maintenance, upgrades of firmware

•


June 4-5, 2012 60

V. Summary & Conclusions

• MicroTCA Advantages being recognized increasingly in

Europe and Asia and more labs are signing on (ESS,

ESSB, GSI, Spring8)

• PICMG industry support is solid and growing, much

stronger in US than lab support

• Standards collaborations under PICMG will continue to

refine timing protocols, software-firmware interfaces as

more users raise practical problems.

• MicroTCA has potential to be physics standard of choice for

new projects, upgrades for next 1-2 decades


June 4-5, 2012 61

END OF SLIDES


Documents

Introduction to MicroTCA - Pages - … 4-5, 2012 4 PICMG-ese Term Definition PICMG PCI Industrial Computer Manufacturers Group, 250 corporations ATCA Advanced Telecommunications Computing