SADT engineering meeting 9Oct13 new template · Click&to&editMaster&/tle&style & 15/10/2013 8 & Regional Science & Engineering Centre(s) Regional Science & Engineering Centre(s) SKA

Click to edit Master /tle style

15/10/2013 1

Signal and Data Transport Consor1um Overview SKA Engineering Mee/ng 9th October 2013

Richard Oberland, SADT project engineer University of Manchester


15/10/2013 2

The SKA and SADT

Ø very large collec1ng area (km2) à Many network connec1ons

Ø very-‐large-‐angle field of view à High Bit Rates

Ø wide frequency range à Varied antenna systems for connec1on with varying connec1on speeds. High Bit Rates

Ø large physical extent à Geographically distributed Network and long distance transmission

Ø SADT is the backbone of the telescope Ø Our end-‐product needs to be transparent but we drive most of the design interfaces


15/10/2013 3 3

SADT Tasks and responsibili/es


15/10/2013 4

SADT Consor/um •  Consor/um Board, chair: Jan Geralt Bij De Vaate •  Lead ins/tute: University of Manchester

–  Leader: Richard Schilizzi –  Deputy Leader: Keith Grainge –  Project Manager: Althea Wilkinson –  System Engineer: Paul Carr –  Project Engineer (SADT): Richard Oberland –  Element specialist (SAT): Simon Garrington

•  ASTRON (Netherlands) •  NPL (UK) •  NCRA (India) •  CSIRO (Australia) •  IT (Portugal) •  JIVE (Netherlands)

•  SKA Africa (South Africa) •  Tsinghua University (China) •  Peking University (China) •  University of Granada (Spain) •  AARNet (Australia) •  DANTE (UK)

Associate members

Full members


15/10/2013 5

SADT Work breakdown structure

4. SKA.TEL.SADT.MGT, project management – Althea Wilkinson, Uman 4. SKA.TEL.SADT.SE, system engineering – Paul Carr, Uman 4. SKA.TEL.SADT.SAT.CLDES, Clock Design – David Hindley, NPL 4. SKA.TEL.SADT.SAT.CLSYS, Clock System – David Hindley, NPL 4. SKA.TEL.SADT.SAT.LMC, SAT Local Monitoring and Control -‐ Yashwant Gupta, NCRA 4. SKA.TEL.SADT.SAT.STFR, Distribu/on of Time, Freq and Phase – Simon Garrington, Uman 4. SKA.TEL.SADT.NWA network architecture – Peter Maat, ASTRON 4. SKA.TEL.SADT.NMGR network manager – Yashwant Gupta, NCRA 4. SKA.TEL.SADT.DDBH digital data back haul – Richard Oberland, Uman 4. SKA.TEL.SADT.TM telescope manager -‐ Yashwant Gupta, NCRA 4. SKA.TEL.SADT.CSP central signal processor – Shaun Amy, CSIRO 4. SKA.TEL.SADT.SDP science data processor – Shaun Amy, CSIRO 4. SKA.TEL.SADT.LINFRA local infrastructure – Bruce Wallace, SKA Africa

•  Individual contact details and responsibili/es will be shared this week


15/10/2013 6 6

SADT Product structure


15/10/2013 7

SADT Product structure

Transponder

W A C Transponder Transponder Transponder

Dispersion Comp/ Amplifier

WDM Mux

A Transponder Transponder Transponder

Dispersion Comp/ Amplifier

WDM Dmux

3 Traffic types (Science Data, M&C, SAT) 3+ Opera1ng modes & network services (e.g. Normal – various science experiments, Commissioning, Cri/cal)

Standardised modular product structure Goal: to reduce total BOM and development effort

Logical data network (OSI layers 2-‐4: protocols, rou/ng, etc.)

Physical media network (OSI layer 1: Op/cal layer architecture)

Transmission links (Mod format, etc.)

-‐ Termina1on Node (e.g. Receptor) -‐

-‐ Termina1on Node (e.g. CSP) -‐ -‐ Op1cal Fibre infrastructure -‐

WDM/OADM Amplifier Cable junc/on Amplifier

Common equipment

Tributary equipment

Tributary equipment Common equipment

3 Segments (Receptor array to CSP, CSP to SDP, SDP to world) 3 Telescopes (Low, Mid, Survey)


15/10/2013 8

RegionalScience &

EngineeringCentre(s)

RegionalScience &

EngineeringCentre(s)

SKA Observatory Global Headquarters

Remote stationson spiral arms

ScienceComputing

RemoteStation

RemoteStation

RemoteStation

Host Country Headquarters

Central Signal Processing

SKA1-low

SKA1-survey

Core Arrays

Australia

Remote stationson spiral arms

ScienceComputing

RemoteStation

RemoteStation

RemoteStation

Host Country Headquarters

Central Signal Processing

Mid-FreqAperture Array

(SKA2)

SKA1-mid

Core Arrays

South Africa

0.45 Tb/s 36 Tb/s

(8.7 Tb/s)* 11 Tb/s 39 Tb/s

3.2 Tb/s

19.6 Tb/s

27 Tb/s

SKA1-‐low SKA1-‐survey SKA1-‐mid

()* – if digi1sed

100Gb/s 100Gb/s

Data Transport Requirements

(47 Tb/s)*


15/10/2013 9

Key SADT cost drivers

LOW MID SURVEY Array core digital interfaces 866 133 18 Array arm digital interfaces 45 57 42 Total array digital interfaces 911 190 60 Output bit rate 10Gbps 90Gbps 864Gbps Max cable distance 70km? 210km? 40km?

•  SADT design strongly dependent on location of •  Network nodes in array segments •  CSP and SDP buildings

•  Costs strongly related to network topology •  Number of nodes and aggregation points •  Cable routing link distance •  Bit rate From baseline design


15/10/2013 10

Receptor array to CSP (DDBH) Concept genera/on routes

o  Cut-‐down turn-‐key solu/ons from telecom systems houses o  Custom-‐built SADT stand-‐alone boxes/card(s), hos/ng

o  Discrete COTS ICs or FPGAs for SerDes, transport protocols, custom framing, etc.

o  COTS op/cal transceiver modules, backplane physical interface?

o  Custom-‐built cross-‐element card(s), hos/ng o  COTS components as above, but implemen/ng func/onality from other

consor/a. Interface boundary located on a shared board/IC e.g. FPGA transceiver core

o  Trade offs to be considered for down-‐selec/on o  e.g. CAPEX/OPEX cost, RFI, development complexity


15/10/2013 11

Central Signal Processor to SDP

o  Benchmark solu/on is ASKAP ultra low-‐loss fibre pairs with intermediate amplifica/on

o  Loca/on of SDP in South Africa needs to be resolved

Solar powered CEV picture (lek) and communica/ons rack (right) installed at Geraldton (WA) (courtesy of S. Amy, CSIRO)


15/10/2013 12

SDP to the outside world

o  Issues to be considered §  Cost not included in cost cap §  Define data products and data rates required

§  Feasibility study needs to be done §  (Support SDP prototyping of data product distribu/on)

§  External interface and access to/distribu/on of data via NRENs

§  Secure, remote, reliable access to SKA Observatory networks for commissioning, debugging and facility opera/on from various loca/ons


15/10/2013 13

Synchronisa/on and Timing (SAT)

Requirements 1.  ensure phase coherence of whole array, at ps

level 2.  provide high precision long-‐term /ming for

pulsars and transients, 1-‐10 ns over 10 yrs 3.  provide absolute /me for

•  System management •  Antenna poin/ng •  Timing /cks, etc

4.  provide frequency standards for LOs etc 5.  VLBI opera/ons


15/10/2013 14

Clock system

o  Central clock reference systems at RSA and AUS sites will contain an ensemble of clocks including exis/ng pre-‐cursor H-‐masers

•  Each site will need at least one ac/ve hydrogen maser clock synchronised to IAT or other /mescales as the primary reference

H-‐maser clock product example ,at NPL UK


15/10/2013 15

Time and frequency distribu/on & compensa/on methods

1.  Two-‐way connec/on to antenna elements, with path measurement

2.  One-‐way connec/on of frequency/clock signal from central sta/on to surrounding element with ac/ve path compensa/on

3.  Independent accurate atomic clocks servicing a sta/on or group of antennas


15/10/2013 16

Frequency transfer and phase error measurement benchmark

Meets Baseline Design requirements •  Amplitude modula/on •  RF phase measurement with off-‐line compensa/on

Implemented on e-‐MERLIN and J-‐VLA

C C

Frequency

-‐ in

Phase measure

A

To Correlator

A

Locked coupled crystal

oscillator

Electrical

Op1cal Electrical

Op1cal

Electrical

Op1cal Electrical

Op1cal

Frequency

-‐ out

Phase error out to compensa/on

scheme

Maser clock in

Clock out to receptor, etc

Receptor side

Fibre link

Central processor side


15/10/2013 17

Local infrastructure: Re/cula/on

o  Cable installa/on design & techniques o  Combine with power re/cula/on? o  Precursor integra/on

-‐4000

-‐3000

-‐2000

-‐1000

0

1000

2000

3000

4000

-‐4000 -‐3000 -‐2000 -‐1000 0 1000 2000 3000 4000

ASKAP Array

Australia South Africa


15/10/2013 18

Telescope Manager transport networks

o  Carries monitoring and control signals for mul/ple LMCs/control planes

o  Opera/on modes include: o  PNET -‐ Produc/on Network for normal opera/on o  ENET – Engineering Network for system diagnosis o  SNET – Safety Network to command a telescope shutdown and report o  These subnets may be carried on separate physical/logical networks

o  TM transport networks could be built using mature COTS switch products o  Significant effort required at TM and infra consor/a interfaces


15/10/2013 19

Network manager & combined network architecture

•  Network Manager (NMGR) : Control plane (‘LMC’) for SADT network equipment –  Strong interac/on with Telescope Manager

•  Network Architecture (NWA) : dedicated work package op/mising architecture considering all products

3 Traffic types (Science Data, M&C, SAT) 3 Segments (Receptor array, CSP to SDP, SDP to world)

3 Telescopes (Low, Mid, Survey)


15/10/2013 20

SADT network architecture

Remote buildings

Core bunker

Core receptor

Core receptor

SDP Core CSP Core

Remote receptor

Beamformer Power

Bunker region

SAT LMC

DDBH LMC Cooling

VOIP phone

Security control

Power

CSP LMC

SDP LMC

Core M&C

Assembly region

Workshop

Office VOIP phone

Cooling

Accommoda1on region

Office Power

Webcam

Cooling

Wireless comms

Feed Receiver

Power Cooling

Receptor region

SAT LMC

DDBH LMC

Weather stn

Wireless comms

SAT LMC

Feed Receiver Power

Cooling Weather stn

Feed Receiver Power

Cooling

Weather stn

Cooling Core M&C

SADT.DDBH LMC SADT.CSP

LMC SADT.CSP

LMC

SADT.SDP LMC

Office

Power

Cooling

Office Power

Remote hut

Cable junc/on

Outside world

M&C Sci Data SAT


15/10/2013 21

Network upgrade challenges SKA Phase 2

o  10x number of antennas/sta/ons

o  At least 10x data rates o  Installing Phase 2

infrastructure over Phase 1? o  Long distances (>1000 km)

-‐20000

-‐15000

-‐10000

-‐5000

0

5000

10000

15000

20000

-‐20000 -‐15000 -‐10000 -‐5000 0 5000 10000 15000 20000

Array ConfigurationsMeerKAT and SKA2

20000

-‐2000

-‐1500

-‐1000

-‐500

0

500

1000

1500

2000

-‐2000 -‐1500 -‐1000 -‐500 0 500 1000 1500 2000

Array ConfigurationsMeerKAT and SKA2

South Africa site illustraEon


15/10/2013 22 22

SADT Work plan and process


15/10/2013 23

Top level SADT Milestones

•  From the recent SKAO nego/a/on phase Stage 1 Milestones Stage 2 Milestones

Milestone Date Milestone Date

1) T0 01/11/2013 1) KO 01/01/2015

2) T0 + 12 weeks SKAO requirement TBDs complete? 01/02/2014

2) Verifica1on plan 01/02/2015

3) Concept genera1on and down selec1on 31/05/2014 3) Prototypes & modelling 01/10/2015

4) ICDs finalised 30/06/2014

4) Test report on prototypes 01/12/2015

5) Baseline design and cost snapshot 15/09/2014

5) Integra1on and tes1ng 01/06/2016

6) PDR document delivery

01/11/2014 6) Report on Integra1on 01/07/2016

7) PDR 01/01/2014

7) Procurement docs for tender 01/10/2016

8) Close PDR 01/02/2015 8) CDR document delivery 01/11/2016

9) CDR 01/01/2017 9) close CDR 01/02/2017


15/10/2013 24

Key informa/on from SKAO


15/10/2013 25

SADT requirements capture process

•  Technical •  Func/onal •  Performance •  Opera/onal •  Interfaces •  Standards compliance

•  Economic •  Legal •  Statutory regula/ons •  Poli/cal •  Environmental •  Sociological

•  Training •  Equipment •  Personnel •  Infrastructure •  Doctrine / Strategy •  Organisa/on •  Industry •  Logis/cs •  Non Func/onal

•  Aim to capture a complete set of requirements types from all stakeholders & end users •  To fuel industry-‐biased concept genera/on •  Long term product lifecycle benefits

Stakeholders Astronomers SKA Office Telescope Opers DSH LFAA CSP SDP INFRA TM AIV STFC GOV AGENCIES SADT – Internal MANUF ENG CONSTR ENG MAINT ENG COMMISS ENG

End users Scien1st Design Engineer Operator/Maintainer Manufacturer Commissioner Constructor/installer


15/10/2013 26

Models and Prototypes

•  Physical layer modelling o  Component and transmission links o  Network planning and topology design

•  Parametric modelling – Cost, etc. •  FPGA/pluggable op/cs evalua/on boards •  Dark fibre tests (e.g. e-‐Merlin, NRENs) •  Detailed design implementa/on and

simula/on tools o  Electronic circuits o  Sokware architecture models o  RFI, thermal, mechanical models o  Ready for manufacture prototypes

o  H/W & S/W development on cut-‐down COTS equipment and custom integrated cards


15/10/2013 27

Cable & trench rou/ng & aggrega/on

Using TrenchCOAT modelling tool


15/10/2013 28

Verifica/on plans

•  Pre-PDR phase o  Customised testing and modelling of component samples

against SADT level requirements

o Pre-CDR phase o  Testing board assemblies against design spec pass/fail criteria o  Network performance margin testing against EOL budgets o  Environmental testing – Thermal, humidity lightning o  EM emission testing o  On/off site system integration level testing

o  e.g. from SKA1-Mid dish equipment to CSP equipment interface, across long-haul distances with mixed traffic loads


15/10/2013 29 29

Conclusions


15/10/2013 30

Key Challenges •  Interfaces with every other element •  Strongly affected by system design changes

–  e.g. 3 order magnitude difference to SADT scope for LFAA concepts –  Good models essen/al

•  Turn-‐key solu/ons within cost cap? –  Custom solu/on materials poten/ally cheaper, but added risk and

development /me considering complete requirements

•  Joint op/misa/on of 3 traffic types and 3 telescopes –  Goal: Reduce BOM and duplicated h/w & s/w development effort

•  Top risks –  Requirements insufficiently defined –  Changes to baseline architecture


15/10/2013 31

SADT aims for this week

o  Make key contacts within consor/a o  Determine responsibili/es and skill sets at an individual level

o  Get preliminary requirements driving ICD development

o  Seek clarity in the engineering and opera/ons use of the networks

o  SADT kick off mee/ng (Fri/Sat) with 27 apendees present all week


15/10/2013 32 32

Ques/ons?


15/10/2013 33

SADT Level 4 work packages

Documents

SADT engineering meeting 9Oct13 new template · Click&to&editMaster&/tle&style & 15/10/2013 8 & Regional Science & Engineering Centre(s) Regional Science & Engineering Centre(s) SKA