EVLA Correlator DRAO NovelTel Jun14-11halpern/chime/workshop... · 2011. 6. 21. · EVLA P j t O iEVLA Project Overview •Upgrade started ~2000-2001. • All new lower-noise receivers

Th EVLA C lThe EVLA CorrelatorB. Carlson

National Research CouncilCanada

Conseil national de recherchesCanada Novel Tel. Workshop, DRAO June 14, 2011

O tliOutline• EVLA project overview.

• What is a correlator?

• EVLA correlator capabilities.EVLA correlator capabilities.

• Basic signal processing.

• System architecture.

• A view inside the correlator room.

• A (very) short survey of current developments/directions.

B. Carlson June 14, 2001 Novel Tel. Workshop DRAO – The EVLA Correlator

2

A (very) short survey of current developments/directions.

• Peter Dewdney (now SPDO)• Peter Dewdney (now SPDO)• Sean Dougherty (NRC-DRAO)• Dave Fort (NRC-DRAO)• Sonja Vrcic (NRC DRAO)

• Bruce Rowen (NRAO-Socorro)• Kevin Ryan (NRAO-Socorro)• Hichem Ben Frej (NRAO-Socorro)

• Sonja Vrcic (NRC-DRAO)• Heng Zhang (NRC-DRAO)• Zoran Ljusic (NRC-DRAO)• Brent Carlson (NRC DRAO)

• Martin Pokorny (NRAO-Socorro)• Ken Sowinski (NRAO-Socorro)• Michael Rupen (NRAO-Socorro)

• Brent Carlson (NRC-DRAO)• Dave Del Rizzo (NRC-DRAO)• Ralph Webber (NRC-DRAO)• Mark Halman (NRC DAO)

• Rick Perley (NRAO-Socorro)• Barry Clark (NRAO-Socorro)• Kerry Shores (NRAO-Socorro)

• Mark Halman (NRC-DAO)• Dale Basnett (Contractor)• Rick Smegal (Contractor)• Amy Fink (frmr NRC DRAO)

• Pete Pelletier (Temp/prod)• Shelley Deakin (Temp/prod)

• Amy Fink (frmr NRC-DRAO)• Donna Morgan (NRC-DRAO)• BreconRidge/Sanmina• iSine Inc ; Altera; Xilinx

• Josh Fink (Temp/ship-rec’v)

• iSine Inc.; Altera; Xilinx

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 3



EVLA P j t O iEVLA Project Overview

• Upgrade started ~2000-2001.pg• All new lower-noise receivers and feeds. Antennas

completely gutted.• Replace waveguide system with digital fiber-optic system.

Samplers at the antennas for stable performance.• Increase BW from ~100 MHz/pol’n to 8 GHz/pol’n• Increase BW from ~100 MHz/pol n to 8 GHz/pol n.• Contiguous frequency coverage from 1-50 GHz.• New frequency-agile, high spectral and temporal resolution q y g , g p p

correlator.• Project completion end of 2012.


• Actually achieved SEFDs (initial spec):

– L-band – 400 Jy (325)– S-band – 350 Jy (235)

C band 300 Jy (245)– C-band – 300 Jy (245)– X-band – 250 Jy (300)– Ku-band – 280 Jy (385)– K-band – 450 Jy (650)– Ka-band – 620 Jy (760)– Q-band – 1100 Jy (1220/2760)y ( )

(Source: R. Perley, upcoming ApJLet)


Source: Rick Perley,NRAO


Wh t i C l t ?What is a Correlator?• Calculates the cross-correlation (cross-power) coefficient for every pair of

t i thantennas in the array.– XF: delay-domain cross-correlation and integration followed by FFT.– FX: real-time FFT, multiply channels and integrate.– These days, its most often all digital…and FX.

• More bandwidth yields more sensitivity…but higher-speed electronics.y y g p

• Spectral channels open up spectral-line observing (velocity, spectral index info)info).




EVLA C l t C bilitiEVLA Correlator Capabilities

• 32 antennas (EVLA 28 + spare capacity).( p p y)

• 8 GHz/pol’n, in 2 GHz bands.– 8-bit sampling, 1 GHz/pol’n x 2 = 2 GHz/pol’n – operational.– 3-bit sampling, 2 GHz/pol’n x 4 = 8 GHz/pol’n – samplers undergoing

testing…g

• 16,384 channels/baseline…up to 4 Mchannels/baseline at narrower bandwidths (≤ 0 5 MHz/sub band x 16 @1 pol’nnarrower bandwidths (≤ 0.5 MHz/sub-band x 16 @1 pol n, 256k channels ea.—2 Hz resolution) with “recirculation”.



• Stable digital sub-band filtering. Original concept was method for efficient parallel processing.– Sub-bands (a.k.a. “spectral windows”) 128 MHz, 64 MHz, …, 31.25 Sub ba ds (a. .a. spec a w dows ) 8 , 6 , …, 3 . 5

kHz.– Implemented in multi-stage digital filter in Xilinx FPGA.– Each sub-band can be on different delay center on the sky (within theEach sub-band can be on different delay center on the sky (within the

main beam of the antenna).

All 16 GH di it l h d t t il bl ll th ti• All 16 GHz digital phased-array output available all the time.



• Quasi-flexible sub-band tuning. “WIDAR” technique limited Q g qto sub-band integer slots (but stage-2 SSB mixer relaxes limitation somewhat).

• Pulsar phase binning: 2 banks of 2000 bins ~200 usec bin• Pulsar phase binning: 2 banks of 2000 bins, ~200 usec bin resolution on every product. Down to ~15 usec bin resolution with reduced bandwidth.


EVLA C l t C biliti

• High time resolution output: 100 msec (limitation due to data

EVLA Correlator Capabilities

g p (storage and backend processing) 10 msec 1GE output capable, 1 msec with 10GE output field upgrade.

Agile dumping/integration Each sub band can have different– Agile dumping/integration. Each sub-band can have different parameters…locked to system timing, or model of “something else” like pulsar ephemeris.

• Sub-arraying, with granularity of 4 antennas.


B i Si l P iBasic Signal Processing

• Introduce small epsilon frequency shift in signal in each antenna before sampler.

• Remove after sub-band filtering during correlation.

• Aliased sub-band transition bands de-correlate with time.– “Moral equivalent” of Walsh-function switching.

• Allows seamless “stitching” of sub-bands.– But, 2X sensitivity loss at sub-band edges.

• All-digital sub-sample delay correction. Stable, predictable.




S t A hit tSystem Architecture

• 8 “Station” racks – Antenna-based processing (wideband p g (delay, digital filtering, signalling generation).

• 16 “Station Boards” in each rack; 4 boards per antenna.

Di t ib t d b it hi f fl ibl ll ti f• Distributed cross-bar switching for flexible allocation of correlation hardware to sub-bands. (a.k.a. coarse filterbank“corner-turner”)

• Point-to-point cables from Station racks to Baseline racks.


Outputs to Baseline Board Pairs in Quadrants 1-4

F:0

F:0

F:2F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:2

F:2

F:10

F:4

F:4

F:4

Outputs to Baseline Board Pairs in Quadrants 1 4

Q4

Q3

Q2

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 BlB Pair

F:0

F:0F:5

F:5 F:3

F:3

Bar

FP

GA

F:8F:8 F:8 F:8 F:8 F:8 F:8F:10F:4

-Bar

Boa

rd-0

-U

Q1

-Bar

Boa

rd-0

-L

-Bar

Boa

rd-1

-U

-Bar

Boa

rd-2

-U

-Bar

Boa

rd-3

-U

-Bar

Boa

rd-4

-U

-Bar

Boa

rd-5

-U

-Bar

Boa

rd-6

-U

-Bar

Boa

rd-7

-U

-Bar

Boa

rd-1

-L

-Bar

Boa

rd-2

-L

-Bar

Boa

rd-3

-L

-Bar

Boa

rd-4

-L

-Bar

Boa

rd-5

-L

-Bar

Boa

rd-6

-L

-Bar

Boa

rd-7

-L

X-Bar BoardConnectionsF:3

1 co

lum

n/sl

ice

= 1

X-B

for 4

ant

enna

s

F:4

F 3

F:10

F:2

F 5 F 5 F 8

F:0

F:5

F 9

F:3

F:0

F:8

F:2

F 0F 8 F 8 F 8 F 8 F 8 Station B

oard O

utputsX- X-X- X- X- X- X- X- X- X- X- X- X- X- X- X-

StationBoard

Output Chip

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 wafer (SBOID)

F:3 F:5 F:5 F:8 F:9 F:0F:8 F:8 F:8 F:8 F:8 Sta

D0,B2,D2

B0,B1,D1 0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

p pX-bars

ards/an

t

C0,A2,C2

A0,A1,C1 0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

Station Board Digital Filters

4 Stat

ion Boar

Inputs from antennas

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 20

Inputs from antennas


S t A hit t

• 8 “Baseline” racks—baseline-based processing (cross-

System Architecture

p g (correlation and phased-array summing).

• 16 boards in each rack, 2 boards correlate a sub-band pair, all 32 antennas.– Can trade-off bandwidth for number of spectral channels.Can trade off bandwidth for number of spectral channels.

• 1 Gigabit Ethernet correlated data output from each board. – Phased-array output on 1 Gigabit Ethernet as well in “VDIF” format

for VLBI recording.



24B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator

















B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 41Credit: Paul Harden, NRAO

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 42Credit: Paul Harden, NRAO






C t D l t /Di tiCurrent Developments/Directions• S/W correlator (NRAO): DiFX – Deller, Brisken et. al. VLBA multi-(~400)

phase center s/w flexibility…new parameter space. 4 Gbps, ~10 antennas.• ASKAP correlator (CSIRO)– multi-Xilinx FPGA ATCA standard.• CASPER (Berkeley) – single-Xilinx FPGA + PPC non-standard FF, cable

interconnect, libraries, fast to “get going”, keep up with tech curve.• Uniboard (JIVE/Astron) – 8 x Altera Stratix-IV recently completed/released.

non-standard FF. Uniboard-2 planned (I believe).• JPL ASIC (D’Addario) – high efficiency/compact T-MUX CMAC array with no

acc m memor Plan on b ilding protoaccum memory. Plan on building proto.• GSA/X421 (DRAO) – physical layout for very large corr, with no cable

interconnects, X421 ASIC baseline-based, time+channel int, VBA for RT IP.• Kermode (DRAO) – multi-Xilinx FPGA ATCA standard with flexibilities for• Kermode (DRAO) – multi-Xilinx FPGA ATCA standard, with flexibilities for

other sig-proc + compute tasks, and interfaces. Commercial BSP planned.• ALL poly-phase FX.


G l t dGeneral trends…• S/W correlation for small-to-medium jobs…High degree of flexibility. If

t ft th i k d l t FPGA l tfnot software, then quick-development FPGA platforms.

• Quasi-general purpose FPGA boards, of varying densities, and varying interconnect schemes for small-to-large jobs.– Keeping up with the technology curve is still a challenge, and is most

easily done with few devices per board, but then limits performance.– Must factor in total system development costs and time, not just board

design and test time.– Boards with FPGAs + memory + MGT fabric.

• Custom designs only for very large systems (e.g. SKA-2)


SSummary

• EVLA project overview.

• What is a correlator? EVLA correlator capabilities.

• Basic signal processing... filterbank followed by c-lag corr.

S hi k bli &C li• System architecture…racks, cabling, M&C, power, cooling.

• A view inside the correlator room…lotsa pictures.

• S/W (small-med); FPGA boards (small-large); custom (very large).



B i Si l P iBasic Signal Processing• All-digital sub-sample delay correction. Stable, predictable.



Credit: Paul Harden, NRAO



2009 – 12 antennas


Awaiting longest baseline (A config) data(A-config) data.

L and Ka-band,2 GHz BW.


Documents

EVLA Correlator DRAO NovelTel Jun14-11halpern/chime/workshop... · 2011. 6. 21. · EVLA P j t O iEVLA Project Overview •Upgrade started ~2000-2001. • All new lower-noise receivers