ASKAP Signal Processing OverviewDIFX Users and Developers Meeting
CSIRO ASTRONOMY AND SPACE SCIENCE
Dr. Grant Hampson | ADE Team Leader4 September 2012
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
ASKAP Overview• 36x12-metre antennas
• 630 baselines
• 188-port PAF receiver• “Tuneable” over 0.7-1.8GHz
• Digitisation • Provides at least 300MHz processed BW
• 30deg2 FoV• 36-dual pol. Beams (varies with frequency)• Carries through to correlator• Tied array outputs also
2 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
ASKAP Differences• BETA (Eng. Test Array)
• Dual heterodyne receiver • ADC at antenna• Dragonfly-2 digitiser (IF sampling)• Redback-2 DSP board inside ATCA chassis
– Xilinx Virtex-6 FPGAs
• ADE (ASKAP Design Enhancements)• RFOF (RF over fibre) from PAF to central site• All digital electronics in central site• Dragonfly-3 digitiser (direct sampling)• Redback-3 DSP board in 1U chassis
– Xilinx Kintex-7 FPGAs
3 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
Digital Back-end Blocks
4 |
Correlator
Beamformer
Digital Receiver
1Beamformer
Correlator
FineFilter Bank
PAFCross
Connect
AntennaCross
Connect
192
36
PAF ADC PhaseSwitch
CoarseFilter Bank
ChannelSelect
ArrayCovariance
MatrixCoarseDelay Fringe
Tied Array
Long TermAccumlator
1
36-antennas192-ports2x16-bits300 MHz=66Tbps=6600 x 10G
36-antennas2x36-beams2x16-bits300 MHz=25Tbps=2500 x 10G
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
PAF Electronics – The Domino• The “Domino” combines:
• Low noise amplifier • RF filtering – defines 3 bands• RF over SM fibre transmitter
• Each Domino contains electronics for 2 PAF ports:• 1.8W power per port• 250grams per port
• Backplanes provide power,control and monitoring
• Duplex LC/APC connections to MTP elite, connect RF to central site
5 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
ADE Backend – Digital Receiver• Upgrade to Xilinx Series-7 FPGAs
• Kintex-325 FPGAs (840-DSP/RAM)• Contains first stage coarse filter bank
• Direct sampling ADCs• NatSemi 12-bit 1600MSPS ADCs
• Avago optical MiniPODs • Multimode 120Gbps modules• FPGA connect directly at 10Gbps
• 1U chassis packaging • No front or rear to boards• 16 ports per 1U• ~10W per PAF port
6 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
ASKAP (BETA) Processed Spectrum
7 |
Sampled Bandwidth = 384MHz
Processed Bandwidth = 304MHz
384MHz 768MHz
Second Nyquist zone
54 x 18.52kHz fine channels (= 1MHz)sent to the correlator
First-stage filterbank output:304 x 1MHz channels
oversampled by 32/27
Second-stage filterbank output:64 x 18.52kHz channels per 1MHz
critically sampled
Stage 1 filterbank
Beamformer
Stage 2 filterbank
-1 -0.5 0 0.5 1 1.5 2-100
-90
-80
-70
-60
-50
-40
-30
-20
-10
0
Frequency (MHz)
Mag
nitu
de (d
B)
Filterbank
Critically sampled filterbank
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
Frequencies?• Can’t have a single 768MHz channel – need to channelise
• Beamformer requires ~1MHz resolution to apply weights• RFI – can remove frequencies that are troublesome – few MHz• Allows looking at frequencies that are well separated• Allows different simultaneous zooms on the same frequencies
• Zooms• Fine delays/fringe stopping require ~100kHz to avoid smearing at band edges• To avoid aliasing near edges of 1MHz channels, and frequency rolloff
– The 1MHz channels are oversampled by 32/27=18.5%
• Tied Array / VLBI• Fine channels are stitched back together to form larger BWs (e.g., 16MHz)• To avoid aliasing and frequency rolloff, the fine channels are planned to also be
oversampled• Don’t require sample rates to be a power of 2
8 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
ADE DSP: Beamformer and Correlator• Common hardware platform
• Xilinx Kintex-420 Series-7 FPGAs• ~1680 DSP/RAM per FPGA• 2 DDR3 modules @ 1066MTS• 120Gbps optical transceiver
• Direct 10Gbps from FPGA to optics transceivers simplifies design
• 1U chassis packaging • No front or rear to boards• 6 FPGAs per Redback board
• Use of optical backplanes• Low capital cost – no power• 1.5Tbps optical backplane• Acts like “fixed” network switch• Scalable and modular
9 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
Why use FPGAs for Beamforming?• Why use FPGAs?
• 10k multipliers per Redback-3 board (@300MHz = 3Tera ops/sec)• We are using 36x6=216 Redback-3 boards in beamformer • Beamforming, ACM and Fringe• Large processed BW – 300MHz with 36 dual pol beams is significant• Large bitwidths – linear up to beamformer output – preserving dynamic range• Lots of IO – guaranteed IO and lots of it• Power efficient
• Why not use a GPU?• Network switch is a killer: 6600 input ports + 6600 output ports
– size, price, power, cabling, ???• Are GPU’s IO constrained for such an application? • How many GPUs per PC?
10 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
Why use FPGAs for Correlator?
• We are using 75 Redback-3 boards in correlator (225TOPS)
• Why not use a GPU?• Network switch: 2500 input ports + 2500 output ports
– size, price, power, cabling, ???• Processing still very regular
• Correlation cells are not that different to beamformers (CMAC with DDR3)• Have to also manage Tied Array outputs, inverse filterbank
• The balance of on-chip resources: logic, RAM, DSP units along with the high IO bandwidth and relatively low power consumption still makes the FPGA a compelling choice in obtaining a cost effective solution for these types of processing applications.
11 |
DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12
My two bits ...• For single pixel feeds
• No beamformer• Correlator is so much smaller• Pretty simple – GPU?
• However, single pixel feeds can be wider BW• Maybe a factor of 10 larger? How many more dishes?
• Issues with GPUs???• Network switch ports and BW• Watching bit widths carefully• Conversion fixed-floating point• Data rates into GPU cards?• Density of processing?• Software tools
12 |
• Issues with FPGAs???• Communications• Programming • Software tools• Fixed interconnections?• Compilation times• Test benches
• ASKAP ~1 Peta Op• Data rates going up
• 10G to 25G• 10GbE pretty slow
• 100GbE needsto become standard
• Density going up• Better cooling
CSIRO ASTRONOMY AND SPACE SCIENCE
Thank you
We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site.
CSIRO Astronomy and Space ScienceDr. Grant HampsonADE Team Leadert +61 2 9372 4647e [email protected] www.csiro.au/cass