Upload
gabriella-higgins
View
213
Download
0
Embed Size (px)
Citation preview
ETA Data Processing
Steve EllingsonLow Frequency Software Workshop – Chicago – Aug 10, 2008
RFI Environment: Bad But Manageable
TV
Ch 4
Ch 3
Ch 2
Ch 5
Ch 6
KEY POINT:
Can observe here – but need good linearity and narrow channelization
~ 100 s of noise-limited sensitivity using > 95% of contiguous 5
MHz band around 38 MHz
Search Range
(29-47 MHz)
Primary threat to linearity – receiver design challenge
Often, but not always
possible.
In-Band RFI Challenges
Wideband junk
Wideband junk
Wideband junkS
elf-
Gen
erat
ed (
PC
)
6-m
Am
ateu
r R
adio
Ionospheric enhancement
Ionospheric enhancement
Cit
izen
’s B
and
, oth
er H
F
NC
Sta
te P
olic
e
Impulsive noise starts to become a problem at
resolutions ~100 s
Galactic background clearly visible underneath sparse RFI
Self-RFI is a relatively minor problem
Offline ProcessingUp to 200 x 1GB (17s) Files7+7 bit complex @ 7.5 MSPS
Data integrity check
Create raw spectragrams
Create baseline spectragrams
Calibrate spectragrams
RFI mitigation
Incoherent dedispersion
Integrate time series
Manual inspection for pulses
Data transfer errors (rare but significant)Sample value histograms / clipping (checking for intermittent RFI swamping)
1K FFT (yields freq-time resolution 7.324 kHz x 136.5 s)Integrate to 8.738 ms (for Crab GP search; also, suppresses impulsive RFI)
Updated every ~7.5 minutes (timed to track Galactic background variation)using spectragrams hand-picked for low RFI
Remove frequency response; Linear interpolation between baseline spectragrams to track Galactic background
Three passes of “plinking” (replacing extreme values with median values):(1) Time-frequency pixels one at a time [th1](2) All freq pixels for a given time, triggered on total power thresholding [th2](3) All time pixels for a given freq, triggered on integrated spectrum thresholding [th3]
Operates on 7.324 kHz x 8.738 ms spectragramsw/o interpolation
In effect, smoothing to expected resolution of scattered-broadened pulse (We use 498 ms for Crab)
Difficult to automate due to RFI and time-domain baseline fluxuations
Possible Incoherent combining of polarizations / dipole signals
Example of RFI Mitigation
Before
3.75 MHz
3600 sAfter
th1 = 0.40 (time-freq)th2 = 0.03 (time)th3 = 0.02 (freq)
< 1% pixels plinked
3.75 MHz
= 7.324 kHz = 498 ms
= 7.324 kHz = 498 ms
38.0 MHz
38.0 MHz
Plotting power; Extreme values in this plot
are typically within a few % of mean
Example Simple Pulse Detection (old toolchain – sorry!)
RFI Mitigation, DM = 56.791 pc/cm3
No RFI Mitigation, No Dedispersion
RFI Mitigation, No Dedispersion
5
5
Duration ~ 2 sPeak DM = 56.791 pc/cm3
Est. flux ~ 876 Jy
DM sweep
Example of Relatively Good RFI Conditions
No RFI Mit, No Dedispersion
RFI Mit, No Dedispersion
RFI Mit, DM = 56.791 pc/cm3
Off-Line Processing Summary Data processing
– Operates on coherently-sampled voltage data (dipoles or beams)– 1 hour of observation is typically about 1 TB raw (data constipation!)– 100% new C-language source code / tool chains– Nothing special for computing (tend to use existing PC cluster to minimize
amount of data transfer)
Lessons Learned (from the perspective of a dispersed pulse hunter)– Value of extensive diagnostic “pre-analysis” to identify problematic data:
Smallest fraction of FLOPS, but greatest fraction of person-hours Weak RFI (histograms over many domains & resolutions) Spurious ionospheric conditions Consistency with sky model (“Error” in time-varying continuum small?) Repeatability (is today within a few tenths of percent of yesterday?)
– Seems to be more productive to reobserve than to try to salvage “subtly problematic” data, even if only portions look bad.
By our standards, we end up throwing out about ½ of data that initially looks good
– Extent of site multipath (self-inflicted), impact– Antenna & cable dispersion, impact– Value in keeping coherent dipole voltage data, despite logistics, to maximally
facilitate reprocessing
9
ETA A/D-RX Board
Analog SignalFrom ARX
120 MHzSystem Clock
Parallel(4b + CLK)LVDS toRCC:
7.5 MSPSI7+Q7, plus in-band data(240 Mb/s)
29 47
3.75
18
Altera Stratix EP1S25 25,560 LEs80 9-bit DSP blocks1,944,576 memory bits
LVDS direct-connects via Mictor connector
12-bit,120 MSPS
digitization
1
Reconfigurable Computing Cluster (RCC)
• 16-node “Virtual FPGA”
• Each node is a development board with Xilinx XC2VP30 FPGA
• Edge nodes (“E”) catch streaming LVDS from digital receivers
• 3.125 Gb/s Infiniband-like interconnects
• Center nodes (“C”) route between RCC nodes & push results to PC cluster
• PPCs internal to FPGAs run Linux, perform GPP-type functions
Xilinx ML310
1
RCC “All Dipoles” Mode
240 MB/s aggregate
(60 MB/s per PC)
Coherent time series, 3.75 MHz BW
Acknowledgements:
John Simonetti PhysCameron Patterson CpE
Zack Boor PhysSean Cutchins PhysKshitija Deshpande EEMahmud Harun EEMike Kavic PhysAnthony Lee EEBrian Martin CpEWyatt Taylor EEVivek Venugopal CpE
Pisgah Astronomical Research Institute
AST-0504677
Supported by:
http://www.ece.vt.edu/swe/eta