Upload
dale-cooper
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Digital Packaging Processor - Overview
Gordon Hurford
Nov 7, 2011
EOVSA Technical Design Meeting - NJIT
Digital Packaging Processor
• Role of DPP
• Overall assumptions and priorities
• Interface Overview
• Tasks and algorithms
• Hardware/software Implementation
-----------------------------------------------------------
• DPP Interface Details
Data System Approach Fundamental Drivers
• Very limited software resources• Non-trivial data rate and volume • Automated analysis pipeline for efficient observing• Must have science-useable system in place by
September 2013• Data products to be readily useable by broader solar
community– Data products with preset parameters– Data products with user-selected parameters– Tools and support for experienced users
Data System Approach Implications
• Monomode observing• Calibrated data archived in application-specific databases• Reliance on existing software packages
– Miriad package for calibration & mapping– RHESSI SolarSoft package for user interface and data product
display– RHESSI database model
• Err on side of over-rejection of data • Limited initial support for ‘nice-to-have’ options• Limited initial support for calibration refinements• Limited support for non-solar applications
Data System Assumptions
• All information required for data analysis is written by the DPP to the Interim Data Base
• Engineering data acquisition, archiving and display is the responsibility of the ACC, and is “largely” decoupled from science data.
Nomenclature
• Data frame = Interval representing data from one correlator cycle (20 ms, ~4000 channels with 500 MHz range)
• Spectral frame = Data corresponding to a complete frequency-agile cycle (nominal 1 second, 10s to 100’s of ‘science channels, 18 GHz rang) Corresponds to a state frame.
• Scan: Observing interval within which target and frequency cycling pattern is unchanged
Role of Digital Packaging Processor
• To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base
• Real time, irreversible processing
DPP Interface Overview
DPP
State Frame
ACC
Correlator
Start / End Scan Commands
Scan-independent Calibration Parameters
Scan Parameters
Frame parameters
Frame status report
<P>, <P2>,
Correlations
Interim Data Base
Miriad
format
Internal RFI
Database1 s timing tick
0.02 s timing tick
RFI
results
DPP Task Timing
• Occasional – non operational– Accept, store and preprocess calibration parameters
• Scan initiation– Accept, store and preprocess scan-specific parameters
• Data frame (20 ms) – filter, and frequency-average correlator output
• Spectral frame (1 s) – Assemble, pre-calibrate, reformat and write data to Interim
database
• TBD – Format results and write to RFI database
DPP – Stage 1 ProcessingEvery data frame (20ms)
• Evaluate kurtosis data to identify RFI-affected subbands as a function of frequency only.
• Save RFI statistics• Combine with pre-flagged subbands to
generate a “destination vector” for each subband
• Apply complex gains at subband level ???• Average subband data into spectral channels • Save 1st 3 moments of averages ???
DPP Stage 2 ProcessingEvery spectral frame (1s)
– Convert antenna-based flags (e.g. slewing) from state frame to baseline-based, frequency-independent flags
– Apply time-independent complex gains if available– Apply baseline corrections– Apply non-linearity corrections– Correct for attenuator settings– Correct for spectral simultaneity
• Miriad format this is no longer optional
– Convert visibility, uv and analysis-relevant state-frame data to Miriad-compatible format
– Write spectral frame to IDB– Report DPP status to state frame
DPP - Implementation
• Original concept was to follow FASR plan for a cluster-based DPP
• Estimate processing requirements for EOVSA DPP at ~100 MIPS = 1/60 of FASR requirements
• Implementation will be based on a single multi-core machine
• Software organization will be compatible with migration to a cluster if necessary
DPP Software Architecture
ACC State Frame Correlator IDB
Coordination Task
I/O, data assembly, no processing per se
Header Processing
Stage 1 Processing
Stage 2 Processing
Pointers within shared memory
Conventional, time-independent processing tasks
C1
C2 C3, C4 C2
Cn = core within a quad core processor or nodes in a cluster
DPP
RFI database
ParameterProcessing
C2
DPP Status• Software architecture and tasks identified
• Detailed definition of interfaces is underway
• EOVSA to Miriad format conversion being tested with FST data– (Fortran 77 for Miriad compatibility)
• Next:1. Complete definition of interfaces2. Code Stage 1 tasks (GH)
• Evaluate timing requirements3 Code Coordination task (JM). 3. Detailed definition of processing algorithms 4. Code of Stage 2 tasks
5. Machine selection and purchase • Development platform?
• Goal: Functional DPP to support prototype testing