Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 1
EVLA Monitor & Control Software PDR
E2E Interfaces: Observation Scheduling
Complaints to: Boyd Waters
John Benson, Barry Clark, Tim Cornwell, Rich Moeser, Peggy Perley , Bruce Rowen, Kevin Ryan, Bill Sahr, Ken Sowinski, Boyd Waters
Originally, Bill Sahr was going to present this topic, but it really makes sense to have an e2e person do it.
The material discussed in this presentation is the result of the work of the EVLA M&C Computing group, and the e2e Team. But I probably got some stuff wrong and left some stuff out, so complaints to Boyd Waters.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 2
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 2
What is “e2e”?
• End-to-End Data Management– Capture and Preserve data… – …in a computer-useable format…– THROUGHOUT THE PROJECT LIFECYCLE!
• Motivation:– Automate where possible– Make Radio Interferometry EASIERPOSSIBLE
Also, there’s been lots of talk about the traditional, “roach motel” model of data archival: data checks in, but never checks out! So we’d like to provide meaningful access to the data that we (NRAO) already have.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 3
e2e Architectural Diagrams
5. Observation Scripting Toolkit
Real-TimeScheduler
ControlScripts
10. Archive Toolkit
11. Pipeline Toolkit
3. Proposal Submission Toolkit
Submit Proposal
SubmitterIntentionalProposal
Generate
ObservingScripts
Test
Engineer
ObservingSystem
Monitor Data
Visibility Data
ArrayOperator
ImagePipeline
Visibility Data
Data
Bus
&
Storage
Observation
M o n i t o r
Search
Archive
?Query
!Results
Monday, November 26, 2001 bwaters - page:1 of 1
End-to-End: More Detail
9. Real-TimeObserving
Toolkit
NormalizedProposal
Scenarios
project
Control
Scripts
4. Proposal Management Toolkit
$
VISIOCORPORATION
PrioritizedProposal
-or-
PrioritizeProposals
$
V I S I OCORPORATION
PrioritizedProposal
NormalizedProposal
NormalizeProposals
NormalizedProposal
Intentional
Proposal
Scriptor
TAC
Data
Wrangler
7. Observation Evaluation Toolkit
EvaluateObservation
ObservationEval. ReportEvaluator
Researcher
ImagesVisibilities
Ancillary data reportsetc.
13. Calibration Source Toolkit
CalibrationData
Calibration
Editing
Moni tor Data
Visibility Data
CalibrationScientist
6. Telescope Simulation Toolkit
Simulate
Telescope
ProposalModeller
conditions
CalibrationData
Monitor Data
Visibility Data
Observ ing
Scripts
ObservingConditions conditions
8. Observation Scheduling Toolkit
Dynamic
Scheduler
conditions
Scenarios
project
Calibration Data is aspecific kind of
“conditions” data.CalibrationData
Scheduler
Visibility Data
CalibrationData
CalibrationData
Monitor Data
ImageControl Scripts -“as observed”
A “Scenario” is anordered list of Projects
The Real-TimeScheduler produces aQueue of Control
Scripts (e.g. crd files)from a Scenario.
The Observing System providesfeedback to the Real-Time Scheduler
by reporting the Control Scripts “asobserved”. The Observing Systemmay also raise events via Monitor
Data. TBD.
5/15/2002 Boyd Waters <[email protected]> Boyd Waters 3
Here is our obligatory e2e architectural diagram.
I don’t expect you to follow this; the point here is that we have thought about the process of observing with NRAO instruments as data work-flow. We have identified major steps along the work flow, and are building toolkits for many of these major steps.
The next two slides are slightly larger versions of this diagram.
(You can get a (theoretically) printable version of this diagram in a larger format from the e2e web site at http://www.nrao.edu/e2e )
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 4
5. Observation Scripting Toolkit
Real-Time
3. Proposal Submission Toolkit
Submit Proposal
SubmitterIntentionalProposal
GenerateObserving
Scripts
Data
Bus
&
St
NormalizedProposal
Scenarios
project
4. Proposal Management Toolkit
$
VISIO
CORPORATION
PrioritizedProposal
-or-
PrioritizeProposals
$
VISIOCORPORATIO
N
PrioritizedProposal
NormalizedProposal
NormalizeProposals
NormalizedProposal
IntentionalProposal
Scriptor
TAC
DataWrangler
7. Observation Evaluation Toolkit
EvaluateObservation
ObservationEval. ReportEvaluator
13. Calibration Source Toolkit
CalibrationData
CalibrationEditing
Monitor Data
Visibility Data
CalibrationScientist
6. Telescope Simulation Toolkit
SimulateTelescope
ProposalModeller
conditions
CalibrationData
Monitor Data
Visibility Data
ObservingScripts
ObservingConditions conditions
8. Observation Scheduling Toolkit
DynamicScheduler
conditions
Scenarios
project
Calibration Data is aspecific kind of“conditions” data.Calibration
Data
Scheduler
Visibility Data
A “Scenario” is anordered list of Projects
Here’s the top half of the e2e Architecture Diagram.
Roughly, the right side of the diagram models interactions with hardware (e.g., the telescope, a weather station), and the left side of the diagram models interactions with people.
The data flow starts at the top-right of the diagram, with the Proposal Submission; Submissions turn into Approved Proposals, which are processed to obtain a Project that contains observation instructions (in a scripting language).
Sets of Projects are examined by the Dynamic Scheduler, which orders the Projects according to observing logic based on current conditions and observation constraints (e.g., good weather for high-frequency observing).
An ordered set of observing instructions – called a Scenario – is passed to a “Real-Time Scheduler” that turns the observing instructions (which areexpressed in a human-readable scripting language) into more-detailed instrument instructions to be executed by the telescope. I’ll talk about this more in a later slide.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 5
Real-TimeScheduler
ControlScripts
10. Archive Toolkit
11. Pipeline Toolkit
TestEngineer
ObservingSystem
Monitor Data
Visibility Data
ArrayOperator
ImagePipeline
Visibility Data
&
Storage
ObservationMonitor
SearchArchive
?Query
!Results
Monday, November 26, 2001 bwaters - page:1 of 1
End-to-End: More Detail
9. Real-TimeObserving
Toolkit
Scenarios
ControlScripts
Researcher
ImagesVisibilitiesAncillary data reportsetc.
13. Calibration Source Toolkit
CalibrationData
CalibrationEditing
Monitor Data
Visibility Data
CalibrationScientist
8. Observation Scheduling Toolkit
DynamicScheduler
Scenarios
projectScheduler
CalibrationData
CalibrationData
Monitor Data
ImageControl Scripts -“as observed”
A “Scenario” is anordered list of Projects
The Real-TimeScheduler produces aQueue of ControlScripts (e.g. crd files)from a Scenario.
The Observing System providesfeedback to the Real-Time Schedulerby reporting the Control Scripts “asobserved”. The Observing Systemmay also raise events via MonitorData. TBD.
Here’s the bottom half of the architecture diagram, which shows the punch-line: the Image Pipeline and the Archive Query.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 6
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 6
e2e/AIPS++ pipeline results
Tim Cornwell wrote this prototype Image Pipeline as a production-rule system implemented as GNU Makefiles driving AIPS++.
This is what we want: every observation from the VLA (and other NRAO instruments) will result in a set of images that have been built automatically. Instead of a Radio-Astronomy Interferometer “black belt” sitting down at an AIPS++ workstation and reducing the data, the image pipeline will try to do the same thing. Can a computer be as smart as Chris Carilli? No. But if the system is supplied with sufficient information about the observation, it can make reasonable choices about the data-reduction process to produce these default datasets.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 7
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 7
e2e:Calibrator Selection
• Prototyped Java interface• Will do user testing very
soon• Expect to deploy soon• Make catalog available in
AIPS++ for processing• Also make available as an
NVO cone search – more later
Honglin Ye wrote this Calibrator Selection Tool prototype with the help of John Benson, input from Greg.
The point here is that we are building tools all along the workflow chain, from proposal submission all the way to data reduction, to ensure that well- formed, complete data is entered into our data archive. A unified data model makes automated pipeline processing possible.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 8
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 8
e2e:Proposal Submission…
Example:
Gemini Proposal
Tool
This is NOAO’s Gemini Telescope “Phase I” Proposal Submission Tool. It’s an example of the kind of submission tool we’re thinking about.
The point here is that, with a some sort of forms-based editor, we can prompt the user for information about the observing proposal such that we obtain complete, structured information. We hope. In theory. Well, the editor will help…
In this screenshot, you can see the Gemini proposal tool as a Java application in the upper-left window. In the window on the right side of the screen, I’veopened the resulting proposal in Notepad (a Windows text editor). You can see that the proposal is simply an XML file. In the bottom-middle window, I’ve opened this same proposal in a popular XML editor to emphasize the document structure.
Writing the proposal as XML allows computer systems to interpret the structured data. We could use some other form like structured ASCII tables; XML is the fun thing to use at the moment. There are plenty of XML parsing tools available.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 9
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 9
e2e-EVLA Interaction Observation
SchedulingSoftware
Monitor &ControlSystem
This is the data-flow diagram that was in the documentation submitted to the National Science Foundation.
I don’t expect you to read all the labels here. The point is that the green stuff is the primary responsibility of the Data Management Group (mostly the e2e Project), while the red stuff is the responsibility of the EVLA Project.
The major interaction between these teams, discussed in this presentation, is between the “Observing layer” and the “Monitor and Control” systems.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 10
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 10
e2e-EVLA II
Diagram Design:B. Clark
Barry Clark presented this diagram on the first day of the EVLA M&C PDR. I’ve added the colors consistent with the previous diagram, to emphasize the similarity between these two diagrams.
This diagram emphasizes that the piece of the “observing layer” that interacts with M&C is the Observation Scheduler.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 11
Scheduling Phases
Real-TimeScheduler
ControlScripts
TestEngineer
ObservingSystem
Monitor Data
Visibility Data
ArrayOperator
ObservationMonitor
9. RemoteObserving
Toolkit
Scenarios
ControlScripts
8. Observation Scheduling Toolkit
DynamicScheduler
conditions
Scenarios
project
Calibration Data is aspecific kind of“conditions” data.
Scheduler
CalibrationData
Control Scripts -“as observed”
5. Observation Scripting Toolkit
GenerateObserving
Scripts
NormalizedProposal
project
$
VISIOCORPORATIO
N
PrioritizedProposal
-or-
Scriptor
Monitor and
Control
The data flow starts at the top-right of the diagram, with the Proposal Submission; Submissions turn into Approved Proposals, which are processed to obtain a Project that contains observation instructions (in a scripting language).
Sets of Projects are examined by the Dynamic Scheduler, which orders the Projects according to observing logic based on current conditions and observation constraints (e.g., good weather for high-frequency observing).
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 12
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 12
Dynamic Scheduling
Structure the Observation……so that we can…
Respond to Events on a Short Time Scale
Changing conditions (e.g. weather)Targets of Opportunity (e.g. GRBs)
…
èBlock-Based Scheduling
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 13
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 13
Observing Block
Preamble– The telescope setup for this
block– Examples:
• Calibrators• LO setup
“Post-amble”– Cleanup, book-keepingObserving BlockObserving Block
Pream
ble
~20(?) minutes nominal“P
ost-amble”
20 minutes is a typical calibration duty cycle. But the blocks can vary in size.
The internal structure used by the observing block is To Be Decided.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 14
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 14
Observing Block Constraints
• “Run this block until the calibrations converge”
• “I can’t run unless the previous block has run successfully”
• “I MUST run at 16:42:30 GST on 30 May 2002”– (fixed scheduling is dynamic scheduling with
time-domain constraints)
These are EXAMPLES, not REQUIREMENTS.
Requirements here have yet to be gathered, but we can collect such and use the requirements to get a better idea of what the observing blocks will need to express.
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 15
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 15
Block-Based Scheduler
Telescope sees ONE BLOCK AT A TIME:Block Queue
Telescope
“ready for next block”“here it is”
Observing BlockObserving Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
Implications:
•Simplifies the telescope state data
•Telescope reports block execution status back to the block queue
•All “observing logic” is maintained by the Block Queue
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 16
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 16
Block Execution
Observing BlockObserving Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
Input Queue:
Execution:Observing Block
Observing Block
Pream
ble
“Post-amble”
3
Observing BlockObserving Block
Pream
ble
“Post-amble”
1
Observing BlockObserving Block
Pream
ble
“Post-amble”
1
Observing BlockObserving Block
Pream
ble
“Post-amble”
2Observing Block
Observing Block
Pream
ble
“Post-amble”
3
Observing BlockObserving Block
Pream
ble
“Post-amble”
2Observing Block
Observing Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
2Observing Block
Observing Block
Pream
ble
“Post-amble”
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 17
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 17
Block Execution: Detail
Input Queue:
Execution:Observing Block
Observing Block
Pream
ble
“Post-amble”
3
Observing BlockObserving Block
Pream
ble
“Post-amble”
1
Observing BlockObserving Block
Pream
ble
“Post-amble”
1
Observing BlockObserving Block
Pream
ble
“Post-amble”
2Observing Block
Observing Block
Pream
ble
“Post-amble”
3
Observing BlockObserving Block
Pream
ble
“Post-amble”
2
Archive:
Observing BlockObserving Block
Pream
ble
“Post-amble”
4Observing Block
Observing Block
Pream
ble
“Post-amble”
5Observing Block
Observing Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
MeasurementSet
MeasurementSet
MeasurementSet
MeasurementSet
Observing BlockObserving Block
Pream
ble
“Post-amble”
3
Observing BlockObserving Block
Pream
ble
“Post-amble”
1Observing Block
Observing Block
Pream
ble
“Post-amble”
2Observing Block
Observing Block
Pream
ble
“Post-amble”
3Observing Block
Observing BlockP
reamble
“Post-amble”
4Observing Block
Observing Block
Pream
ble
“Post-amble”
ready ready ready ready
Pream
ble 2Observing Block
Observing BlockObserving Block
Pream
ble
“Post-amble”
Observing BlockObserving Block
Pream
ble
“Post-amble”
2
ok ok ok ok
readyFailed!
?
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 18
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 18
Block Templates
• Debug blocks• Template blocks• “Default” blocks
e2e <-> EVLA Observation Scheduling (EVLA M&C SW PDR)
5/15/2002
Boyd Waters <[email protected]> 19
May 15, 2002 EVLA Monitor & Control Software PDR
Boyd Waters 19
Next Steps
• Gather and Codify Requirements– Observing Block constraints
• M&C óObserving System Interaction– Formal Model
• Observing System Scripting– How instrument commands and observing
constraints are expressed