Upload
ambrose-perkins
View
218
Download
1
Embed Size (px)
Citation preview
1 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Aquarius Instrument Operations
Aquarius/SAC-DMission Operations Review
February 8 – 10, 2010
2 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Agenda
Peer Review Results Key RequirementsProcesses/Operations OverviewAquarius Command and Control System ArchitectureTesting StatusProcedures/Status/PlansRisksRisk MitigationDocumentation/Status/Plans for completion
3 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
PEER REVIEW RESULTSAquarius Instrument Operations
4 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Recommended Actions
Clearly define the final set of instrument operations procedures Identify procedures to be written Identify what procedures must be verified on the Observatory before
flight
The project has been addressing these since the Peer Review.Concern remains that, due to project demands, personnel may not
be available to rehearse critical procedures
5 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
KEY REQUIREMENTSAquarius Instrument Operations
6 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Key Requirements
Mechanics of the Ground System DesignUse of the Oceans Biology System
ACCS relies on its facilities, networks, security, and software supportUse of CONAE for Mission Operations
The ACCS Scheduler Client and Telemetry Viewer are CONAE deliverablesThe AQ Test-bed also interfaces with SInter, the CONAE commanding tool.ACCS processes and procedures must align with CONAE’s
Instrument Operations - Maximizing Quality and Quantity of Science Data CollectionScience data loss budget of 8%Science data accuracy to 0.2 PSU
Perform Aquarius command planning within flight rule constraintsMonitor Aquarius science and telemetry to determine state of healthPerform Aquarius anomaly handling
7 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Key Requirements Flow-down Summary
All real-time tasks and all SAC-D tasks are the responsibility of the SAC-D Flight Operations Team
As captured in the documents:Aquarius Ground System Level-3 Requirements (AQ-326-0152)L4 AQ Command and Control System & Test Requirements (AQ-426-0197)
8 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
PROCESSESAquarius Operations
9 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Aquarius Instrument Operations Processes
The Aquarius Command and Control System (ACCS) supports:
Aquarius command planningStandard: daily science data downloads; monthly Cold Sky CalibrationsSpecial: science requests such as Look Up Table changesEmergency: anomaly response
Aquarius offline state-of-health analysisReceipt of Aquarius/SAC-D science and housekeeping data from the ADPS. Receipt of Aquarius/SAC-D pass prediction files from the ADPS.Processing of Aquarius/SAC-D housekeeping and Aquarius science data for the purpose of monitoring instrument trends and state-of-health.Publishing of telemetry trends and alarms to Web-based tools to enable off-site health checks.
Aquarius anomaly handlingAquarius instrument anomaly tracking and resolution.
10 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Command Planning Overview
Two standard command planning activities: Perform science data downloads at least once every 12 hours
Plan 2 weeks of downloads every weekCONAE has the flexibility to send a subset of these every few daysCONAE performs C&DH management
Perform a Cold Sky CalibrationNo Aquarius commands; purely a request for the S/P state of “Cold
Sky Calibration”
Science may request other command planning activities such as: Thermal set-point change Adjust the radiometer receive window
11 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Command Planning Steps Daily: Check for science requests
Include any science requests in the upcoming command plan Create a stand-alone command plan if urgent
Weekly: Generate the command plan Two weeks of downloads
Command plan upload to the S/P is base-lined as being two weeks in the future Command plan execution by the S/P is base-lined as being 2 weeks + 3 days in the future
Any science requests Enter command plan into the Scheduler Client
A command plan is an “Action Request” to the Scheduler Client. An “Action Request” is composed of “Basic Action Components”, “Massive Upload Components” and
“Extended Condition Components”. Example Basic Action Component: start downlink/stop downlink command pair Example Massive Upload: Aquarius Flight Software Patch Example Extended Condition Component: X-band transmitter on/off command pair
Scheduler Client is a CONAE deliverable tool Interfaces with the CONAE Spacecraft Operations Planning tool (SOP)
CONAE SOP provides progress reports on Action Request planning via the Scheduler Client: Action Request Received by the SOP Action Request Accepted by the SOP
ACCS verifies reports, then verifies execution via telemetry/downlink file reception
12 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Standard Command Planning Process at a Glance
W T F S S M T W T F S S M T W T F S S M T W T
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Aquarius Commands Planned at the ACCS
Aquarius Action Requests Submitted to the SOP
ACCS Receives Action Report: Action Request Received
ACCS Receives Action Report: Action Request Accepted
13 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Command Planning - Emergency
The emergency command planning process is defined as follows: Utilize emergency channels to notify the CONAE MOC of an Aquarius anomaly:
PhoneEmail
And also enter a ticket into the:Aquarius Reporting and Tracking Tool
Generate the (contingency) Aquarius command plan as per standard procedure, but with accelerated implementation:
CONAE will schedule NEN supportCONAE will process received Aquarius command plans as quickly as possible
14 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Daily Monitoring & Trending Process
Alarm and Error Notification Tool: Verify expected download files have been received
Download file times match commanded download times File sizes match expected file sizes No Cyclic Redundancy Check errors have occurred
Note: Data integrity (ICDS CRC and contiguous blocks) checked automatically Verify No Alarms present
If Alarms present, perform Anomaly Resolution process Web-based graphs
Examine science and telemetry trends on various scales: Daily Monthly, Seasonal Yearly, Mission
Telemetry Viewer Verify Alarms in agreement with Analysis Tool Verify instrument configuration is as expected Review data for software messages
15 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Anomaly Response
The SAC-D and Aquarius teams generated a common anomaly response flow.Aquarius Operations has designed an anomaly process aligned with that flow.
16 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
AS Anomaly Process
17 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
I. Discovery a. CONAE FOT or AQ OPS detects telemetry Out of Limits
Team has procedures and authority to SAFE Aquariusb. AQ OPS detects abnormal science or telemetry trendc. Science detects abnormal science performance
II. Notificationa. If detected by CONAE or Science, party notifies AQ OPS via email/phoneb. AQ OPS notifies AQ GS Manager, CONAE Manager, AQ PI by submitting a TRAC anomaly
ticketIII. Gather Investigation Team
a. AQ OPS asks AQ GS Manager for additional supportb. AQ OPS updates TRAC with criticality, preliminary investigation report; updates TRAC status
to “team formation”. (Publishes TRAC ticket and report to AQ/Science/CONAE management)c. AQ GS Manager asks the AQ PI to authorize funds for JPL supportd. AQ PI asks JPL science liaison to recruit resolution teame. JPL science liaison recruits team, notifies AQ PI, AQ GS Mgr, AQ OPS of team membersf. AQ OPS updates TRAC with team member list; updates TRAC status to “in work”. (Publishes
TRAC ticket and report to entire team).
Aquarius Specific Anomaly Resolution Process
18 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
IV. Develop Solutiona. Technical team uses TRAC to confer, exchange data, develop solution and verification
criteria, exercise with Test-bed.b. When ready, technical team buy-in confirmed by setting TRAC status to “solution
proposed”.V. Obtain Approvals
a. Upon team acceptance, technical team leader updates TRAC status to “solution accepted, technical”.
b. CONAE exercises procedure against Observatory EM; forwards accept/reject to CONAE Mgmt
c. If “reject”, return to step V.d. Management team (AQ GS/AQ PI/CONAE) reviews solution; upon acceptance, AQ PI
updates TRAC status to “solution accepted, management”.VI. Execute and Verify
a. AQ OPS plans commands according to procedure b. CONAE FOT executes c. Full team verifies per step V criteria.
VII. ReportAQ OPS updates TRAC with Final Report and Resolution status: completed/reissued. (If
reissued, go to step V). Aquarius operations documents and procedures updated accordingly.
Aquarius Specific Anomaly Resolution Process
19 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
ARCHITECTUREAquarius Command and Control System
20 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Aquarius Ground System
The Aquarius Ground System is contained within the Oceans Biology Data System The Aquarius Ground System is made up of 2 segments:
The Aquarius Data Processing System (ADPS) The Aquarius Command and Control System (ACCS)
The Aquarius Data Processing System receives Aquarius science data, SAC-D auxiliary data, and other science ancillary data to generate Aquarius science products.
The Aquarius Command and Control System receives Aquarius science data, SAC-D and Aquarius House Keeping Telemetry, and pass prediction files in order to monitor Aquarius health and status, plan commands, and respond to anomalies.
21 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Oceans Biology Data System Support to the Aquarius Command and Control System
Functions/services provided:FacilitiesInfrastructureStaff IT network and securityInherent redundancy (except CPT; hot backup)Use of automationConfiguration ManagementSoftware maintenanceData storageUse of Oceans Biology Data System interface with CONAE
22 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Aquarius Command and Control tools and interfaces
ACCS Architecture
23 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
TESTINGPlans and Status of
24 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
ACCS and Aquarius Instrument Operations Test List
ACCS Ground System TestingSoftware Tools
ComponentInternal Interfaces Functional External Interfaces
Command and Telemetry TestingCommand Database Telemetry DatabaseCross-Tool Verification
Mission Scenarios
25 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Testing Status
COMPLETED TESTSSoftware Tools
Component – baseline completed January 11, 2010Internal Interfaces – baseline completed January 14, 2010Functional – baseline completed January 14, 2010
Command and Telemetry TestingCommand Database – completed June 15, 2009Telemetry Database – completed in instrument I&T at JPL
UPCOMING TESTSCross-Tool Verification – test design meeting scheduled February 15, 2010External Interfaces – will participate in CONAE led tests
Mission Scenarios – under development by the SAC-D and Aquarius Mission System Engineers.
26 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Analysis Tool Alarm and Error Notification Tool Command Planning Tool Web-based graphs Reporting and Tracking Tool
JPL performed the Test-Bed functional test
Component Testing included the:
ACCS Components Tested
27 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Interface testing included the:L0 file deliveryLevel 1A reportsAlarm NotificationsHKT file deliveryText files to Web graphsWeb graph publicationDownload time file delivery
ACCS Interfaces Tested
28 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
ACCS Functional Test DescriptionACCS Functional Test – encompasses component and internal interface tests
100% of Level 4 Requirements verifiedData Monitoring Test:
Correlated science and HKT data files from I&T processed through the full ACCS system Included a DPU memory download
Command Planning TestAll Aquarius commands and known sequences planned
Each singular commandDownlinksLUT programmingDPU memory downloadFlight software patchIntroduced illegal times, sequences, parameters
Anomaly Handling TestSimulated use of Recording and Tracking tool in anomaly resolution process
Included file exchanges, milestones, notificationsSimulated science and instrument team external access
29 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Command and Telemetry Database Tests
Command and Telemetry Database TestingCommand Database
Worked with JPL to verify hex output of CONAE SInter tool matched output of I&T commanding toolExercised all legal commands and wide range of legal and illegal command parameters
Telemetry Database – completed in instrument I&T at JPLThe telemetry database is the telemetry dictionary as captured in Telemetry Viewer scripts.Scripts define how:
Telemetry bytes are extracted from telemetry framesHex values are converted to Engineering UnitsAlarms are applied
Telemetry Viewer pages display hex, EU values and alarmsUltimately compared to values from trusted sources:
Power metersCalibrated sensorsSoftware tools
30 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
AQ/SAC-D GS External Interface and Verification Test Lists
SDGS-GSD-I-0042-a Science data deliverySDGS-GSD-I-0044-a ST TM deliverySDGS-GSD-I-0024-a AR Reception and Confirmation Integration TestSDGS-GSD-I-0025-a Action Reports Request and Response Integration TestSDGS-GSD-I-0028-a Voice Communication Integration TestSDGS-GSD-I-0026-a Orbit Products delivery
SDGS-GSD-V-0002 X band data publicationSDGS-GSD-V-0004 MOC ancillary data production and deliverySDGS-GSD-V-0006 Pass plan generationSDGS-GSD-V-0008 Script validation and compilationSDGS-GSD-V-0012 Execution report generationSDGS-GSD-V-0011 Flight Software replacement
These external interface and verification tests are CONAE-led and will verify Aquarius Ground System Level 3 Requirements:
31 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
PROCEDURESStatus of
32 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Key Aquarius Operations Procedures
The key procedures for Aquarius instrument operations are:
Aquarius Command PlanningAquarius Monitoring and TrendingAquarius Anomaly ResponseACCS Configuration Management
And the individual standard and contingency instrument procedures.
33 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
ACCS Procedures: Status and Plans for Completion
Key ACCS procedures are in preliminary form.Extracted from the ACCS Functional Test Procedure.Exception: Configuration Management Procedure
This procedure provides the steps for performing configuration managementIt also identifies what items are to be maintained under CM:
Processes, ProceduresTool software (including CONAE deliverables)Flight Software/patchesCommand and Telemetry scripts (databases)Tests and test reports
ACCS Procedures must be updated in lockstep with the Functional Test. This is expected with:
Expansion of Functional Test to encompass Joint TestsCorrections to Functional Test as Joint and Mission Tests progress
34 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Aquarius Instrument Procedures, Standard and Contingency
35 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Individual Aquarius procedures: Have been drafted by the instrument team as flowcharts (where
needed) with command sequences and telemetry responses identified. AND/OR Exist as ATLO scripts/procedures
Work to be done: The Aquarius team should come to consensus on which procedures
are operations-critical. The SAC-D Flight Operations Team and the Aquarius Instrument Ops
Team should rehearse/observe these critical procedures.
Instrument Operations Procedures Status
36 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Aquarius flows and ATLO scripts will be transferred to operations procedures The transfer steps from ATLO scripts to operations procedures are:
Correlate specific ATLO/I&T scripts to each operations procedure Parse the scripts by decision point. ACCS procedures direct the use of the
CONAE Aquarius procedures. Thus: ACCS procedures include the decisions within a given ATLO script CONAE procedures include the steps between decisions
Write the ACCS procedures and the CONAE procedures (scripts) according to the parsed ATLO/I&T scripts
Rehearse/observe the ACCS and CONAE Aquarius procedures against the Observatory.
AND/OR Rehearse the ACCS and CONAE procedures in mission scenarios.
Generating Instrument Operations Procedures
37 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
ACCS vs CONAE Procedures
37
Two types of instrument procedures:ACCSCONAE Flight Operations
ACCS procedures are documents in step-by-step formatAll decision points occur within the ACCS procedures
CONAE procedures are scriptsCONAE Aquarius scripts are written in the Sinter scripting languageThe scripts consist of:
Aquarius commands established in known sequencesAssociated telemetry responsesAny other real-time activities required
Examples:Inserted delays between commandsSpecial data loggingInformation to note before proceeding
38 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
RISKS/CONCERNS
39 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Risk Assessment
After an extensive risk/concern discussion during the Aquarius Instrument Peer Review and follow-up discussions with the Aquarius team, two risks/concerns remain.
The two risks/concerns are listed by priority from the instrument operations perspective.
40 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Risk
RISK: Personnel availability may be too limited to allow for implementing the existing command sequences as procedures and rehearsing them.
IMPACT: The Operations Team would then be inadequately trained for Aquarius operations.
Mitigation: Project is planning a workshop in Bariloche in the April timeframe to rehearse critical procedures.
41 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Concern
CONCERN: Awaiting Scheduler Client delivery.
IMPACT: Unable to develop adequate Aquarius command planning procedure in time for interface tests and scenarios. Interface testing of the command planning link could be delayed.
Mitigation: Scheduler Client User’s Guide and ongoing conversations with developers have provided baseline information for ACCS tool and procedure development.
42 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
DOCUMENTATIONDescription and Status of
42
43 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Documentation/Status
Name Status Version Date
Aquarius Ground System Level-3 Requirements (AQ-326-0152)
Signed
ACCS Level-4 Requirements (AQ-426-0197)
Signed January 19, 2010
Aquarius Command and Control System Software Management Plan (AQ-316-0239)
Ready for Signature February 25, 2009
Aquarius Ground System Integration Test & Training Plan (AQ-336-0244)
Revision January 18, 2010
SAC-D Ground System to Aquarius Ground System ICD (AS-336-0151b)
Review July 14, 2008
Aquarius Ground System to Instrument Team Anomaly ICD (AQ-336-0522)
Ready for Signature December 7, 2009
Aquarius Science Team to Ground System ICD (AQ-336-0243)
Ready for Signature December 7, 2009
Aquarius Flight Operations Plan (AQ-316-0157)
Ready for Signature December 3, 2009
ACCS Operations Handbook Preliminary December 28, 2009
Aquarius Standard Procedures Preliminary January 4, 2010
Aquarius Anomaly Procedures Draft
ACCS User’s Guides Final January 6, 2010
44 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
ACCS – CONAE Rec/Dels - Status
Nº: TITLE ACCS Rec/Del
STATUS
1 CAS SCL Reference Manual Rec Received
2 CAS SCL User’s Guide Rec Received
3 Aquarius Telemetry Descriptions Del PendingDue: April 23, 2010
4 Aquarius Command Descriptions Del PendingDue: April 23, 2010
5 SAC-D HKTM Dictionary (TBC) Rec Pending
6 Aquarius Operating Procedures (TBC)
Del PendingDue: June 10, 2010
7 Scheduler Client User Manual (TBC)
Rec Received
8 TM Viewer User Manual (TBC) Rec Received
9 Aquarius Planning Restrictions (TBC)
Del PendingDue: April 23, 2010
10 Aquarius TM Pages and Decoding scripts (TBC)
Del Sent
11 Sinter Command Tool Rec Received
12 Telemetry Viewer Tool Rec Received
13 Scheduler Client Tool Rec Received
45 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Instrument Team – ACCS Rec/Dels - Status
Nº: TITLE ACCS Rec/Del
STATUS
14 AQ Flight Software User’s Guide
Rec Received
15 AQ Instrument Operations Handbook
Rec Received
16 Aquarius Telemetry Dictionary
Rec Received
17 Aquarius Command Dictionary
Rec Received
18 Aquarius Test-Bed User’s Guide
Rec Received
19 Aquarius Test-Bed Maintenance Manual
Rec Received
46 of XXFebruary 8-10, 2010
Section 09 – Aquarius Command and Control SystemMission Operations Review
Summary
The Aquarius instrument operations team and the ACCS tools are ready to move into the operations testing phase.
The testing, process development and procedure development is working in conjunction with SAC-D ground system and Aquarius instrument testing.