Overview of: Mission and System Design Verification and Validation · 2010-09-17 · Mission and System Design Verification and Validation SDOE 633 Presented by ... validation plan

Overview of:Mission and System Design

Verification and ValidationSDOE 633

Presented by

With our Academic Partner

SDOE 633 Course Overview

The Challenge...

2

The US Forest Service needs a more effective means to detect and monitor potentially dangerous wildfires

Turning this...

Into This...Mission Operations Element

ESA’s Space Operations Center. Courtesy ESA

Communication Architecture. US Fig. 15-11.

Subject Element: Wildfires

Space Element

Orbits & TrajectoriesLaunch Element

3.1 Form shall be...3.2 Fit shall be...3.3 Function shall be...4.1 Form shall be verified by...

System Requirements Document

Valid Baseline Requirements


The Challenge...

3



Valid Baseline Requirements

Turning this... ...into this

While Managing all of this...

VerificationClosure Notices

Test & Verification Requirements (TVRs)

Procedures

Facilities &Equipment

Personnel

Master Verification Plan


Course Description

SDOE 633 provides hands-on opportunities to apply key principals of space systems engineering. Students are given a set of customer expectations in the form of broad mission objectives. Using state-of-the-industry mission design and analysis tools, students apply systems engineering processes to define top-level system requirements, design key elements, and conclude with a system design review. In V&V, participants experience system realization processes first hand by integrating, verifying, validating, and delivering the shoe box–sized EyasSAT satellite. From the part-level to the system level, participants implement a rigorous assembly, integration, verification, and validation plan on space hardware/software applying “test like you fly, fly like you test” principles.

4


System Engineering Life-cycle Processes

5

Define,Goals &

Objetctives

DevelopConcept ofOperations

Engineer &Manage

Requirements

CreateFunctional

&Physical

Architectures

DESIGN

IntegrateImplement

Verify &Validate

Transition

NEEDSCAPABILITIES

REALIZEMANAGE

Build-to/Code-to Baseline

Mission

Baseline

System

Baseline

Functional

Baseline

Design-to

Baseline

As-built

Baseline

As-deployed

Baseline

We’ll use this model of Systems Engineering Processes to guide our discussionThe purpose of this course

is to help bridge the gap between the “left side” and “right side” of the SE “V”


SDOE 633 Part A• Given: a brief Request for Proposal or Announcement of

Opportunity...• Using: Integrated Mission Design Spreadsheet Tool • Satellite Tool Kit• Complete the Conceptual Design for an entire mission...

– Orbit design– Spacecraft design– Launch vehicle selection– Operations Concept– Driving Requirements and Key Performance Parameters

6




SDOE 633 Part A Objectives

• Cultivate a better understanding of the overall space systems engineering process– Technical processes, tools and information available– Interpersonal skills and distributed collaborative efforts

• Enhance space system engineering skills—system engineering management, technical integrity and technical leadership

• Integrate all elements of a successful mission • Explore key trajectory constraints on space mission design• Establish a process to refine requirements and define parameters to

meet mission objectives at acceptable cost and risk• Practical application of the information and processes in a non-

threatening environment• Promote system-thinking by all participants

Start with a blank sheet of paper and develop a detailed Mission Concept Review (MCR) to meet a set of broad objectives

7


SDOE 633 Part A Description• A learning laboratory for participants to explore and use the

competencies learned on the job and in the Space Mission Analysis and Design, Designing Cost-Effective Space Missions, Understanding Space, and Human Spaceflight courses and workshops – Interpersonal skills– Space system engineering– Collaborative design techniques

• During the design exercise, the team…– Determines the requirements from the customer– Develops an appropriate mission concept to meet the requirements– Completes the conceptual design of a mission and the associated

space and ground assets to provide the necessary products or services to identified customers and users

8


SDOE 633 Part A Description (cont’d)

• Participants… – Exercise the Space Mission Analysis and Design processes and

use space system engineering fundamentals to create a reasonable science mission opportunity

– Learn the ties between customer requirements, mission and system design and lifecycle cost

– Develop an instinct for the technical concepts and parameters to adjust to make their newly conceived mission more cost-effective

– Learn to think and work by way of a practical, end-to-end example – beginning with customer needs to create a viable space-based product or service

• The design exercise, hence, serves as a concrete and practical means to apply and test system engineering techniques in a non-threatening, real-life environment

9


SDOE 633 Part A Flow

• Getting Started– Overview of the SE and conceptual mission design processes– Introduction to Space Space Systems Engineering– Conceptual Mission Design– Review of Science Mission & System Design & Operations– Introduce Design Exercise “Announcement of Opportunity”– Firesat case study example– Team assignments– Introduction to Space Mission Analysis and Design (SMAD) and STK software tools

using Firesat example– Begin concurrent design sessions (requirements definition)

• Getting Down to Work– Firesat case study example (cont.)– Concurrent design sessions and status reviews

• Finishing Up– Concurrent design sessions– Working lunch– Team final presentations and feedback (after lunch)– Wrap-up/Critique

10


Design Tools...The SMAD Spreadsheet

• Integrates 100’s of equations as well as cost and operational complexity models from Space Mission Analysis and Design text to create a fully-integrated Excel-based system modeling tool for rapid trade studies– Shallow learning curve allows students with basic background in

space mission design to quickly begin analyzing trade-offs between

11


SMAD Spreadsheet Navigator

12


SMAD Spreadsheet Example Output

13


SDOE 633 Part B

14

Verification Planning

EyasSAT Exercises

REQUIREMENT VERIFY

METHOD

(LEVEL)

EVENT(S) SUCCESS CRITERIA VALID REQ’T

?

VERIFY

STATUS?

VERIFY

STATUS?

VERIFY

STATUS?

VERIFY

STATUS?

VERIFY

STATUS?

COMMENTS

3.1 System Characteristics: EyasSAT System characteristics shall be as refined by the following:

Inspection

(SYSTEM)

System Acceptance Review

If verification of all characteristic requirements have been successfully completed.

3.1.1. System Definition: EyasSAT system major components shall include the following: (1) Structure & Integration Subsystem (SIS), (2) Electrical Power Subsystem (EPS) Module, (3) Data Handling Subsystem (DHS) Module, (4) Communication Module (Comm), and (5) Attitude Determination & Control Subsystem (ADCS) Module, LED Test Module assembled as per specifications

Inspection

(SYSTEM)

Subsystem Baseline Physical Inspections AND System Baseline Physical Inspection

If all specified major components are included

3.1.2. System Mass: Total system mass shall not exceed 3.0 kg, Subsystem mass is allocated as follows:

Inspection

(SYSTEM)

System Baseline Physical Inspection

If system mass does not exceed 3.0 kg.

3.1.2.1 SIS Mass: SIS mass shall not exceed 1.5 kg.

Inspection

(SUBSYSTEM)

Subsystem Baseline Physical Inspections

If SIS mass does not exceed 1.5 kg

3.1.2.2 EPS Mass: EPS Module mass, including LED Test Module, shall not exceed 0.5 kg.

Inspection

(SUBSYSTEM)

Subsystem Baseline Physical Inspections

If EPS mass does not exceed 0.5 kg

EyasSAT Requirements Verification Matrix

Rev. 8.1 August 21, 2009 1

Requirements Validation

Part-level Verification

Software Verification & Validation

SubsystemVerification

SystemIntegration

• A combination of lecture and extensive hands-on exercises using the EyasSAT Educational Satellite System

– To teach space system verification and validation principles and practices

SDOE 633 Course Overview 15

SDOE 633 Part B Objectives

• At the end of this course you should be able to:– Explain the end-to-end SE process and how it applies to system (and

lower level) requirements definition, allocation, validation and verification.– Describe the purpose and scope of key documents required in the

validation and verification processes, and describe typical errors committed.– Describe various methods of verification, when they are appropriate. and

how they are used as part of a verification plan for a system of interest– Determine appropriate circumstances and applicability of verification

methods to prototype and proto-flight systems. – Analyze representative verification plans, test sequences and activities

for an example system of interest (spacecraft).– Develop, evaluate and implement a master verification plan for a space

system including hardware, software and associated ground support equipment (GSE).

– Apply processes and techniques in a hands-on workshop associated with a system of interest.

– Describe applicable NASA, ECSS, DoD and Industry Standards and lessons learned to support system verification decisions and activities.


Why Focus on V&V?

16

Mission/Year Mishap Verification & Validation Contributing Factors

Genesis/2004 G-switch installed backwards !parachute not deployed ! hard landing

“no system-level test” (of G-switch)

Columbia/2003 debris damaged thermal tiles ! loss of crew and vehicle

“current [modeling] tools, including the Crater model, are inadequate...”“flight configuration was validated using extrapolated test data...rather than direct testing”

Comet Nucleus Tour (CONTOUR)/2002

overheating of s/c by solid rocket motor plume ! vehicle lost

“Project reliance on analysis by similarity”

Wide Field InfraRed Explorer (WIRE)/1999

electronic startup transient !early cover jettison !cryogen boil-off science mission lost

“failure to correctly identify the source of the signal which caused Electro Explosive Device (EED) Simulator to ‘latch’ upon Pyrotechnics Box power-up during spacecraft integration testing”

Mars Polar Lander/1998

software flaw !descent engine shut-off too soon !vehicle lost

“employed analysis as a substitute for test in the verification and validation of total system performance...tests employed to develop or validate the constituent models were not of adequate fidelity.”

ASSE Table 11-2


V&V Inputs & Outputs

17

Inputs Verification, Validation & Certification

Activities

Outputs

•Un-validated Requirements Requirements Validation •Validated Requirements

•Un-validated Mission Critical Models•Validated Model Requirements Model Validation

•Validated Models•Model Uncertainty Factors (MUFs)•List of Model idiosyncrasies

•Validated Requirements•Un-verified End Product(s)•Verification Plan (including incompressible test list)•Verification Enabling Products (e.g. validated models)

Product Verification

•Verification Plan (as implemented)•Verified End Product(s)•Verification Products (data, reports, verification closure notices, work products)

•Verified End Product(s)•Customer Expectations (e.g. MOEs and other acceptance criteria)•Operations Concept•Validation Plan•Validation Enabling Products

Product Validation

•Validated Product(s)•Validation Products (e.g. data, test reports, work products)

•Verified & Validated Product(s)•Verification & Validation Product(s)•Real-world Operational Context for End Product(s)

Flight Certification

•Certified Product•Certification Products (e.g. signed DD250, completed functional config. audit (FCA, physical config. audit (PCA), mission rules)

This course will explore the activities and tools needed to achieve these inputs and outputs

SDOE 633 Course Overview 18

SDOE 633 Part B Agenda

LECTURES• Intro to Space Systems Engineering• The EyasSAT System of Interest• Validating Requirements & Models• Verifying Products• Verification of COTS/NDI• Software Verification & Validation• Validating Products and Flight

Certification

Goal: Achieve an ability to analyze, synthesize and critically evaluate V&V plans and real-world implementations through interactive lectures and hands-on exercises

EXERCISES• SRPL• EyasSAT Requirements Validation• EyasSAT Verification Planning• EyasSAT Parts-Level Verification

Events• EyasSAT Software V&V Event• EyasSAT Subsystem Verification

Events• EyasSAT System Verification Events• EyasSAT System Validation Events• EyasSAT System Acceptance Review


SDOE 633 Admin

• Prerequisites– Completion of SDOE 632 or 635 (or equivalent space mission design

experience)• Grading

– Mission Concept Review, 40%» combined oral presentation and written design package in Powerpoint

– Final Exam (online), 60%» Focused on key concepts and practices in Space System Verification and

19

102

G.3 Mission Concept Review

The MCR affirms the mission need and examines the proposed mission’s objectives and the

concept for meeting those objectives.

Table G-3 – MCR Entrance and Success Criteria

Mission Concept Review

Entrance Criteria Success Criteria

1. Mission goals and objectives.

2. Analysis of alternative concepts to show

at least one is feasible.

3. Concept of operations.

4. Preliminary mission descope options.

5. Preliminary risk assessment, including

technologies and associated risk

management/mitigation strategies and

options.

6. Conceptual test and evaluation strategy.

7. Preliminary technical plans to achieve

next phase.

8. Defined MOEs and MOPs.

9. Conceptual life-cycle support strategies

(logistics, manufacturing, and

operation).

1. Mission objectives are clearly defined and stated and are

unambiguous and internally consistent.

2. The preliminary set of requirements satisfactorily provides a

system that will meet the mission objectives.

3. The mission is feasible. A solution has been identified that is

technically feasible. A rough cost estimate is within an

acceptable cost range.

4. The concept evaluation criteria to be used in candidate

systems evaluation have been identified and prioritized.

5. The need for the mission has been clearly identified.

6. The cost and schedule estimates are credible.

7. An updated technical search was done to identify existing

assets or products that could satisfy the mission or parts of

the mission.

8. Technical planning is sufficient to proceed to the next phase.

9. Risk and mitigation strategies have been identified and are

acceptable based on technical risk assessments.

G.4 System Requirements Review

The SRR examines the functional and performance requirements defined for the system and the

preliminary program or project plan and ensures that the requirements and the selected concept

will satisfy the mission.

Table G-4 – SRR Entrance and Success Criteria

System Requirements Review

Entrance Criteria Success Criteria

1. Successful completion of the MCR and responses made to all MCR

Requests for Actions (RFAs) and Review Item Discrepancies (RIDs).

2. A preliminary SRR agenda, success criteria, and charge to the board

have been agreed to by the technical team, project manager, and

review chair prior to the SRR.

3. The following technical products for hardware and software system

elements are available to the cognizant participants prior to the

review:

a. system requirements document;

b. system software functionality description;

c. updated concept of operations;

d. updated mission requirements, if applicable;

e. baselined SEMP;

f. risk management plan;

g. preliminary system requirements allocation to the next lower level

system;

1. The project utilizes a sound

process for the allocation and

control of requirements

throughout all levels, and a

plan has been defined to

complete the definition activity

within schedule constraints.

2. Requirements definition is

complete with respect to top-

level mission and science

requirements, and interfaces

with external entities and

between major internal

elements have been defined.

3. Requirements allocation and

flow down of key driving

Overview of:Mission and System Design

Verification and ValidationSDOE 633

Presented by

With our Academic Partner

Documents

Overview of: Mission and System Design Verification and Validation · 2010-09-17 · Mission and System Design Verification and Validation SDOE 633 Presented by ... validation plan