February 9 – 10, 2010

http://nsc.nasa.gov/

National Aeronautics and Space Administration

Improving Project Success throughTechnical Excellence

John D. MarinaroMark M. Kowaleski

1Used with Permission

Agenda• SMA Technical Excellence Program (25 min.)

– NSC Background

– Purpose of STEP

– How the STEP Curriculum was Developed

– STEP Curriculum Elements

– Extension to other Agency training needs and risk reduction

– STEP Level 1 Roll-out and Metrics

– STEP Levels 2 – 4 Planning

• System Safety Discipline Overview (20 min.)– System Safety & Program/Project Interactions

– System Safety Competencies

– System Safety Lifecycle Products and Services

– Community of Practice

– System Safety State-of-the-Discipline

– System Safety and Program/Project Risk Reduction

• Q&A (10 min.)

• Back-up Materials– SMA Discipline Curriculums

2

SMA Technical Excellence Program

3

http://nsc.nasa.gov/ 4

NASA SAFETY CENTEROverview

NSC develops SMA personnel, processes, and tools needed for the safe achievement of NASA’s strategic goals through:

2/18/2010

Learning from our mishaps Ensuring that we are informed risk-takers

Benchmarking from the best and bringing their best practices into our workplace

Managing the routine risks of the workplace effectively

Preserving our resources for the NASA mission

Establishing a learning environment

http://nsc.nasa.gov/ 5

NASA Safety CenterFOUR AREAS OF SUPPORT

2/18/2010

http://nsc.nasa.gov/ 6

THE PEOPLE OF THE NSC

2/18/2010

Agency RangeSafety Efforts

SARD/KSC

OCELiaison

Chief,OSMA Technical

DisciplineFellows

NASA SafetyTraining Center

GRCOCIO

NESCLiaison

Kerry RempCOTR

Curriculum Development

Diane ChapmanTechnical Support/Discipline

John MarinaroTechnical

Excellence Director

Homayoon Dezfuli

Systems Safety ST

Audit support from the field

centers, other HQs offices/SSCs

EDUCATEAlphaport

NASA Safety Center

Alan H. PhillipsDirector

Rafael SanabriaDeputy Director

Dennie GoniaSecretary

Sandra L. HardyBusiness Manager

Charlene L. AndersonAdministrative Officer

Mark KowaleskiSystems Safety

Dr. Natesan JambulingamReliability &

Maintainability

Ray KacmarQuality

Engineering

VacantSoftware Assurance

VacantOperational Aviation

Safety

Fayssal SafieReliability &

Maintainability ST

Brian HughittQuality

Engineering ST

Martha Wetherholt

Software Assurance ST

Carolyn VanDrei

Enterprise Architecture

Manager

Sue OteroIRIS COTR

IRISEX3

Karen MeinertKnowledge Management

Systems Manager

Dr. Jeffrey DawsonData Analysis &

Trending Analyst

Sallie KeithInformation

Dissemination

Michael MilbertAnalyst

Art LeeInstitutional/ Facility

Operational Audit Manager HQ

Paul BoldonInstitutional/

Programmatic Support Audit Manager HQ

Robert CrossAnalyst

Cheevon (Mi-Mi) LauAudits & Assessments Director

Brad KaustinenNCAS COTR

NCAS Honeywell

Mark GeorgeMIS Administrative Support

Michael DodsonMishap Investigator

Eastern Region (GRC, GSFC, HQ, KSC, LaRC )

Kristie FrenchMishap Investigator

Southern Region(JSC, MFSC, SSC)

Dennis MorehouseMishap Investigator

Western Region(ARC, DFRC, JPL)

Israel GreenfeldIRIS Requirements Liaison

with MIBs/MIWGSteve LilleyMIB Technical Report Reviews & Corrective

Action Tracking

Kenneth O’ConnorMishap Investigation

Support Manager

Brian FollenPBMACOTR

PBMAARES

Brian JacksonAuditor

Safety & Mission AssuranceTechnical Excellence• Within NASA, SMA professionals

provide a vital role in support of NASA’s design and operations focus.

• SMA advises mission specialists on matters related to risk, safety, and mission success.

• SMA encompasses all Agency safety, reliability, maintainability, and quality engineering and assurance activities.

7

STEP was Designed to be…

1. …a way to assist the Agency (specifically SMA) as it transitions from an operational focus back to a designfocus.

8

STEP was Designed to be…

2. …both a source of knowledge and an ongoing reference tool in providing SMA professionals with structured, advanced SMA educational opportunities in support of that design focus.

9

STEP Curriculum Development

Technical Discipline Teams developed a competency-driven curriculum structure for each discipline that included Education, Training & Lifelong Learning.

Job of SMA Professional

SMA Disciplines

CompetencyWheel

SMA Products & Services

EDUCATION TRAINING LIFELONG LEARNING10

STEP Curriculum

1 Introduction to STEP

2 Introductory Discipline Concepts

[Doer]

3 Intermediate Discipline

Concepts or Specialty

[Team Leader]

4 Advanced Discipline

Concepts or Specialty

[Expert]

• The STEP Curriculum is delivered across four levels with increasing specialization.

Generalized Specialized

11

A Typical STEP Program Commitment

STEP Training Elements Level 1 Level 2 Level 3 Level 4

Core Training 3 24 40 N/A

Discipline Training 18 100 137 137

Domain Training (by others) 4 40 TBD TBD

Total Course Hours 25 164 177 (+TBD) 137 (+TBD)

OJT N/A 200 400 400

Years of Experience N/A 2 4 8

Test and/or Peer Board N/A N/A Yes Yes

CEU’s N/A 16 24 32

Estimated Hours per STEP Level

12

STEP Training Elements

13

STEP’s Benefit toNASA’s Programs/Projects

• High confidence in SMA’s competence and capability.

• High confidence in SMA’s products & services.

• Improved technical program/project performance.

• Risk Reduction:– Increased success likelihood.

– Improved risk characterization.

– Improved risk-based decision-making.

• Serves as showcase for engineering excellence.

• Complements Agency training initiatives:– Academy of Program/Project and Engineering Leadership (APPEL).

– System Engineering Leadership Development Program (SELDP).

– Senior Executive Service Career Development Program (SES CDP).

14

STEP Level 1

Curriculum• One curriculum across all six disciplines• Total of 25 contact hours

– Each course is one hour on average• Topics covered:

– SMA Implementation – Core – Domain – Disciplines – Case Studies

• On-demand and self-paced

15

• Agency-wide roll-out

• Online Courses

• Handbook

• Quick-start guide

• Course DVD

• Brochures

16

Level 1 Initiatives

Level 1 Participation

17

Post Rollout• Over 1,300 individuals have

been assigned or registered for the curricula

• 361 graduates as of Nov 1

• 26% of all enrollees have successfully completed Level 1 – two months after rollout!

• Feedback: nearly 70% rate the program as “high value”

STEP Levels 2 - 4

Curriculum Features• Unlike Level 1’s unified curriculum, Levels 2 - 4 provide a unique curriculum

for each discipline

• Participants will choose a discipline and receive a Course of Study to guide them through each level.

• Levels 2-4 allow for electives & discipline specialization

• The course of study for each discipline includes a variety of course types:– Online/e-learning via SATERN

– Instructor-led workshops

– APPEL, NSTC and SARD courses

– Commercially-available courses

– Courses created by and for the NASA Safety Center

18

Levels 2 – 4 also encourage OJT, Enrichment Experiences, Career Training & Continuing Education.

Competency Wheel

19

• The planning process started with the Competency Wheel for each discipline, from Level 1.

• The Competency Wheel lists all the competencies an SMA professional needs to perform.– The inner ring lists the major

competencies. – The outer ring provides a

further level of detail to each of the major competencies.

20

Performance Objectives

Level 2• Recognize QE Quality

Management functions.• Recognize Quality

Management as one of four (4) second tier competencies of the Perform Quality Management & Planning Competency.

• Recognize the six (6) third tier competencies in Quality Management.

Level 3• Perform QE Quality

Management functions.• Apply Quality

Management principles to develop Quality Management Systems (QMS).

Level 4• Evaluate Quality

Management functions and Systems.

For each competency, detailed objectives were written to describe exactly what performance is expected at each level. (Typical performance objectives shown)

SMA Work Products and Services

SMA Work Products and Services were identified for each discipline. These are the things SMA Professionals actually do on the job.

21

Course of Study

The Result: • A complete curriculum

plan – a Course of Study –was developed for each discipline, by level

• The course of study includes a list of – Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

22

STEP – A Complete Curriculum

• Overall, STEP will be a complete curriculum tailored specifically for the SMA Professional.

• STEP is specifically designed to address the technical needs ofNASA’s programs and projects.

23

OVERVIEW

System Safety Discipline

24

25

Technical Excellence does pay off……and you do get what you pay for.

Coach 1st Class

US Airways Flight # 1549

26

SS Discipline – Where are we today?

• Most SS professional’s training is ad hoc training and professional development (not many opportunities for SMA degrees at universities).

• We “learn as we go” and heavy emphasis on unstructured OJT.• We do a lot of “reviewing” but not a lot of actually “doing” and this is

perceived as a negative thing.• Don’t address new hires’ training needs.• Don’t have mentoring for entry or mid-level employees.• Training needs to apply to “real” work vs. perceived or ideal work that we

think we should be doing.• Don’t have structured OJT with a clear understanding of what it is that is

expected to be learned.• Training does not address softer SS discipline areas like resource

estimation, safety in contracts, and SS integration with system and program lifecycle.

• Training is presently focused on qualitative methods and we would benefit as a community from more analytical-based training.

What Do System Safety Engineers Do?

Fundamentally, NASA System Safety Engineers, at some point in their careers, during any phase of the system life cycle, do any combination of three basic activities:

1. Assess any situation with safety consequences.

2. Manage safety in acquisitions.

3. Manage, leverage, and interact with safety organizations and NASA programs.

From this premise, all competencies and products and services are derived.

System Safety’s Interactionswith Programs/Projects

SS Engineer Decision Maker

System Safety Products (E.g., Hazard Analysis, FTA, PRA, CRM, etc.)SS Engineer applies

competencies to develop SS products

SS products and Framework influences decisions

Decisions influence the outcome of Programs

SS products support milestones in the Program lifecycle

Competencies define how SS Engineers do

their job at NASA1

4

5

System Safetymonitors lifecycle

Safety Performance

6

23

Draft 0 of Overview of System Safety – Not to be Distributed Outside of SS Working Group29

Overall System Safety Framework

30

System Safety Student Body Estimates

CenterSMA - Civil

ServantSMA -

Contractor TotalGRC 14 6 20JSC 110 233 343MSFC 32 35 67LaRC* 10 20 30DFRC 2 0 2JPL n/a 10 10ARC* 5 10 ?SSC 4 10 14GSFC 31 73 104KSC 28 25 53HQ/NSC* 10 2 12* EstimatedTotal 246 424 670

System Safety STEP participantsCivil Servant SMA & SMA's Support Contractor

SS Competency Wheel

31

A SMA System Safety professional needs to possess these 34 competencies:

Major Competencies

Technical• System Safety Mathematical Skills

• System Safety Analytical Methods

• System Safety Rationale

Managerial• System Safety in Acquisition Management

• System Safety in Organizational Management

Competency Performance Objectivesand Training, OJT, and Reference Material Allocations(1 of 34 competencies)

32

Competency: Hazard AnalysisDescription: The analytical method and approach used to identify, mitigate, and manage safety hazards found in systems.Performed by: System Safety EngineerSTEP Level 2 STEP Level 3 STEP Level 4 • Understand and explain the

basic engineering principles, steps, and elements of a Hazard Analysis (HA).

• Identify and explain NASA policy and procedures governing the conduct of HA.

• Prepare a qualitative hazard analysis on a simple, isolated NASA space subsystem.

• Document and present a HA to a senior official.

• Understand and explain the relationships of HA to fault tree analysis and the relationship of the HA with other SMA disciplines.

• Prepare a quantitative HA on a NASA space subsystem or system.

• Prepare a Hazard Report (HR) with all required content.

• Identify, document, and monitor requirements associated with a HR.

• Monitor the implementation and performance of Hazard Controls throughout the lifecycle.

• Evaluate HR/HA prepared by contractors.

• Present a HA/HR to decision makers.

• Monitor the performance of HA management systems and processes used by the organization.

• Conduct analysis of supporting HA data for trends, indicators, and pitfalls.

• Interpret, apply, communicate, and monitor requirements related to HA for NASA programs and projects.

• Influence policy or procedures to improve the conduct of HA/HR.

Training OJTReferenceand/or and/or equals COMPETENCY

33

Core and Advanced System Safety Products

Core System Safety Products & Services•Are products & services that all NASA’s System Safety Engineers are

expected to be able to produce at any time.–E.g., Create a Hazard Analysis.

•Are generally independent of specific job location, project, or job function.

Advanced System Safety Products & Services•Are products & services that NASA’s System Safety Engineers are expected

to be able to produce if their job assignments demand it or as their careers progress and their responsibilities escalate.

–E.g., Support a SEB for a major procurement/acquisition activity.

•Are generally dependent on specific job location, project, and job function.

SS Products & ServicesDistributed across Levels

STEP Level 2 STEP Level 3 STEP Level 4 • Conduct Hazard Analysis• Conduct Fault Tree Analysis• Prepare Safety Rationale• Prepare Safety Data Package• Support Anomaly Resolution• Provide Decision Maker Input• Communicate Safety

Requirements• Conduct Surveillance &

Compliance Verification• Safety Performance Metrics

Monitoring

• Conduct Hazard Analysis• Conduct Fault Tree Analysis• Prepare Safety Rationale• Prepare Safety Data Package• Support Anomaly Resolution• Provide Decision Maker Input• Communicate Safety

Requirements• Conduct Surveillance &

Compliance Verification• Safety Performance Metrics

Monitoring----------------------------------------------• Safety Acquisition Management• Probabilistic Risk Assessment• Risk Analysis• Integrated Safety Data Package• Support Certification of Flight

Readiness

• Conduct Hazard Analysis• Conduct Fault Tree Analysis• Prepare Safety Rationale• Prepare Safety Data Package• Support Anomaly Resolution• Provide Decision Maker Input• Communicate Safety

Requirements• Conduct Surveillance &

Compliance Verification• Safety Performance Metrics

Monitoring---------------------------------------------• Safety Acquisition Management• Probabilistic Risk Assessment• Risk Analysis• Integrated Safety Data Package• Support Certification of Flight

Readiness----------------------------------------------• Safety Organizational

Management• Support Lessons Learned• Manage Policy/Standards &

Requirements• Support Special Studies• Support Mishap/Close-call

investigations• Support Audits & Assessments34

Green = CoreRed = Advanced

SS Community of PracticeQ: What is the SS COP?

A: Anything associated with the conduct of System Safety activities that folks need to know, share, keep, or access.

Examples:

1. Agency “go-to” SS Website.

2. Mass email notification list.

3. Guest Lecture Series.

4. Webcasts, Podcasts, etc.

5. Blogs/Papers with ratings.

6. Clearing House ConceptTools, Documentation, Reference,

Resources, Conferences, Speakers,

Lessons Learned, Training, Etc.35

35

SS Website

36

STEP enables Program/ProjectRisk Reduction through…• Providing a competent and highly capable System Safety

workforce.

• Providing predictable, consistent, and value-added System Safety products & services to programs/projects.

• Analytical tools proliferation (e.g., SAPHIRE/QRAS).

• Enabling risk-informed decision-making (RIDM).

• Providing safety throughout the entire program acquisition lifecycle.

• Providing a forum for a community of practice to encourage lessons learned and communications.

• Proactive human capital management.

37

38

State-of-the-Discipline (SOD)

Asks: “What is the health status of the SS Discipline?” and“What are we doing about it?”

Sample SOD Metrics• STEP Qualification distribution.• Training metrics/rates/distribution.• Integrity of SS products over time.• Analytical tools proliferation.• Proactive Human Capital Management• Resources, staffing, work-load.• Rate of Hazard Controls violations.• Safety-related Technology Readiness.

39

State-of-the-Discipline Example

Metric: Proactive Human Capital Management

• What is the competency relationship between System Safety and System Engineering at NASA?

• Data supports observation that SMA and SE could be better integrated.

• Is an overall 2% cross competency relationship acceptable?

• What are we doing about it?

Agency CMS Competency Level 1 Level 2 Level 3 Level 4 Total"SAFENG" 181 321 288 197 987"SYSTEMENG" 325 742 930 503 2500

506 1063 1218 700 3487

Level 1 Level 2 Level 3 Level 4 TotalCombined SAFENG and SYSTEMENG 17 20 21 26 84

3% 2% 2% 4% 2%

Agency CMS Competency Level

Combined CMS Competencies Levels

Desired SODmetricimprovement

Desired

STEPPING FORWARD

40

• Providing Education for NASA’s Future…

… one STEP at a time.

Back-up

41

CURRICULUM BY DISCIPLINESMA Technical Excellence Program

42

SOFTWARE ASSURANCEKAREN MEINERT

STEP Internal Review

43

SA Technical Discipline Team

• NSC TDT Lead – Karen Meinert• NSC SSC – Jennifer Jones• HQ – M. Wetherholt, M. Bodeau• ARC – C. Chow, G. DelCarmen• DFRC – D. Tran• GRC – T. Luu• GSFC- S. Harris• IV&V – L. Montgomery

• JPL – S. Flanagan, J. Wilf• JSC – R. Hugger• KSC – D. Victor• LaRC – L. Johnson• MSFC – R. Strickland• SSC – C. Fallo

SA Competency Wheel

• Software Assurance is comprised of the following competencies: – Software Quality – Software Safety – Software Reliability – Software Verification &

Validation – Software Independent

Verification & Validation– Software Assurance

Planning & Management

45

Competencies by Level

46This is one of six competencies for this discipline

Competency: Software SafetyDescription: Knowledge, skills and abilities in identifying, analyzing, tracking, mitigating and controlling software hazards and hazardous functions to ensure safer operation within a system. Activities performed ensure that safety is designed into the software.

STEP Level 2 STEP Level 3 STEP Level 4Performance Expectation

Observe and participate with supervision in software safety activities throughout the lifecycle including:

o Perform software safety litmus test o Assist in reviewing: system PHArequirements, designs, code, analyses, change requests, testing documents, planning documents, lessons learned, and traceability of safety requirements

o Participate in peer reviewso Attend/observe milestone reviews

and safety reviewso Assist in assurance of CM of safety

requirementso Assist in software acquisition

activities and evaluation of contractor deliverables.

Performance ExpectationUnder general awareness of supervisor, perform software safety activities throughout the lifecycle including:

o Identify safety critical softwareo Review system hazard analyseso Perform or review software safety

analyseso Verify compliance to appropriate

safety standardso Review software safety planso Assure V&V requirements are meto Review/assure software safety

requirements are documented and tracked throughout the life cycle

o Participate in reviews of designs and code

o Review test documentationo Identify potential risk/safety issues

and verify issues are identified and tracked to resolution

(…continued)

Performance Expectation:o Determines criteria for safety critical

software classificationo Develop/review safety standardso Identify safety critical software o Complete hazard reports with Systems

Engineeringo Develop software hazard reportso Assure hazards verifications are followedo Assess adequacy of safety requirements

and software safety documentationo Plan and manage software safety effortso Perform software safety activities

throughout the lifecycle, including:o Develop/review/assess

requirements, designs, code, and test documents;

o Assess development and test environments;

o Determine adequacy of V&V activities;

(…continued)

Mapping Training to Competencies

47

Software Assurance Levels 2-4Discipline/Course Type(30 Courses)

Training Available

Needs Modification New Development

Instructor-Led Courses 93% 7% 17%

Online Courses 7% .5% 1%

Total 100% 7.5% 18%

Hours 374 28 68

Current Status

Course of Study

• The Course of Studyincludes a list of – 30 Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

49

QUALITY ENGINEERINGRAY KACMAR

STEP Forward

50

51

QE Technical Discipline Team

• NSC TDT Lead – Ray Kacmar• NSC SSC – Jerome Anderson• HQ – Brian Hughitt• ARC - Steve Jara, Damon Flansburg• DFRC - Gary May, Kevin Reilly• GRC - Bob Makovec• GSFC - Louis Thomas, Mike Kelly

• JPL - Thomas Smith, Tony Gutierrez• JSC - Dave Dyer• KSC - Robert Ellison• LaRC - Elijah Kent, Charlie Greenhalgh• MSFC - Ken Crane• SSC - Tim White• NCAS - Charles Dionne

QE Competency Wheel

52

An SMA Quality Engineering professional needs to possess these major competencies:

• Perform Quality Assurance Management & Planning

• Assure Adequate Design & Development Controls

• Conduct Procurement QE• Define and Implement Production

Material & Process Controls• Define and Perform Inspection

Testing & Acceptance Activities• Implement Problem Reporting &

Corrective Action• Conduct Quality Evaluations & Data

Analysis The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies.

Competency: Quality ManagementDescription: Recognize, Perform and Evaluate Quality Management Functions

STEP Level 2 STEP Level 3 STEP Level 4

• Recognize QE Quality Management functions.

• Recognize Quality Management is one of four (4) second tier competencies of the Perform Quality Management & Planning Competency.

• Recognize the six (6) third tier competencies in Quality Management.

• Understand the concepts of quality management and quality management systems.

• Understand the concepts of strategic planning for quality.

• Understand communication mechanisms.

• Recognize the factors that influence the work environment.

• Recognize NASA polices on Human Factors.

• Recognize the importance of NASA’s training & certification requirements related to quality and identify related training.

• Recognize related sources of information.

• Perform QE Quality Management functions.

• Apply Quality Management principles to develop Quality Management Systems (QMS).

• Based on a review of the quality requirements, plan the QMS (to include objectives, scope, schedule, and technical expertise).

• Document the results of the QMS, to include any findings, observations, or assessments of risk.

• Perform strategic planning (Elective).• Develop effective communication

mechanisms (Elective).• Identify human factor issues and

human error prevention techniques (Elective).

• Evaluate Quality Management Functions and Systems.

• Perform a complete quality management system evaluation to ensure compliance to all applicable QMS requirements defined in the planned evaluation.

• Become a Certified Quality Engineer (Elective).

• Evaluate human factor issues and human error prevention techniques. (Elective).

• Provide instruction to operators and/or inspectors in the methods of product/service verification, throughout the value-stream.

Competencies by Level

This is 1of 48 competencies for this discipline

54

Mapping Training to Competencies

Training CourseTraining Hours Pe

rform

Qua

lity

Assu

rance

Ma

nage

ment

and

Plann

ing

Quali

ty Ma

nage

ment

Prog

ram Q

uality

Int

egrat

ionTe

chnic

al As

sess

ment

Quali

ty To

olsAs

sure

Adeq

uate

Desig

n &

Deve

lopme

nt Co

ntrols

Desig

n Inp

uts an

d Re

views

Workm

ansh

ip Sta

ndard

sDe

sign o

f Ex

perim

ents

Draw

ings,

Dimen

sions

& To

leran

ces

Desig

n Fail

ure

Mode

s & Ef

fects

Analy

sis (F

MEA)

and

Critic

al Ite

ms Li

st (C

IL)Ha

zard

Analy

sisFa

ult Tr

ee An

alysis

Risk M

anag

emen

tCo

nfigu

ration

Ma

nage

ment

Data

Mana

geme

ntCr

itical

Syste

msCo

nduc

t Pr

ocure

ment

Quali

ty En

ginee

ring

Risk B

ased

Ac

quisi

tion

Mana

geme

ntPu

rchas

e Orde

r rev

iewPr

e-awa

rd/Po

st aw

ard Su

r

Quality Management Overview 3 2 2

Quality Management Systems 8 3Workforce Development, Training & Certification 0.5Program Quality Integration Overview 1 2

Program Quality Integration 4 3

Quality Assurance Plans 3 2

Lessons Learned 1 2

Technical Assessment Overview 1.5 2

Technical Assessment 4 3

Audits 0.5 2

AS 9100 Lead Auditor Class 40 4

Quality Assurance Surveillance Plan 1 2

Quality Tools Overview 1.5 3

Quality Tools 5 4

Quality Tools - NASA Addendum 2 4

Lean Six Sigma 8 4

Quality Function Deployment 6 4

Design Inputs and Reviews Overview 4 2 2

Design Inputs and Reviews 8 3 3Mistake-Proofing Executive Overview 4 2

Mistake Proofing 16 3

Workmanship Standards Overview 1 2Design of Experiments Executive Overview 4 3

Design of Experiments 32 4Drawings, Dimensions, Tolerances Overview 1 2Drawings, Dimensions & Tolerancing - S-44 40 3Failure Modes Effects Analysis & Critical Items List (SATERN) SMA-017-01 1 2

System Safety I 8 3 3Continuous Risk Management - APPEL - CRM 24 4

Configuration Management 4 2Functional Configuration Audit / Physical Configuration Audit 0.5 2

Data Management 2 2

Parts Control Overview 1

Procurement Quality Overview 8 2 2 2 2

NASA Letter of Delegation 2

Quality Engineering Levels 2-4Discipline/Course Type(66 Courses)

Training Available

Needs Modification New Development

Instructor-Led Courses 58% 0% 0%

Online Courses 18% 14% 10%

Total 76% 14% 10%

Hours 282.5 53.5 38

Current Status

Course of Study

• The Quality Engineering Course of Study includes a list of – 66 Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

56

RELIABILITY & MAINTAINABILITYDR. NAT JAMBULINGAM

STEP Forward

57

58

R & M Technical Discipline Team

• NSC TDT Lead – Nat Jambulingam• NSC SSC – Maria Havenhill, Dan Deans, Bob Loomis• HQ – Frank Groen • HQ Tech Fellow – Fayssal Safie• ARC – N/A• DFRC – N/A• GRC - Edward Zampino• GSFC - Tony Diventi, Matthew Samuel• JPL - Chau Brown • JSC - Samadar Roshan-Zamir

• KSC - Chi Yeh, Tim Adams• LaRC – Pete Parker• MSFC - Richard Stutts• SSC - Mike Rewis

R & M Competency Wheel

59

A Reliability & Maintainability professional needs to possess these major competencies:

• Reliability Management

• R&M in Acquisition

• R&M Methods in Design & Development

• Reliability Testing and Demonstration

• Use of Database and Reports for R&M

• Mathematical Tools

The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies.

Competencies by Level

60This is one of 41 competencies for this discipline

Competency: FMEA/CIL, FMECADescription: Systematically identifies system’s potential failure modes, and evaluates their causes and effects on the system’s ability to perform its functions.STEP Level 2 STEP Level 3 STEP Level 4 • Define failure modes, integration

characteristics, propagation• Describe the contents of FMEAs, Critical

Items List (CIL) and FMECAs.• Describe how and why Critical Items are

created out of the FMEA analyses.• Locate and read program requirements.• Describe the FMEA, Critical items and

FMECA process flow.• Describe the relationship of CIL to

hazard reports.• Identify simple generic failure modes

and postulate the effects based on an understanding of component functionality.

• Identify simple failure mode effects at the system level

• Interpret and apply FMEA or FMECA program requirements.

• Perform a FMEA or FMECA on assigned system/subsystem/component.

• Evaluate a FMEA and Critical Item List analysis & coordinate comments with the subsystem engineer.

• Assess compliance to the reliability design requirements verified by the FMEA or FMECA.

• Identify common cause failure modes in addition to system failure modes.

• Identify failure modes that are specific to the design configuration under analysis.

• Trace the effects of failure modes to the system level.

• Coordinate with design engineering to develop retention rationale for the critical items.

• For those engineers involved in diagnostics, describe the relationship between FMEA, FMECA and fault detection and isolation.

• Perform/evaluate FMEA or FMECA on increasingly complex systems.

• Accurately & thoroughly document FMEA and Critical Item List analysis performed & describe the implications the conclusions may have.

• Explain nuances not readily evident to less experienced personnel.

• Assess compliance of submitted FMEA/CIL, FMECA to requirements document.

• Determine compliance to Reliability design requirements verified by the FMEA or FMECA.

• Lead a team in developing a FMEA and a Critical Item List.

• Coordinate the team activity on a complex system so that the interfaces between subsystems have coverage and the Critical Item retention rationale meets all the requirements of the Program FMEA/CIL or FMECA requirements document.

61

Mapping Training to CompetenciesCompetencies

Potential Source

Training CoursesTraining Hours

Probability

Statistics

Uncertainty

Analysis

Sensitivity Analysis

Decision Analysis

Boolean Algebra

Data Analysis

Markov Analysis

R&M Planning

R&M Program Management

R&M Starategic Management

Managing and Evaluating the elements of a contractor's R&M

R&M into contracts, SOWs, specifications, deliverables, award structure, and other acquisition elements

Defining and controlling R&M requirements for program/project acquisition

Reliability Block Diagram

Fault Tree Analysis

FMEA/CIL, FMECA

Root Cause Analysis

Probabilistic Analysis

Risk Analysis

Parts Stress Analysis

Reliability Modeling

Maintainability Modeling

Availability Modeling

Mechanical Reliability

Software Reliability

Human Reliability

Data Analysis and Trending

Parts and Material evaluation

Reliability Test Planning

Design of Experiment

Reliability Test Development

Product Life Testing

Reliability Growth

NASA PRACA System

NASA Lessons Learned System

NASA Internal Reports

Government-Industry Data Exchange Program (GIDEP)

Material Review Board Reports

Contractors Test Reports

Commercial and other Givernment agencies data

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

1 OSMA+NSC R&M Principles 24 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

2 NSC Design for Reliability 32 2 2 2 2 2 2 2

3 NSC Design for Maintainability 32 2 2 2 2 2 2 2

4NSC Reliability, Availability, and Maintainability (RAM) Planning for

Programs and Projects 14 2 2 2

5 Commercial Data Collection and Analysis I* 16 3 3 3 3 3 3 3 3

6NSC Reliability, Availability, and Maintainability Modeling (including

success tree analysis) I 16 3 3 3

7

GSFC/JPL Parts and Materials Assessment (EEE and Mechanical), Parts Stress/Derating and also incl. Thermal Analysis and Structural Analysis I 16 3 3

8NSC

Maintainability and Supportability Analysis and Integration* 16 3

9 SATERN/JPL FMEA/CIL, FMECA 16 3

10 NSC R&M Testing and Demonstration I* 16 3 3 3 3

11 Commercial Reliability Growth 16 3

12 NSC Dependency and Common Cause 16 3 3 3

13JPL

Single Event Upset/Effects Analysis, Radiation Effects I 16 3 3 3 3

14 Commercial Sneak Circuit Analysis 16 3 3 3

15OSMA

Root Cause Investigation and Corrective Action/Failure Analysis 8 3

16 NSC GIDEP 8 3 3

17 Commercial Certified Reliability Engineer (CRE) Refresher 16

18 SS Probabilistic Risk Assessment (SS) 24 3 3 3

19 QE Problem Reporting (QE) 4 3 3 3

20 SS Hazard Analysis (SS) 8 3 3 3 3

21 SS Fault Tree Analysis (SS) 24 3

22 NSC Data Collection and Analysis* II 16 4 4 4 4 4 4 4 4

23NSC Reliability, Availability, and Maintainability Modeling (including

success tree analysis) II 16 4 4 4

24

NSC Parts and Materials Assessment (EEE and Mechanical), Parts Stress/Derating and also incl. Thermal Analysis and Structural Analysis II 8 4 4

25 NSC R&M Testing and Demonstration II* 16 4 4 4 4

26 Commercial Reliability Growth 16 4

27 SA Software Reliability (SA) 16 4

28 Commercial Physics of Failure 16 4 4 4 4 4

29JPL

Single Event Upset/Effects Analysis, Radiation Effects II 16 4 4 4 4

30 JPL Worst Case Analysis 24 4 4

31Commercial (NRC)

Human Reliability 28 3

32NSC/ Commercial

Design of Experiments 40 4

**

Cross Discipline Course Probability and Statistics Basics : Source Document: NASA SP-

2009-659 (SS) 3 3

TOTAL COURSES ADDRESSING COMPETENCY 2 3 1 1 1 1 5 1 3 3 3 3 3 3 3 1 4 2 1 2 2 8 8 9 4 1 1 3 2 3 1 3 3 5 1 2 2 3 2 2 1

R & M Levels 2-4 DisciplineDiscipline/Course Type(32 Courses)

Training Available

Needs Modification New Development

Instructor-Led Courses 19% 0% 15%

Online Courses 25% 9% 32%

Total 44% 9% 47%

Hours 166 32 176

Current Status

Total Discipline Hours = 374 [302 hours are required; 72 hours are elective]

Course of Study

• The R & M Course of Study includes a list of – 32 Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

63

SYSTEM SAFETYMARK M. KOWALESKI

STEP Forward

64

65

SS Technical Discipline Team

• NSC TDT Lead – Mark Kowaleski• NSC SSC – Maria Havenhill, Diane

Chapman, Jennifer Baumeister, Kathy Kraft• HQ – Homayoon Dezfuli• ARC – N/A• DFRC - Jonathan Brown• GRC - Bill Schoren, Carrie Green• GSFC - Bo Lewis, Angela Melito

• JPL - Kirk Barrow• JSC - Gerald Readore, Larry Gregg• KSC - Ronald Long, Robert Schwader• LaRC - John Greco, Jose Caraballo• MSFC - Herb Shivers, Keith Layne• SSC - Robert Gargiulo

SS Competency Wheel

66

An SMA System Safety professional needs to possess these major competencies:

Technical• System Safety Rationale

• System Safety Analytical Methods

• System Safety Mathematical Skills

Managerial

• System Safety in Acquisition Management

• System Safety in Organizational Management

The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies.

Competencies by Level

67This is 1 of 34 competencies for this discipline.

Competency: Hazard AnalysisDescription: The analytical method and approach used to identify, mitigate, manage and communicate safety hazards found in systems.

STEP Level 2 STEP Level 3 STEP Level 4 • Understand and explain the basic

engineering principles, steps, and elements of a Hazard Analysis (HA).

• Identify and explain NASA policy and procedures governing the conduct of HA.

• Prepare a qualitative hazard analysis on a simple, isolated NASA space, ground, and/or test hardware/operations subsystem.

• Document and present a HA to a senior official.

• Understand and explain the relationships of HA to fault tree analysis and other relevant analytical techniques, and the relationship of the HA with other SMA disciplines.

• Prepare a quantitative HA on a NASA space subsystem or system.

• Prepare a Hazard Report (HR) with all required content.

• Identify, document, and monitor requirements associated with a HR.

• Monitor the implementation and performance of Hazard Controls throughout the lifecycle.

• Evaluate HR/HA prepared by contractors.

• Present a HA/HR to decision makers.

• Utilize anomaly resolution input to evaluate impacts to system-level hazard risk through the use of qualitative and/or quantitative tools.

• Utilize Fault Tree Analysis (FTA) or an equivalent tool to map hazard causes and controls to other safety and reliability data products (e.g. FMEA/CIL).

• Assess the performance of HA management systems and processes used by the organization.

• Compare actual vs. predicted system performance and formulate and manage corrective actions, as necessary, to ensure efficacy of hazard control.

• Conduct analysis of supporting HA data for trends, indicators, and pitfalls.

• Interpret, apply, communicate, and monitor requirements related to HA for NASA programs and projects.

• Influence policy, procedures, and/or requirements documents to improve, standardize and encourage the conduct of HA/HR for NASA programs and projects.

Mapping Training to Competencies

68

Source

Training Element Prob

abilit

ySt

atis

tics

Bool

ean

Alg

ebra

Util

ity A

nalys

isLi

near

Pro

gram

min

gH

azar

d An

alys

isFa

ult T

ree

Ana

lysi

sPr

obab

ilistic

Ris

k As

sess

men

tR

isk

Ana

lysi

sEv

ent S

eque

nce

Dia

gram

sH

uman

Erro

r Ana

lysi

sSo

ftwar

e Fa

ilure

Ana

lysi

sH

ardw

are

Rel

iabi

lity

Anal

ysis

Dat

a A

naly

sis

Sour

ce/M

echa

nism

/Out

com

e M

eth.

Acci

dent

Sce

nario

Dev

elop

men

tLi

kelih

ood

Estim

atio

nC

onse

quen

ce A

naly

sisU

ncer

tain

ty A

naly

sis

Sens

itivi

ty A

naly

sis

Dec

isio

n A

naly

sis

Elem

ents

of F

light

Rat

iona

lePr

esen

tatio

n an

d C

omm

unic

atio

nEn

dors

emen

t of F

light

Rea

dine

ss.

Syst

em S

afet

y re

quire

men

ts (A

cqu)

Prop

osal

/con

tract

s/S

OW

, etc

.M

anag

e/ev

alua

te s

afet

y de

liver

able

sM

anag

ing

orga

niza

tions

Man

agin

g re

sour

ces

and

budg

etM

anag

e pr

oduc

ts

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30Competencies for Core

Products/Services

Class TrainingOSMA Probability and Statistics Basics I: Source

Document: NASA SP-2009-659 2 2 3 3 3 3 3 3 3 3

OSMARisk Management; Source: NASA RIDM Handbook (under development) and CRM Handbook (to be developed) 3 3 3 3 3 3 3 3 3 3 3

OSMAScenario-based System Safety Analysis: Source: NASA System Safety Standard (under development) 3 3 3 3 3 3 3 3 3

OSMA Basic Fault Tree Analysis I; Source Document: NASA Fault Tree Handbook 2 2 2 2 2 2 2

OSMA System Safety I: Source Document: NASA System Safety Standard (under development) 2 2 2 2 2 2 2 2 2 2 2 2

NSTC SMA-SAFE-NSTC-0002, System Safety Fundamentals 3 3 2 2 3 3 2 2 2 2 2 3 2 3

OSMAProbabilistic Risk Assessment Methods (PRAM); Source Document: NASA PRA Procedures Guide 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

OSMA Probability and Statistics Basics II: Source Document: NASA SP-2009-659 3 3

OSMA RIDM Source: NASA RIDM Handbook (under development) 3 3 3 3 3

OSMA Basic Fault Tree Analysis II; Source Document: NASA Fault Tree Handbook 3 3 3 3 3 3 3

OSMA System Safety II: Source Document: NASA System Safety Standard 3 3 3 3 3 3 3 3 3 3 3 3

APPEL Project Planning Analysis and Control (PPAC) 2 2 2 3 3 3

APPEL Management of Space Technology Programs (APPEL-MSTP) 2 2 2

OSMA Application of Bayesian Probability Inference to Risk and Reliability Analysis of Space Systems; Source Document: NASA SP-2009-659 3 3 3 3 3 3 3 3 3 3 3 3

NSTC SMA-SAFE-NSTC-0040, System Safety Engineering 3 3 3 3 2 3 3 2 3 3 3 3 3 3

APPEL Assessing Project Performance (APPEL-APP)

APPEL Performance Based-Statement of Work (APPEL-PBSOW) 3 2

OSMAApplication of SAPHIRE to PRA: Source: NASA PRA Procedure Guide and SAPHIRE Users Manual 3 3 3 3 3 3 3 3 3 3 3 3 3 3

NSTC SMA-SAFE-NSTC-0023, System Safety in Acquisition 2 2 3 3 3

TOTAL COMPETENCIES ADDRESSED 5 5 4 1 0 10 11 11 12 5 5 4 8 4 4 8 12 7 9 5 2 3 5 3 7 5 6 1 1 1

System Safety Levels 2-4Discipline/Course Type(19 Courses)

Training Available

Needs Modification New Development

Instructor-Led Courses 100% 15% 10%

Online Courses 0% 0% 0%

Total 100% 15% 10%

Hours 374 94 0

Current Status

Course of Study

• The System Safety Course of Study includes a list of – 19 Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

70

OPERATIONAL SAFETYVANESSA PELLEGRINO

STEP Forward

71

72

OS Technical Discipline Team

• NSC TDT Lead – Vacant• NSC SSC - Vanessa Pellegrino• HQ (SARD) – Jonathan Mullin• HQ - Matt Deptola• ARC – David King• DFRC – Leonard Mbuko• GRC – Angela Windau• GSFC – Pat Hancock• IV&V – Kurt Kehl, Dave Sheldon

• JPL – Peter Robles• JSC – Bubba Johnson• KSC – Cal Staubus• LaRC – Jose Caraballo• MSFC – Dennis Davis• NSTC – Tim Evans• SSC – David Lorance• WFF – Kenny Volante

OS Competency Wheel

73

• An SMA Operational Safety professional needs to possess these major competencies:

• Recognizing Hazards

• Measuring, Evaluating and Controlling Hazards

• Safety Management

• Safety Metrics

• Collecting Risk Information

• Assessing Risk

• Managing Risk

• Hazards Specific to NASA The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies.

Competencies by Level

74This is one of 38 competencies for this discipline

Competency: Personal Protective Equipment (PPE)Description: Provide oversight and guidance for the Personal Protective Equipment program to ensure that employees are protected from the hazards in the workplace.

STEP Level 2 STEP Level 3 STEP Level 4• Understand and describe the

Hierarchy of Controls • Describe the purpose of PPE• Understand the usage of the

different types of PPE • Understand how to maintain

different types of PPE• Know the components and

requirements of an effective PPE program as outlined in 29 CFR 1910.32 & 1910.33 and 1910.134-138

• Understand the requirementsin NPR 8715.3 Chapter 3, Section 3.3

• List in order the common steps performed during a hazard assessment

• Understand how each type of PPE is designed to protect people

• Inspect employees' PPE • Perform a PPE Assessment• Perform training on

donning/doffing, cleaning, storage, use and limitations of PPE

• Perform selecting and testing of PPE

• Purchase appropriate PPE

• Evaluate that PPE is being worn properly

• Evaluate and update a Personal Protective Assessment Plan

Mapping Training to Competencies

75

Training Hours Bi

olog

ical

Haz

ards

Chem

ical

Haz

ards

Elec

tric

al H

azar

dsN

atur

al H

azar

dsRa

diat

ion

Haz

ards

Stru

ctur

al a

nd

Mec

hnan

ical

Haz

ards

Haz

ards

Rel

ated

to F

ire &

Ex

plos

ions

Haz

ards

Rel

ated

to

Hum

an F

acto

rs &

Er

gono

mic

sM

easu

rem

ent a

nd

Mon

itor

ing

Engi

neer

ing

Cont

rols

Adm

inis

trat

ive

Cont

rols

Pers

onal

Pro

tect

ive

Equi

pmen

tM

anag

emen

t Pro

cess

esIn

spec

tion

s and

Aud

iting

Proj

ect M

anag

emen

tRi

sk M

anag

emen

tSa

fety

Man

agem

ent

Syst

ems

Trai

ning

and

Co

mm

unic

atio

n M

etho

dsG

roup

Dyn

amic

s

Basi

c Fi

nanc

ial P

rinci

ples

Prob

abili

ty a

nd S

tatis

tics

Perf

orm

ance

Met

rics

and

Indi

cato

rsId

enti

fy a

nd c

hara

cter

ize

haza

rds

Des

ign

and

use

data

m

anag

emne

t sys

tem

s to

co

llect

and

val

idat

e ri

sk

info

rmat

ion

Colle

ct a

nd v

alid

ate

risk

info

rmat

ion

by s

tudy

ing

cultu

re, m

anag

emen

t st

yle,

bus

ines

s clim

ate,

fin

anci

al c

ondi

tion

s &

th

e av

aila

bilit

y of

re

sour

ces

Rese

arch

app

licab

le

law

s, p

olic

ies,

con

sens

us

stan

dard

s an

d be

st

prac

tices

to d

evel

op

benc

hmar

ks fo

r ass

essi

ng

an o

rgan

izat

ion'

s saf

ety

perf

orm

ance

Eval

uate

the

ris

k of

in

jury

/pro

pert

y da

mag

e w

ith

an o

rgan

izat

ion'

s fa

cilit

ies,

pro

duct

s,

syst

ems,

pro

cess

es,

equi

pmen

t &

em

ploy

ees

Aud

it m

anag

emen

t sy

stem

s to

com

pare

ag

ains

t est

ablis

hed

stan

dard

sA

naly

ze tr

ends

in le

adin

g an

d la

ggin

g pe

rfor

man

c in

dica

tors

usi

ng

hist

oric

al in

form

atio

n an

d st

atis

tica

l met

hods

Training Course 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29Ground Nuclear Safety 1 2

Refresher for Explosives Handlers and Operational Personnel 7Crane Operations and Rigging Safety Refresher 4 2

Fall Protection Refresher 3 2Explosive Safety Management and Engineering 21

Fire Hazards in Oxygen Systems 12 3

Aerial Platform 3 3

Stairw ays and Ladders 3 2

Hazardous Locations 14 3

Basic Explosives Safety 14Overhead Cranes and Material Handling 14 2Fire Protection - Theory and Practice 21 2

Mobile Crane Safety 14 2

Forklif t Safety 3 2

Life Safety Code 21 2

Laboratory Safety and Health 14 2

Electrical Safety Standards 21 2

Electrical Safety Refresher 3 2

Fall Protection 14 2

Safety in High Pressure Systems 14 2

Safety in High Pressure Operations 7 2

Operational Safety Levels 2-4Discipline/Course Type(TBD Courses)

Training Available

Needs Modification New Development

Instructor-Led Courses TBD TBD TBD

Online Courses TBD TBD TBD

Total TBD TBD TBD

Hours TBD TBD TBD

Current Status

Course of Study

• The Course of Studyincludes a list of – Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

77

AVIATION SAFETYDARREN CHILDERS

STEP Forward

78

AS Technical Discipline Team

• NSC TDT Lead – Vacant• NSC SSC – Darren Childers• HQ – Richard Yasky• DFRC – Jonathan Brown• GRC – Mike Ernst/Paul Tavernelli• JSC – Jill Brigham• LaRC – Noel West/Evan Horowitz

• ARC – N/A• GSFC – N/A • IV&V – N/A• JPL – N/A• KSC – N/A• MSFC – N/A• SSC – N/A

AS Competency Wheel

80

An Aviation Safety professional needs to possess these major competencies:

• Aviation System Safety Analytical Methods

• Program Reviews

• Unmanned Aircraft Systems (UAS)

• Test Management/Operational Test and Evaluation

• Basic Performance Flight Testing

• Vehicle Structures, Loads and Flutter Testing

• System Safety

• Modern Flight Control Systems

• Advanced Performance Flight Testing

The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies.

81Note: See STEP Forward binder with details of 54 AS Competencies.

AS Competency Performance Objectives

Competency: Modern Flight Control Systems Description: In-depth understanding of linear control that includes the impact that controls and displays have upon flight control evaluations. Extensive knowledge of frequency domain testing, parameter identification and flight test instrumentation requirements STEP Level 2 STEP Level 3 STEP Level 4

• Recognize the pros and cons of different data collection and processing techniques.

• Identify and mitigate flight test risk.

• Identify system problems resulting from flight test and assist in troubleshooting.

• Understand principles of linear control theory.

• Predict results, compare results to flight test data and resolve anomalies.

• Reduce, analyze and explain flight test data.

• Predict the performance and flying qualities of an aircraft from first principles

• Defend conclusions based upon valid flight test data.

• Demonstrate the importance of credible test and evaluation.

Mapping Training to Competencies

82

Training Hours

Iden

tify,

Ana

lyze

, E

limin

ate

and

Rep

ort

Haz

ard

sR

isk

Ass

essm

ent/R

isk

Man

age

men

tF

ault

Tree

Ana

lysi

sF

ailu

re M

odes

and

E

ffect

s A

naly

sis

(FM

EA

)F

ailu

re M

odes

, Effe

cts

and

Cri

tical

ity A

nal

ysis

(F

ME

CA

)A

irwo

rthin

ess

and

F

light

Saf

ety

Rev

iew

B

oard

sD

esig

n R

evie

ws,

A

ircra

ft C

onfig

urat

ion

Man

age

men

t and

F

light

/Tes

t Rea

dine

ss

Rev

iew

sD

esig

n R

evie

ws

of

Airc

raft

Con

figur

atio

n M

ana

gem

ent

Mis

hap

Pre

vent

ion

Sen

sors

Hea

lth M

onito

ring

Sys

tem

sD

ata

Link

sM

issi

on C

ontro

l E

lem

ents

Con

trol

and

Dis

pla

ysE

nvel

ope

Pla

nnin

gIn

-flig

ht S

imul

atio

nH

igh

Fide

lity

Mod

els

Sys

tem

Def

icie

nci

es

Mis

sion

Pla

nnin

gC

ontin

genc

y Lo

gic

Nat

iona

l Airs

pac

e S

yste

m In

terf

ace

Ops

& D

evel

opm

ent

Tes

t, E

valu

atio

n an

d P

lann

ing

Req

uire

me

# Source Training Course Course Code 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

1 APPEL Introduction to Aeronautics APPEL-I-AERO 32

2 APPELProject Review Processes and Strategies APPEL-PRPS 40 2 2 2

3 APPEL Assessing Project Performance APPEL-APP 16

4 APPEL Performance Based-Statement of Work

APPEL-PBSOW 16

5 APPELManagement of Space Technology Programs APPEL-MSTP 24

1 OSMA Probability and Statistics Basics I N/A 8

2 OSMA Basic Fault Tree Analysis I N/A 8 2

3 OSMA System Safety I N/A 8

4 OSMAProbabilistic Risk Assessment Methods (PRAM) N/A 28

5 OSMA Probability and Statistics Basics II N/A 24

6 OSMA Basic Fault Tree Analysis II N/A 16

7 OSMAApplication of Bayesian Probability Inference to Risk and Reliability

f S S

N/A 24

1 NSTC System Safety EngineeringSMA-SAFE-NSTC-0040 36

2 NSTC System Safety in AcquisitionSMA-SAFE-NSTC-0023 20

1 Practical AeronauticsIntroduction to Jet Engines -- A Practical Perspective N/A 16

2 Practical AeronauticsAeronautics for Scientist and Engineers N/A 40

1National Test Pilots

SchoolTest Management/Operational Test and Evaluation T&E 4002 40 2 2

2 National Test Pilots School Performance Flight Testing I T&E 4101 40

3 National Test Pilots School Performance Flight Testing II T&E 4102 40

4 National Test Pilots School Flying Qualities Flight Testing I T&E 4103 40

5 National Test Pilots School Flying Qualities Flight Testing II T&E 4104 40

Aviation Safety Levels 2-4Discipline/Course Type(20 Courses)

Training Available

Needs Modification New Development

Instructor-Led Courses 100% 8% 0%

Online Courses 0% 0% 0%

Total 100% 8% 0%

Hours 374 31 0

Current Status

Course of Study

• The Aviation Safety Course of Study includes a list of – 16 Courses

– Readings & Resources

– OJT Experiences

– Lifelong Learning Activities

84

AbstractPresentation Title: Improving Project Success through Technical Excellence

Synopsis: This presentation discusses NASA’s approach to developing Technical Excellence (TE) in Safety and Mission Assurance (SMA) and addresses how TE enhances safety and mission success for NASA’s projects.

Abstract: NASA performs high-risk functions in the process of achieving its goals and objectives and SMA personnel provide valuable risk mitigation products and services to projects that enhance safety and mission success. The NASA Safety Center (NSC) has been charged with developing an Agency-wide SMA Technical Excellence Program (STEP). STEP is a structured professional development program that develops increasing levels of competencies in various SMA disciplines. STEP also has implications that are of interest to the project management and system engineering communities and may serve as a model to enhance the technical excellence of those communities as well. For example, STEP will give projects increased confidence that various SMA products, such as Hazard Reports, Fault Trees, Risk Assessments, etc., are of the highest quality and caliber achievable. Further, projects will be assured that SMA products are integrated and treated holistically as part of the System Engineering and Project Management life-cycles. The tools and methods to deliver focused and effective professional training may be extended to meet other Agency training needs and to enhance project communications. In these ways, this presentation will convey to the audience that STEP is truly a window of opportunity to improve the state-of-SMA in the Agency and to provide a direct benefit to the Agency’s Projects. 85

BiographiesName: John D. Marinaro

Title: Technical Excellence Director

Company/Organization: NASA Safety Center

Voice: 440-962-3092 Fax: 440-962-3110 E-mail: john.d.marinaro@nasa.gov

Mr. John D. Marinaro is the Director of Technical Excellence at the NASA Safety Center, the former Executive Director of the NASA Aerospace Safety Advisory Panel, NASA Independent Verification and Validation Facility Chief Engineer, and a retired Army Helicopter Pilot and Aviation Officer. For more than 25 years, Mr. Marinaro has held management positions devoted to fields such as: aerospace and electrical engineering; safety management, particularly in NASA operations; software development and validation; airplane and helicopter flight operations; and other project management responsibilities. His career has included service with NASA, the Federal Aviation Administration, the Naval Air Systems Command, and a former active and reserve member of the United States Marine Corps, Navy Reserves, and Army National Guard.

Name: Mark M. Kowaleski

Title: System Safety Discipline Lead

Company/Organization: NASA Safety Center

Voice: 440-962-3044 Fax: 440-962-3110 E-mail: mark.m.kowaleski@nasa.gov

Mr. Mark M. Kowaleski has 21 years of experience in space systems with NASA, including spacecraft, instruments, flight operations, ground systems, Space Shuttle, and System Safety. Mark has worked as a satellite operations manager, as a system engineer, project manager, Space Launch Initiative Safety Manager, Space Shuttle Safety Manager, and as the Aerospace Safety Advisory Panel’s Executive Director. Mark served as the Executive Secretary for the NASA Return-to-Flight team, served on the Space Shuttle Crew Survivability Study, and was a member of the STS-114 Gap Filler close-call investigation. Mark is presently working at the NASA Safety Center, being responsible for defining and implementing NASA’s System Safety Technical Excellence Program. 86