0 NASA Technical Standards Program Office Lessons Standards Training Communicate Knowledge Projects Generate Knowledge Effective Use of Lessons Learned:

1

NASA Technical Standards Program Office

Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Effective Use of Lessons Learned:A Key Element for Mission Success

Presented byPaul S. GillManager,

NASA Technical Standards Program Office June 18, 2003

“Try to make original mistakes rather than repeating those of your predecessor.”

Secretary of Defense Donald Rumsfeld

2


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e Lessons Learned Are Crucial To The Agency’s Continued Success

They Capture The Knowledge And Experiences From NASA’s Programs And Projects A Principal Component Of Any Organizational Culture Committed To Excellence

NASA Recognizes The Need To Systematically Document And Utilize The Knowledge Gained From Past Experiences.

NASA Policy And Guideline (NPG) 7120.5B, “NASA Program And Project Management Processes And Requirements”

NASA Policy Directive (NPD) 8730.3, “NASA’s Quality Management System Policy”

GAO-02-195 Revealed Fundamental Weaknesses In How NASA Performs The Collection And Sharing Of Lessons Learned Information.

Lessons Learned Not Routinely Identified, Collected Or Shared By Program And Project Managers

Lack Of Proper Review For Lessons Learned

Need For More Positive Lessons Learned (Generated Not As a Failure)

Background

3


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Develop and Initiate a Process to Facilitate the Identification and Input of Lessons Learned Information

Through the utilization of Chartered Technical Discipline Working Groups, Review and Categorize Submitted Lessons Learned Information

Consolidate and Link Lessons Learned Databases That May Exist at Different NASA Facilities and NASA Contractors

Utilize Automated Workflow Capability to Manage and Monitor Tasks

Facilitate the Dissemination of This Information Across a Broad Spectrum Via a Portal Within the Agency’s Knowledge Management Initiative, Thus Providing Programs and Projects With Advanced Search and Retrieval Mechanisms

Obtain Current Performance Metrics to Evaluate System Effectiveness

Approach

Proposed Initiative Leverages State-of-the-art Information Technologies and Organizational Processes to Optimize the Capture, Management, and Dissemination of Lessons Learned

4


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Lessons

Standards and Practices

Training

Time

Update/Create Standards

Update/Create Training

Communicate Knowledge –

Infuse into Practice

Projects

Generate and Acquire Lessons

Generation and Communication of Knowledge Begins With Lessons

Project 1 Project N

Approval Process

Approval Process

Gather Lessons at Control Gates

5


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Lessons Learned Implementation Process

ATPSRR

PDRCDR

DCR

FRRPost

Flig

ht

Generate and Communicate Lessons Learned Lessons Learned

Databases

Technical Discipline Working Group Review

PROGRAMS AND PROJECTS (NPG 7120.5)

IMPLEMENTATIONFORMULATION

Screened And

Approved Lessons

SYSTEMS ENGINEERING (NPG 71xx.x and SP-6105)

CONCEPTDEVELOPMENT

DESIGNPHASE

DEVELOPMENT& OPERATIONS

ProposedLesson

6


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e Programs and Projects

Link ToRequirements

Infuse IntoEngineering Process Link the Lessons

Learned to System

Development Lifecycle

Requirements

Infuse the Lessons Learned Into the

Engineering Process

EstablishFeedbackSystem

Feedbacks Needed to Provide User Based

Enhancements

CO

MM

UN

ICA

TE K

NO

WLE

DG

E

Generate & Identify

Document

Categorize

Characterize

Evaluate

Generate & Identify Lessons Learned

Document Lessons Learned

Categorize Into Discipline Categories

Characterize into Programmatic Versus

Technical

Lessons Learned Validity is

Determined by the Working

Group

GE

NE

RA

TE K

NO

WLE

DG

E

Lessons Learned Closed-Loop Process

7


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Key Agency Roles

Success of a Lessons Learned Program Will Require the Support and Participation of the Following:

Office of the Chief Engineer (Code AE) Overall Management and Oversight Responsibility (i.e. NPD, NPG, and Program Plan) Charter the Technical Discipline Working Groups

Chief Information Officer (CIO) Responsible for Agency Information Technology Infrastructure

Safety and Mission Assurance Office (Code Q) Responsible for the Collection and Archiving of Mishap Reports and Failure Investigation

Reports

Enterprises/NASA Centers/Agency Contractors Source and Users of the Lessons Learned Information Membership in Technical Discipline Working Groups

Lesson Learned Program Office Agency Lessons Learned Working Group Lessons Learned Information System Engineering Discipline Working Groups Responsible for the Collection, Analysis, and Dissemination of Lessons Learned Information

Training (Code FT) Advanced Program and Project Leadership (APPL) NASA Engineering Training (NET)

8


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Programs and Projects Generate and Use Knowledge

NASA Technical Discipline Working Groups Acquire Lessons (Develop, Review, Screen) Place Lesson in Discipline Taxonomy Update/create Standards to Incorporate Lessons

NASA Lessons Learned Program Facilitate Capture and Use of Lessons Learned Provide On-line Access to Lessons Learned System

NASA Technical Standards Program Identify Lessons for Potential Conversion Into Technical Standard Link Lesson to Relevant NASA Preferred Technical Standards

NASA Training Incorporate Lessons and Standards Into Training Materials Provide Training on Use of Lessons Learned System

Key Program Roles

9


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Lesson Learned Program Proposed Organizational

Structure

Office of the NASA Chief Engineer

Theron Bradley

Lessons Learned ExecutiveGregory Robinson

Office of the NASA Chief Engineer

Theron Bradley

Lessons Learned ExecutiveGregory Robinson

Lessons Learned Program Office

Lessons Learned Program Office

NASA Lessons LearnedWorking Group

(Senior Representative From Each Center/JPL)

NASA Lessons LearnedWorking Group

(Senior Representative From Each Center/JPL)

NASA Engineering Discipline Working Groups

NASA Engineering Discipline Working Groups

Lessons Learned Systems

Lessons Learned Systems

NASA Engineering Management Board

NASA Engineering Management Board

10


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

ePlans Initiated in FY03:

Enhance Existing Taxonomy of Categories/Subcategories (Technical and Programmatic)

Categorize Existing Lessons Learned Databases or Raster Images (PDF, etc.)

Identify NASA Failure Mishap Reports

Initiate the Development of a Process for Formulating Discipline Working Groups for

Screening, Validating and Categorizing Existing Lessons Learned and to Facilitate

Searching, and Identifying New Lessons Learned

Initiate and Develop a Common Lessons Learned Process and Format

Lessons Learned To be Infused Into the Agency's Programs and Projects at the

Appropriate Stages During the Lifecycle

Inputs From Program and Project Managers Will Help Define the Requirements to

Implement Lessons Learned Into Product Lifecycle

Define Agencywide System Architecture

Proposed Tasks

11


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

ePlans for FY04 Through FY08:

Establish an Agency Lessons Learned Working Group

Comprised of Senior Representatives From All NASA Field Facilities and Enterprises

Establish Agencywide Technical Discipline Working Groups

Catalyst for Infusion of Lessons Learned into Product Life Cycle

Working Groups Defined By The Ten Discipline Categories Established by the NASA

Technical Standards Program

Develop, Review, and Screen Lessons Learned for Endorsement by the Agency

Working Groups Provide Agencywide Expertise in Specific Technical Areas

DISCIPLINE WORKING GROUPS ARE THE KEY TO SUCCESS

Proposed Tasks (Cont’d)

12


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

ePlans for FY04 Through FY08 (Cont’d)

Implementation and Maintenance of the Information System Architecture

Including Advanced Search Engine That Meets System Requirements

Promote the Agency Lessons Learned Initiative Through an Agencywide

Awareness and Training Initiative

Designed to Reach All NASA Employees and Supporting Contractors

Ensure Lessons Learned Are Available From a User Friendly Website

Develop Metrics to Ascertain the Effectiveness of the System Usage at the

Agency Level

Establish a Feedback System for Inputs Needed to Provide User Based

Enhancements

Proposed Tasks (Cont’d)

13


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

NASA Lessons Learned Working Group NASA Engineering Discipline Working Groups NASA Lessons Learned Program Website for Agency Wide Use With Timely

Feedback Response System Including Integration of Legacy Lessons Learned Systems

Common Lessons Learned Format Awareness Initiative Metrics on Usage of Lessons Learned A Process Whereby Lessons Learned May Be Linked to the Applicable

Period/Phase in the Lifecycle of a Program or Project Periodic Brief to the NASA Chief Engineer’s Office

Deliverables

The Anticipated Products of the Proposed Lessons Learned Initiative Are Contained in the Following List:

14


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Challenges to Overcome

Establishment of Technical Discipline Working Groups Chartered by Code AE Identify, Categorize, and Promote Incorporation and Use of Lessons Learned Resources Required (Dollars and Manpower)

Standardization System Requirements of Standardized Electronic Templates (Data Format

Standards) to Facilitate the Capture of Metadata and Integration of Legacy

Lessons Learned Development of Large–Scale Advanced Technology Information System Get It To the People Agencywide

Education and Awareness on Value of Generating and Using Lessons Learned Develop Personalization Profile Infuse Into Program/Project Execution Oversight of Use in Programs/Projects

15


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e Lessons Learned /Best Practices Identified

Over 1,700 Lessons Learned/Best Practices Accessible on the NASA Technical Standards Program Website

Over 245 Preferred Technical Standards Are Linked to 477 Unique Lessons Learned

Lessons Learned Sources Have Been Identified From Databases, Technical Papers and Memorandums, Mishap Reports, NASA Special Publications (Monographs)

Lessons Learned Search Screen Capability on NASA Technical Standards Program Website

Over 1 Million Keywords Contained in the Database

Enhanced Search Permits Users to Search by

Multiple Keywords

Phrases

Functional Categories

Organizations

Project

Date

Accomplishments To Date

16


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Correlation of Lessons Learned With NASA Technical Standards Taxonomy

Discipline Category

Title(Several Lessons Learned Fall Into Multiple Categories)

Number of Lessons

0000Documentation and Configuration Management, Program

Management 412

1000Systems Engineering and Integration, Aerospace

Environments, Celestial Mechanics 456

2000 Computer Systems, Software, Information Systems 236

3000 Human Factors and Health 205

4000Electrical Systems, Electronics, Avionics/Control Systems,

Optics 323

5000Structures/Mechanical Systems, Fluid Dynamics, Thermal,

Propulsion, Aerodynamics 730

6000 Materials and Processes, Parts 564

7000 System Test, Analysis, Modeling, Evaluation 673

8000 Safety, Quality, Reliability, Maintainability 829

9000Operations, Command, Control, Telemetry/Data Systems,

Communications 482

Accomplishments To Date (Cont’d)

17


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Summary

Lessons Learned Are Necessary to Preserve and Enhance the Agency’s Corporate

Knowledge The Effort Must Be Optimized to Meet Users Needs and Fully Take Advantage of

Available Information Technology Assets

Success of This Effort Will Be Highly Dependent on Buy-in From All

Enterprises/centers Agency Level Initiative, Including NASA Contractors

NASA Technical Standards Program Office Is Best Positioned to Launch and Lead

Effort Has Effectively Implemented Agency-wide Working Groups Very Successful Linking Lessons Learned to Technical Standards Has Understanding of Task/Challenges – Similar to Technical Standards

Program, But of Greater Magnitude

A Must Do Task

18


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

This Is Our Opportunity To Make Lessons LearnedThis Is Our Opportunity To Make Lessons LearnedA Viable Element of the NASA Culture!A Viable Element of the NASA Culture!

19


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Backup

20


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

eNPG 71xx.x Systems Engineering

Processes and Requirements

Concept Development Prod’n Oper’n Retire

NPG 7120.5B Program/Project Mgmt.Processes and Requirements

StandardsLessonsLearned

Capture and Deliver Lessons Learned in their Context of Use

Advanced Search Based on Discipline, Activity, Phase, Etc.

Interactive, Web-Based Documents

21


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

Start Observation of Lesson Learned/Best Practice (LL/BP)

Submit LL/BP to Lessons Learned (LL) Program Office

LL Program Office Initial Screening

Submit to Technical Discipline Working Group (TDWG) for Review of Value to NASA

Valid?

Lesson May Be Sent Back to Author for

Editing or Removed

From System

No

Disseminate the Lessons through the Application

Server

• Insufficient Information

• Not Technically Accurate

• Duplication

• Proprietary Information

• ITAR/EAR Information

Lesson Learned Work Flow

Lesson May Be Sent Back to Author for

Editing or Removed From System

Yes

NASA Lessons Learned Databases

Users Search for Lessons Learned

Metrics, Feedback, and Analysis

Approved and Recommend

Development/Change of a NASA Standard

based on LL/.BP?

Review Existing Standards for Applicability

Yes

No

22


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

User Requirements

(ICD) & Concept of Operations

System Requirements &

Architecture

Refine Design (SRR, PDR,

CDR)

Build and Deliver Increments

Component Integration and

Test

System Integration and

Test

System Demonstration, Verification,and

Validation

“Vee” Model Of System Development ConceptIdentification and

Integration of Technical Standards and Lessons

Learned into Appropriate Stages of the Product Life

Cycle

23


Lessons

Standards

Training

Com

mun

icat

e

Kno

wle

dge

Projects

Gen

erate

Kn

ow

ledg

e

NASAStandards

andPractices

LegacyLessonsLearned

LegacyLessonsLearned

Agency-wide

LessonsLearned

Knowledge Metadata- Document Content and Structure- Taxonomy and Ontology

- Technical Discipline and Discipline Ontology

- Etc.

ExternalStandards

Advanced Common Interface

• Integrated Access to NASA Knowledge• Knowledge-based Search Through Lessons and Standards• Discover and Link Related Knowledge• Prompt Potential Users That Knowledge Is Available• Generate New Knowledge as Part of Project Life Cycle

XML-Based Data Exchange In A Common Format

A Vision Enabled by Integrated Shared Knowledge

Interactive, Web-based Documents Deliver Relevant Standards and Lessons

Learned in the Context of Use Gather New Lessons at Critical Project

Gates and Engineering Milestones