Keer.beth

Cross-Cutting Look at OCE Policy Compliance within NASA

(February 2010)

Presented By: Beth KeerNASA HEADQUARTERSOFFICE of the CHIEF ENGINEER

OCEOffice of the Chief Engineer




OCEOffice of the Chief Engineer Used with Permission

2

OCENASA Engineering &Program/ProjectManagement

Survey Team Members

Philosophy:• Involve personnel that are knowledgeable, experienced and able to

take lessons back to home Center to institute change.• Take advantage of symmetry with other activities as much as

possible.• Some consistency with some positions held for POC and unique

skills specific to the Center.

Team Members:• Beth Keer – OCE Survey Team Manager• Jim Lawrence – OCE (PSGS/Dell)• John Kelly – OCE Software Manager• Dr. David Liskowsky – OCHMO (Part time) • Participant from Center to be Surveyed• Participant from Center with interest in chosen Projects• * Patti Stockman – NASA HQ CIO (Part time)• * EVMWGChampion: Mike Ryschkewitsch Chief EngineerSponsor: Sandra Smalley

3


Survey Objectives

• Review Center processes and infrastructure for compliance with OCE requirements, policy, procedures, processes, statutes, and regulations

• Review at least two Projects/Programs’ documents and processes for compliance with OCE requirements, policy, procedures, processes, statutes, and regulations

• Identify systemic problems or deficiencies • Recognize areas of excellence/best practices• Receive Center feedback regarding areas where

Agency policy and requirements should be modified• Results are used in Agency response to OMB/GAO/

IG issue on oversight of implementation of requirements

4


Survey Basis

The basis for the survey: • NPR 7120.5D, NASA Space Flight Program and

Project Management Requirements; • NPD 2820.1, NASA Software Policy; • NPR 7150.2, NASA Software Engineering

Requirements; • NPR 7120.6, Lessons Learned;• NPD 8070.6, Technical Standards; • NPR 7120.8, NASA Research and Technology

Program and Project Management Requirements (Planning Perspective);

• NPR 7123.1, NASA Systems Engineering Processes and Requirements (beginning with DFRC in 2010).

5


Findings Definition Guidelines

Strength: A finding of OCE Policy implementation that results in reduced risk to the Program or Project.

Weakness: A finding of OCE Policy implementation that results in risk to the Program or Project.

Observation: A finding that is a potential weakness or potential risk to the Program or Project.

Opportunity: A finding of OCE Policy Implementation that is potentially a strength.

Non-Conformance: A finding of OCE Policy not being implemented and with no waiver in place.

6


Survey Core Elements

• Common framework for unified program and project life cycle

• Program and project review structure• Technical Authority implementation• Dissenting Opinions/ Waiver Processes• Lessons Learned• Technical Standards• Software Engineering Management• Research and Technology (NPR 7120.8)

7


ScheduleGSFC – April 2008KSC – July 2008MSFC – December 2008**JSC – February 2009**LaRC – March 2009**JPL – August 2009**ARC – October 2009**

(Current Status) GRC – November 2009DFRC – *February 2010

(Round 1 complete) SSC - March 2010HQ MD or PSO – May 2010GSFC – July 2010KSC – September 2010

* Planning underway ** Dates consistent with IPS

8


Common Framework

Management Reporting:• Various degrees of strength to the CMC as far as

Management Reporting of Projects.– Meet commitments of Center vs Roads and camodes

• Integrated CMCs– Strong implementation for instances with robotic missions – Originally had good reports on CxP, (JSC, MSFC, LaRC) – ARC response not as strong (not clear if external

environment change for CxP or other factors).

Excellent: GSFC, MSFC, LaRC, JPL

9


Common Framework

Configuration Management:• CM/DM/RM practices are inconsistent across the

Centers.– Implementation is strong on projects.

• Most Centers have one or two tools that they “bless” for use but do not require those tools to be used.

– Program driven tool (Windchill) viewed as not user-friendly.• Some use as repository, not an improvement over working systems.• Potential for confusion/risk by using separate systems.

– Records Management awareness is improving.• Records retention schedules are not consistently being completed.• Records Management planning is not clearly considered when setting

up CM systems.• Records Management responsibility is consistently being assigned as

the CM Officer or CM lead on the projects.

10


Common Framework

Information Management:• Directives Management Systems not up to date to

reflect Policy (update processes not perfect).– HQ requirements are too many and at different levels of

detailed description (what vs how).Excellent: JPL has contract with definite applicability of policy

AND flowdown of requirements into JPL Rules! and JPL FPP.

Earned Value Management:• Centers do not have validated EVMS (JPL is

exception).• Implementation within the Projects not being

institutionalized within the Centers. • Value of the data is inconsistent to the Managers.

11


Common Framework

Risk Management (RM):• Inconsistent use of RM as part of the decision making

process across the Centers.– Some use to allocate reserve for risk mitigation.– Strongest CMCs include Risk reporting and have strong

discussions at monthly CMCs. (GSFC, MSFC, JPL, LaRC)

• Use of IRMA on CxP is a strength. – Possible false confidence that risks are being reported.– Focus is on identifying and tracking, rather than managing

the risks by working levels.– Perspective is different from working level to Project level

(i.e. rating).– Separate risk systems maintained to use the risks to

manage.

12


Program and Project Review

Implementation Issues: • Direction vs Recommendation• Loss of understanding that SRB is

meeting objectives of TA (OCE/Centers), MD, and AA (PA&E).

Acceptance of SRB requirements.

13


Technical Authority

Two Implementations on the Projects:1) Mission/Lead System Engineer is the TA-Eng and

independently funded. (GSFC & MSFC)• Performs TA as a collaborative effort with the Systems

Engineers on the Project

2) Chief Engineer is the TA-Eng and independently funded.

• Focus of TA is strongly on the independent funding and requirements ownership (technical voice not prevalent)

• Performs special studies• Independent of Systems Engineering functions of the

Project/Program• Defines specific roles between TA and SE (potential to

have a gap)

14


Technical Authority

• TA-Software responsibilities are not widely understood.

• TA Implementation Plans require updating– Most Centers addressed only TA-Eng– Surveys prompting updates to include SMA and

HMTA• HMTA Infrastructure is limited to JSC.

– Infrastructure that can be leveraged does exist– Training/awareness needs to be increased

• Non-Conformance: JPL is submitting a waiver to HQ for independently funded TA-SMA.

15


Waivers/Dissenting Opinions

• Waiver Process at the Centers is not clearly understood by Projects and Programs– Weaknesses exist in content for waivers within Center

documentation (inclusion of rationale)

• Dissenting Opinions Process– Some Centers have a documented process within specific

areas (i.e. Engineering)– No Center has a documented process that covers all

possible dissenting opinions (SMA, Eng, procurement, programmatic, etc.)

16


Technical Standards/Lessons Learned

• Tech standards:– Overall implementation is good– Documentation of the working process

• Lessons Learned:– Overall the requirements of the NPR are

being met.– Personnel get the lessons they need for

their next Program/Project through informal means.

17


Software Engineering Management

• Overall implementation strengths exist at the Centers.

• Policy is being taken seriously.• NASA workforce is working to assure

requirements are flowed down to Projects.

18


Software Engineering ManagementNon-Compliances:• KSC: Risk Analysis and Waiver approved for Ares 1-X GS• JSC: SW Engineering Improvement Plan is out of date-working.

Class A s/w being developed by organizations not having CMMI level 2 rating or higher - (in-house CMMI corrected, contractor levels are being tracked and reported).

• LaRC: Required 7150.2 Compliance matrix not completed (corrected)Firmware development is proceeding without applicable

s/w requirements.• HQ/JPL: NPR 7150.2 requirements are not flowed to the Contract

Meet the intent and have the ability to apply at the task level.• ARC: SW Engineering Improvement Plan does not exist.• GRC: NPR 7150.2 requirements are not flowed to some contracts and

procurement efforts.Class A software development with no current CMMI rating in the required process areas performed by the Center.

19


7120.5E Suggestions

• Address all requirements, but allow “Tailoring” of the approach to implementation of the requirements consistent with Program/Project characteristics such as scope, complexity, visibility, cost, safety, and acceptable risk of a project.

• Address how requirements are determined to be n/a for work at a Center (elements within the Project).

• Training Requirement on Project Management teams to have annual training.

• Collecting Lessons learned as go through the life cycle does not take priority.

• Inclusion of context within policy is viewed as positive.

20


7120.5E Suggestions

• TA-Eng for Software (3.4.1.1 including TA for Software as described in NPR 7150.2)

• TA implementation focus is on the independent funding of the technical authority and waiving the technical requirements at the Project level (good).– Add language that an additional role of the independently

funded TA is to be the voice of working level engineers that have not been designated as TA.

• Include table to describe and point to different classifications (7120.5, 8705.4, 7150.2, etc) and context for the need of the classifications.

21


7120.5E Suggestions

SRB Implementation Issues:• Implementation of SRBs is not clear to Projects that all

convening authorities objectives are being met .• SRBs act as a decisional body rather than as a recommendation

body. (Identified as an implementation issue, not a policy issue)• Inconsistent implementation of SRB process (personality

dependent).• Timeframe to complete the report-out process is too long.

– Policy change to allow parallel activities– Describe process to authorize continuation into the next phase

while the road to the KDP continues.

22


Additional OCE Actions

• Policy-related letters/correspondence to be attached to the affected NPR/NPD(s) within NODIS

• Use of SMART Id’s on OCE Policy to assist Centers with requirements flowdown

• Blanket waiver for Class D missions

23


Topic for Discussion

ISSUE: Class D Payload classification and implementation of Class B Software requirements in a resource constrained environment.

1. Comply2. Ctr level waivers project by project for delegated requirements

(80+% of NPR 7150.2) and implement remaining non-delegated requirements

3. Ctr level waivers project by project for delegated requirements (80+% of NPR 7150.2) and submit waiver to HQ for relief on remaining requirements.

4. Submit blanket waiver to HQ to implement alternate Class B Software requirements that do not meet or exceed the requirements of NPR 7150.2.• Describe the Center process to identify, mitigate,

communicate and accept project risk associated with the implementation approach proposed by the Center.

24


Generic/Blanket Tech Authority Waiver capabilityin NPR 7150.2A*, Chapter 6

“6.1.1 For those cases in which a Center or project desires a general exclusion from requirement(s) in this NPR or desires to generically apply specific alternate requirements that do not meet or exceed the requirements of this NPR, the requester shall submit a waiver for those exclusions or alternate requirements for approval by the NASA Headquarters Chief Engineer with appropriate justification. [SWE-120]”

“Note: This type of waiver (which is approved by the NASA Headquarters Chief Engineer) is for generic/blanket relief from a requirement for a Center, Center organization, or multiple projects over an extended time. Generic/blanket waivers are not to be confused with normal waivers that address relief from a requirement on a single project or in a specific instance (which can be approved at the Center level if so specified in last column of Appendix D).”

* This document has passed NODIS review an is in the Administrator’s suite for signature

25


Research and Technology

• Definition of roles and responsibilities for implementation of Technical Authority in R&T Programs and Projects is needed by HQ. (ARMD TA, Program TA, Project TA, PI, and Center TAs)

• Roles and responsibilities of the Center TAs need to be clearly defined in Center documentation.

• R&T Project Management Structure is complex from a communication perspective.

• Checks and Balances developing within the organization and Project specific to the R&T management structure.

26


Topic for Discussion

• Transition from R&T project (NPR 7120.8) into a flight project (NPR 7120.5). – Clarity required– Implementation of NPR 7123.1 requirements– Implementation of SMA requirements

• ARC and GRC have working processes for projects that transition from R&T to flight.

• OCE/MDs evaluation of NPR 7120.8 and NPR 7123.1 for potential clarifications

27


Back Up Charts

28


Survey/Audit Differences

• Higher level of documentation review• Focus on implementation of OCE

requirements• Fewer interviews conducted• Response to findings is different• Actively looking for Center feedback on areas

where Agency policy and requirements should be modified

29


Survey Ground Rules

1. The basis for the OCE Survey is NPR 7120.5D - NASA Space Flight Program and Project Management Requirements, NPD

2820.1 - NASA Software Policy, NPD 8070.6 – Technical Standards, NPR 7120.6 - Lessons Learned Process and NPR

7150.2, NASA Software Engineering Requirements. NPR 7120.8 – NASA Research and Technology Program and

Project Management Requirements.2. The Survey will be conducted using the OCE Requirements

Survey Questions checklist developed for the Center. 3. The Center interface point of contact assists the Survey

Team in obtaining documentation and establishing interview schedules for the areas being surveyed.

30


Survey Ground Rules (cont.)

4. The OCE Survey is being conducted in coordination with the CxPSA Level 2 Audit. Some Survey areas (such as TA-SMA, Risk

Management, Design documentation, etc.) also have attributes which are within the scope of the CxP Audit Team. In these areas, the Audit teams will also review the OCE Survey attributes. The

OCIO Records Management Assessment is also coordinated with the Survey activities. The integrated effort is taken to ensure

complete coverage of all OCE requirements without duplication of reviews by multiple teams.

5. Interviews will be conducted. Designated Program/Project managers, technical authorities, records managers, configuration

managers, institutional directives manager, procurement acquisition managers and software managers. Additional

interviews may be conducted based on issues found during the Survey and will be arranged by the Center interface point of

contact.

31



6. The Survey Team (in conjunction with the Audit/Assessment Teams) will conduct a brief meeting on Tuesday and

Wednesday afternoon. The team members will discuss their findings from that day, their planned activities for the

following day and any support needed from the Center. Survey efforts may be refocused or additional areas may be added at the daily meeting based on findings generated by the team members. The Center interface point of contact is

requested to attend the daily team meeting to maintain close coordination with the Survey Team.

7. The Survey Team will conduct a meeting as early in the day on Thursday to discuss the findings with the Center

interface point of contact and any other Center personnel invited by the POC.

32



8. OCE Survey Manager will concur with all survey findings to ensure consistent interpretation of

requirements.

9. An Out-brief will be held on Friday for OCE, SMA, OCHMO and Center personnel marking the close

of the on-site Survey.

10. The Center POC is provided access to the detailed dBase (SAARIS) of findings from the

Survey.

11. OCE will work with the Center POC to ensure understanding of the findings and the responses.