Reliability Assurance Initiative

(RAI) ReliabilityFirst Activity

1

Bob Wargo Director – Analytics & Enforcement

July 10, 2013

Reliability Assurance Initiative – Concept

A Value-Centered Approach using a risk/performance assessment to shape and align activities within a thoughtful, purposed framework that proactively promotes positive reliability outcomes Simply Put Mitigate risks BEFORE they become violations

2

Participation

ReliabilityFirst is an active participant with other Regions and NERC in the development of the RAI

SERC, MRO & ReliabilityFirst are working on different aspects of the Internal Control appraisal effort of the RAI

ReliabilityFirst is focused on developing the attribute framework of Internal Controls. i.e., how a Registered Entity’s Internal Control Program will be interpreted for the purposes of compliance monitoring and enforcement decision-making

3

Areas of Regional Focus to Date

4

Inherent Risk Assessment

Internal Control Attribute Framework

Appraisal of Registered Entity Internal Controls

Analysis of Internal Control Artifacts

Monitoring & Non-Compliance Decision Making

Audit Scope & Frequency

Disposition of Non-Compliances

MRO ReliabilityFirst

Approach

5

Regions have examined 1000’s of entity organizations, programs and structures and analyzed 1000’s of violations and has identified core competency areas where weaknesses have lead to non-compliances and reliability risks. Many Registered Entities, have successfully implemented internal measures that have led to effective, value added, reliability focused, compliance programs. PJM, AEP, PPL and IMPA (as a Small Entity representative) Are working with ReliabilityFirst in the development of the attribute framework for the appraisal of Internal Controls.

To Assure Scalability!!

Partnership with Registered Entities on “RAI” Type Activities

ReliabilityFirst efforts with Registered Entities before RAI included:

• Assist Visits • Failure Mode Effects Analysis (FMEA) to Identify,

Quantify and Prioritize Risks • Use of Maturity Models to Assess Performance

Capabilities in Critical Areas • Risk-Harm Analytics Training • Root Cause Analysis

As the RAI effort began, ReliabilityFirst sought to build off of those experiences and leverage the cooperative spirit of those interactions.

6

Ground Rules for the ReliabilityFirst RAI Project

“Thinking Out Loud” is encouraged – No Knee Jerk Reactions to Ideas! The end product has to work and meet the needs for everyone involved – Regions & Registered Entities. The “RAI” has to have an “ROI” for Registered Entities. The end product has to be efficient, effective, have a sound analytical basis and crystal clarity on criteria and procedure. The end product should encourage, enable and reward continuous improvement.

7

ReliabilityFirst Policy on Appraising Internal Controls

ReliabilityFirst has no intention to mandate a certain control methodology or practice. Rather, ReliabilityFirst is committed to be knowledgeable and competent in the field of assessing whatever internal control strategy an entity has chosen or chooses to use. ReliabilityFirst further commits to be transparent in identifying the attributes that will be used in assessing the strength of any particular internal control program an entity might have (see a partial list of attributes on next slide).

8

Partial List of Global Attributes Used for Appraising Internal Controls

Are the instituted internal controls the result of a careful approach that considered the likely possible failure modes? What is the likelihood that the internal controls will be effective in preventing a non-compliance? What is the likelihood that the internal controls will be effective in timely detecting a non-compliance? And finally, given the standard/requirement and the entity, what is the likely harm a non-compliance might have if it occurs and remains undetected? (Tells ReliabilityFirst how “hard” to grade/look at the above questions) 9

Leveraging Existing Internal Control/Performance Models for the Purpose of Constructing an Attribute List

10

Six models were selected as primary “feed stock” from over 40 candidates1

1. See “Assessing Organizational Capabilities: Reviewing and Guiding the Development of Maturity Grids” by Anja M. Maier, James Moultrie, and P. John Clarkson, IEEE Transaction on Engineering Management, Vol. 59, No. 1, February 2012

11

Opportunity: Utilize Existing Model Resources as a Source for Development of Model for the Appraisal of Internal

Controls

ReliabilityFirst INCOSE RMM CMMI SGMM ES-C2M2 INPO MRO Critical Process Areas V3.2.2 Oct

2011 CERT-RMM DEV May-05 Attributes

RM ‐ Risk Management RM ‐ Risk Management

RISK - Risk Management

CAR - Causal Analysis and Resolution RISK - Risk Management PI.2 Corrective Action Corrective Action

Program

MON - Monitoring RSKM - Risk Management

Internal Controls and Risk

Prioritization

VAR - Vulnerability Analysis and Resolution

THREAT - Threat and Vulnerability Management

Example of rolling up the domain/process areas of the

various models into 1 Process Area

27 Possible Core Competency

Areas Were Identified

12

Recognize Distribution of Core Competencies Demanded by the Reliability Standards

STD DESC

EXDM

– External Dependencies M

anagement

SUP – Supply

AC

Q ‐ A

cquisition

PP ‐ Project Planning

WM

‐ Work M

anagement

DM

‐ Decision M

anagement

RM

‐ Risk M

anagement

CM

‐ Configuration M

anagement

INFO

M ‐ Inform

ation Managem

ent

MEA

S ‐ Measurem

ent

SRD

‐ Stakeholder Requirem

ents Definition

RA

‐ Requirem

ents Analysis

AD

‐ Architectural D

esign

IMPL ‐ Im

plementation

INT ‐ Integration

VER ‐ Verification

TRA

N ‐ Transition

VAL ‐ Validation

OPER

‐ Operation

MA

INT ‐ M

aintenance

DISP ‐ D

isposal

LCM

M – Life C

ycle Model M

anagement

INFR

AM

‐ Infrastructure Managem

ent

PPM – Project Portfolio M

anagement

WFM

– Workforce M

anagement

QM

‐ Quality M

anagement

TLR – Tailoring.

Count:

12%

0%

0%

25%

0%

0%

4%

12%

65%

36%

3%

0%

0%

0%

1%

16%

0%

12%

47%

10%

1%

2%

0%

0%

7%

0%

0%

BAL-001 Real Power Balancing Control Performance 1 1 BAL-002 Disturbance Control Performance 1

BAL-002-WECC Contingency Reserve (WECC) 1 BAL-003 Frequency Response and Bias 1 1 BAL-004 Time Error Correction 1 1

BAL-004-WECC Automatic Time Error Correction (WECC) 1 BAL-005 Automatic Generation Control 1 1 1 BAL-006 Inadvertent Interchange 1 1 1

BAL-502-RFC Planned Resource Adequacy Assessment (RFC) 1 1 1 BAL-STD Operating Reserves (WECC) 1 COM-001 Telecommunications 1 1 1 1 COM-002 Communications and Coordination 1 1 1 1 CIP-001 Sabotage Reporting 1 1 1

Example Spreadsheet Of How Core Competency Activities are Connected to

Standards

Core Competency Areas Recognition Example

FAC-008-3

13

R6. Each Transmission Owner and Generator Owner shall have Facility Ratings for its solely and jointly owned Facilities that are consistent with the associated Facility Ratings methodology or documentation for determining its Facility Ratings. R7. Each Generator Owner shall provide Facility Ratings (for its solely and jointly owned Facilities that are existing Facilities, new Facilities, modifications to existing Facilities and re-ratings of existing Facilities) to its associated Reliability Coordinator(s), Planning Coordinator(s), Transmission Planner(s), Transmission Owner(s) and Transmission Operator(s) as scheduled by such requesting entities.

Verification Making Sure They Rated the Thing Right, Used

the Calculations Properly

Information Management

Root Cause Distribution By Core Competency Area for ReliabilityFirst Violations

14

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

645 Violations (Most Recent)

Internal Control Process Areas

15

ReliabilityFirst

Critical Process Areas EXDM - External Dependencies Management

PP ‐ Project Planning

WM - Work Management

DM ‐ Decision Management

RM ‐ Risk Management

CM ‐ Configuration Management

INFOM ‐ Information Management

MEAS ‐ Measurement

IMPL ‐ Implementation

INT ‐ Integration

VER ‐ Verification

VAL ‐ Validation

OPER ‐ Operation

MAINT ‐ Maintenance

WFM – Workforce Management

QM ‐ Quality Management

100% of the

Reliability Standards Are Covered by

these 16 Areas of Core Competencies

98% of the Violations

Analyzed for Root Cause had Weaknesses in 1 or more

these 16 Areas as a Contributor To the Violation

Selected 16 out of the 27

Possible Core Competency

Areas

Example: Mapping COSO to Selected Core Competency Areas

16

Objectives & Components from COSO Internal Control Framework

ReliabilityFirst

Critical Process Areas EXDM - External Dependencies Management

PP ‐ Project Planning

WM - Work Management

DM ‐ Decision Management

RM ‐ Risk Management

CM ‐ Configuration Management

INFOM ‐ Information Management

MEAS ‐ Measurement

IMPL ‐ Implementation

INT ‐ Integration

VER ‐ Verification

VAL ‐ Validation

OPER ‐ Operation

MAINT ‐ Maintenance

WFM – Workforce Management

QM ‐ Quality Management

Looking at the “Big Stuff” Continues to Influence Positive Performance in the “Small Stuff”

17

ReliabilityFirst Critical Process Areas

EXDM - External Dependencies Management

PP ‐ Project Planning

WM - Work Management

DM ‐ Decision Management

RM ‐ Risk Management

CM ‐ Configuration Management

INFOM ‐ Information Management

MEAS ‐ Measurement

IMPL ‐ Implementation

INT ‐ Integration

VER ‐ Verification

VAL ‐ Validation

OPER ‐ Operation

MAINT ‐ Maintenance

WFM – Workforce Management

QM ‐ Quality Management

Increasing Impact

Low Impact/Risk Standards for Entity “A”

High Impact/Risk

Standards for Entity “A”

Capability in Critical Process Areas Influences Entity’s

Performance in the Highest Impact

Standards/Requirements

But Since the Same Type of Activities or Processes are

Common throughout ALL the Standards, the Regions Will Have Insight into an Entity’s Capability

Throughout the Pyramid!!!

Grid Reliability Improvement and Performance Model (GRIPM)

Next Phase Completion By August 7, 2013

A Multi-Regional/Multi-Registered Entity Team will: 1. Develop abstracts for all 16 Process Areas.

2. Develop the Configuration Management Process Area

fully with Expected Components (or attributes) list that would be expected in a program to achieve that target competency.

3. Develop the Configuration Management Process Area Informative (or library of examples) components list to aid the Registered Entity.

Grid Reliability Improvement and Performance Model (GRIPM)

Next Phase Completion By August 7, 2013 (cont.) 4. Develop the overall Capability Appraisal Method to be

used to illicit artifacts from Registered Entities (borrow from MRO and other sources).

5. Develop first iteration of the Audit Scope List generated based on entity risk and capabilities in critical process areas.

6. Start Test Appraisal of Entity in Configuration Management Process Area.

Note: Educational Materials will be compiled as the project unfolds so it is not an after thought.

20

Strategic Awareness

Quality Assurance

Tactical Execution

Knowledge Coordination

Maturity Level 5 Project Planning

Maturity Level 4 Decision Making Quality Management External Dependencies Integration

Maturity Level 3 Risk Management Verification Validation

Work Management Workforce Management Implementation

Maturity Level 2 Measurement Operations Maintenance

Configuration Management Information Management

Maturity Level 1

An Example of How this Might Work Achieving Capability in Process Areas Leading to Higher Maturity Levels and

Reduced Monitoring and Favorable Non-Compliance Treatment

Need to achieve capability in

each Process on a Level to achieve that

Maturity Level

No Internal Controls = Full Audit on Regular

Schedule

Achieving Level 2 Allows for Some

Reduction in Audit Scope

Achieving Level 3 Allows for Substantial

Reduction in Audit Scope

Achieving Level 4 Allows for Substantial

Reduction in Audit Scope & Less

Frequent Monitoring

More Likely to “Decline to Pursue” A Non-Compliance